0
url_list = ['www.scrape.com/file1', 'www.scrape.com/file2', ''www.scrape.com/file3'] 

category_id_list = ['12345','abcde','ABCDE']

zip_list = ['10075','10017','10028']

I have three variables I use to create a URL to be requested. in the order: url_list+zip+categoryid

the url is then passed into a function which has the scrape code

I was using 3 for loops to iterate over these lists but that is highly redundant

for url_ in url_list:
   for category_id in category_id_list:
       for zip_ in zip_list:

           request_url = url_+category_zip_
           func(request_url)

This does the job, but is there a more optimal way to do it? Thank you!

3

3 Answers 3

3

You may use itertools.product

import itertools

for url in (str.join("",url) for url in itertools.product(url_list,category_id_list,zip_list)):
    func(url)
Sign up to request clarification or add additional context in comments.

Comments

0

This might be a bit late, but this is the way I did it:

cats = ["a","b","c","d"]

zips = ["25320","53902","59607","53123"]

base = "https://example.com"

for i in range(4):
   url = "{}/{}/{}".format(base, cats[i], zips[i])
   print(url)

Output:

https://example.com/a/25320
https://example.com/b/53902
https://example.com/c/59607
https://example.com/d/53123

Comments

0

One way to avoid writing multiple for loops is using zip.It allows you to access ith element from each list at once. So you can do something like:

url_list = ['www.scrape.com/file1', 'www.scrape.com/file2', 'www.scrape.com/file3']
category_id_list = ['12345','abcde','ABCDE']
zip_list = ['10075','10017','10028']

for url, id, zip in zip(url_list, category_id_list, zip_list):
    request_url = url + id + zip
    func(request_url)

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.