How to loop multiple page on selenium python BeautifulSoup

Question

Right now my this code able to click on every single product of this page "https://www.daraz.com.bd/audio/?page=1&spm=a2a0e.home.cate_2.2.49c74591NNpWDU%27" which taking me to product details page of each item. Can anybody tell me how I will loop multiple pages such as page2, page3, page4? here is my code

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from bs4 import BeautifulSoup

#argument for incognito Chrome
option = Options()
option.add_argument("--incognito")


browser = webdriver.Chrome(options=option)

browser.get("https://www.daraz.com.bd/audio/?page=1&spm=a2a0e.home.cate_2.2.49c74591NNpWDU%27")


# Wait 20 seconds for page to load
timeout = 20
try:
    WebDriverWait(browser, timeout).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='c16H9d']")))
except TimeoutException:
    print("Timed out waiting for page to load")
    browser.quit()


soup = BeautifulSoup(browser.page_source, "html.parser")

product_items = soup.find_all("div", attrs={"data-qa-locator": "product-item"})
for item in product_items:
    item_url = f"https:{item.find('a')['href']}"
    print(item_url)

    browser.get(item_url)

    item_soup = BeautifulSoup(browser.page_source, "html.parser")
    # Use the item_soup to find details about the item from its url.
    container = item_soup.find_all("div",attrs={"id":"container"})
    for items in container:
        title = items.find("div",{"class":"pdp-product-title"})
        print(title)



browser.quit()

now it getting information only from page1. I want it will also collect information from other pages such as page2,page3,page4,page5

Each page has a pagination object to take you to the previous/next page, etc. — Reti43
– Reti43, Commented Mar 19, 2020 at 20:43
how I will do it? please explain with code if possible. Thanks for your comment — Farhan Ahmed
– Farhan Ahmed, Commented Mar 19, 2020 at 22:44
@FarhanAhmed Stack Overflow is not a substitute for guides, tutorials or documentation. — AMC
– AMC, Commented Mar 20, 2020 at 22:37

Kurosh Ghanizadeh · Accepted Answer · 2020-03-20 15:40:51Z

1

as you can see in website when you click on page 2 in the URL the only thing that changes is page number so this can be easily done by loops. for a better code you can make a variable called url and change that for each page:

for page_num in range(1, 10): # change the range as you want to
    url = "https://www.daraz.com.bd/audio/?page={}&spm=a2a0e.home.cate_2.2.49c74591NNpWDU%27".format(page_num)

and put the rest of code inside this loop (except for the last line)

answered Mar 20, 2020 at 15:40

Kurosh Ghanizadeh

5791 gold badge4 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Farhan Ahmed Over a year ago

Thanks Kurosh Ghanizadeh. It worked. Can please tell me how to get output in a csv file? what code I need to be write to get output?

Kurosh Ghanizadeh Over a year ago

I haven't worked too much with CSV so I suggest just read some documentation. It's not a complicated thing you can work with it easily

Collectives™ on Stack Overflow

How to loop multiple page on selenium python BeautifulSoup

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related