How to click on "next" button using selenium or scrapy in python

Question

I was trying to scrape some data from flipkart.com using scrapy. I got everything other than going to the next page. first I tried using scrapy followed by selenium. actually, a class has 2links for the previous page and for the next page.

with scrapy: I am unable to fetch any result. I am getting the blank output.

with selenium: whenever I try to click on the next page it works fine from page 1 to page 2. but then instead of going on the 3rd page, it's coming back to the second page.

is there any other way to do this? how should I distinguish the URLs for the previous page and the next page?

You can change the page number directly in the URL: https://www.flipkart.com/...&page=3. — Yevhen Kuzmovych
– Yevhen Kuzmovych, Commented Feb 9, 2021 at 17:48
I'd suggest changing the URL like @YevhenKuzmovych says (preferably using scrapy), since the xpath of elements might change everytime you click next. The xpath may also vary depending on the products category — Rohalt
– Rohalt, Commented Feb 9, 2021 at 17:52

Samsul Islam · Accepted Answer · 2021-05-30 16:03:03Z

1

just append [-1] so that driver only fetches the last occurring result for the given class/id for flipkart the code is as follows

driver.find_elements_by_xpath('//a[@class="_1LKTO3"]')[-1].click()

edited May 30, 2021 at 16:03

Samsul Islam

2,6192 gold badges20 silver badges24 bronze badges

answered May 30, 2021 at 15:49

amey

111 bronze badge

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to click on "next" button using selenium or scrapy in python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related