Web Scraping by using "Next" button Python

Question

I am getting reviews for a webpage and need to scan each page until there are no longer any reviews left. Reviews page have multiple pages, and my first thought was to use a While loop, however, I am unsure where to start. HTML code of the web page looks something similar to this.

HTML Code in last page;

Any help is appreciated.

Show code you have tried so far. Also share HTML of the last page — Andersson
– Andersson, Commented Dec 25, 2016 at 11:45
Check if URL has page as a parameter. If it does, you don't need to use next button. And for parsing html I would suggest BeautifulSoup — Yevhen Kuzmovych
– Yevhen Kuzmovych, Commented Dec 25, 2016 at 11:52
URL doesn't have a page parameter. I couldn't come up with something for looping. Any code suggestions are very welcomed. @Andersson — AlpU
– AlpU, Commented Dec 25, 2016 at 11:59
On the very last page next button does not exist, therefore no HTML code for that. I just edited my question and added the html code for last page. @Andersson — AlpU
– AlpU, Commented Dec 25, 2016 at 12:13

Andersson · Accepted Answer · 2016-12-25 20:17:30Z

5

Try to click Next while it can be found:

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException

driver = webdriver.Firefox()
driver.get("http://www.some_site.com")
while True:
    # do whatever you want
    try:
        driver.find_element_by_xpath('//a/span[text()="Next"]').click()
    except NoSuchElementException:
        break

edited Dec 25, 2016 at 20:17

answered Dec 25, 2016 at 12:19

Andersson

52.8k18 gold badges83 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

AlpU Over a year ago

I am getting this error: NameError: global name 'driver' is not defined. I did not use Selenium before, my code uses urllib2 to request and BeautifulSoup to parse. @Andersson

Andersson Over a year ago

Answer updated. If you use http requests to get page HTML source, why do you need selenium at all?

AlpU Over a year ago

in order to use the html code browser-like, to click on next buttons. I could not find any other direct way to do this than Selenium.

Andersson Over a year ago

Does my answer was helpful? Did you get rid of exception?

Collectives™ on Stack Overflow

Web Scraping by using "Next" button Python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related