How to make Selenium script scrape additional urls after either end script or error?

Question

Im new to coding! I found a large part of this script in a github that seems inactive now. Basically this script is scraping reviews of only one Tripadvisor restaurant. Depending on number of pages (same restaurant) it has to scrape the script either finishes normally or gives the following error when there is no more next button:

selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: .//a[@class="nav next ui_button primary"]

I would like the script to continue on to another URL in case it is finished with first restaurant or gets error for no next button. Help me guys!

import sys
import csv
from selenium import webdriver
import time
path_to_file = "entirereview.csv"
num_page = 12
url = "https://www.tripadvisor.com/Restaurant_Review-g187147-d23117361-Reviews-Ya_Bayte-Paris_Ile_de_France.html"
if (len(sys.argv) == 4):
    path_to_file = sys.argv[1]
    num_page = int(sys.argv[2])
    url = sys.argv[3]
driver = webdriver.Firefox()
driver.get(url)
driver.find_element_by_id('_evidon-accept-button').click()
driver.find_element_by_id('filters_detail_language_filterLang_ALL').click()
csvFile = open(path_to_file, 'a', encoding="utf-8")
csvWriter = csv.writer(csvFile)
for i in range(0, num_page):
    time.sleep(3)
    driver.find_element_by_xpath("//span[@class='taLnk ulBlueLinks']").click()
    time.sleep(3)
    container = driver.find_elements_by_xpath(".//div[@class='review-container']")
    for j in range(len(container)):
        title = container[j].find_element_by_xpath(".//span[@class='noQuotes']").text
        date = container[j].find_element_by_xpath(".//span[contains(@class, 'ratingDate')]").get_attribute("title")
        rating = container[j].find_element_by_xpath(".//span[contains(@class, 'ui_bubble_rating bubble_')]").get_attribute("class").split("_")[3]
        review = container[j].find_element_by_xpath(".//p[@class='partial_entry']").text.replace("\n", " ")
        csvWriter.writerow([date, rating, title, review])
    driver.find_element_by_xpath('.//a[@class="nav next ui_button primary"]').click()
driver.close()

if you get error then you should use try/except to catch it. — furas
– furas, Commented Jun 16, 2021 at 20:37

C. Peck · Accepted Answer · 2021-06-17 22:18:43Z

1

You can surround it with try - except if you want to continue gracefully after that error.

try:
    driver.find_element_by_xpath('.//a[@class="nav next ui_button primary"]').click()
except NoSuchElementException:
    # do nothing here?

edited Jun 17, 2021 at 22:18

answered Jun 16, 2021 at 18:43

C. Peck

3,7193 gold badges21 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

matrimm Over a year ago

Thank you! this worked. However I kept getting errors for the "catch" saying its an invalid syntax. So I tried "except" after many different solutions I found of google, just did it on a whim... and it worked!

try:         driver.find_element_by_xpath('.//a[@class="nav next ui_button primary"]').click()     except NoSuchElementException:

Do you have any ideas on how I may make the script do multiple urls. Im searching the answer myelf but hints are always welcome :)

Collectives™ on Stack Overflow

How to make Selenium script scrape additional urls after either end script or error?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related