Handling website errors with selenium python

Question

I am scraping a website with selenium and send an alert, if something specific happens. Generally, my code works fine, but sometimes the website doesn't load the elements or the website has an error message like: "Sorry, something went wrong! Please refresh the page and try again!" Both times, my script waits until elements are loaded, but they don't and then my program doesn't do anything. I usually use requests and Beautifulsoup for web scraping, so I am not that familiar with selenium and I am not sure how to handle these errors, because my code doesn't send an error message and just waits, until the elements load, which will likely never happen. If I manually refresh the page, the program continues to work. My idea would be something like: If it takes more than 10 seconds to load, refresh the page and try again. My code looks somewhat like this:

def get_data():
    data_list = []
    while len(data_list) < 3:
        try:
            data = driver.find_elements_by_class_name('text-color-main-secondary.text-sm.font-bold.text-left')
            count = len(data)
            data_list.append(data)
            driver.implicitly_wait(2)
            time.sleep(.05)
            driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
            WebDriverWait(driver, 3).until(EC.visibility_of_element_located((By.CLASS_NAME,
                                                                         'text-color-main-secondary.text-sm.font-bold.text-left'.format(
                                                                             str(
                                                                                 count + 1)))))
        except TimeoutException:
            break

    text = []
    elements = []
    for i in range(len(data_list)):
        for j in range(len(data_list[i])):
            t = data_list[i][j].text
            elements.append(data_list[i][j])
            for word in t.split():
                if '#' in word:
                    text.append(word)
    return text, elements


option = webdriver.ChromeOptions()
option.add_extension('')
path = ''
driver = webdriver.Chrome(executable_path=path, options=option)
driver.get('')
login(passphrase)
driver.switch_to.window(driver.window_handles[0])
while True:
    try:
        infos, elements = get_data()
        data, message = check_data(infos, elements)
        if data:
            send_alert(message)
        time.sleep(600)
        driver.refresh()
    except Exception as e:
        exception_type, exception_object, exception_traceback = sys.exc_info()
        line_number = exception_traceback.tb_lineno
        print("an exception occured - {}".format(e) + " in line: " + str(line_number))

Darkknight · Accepted Answer · 2021-10-14 06:29:26Z

1

You can use try and except to overcome this problem. First, let's locate the element with a 10s waiting time if the element is not presented you can refresh the page. here is the basic version of the code

try:
  # wait for 10s to load element if it did not load then it will redirect to except block
  WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CLASS_NAME,'text-color-main-secondary.text-sm.font-bold.text-left'.format(str(count + 1)))))
except:
  driver.refresh()
  # locate the elemnt here again

answered Oct 14, 2021 at 6:29

Darkknight

1,87613 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

questionmaster69 Over a year ago

Thank you very much for your answer! Now that I look at it, I could have figured it out myself haha

Collectives™ on Stack Overflow

Handling website errors with selenium python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related