Selenium Timeout Exception in Python

Question

I have the following function:

def get_info(url):
    options = webdriver.ChromeOptions()
    options.headless = True
    chrome_browser = webdriver.Chrome('./chromedriver', chrome_options=options)
    chrome_browser.get(url)
    name = chrome_browser.find_element_by_xpath("//h1[contains(@class,'text-center medium-text-left')]").text
    winter = WebDriverWait(chrome_browser, 10).until(
        EC.presence_of_element_located((By.XPATH, '//*[@id="main-content"]/div[1]/div[1]/div/div[2]/div[5]/div['
                                                  '2]/div/div[1]/div[3]/div')))
    chrome_browser.quit()
    return winter, name

I want to get the width percentage from the winter/spring/summer etc charts on this site: https://www.fragrantica.com/perfume/Christian-Dior/Sauvage-Eau-de-Parfum-48100.html

So I want this function to return the name of the fragrance and the winter row on the HTML page. The season ratings seem to load a bit slower on the site so I've tried to add wait until the HTML row appears. When I click inspect on the winter ratings chart I am given this element:

<div style="border-radius: 0.2rem; height: 0.3rem; background: rgb(120, 214, 240); width: 90.3491%; opacity: 1;"></div>

Firstly, BeautifulSoup did not find it so I tried Selenium. Selenium does not find it, and when using the WebDriverWait it just shows me this error:

Traceback (most recent call last):
  File "D:/Fragrance selector/main.py", line 16, in <module>
    s = info.get_info('https://www.fragrantica.com/perfume/Christian-Dior/Sauvage-Eau-de-Parfum-48100.html')
  File "D:\Fragrance selector\fragrance_info_from_net.py", line 24, in get_info
    winter = WebDriverWait(chrome_browser, 10).until(
  File "D:\Fragrance selector\venv\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

I am genuinely out of ideas with this problem. I have no more ideas on how to get that width percentage from the ratings. I would really appreciate it if some of you could help me figure this out.

Maybe load the page in your browser, save it to a file and see if you can use the same logic with BeautifulSoup4. It might be as simple as the XPath being wrong. Also, use a try ... except statement to try your WebDriverWait in, so you can handle the exception in your program when it occurs. — VirtualScooter
– VirtualScooter, Commented Feb 15, 2021 at 19:44

Ryan Burch · Accepted Answer · 2021-02-16 16:14:09Z

2

I modified the wait in your WebDriverWait to 30 seconds and added a get_attribute to return the correct data. See below for an example:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.wait import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument('--no-sandbox')
options.add_argument('--start-maximized')
options.add_argument('--disable-extensions')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--ignore-certificate-errors')
service = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=service, options=options)
driver.get('https://www.fragrantica.com/perfume/Christian-Dior/Sauvage-Eau-de-Parfum-48100.html')
name = driver.find_element(By.XPATH, "//h1[contains(@class,'text-center medium-text-left')]").text
winter = WebDriverWait(driver, 30).until(ec.presence_of_element_located((By.XPATH, '//*[@id="main-content"]/div[1]/div[1]/div/div[2]/div[4]/div[2]/div/div[1]/div[3]/div/div')))
t = winter.get_attribute('style')
driver.quit()

To get all of the ratings (Faces and Seasons) into a list, please use the following code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.wait import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options

new = []
options = Options()
options.add_argument('--no-sandbox')
options.add_argument('--start-maximized')
options.add_argument('--disable-extensions')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--ignore-certificate-errors')
service = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=service, options=options)
driver.get('https://www.fragrantica.com/perfume/Christian-Dior/Sauvage-Eau-de-Parfum-48100.html')
name = driver.find_element(By.XPATH, "//h1[contains(@class,'text-center medium-text-left')]").text
WebDriverWait(driver, 30).until(ec.presence_of_element_located((By.XPATH, '//*[@id="main-content"]/div[1]/div[1]/div/div[2]/div[4]/div[2]/div/div[1]/div[3]/div/div')))
ratings = driver.find_elements(By.XPATH, './/div[@style="width: 100%; height: 0.3rem; border-radius: 0.2rem; background: rgba(204, 224, 239, 0.4);"]')
for style in ratings:
    new.append(style.find_element(By.TAG_NAME, 'div').get_attribute('style'))


print(new)
driver.quit()

This will give an ordered list (starting with the Faces, then moving to the Seasons) of the % which is shown by the bar.

edited Feb 16, 2021 at 16:14

answered Feb 15, 2021 at 19:45

Ryan Burch

5307 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Alexandru - Gabriel Badea Over a year ago

So I've tried copy pasting your solution and i get this error: driver = webdriver.Chrome(service=service, options=options) TypeError: __init__() got an unexpected keyword argument 'service'

Alexandru - Gabriel Badea Over a year ago

okay so i deleted Service from ChromeDriver manager and it seems to work now. Thank you

Alexandru - Gabriel Badea Over a year ago

hey since I want to get all of the seasons I've added after winter this: spring = WebDriverWait(driver, 30).until(ec.presence_of_element_located((By.XPATH, <spring element path>))). But I get a selenium.common.exceptions.TimeoutException. Do you have any idea how to go around this?

Ryan Burch Over a year ago

@Alexandru-GabrielBadea see my edited answer.

Alexandru - Gabriel Badea Over a year ago

You, sir, are a savior. Thank you very much. I have learned a lot from your answers.

Collectives™ on Stack Overflow

Selenium Timeout Exception in Python

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related