PYTHON : Selenium loop selenium.common.exceptions.TimeoutException

Question

I'm trying to scrape product details from each individual product page. I try to append an integer in between xpath for 'elements'. The for i in range loop run without problem for j = 1, print getproductname but throw error for j = 2.

How do you fix the loop ? I couldn't figure out what seems to be the issue here :

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

CHROMEDRIVER_PATH = '/Users/reezalaq/PycharmProjects/wholesale/driver/chromedriver'
chrome_options = webdriver.ChromeOptions()
driver = webdriver.Chrome(CHROMEDRIVER_PATH, options=chrome_options)
chrome_options.accept_untrusted_certs = True
chrome_options.assume_untrusted_cert_issuer = True
chrome_options.headless = False

driver.get('https://www.skinnymixes.com/collections/skinny-syrups')
for i in range(1,20):
    j = str(i)
    print(j)
    elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, '#collection-content > div.collection-listing > div > div.row.product-list.text-center.mb-3.in-view.in-view--active.in-view--loaded > div:nth-child('+ j +') > a')))
    print(elements)
    for element in elements:
        url = element.get_attribute('href')
        print(url)
        #open new tab with specific url
        driver.execute_script("window.open('" +url +"');")
        #switch to new tab
        driver.switch_to.window(driver.window_handles[1])
        getproductname = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, '//*[@id="shopify-section-product"]/div/div[1]/div/div[2]/h1')))
        print(getproductname.text)

Traceback (most recent call last):
  File "/Users/reezalaq/PycharmProjects/legasimall/skinny/getproducts.py", line 70, in <module>
    elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, '#collection-content > div.collection-listing > div > div.row.product-list.text-center.mb-3.in-view.in-view--active.in-view--loaded > div:nth-child('+ j +') > a')))
  File "/Users/reezalaq/PycharmProjects/legasimall/venv/lib/python3.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

Vikas Ojha · Accepted Answer · 2019-11-03 07:24:31Z

0

The problem with your code is that

it opens a product in a new window, switches driver to it and then it scrapes the information
but after that it is NOT switching the driver back to the original window.

So, basically at the end of the loop you need to switch it back to original window, i.e.,

either,

driver.switch_to.default_content()

or, save the original window handle before switching to new window and use that to switch back the driver to it.

answered Nov 3, 2019 at 7:24

Vikas Ojha

7,0006 gold badges24 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

PYTHON : Selenium loop selenium.common.exceptions.TimeoutException

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related