How to collect data python using selenium geckodriver

Question

I tried to scrap data from website using selenium firefox and geckodriver. I want to navigate from one page to another and extract informations. I am unable to move from current page to another, and force driver to back to the original page and move to next element in the list. I created the code bellow which click on first element and go to the specific page to collect data. Thank you

   binary = FirefoxBinary('/usr/bin/firefox')
    driver = webdriver.Firefox(firefox_binary=binary, executable_path=r'/home/twitter/geckodriver')
    try:
        driver.get('https://www..........')  
        list_element = driver.find_elements_by_xpath("//span[@class='icon icon-email']")
for element in list_element :         
        x = driver.current_url
        element.click()
        time.sleep(5)
        for ex in driver.find_elements_by_xpath('//span[@class = "valeur"]'):
            print (ex.text)
        driver.back()
except Exception as e:
         print (e)
driver.quit()

What Exception are you getting? Does clicking the link open the link in a new tab/window? — Chuk Ultima
– Chuk Ultima, Commented Feb 26, 2018 at 9:52

Abdullah Ahmed Ghaznavi · Accepted Answer · 2018-02-26 09:52:30Z

2

This might happens because your driver/browser didn't get the new page (current page). add one line after element.click() or time.sleep(5) :

driver.switch_to.window(driver.current_window_handle)

then try to run your code again. Hope this helps you! :)

answered Feb 26, 2018 at 9:52

Abdullah Ahmed Ghaznavi

2,0953 gold badges19 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

undetected Selenium Over a year ago

Good answer. Try to avoid time.sleep(5) and suggest to induce WebDriverWait (+1)

Abdullah Ahmed Ghaznavi Over a year ago

Thankyou! Yes you are right WebDriverWait() is better option here.

Collectives™ on Stack Overflow

How to collect data python using selenium geckodriver

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related