Trying to loop links with selenium in python

Question

I would like to loop a set of links with selenium in python. I have tried to follow this explanation with no success. I keep getting the "stale element reference error(I am trying to use WebDriverWait). my code is as follows:

list_of_links = mydriver.find_elements_by_xpath('//ul[@class="directory dir-col"]/li/a')
for link in list_of_links:
UI.WebDriverWait(mydriver, 30).until(lambda mydriver:mydriver.find_element_by_xpath('//ul[@class="directory dir-col"]/li/a'))        
link.click()
mydriver.back()

I did try placing the webdriver wait command before and after the click and back commands with no success. any help will be highly appreciated.

Richard · Accepted Answer · 2014-04-23 15:19:11Z

1

The issue is that once you leave the page, the elements in list_of_links become stale.

This approach should work for you, assuming each link has different text:

list_of_links = mydriver.find_elements_by_xpath('//ul[@class="directory dir-col"]/li/a')
list_of_linktext = []
for link in list_of_links:
    list_of_linktext.append(link.text)

for linktext in list_of_linktext:
    mydriver.find_element_by_link_text(linktext).click()
    mydriver.back()

answered Apr 23, 2014 at 15:19

Richard

9,0393 gold badges41 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

CodePro · Accepted Answer · 2014-04-23 21:08:34Z

0

Based on Richard idea I decided to loop on xPath values and not link text(which can be the same) the code I ended up with is:

import lxml.html as lh
import urllib2
from selenium import webdriver

htmlObject = lh.parse(urllib2.urlopen(start_url))
listOfPaths = htmlObject.xpath('//ul[@class="directory dir-col"]/li/a')
listOfLinkPathes = []
for link in listOfPaths:
    listOfLinkPathes.append(htmlObject.getpath(link))
for linkPath in listOfLinkPathes:
   mydriver.find_element_by_xpath(linkPath).click()
   mydriver.back()

answered Apr 23, 2014 at 21:08

CodePro

1591 silver badge12 bronze badges

Comments

seagulf · Accepted Answer · 2014-04-23 16:04:12Z

-2

For such tasks, I suggest you use irobotsoft web scraper. This video shows how to do it: http://irobotsoft.com/help/record%20robot.swf

answered Apr 23, 2014 at 16:04

seagulf

3783 silver badges5 bronze badges

Collectives™ on Stack Overflow

Trying to loop links with selenium in python

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related