Python - Using Selenium to extract data from Webpage

Question

I am trying to extract IP address information from a webpage using Selenium, here is a portion of the code

removals = []
ServerNames = driver.find_elements_by_xpath("//span[contains(text(), 'Microsoft') or contains(text(), 'Solaris') or contains(text(), 'Linux') or contains(text(), 'Server') or contains(text(), 'Windows') or contains(text(), 'AIX') or contains(text(), 'VMware') or contains(text(), 'ESX')]")
Backups = driver.find_elements_by_xpath("//span[contains(text(), '|locationInherited=true')]")
ips = driver.find_elements_by_xpath("//span[contains(text(), '10.') or contains(text(), '167.') or contains(text(), '170.') or contains(text(), '192.') or contains(text(), '172.') or contains(text(), '204.')]")
for x, y in itertools.zip_longest(ServerNames, Backups, fillvalue=''):
        x.click()
        ips
        y.click()
        ips
for ip in ips:
        removals.append(ip.text)

When a 'ServerNames' object is found and clicked, the webpage will display both the primary and backup IP addresses for that particular server, with that said, need the script to:

1)Iterate through every found "ServerNames" object 2) Click on the object 3) Grab the associated Primary IP 4) Click on the associated "Backups" object found 5) Grab that IP as well

Then move on to the next ServerNames object and repeat these steps until all IP addresses for all severs have been extracted. I understand that the driver.find_elements method essentially create lists, so basically I need a way to alternate clicking & extracting data for each iteration through the lists. Thanks,

with the code above I get this error:

c:\Python35\Scripts>python Remedy2.py
Traceback (most recent call last):
  File "Remedy2.py", line 73, in <module>
    y.click()
AttributeError: 'str' object has no attribute 'click'

with this code:

for x in ServerNames:
        x.click()
        ips = driver.find_elements_by_xpath("//span[contains(text(), '10.') or contains(text(), '167.') or contains(text(), '170.') or contains(text(), '192.') or contains(text(), '172.') or contains(text(), '204.')]")
        if Backups:
            for y in Backups:
                y.click()
                ips = driver.find_elements_by_xpath("//span[contains(text(), '10.') or contains(text(), '167.') or contains(text(), '170.') or contains(text(), '192.') or contains(text(), '172.') or contains(text(), '204.')]")

I get a 'stale element error'

Amit · Accepted Answer · 2017-10-04 19:19:18Z

1

You have to find the element again if the page changes after an action. The stale element erorr is caused probably because the element is no longer attached to the DOM. An approach is given below:

for i in range(len(ServerNames)):
        server = driver.find_elements_by_xpath("//span[contains(text(), 'Microsoft') or contains(text(), 'Solaris') or contains(text(), 'Linux') or contains(text(), 'Server') or contains(text(), 'Windows') or contains(text(), 'AIX') or contains(text(), 'VMware') or contains(text(), 'ESX')]")[i]
        server.click()
        ips = driver.find_elements_by_xpath("//span[contains(text(), '10.') or contains(text(), '167.') or contains(text(), '170.') or contains(text(), '192.') or contains(text(), '172.') or contains(text(), '204.')]")
        if Backups:
            for j in range(len(Backups)):
                backup = driver.find_elements_by_xpath("//span[contains(text(), '|locationInherited=true')]")[j]
                backup.click()
                ips = driver.find_elements_by_xpath("//span[contains(text(), '10.') or contains(text(), '167.') or contains(text(), '170.') or contains(text(), '192.') or contains(text(), '172.') or contains(text(), '204.')]"

answered Oct 4, 2017 at 19:19

Amit

20.6k7 gold badges51 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

I.T. Guy Over a year ago

Excellent. Really appreciate the assistance.

I.T. Guy Over a year ago

Got this error: c:\Python35\Scripts>python Remedy2.py Traceback (most recent call last): File "Remedy2.py", line 75, in <module> server.click() AttributeError: 'list' object has no attribute 'click'

Amit Over a year ago

Make sure you are not missing the [i] at the end of line previous to server.click()

I.T. Guy Over a year ago

I see, thank you.. Just for my knowledge, what does the [i] and [j] tell the script to do?

Amit Over a year ago

find_elements methods returns a list of elements and [i] & [j] is used to assign the ith & jth element respectively to the variable.

|

Collectives™ on Stack Overflow

Python - Using Selenium to extract data from Webpage

1 Answer 1

9 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Related