Inconsistent results retrieving data with Selenium in Python

Question

driver = webdriver.Chrome(driver_path, options=chrome_options)
wait = WebDriverWait(driver, 20)

driver.get('https://%s/' % asset_id)

wait.until(EC.presence_of_element_located((By.XPATH, "//*[@id='dev_diaginfo_fid']")))

print(driver.find_element_by_xpath('//*[@id='dev_diaginfo_fid']").get_attribute=("innerHTML"))

I'm able to log into the website and Selenium returns the WebElement but it is not consistent when returning the text from that WebElement. Sometimes it returns it and other times it seems like it isn't loading fast enough (slow network where this is being utilized) and returns no data at all but I can still see the WebElement itself just not the data. The data is dynamically loaded via JS. Probably not relevant but I am using send_keys to pass the credentials needed to login and then the page with the version is loaded.

Is there a way to use an ExpectedCondition (EC) to wait until it sees text before moving on? I'm attempting to pull the firmware version from a network device and it finds the Firmware element but it is not consistent when returning the actual firmware version. As stated before, there are issues with network speeds occasionally so my suspicion is that it's moving on before loading the firmware number. This device does not have internet access so I can't share the URL. I can confirm that I have pulled the firmware version but it's just not consistent.

I have tried passing it to beautifulsoup and can verify that it sees Firmware Version: but the inner tags are empty.

Edit: I have tried EC.visibility_of_all_elements and EC.visibility_of_element as well with no luck.

Not sure that would work, but try using EC.visibility_of_element_located. — RonKon
– RonKon, Commented Oct 19, 2021 at 12:08
I have tried EC.visibility_of_element_located as well. I'll edit the original and add that — ex28
– ex28, Commented Oct 19, 2021 at 12:57

Alichino · Accepted Answer · 2021-10-20 14:27:00Z

1

Here's an idea.

Try a while loop until you see the text.

counter = 0

elem = driver.find_element_by_xpath("//*[@id='dev_diaginfo_fid']").get_attribute=("innerHTML")

while "" in elem:
    pause(500)
    elem = driver.find_element_by_xpath("//*[@id='dev_diaginfo_fid']").get_attribute=("innerHTML")
    if "" not in elem:
        print("Success! The text is: " + elem)
        break
    if counter > 20:
        print("Text still not found!")
        break
    counter += 1

Obviously, adjust the loop to suit your needs.

answered Oct 20, 2021 at 14:27

Alichino

1,7362 gold badges18 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ex28 Over a year ago

This doesn't run the success loop correctly for whatever reason but by running the loop it does let the data load and can be picked up after so I'm marking as solved. Thanks!

Alichino Over a year ago

Perhaps the condition should be >> while not elem: << and later in the loop >> if not elem: << But anyway, glad it works ;-)

Collectives™ on Stack Overflow

Inconsistent results retrieving data with Selenium in Python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related