Selenium Python Find Elements by XPath does not return all expected elements

Question

I am working on a scraping data from a series of tables on an HTML website. The website has varying numbers of tables depending on the input, so I am trying to use: .find_elements_by_xpath("//table") to simply get a list with all the table elements on the page.

However, it is only returning the first table on the page in this case. When I use find_element_by_xpath(//table[2]), it returns the other table, but using find_elements does not.

The website portal is here, just press send at the bottom. (I am trying to get the data from the results page). I am using Selenium in Python on the Firefox Webbrowser.

Interesting note, the header table has an XPath of table[1], which begs the question of where is table[0].

how do you know selenium does not return all elements? Please show code, as it should return 2 tables. About xpath [1] - counter in xpath starts from 1, not 0. So [1] is really first table. and //table[0] does not exist =) — Vitaliy Moskalyuk
– Vitaliy Moskalyuk, Commented Aug 26, 2020 at 18:57
Hey there, so when I run the following code, it only returns a value of 1, as opposed to the 2 expected. (One table for the header of the page and the other for the data table). count = driver.find_elements_by_xpath("//table") print(len(count)) — Tomasz Tabernacki
– Tomasz Tabernacki, Commented Aug 26, 2020 at 22:29
@AmrutaPande Hi, Yes I am using an explicit wait before I take count. Again, if I use EC. presence_of_element_located(By. XPATH, "//table[2]) It loads table 2 just fine, but still, the other tables are not found by EC.presence_of_all_elements_located — Tomasz Tabernacki
– Tomasz Tabernacki, Commented Aug 28, 2020 at 1:50

rahul rai · Accepted Answer · 2020-08-28 06:21:03Z

2

The Key here is to wait properly before you navigate to new tab. As after clicking on send a new tab is getting opened. See below code:

driver.get('http://imed.med.ucm.es/Tools/rankpep.html')
WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.XPATH, "//img[@src='/icons/ig_logo.png']")))

send = driver.find_element_by_xpath("//input[@value='Send']")
driver.execute_script("arguments[0].scrollIntoView();", send)
driver.execute_script("arguments[0].click();", send)

WebDriverWait(driver, 20).until(EC.number_of_windows_to_be(2)) #Wait for 2nd tab to open
driver.switch_to.window(driver.window_handles[1])
count = driver.find_elements_by_xpath("//table")
print(len(count))

Out Put: As on new page there are two tables, it has printed 2

answered Aug 28, 2020 at 6:21

rahul rai

2,3361 gold badge9 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Tomasz Tabernacki Over a year ago

Hey thanks for this. Strangely, this code didn't work with a Firefox webdriver, but it inspired me to try Chromedriver. Somewhat inexplicably all the code works now. Thanks for your help!

Collectives™ on Stack Overflow

Selenium Python Find Elements by XPath does not return all expected elements

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related