Select element using XPATH with Python?

Question

I am trying to determine the number of pages of data generated by the Indian Central Pollution Controal Board. Here is an example of output. Following https://github.com/RachitKamdar/Python-Scraper, I used selenium/python

maxpage = int(browser.find_elements(By.XPATH,"//*[@id='DataTables_Table_0_paginate']/span/a")[-1].text)

but this produces an empty array. I am really not sure what I am doing wrong. Any help would be greatly appreciated. Thanks

Prophet · Accepted Answer · 2021-06-15 21:14:11Z

1

You have to add expected condition to wait until the page loaded the data.
You can wait for visibility of element you are using and after that get it's text, like this:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.XPATH, "//*[@id='DataTables_Table_0_paginate']/span/a")))
maxpage = int(browser.find_elements(By.XPATH,"//*[@id='DataTables_Table_0_paginate']/span/a")[-1].text)

answered Jun 15, 2021 at 21:14

Prophet

33.5k28 gold badges58 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Rohan · Accepted Answer · 2021-06-16 10:53:36Z

0

You might want to try getattribute('textContent')

In your case:

maxpage=browser.find_element_by_xpath("(//*[@id='DataTables_Table_0_paginate']/span/a)[last()]").getattribute('textContent')

answered Jun 16, 2021 at 10:53

Rohan

647 bronze badges

Collectives™ on Stack Overflow

Select element using XPATH with Python?

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related