Facing issues while scraping data from a table using python with selenium

Question

I've written a script using python in combination with selenium to parse table from a target page which can be reached out following some steps I've tried to describe below for the clarity. It does reach the destination but at the time of scraping data from that table It throws an error showing in the console "Unable to locate element". I tried with online xpath tester to see if it is wrong but I found that the xpath I've used in my script for "td_data" is right. I suppose, what I'm missing here is beyond my knowledge. Hope there is somebody to take a look into it and provide me with a workaround. Btw, the site link is given in my script.

Link to see the html contents for the table: "https://www.dropbox.com/s/kaom5qzk78xndqn/Partial%20Html%20content%20for%20the%20table.txt?dl=0"

Steps to reach the target page which my script is able to maintain:

Selecting "I've read and understand above"
Putting this keyword "pump" in the inputbox located right below "Select medical devices".
Selecting the checkbox "Devices found for "pump".
Finally, pressing the search button

Script I've tried with so far:

from selenium import webdriver
import time

driver = webdriver.Chrome()
driver.get('http://apps.tga.gov.au/Prod/devices/daen-entry.aspx')

driver.find_element_by_id('disclaimer-accept').click()
time.sleep(5)

driver.find_element_by_id('medicine-name').send_keys('pump')
time.sleep(8)

driver.find_element_by_id('medicines-header-text').click()

driver.find_element_by_id('submit-button').click()
time.sleep(7)

for item in driver.find_elements_by_xpath('//div[@class="table-responsive"]'):
    for tr_data in item.find_elements_by_xpath('.//tr'):
        td_data = tr_data.find_element_by_xpath('.//span[@class="hovertext"]//a')
        print(td_data.text)

driver.close()

Can you please share the HTML so that I can validate the logic? — Monika
– Monika, Commented Jun 24, 2017 at 11:06
Thanks for your response, Monika. I've updated my post with a link to the html content. — SIM
– SIM, Commented Jun 24, 2017 at 11:22

James Schinner · Accepted Answer · 2017-06-24 12:26:52Z

1

Why don't you just do this:

from selenium import webdriver
import time

driver = webdriver.Chrome()
driver.get('http://apps.tga.gov.au/Prod/devices/daen-entry.aspx')

driver.find_element_by_id('disclaimer-accept').click()
time.sleep(5)

driver.find_element_by_id('medicine-name').send_keys('pump')
time.sleep(8)

driver.find_element_by_id('medicines-header-text').click()

driver.find_element_by_id('submit-button').click()
time.sleep(7)

for item in driver.find_elements_by_xpath(
'//table[@id]/tbody/tr/td[@class]/span[@class]/a[@id]'
):
    print(item.text)

driver.close()

Output:

Maybe you want to think about saving the page with driver.page_source, pull out the table, save it as a html file. Then use pandas from html to open the table into a dataframe

edited Jun 24, 2017 at 12:26

answered Jun 24, 2017 at 11:55

James Schinner

1,5791 gold badge21 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

James Schinner Over a year ago

This xpath will give you all the rows:"""//body[@class]/div[@id]/div[@id]/div[@class]/div[@id]/div[@class]/form[@id]/div[@id]/div[@class]/div[@id]/div/div[@class]/table[@id]/tbody/tr"""

SIM Over a year ago

Thanks for the solution, James Schinner. I never worked with table data using selenium that is why i messed up with my script.

SIM Over a year ago

Thanks James Schinner, for your answer. Please replace your xpath with '//div[@class="table-responsive"]//td[@class="row-odd"]' in your script so that I can accept your answer. The one you provided just now is very fragile and will break if any sort of change is made in that webpage.

James Schinner Over a year ago

Done, though it gave me an error:selenium.common.exceptions.ElementNotVisibleException: Message: element not visible (Session info: chrome=59.0.3071.109) (Driver info: chromedriver=2.30.477700 (0057494ad8732195794a7b32078424f92a5fce41),platform=Windows NT 10.0.15063 x86_64)

James Schinner Over a year ago

I'm going to leave it i'm afraid. I'm glad it solved your problem though

Collectives™ on Stack Overflow

Facing issues while scraping data from a table using python with selenium

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related