How can I get table with dynamic data from a url using Selenium and xpath, or other alternative?

Question

I am trying to retrieve the main "Network Status" table data from this site:

http://dev.cryptolions.io/mainnet/

I have tried many combinations of selenium functions populated with Chrome's copy selector and copy Xpath, but am not having any luck. The site opens with selenium, but I keep getting errors similar to this. What am I missing here?

Getting the table in any number of data structures would be ok. But a dictionary, JSON, or something I can load into a pandas df would be ideal.

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="bpTableSystem"]"}
(Session info: chrome=67.0.3396.87) (Driver info: chromedriver=2.40.565498 (ea082db3280dd6843ebfb08a625e3eb905c4f5ab),platform=Windows NT 10.0.17134 x86_64)

My Code:

driver = webdriver.Chrome(r'C:\chromedriver_win32\chromedriver.exe') 
browser = webdriver.Chrome()

browser.get('http://dev.cryptolions.io/mainnet/')

time.sleep(30)
content = driver.find_element_by_xpath('//*[@id="bpTableSystem"]')
print content.text

Andersson · Accepted Answer · 2018-06-17 07:26:17Z

1

The problem is that you've defined two Chrome instances: browser and driver. You navigate to target page with browser while trying to find table with driver.

Simply replace

browser.get('http://dev.cryptolions.io/mainnet/')

with

driver.get('http://dev.cryptolions.io/mainnet/')

answered Jun 17, 2018 at 7:26

Andersson

52.8k18 gold badges83 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Matthew Borish Over a year ago

Your comment solved my problem instantly. Many Thanks!

Collectives™ on Stack Overflow

How can I get table with dynamic data from a url using Selenium and xpath, or other alternative?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related