Extracting element from table using selenium web scraping

Question

Using selenium I am trying to scrape a table from a website, however, data appears compressed into one single column rather than two separate columns; Date and Value- help would be greatly appreciated - now amended to included further improvement

driver.get("https://www.multpl.com/shiller-pe/table/by-year/")

table_trs = driver.find_elements(By.XPATH, '//table[@id="datatable"]/tbody/tr')

value_list = []
for v in range(len(table_trs )):
    value_list.append(table_trs [v].text)

print(value_list)
df = pd.DataFrame(value_list)

The table I am trying to scrape appears on the website as follows....

..and the section of html associated to it.....

And where is the code you wrote to scrape it?

mplungjan
– mplungjan

2022-02-06 18:57:35 +00:00
Commented Feb 6, 2022 at 18:57 — mplungjan
– mplungjan, Commented Feb 6, 2022 at 18:57
I have updated question to show the code i've used

Diop Chopra
– Diop Chopra

2022-02-06 19:30:09 +00:00
Commented Feb 6, 2022 at 19:30 — Diop Chopra
– Diop Chopra, Commented Feb 6, 2022 at 19:30

HedgeHog · Accepted Answer · 2022-02-06 20:57:05Z

3

Note: Answer is focused on correct usage of xpath and only based on your screenshot - Improving your question and posting code and examples as text would generate more specific answers

To get the <tr>s of the table by xpath change it to //table[@id="datatable"]/tbody/tr

from selenium.webdriver.common.by import By

table_trs = driver.find_elements(By.XPATH, '//table[@id="datatable"]/tbody/tr')

EDIT

Based on your improvements you can go with pandas.read_html() to convert the table directly into a dataframe. Just have to rename the columns, cause there are two span tag in the <th> that will lead to column name "Value Value":

driver.get("https://www.multpl.com/shiller-pe/table/by-year/")

df = pd.read_html(driver.page_source)[0]
df.columns = ['Date','Value']

As alternative you could iterate like this:

driver.get("https://www.multpl.com/shiller-pe/table/by-year/")

table_trs = driver.find_elements(By.XPATH, '//table[@id="datatable"]/tbody/tr')
value_list = []
for row in table_trs[1:]:
    value_list.append({
        'Date':row.find_elements(By.TAG_NAME, "td")[0].text,
        'Value':row.find_elements(By.TAG_NAME, "td")[1].text
    })

df = pd.DataFrame(value_list)

Output

Date	Value
Feb 4, 2022	37.18
Jan 1, 2022	39.63
Jan 1, 2021	34.51
Jan 1, 2020	30.99
Jan 1, 2019	28.38
Jan 1, 2018	33.31
Jan 1, 2017	28.06
Jan 1, 2016	24.21
Jan 1, 2015	26.49
...	...

edited Feb 6, 2022 at 20:57

answered Feb 6, 2022 at 19:01

HedgeHog

25.4k5 gold badges18 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Diop Chopra Over a year ago

Thanks, but this provides a very strange output.... [<selenium.webdriver.remote.webelement.WebElement (session=.......

HedgeHog Over a year ago

It is a result set of the trs you have to iterate - As mentioned, improving your question would lead to more specific answers.

Diop Chopra Over a year ago

Iteration now added and converted to df but unable to split single column

HedgeHog Over a year ago

Thanks for your update - I added an edit to my answer, take a look and give it a try.

Diop Chopra Over a year ago

that came out exactly as i'd hoped - thanks!!!

Gaj Julije · Accepted Answer · 2022-02-06 18:57:41Z

1

Something is missing here or you put wrong xpath. Valid xpath will be (based on picture) "//div[@id="datatable"]/tbody/tr", but that will only give you the rows. You can iterate by getting the all row nubers and colums, something like //div[@id="datatable"]/tbody/tr[i]/td[j] and then get text fro each element.

answered Feb 6, 2022 at 18:57

Gaj Julije

2,1951 gold badge22 silver badges38 bronze badges

Collectives™ on Stack Overflow

Extracting element from table using selenium web scraping

2 Answers 2

EDIT

Output

5 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

EDIT

Output

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related