Python Selenium Web Scraping the data from the charts

Question

I am trying to web-scrape the chart data from this page https://www.focusireland.ie/resource-hub/latest-figures-homelessness-ireland/

On the right top, there is a chart which details Number of homeless people in Ireland.

I am trying to web-scrape the axis which is Jul - 14 and it's corresponding value 3258. But, I couldn't do locate those element and it's value using x-path.

I tried something like this to web-scrape the value

Val1 = driver.find_element_by_xpath("//*[@id='chart']/div[7]").text

& it didn't work.

And tried this:-

element_to_hover_over = driver.find_element_by_xpath("//div[@class='label series smaller' | @class='label series smaller hover']/span[text()='Jul-14'").text
print(element_to_hover_over)

It didn't work either for me and returned an error message as

Unable to locate an element with the xpath expression //div[@class='label series smaller' | @class='label series smaller hover']/span[text()='Jul-14' because of the following error:
SyntaxError: Failed to execute 'evaluate' on 'Document': The string '//div[@class='label series smaller' | @class='label series smaller hover']/span[text()='Jul-14'' is not a valid XPath expression.

Nestor Yanchuk · Accepted Answer · 2018-04-23 14:11:15Z

1

Tha data are not on that website. You have to parse source iframe of the chart ( datawrapper.dwcdn.net/** ). Then you have to extract and parse JS variable (__dw.params.data).

This is not so hard, see this.

answered Apr 23, 2018 at 14:11

Nestor Yanchuk

1,2869 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Student of the Digital World Over a year ago

:- Can you show that for one of the axis please. I am finding it really hard to understand this with no background to JS.

Nestor Yanchuk Over a year ago

Look at the link I've provided, then just run wd.execute_script("return __dw.params.data"). It is going to be an easy-to-parse string.

Ankur Singh · Accepted Answer · 2018-04-23 14:48:14Z

1

You can try below code to get text :

element_to_hover_over = firefox.find_element_by_xpath("//span[text()='Jul-14']")

hover = ActionChains(firefox).move_to_element(element_to_hover_over)
hover.perform()

element_to_hover_over = firefox.find_element_by_xpath("//div[@class='label value outline  showOnHover hover']/span")

element_to_hover_over.text

Your second chart is inside frame first you need to switch into frame :-

frame :- datawrapper-chart-8ZOKk

then try below xpath to get value in list

list <webElement> = (//div[@class='label series smaller' and @data-column='X.1'])[1]/span

then iterate through every row and get value

edited Apr 23, 2018 at 14:48

answered Apr 23, 2018 at 14:20

Ankur Singh

1,2891 gold badge8 silver badges20 bronze badges

7 Comments

Student of the Digital World Over a year ago

this would get me the value of 3258. But getting the value of Jul-14 is still a task here.

Ankur Singh Over a year ago

you can element_to_hover_over .text to get value

Ankur Singh Over a year ago

try this :- //div[@class='label series smaller' | @class='label series smaller hover']/span[text()="Jul-14"]

Student of the Digital World Over a year ago

Sorry, but I think you didn't get my point. I am looking to go back from chart to table. So, I would like to write a code that is dynamic - here we are putting in the variable as Jul-14 in element_to_hover_over

Ankur Singh Over a year ago

Updated my answer

|

Collectives™ on Stack Overflow

Python Selenium Web Scraping the data from the charts

2 Answers 2

2 Comments

element_to_hover_over.text

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

element_to_hover_over.text

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related