Scraping hover over figure using Python and Selenium

Question

I am trying to scrape the data from http://fuelinsights.gasbuddy.com/Charts using Python and Selenium. The difficult part is that the data only appear when a point on the line graph is hovered over. Currently, my issue is an inability to create a list of all the hover over objects. My code so far is below:

from selenium import webdriver as web
from selenium.webdriver.common.action_chains import ActionChains

driver = web.Chrome('driver path')

driver.get('http://fuelinsights.gasbuddy.com/Charts')

test= driver.find_elements_by_xpath('//*[@class="highcharts-markers"]')

print(test)

`

which gives me test=[]. Previously, I have used beautifulsoup for all of my scraping projects, but I have redone some of my previous projects to make sure that I understand how Selenium works and haven't had issues.

If anyone can help me solve this issue so I can create a list of the items that I can use ActionChains to hover over and extract the price and date from it would be much appreciated.

Thank you!

****EDIT**** To clarify, I have looked over numerous other posts concerning SVG and g elements and Highcharts, but I am still short on a solution to this problem. I have tried numerous Xpaths (and other find_elements_by options), but have only been able to come to two results: (1) the Xpath is valid, but does not contain any elements, or (2) InvalidSelectorException indicating that I was unable to locate an element with the xpath expression. I believe this comes down to simply incorrectly specifying my Xpath, but I am at a loss for how to find the correct Xpath.

The highcharts-markers class is within a <g> tag which is within a svg tag. Possible duplicate of Selenium WebDriver [Java]: How to Click on elements within an SVG using XPath — undetected Selenium
– undetected Selenium, Commented Jan 10, 2018 at 7:55
Thanks DebanjanB this has provided some useful insight. These are very similar questions, but I have been unable to solve my problem using the proposed solutions provided in the link (admittedly this could be due to my own shortcomings converting from Java to Python though). — Let's Boogie
– Let's Boogie, Commented Jan 10, 2018 at 21:23

cruisepandey · Accepted Answer · 2021-05-12 15:51:48Z

5

You can't use the Xpath which you have mentioned above for locating the elements inside the svg tag.

Xpath which you can use to create a list of hover objects is:

//*[name()='svg']//*[name()='g' and @class='highcharts-markers']/*[name()='path']

I have written a java program for getting the text of all the tool-tip elements. You can use the logic and write a corresponding python code:

1. Get List of tooltip Elements

 List <WebElement> highChartElements= driver.findElements(By.xpath("//*[name()='svg']//*[name()='g' and @class='highcharts-markers']/*[name()='path']"));

2. Iterate through the list and use action class for moving and clicking on all the tooltip Elements

3. Get the text of the tooltip elements.

for(WebElement element:highChartElements){
        Actions action = new Actions(driver);
        action.moveToElement(element).click().perform();
        Thread.sleep(3000);
        List<WebElement> highChartToolTipTextElements= driver.findElements(By.xpath("//*[name()='svg']//*[name()='g' and @class='highcharts-tooltip']/*[name()='text']/*[name()='tspan']"));
        for(WebElement toolTipElement:highChartToolTipTextElements){
            System.out.println("The text for the elements is"+toolTipElement.getText());
        }
    }

edited May 12, 2021 at 15:51

cruisepandey

29.5k6 gold badges23 silver badges43 bronze badges

answered Jan 10, 2018 at 8:42

Monika

7321 gold badge4 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Let's Boogie Over a year ago

Monika, thanks for the pointers. I had to tweak the Xpath you provided slightly for it to run, but I still am getting an empty list back from the driver.find_elements_by_xpath command. My Xpath is now '//*[name()="svg"]//*[name()="g"]//*[@class="highcharts-markers"]/*[name="path"]' do you have any other advice on how to fix this?

Monika Over a year ago

Don't change the xpath, use the one I have provided. The xpath should work, I have executed the program on my machine and it worked.Why did you tweak the Xpath?

Let's Boogie Over a year ago

The Xpath you specified when used in python in the following command test= driver.find_elements_by_xpath("//*[name()='svg']//*[name()='g' and @class='highcharts-markers']/*[name()='path']") returns test=[]. I'm not sure why this provides you with results, but returns nothing to me. Any further guidance is much appreciated!

ce.teuf · Accepted Answer · 2020-05-31 11:21:37Z

Thanks You ! 2 years later, i'm facing to an equivalent project and used your example to learn how to get the job done with python & Firefox. Perhaps the following code will be useful to some people.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains

driver = webdriver.Firefox(executable_path=r'path\to\the\geckodriver.exe')
driver.get('http://fuelinsights.gasbuddy.com/Charts')
element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "highcharts-markers")))
test = driver.find_elements_by_xpath("//*[name()='svg']//*[name()='g' and @class='highcharts-markers']/*[name()='path']")

res = []
for el in test:
    hover = ActionChains(driver).move_to_element(el)
    hover.perform()
    date = driver.find_elements_by_css_selector(".highcharts-tooltip > text:nth-child(5) > tspan:nth-child(1)")
    price = driver.find_elements_by_css_selector(".highcharts-tooltip > text:nth-child(5) > tspan:nth-child(4)")
    res.append((date[0].text, price[0].text))

"res" contains :

('Saturday, May 30, 2020', '1.978 $/gal')
('Friday, May 29, 2020', '1.979 $/gal')
('Thursday, May 28, 2020', '1.977 $/gal')
('Wednesday, May 27, 2020', '1.972 $/gal')
('Tuesday, May 26, 2020', '1.965 $/gal')
.......

I am stuck on the same issue but on a different website. This solution didn't help me

Collectives™ on Stack Overflow

Scraping hover over figure using Python and Selenium

2 Answers 2

3 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related