Webscraping question in python using Selenium

Question

I am trying to scrape using selenium in python. I want the solar data from this site and section: https://www.caiso.com/TodaysOutlook/Pages/supply.html#section-renewables-trend

I think the problem I'm having is that the Chart data (CSV) menu option does not function as a button so clicking it doesn't work. This is what I see when I inspect the element before and after clicking it the "Chart data (CSV)" menu option.

Before: <a class="dropdown-item mb-0" id="downloadRenewablesCSV" data-type="text/csv">Chart data (CSV)</a>

After: <a class="dropdown-item mb-0" id="downloadRenewablesCSV" data-type="text/csv" href="data:text/csv;charset=utf8,Renewables%2007%2F20%2 ... [alot of encoded data] ...2C209%2C211%2C211%2C211%2C212%2C211%2C211%2C210%0A" download="CAISO-renewables-20220720.csv">Chart data (CSV)</a>

originally I assumed it was just a button element that would download the csv file and was trying to do this:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome(executable_path='PATH')
driver.get('https://www.caiso.com/TodaysOutlook/Pages/supply.html')
button = driver.find_element(by='xpath',value='/html/body/div[1]/div[3]/div[8]/div/div/div[2]/nav/div[3]/div/a[1]')
button.click()

This isn't working. Any advice? I am very new to selenium sorry.

Yes, I was wrong, so I deleted the comment. The CSV data is embedded in the href attribute. You should be able to fetch that attribute and URL-decode it to fetch your data. — Tim Roberts
– Tim Roberts, Commented Jul 20, 2022 at 17:41
@TimRoberts ah okay. My problem is that the encoded data doesn't appear until after the Chart data (CSV) is clicked on. Is there a way to get it without figuring out how to click the Chart data (CSV) option first? — Zachary McArthur
– Zachary McArthur, Commented Jul 20, 2022 at 17:44
If it doesn't redraw the whole page, then either the data is embedded in the Javascript, or it makes an AJAX request to fetch it. You may have to slog through the web page code to find that. Have you watched the network traffic in your browser's Developer Tools to see what requests are made? — Tim Roberts
– Tim Roberts, Commented Jul 20, 2022 at 17:47

undetected Selenium · Accepted Answer · 2022-07-20 21:03:21Z

2

You were trying to click on download button without actually expanding the drop down, the element becomes interactable upon clicking the dropdown.

The show class is added dynamically to the div only once the <button> with text Download is clicked.

The below code should work after clicking on the dropdown button

dropdown = driver.find_element(By.XPATH, "//button[@id='dropdownMenuRenewables']")
dropdown.click()
download_b = driver.find_element(By.XPATH, "//a[@id='downloadRenewablesCSV']")
download_b.click()

This will download the file for you

edited Jul 20, 2022 at 21:03

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

answered Jul 20, 2022 at 18:04

Himanshu Poddar

7,90116 gold badges60 silver badges120 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Luke Hamilton · Accepted Answer · 2022-07-20 17:56:04Z

1

JS Path Interaction:

Xpath selectors can be a bit finicky, I would revert to the basics and try to interact with the element via the JS Path. I was able to reproduce the error and download the report using the JS Path instead. Implement the following updated code:

driver.get('https://www.caiso.com/TodaysOutlook/Pages/supply.html')
driver.execute_script("el = document.querySelector('#downloadRenewablesCSV');el.click();")

answered Jul 20, 2022 at 17:56

Luke Hamilton

7377 silver badges20 bronze badges

Collectives™ on Stack Overflow

Webscraping question in python using Selenium

2 Answers 2

Comments

JS Path Interaction:

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

JS Path Interaction:

Comments

Your Answer

Sign up or log in

Post as a guest

Related