I am trying to scrape using selenium in python. I want the solar data from this site and section: https://www.caiso.com/TodaysOutlook/Pages/supply.html#section-renewables-trend
I think the problem I’m having is that the
Chart data (CSV) menu option does not function as a button so clicking it doesn’t work. This is what I see when I inspect the element before and after clicking it the "Chart data (CSV)" menu option.
<a class="dropdown-item mb-0" id="downloadRenewablesCSV" data-type="text/csv">Chart data (CSV)</a>
<a class="dropdown-item mb-0" id="downloadRenewablesCSV" data-type="text/csv" href="data:text/csv;charset=utf8,Renewables%2007%2F20%2 ... [alot of encoded data] ...2C209%2C211%2C211%2C211%2C212%2C211%2C211%2C210%0A" download="CAISO-renewables-20220720.csv">Chart data (CSV)</a>
originally I assumed it was just a button element that would download the csv file and was trying to do this:
from selenium import webdriver from selenium.webdriver.common.keys import Keys driver = webdriver.Chrome(executable_path='PATH') driver.get('https://www.caiso.com/TodaysOutlook/Pages/supply.html') button = driver.find_element(by='xpath',value='/html/body/div/div/div/div/div/div/nav/div/div/a') button.click()
This isn’t working. Any advice? I am very new to selenium sorry.
JS Path Interaction:
Xpath selectors can be a bit finicky, I would revert to the basics and try to interact with the element via the JS Path. I was able to reproduce the error and download the report using the JS Path instead. Implement the following updated code:
driver.get('https://www.caiso.com/TodaysOutlook/Pages/supply.html') driver.execute_script("el = document.querySelector('#downloadRenewablesCSV');el.click();")
Answered By – Luke Hamilton