Issue
I am trying to scrape using selenium in python. I want the solar data from this site and section: https://www.caiso.com/TodaysOutlook/Pages/supply.html#section-renewables-trend 
I think the problem I’m having is that the Chart data (CSV) menu option does not function as a button so clicking it doesn’t work. This is what I see when I inspect the element before and after clicking it the "Chart data (CSV)" menu option.
Before: <a class="dropdown-item mb-0" id="downloadRenewablesCSV" data-type="text/csv">Chart data (CSV)</a>
After: <a class="dropdown-item mb-0" id="downloadRenewablesCSV" data-type="text/csv" href="data:text/csv;charset=utf8,Renewables%2007%2F20%2 ... [alot of encoded data] ...2C209%2C211%2C211%2C211%2C212%2C211%2C211%2C210%0A" download="CAISO-renewables-20220720.csv">Chart data (CSV)</a>
originally I assumed it was just a button element that would download the csv file and was trying to do this:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome(executable_path='PATH')
driver.get('https://www.caiso.com/TodaysOutlook/Pages/supply.html')
button = driver.find_element(by='xpath',value='/html/body/div[1]/div[3]/div[8]/div/div/div[2]/nav/div[3]/div/a[1]')
button.click()
This isn’t working. Any advice? I am very new to selenium sorry.
Solution
JS Path Interaction:
Xpath selectors can be a bit finicky, I would revert to the basics and try to interact with the element via the JS Path. I was able to reproduce the error and download the report using the JS Path instead. Implement the following updated code:
driver.get('https://www.caiso.com/TodaysOutlook/Pages/supply.html')
driver.execute_script("el = document.querySelector('#downloadRenewablesCSV');el.click();")
Answered By – Luke Hamilton
This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0