I am newbie to web scraping using selenium and I am scraping seetickets.us
My scraper works as follows.
- sign in
- search for events
- click on each event
- scrape data
- come back
- click on next event
Now the problem is that some of the events do not contain some elements such as
this event: https://wl.seetickets.us/event/Beta-Hi-Fi/484490?afflky=WorldCafeLive
which does not contain pricing table
but this one does
so I have used try except blocks
try: find element except: return none
but if it doesnt found the element in try, it takes 5 seconds to go to except because I have used
Now , if any page does not contain multiple elements , the selenium takes very much time to scrape that page.
I have thousands of pages to scrape. What should be done to speed up the process.
To speedup web scraping using Selenium:
- Remove implicitwait() totally.
- Induce WebDriverWait to synchronise the webdriver instance with the WebBrowser instance for either of the following element states:
Your effective code block will be:
try: element = WebDriverWait(driver, 3).until(EC.visibility_of_element_located((By.ID, "input")))) print("Element is visible") except TimeoutException: print("Element is not visible")
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC
Answered By – undetected Selenium