Speedup web scraping with selenium


I am newbie to web scraping using selenium and I am scraping seetickets.us
My scraper works as follows.

  • sign in
  • search for events
  • click on each event
  • scrape data
  • come back
  • click on next event
  • repeat

Now the problem is that some of the events do not contain some elements such as
this event: https://wl.seetickets.us/event/Beta-Hi-Fi/484490?afflky=WorldCafeLive

which does not contain pricing table
but this one does


so I have used try except blocks

   find element 
   return none

but if it doesnt found the element in try, it takes 5 seconds to go to except because I have used


Now , if any page does not contain multiple elements , the selenium takes very much time to scrape that page.

I have thousands of pages to scrape. What should be done to speed up the process.



To speedup web scraping using Selenium:

Your effective code block will be:

   element = WebDriverWait(driver, 3).until(EC.visibility_of_element_located((By.ID, "input")))) 
   print("Element is visible")
except TimeoutException:
   print("Element is not visible")

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Answered By – undetected Selenium

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply

(*) Required, Your email will not be published