Speedup web scraping with selenium

Issue

I am newbie to web scraping using selenium and I am scraping seetickets.us
My scraper works as follows.

  • sign in
  • search for events
  • click on each event
  • scrape data
  • come back
  • click on next event
  • repeat

Now the problem is that some of the events do not contain some elements such as
this event: https://wl.seetickets.us/event/Beta-Hi-Fi/484490?afflky=WorldCafeLive

which does not contain pricing table
but this one does

https://www.seetickets.us/event/Wake-Up-Daisy-1100AM/477633

so I have used try except blocks

try:
   find element 
except:
   return none

but if it doesnt found the element in try, it takes 5 seconds to go to except because I have used

webdriver.implicitwait(5)

Now , if any page does not contain multiple elements , the selenium takes very much time to scrape that page.

I have thousands of pages to scrape. What should be done to speed up the process.

Thanks

Solution

To speedup web scraping using Selenium:

Your effective code block will be:

try:
   element = WebDriverWait(driver, 3).until(EC.visibility_of_element_located((By.ID, "input")))) 
   print("Element is visible")
except TimeoutException:
   print("Element is not visible")

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Answered By – undetected Selenium

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply

(*) Required, Your email will not be published