Scrolling infinite page with Python/PhantomJS/Selenium

Issue

I’m trying to scrape this one (infinite) page (www.mydealz.de) but I cannot get my webdriver to scroll down the page.
Im using Python (3.5), Selenium (3.6) and PhantomJS.
I already tried several approaches but the webdriver just wont scroll – it just gives me the first page.

1st approach (the ususal scrolling approach):

last_height = driver.execute_script("return document.body.scrollHeight")
while True:
  driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
  time.sleep(1)
  new_height = driver.execute_script("return document.body.scrollHeight")
  if new_height == last_height:
       break
  last_height = new_height

2nd approach (just pressing the down key several times and release it, tried wating inbetween presses, too):

ActionChains(driver).key_down(Keys.ARROW_DOWN).perform()
ActionChains(driver).key_up(Keys.ARROW_DOWN).perform()

3rd approach (find the last element in the “scrolling list” and scroll to its view to force scrolling):

posts = driver.find_elements_by_css_selector("div.threadGrid")
driver.execute_script("arguments[0].scrollIntoView();", posts[-1])

Nothing worked so far, does anybody know if there is another approach or where I made an error?

Solution

To scroll through the webpage untill the url is mydealz.de/?page=3 you can use the following block of code :

from selenium import webdriver

driver = webdriver.PhantomJS(executable_path=r'C:\\Utility\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe')
driver.set_window_size(1400,1000)
driver.get("https://www.mydealz.de")
while ("3" not in driver.current_url) :
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
print(driver.current_url)
driver.quit()

Console Output :

https://www.mydealz.de/?page=3

Answered By – DebanjanB

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply

(*) Required, Your email will not be published