Issue
I’m trying to scrape this one (infinite) page (www.mydealz.de) but I cannot get my webdriver to scroll down the page.
Im using Python (3.5), Selenium (3.6) and PhantomJS.
I already tried several approaches but the webdriver just wont scroll – it just gives me the first page.
1st approach (the ususal scrolling approach):
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(1)
new_height = driver.execute_script("return document.body.scrollHeight")
if new_height == last_height:
break
last_height = new_height
2nd approach (just pressing the down key several times and release it, tried wating inbetween presses, too):
ActionChains(driver).key_down(Keys.ARROW_DOWN).perform()
ActionChains(driver).key_up(Keys.ARROW_DOWN).perform()
3rd approach (find the last element in the “scrolling list” and scroll to its view to force scrolling):
posts = driver.find_elements_by_css_selector("div.threadGrid")
driver.execute_script("arguments[0].scrollIntoView();", posts[-1])
Nothing worked so far, does anybody know if there is another approach or where I made an error?
Solution
To scroll through the webpage untill the url is mydealz.de/?page=3 you can use the following block of code :
from selenium import webdriver
driver = webdriver.PhantomJS(executable_path=r'C:\\Utility\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe')
driver.set_window_size(1400,1000)
driver.get("https://www.mydealz.de")
while ("3" not in driver.current_url) :
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
print(driver.current_url)
driver.quit()
Console Output :
https://www.mydealz.de/?page=3
Answered By – DebanjanB
This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0