I’m trying to scrape this one (infinite) page (www.mydealz.de) but I cannot get my webdriver to scroll down the page.
Im using Python (3.5), Selenium (3.6) and PhantomJS.
I already tried several approaches but the webdriver just wont scroll – it just gives me the first page.
1st approach (the ususal scrolling approach):
last_height = driver.execute_script("return document.body.scrollHeight") while True: driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") time.sleep(1) new_height = driver.execute_script("return document.body.scrollHeight") if new_height == last_height: break last_height = new_height
2nd approach (just pressing the down key several times and release it, tried wating inbetween presses, too):
3rd approach (find the last element in the “scrolling list” and scroll to its view to force scrolling):
posts = driver.find_elements_by_css_selector("div.threadGrid") driver.execute_script("arguments.scrollIntoView();", posts[-1])
Nothing worked so far, does anybody know if there is another approach or where I made an error?
To scroll through the webpage untill
the url is mydealz.de/?page=3 you can use the following block of code :
from selenium import webdriver driver = webdriver.PhantomJS(executable_path=r'C:\\Utility\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe') driver.set_window_size(1400,1000) driver.get("https://www.mydealz.de") while ("3" not in driver.current_url) : driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") print(driver.current_url) driver.quit()
Console Output :
Answered By – DebanjanB