Scroll to the end of the infinite loading page using selenium Python

Issue

I’m scraping the follower names from the twitter using Selenium and that page is infinite, whenever I scroll down I can see new followers.
Somehow I want to go to the bottom of the page so that I can scrape all the followers.

while number != 5:
   driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")
   number = number + 1
   time.sleep(5)

usernames = driver.find_elements_by_class_name(
       "css-4rbku5.css-18t94o4.css-1dbjc4n.r-1loqt21.r-1wbh5a2.r-dnmrzs.r-1ny4l3l")
for username in usernames:
   print(username.get_attribute("href"))

Right now code is scrolling 5 times. I have put a static value but I don’t know how much scrolls are needed to reach the bottom of the page.

Solution

Use below code for infinite loading. It will keep scrolling until new elements are getting loaded i.e. page size is changing.

# Get scroll height after first time page load
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    # Wait to load page
    time.sleep(2)
    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

Answered By – rahul rai

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply

(*) Required, Your email will not be published