Python Beautifulsoup findAll finds some but not all

Issue

With a bit of Python knowledge I tried to do some LinkedIn company posts scraping.

Using the code below, that I took from this website, all posts on a company’s LinkedIn page are found first before its contents are extracted. The issue is that I know, I’ve counted, that there are more posts than that the findAll function returns, regardless of which of the parsers lxml, html5lib or html.parser I use. In one case, it returns 43 out of 67 posts, in another case it returns 10 out of 14. Typically, it finds about 3 or 4, then it skips 4 or 5 posts, then it finds a few again, etc.

How can I find out why this is happening?

#!/usr/bin/env python
# coding: utf-8

# Import
from selenium import webdriver
from bs4 import BeautifulSoup as bs
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

# Get credentials to log in to LinkedIn
username = input('Enter your linkedin username: ')
password = input('Enter your linkedin password: ')
company_name = input('Name of the company: ')

# Access Webdriver
s=Service(ChromeDriverManager().install())
browser = webdriver.Chrome(service=s)
browser.maximize_window()

# Define page to open
page = "https://www.linkedin.com/company/{}/posts/?feedView=all".format(company_name)

# Open login page
browser.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')

# Enter login info:
elementID = browser.find_element_by_id('username')
elementID.send_keys(username)
elementID = browser.find_element_by_id('password')
elementID.send_keys(password)
elementID.submit()

# Go to webpage
browser.get(page + 'posts/')

# Define scrolling time
SCROLL_PAUSE_TIME = 1.5

# Get scroll height
last_height = browser.execute_script("return document.body.scrollHeight")

# Scroll all the way to the bottom of the page
while True:

    # Scroll down to bottom
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    
    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    new_height = browser.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

# Get content of page
content = browser.page_source.encode('utf-8').strip()

# Create soup
linkedin_soup = bs(content, "html5lib")
linkedin_soup.prettify()

# Find entities that contain posts
containers = linkedin_soup.findAll("div",{"class":"occludable-update ember-view"})

Solution

The issue is when you scroll down to the bottom, it sort of skips some of the posts to render. There likely a better way to do this, but basically I have scroll 1/4 of the way, then 1/2, then full (hoping to catch all the posts). Try this adjustment:

# Scroll all the way to the bottom of the page
while True:

    # Scroll down to bottom
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4);")
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight/2);")
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    
    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    new_height = browser.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

Answered By – chitown88

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply

(*) Required, Your email will not be published