Here is my code.
... from selenium import webdriver from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.chrome.options import Options from bs4 import BeautifulSoup options = Options() options.headless = True driver = webdriver.Chrome(ChromeDriverManager().install(),options=options) driver.get("https://www.inc.com/inc5000/2021") data=driver.page_source print(data) ...
Why do you need beautiful soup, you just could use selenium:
from selenium import webdriver from selenium.webdriver.common.by import By driver = webdriver.Chrome() driver.get("https://www.inc.com/inc5000/2021") companies = [e.text for e in driver.find_elements(By.CLASS_NAME, "company")]
This will only give you the elements in the viewport. You need to improve on that by scrolling.
Answered By – dosas