scraping amazon deals page 40 product but get 15 product


I want to scrape all products(40) from deals page in amazon but just get 16 one I searched a lot and found that I should use scroll but I got the same value and scroll didn’t work


# -*- coding: utf-8 -*-
import requests
import time
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Chrome(executable_path='C:\\Users\\Compu City\\Desktop\\chromedriver.exe')

res = driver.execute_script("return document.documentElement.outerHTML",'window.scrollBy(0,2000)')
soup = BeautifulSoup(res , 'lxml')
for x in soup.find_all('a',{'class':'a-size-base a-link-normal dealTitleTwoLine singleCellTitle autoHeight'}):
    for y in x.find_all('span',{'class':'a-declarative'}):
        print('\n >>>'+y.text+'\n')


You can use the below css to get all 40 items.

div[class^='a-section a-spacing-none tallCellView gridColumn']
# below is the line to get all the products"div[class^='a-section a-spacing-none tallCellView gridColumn']")


enter image description here

Answered By – supputuri

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply

(*) Required, Your email will not be published