selenium python load more button not working

Issue

I am trying to click load more button infinite time until load all products. But my load more button not working. here is my code:

url = 'https://www.pursemall.ru/vuitton/damier-azur'
driver = webdriver.Chrome()
driver.maximize_window()
driver.get(url)


try:
    for i in range(1000):
     load_more_button = driver.find_element_by_xpath("//span[text()='Load More Products']")
     load_more_button.click()
except:
    pass
    print("task load more button completed") 

why my load more button not working? where I am doing mistake?

Solution

What you can do is the below. Each call inside the loop will fetch a set of products. You can scrape each response and collect the data.

In the browser do F12 -> Network -> XHR and see how the browser does the HTTP call every time you hit Load More Products

import requests


for page in range(1,4):
    url = f'https://www.pursemall.ru/vuitton/damier-azur?page={page}'
    print(url)
    r = requests.get(url)
    print(r.status_code)

Answered By – balderman

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0