Python : find_all() return an empty list

Question

I'm trying to make a bot that send me an email once a new product is online on a website.

I tried to do that with requests and beautifulSoup.

This is my code :

import requests
from bs4 import BeautifulSoup

URL = 'https://www.vinted.fr/vetements?search_text=football&size_id[]=207&price_from=0&price_to=15&order=newest_first'

headers = {'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36"}

page = requests.get(URL, headers=headers)

soup = BeautifulSoup(page.content, 'html.parser')

products = soup.find_all("div", class_="c-box")

print(len(products))

Next, I'll want to compare the number of products before and after my new request in a loop.

But when I try to see the number of products that I found, I get an empty list : []

I don't know how to fix that ...

The div that I use is in others div, I don't know if it has a relation

Thanks by advance

I'm getting 403 when I ran your script. So, probably the soup object doesn't have the required dom to parse — bigbounty
– bigbounty, Commented Jul 23, 2020 at 23:42

cylee · Accepted Answer · 2020-07-23 23:36:45Z

1

You have problem with the website that you are trying to parse.

The website in your code generates elements you are looking for(div.c-box) after the website is fully loaded, using javascript, at the client-side. So it's like:

Browser gets HTML source from server --(1)--> JS files loaded as browser loads html source --> JS files add elements to the HTML source --(2)--> Those elements are loaded to the browser

You cannot fetch the data you want by requests.get because requests.get method can only get HTML source at point (1), but the website loads the data at (2) point. To fetch such data, you should use automated browser modules such as selenium.

answered Jul 23, 2020 at 23:36

cylee

5701 gold badge6 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Bastien Over a year ago

Thanks, I'm gonna see tutorials for selenium !

eren · Accepted Answer · 2020-07-24 00:01:36Z

1

You should always check the data.

Convert your BeautifulSoup object to string with soup.decode('utf-8') and write it on a file. Then check what you get from the website. In this case, there is no element with c-box class.

You should use selenium instead of requests.

edited Jul 24, 2020 at 0:01

answered Jul 23, 2020 at 23:47

eren

311 silver badge5 bronze badges

Collectives™ on Stack Overflow

Python : find_all() return an empty list

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related