Python Requests HTML - Result of found item doesn't return content

Question

Hi im scraping a webpage with my script, the problem is only one item (title) can be found correctly, other items only throw back html when grabbed like that:

[<Element 'div' class=('rd-product-description__top-accordion-content-description',)>]

My Script:

from requests_html import HTMLSession

url = 'https://www.kaufland.de/product/324406424/?search_value=staubsauger'
s = HTMLSession()
r = s.get(urlos)
r.html.render(sleep=1)
titel  = str(r.html.find('h1.rd-title')).replace("[<Element 'h1' title='", "").replace("' class=('rd-title',)>]", "")
price = r.html.find('span.rd-price-information__price')
description = r.html.find('div.rd-product-description__top-accordion-content-description', first=True)
print(description)

The Webpage to scrape: https://www.kaufland.de/product/324406424/?search_value=staubsauger

What is the reason i cant get anything out of it? Trying it with .text also doesnt work, as the element has no text object.

Andrej Kesely · Accepted Answer · 2022-07-05 14:56:55Z

The data you see on the page is stored inside <script> and processed by JavaScript. To get price, description, etc. you can use js2py module. For example:

import re
import js2py
import requests
from bs4 import BeautifulSoup


url = "https://www.kaufland.de/product/324406424/?search_value=staubsauger"

data = re.search(r"__PDPFRONTEND__=(.*?\));", requests.get(url).text).group(1)
data = js2py.eval_js(data)

price = data["state"]["product-store"]["price$"]
desc = data["state"]["product-store"]["description$"]["descriptionHtml"]

desc = BeautifulSoup(desc, "html.parser")

print(price)
print()
print(desc.get_text(strip=True))

Prints:

119.99

Bosch BGN2CHAMP Bodenstaubsauger 600W Aktionsradius 9m Hygienefilter Umschaltbare Rollendüse FilterwechselanzeigeHiSpin MotorStaubsaugen klappt mit einem starken Motor mühelos. Der leistungsstarke und präzise kalibrierte HiSpin Motor sorgt hier für eine gründliche Reinigungsleistung bei geringem Energieverbrauch.Super kompaktStaubsaugen ist jetzt viel einfacher geworden! Dieses kompakte, kleine und leichte Gerät macht 

...and so on.

Collectives™ on Stack Overflow

Python Requests HTML - Result of found item doesn't return content

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related