Extracting string data from a html source

Question

from bs4 import BeautifulSoup
import urllib.request

page = urllib.request.urlopen('https://www.applied.com/categories/bearings/accessories/adapter-sleeves/c/1580?q=%3Arelevance&page=1')
html = page.read()
soup = BeautifulSoup(html)

items = soup.find_all(class_= 'product product--list ')

for i in items[0:1]:
    product_name = i.find(class_="product__name").a.string.strip()
    print(product_name)
    product_url = i.find(class_="product__name").a['href']
    print(product_url)
    price = i.find(itemprop="price").string
    print(price)

Using the above code I tried to get the price for each product in that page. But when i tried, the output for price variable is showing as none.

When I inspect the html source for the price in a browser it is showing the price as a normal text as how I got for product_name variable.

Can someone guide me on how to get the price for the products in that page.

I've tested your code: the product price is initially empty in page content: <span class="price" itemprop="price"> <span itemprop="offers" itemscope="" itemtype="http://schema.org/Offer"> <span itemprop="price"> </span>. Voting to close — RomanPerekhrest
– RomanPerekhrest, Commented Sep 18, 2017 at 7:27
The price is loaded by Javascript. Try this: stackoverflow.com/questions/8049520/… — SajidSalim
– SajidSalim, Commented Sep 18, 2017 at 7:38

Himanshu dua · Accepted Answer · 2017-09-18 07:39:49Z

1

Price is loaded by Ajax(https://www.applied.com/getprices) after page is loaded that's why it is not in HTML.

Use https://www.applied.com/getprices to get the price of an item You have to send post request with following params for getting the price of the product.

{
  "productCodes": "100731658",
  "page": "PLP",
  "productCode": "100731658",
  "CSRFToken": "172c7073-742f-4d7d-9c97-358e0d9e631e"
}

answered Sep 18, 2017 at 7:39

Himanshu dua

2,5141 gold badge22 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Extracting string data from a html source

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related