web scraping error while using Beautiful soup and requests in python

Question

I'm trying to write the code for tracking the amazon price of a product.The code is below

import requests
from bs4 import BeautifulSoup
url='https://www.amazon.com/LunaJany-Womens-Striped-Office-Career/dp/B01DPLT4AC/ref=sxin_7_ac_d_rm?ac_md=2-2-ZHJlc3NlcyBmb3Igd29tZW4gd29yayBjYXN1YWw%3D-ac_d_rm&crid=1POYCFAFYAR8B&cv_ct_cx=dresses+for+women+casual+summer&dchild=1&keywords=dresses+for+women+casual+summer&pd_rd_i=B01DPLT4AC&pd_rd_r=0b613dda-1077-46d2-b403-af7e15840645&pd_rd_w=7Mp2P&pd_rd_wg=rNofK&pf_rd_p=a0516f22-66df-4efd-8b9a-279a864d1512&pf_rd_r=1P30PXW75XA27N3M6VDK&psc=1&qid=1592310609&sprefix=dre%2Caps%2C440&sr=1-3-12d4272d-8adb-4121-8624-135149aa9081'
        header={"user-agent":'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'}
        page=requests.get(url,headers=header)
        soup1=BeautifulSoup(page.content,"html.parser")
        soup2=BeautifulSoup(soup1.prettify(),"html.parser")
        title=soup2.find(id="productTitle").get_text()
        print(title)

While trying to print the title I'm getting error as

Traceback (most recent call last):
  File "C:/Users/Patterns/PycharmProjects/RUBI/Tracks amozon prices.py", line 8, in <module>
    title=soup2.find(id="productTitle").getText()
AttributeError: 'NoneType' object has no attribute 'getText'

Could anyone help me out??

The error shows getText but your code says get_text(), Which one are you using? — Juanje
– Juanje, Commented Jun 16, 2020 at 13:38
The posted code has .get_text(), but the error message says you're actually using .getText(). Please post your real code. — John Gordon
– John Gordon, Commented Jun 16, 2020 at 13:38
I'm sorry that previously i have used getText and i got the above error and suddenly i thought to try get_text() but forget to replace the error here. — Rubini
– Rubini, Commented Jun 16, 2020 at 13:42
Traceback (most recent call last): File "C:/Users/Patterns/PycharmProjects/RUBI/Tracks amozon prices.py", line 8, in <module> title=soup2.find(id="productTitle").get_text() AttributeError: 'NoneType' object has no attribute 'get_text' — Rubini
– Rubini, Commented Jun 16, 2020 at 13:42
lots of content is generated via JavaScript, capability that BeautifulSoup don't have. but you can get luck by scrapping the json objects on the source code, but just not from the HTML directly. — HoneyPoop
– HoneyPoop, Commented Jun 16, 2020 at 13:49

astroanax · Accepted Answer · 2020-06-16 13:55:16Z

1

It says NoneType has no attribute "get_text", implying that no matching element with the id "productTitle" was found, hence returning None. None is a NoneType object has therefore has no "get_text" attribute.

Tip - Try tweaking productTitle. I am not sure, but it might not be the element for price of the item you are trying to track the price of.

edited Jun 16, 2020 at 13:55

answered Jun 16, 2020 at 13:46

astroanax

992 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Harshit · Accepted Answer · 2020-06-16 13:56:14Z

1

Its because there is no such element like productTitle. When you use beautifull soup to load the page content there is a robot check page that loads instead. Like this one. Robot Check

Just try to print page content by - print(soup2) you will get to know the reason of error.

answered Jun 16, 2020 at 13:56

Harshit

1281 silver badge9 bronze badges

Collectives™ on Stack Overflow

web scraping error while using Beautiful soup and requests in python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related