Trying to use BeautifulSoup to learn python

Question

I am trying to use BeautifulSoup to scrape basically any website just to learn, and when trying, I never end up getting all instances of each parameter set. Attached is my code, please let me know what I'm doing wrong:

import requests

url = "https://www.newegg.com/core-i7-8th-gen-intel-core-i7-8700k/p/N82E16819117827?Item=N82E16819117827"

result = requests.get(url)
doc = BeautifulSoup(result.text, "html.parser")

price = doc.find_all(string="$")
print(price)

#### WHY DOES BEAUTIFULSOUP NOT RETURN ALL INSTANCES!?!?!?```

are you trying to know bs4 or python? i think 'use BeautifulSoup to learn python' makes little sense. — Lei Yang
– Lei Yang, Commented Jan 17, 2022 at 4:52
trying to learn python, was just watching some youtube vids where they web scraped with beautifulsoup, so I wanted to practice on it — gulade47
– gulade47, Commented Jan 17, 2022 at 9:29

Subbu VidyaSekar · Accepted Answer · 2022-01-17 06:48:31Z

1

as per the url provided in the question, I could see the price with $ symbol is available in the price-current class name.

So I have used a find_all() to get all the prices.

Use the below code:

import requests
from bs4 import BeautifulSoup

url = "https://www.newegg.com/core-i7-8th-gen-intel-core-i7-8700k/p/N82E16819117827?Item=N82E16819117827"

result = requests.get(url)
doc = BeautifulSoup(result.text, "html.parser")

price = doc.find_all(attrs={'class': "price-current"})
for p in price:
    print(p.text)

output:

$399.99

$400.33
$403.09
$412.00

edited Jan 17, 2022 at 6:48

answered Jan 17, 2022 at 0:17

Subbu VidyaSekar

2,6413 gold badges26 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Shadow Over a year ago

This answer could be improved by explaining what you did, and why. In its currently form, it does little to help the asker learn python.

codyho · Accepted Answer · 2022-01-17 04:48:52Z

I'm not sure what you mean by "all instances of each parameter set." One reason that your code block may not be working, however, is that you forgot to import the BeautifulSoup library.

from bs4 import BeautifulSoup

Also, it's not the best practice to scrape live sites. I would highly recommend the website toscrape.com. It's a really great resource for newbies. I still use it to this day to hone my scraping skills and expand them.

Lastly, BeautifulSoup works best when you have a decent grasp of HTML and CSS, especially the selector syntax. If you don't know those two, you will struggle a little bit no matter how much Python you know. The BeautifulSoup documentation can give you some insight on how to navigate the HTML and CSS if you are not well versed in those.

Collectives™ on Stack Overflow

Trying to use BeautifulSoup to learn python

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related