Python web scraping, using html-requests to find a specific element and extract text

Question

I am using python for webscraping (new to this) and am trying to grab the brand name from a website. It is not visible on the website but I have found the element for it:

<span itemprop="Brand" style="display:none;">Revlon</span>

I want to extract the "Revlon" text in the HTML. I am currently using html requests and have tried grabbing the selector (CSS) and text:

brandname = r.html.find('body > div:nth-child(96) > span:nth-child(2)', first=True).text.strip()

but this returns None and an error. I am not sure how to extract this specifically. Any help would be appreciated.

The website URL is:boots.com/revlon-colorstay-makeup-for-normal-dry-skin-10212694 — need_halp
– need_halp, Commented Apr 15, 2021 at 13:00

Adil kasbaoui · Accepted Answer · 2021-04-15 13:20:09Z

3

Here is a working solution with Selenium:

from seleniumwire import webdriver
from webdriver_manager.chrome import ChromeDriverManager


driver = webdriver.Chrome(ChromeDriverManager().install())

website = 'https://www.boots.com/revlon-colorstay-makeup-for-normal-dry-skin-10212694'

driver.get(website)

brand_name = driver.find_element_by_xpath('//*[@id="estore_product_title"]/h1')

print('brand name: '+brand_name.text.split(' ')[0])

You can also use beautifulsoup for that:

from bs4 import BeautifulSoup
import requests


urlpage = 'https://www.boots.com/revlon-colorstay-makeup-for-normal-dry-skin-10212694'

# query the website and return the html to the variable 'page'
page = requests.get(urlpage)
# parse the html using beautiful soup and store in variable 'soup'
soup = BeautifulSoup(page.content, 'html.parser')
name = soup.find(id='estore_product_title')
print(name.text.split(' ')[0])

edited Apr 15, 2021 at 13:20

answered Apr 15, 2021 at 13:12

Adil kasbaoui

6738 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

MK Developer · Accepted Answer · 2021-04-15 13:26:24Z

1

try this method .find("span", itemprop="Brand") I think it's work

from bs4 import BeautifulSoup
import requests


urlpage = 'https://www.boots.com/revlon-colorstay-makeup-for-normal-dry-skin-10212694'

page = requests.get(urlpage)
# parse the html using beautiful soup and store in variable 'soup'
soup = BeautifulSoup(page.content, 'html.parser')
print(soup.find("span", itemprop="Brand").text)

edited Apr 15, 2021 at 13:26

answered Apr 15, 2021 at 13:07

MK Developer

362 bronze badges

1 Comment

yav dat Over a year ago

This approach does not work with JavaScript created or dynamic pages. What should we do if we used r.html.render and then r.html.find?

Collectives™ on Stack Overflow

Python web scraping, using html-requests to find a specific element and extract text

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related