Selenium 'list' object has no attribute 'text'

Question

So I'm using python and selenium to scrape the product titles on a sephora page.

url = 'https://www.sephora.com/ca/en/shop/face-makeup'

driver.get(url)  
time.sleep(2)
   
browser = scrollDown(driver, 20)

# this selected the div for every product on the page
products = driver.find_elements_by_class_name('css-79elbk')

for product in products:
    title = product.find_elements_by_xpath('/html/body/div[1]/div[2]/div/div/div/div[2]/div[1]/main/div[3]/div/div[1]/div[1]/div[1]/a/div/div[4]/span[2]').text
    print(title)

The problem is that when I run it I get Line 48: AttributeError: 'list' object has no attribute 'text'. The title is in a span that is nested in a div. I've tried this on a normal div with text inside and it scrapes it no problem.

Note that the function you are using finds elements by xpath (note the "s" in find_elements_by_xpath). The function returns a list, not a single object. — Shane Bishop
– Shane Bishop, Commented Jan 22, 2021 at 1:40
@ShaneBishop that got rid of the error I was mentioning but now it just returns the first product price x amount of products on the page. — nick
– nick, Commented Jan 22, 2021 at 1:51

frianH · Accepted Answer · 2021-01-22 04:07:43Z

1

The error appear because this line:

.find_elements_by_xpath('/html/body/div[1]/div[2]/div/div/div/div[2]/div[1]/main/div[3]/div/div[1]/div[1]/div[1]/a/div/div[4]/span[2]').text

The above return a list.

.text method utilize to .find_element_* (without s)

But in simple you can scrape the title by css selector with this value : div[data-comp="ProductDisplayName "] span[data-at="sku_item_name"]

Try following code:

url = 'https://www.sephora.com/ca/en/shop/face-makeup'

driver.get(url)  
time.sleep(2)
   
browser = scrollDown(driver, 20)

titles = driver.find_elements_by_css_selector('div[data-comp="ProductDisplayName "] span[data-at="sku_item_name"]')
for title in titles:
    print(title.text)

To scrape the brand, you only change selector to be : div[data-comp="ProductDisplayName "] span[data-at="sku_item_brand"]

answered Jan 22, 2021 at 4:07

frianH

7,5916 gold badges26 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

nick Over a year ago

thank you! I didn't even know you could scrape with data attributes.

Collectives™ on Stack Overflow

Selenium 'list' object has no attribute 'text'

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related