Scrapy returning empty lists when using css

Question

I am trying to scrape nordstrom product descriptions. I got all the item links (stored in local mongodb db) and now am itertating through them and here is an example link https://www.nordstrom.ca/s/leith-ruched-body-con-tank-dress/5420732?origin=category-personalizedsort&breadcrumb=Home%2FWomen%2FClothing%2FDresses&color=001

My code for the spider is:

def parse(self, response):
    items = NordstromItem()
    description = response.css("div._26GPU").css("div::text").extract()
    items['description'] = description
    yield items

I also tried scrapy shell and the returned page is blank. I am also using scrapy random agents.

“I also tried scrapy shell and the returned page is blank”. Sounds like you are getting detected as a bot and given a bad response. That’s what you need to fix first. — Gallaecio
– Gallaecio, Commented May 6, 2020 at 9:51

Karla · Accepted Answer · 2020-05-02 19:22:23Z

1

I suggest you to use css or xpath selector to get the info you want. Here's more about it: https://docs.scrapy.org/en/latest/topics/selectors.html

And you can also use css/xpath checker to help identify if the selector gets the info you want. Like this Chrome extesion: https://autonomiq.io/chropath/

edited May 2, 2020 at 19:22

answered May 2, 2020 at 19:16

Karla

961 gold badge1 silver badge6 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Scrapy returning empty lists when using css

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related