Using multiple CSS selectors for the same ArticleItem in Scrapy

Question

The site I am scraping has an inconsistent layout. I'm currently using this but its not returning all the titles -

article['title'] = sel.css('p[class=title] ::text').extract()

I need to use this to scrape span classes also -

article['title'] = sel.css('span[class=newstitle] ::text').extract()

Is there a way to combine two css selectors in a single ArticleItem?

Arthur Burkhardt · Accepted Answer · 2014-07-29 20:01:00Z

4

As simple as list concatenation:

article['title'] = response.css("p.title ::text").extract() + \
                   response.css("span.newstitle ::text").extract()

answered Jul 29, 2014 at 20:01

Arthur Burkhardt

7004 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

an even simpler: response.css("p.title ::text, span.newstitle ::text").extract()