Scrapy xpath not working - only in combination with css-selector?

Question

i try to scrape the following site with scrapy and try something with the scrapy shell -

This is the basis spider:

import scrapy

class ZoosSpider(scrapy.Spider):
    name = 'zoos'
    allowed_domains = ['https://www.tripadvisor.co.uk/Attractions-g186216-Activities-c48-a_allAttractions.true-United_Kingdom.html']
    start_urls = ['http://https://www.tripadvisor.co.uk/Attractions-g186216-Activities-c48-a_allAttractions.true-United_Kingdom.html/']

    def parse(self, response):
        tmpSEC = response.xpath("//section[@data-automation='AppPresentation_SingleFlexCardSection']")
        for elem in tmpSEC:
          pass

I get all relevant sections with this xpath: (when i try len(tmpSEC) i get 30 which seems ok for me)

tmpSEC = response.xpath("//section[@data-automation='AppPresentation_SingleFlexCardSection']")

Now i want to extract the very first href-tag and tried it with this xpath: (but with that i only get "/" as result)

>>> tmpSEC[0].xpath("//a/@href").get()  
'/'

and also with

>>> tmpSEC[0].xpath("(//a)[1]/@href").get()  
'/'

but only with an css-selector this is working fine

>>> tmpSEC[0].css("a::attr(href)").get() 
'/Attraction_Review-g186332-d216481-Reviews-Blackpool_Zoo-Blackpool_Lancashire_England.html'

Why is this only working with an css-selector and not with an xpath-selector?

Md. Fazlul Hoque · Accepted Answer · 2021-11-16 22:12:26Z

2

Here is the working solution using xpath. You need to inject dot(.) like as follows:

import scrapy


class ZoosSpider(scrapy.Spider):
    name = 'zoos'
    
    start_urls = ['https://www.tripadvisor.co.uk/Attractions-g186216-Activities-c48-a_allAttractions.true-United_Kingdom.html/']

    def parse(self, response):
        tmpSEC = response.xpath(
            "//section[@data-automation='AppPresentation_SingleFlexCardSection']")
        #for elem in tmpSEC:
        yield {
            'link':tmpSEC[0].xpath(".//a/@href").get() 
            }

Output:

{'link': '/Attraction_Review-g186332-d216481-Reviews-Blackpool_Zoo-Blackpool_Lancashire_England.html'}

answered Nov 16, 2021 at 22:12

Md. Fazlul Hoque

16.2k5 gold badges15 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Scrapy xpath not working - only in combination with css-selector?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related