Scraping based on "nested property"

Question

After having created a few different spiders I thought I could scrape practically anything, but I've hit a roadblock.

Given the following code snippet:

<div class="col-md-4">
    <div class="tab-title">Homepage</div>
    <p>
        <a target="_blank" rel="nofollow" 
         href="http://www.bitcoin.org">http://www.bitcoin.org
        </a>
    </p>
</div>

How would you go about selecting the link that is in within <a ... </a> based on the text within the tab-title div?

The reason that I require that condition is because there are several other links that fit this condition:

response.css('div.col-md-4 a::attr(href)').extract()

My best guess is the following:

response.css('div.col-md-4 div.tab-title:contains("Homepage") a::attr(href)').extract()

Any insights are appreciated! Thank you in advance.

Note: I am using Scrapy.

Tomáš Linhart · Accepted Answer · 2018-02-15 20:58:43Z

2

How about this using XPath:

response.xpath('//div[@class="tab-title" and contains(., "Homepage")]/..//a/@href')

Find a div with class tab-title which contains Homepage inside, then step up to the parent and look for a child on any level.

EDIT: Using CSS, you should be able to do it like this:

response.css('div.tab-title:contains("Homepage") ~ * a::attr(href)')

edited Feb 15, 2018 at 20:58

answered Feb 15, 2018 at 7:00

Tomáš Linhart

10.2k1 gold badge30 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

vcovo Over a year ago

Thank you. Can you "step up" using css selectors or is that functionality limited to XPath selectors?

vcovo Over a year ago

Very nice, I was unaware of the ~ and * symbols for css selectors. If you don't mind would you briefly explain what they do with regards to this example?

Tomáš Linhart Over a year ago

Basically it says to select all elements preceded by div with class tab-title and containing Homepage string. Check out the reference at W3Schools.

vcovo Over a year ago

I was referring to the symbols, but thank you regardless.

Collectives™ on Stack Overflow

Scraping based on "nested property"

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related