Scrapy and Xpath to extract data from javascript code

Question

I am in the process of learning and simultaneously building a web spider using scrapy. I need help with extracting some information from the following javascript code:

<script language="JavaScript" type="text/javascript+gk-onload">

    SKART = (SKART) ? SKART : {};
    SKART.analytics = SKART.analytics || {};
    SKART.analytics["category"] = "television";
    SKART.analytics["vertical"] = "television";
    SKART.analytics["supercategory"] = "homeentertainmentlarge";
    SKART.analytics["subcategory"] = "television";

</script>

I wish to extract the category information as television using Xpath. Please help me with the selectors I should use.

alecxe · Accepted Answer · 2015-03-20 13:15:15Z

2

You can use the Selector's built-in support for regular expressions through re():

pattern = r'SKART\.analytics\["category"\] = "(\w+)";'
response.xpath('//script[@type="text/javascript+gk-onload"]').re(pattern)

Demo (using scrapy shell):

$ scrapy shell index.html
In [1]: pattern = r'SKART\.analytics\["category"\] = "(\w+)";'

In [2]: response.xpath('//script[@type="text/javascript+gk-onload"]').re(pattern)
Out[2]: [u'television']

answered Mar 20, 2015 at 13:15

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Scrapy and Xpath to extract data from javascript code

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related