How to extract Javascript variables from a script tag in a Scrapy response?

Question

<script type="text/javascript">

define('roomsAndRatesData', function() {
var roomsAndRatePlans = {"propertyData":{"bookingFeeMessageEnabled":true}},{"rooms":{"24-285501":{"locale":"en"}}}

};
</script>

There where other variable also define in that script. i just need data that are inside "var roomsAndRatePlans" variable can any one suggest solution to get only that variable data.

i have try other solutions but always retrieve all data inside that script tag and converting that data throws error.

so please provide me solution to get only particular data of that script

Thanks

You cannot retrieve it by XPath, you have to extract it by Regex. — thirdDeveloper
– thirdDeveloper, Commented Aug 6, 2018 at 11:33
Can you help me to create that Regex, as i don't have much idea about how to use Regex @thirdDeveloper — Milan Hirpara
– Milan Hirpara, Commented Aug 6, 2018 at 11:51
For me this looks like a nested dictionary. So try to separate the stuff after the = and then access like an dictionary — Thomas Strub
– Thomas Strub, Commented Aug 6, 2018 at 12:02

Valdir Stumm Junior · Accepted Answer · 2018-08-06 14:14:18Z

4

You can use the js2xml library:

>>> import js2xml
>>> from js2xml.utils.vars import get_vars
>>> snippet = '''
var product = {"price": 10.93, "stock": false, "name": "JBL"};
var analytics = {"id": 1234};
'''
>>> get_vars(js2xml.parse(snippet))
{
    'product': {'price': 10.93, 'stock': False, 'name': 'JBL'},
    'analytics': {'id': 1234}
}

Your JS snippet seems to be incomplete, so I used a dummy snippet here. In your case, you may first have to get the JS code via:

>>> snippet = response.css('script:contains("roomsAndRatesData")::text').get()

answered Aug 6, 2018 at 14:14

Valdir Stumm Junior

4,6671 gold badge25 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to extract Javascript variables from a script tag in a Scrapy response?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related