Parsing variable data out of a javascript tag using python

Question

I am scraping some websites using BeautifulSoup and Requests. There is one page that I am examining that has its data inside of a <script language="JavaScript" type="text/javascript"> tag. It looks like this:

<script language="JavaScript" type="text/javascript">
var page_data = {
   "default_sku" : "SKU12345",
   "get_together" : {
      "imageLargeURL" : "http://null.null/pictures/large.jpg",
      "URL" : "http://null.null/index.tmpl",
      "name" : "Paints",
      "description" : "Here is a description and it works pretty well",
      "canFavorite" : 1,
      "id" : 1234,
      "type" : 2,
      "category" : "faded",
      "imageThumbnailURL" : "http://null.null/small9.jpg"
       ......

Is there a way that I can create a python dictionary or json object out of the page_data variable within this script tag? That would be much nicer then trying to obtain values with BeautifulSoup.

Martijn Pieters · Accepted Answer · 2018-09-06 15:39:36Z

24

If you use BeautifulSoup to get the contents of the <script> tag, the json module can do the rest with a bit of string magic:

 jsonValue = '{%s}' % (textValue.partition('{')[2].rpartition('}')[0],)
 value = json.loads(jsonValue)

The .partition() and .rpartition() combo above split the text on the first { and on the last } in the JavaScript text block, which should be your object definition. By adding the braces back to the text we can feed it to json.loads() and get a python structure from it.

This works because JSON is basically the Javascript literal syntax objects, arrays, numbers, booleans and nulls.

Demonstration:

>>> import json
>>> text = '''
... var page_data = {
...    "default_sku" : "SKU12345",
...    "get_together" : {
...       "imageLargeURL" : "http://null.null/pictures/large.jpg",
...       "URL" : "http://null.null/index.tmpl",
...       "name" : "Paints",
...       "description" : "Here is a description and it works pretty well",
...       "canFavorite" : 1,
...       "id" : 1234,
...       "type" : 2,
...       "category" : "faded",
...       "imageThumbnailURL" : "http://null.null/small9.jpg"
...    }
... };
... '''
>>> json_text = '{%s}' % (text.partition('{')[2].rpartition('}')[0],)
>>> value = json.loads(json_text)
>>> value
{'default_sku': 'SKU12345', 'get_together': {'imageLargeURL': 'http://null.null/pictures/large.jpg', 'URL': 'http://null.null/index.tmpl', 'name': 'Paints', 'description': 'Here is a description and it works pretty well', 'canFavorite': 1, 'id': 1234, 'type': 2, 'category': 'faded', 'imageThumbnailURL': 'http://null.null/small9.jpg'}}
>>> import pprint
>>> pprint.pprint(value)
{'default_sku': 'SKU12345',
 'get_together': {'URL': 'http://null.null/index.tmpl',
                  'canFavorite': 1,
                  'category': 'faded',
                  'description': 'Here is a description and it works pretty '
                                 'well',
                  'id': 1234,
                  'imageLargeURL': 'http://null.null/pictures/large.jpg',
                  'imageThumbnailURL': 'http://null.null/small9.jpg',
                  'name': 'Paints',
                  'type': 2}}

edited Sep 6, 2018 at 15:39

answered Nov 8, 2012 at 21:36

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

2rs2ts Over a year ago

I'd love to know how I can repurpose this for an object declaration which does not use quotes to denote the keys of the object, e.g. default_sku: "SKU12345", .... It'd probably just take a regex...

Martijn Pieters Over a year ago

@2rs2ts: See Issue with html tags while scraping data using beautiful soup for a previous answer that adds in quotes to make something valid JSON.

2rs2ts Over a year ago

Although that regex didn't work for me, I see that using a regex is definitely the appropriate approach. Thank you :)

Martijn Pieters Over a year ago

Yes, that regex is somewhat specific; it doesn't allow for whitespace between the opening { or the comma. It also assumes that there are no opening curly braces and commas in the string values anywhere. Add in some whitespace allowance (\s*) and it maintain the assumption about { and , not appearing in values, and you should be able to use a regular expression to turn a Javascript object into JSON.

2rs2ts Over a year ago

Precisely the obstacles I encountered!

Collectives™ on Stack Overflow

Parsing variable data out of a javascript tag using python

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related