Python: Parse JSON string with embedded Javascript constants/variables

Question

How can I parse a JSON string that has embedded Javascript constants or variables?

For example, how to parse a JSON string like this one?

  {
    "menu": {
      "id": "file",
      "value": "File",
      "popup": {
        "menuitem": [
          {
            "value": "New",
            "onclick": Handlers.NEW
          },
          {
            "value": "Open",
            "onclick": Handlers.OPEN
          },
          {
            "value": "Custom",
            "onclick": "function(){doSomething(Handlers.OPEN);}"

          }
        ]
      }
    }
  }

All validators of course consider the JSON to be invalid, yet it is perfectly valid when evaluated in a context where the corresponding Javascript objects are defined.

The first thing that comes to mind is to pre-process the string before feeding it to the JSON parser, but that is tricky, since the same strings can occur inside existing strings (as shown in the sample JSON), and it would require some regex fiddling in order to reliably detect whether e.g. Handlers.NEW is used as an undecorated value, or inside an existing string value.

Is there a clean way to handle this use case without having to do manual regex replacements?

It is in fact not valid JSON (although it is a valid Javascript object). You might need to write your own parser for this. — L3viathan
– L3viathan, Commented Feb 23, 2016 at 17:47

Daniel · Accepted Answer · 2016-02-23 18:12:30Z

2

You can use the ast-module:

import ast

data = """{
    "menu": {
      "id": "file",
      "value": "File",
      "popup": {
        "menuitem": [
          {
            "value": "New",
            "onclick": Handlers.NEW
          },
          {
            "value": "Open",
            "onclick": Handlers.OPEN
          },
          {
            "value": "Custom",
            "onclick": "function(){doSomething(Handlers.OPEN);}"

          }
        ]
      }
    }
  }"""

def transform(item):
    if isinstance(item, ast.Dict):
        return dict(zip(map(transform,item.keys), map(transform, item.values)))
    elif isinstance(item, ast.List):
        return map(transform, item.elts)
    elif isinstance(item, ast.Str):
        return item.s
    else:
        return item

print transform(ast.parse(data).body[0].value)

answered Feb 23, 2016 at 18:12

Daniel

42.9k4 gold badges57 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

L3viathan · Accepted Answer · 2016-02-23 19:23:04Z

1

import ast

s="""{
"menu": {
  "id": "file",
  "value": "File",
  "popup": {
    "menuitem": [
      {
        "value": "New",
        "onclick": Handlers.NEW
      },
      {
        "value": "Open",
        "onclick": Handlers.OPEN
      },
      {
        "value": "Custom",
        "onclick": "function(){doSomething(Handlers.OPEN);}"

      }
    ]
  }
}
}"""

def evaluate(obj):
    if isinstance(obj, ast.Module):
        return evaluate(obj.body[0])
    elif isinstance(obj, ast.Expr):
        return evaluate(obj.value)
    elif isinstance(obj, ast.Dict):
        return {key.s: parse(value) for key, value in zip(obj.keys, obj.values)}
    elif isinstance(obj, ast.List):
        return [parse(element) for element in obj.elts]
    elif isinstance(obj, ast.Str):
        return obj.s
    elif isinstance(obj, ast.Attribute):
        return evaluate(obj.value) + "." + obj.attr
    elif isinstance(obj, ast.Name):
        return obj.id
    elif isinstance(obj, ast.Num):
        return obj.n
    else:
        print("I apparently forgot", type(obj))

x = evaluate(ast.parse(s))
print(x)

This parses the string into an Abstract Syntax Tree, and then recursively builds a Python object out of it, converting attributes into strings.

edited Feb 23, 2016 at 19:23

answered Feb 23, 2016 at 18:08

L3viathan

27.5k2 gold badges63 silver badges84 bronze badges

2 Comments

L3viathan Over a year ago

@ccpizza Thanks! If you find a case where it breaks, let me know. The only cases that would be unfixable would be an object that isn't syntactically valid Python.

ccpizza Over a year ago

works like a charm with a pretty complex JSON that is almost 1MB in size. Thank you so much!

Collectives™ on Stack Overflow

Python: Parse JSON string with embedded Javascript constants/variables

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related