Fastest way to filter json string in Python

Question

I have a json structure like

{
    "a": "1",
    "b": "2",
    "c": {
        "d": "3"
    }
}

What I want is to only keep the 1st level of the json, i.e. remove if 1st level's value is not a string, so I have a program like

import json

s = ''' {
    "a": "1",
    "b": "2",
    "c": {
        "d": "3"
    } } '''

data = json.loads(s) 
ret = {}

for k, v in data.items():
    if (isinstance(v, basestring)):
        ret[k] = v

print json.dumps(ret)

Since I need to process huge amount of json string like that, I am looking for if any fastest way or more elegant way to do the same thing in Python

be careful when you use json string verbatim inside a Python string literal. Use raw-string literal r'' to avoid interpolating backslashes inside json. — jfs
– jfs, Commented May 5, 2014 at 17:26
if the question is about performance then you should provide a basic benchmark and determine how fast is fast enough in your case. — jfs
– jfs, Commented May 5, 2014 at 17:30

Martijn Pieters · Accepted Answer · 2014-05-05 15:27:07Z

4

Use a dict comprehension:

ret = {k: v for k, v in json.loads(s).iteritems() if isinstance(v, basestring)}

The dict.iteritems() call here prevents a full list being built first.

If your JSON input is truly huge, consider switching to an iterative JSON parser like ijson, and parse your JSON with an event-driven interface:

import ijson

ret = {}
key = None

with open(some_large_jsonfile) as json_file:
    for prefix, type, value in ijson.parse(json_file):
        if prefix and not '.' in prefix and type == 'string':
            # only top-level string values
            ret[prefix] = value

but it could be a good idea to process the key-value pairs right there and then rather than build up a full dictionary.

edited May 5, 2014 at 15:27

answered May 5, 2014 at 15:01

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Ryan Over a year ago

my json is not huge, but I have many lines of json need to process.

jfs Over a year ago

@Ryan: dict comprehension might be slower than an explicit loop (here's an example where a generator expression (related concept) is slower than an explicit for-loop). If an individual json object is small then it is not clear what would be faster a loop that uses .iteritems() or .items() (all items have to be created anyway, same logic as xrange() vs range()). Without a benchmark it is hard to say. unicode could be used instead of basestring. Do you mean that you have many small json objects (one per line) e.g., like tweet stream?

Ryan Over a year ago

@J.F.Sebastian, my json is around 4K in size (average), anyway, I will do the benchmark first. Thanks.

Collectives™ on Stack Overflow

Fastest way to filter json string in Python

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related