Extract particular fields from json in python

Question

Say I have a lot of json lines to process and I only care about the specific fields in a json line.

{blablabla, 'whatICare': 1, blablabla}
{blablabla, 'whatICare': 2, blablabla}
....

Is there any way to extract whatICare from these json lines withoud loads them? Since the json lines are very long it may be slow to build objects from json..

Another option! if you have huge Json file. Store the Json file to MYSQL DB, you can optimize your queries to get what you care about. However, I am not sure if it's the best way to do it comparing the below-mentioned options. — user3378649
– user3378649, Commented Oct 21, 2014 at 22:41

wim · Accepted Answer · 2014-10-21 22:32:45Z

2

Not any reliable way without writing your own parsing code.

But check out ujson! It can be 10x faster than python's built in json library, which is a bit on the slow side.

answered Oct 21, 2014 at 22:32

wim

368k114 gold badges681 silver badges816 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

poke Over a year ago

In my previous experience with large JSON data, decoding was actually fast enough; the bottleneck was still reading the file from disk.

poke · Accepted Answer · 2014-10-21 22:35:43Z

0

No, you will have to load and parse the JSON before you know what’s inside and to be able to filter out the desired elements.

That being said, if you worry about memory, you could use ijson which is an iterative parser. Instead of loading all the content at once, it is able to load only what’s necessary for the next iteration. So if you your file contains an array of objects, you can load and parse one object at a time, reducing the memory impact (as you only need to keep one object in memory, plus the data you actually care about). But it won’t become faster, and it also won’t magically skip data you are not interested in.

answered Oct 21, 2014 at 22:35

poke

392k80 gold badges596 silver badges632 bronze badges

Collectives™ on Stack Overflow

Extract particular fields from json in python

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related