JSON list/dictionary parsing from API

Question

I have developed a small library and am interested in making it easier on users to retrieve data from the JSON lists/dictionaries that are returned. I have created functions that handle the calls using requests. Now suppose I call this function and pass in a few parameters:

precip = precipitation_obs(stid='kfnl', start='201504261800', end='201504271200', units='precip|in')

This will return the following JSON:

{ 'STATION': [ { 'ELEVATION': '5016',
                 'ID': '192',
                 'LATITUDE': '40.45',
                 'LONGITUDE': '-105.01667',
                 'MNET_ID': '1',
                 'NAME': 'Fort Collins/Loveland, Fort Collins-Loveland '
                         'Municipal Airport',
                 'OBSERVATIONS': { 'count_1': 6,
                                   'ob_end_time_1': '2015-04-27T00:55:00Z',
                                   'ob_start_time_1': '2015-04-26T18:55:00Z',
                                   'total_precip_value_1': 0.13,
                                   'vids case4': ['39', '51', '40', '52']},
                 'STATE': 'CO',
                 'STATUS': 'ACTIVE',
                 'STID': 'KFNL',
                 'TIMEZONE': 'US/Mountain'}],
  'SUMMARY': { 'METADATA_RESPONSE_TIME': '5.22613525391 ms',
               'NUMBER_OF_OBJECTS': 1,
               'RESPONSE_CODE': 1,
               'RESPONSE_MESSAGE': 'OK',
               'TOTAL_TIME': '57.6429367065 ms'}}

Now, I want the user to be able to just drill down through the dictionary, but STATION is a list and requires me to do the following:

output =  precip['STATION'][0]['OBSERVATIONS']['ob_start_time_1'] 
print(output)
# returns 2015-04-26T18:55:00Z

where I have to include the [0] to avoid:

TypeError: list indices must be integers, not str

Is there anyway around this? Adding that [0] in there really jacks things up so to say. Or even having to specify ['STATION'] every time is a bit of a nuisance. Should I use the simpleJSON module to help here? Any tips on making this a bit easier would be great, thanks!

If this is your API, why not simplify the output then? Is there only ever one station, then don't use a list. If accessing the station elements is common, and there is just one station, don't use a nested structure but move all keys to the top-level object, etc. — Martijn Pieters
– Martijn Pieters, Commented Apr 27, 2015 at 19:57
No simplejson is not going to help here, because it just parses JSON to the corresponding Python structure. — Martijn Pieters
– Martijn Pieters, Commented Apr 27, 2015 at 19:57
And conversely, if there can be more than one station, you can't get rid of the [0] step, because if there are 3 stations, you need to be able to access #0, #1, and #2. You could change that list into a dict keyed by, say, each station's 'STID' value (so it would be precip['STATION']['KFNL']['OBSERVATIONS']['ob_start_time_1']), but you can't get rid of it. — abarnert
– abarnert, Commented Apr 27, 2015 at 20:03

abarnert · Accepted Answer · 2015-04-27 20:12:36Z

2

Adding that [0] in there really jacks things up so to say. Or even having to specify ['STATION'] every time is a bit of a nuisance.

So just store precip['STATION'][0] in a variable:

>>> precip0 = precip['STATION'][0]

And now, you can use it repeatedly:

>>> precip0['OBSERVATIONS']['ob_start_time_1']
2015-04-26T18:55:00Z

If you know that the API is always going to return exactly one station, and you're never going to want anything other than the data for that station, you can put that in your wrapper function:

def precipitation_obs(stid, start, end, units):
    # your existing code, which assigns something to result
    return result['STATION'][0]

If you're worried about "efficiency" here, don't worry. First, this isn't making a copy of anything, it's just making another reference to the same object that already exists—it takes under a microsecond, and wastes about 8 bytes. In fact, it saves you memory, because if you're not storing the whole dict, just the sub-dict, Python can garbage-collect the rest of the structure. And, more importantly, that kind of micro-optimization isn't worth worrying about in the first place until (1) your code is working and (2) you know it's a bottleneck.

Should I use the simpleJSON module to help here?

Why would you think that would help? As its readme says:

simplejson is the externally maintained development version of the json library included with Python 2.6 and Python 3.0, but maintains backwards compatibility with Python 2.5.

In other words, it's either the same code you've already got in the stdlib, or an older version of that code.

Sometimes a different library, like ijson, can help—e.g., if the JSON structures are so big that you can't parse the whole thing into memory, or so complex that it's easier to describe what you want upside-down in SAX style. But that's not relevant here.

edited Apr 27, 2015 at 20:12

answered Apr 27, 2015 at 20:06

abarnert

368k54 gold badges626 silver badges691 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

jclark754 Over a year ago

Thank you kindly for a thorough response. First bit of code I've written that involves JSON/making API calls. I appreciate the lesson!

abarnert Over a year ago

@jclark754: There's a Firefox extension that an old coworker of mine used to use. It displays the JSON document as a tree, lets you click anywhere you want, guides you through a path from there up to the top, and then gives you the (JavaScript, not Python… but it's similar) code for the indexing and/or iteration steps to get there. Unfortunately, I can't remember what it was called, but you may want to look for something like that. He found it helpful to speed up his development even as a JS wiz (although I didn't), and as a novice it might be even more helpful to visualize what's happening.

Collectives™ on Stack Overflow

JSON list/dictionary parsing from API

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related