Python List comprehension and JSON parsing

Question

I'm new to Python and trying to figure out the best way to parse the values of a JSON object into an array, using a list comprehension.

Here is my code - I'm querying the publicly available iNaturalist API and would like to take the JSON object that it returns, so that I take specific parts of the JSON object into a bumpy array:

import json
import urllib2

#Set Observations URL request for Resplendent Quetzal of Costa Rica
query = urllib2.urlopen("http://api.inaturalist.org/v1/observations?place_id=6924&taxon_id=20856&per_page=200&order=desc&order_by=created_at")
obSet = json.load(query)

#Print out Lat Long of observation
n = obSet['total_results']

for i in range(n) :
    print obSet['results'][i]['location']

This all works fine and gives the following output:

9.5142456535,-83.8011438905
10.2335478381,-84.8517773638
10.3358965682,-84.9964271008
10.3744851815,-84.9871494128
10.2468720343,-84.9298072822
...

What I'd like to do next is replace the for loop with a list comprehension, and store the location value in a tuple. I'm struggling with the syntax in that I'm guessing it's something like this:

[(long,lat) for i in range(n) for (long,lat) in obSet['results'][i]['location']]

But this doesn't work...thanks for any help.

Also as you're new to Python, is there any particular reason for you to use Python 2, now that Python 3 has been out for 8 years? — Antti Haapala
– Antti Haapala, Commented Aug 28, 2016 at 8:22

Antti Haapala · Accepted Answer · 2016-08-28 08:30:56Z

6

obSet['results'] is a list, no need to use range to iterate over it:

for item in obSet['results']:
    print(item['location'])

To make this into list comprehension you can write:

[item['location'] for item in obSet['results']]

But, each location is coded as a string, instead of list or tuple of floats. To get it to the proper format, use

[tuple(float(coord) for coord in item['location'].split(','))
 for item in obSet['results']]

That is, split the item['location'] string into parts using , as the delimiter, then convert each part into a float, and make a tuple of these float coordinates.

answered Aug 28, 2016 at 8:30

Antti Haapala

135k23 gold badges297 silver badges349 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

DeepSpace · Accepted Answer · 2016-08-28 08:27:55Z

2

You can iterate over the list of results directly:

print([tuple(result['location'].split(',')) for result in obSet['results']])
>> [('9.5142456535', '-83.8011438905'), ('10.2335478381', '-84.8517773638'), ... ]

edited Aug 28, 2016 at 8:27

answered Aug 28, 2016 at 8:20

DeepSpace

82.1k12 gold badges119 silver badges166 bronze badges

3 Comments

Ahasanul Haque Over a year ago

you are returning a list of string, not list of tuple.

Ahasanul Haque Over a year ago

What I'd like to do next is replace the for loop with a list comprehension, and store the location value in a tuple. - This is OP's requirement.

DeepSpace Over a year ago

@AhsanulHaque Fixed.

Martijn Pieters · Accepted Answer · 2016-08-28 08:34:00Z

The direct translation of your code into a list comprehension is:

positions = [obSet['results'][i]['location'] for i in range(obSet['total_results'])]

The obSet['total_results'] is informative but not needed, you could just loop over obSet['results'] directly and use each resulting dictionary:

positions = [res['location'] for res in obSet['results']]

Now you have a list of strings however, as each 'location' is still the long,lat formatted string you printed before.

Split that string and convert the result into a sequence of floats:

positions = [map(float, res['location'].split(',')) for res in obSet['results']]

Now you have a list of lists with floating point values:

>>> [map(float, res['location'].split(',')) for res in obSet['results']]
[[9.5142456535, -83.8011438905], [10.2335478381, -84.8517773638], [10.3358965682, -84.9964271008], [10.3744851815, -84.9871494128], [10.2468720343, -84.9298072822], [10.3456659939, -84.9451804822], [10.3611732346, -84.9450302597], [10.3174360636, -84.8798676791], [10.325110706, -84.939710318], [9.4098152454, -83.9255607577], [9.4907141714, -83.9240819199], [9.562637289, -83.8170178428], [9.4373885911, -83.8312881263], [9.4766746409, -83.8120952573], [10.2651190176, -84.6360466565], [9.6572995298, -83.8322965118], [9.6997991784, -83.9076919066], [9.6811177044, -83.8487647156], [9.7416717045, -83.929327673], [9.4885099275, -83.9583968683], [10.1233252667, -84.5751029683], [9.4411815757, -83.824401543], [9.4202687169, -83.9550344212], [9.4620656621, -83.665183105], [9.5861809119, -83.8358881552], [9.4508914243, -83.9054016165], [9.4798058284, -83.9362558497], [9.5970449879, -83.8969131893], [9.5855562829, -83.8354434596], [10.2366179555, -84.854847472], [9.718459702, -83.8910277016], [9.4424384874, -83.8880459793], [9.5535916157, -83.9578166199], [10.4124554163, -84.9796942349], [10.0476688795, -84.298227929], [10.2129436252, -84.8384097435], [10.2052632717, -84.6053701877], [10.3835784147, -84.8677930134], [9.6079669672, -83.9084281155], [10.3583643315, -84.8069762134], [10.3975986735, -84.9196996767], [10.2060835381, -84.9698814407], [10.3322929317, -84.8805587129], [9.4756504472, -83.963818143], [10.3997876964, -84.9127311339], [10.1777433853, -84.0673088686], [10.3346128571, -84.9306278215], [9.5193346195, -83.9404786293], [9.421538224, -83.7689452093], [9.430427837, -83.9532672942], [10.3243212895, -84.9653175843], [10.021698503, -83.885674888]]

If you must have tuples rather than lists, add a tuple() call:

positions = [tuple(map(float, res['location'].split(',')))
             for res in obSet['results']]

The latter also makes sure the expression works in Python 3 (where map() returns an iterator, not a list); you'd otherwise have to use a nested list comprehension:

# produce a list of lists in Python 3
positions = [[float(p) for p in res['location'].split(',')] for res in obSet['results']]

Thanks for teaching OP about the use of map just after I persuaded him to try out Python 3 ;)
@AnttiHaapala: well, the tuple() call will sort that out nicely ;-)

Nehal J Wani · Accepted Answer · 2016-08-28 08:25:45Z

Another way to get list of [long, lat] without list comprehension:

In [14]: map(lambda x: obSet['results'][x]['location'].split(','), range(obSet['total_results']))
Out[14]: 
[[u'9.5142456535', u'-83.8011438905'],
 [u'10.2335478381', u'-84.8517773638'],
 [u'10.3358965682', u'-84.9964271008'],
 [u'10.3744851815', u'-84.9871494128'],
...

If you would like list of tuples instead:

In [14]: map(lambda x: tuple(obSet['results'][x]['location'].split(',')), range(obSet['total_results']))
Out[14]: 
[[u'9.5142456535', u'-83.8011438905'],
 [u'10.2335478381', u'-84.8517773638'],
 [u'10.3358965682', u'-84.9964271008'],
 [u'10.3744851815', u'-84.9871494128'],
...

If you want to convert to floats too:

In [17]: map(lambda x: tuple(map(float, obSet['results'][x]['location'].split(','))), range(obSet['total_results']))
Out[17]: 
[(9.5142456535, -83.8011438905),
 (10.2335478381, -84.8517773638),
 (10.3358965682, -84.9964271008),
 (10.3744851815, -84.9871494128),
 (10.2468720343, -84.9298072822),
 (10.3456659939, -84.9451804822),
 ...

syntagma · Accepted Answer · 2016-08-28 08:29:18Z

1

To correct way to get a list of tuples using list comprehensions would be:

def to_tuple(coords_str):
    return tuple(coords_str.split(','))

output_list = [to_tuple(obSet['results'][i]['location']) for i in range(obSet['total_results'])]

You can of course replace to_tuple() with a lambda function, I just wanted to make the example clear. Moreover, you could use map() to have a tuple with floats instead of string: return tuple(map(float,coords_str.split(','))).

answered Aug 28, 2016 at 8:29

syntagma

24.6k17 gold badges85 silver badges142 bronze badges

Comments

Ahasanul Haque · Accepted Answer · 2016-08-28 08:33:21Z

1

[tuple(obSet['results'][i]['location'].split(',')) for i in range(n)]

This will return a list of tuple, elements of the tuples are unicode.

If you want that the elements of tuples as floats, do the following:

[tuple(map(float,obSet['results'][i]['location'].split(','))) for i in range(n)]

edited Aug 28, 2016 at 8:33

answered Aug 28, 2016 at 8:20

Ahasanul Haque

11.2k6 gold badges45 silver badges61 bronze badges

Comments

Alex S. · Accepted Answer · 2016-08-28 08:35:55Z

Let's try to give this a shot, starting with just 1 location:

>>> (long, lat) = obSet['results'][0]['location']

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: too many values to unpack

Alright, so that didn't work, but why? It's because the longitude and latitude coordinates are just 1 string, so you can't unpack it immediately as a tuple. We must first separate it into two different strings.

>>> (long, lat) = obSet['results'][0]['location'].split(",")

From here we will want to iterate through the whole set of results, which we know are indexed from 0 to n. tuple(obSet['results'][i]['location'].split(",")) will give us the tuple of longitude, latitude for the result at index i, so:
>>> [tuple(obSet['results'][i]['location'].split(",")) for i in range(n)]
ought to give us the set of tuples we want.

Collectives™ on Stack Overflow

Python List comprehension and JSON parsing

7 Answers 7

Comments

3 Comments

2 Comments

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

Comments

3 Comments

2 Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related