Is there an efficient way to remove duplicates 'person_id' fields from this data with python? In this case just keep the first occurrence.
{
{obj_id: 123,
location: {
x: 123,
y: 323,
},
{obj_id: 13,
location: {
x: 23,
y: 333,
},
{obj_id: 123,
location: {
x: 122,
y: 133,
},
}
Should become:
{
{obj_id: 123,
location: {
x: 123,
y: 323,
},
{obj_id: 13,
location: {
x: 23,
y: 333,
},
}
Presuming your JSON is valid syntax and you are indeed requesting help for Python you will need to do something like this
import json
ds = json.loads(json_data_string) #this contains the json
unique_stuff = { each['obj_id'] : each for each in ds }.values()
if you want to always retain the first occurrence, you will need to do something like this
all_ids = [ each['obj_id'] for each in ds ] # get 'ds' from above snippet
unique_stuff = [ ds[ all_ids.index(id) ] for id in set(ids) ]
ds is an array of dictionaries (as that's the option that makes most sense) this works well, but keeps the last occurrence instead of the first one.
.values()/set() may return objects in any order. Assuming the input is json array then it might matter. Here's order preserving algorithmobj and json_array might be better names than each and ds
json.loads(...) and json.load(...)?
json.loads(...) loads from string (loads:load string); json.load(...) loads from a file handler (or a read() supporting file-like object)Here's an implementation that preserves order of input json objects and keeps the first occurrence of objects with the same id:
import json
import sys
from collections import OrderedDict
L = json.load(sys.stdin, object_pairs_hook=OrderedDict)
seen = OrderedDict()
for d in L:
oid = d["obj_id"]
if oid not in seen:
seen[oid] = d
json.dump(seen.values(), sys.stdout, indent=2)
[
{
"obj_id": 123,
"location": {
"x": 123,
"y": 323
}
},
{
"obj_id": 13,
"location": {
"x": 23,
"y": 333
}
},
{
"obj_id": 123,
"location": {
"x": 122,
"y": 133
}
}
]
[
{
"obj_id": 123,
"location": {
"x": 123,
"y": 323
}
},
{
"obj_id": 13,
"location": {
"x": 23,
"y": 333
}
}
]
(if you had valid json)
from simplejson import loads, dumps
dumps(loads(my_json))
This is not valid JSON. On a valid JSON (Array), You can use jQuery $.each and look at the Obj_id to find and remove duplicates.
Something like this:
$.each(myArrayOfObjects, function(i, v)
{
// check for duplicate and add non-repeatings to a new array
});
python tag. Not every things in the world is jQuery.