Query and update JSON within Python

As Scott indicates, the object you've got there is not strictly speaking "a JSON object". It's a perfectly ordinary Python dict containing a perfectly ordinary list which contains perfectly ordinary dicts, so you can manipulate it using ordinary iteration/indexed assignment/etc, no frameworks required.

d = {
  "people": [
    {
      "general": {
        "id": 100,
        "age": 20
      },
      "history": {
      }
    },
    {
      "general": {
        "id": 101,
        "age": 30
      },
      "history": {
      }
    },
    {
      "general": {
        "id": 100,
        "age": 30
      },
      "history": {
      }
    }
  ]
}

#add new person
d["people"].insert(0, {
    "general": {
        "id": 100,
        "age": 20,
    },
    "history": {}
})

#copy `id` over to `identifier` for each person,
#and delete `id`
for person in d["people"]:
    person["general"]["identifier"] = person["general"]["id"]
    del person["general"]["id"]

print(d)

Result:

{'people': [{'general': {'age': 20, 'identifier': 100}, 'history': {}}, {'general': {'age': 20, 'identifier': 100}, 'history': {}}, {'general': {'age': 30, 'identifier': 101}, 'history': {}}, {'general': {'age': 30, 'identifier': 100}, 'history': {}}]}

Adding whitespace, you get

{
    'people': [
        {
            'general': {
                'age': 20, 
                'identifier': 100
            }, 
            'history': {}
        }, 
        {
            'general': {
                'age': 20, 
                'identifier': 100
            }, 
            'history': {}
        }, 
        {
            'general': {
                'age': 30, 
                'identifier': 101
            }, 
            'history': {}
        }, 
        {
            'general': {
                'age': 30, 
                'identifier': 100
            }, 
            'history': {}
        }
    ]
}

Of course, this approach only works if you know the structure of the object. If the person sending you this data is allowed to change the structure whenever they want, then your code is very likely to break right away.

I think what you're hoping for is some kind of "smart parser" that can interpret an object with arbitrary structure and understand the conceptual meaning of each component. As far as I know, no such library exists, because it would need human-level intelligence (or better) to make effective guesses for anything other than simple cases.

...That said, you might be able to handle some level of changing structure as long as you can make certain guarantees about the data. Let's say that there will always be a "people" key that you want to append to, and there will always be an "id" key that you want to rename. If these facts stay constant, then you can walk through the dict and find the objects you need no matter where they are.

import copy

def find_key_item_pairs(obj, criteria):
    if isinstance(obj, dict):
        for key_and_value in obj.items():
            if criteria(key_and_value):
                yield key_and_value
            else:
                value = key_and_value[1]
                yield from find_key_item_pairs(value, criteria)
    elif isinstance(obj, list):
        for item in obj:
            yield from find_key_item_pairs(item, criteria)

d = {
  "people": [
    {
      "general": {
        "id": 100,
        "age": 20
      },
      "history": {
      }
    },
    {
      "general": {
        "id": 101,
        "age": 30
      },
      "history": {
      }
    },
    {
      "general": {
        "id": 100,
        "age": 30
      },
      "history": {
      }
    }
  ]
}

#dynamically locate all people lists
for _, people_list in find_key_item_pairs(d, lambda kv: kv[0] == "people" and isinstance(kv[1], list)):
    #duplicate the first entry and insert
    people_list.insert(0, copy.deepcopy(people_list[0]))

#dynamically locate all dicts containing "id"
for _, person in find_key_item_pairs(d, lambda kv: isinstance(kv[1], dict) and "id" in kv[1]):
    #swap out "id" for "identifier"
    person["identifier"] = person["id"]
    del person["id"]

print(d)

Let's read in the structure:

dictPeople = {
  "people": [
    {
      "general": {
        "id": 100,
        "age": 20
      },
      "history": {
      }
    },
    {
      "general": {
        "id": 101,
        "age": 30
      },
      "history": {
      }
    },
    {
      "general": {
        "id": 100,
        "age": 30
      },
      "history": {
      }
    }
  ]
}

It's a Python dictionary.

1.To add a new object under people, we use standard Python's dictionary functionality. No JSON querying is needed:

dictAdd = {
      "general": {
        "identifier": 100,
        "age": 30
      },
      "history": {
      }
    }
dictPeople['people'].append(dictAdd)

2.To update the field name which is in an unknown place in the structure, we can first use JSON querying (JSONPath) to localize it, using one of the libraries e.g. jsonpath-ng. After we get the smallest structure (a dictionary) with that field, we can update it using standard dictionary functionality of updating the key. After that, we apply the updated dictionary back to the JSON structure. The code is as follows:

import json 
from jsonpath_ng import jsonpath
from jsonpath_ng.ext import parse

# localize the fields to update based on JSONPath query
jsExp = parse("$..people[?general.id].general")
match = jsExp.find(dictPeople)
for obj in match:
    print(obj.value)
    # get the smallest structure with the field
    dictGeneral = obj.value # the structure is a dictionary
    # update the field
    dictGeneral['identifier'] = dictGeneral['id']
    del dictGeneral['id']
    # update JSON structure
    jsExp.update(dictPeople,dictGeneral)

print(dictPeople)

I had the same problem.

I think you may find glom useful and more pythonic, besides you can still use jmespath to complement different uses.