How to handle missing JSON nested keys from an API response in python?

Question

Here is the JSON response I get from an API request:

{  
    "associates": [
        {
            "name":"DOE",
            "fname":"John",
            "direct_shares":50,
            "direct_shares_details":{
               "shares_PP":25,
               "shares_NP":25
            },
            "indirect_shares":50,
            "indirect_shares_details": {
                  "first_type": {
                      "shares_PP": 25,
                      "shares_NP": 0
                  },
                  "second_type": {
                      "shares_PP": 25,
                      "shares_NP": 0
                  }
            }
        } 
    ]
}

However, in some occasions, some values will be equal to None. In that case, I handle it in my function for all the values that I know will be integers. But it doesn't work in this scenario for the nested keys inside indirect_shares_details:

{  
    "associates": [
        {
            "name":"DOE",
            "fname":"John",
            "direct_shares":50,
            "direct_shares_details":{
               "shares_PP":25,
               "shares_NP":25
            },
            "indirect_shares":None,
            "indirect_shares_details": None
            }
        } 
    ]
}

So when I run my function to get the API values and put them in a custom dict, I get an error because the keys are simply inexistant in the response.

def get_shares_data(response):
    associate_from_api = []
        for i in response["associates"]:
            associate_data = {
            "PM_shares":  round(company["Shares"], 2),
            "full_name": i["name"] + " " + ["fname"]
            "details": {
                    "shares_in_PM":    i["direct_shares"],
                    "shares_PP_in_PM": i["direct_shares_details"]["shares_PP"],
                    "shares_NP_in_PM": i["direct_shares_details"]["shares_NP"],
                    "shares_directe":  i["indirect_shares"],
                    "shares_indir_PP_1": i["indirect_shares_details"]["first_type"]["shares_PP"],
                    "shares_indir_NP_1": i["indirect_shares_details"]["first_type"]["shares_NP"],
                    "shares_indir_PP_2": i["indirect_shares_details"]["second_type"]["shares_PP"],
                    "shares_indir_NP_2": i["indirect_shares_details"]["second_type"]["shares_NP"],
                }
            }
            for key,value in associate_data["details"].items():
                if value != None:
                    associate_data["details"][key] = value * associate_data["PM_shares"] / 100
                else:
                    associate_data["calculs"][key] = 0.0
            associate_from_api.append(associate_data)
    return associate_from_api

I've tried conditioning the access of the nested keys only if the parent key wasn't equal to None but I ended up declaring 3 different dictionaries inside if/else conditions and it turned into a mess, is there an efficient way to achieve this?

Victor Valente · Accepted Answer · 2022-08-12 15:37:07Z

1

You can try accessing the values using dict.get('key') instead of accessing them directly, as in dict['key'].

Using the first approach, you will get None instead of KeyError if the key is not there.

EDIT: tested using the dictionary from the question:

edited Aug 12, 2022 at 15:37

answered Aug 12, 2022 at 14:16

Victor Valente

8111 gold badge10 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user16779293 Over a year ago

I thought this worked but it does not work when I go into a missing nested key like this: "shares_indir_PP_1": i.get("indirect_shares_details", {}).get("first_type", {}).get("shares_PP") I get AttributeError: 'NoneType' object has no attribute 'get', is it normal or do I need to change my syntax? In the exemple where indirect_shares_details: None

Victor Valente Over a year ago

Hmmm that's strange. I tested using the dict from your question and it returns None. See changes in my answer. Maybe you're having some trouble managing all these nested structures?

user16779293 Over a year ago

Actually it works when I do it like you on a local dict but when I loop through the real API response that's when I get an error... Which is strange because it's the exact same syntax

OriginCoding · Accepted Answer · 2022-08-12 14:56:07Z

0

You can try pydantic

Install pydantic

pip install pydantic

# OR

conda install pydantic -c conda-forge

Define some models based on your response structure

from pydantic import BaseModel
from typing import List, Optional


# There are some common fields in your json response.
# So you can put them together.
class ShareDetail(BaseModel):
    shares_PP: int
    shares_NP: int


class IndirectSharesDetails(BaseModel):
    first_type: ShareDetail
    second_type: ShareDetail


class Associate(BaseModel):
    name: str
    fname: str
    direct_shares: int
    direct_shares_details: ShareDetail
    indirect_shares: int = 0  # Sets a default value for this field.
    indirect_shares_details: Optional[IndirectSharesDetails] = None


class ResponseModel(BaseModel):
    associates: List[Associate]

use ResponseModel.parse_xxx functions to parse response.

Here I use parse_file funtion, you can also use parse_json function

See: https://pydantic-docs.helpmanual.io/usage/models/#helper-functions

def main():
    res = ResponseModel.parse_file("./NullResponse.json",
                                   content_type="application/json")

    print(res.dict())


if __name__ == "__main__":
    main()

Then the response can be successfully parsed. And it automatically validates the input.

answered Aug 12, 2022 at 14:56

OriginCoding

231 silver badge4 bronze badges

1 Comment

OriginCoding Over a year ago

The "NullResponse.json" is the second response you post. But there is no field "indirect_shares" and "indirect_shares_details".

Collectives™ on Stack Overflow

How to handle missing JSON nested keys from an API response in python?

2 Answers 2

3 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related