Finding missing value in JSON using python

Question

I am facing this problem, I want to separate the dataset that has completed and not complete. So, I want to put flag like 'complete' in the JSON. Example as in output.

This is the data that i have

data=[{'id': 'abc001',
        'demo':{'gender':'1',
               'job':'6',
               'area':'3',
               'study':'3'},
         'ex_data':{'fam':'small',
                    'scholar':'2'}},
       {'id': 'abc002',
        'demo':{'gender':'1',
               'edu':'6',
               'qual':'3',
               'living':'3'},
        'ex_data':{'fam':'',
                   'scholar':''}},
       {'id': 'abc003',
        'demo':{'gender':'1',
               'edu':'6',
               'area':'3',
               'sal':'3'}
        'ex_data':{'fam':'big',
                   'scholar':NaN}}]

Output

How can I put the flag and also detect NaN and NULL in JSON?

Output=[{'id': 'abc001',
        'completed':'yes',
        'demo':{'gender':'1',
               'job':'6',
               'area':'3',
               'study':'3'},
         'ex_data':{'fam':'small',
                    'scholar':'2'}},
       {'id': 'abc002',
        'completed':'no',
        'demo':{'gender':'1',
               'edu':'6',
               'qual':'3',
               'living':'3'},
        'ex_data':{'fam':'',
                   'scholar':''}},
       {'id': 'abc003',
        'completed':'no',
        'demo':{'gender':'1',
               'edu':'6',
               'area':'3',
               'sal':'3'}
        'ex_data':{'fam':'big',
                   'scholar':NaN}}]

Use if loop to check if key having value as null , you need to iterate on all items. — Vikas Sharma
– Vikas Sharma, Commented Jan 8, 2020 at 8:57
To deal with observing NaN you can convert the json object to a normal python data type(dictionary or list) and handle all the computations that way. With that you should be able to break it down as low as it needs to get to obtain just about anything inside — maestro.inc
– maestro.inc, Commented Jan 8, 2020 at 8:58
The NaN value this creates a problem, if you could make it 'NaN' wherever it appears I can solve your question. Waiting for your reply @farahamiramh — Sido4odus
– Sido4odus, Commented Jan 10, 2020 at 21:33

Arkenys · Accepted Answer · 2020-01-08 09:44:11Z

1

Something like this should work for you:

data = [
    {
        'id': 'abc001',
        'demo': {
            'gender': '1',
            'job': '6',
            'area': '3',
            'study': '3'},
        'ex_data': {'fam': 'small',
                    'scholar': '2'}
    },
    {
        'id': 'abc002',
        'demo': {
            'gender': '1',
            'edu': '6',
            'qual': '3',
            'living': '3'},
        'ex_data': {'fam': '',
                    'scholar': ''}},
    {
        'id': 'abc003',
        'demo': {
            'gender': '1',
            'edu': '6',
            'area': '3',
            'sal': '3'},
        'ex_data': {'fam': 'big',
                    'scholar': None}
    }
]


def browse_dict(dico):
    empty_values = 0
    for key in dico:
        if dico[key] is None or dico[key] == "":
            empty_values += 1

        if isinstance(dico[key], dict):
            for k in dico[key]:
                if dico[key][k] is None or dico[key][k] == "":
                    empty_values += 1

    if empty_values == 0:
        dico["completed"] = "yes"
    else:
        dico["completed"] = "no"


for d in data:
    browse_dict(d)
    print(d)

Output :

{'id': 'abc001', 'demo': {'gender': '1', 'job': '6', 'area': '3', 'study': '3'}, 'ex_data': {'fam': 'small', 'scholar': '2'}, 'completed': 'yes'}
{'id': 'abc002', 'demo': {'gender': '1', 'edu': '6', 'qual': '3', 'living': '3'}, 'ex_data': {'fam': '', 'scholar': ''}, 'completed': 'no'}
{'id': 'abc003', 'demo': {'gender': '1', 'edu': '6', 'area': '3', 'sal': '3'}, 'ex_data': {'fam': 'big', 'scholar': None}, 'completed': 'no'}

Note that I changed NaN to None, because here you are most likely showing a python dictionary, not a JSON file since you are using data =

In a dictionary, the NaN value would be changed for None. If you have to convert your JSON to a dictionary, refer to the JSON module documentation.

Also please check your dictionary syntax. You missed several commas to separate data.

answered Jan 8, 2020 at 9:44

Arkenys

3531 silver badge12 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

FrahChan04 Over a year ago

Basically, my data are bigger than this. And all of it showing NaN. How to replace all NaN data to None?

Arkenys Over a year ago

Your data is in a Json file, so when you import it into Python using the json module (check the doc in my above post), it will automatically replace NaN to None. You can't have a dict containing NaN value as NaN is not known in Python.

Jaldhi Mehta · Accepted Answer · 2020-01-08 11:02:49Z

You should try

The Input is

data = [{'demo': {'gender': '1', 'job': '6', 'study': '3', 'area': '3'}, 'id': 'abc001', 'ex_data': {'scholar': '2', 'fam': 'small'}}, {'demo': {'living': '3', 'gender': '1', 'qual': '3', 'edu': '6'}, 'id': 'abc002', 'ex_data': {'scholar': '', 'fam': ''}}, {'demo': {'gender': '1', 'area': '3', 'sal': '3', 'edu': '6'}, 'id': 'abc003', 'ex_data': {'scholar': None, 'fam': 'big'}}]

Also, Nan will not work in Python. So, instead of Nan we have used None.

for item in data:
   item["completed"] = 'yes'
   for key in item.keys():
      if isinstance(item[key],dict):
          for inner_key in item[key].keys():
              if (not item[key][inner_key]):
                  item["completed"] = "no"
                  break
      else:
          if (not item[key]):
              item["completed"] = "no"
              break

The Output will be

 data = [{'demo': {'gender': '1', 'job': '6', 'study': '3', 'area': '3'}, 'completed': 'yes', 'id': 'abc001', 'ex_data': {'scholar': '2', 'fam': 'small'}}, {'demo': {'living': '3', 'edu': '6', 'qual': '3', 'gender': '1'}, 'completed': 'no', 'id': 'abc002', 'ex_data': {'scholar': '', 'fam': ''}}, {'demo': {'edu': '6', 'gender': '1', 'sal': '3', 'area': '3'}, 'completed': 'no', 'id': 'abc003', 'ex_data': {'scholar': None, 'fam': 'big'}}]

Collectives™ on Stack Overflow

Finding missing value in JSON using python

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related