converting JSON object ot array of JSON objects on basis of key - Code in Python

Question

I have an array of JSON object. The sample array is as follows:

[
  {
    "evt.category": "file",
    "evt.cpu": 0,
    "evt.num": 10078507,
    "evt.res": "SUCCESS",
    "evt.time": 1532841047277584400,
    "evt.type": "read",
    "fd.filename": "libnss_files.so.2",
    "fd.name": "/lib/x86_64-linux-gnu/libnss_files.so.2",
    "fd.num": 13,
    "fd.type": "file",
    "fd.uid": "1996913",
    "proc.loginshellid": 19968,
    "proc.name": "last",
    "proc.pid": 19969,
    "thread.ismain": true,
    "thread.tid": 19969
  },
  {
    "evt.buffer": "1000",
    "evt.category": "file",
    "evt.cpu": 0,
    "evt.num": 10078564,
    "evt.res": "SUCCESS",
    "evt.time": 1532841047277731300,
    "evt.type": "read",
    "fd.filename": "loginuid",
    "fd.name": "/proc/16009/loginuid",
    "fd.num": 13,
    "fd.type": "file",
    "fd.uid": "1996913",
    "proc.loginshellid": 19968,
    "proc.name": "last",
    "proc.pid": 19969,
    "thread.ismain": true,
    "thread.tid": 19969
  },
  {
    "evt.buffer": "",
    "evt.category": "file",
    "evt.cpu": 0,
    "evt.num": 10078566,
    "evt.res": "SUCCESS",
    "evt.time": 1532841047277733400,
    "evt.type": "read",
    "fd.filename": "loginuid",
    "fd.name": "/proc/16009/loginuid",
    "fd.num": 13,
    "fd.type": "file",
    "fd.uid": "1996913",
    "proc.loginshellid": 19968,
    "proc.name": "last",
    "proc.pid": 19969,
    "thread.ismain": true,
    "thread.tid": 19969
  }
]

I want to re-structure this array such that each object is converted into another array and each array should contain JSON objects on the basis of these keys like a JSON object of evt , proc thread etc.

I tried some online websites to do that but none of it work.

Please Help.

EDIT: My desired output is as follows:

[
  {
    "evt": {
      "category": "file",
      "cpu": 0,
      "num": 10078507,
      "res": "SUCCESS",
      "time": 1532841047277584400,
      "type": "read"
    },
    "fd": {
      "filename": "libnss_files.so.2",
      "name": "/lib/x86_64-linux-gnu/libnss_files.so.2",
      "num": 13,
      "type": "file",
      "uid": "1996913"
    },
    "proc": {
      "loginshellid": 19968,
      "name": "last",
      "pid": 19969
    },
    "thread": {
      "ismain": true,
      "tid": 19969
    }
  },
  {
    "evt": {
      "buffer": "1000",
    "category": "file",
    "cpu": 0,
    "num": 10078564,
    "res": "SUCCESS",
    "time": 1532841047277731300,
    "type": "read"
    },
    "fd": {
    "filename": "loginuid",
    "name": "/proc/16009/loginuid",
    "num": 13,
    "type": "file",
    "uid": "1996913"
    },
    "proc": {
      "loginshellid": 19968,
    "name": "last",
    "pid": 19969
    },
    "thread" : {
    "ismain": true,
    "tid": 19969
    }
  }
]

@DavidWinder I have added it in the question now. Hope now my question is more clear. Thanks. — cstayyab
– cstayyab, Commented Jul 29, 2018 at 6:25
@cstayyab, which programming language would you like to do this conversion? — Bráulio Figueiredo
– Bráulio Figueiredo, Commented Jul 29, 2018 at 6:27
@BráulioFigueiredo , I didn't mention specific programming language because in my view there may be a generic way to do it ( e.g. loops). But to be specific, it's better to be in python 3 — cstayyab
– cstayyab, Commented Jul 29, 2018 at 6:44
@cstayyab Unfortunatly I don't know python, but I solve your probem in Javascript. Check my answer if it helps — Bráulio Figueiredo
– Bráulio Figueiredo, Commented Jul 29, 2018 at 7:09

NobbyNobbs · Accepted Answer · 2018-07-29 08:14:31Z

2

import json

list_of_objects = json.loads(json_string)

new_list = []
for complex_object in list_of_objects:
    new_object = {}
    for composite_key, value in complex_object.items():
        key, subkey = composite_key.split(".")
        if key not in new_object:
            new_object[key] = {}
        new_object[key][subkey] = value
    new_list.append(new_object)

json_string = json.dumps(new_list)

edited Jul 29, 2018 at 8:14

answered Jul 29, 2018 at 8:01

NobbyNobbs

1,4441 gold badge12 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Bráulio Figueiredo · Accepted Answer · 2018-07-29 07:24:22Z

var obj = [{
        "evt.category": "file",
        "evt.cpu": 0,
        "evt.num": 10078507,
        "evt.res": "SUCCESS",
        "evt.time": 1532841047277584400,
        "evt.type": "read",
        "fd.filename": "libnss_files.so.2",
        "fd.name": "/lib/x86_64-linux-gnu/libnss_files.so.2",
        "fd.num": 13,
        "fd.type": "file",
        "fd.uid": "1996913",
        "proc.loginshellid": 19968,
        "proc.name": "last",
        "proc.pid": 19969,
        "thread.ismain": true,
        "thread.tid": 19969
    },
    {
        "evt.buffer": "1000",
        "evt.category": "file",
        "evt.cpu": 0,
        "evt.num": 10078564,
        "evt.res": "SUCCESS",
        "evt.time": 1532841047277731300,
        "evt.type": "read",
        "fd.filename": "loginuid",
        "fd.name": "/proc/16009/loginuid",
        "fd.num": 13,
        "fd.type": "file",
        "fd.uid": "1996913",
        "proc.loginshellid": 19968,
        "proc.name": "last",
        "proc.pid": 19969,
        "thread.ismain": true,
        "thread.tid": 19969
    },
    {
        "evt.buffer": "",
        "evt.category": "file",
        "evt.cpu": 0,
        "evt.num": 10078566,
        "evt.res": "SUCCESS",
        "evt.time": 1532841047277733400,
        "evt.type": "read",
        "fd.filename": "loginuid",
        "fd.name": "/proc/16009/loginuid",
        "fd.num": 13,
        "fd.type": "file",
        "fd.uid": "1996913",
        "proc.loginshellid": 19968,
        "proc.name": "last",
        "proc.pid": 19969,
        "thread.ismain": true,
        "thread.tid": 19969
    }
];

function convertObj(obj) {
    var resultArray = [];
    obj.forEach(item => {
        var resultObj = {};
        for (var property in item) {
            var array = property.split('.');
            var reference = resultObj;
            for (var i = 0; i < array.length-1; i++) {
                if (!reference[array[i]]){
                    reference[array[i]] = {};
                }
                reference = reference[array[i]];
            }
            reference[array[array.length-1]] = item[property];
        }
        resultArray.push(resultObj);
    });
    return resultArray;
}

console.log(convertObj(obj));

It is not giving required output. It's showing only first sub key value in each object.

peak · Accepted Answer · 2018-07-29 08:01:01Z

The task can be accomplished easily enough using jq, e.g. using the jq executable, the python jq module, or the python pyjq module.

Whichever mode you choose, a suitable filter would be:

def restructure:
  def splitup: index(".") as $ix | [ .[0:$ix], .[1+$ix:] ];
  to_entries
  | map( (.key|splitup) + [.value] )
  | reduce .[] as $x ({}; .[$x[0]][$x[1]] = $x[2]) ;

map(restructure)

Example (using the jq executable)

With the sample input, the invocation:

jq -f program.jq input.jq

yields:

[
  {
    "evt": {
      "category": "file",
      "cpu": 0,
      "num": 10078507,
      "res": "SUCCESS",
      "time": 1532841047277584400,
      "type": "read"
    },
    "fd": {
      "num": 13,
      "name": "/lib/x86_64-linux-gnu/libnss_files.so.2",
      "filename": "libnss_files.so.2",
      "type": "file",
      "uid": "1996913"
    },
    "proc": {
      "loginshellid": 19968,
      "name": "last",
      "pid": 19969
    },
    "thread": {
      "ismain": true,
      "tid": 19969
    }
  },
  {
    "evt": {
      "buffer": "1000",
      "category": "file",
      "cpu": 0,
      "num": 10078564,
      "res": "SUCCESS",
      "time": 1532841047277731300,
      "type": "read"
    },
    "fd": {
      "name": "/proc/16009/loginuid",
      "filename": "loginuid",
      "num": 13,
      "type": "file",
      "uid": "1996913"
    },
    "proc": {
      "loginshellid": 19968,
      "name": "last",
      "pid": 19969
    },
    "thread": {
      "ismain": true,
      "tid": 19969
    }
  },
  {
    "evt": {
      "buffer": "",
      "category": "file",
      "cpu": 0,
      "num": 10078566,
      "res": "SUCCESS",
      "time": 1532841047277733400,
      "type": "read"
    },
    "fd": {
      "name": "/proc/16009/loginuid",
      "filename": "loginuid",
      "num": 13,
      "type": "file",
      "uid": "1996913"
    },
    "proc": {
      "loginshellid": 19968,
      "name": "last",
      "pid": 19969
    },
    "thread": {
      "ismain": true,
      "tid": 19969
    }
  }
]

Robustification

The above assumes all the key names have at least one ".". If that cannot be assumed, then the following variant of restructure could be used:

def restructure:
  def splitup: index(".") as $ix
    | if $ix then [ .[0:$ix], .[1+$ix:] ]
      else [ ., null] 
      end;
  to_entries
  | map( (.key|splitup) + [.value] )
  | reduce .[] as $x ({};
      if $x[1] == null then .[$x[0]] = $x[2]
      else .[$x[0]][$x[1]] = $x[2]
      end ) ;

cstayyab · Accepted Answer · 2018-07-29 09:41:11Z

I don't know about python but this is how you can do it with JS

const arr = [
  {
    "evt.category": "file",
    "evt.cpu": 0,
    "evt.num": 10078507,
    "evt.res": "SUCCESS",
    "evt.time": 1532841047277584400,
    "evt.type": "read",
    "fd.filename": "libnss_files.so.2",
    "fd.name": "/lib/x86_64-linux-gnu/libnss_files.so.2",
    "fd.num": 13,
    "fd.type": "file",
    "fd.uid": "1996913",
    "proc.loginshellid": 19968,
    "proc.name": "last",
    "proc.pid": 19969,
    "thread.ismain": true,
    "thread.tid": 19969
  },
  {
    "evt.buffer": "1000",
    "evt.category": "file",
    "evt.cpu": 0,
    "evt.num": 10078564,
    "evt.res": "SUCCESS",
    "evt.time": 1532841047277731300,
    "evt.type": "read",
    "fd.filename": "loginuid",
    "fd.name": "/proc/16009/loginuid",
    "fd.num": 13,
    "fd.type": "file",
    "fd.uid": "1996913",
    "proc.loginshellid": 19968,
    "proc.name": "last",
    "proc.pid": 19969,
    "thread.ismain": true,
    "thread.tid": 19969
  },
  {
    "evt.buffer": "",
    "evt.category": "file",
    "evt.cpu": 0,
    "evt.num": 10078566,
    "evt.res": "SUCCESS",
    "evt.time": 1532841047277733400,
    "evt.type": "read",
    "fd.filename": "loginuid",
    "fd.name": "/proc/16009/loginuid",
    "fd.num": 13,
    "fd.type": "file",
    "fd.uid": "1996913",
    "proc.loginshellid": 19968,
    "proc.name": "last",
    "proc.pid": 19969,
    "thread.ismain": true,
    "thread.tid": 19969
  }
]


const newArr = arr.map(obj =>{
  
    const newObj = {};

    for(let key in obj){
      const [mainKey,subKey] =  key.split('.')
      if(newObj[mainKey] === undefined){
        newObj[mainKey] = {}
      } 
       newObj[mainKey][subKey] = obj[key];
    }
    
    return newObj
    
    
})

console.log(newArr)

I like the way you did the thing but I am passing an output of program (Sysdig) to my program and sadly, sysdig cannot call a JS file. Besides I also have to do more things after conversion which I cannot do in JS.

Collectives™ on Stack Overflow

converting JSON object ot array of JSON objects on basis of key - Code in Python

4 Answers 4

Comments

2 Comments

Example (using the jq executable)

Robustification

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

2 Comments

Example (using the jq executable)

Robustification

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related