Use Python to iterate through nested JSON data

Question

TASK: I am using a API call to get JSON data from our TeamCity CI tool. We need to identify all those builds which are using old version of msbuild. We can identify from this API call data { "name": "msbuild_version", "value": "15.0" } At the moment i am saving the entire API call data to a file; however i will later integrate the API call to the same script. Now to the question at hand; How can i filter this above property i.e. msbuild_version, to say msbuild_version < 15.0 (i.e. all msbuild less than version 15.0) and display the corresponding 'id' and 'projectName' under 'buildType'; e.g.

"id": "AIntegration_BTool_BToolBuilds_DraftBuild",
"projectName": "A Integration / B Tool / VAR Builds",

here is a part of the JSON data file:-

{
    "project": [{
        "id": "_Root",
        "buildTypes": {
            "buildType": []
        }
    }, {
        "id": "AI_BTool_BToolBuilds",
        "buildTypes": {
            "buildType": [{
                "id": "AI_BTool_BToolBuilds_DraftBuild",
                "projectName": "A I / B Tool / VAR Builds",
                "steps": {
                    "step": [ {
                        "id": "RUNNER_213",
                        "name": "Build",
                        "type": "MSBuild",
                        "properties": {
                            "property": [ {
                                "name": "msbuild_version",
                                "value": "16.0"
                            }, {
                                "name": "run-platform",
                                "value": "x64"
                            }, {
                                "name": "targets",
                                "value": "Build"
                            }, {
                                "name": "teamcity.step.mode",
                                "value": "default"
                            }, {
                                "name": "toolsVersion",
                                "value": "15.0"
                            }]
                        }
                    }, {
                        "id": "RUNNER_228",
                        "name": "temp",
                        "type": "VS.Solution",
                        "properties": {
                            "property": [{
                                "name": "build-file-path",
                                "value": "x"
                            }, {
                                "name": "msbuild_version",
                                "value": "16.0"
                            }, {
                                "name": "vs.version",
                                "value": "vs2019"
                            }]
                        }
                    }]
                }
            }, {
                "id": "AI_BTool_BToolBuilds_ContinuousBuildWithNexusI",
                "projectName": "A I / B Tool / VAR Builds",
                "steps": {
                    "step": [ {
                        "id": "RUNNER_22791",
                        "name": "Build",
                        "type": "MSBuild",
                        "properties": {
                            "property": [{
                                "name": "msbuild_version",
                                "value": "16.0"
                            }, {
                                "name": "run-platform",
                                "value": "x86"
                            }, {
                                "name": "teamcity.step.mode",
                                "value": "default"
                            }, {
                                "name": "toolsVersion",
                                "value": "15.0"
                            }]
                        }
                    }]
                }
            }]
        }
    }, {
        "id": "AI_BTool_BToolBuilds_VARApiBuilds",
        "buildTypes": {
            "buildType": [{
                "id": "AI_BTool_BToolBuilds_CiVARNewSolutionContinuousBuild",
                "projectName": "A I / B Tool / VAR Builds / VAR API builds",
                "steps": {
                    "step": [ {
                        "id": "RUNNER_22791",
                        "name": "Build",
                        "type": "MSBuild",
                        "properties": {
                            "property": [{
                                "name": "msbuilds_version",
                                "value": "15.0"
                            }, {
                                "name": "toolsVersion",
                                "value": "15.0"
                            }]
                        }
                    }]
                }
            }, {
                "id": "AI_BTool_BToolBuilds_VARApiBuilds_CiVARIngestionWindowsServiceNonReleaseBranchBuild",
                "projectName": "A I / B Tool / VAR Builds / VAR API builds",
                "steps": {
                    "step": [{
                        "id": "RUNNER_22790",
                        "name": "Nuget Installer",
                        "type": "jb.nuget.installer",
                        "properties": {
                            "property": [{
                                "name": "nuget.path",
                                "value": "%teamcity.tool.NuGet.CommandLine.4.9.2%"
                            }, {
                                "name": "msbuilds_version",
                                "value": "16.0"
                            }, {
                                "name": "nuget.use.restore",
                                "value": "restore"
                            }, {
                                "name": "sln.path",
                                "value": "VAR.sln"
                            }, {
                                "name": "teamcity.step.mode",
                                "value": "default"
                            }]
                        }
                    }]
                }
            }]
        }
    }]
}

My Solution till now And my code snippet till now

import json

with open('UnArchivedBuilds.txt') as api_call:
  read_content = json.load(api_call)

#for project in read_content['project']:
#   print (project.get('buildTypes'))


for project in read_content['project']:
   # print (project['id'])
   print (project['buildTypes']['buildType'])

I am not able to decide on the hierarchy of the JSON to print the relevant data (i.e id and projectName) wherever msbuild_version is less than 15.0

There are several starting points to approach this type of problem. First being recursion: You can go down the "JSON tree" via a recursive function and extract the data that way (which is handy when data structure varies and results in varying levels). Second approach would be schema validation: Assuming you know all possible schema / data tree variations you could define a schema for each of it and call a data handling function depending on the validation/detection result. Cerberus is a quite versatile schema validator, but has a bit of a learning curve and might rather be heavy-lifting. — albert
– albert, Commented Mar 16, 2020 at 10:32
@albert, thanks. If you note the snippet i have got above, i am trying to get the data iterating through the JSON tree. This is where i am getting issue. Is there a way to iterate through the objects? or do you have any tip to make this any easier — maverick
– maverick, Commented Mar 16, 2020 at 12:00

albert · Accepted Answer · 2020-03-17 19:35:00Z

2

I had a look at your JSON data which was broken. In order to work with the snippet you provided I fixed the malformed data and removed unneeded parts to decrease clutter:

{
    "project": [{
        "buildTypes": {
            "buildType": [{
                "id": "AIntegration_BTool_BToolBuilds_DraftBuild",
                "projectName": "A Integration / B Tool / VAR Builds"
            },
            {
                "id": "AIntegration_BTool_BToolBuilds_ContinuousBuildIntegration",
                "projectName": "A Integration / B Tool / VAR Builds"
            }]
        }
    }]
}

As per my comment above I suggested recursion or using a schema validator to boil the JSON data down. However, as a quick-and-dirty approach and due to the weird structure of your dataset, I decided to do a multi-iteration over some parts of your data. Iterating over the same data is quite ugly and considered to be bad practice in most cases.

Assuming the data is stored in a file called input.json, the following snippet should give you the desired output:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import json

with open('input.json') as f:
    data = json.load(f)

projects = (element for element in data.get('project'))
build_types = (element.get('buildTypes') for element in projects)
build_types = (element.get('buildType') for element in build_types)

for item in build_types:
    for element in item:
        identifier = element.get('id')
        project_name = element.get('projectName')
        print('{} --> {}'.format(identifier, project_name))

Printing:

AIntegration_BTool_BToolBuilds_DraftBuild --> A Integration / B Tool / VAR Builds
AIntegration_BTool_BToolBuilds_ContinuousBuildIntegration --> A Integration / B Tool / VAR Builds

answered Mar 17, 2020 at 19:35

albert

8,69111 gold badges59 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

maverick Over a year ago

Thanks @albert. I agree, the JSON file had incorrect format and too cluttered. Please note now, i have updated the original question, with a valid JSON file. Your code works and prints the id and project name as desired. However my use case is only print those id/project name where the JSON tree has this structure /value ` steps> step> properties> property> { "name": "toolsVersion", "value": "15.0" }`

maverick Over a year ago

I tried extending your code as this;

....  print('{0} --> {1}'.format(identifier, project_name)) 	for item1 in item: 	steps = (element.get('steps') for element in build_type) 	print("element.get(steps) :  %s" % type(steps)) 	step = (element.get('step') for element in steps) 	print("element.get(step) :  %s" % type(step)) 	for item1 in step: 		for element1 in item1: 			step_id = element1.get('id') 			print type(step_id)

however it dosent work

maverick Over a year ago

I was able to iterate further using build_type_step = element.get('steps').get('step') and properties_property = (element.get('properties').get('property') for element in build_type_step); . And deriving the value from properties_property for msbuild version. Also when trying to get the specific step id, or name which is unicode value not able to get, build_type_step = element.get('steps').get('step').get('id') or build_type_step = element.get('steps').get('step').['id']. How can we get this specific value

Collectives™ on Stack Overflow

Use Python to iterate through nested JSON data

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related