3

I'm trying to dynamically parse & build-up a data structure of some incoming JSON files I'm to be supplied with (that'll be in non-standard structure) via Powershell to then process the data in those files & hand them over to the next step.

As part of that, I'm trying to build up the data structure of the JSON file into essentially a list of of data-paths for me to parse through & grab the data out of, so that I can cope with arrays, nested JSON objects and so on. So far so good.

Where I fall into some sort of Powershell peculiarity is in handling 2+ levels of depth via a variable. Let me give you a nice code-block to demonstrate the problem...

# Generate a Quick JSON file with different data types & levels
[object]$QuickJson = @'
{
    "Name" : "I am a JSON",
    "Version" : "1.2.3.4",
    "SomeBool" : true,
    "NULLValue" : null,
    "ArrayOfVersions" : [1.0,2.0,3.0],
    "MyInteger" : 69,
    "NestedJSON" : {
        "Version" : 5.0,
        "IsReady" : false
    },
    "DoubleNestedJSON" : {
        "FirstLevel" : 1,
        "DataValue" : "I am at first nested JSON level!",
        "Second_JSON_Level" : {
            "SecondLevel" : 2,
            "SecondDataValue" : "I am on the 2nd nested level"
        }
    }
}
'@

# Import our JSON file into Powershell
[object]$MyPSJson = ConvertFrom-Json -InputObject $QuickJson
# Two quick string variables to access our JSON data paths
[string]$ShortJsonPath = "Name"
[string]$NestedJsonPath = "NestedJson.Version"
# Long string to access a double-nested JSON object
[string]$LongNestedJsonPath = "DoubleNestedJSON.Second_JSON_Level.SecondDataValue"

# Both of these work fine
Write-Host ("JSON Name (Direct) ==> " + $MyPSJson.Name)
Write-Host ("JSON Name (via Variable) ==> " + $MyPSJson.$ShortJsonPath)

# The following way to access a single nested Json Path works fine
Write-Host ("Nested JSON Version (via direct path) ==> " + $MyPSJson.NestedJson.Version)
# And THIS returns an empty line / is where I fall afoul of something in Powershell
Write-Host ("Nested JSON Version (via variable) ==> " + $MyPSJson.$NestedJsonPath)

# Other things I tried -- all returning an empty line / failing in effect
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson.$NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $MyPSJson.$($NestedJsonPath))
Write-Host ("Alternate Nested JSON Version ==> " + $($MyPSJson).$($NestedJsonPath))

# Similarly, while THIS works...
$MyPSJson | select-object -Property NestedJSON
# This will fail / return me nothing
$MyPSJson | select-object -Property NestedJSON.Version

... in doing a bunch of research around this, I came across a suggestion to transform this into a Hashtable -- but that has the same problem, sadly. So with the above code-snippet, the following will transform the JSON object into a hashtable.

# Same problem with a hash-table if constructed from the JSON file...
[hashtable]$MyHash = @{}
# Populate $MyHash with the data from our quickie JSON file...
$QuickJson | get-member -MemberType NoteProperty | Where-Object{ -not [string]::IsNullOrEmpty($QuickJson."$($_.name)")} | ForEach-Object {$MyHash.add($_.name, $QuickJson."$($_.name)")}

# ... and even then -- $MyHash."$($NestedJsonPath)" -- fails, while a single level deep string works fine in the variable! :(

So it's pretty clear that I'm running into "something" of a Powershell internal logic problem, but I can't get Powershell to be overly helpful in WHY that is. Adding a '-debug' or similar in an attempt to increase verbosity hasn't helped shed light on this.

I suspect it's something akin to the items raised in this article here ( https://blogs.technet.microsoft.com/heyscriptingguy/2011/10/16/dealing-with-powershell-hash-table-quirks/ ) but just specific with variables.

I've not had any luck in finding anything obvious in the Powershell language specification (3.0 still being the latest from here as far as I can tell -- https://www.microsoft.com/en-usdownload/details.aspx?id=36389 ) either. It may be in there, I may just miss it.

Any advice in how to get Powershell to play nice with this would be greatly appreciated. I'm not sure how / why Powershell is fine with a simple string but seems to have issues with a 'something.somethingelse' type string here.

Thank you.

Further notes & addenda to the original:

It seems there are several issues to attack. One is "dealing with a single nested level". The "quick fix" for that seems to be using "Invoke-Expression" to resolve the statement, so for instance (IMPORTANT - take note of the back-tick with the first variable!):

iex "`$MyPSJson.$NestedJsonPath"

That use of Invoke-Expression also works with multi-nested situations:

iex "`$MyPSJson.$LongNestedJsonPath"

An alternative approach that was mentioned is the use of multiple select statements ... but I've not been able to get that to work with multi-nested objects (Powershell seems to not resolve those properly for some reason).

So for instance in this scenario:

($MyComp | select $_.DoubleNestedJSON | select FirstLevel)

Powershell returns

FirstLevel    
---------- 

... instead of the actual data value. So - for now, it seems that selects won't work with multi-level nested objects due to Powershell apparently not resolving them?

4
  • 1
    Invoke-Expression is the simplest (also slowest, I guess) solution: iex "`$MyPSJson.$NestedJsonPath" Commented Jan 26, 2017 at 14:50
  • I can live with slow - as long as I can get it to behave / work to begin with. Commented Jan 26, 2017 at 15:06
  • <Meh - can't edit my previous comment ... anyway - thanks most kindly for getting me un-stuck. That'll give me something to look into as well as to why 'iex' works. but the regular substitution does not! Big thanks!> Commented Jan 26, 2017 at 15:15
  • Still no closer to understanding why iex works, but dot-walking does not ... but just wanted to say (separately) a big thank you for getting me out of that bind. Your suggestion works with multiple levels of nesting perfectly. :) Commented Jan 26, 2017 at 16:33

5 Answers 5

6

When you write something like

$MyPSJson.Name

this will attempt to retrieve the member named Name from the object $MyPSJson. If there is no such member, you'll get $null.

Now, when you do that with variables for the member name:

$MyPSJson.$ShortJsonPath

this works pretty much identical in that the member with the name stored in $ShortJsonPath is looked up and its value retrieved. No surprises here.

When you try that with a member that doesn't exist on the object, such as

$MyPSJson.$NestedJsonPath
# equivalent to
# $MyPSJson.'NestedJSON.Version'

you'll get $null, as detailed before. The . operator will only ever access a member of the exact object that is the result of its left-hand-side expression. It will never go through a member hierarchy in the way you seem to expect it to do. To be frank, I'm not aware of a language that works that way.

The reason it works with Invoke-Expression is, that you effectively converting the $NestedJsonPath string into part of an expression resulting in:

$MyPSJson.NestedJSON.Version

which Invoke-Expression then evaluates.

You can, of course, define your own function that works that way (and I'd much prefer that instead of using Invoke-Expression, a cmdlet that should rarely, if ever, used (heck, it's eval for PowerShell – few languages with eval advocate its use)):

function Get-DeepProperty([object] $InputObject, [string] $Property) {
  $path = $Property -split '\.'
  $obj = $InputObject
  $path | %{ $obj = $obj.$_ }
  $obj
}

PS> Get-DeepProperty $MyPSJson NestedJson.Version
5,0

You could even make it a filter, so you can use it more naturally on the pipeline:

filter Get-DeepProperty([string] $Property) {
  $path = $Property -split '\.'
  $obj = $_
  $path | %{ $obj = $obj.$_ }
  $obj
}

PS> $MyPSJson | Get-DeepProperty nestedjson.version
5,0
Sign up to request clarification or add additional context in comments.

1 Comment

Ah - OK, that makes a lot of sense, put that way. Thanks a tonne for that (being mainly a script-guy rather than a programmer, the dot-notation didn't seem to be a problem for me). Using a filter is not something I've thought of either. Very promising & interesting advice here. I understand the hesitation around using Invoke-Expression - no quibble there. My first priority was in "getting the darn thing to work" to begin with. Am quite liking your variant approach with the filter. Also (again) big thanks for making sense of why this was a problem in the first place :).
2

Why this doesn't work

When you provide the properties that you'd like within a string, like this

[string]$NestedJsonPath = "NestedJson.Version"

Powershell looks for a property called NestedJSon.Version. It's not actually traversing the properties, but looking for a string literal which contains a period. In fact, if I add a property like that to your JSON like so.

[object]$QuickJson = @'
{
    "Name" : "I am a JSON",
    "Version" : "1.2.3.4",
    "SomeBool" : true,
    "NULLValue" : null,
    "ArrayOfVersions" : [1.0,2.0,3.0],
    "MyInteger" : 69,
    "NestedJSON.Version" : 69,
    "NestedJSON" : {
        "Version" : 5.0,
        "IsReady" : false
    }
}

I now get a value back, like so:

>$MyPSJson.$NestedJsonPath
69

The best way to get your values back is to use two separate variables, like this.

$NestedJson = "NestedJson"
$property   = "Version"

>$MyPSJson.$NestedJson.$property
5.0

Or, alternatively, you could use select statements, as seen in the original answer below.


$MyPSJson | select $_.NestedJSON | select Version
Version
-------
1.2.3.4

If you use multiple Select-Object statements, they'll discard the other properties and allow you to more easily drill down to the value you'd like.

6 Comments

Interesting ... it seems that Powershell is handling certain objects in a really ... "odd" way. The Invoke-Expression route works fine for a single level of nesting. I've played with a double nested JSON file and that is having issues with both - the "Invoke Expression" as well as the suggested multi-level select approach. I'll dig into it some more, you folks have given me a few ideas. I'll also edit the JSON snippet & problem description for more detail as it makes sense! Thanks for all the thoughts so far. :)
OK - so, it seems that Powershell is handling nested objects really strangely. While your suggestion seems to work with a single-level nesting, it seems to fall apart at double-nesting ... PS just returns the attribute, but not the data value. I've edited my question with the new information. Invoke-Expression does do the trick however, even through multiple levels of nesting (no idea why). It'd be nice if PS would give a bit more indication as to why, but oh well. So - for now, I'll put the "iex" route as an answer, as it works with multi-level nesting as well.
Note that your select pipeline element there cannot work as expected, unless you're within a script block that actually has $_ defined. Perhaps you meant select NestedJSON?
Good point Joey, in his example, we were within a big for each loop.
ah, I've updated the answer. The awesome Steven Murawski took a look and helped me to understand what's happening.
|
2

I followed Joey's filter example. However, I found it did not support accessing arrays. Sharing the code that I got to work for this. Hopefully it will help others as well. Awesome thread!

filter Get-DeepProperty([string] $Property) {
$path = $Property -split '\.'
$obj = $_
    foreach($node in $path){
        if($node -match '.*\[\d*\]'){
            $keyPieces = $node -split ('\[')
            $arrayKey = $keyPieces[0]
            $arrayIndex = $keyPieces[1] -replace ('\]','')
            $obj = $obj.$arrayKey[$arrayIndex]
        } else { 
            $obj = $obj.$node 
        }
    }
$obj
}

Example usage:

$path = "nested.nestedtwo.steps[2]"
$payload | Get-DeepProperty $path

Comments

1

I had the same problem, so I wrote a function that does the trick. It enables accessing any level of the json by variable path (string):

function getNestedJsonValue() {
    param(
        [Parameter(Mandatory = $true, ValueFromPipeline)] [PSCustomObject] $inputObj,
        [Parameter(Mandatory = $true)] [string] $valuePath
    )
    if (($valuePath -eq $null) -or ($valuePath.length -eq 0) -or ($inputObj -eq $null)) {
        return $inputObj
    }

    [System.Array] $nodes = "$valuePath" -split '\.'
    foreach ($node in $nodes) {
        if (($node -ne $null) -and ($node.length -gt 0) -and ($inputObj -ne $null)) {
            $inputObj = $inputObj.$node
        } else {
            return $inputObj
        }
    }
    return $inputObj
}
  • Usage: getNestedJsonValue -valuePath $nestedValuePath -inputObj $someJson
  • Pipe usage: $someJson | getNestedJsonValue -valuePath $nestedValuePath
  • An example nestedValuePath would be $nestedValuePath="some.nested.path"

2 Comments

Ooo - very nice. I shall give that a whirl (currently using python mainly to handle JSON-s). Thank you - appreciate the share :).
*made a minor change to fix behaviour of not returning the $null value of nested elements (but the parent's value instead).
0

Credit to wOxxOm for getting things on the right track.

Invoke-Expression does seem to work perfectly for this situation (if somewhat expensive, but that's fine in my personal example & situation), and it can cope with multiple levels of nesting.

So as examples for the above code snippet, the following will resolve just fine (Key point - pay attention to the initial back-tick. That caught me off guard):

Write-Host ("Single level JSON test ==> " + (iex "`$MyPSJson.$NestedJsonPath"))
Write-Host ("Double level JSON test ==> " + (iex "`$MyPSJson.$LongNestedJsonPath"))

That'll return our desired results:

Single level JSON test ==> 5.0
Double level JSON test ==> I am on the 2nd nested level

FoxDeploy's answer of using multi-level selects doesn't seem to work with 2+ levels of nesting, unfortunately for some bizarre reason.

Using:

($MyPSJson | select $_.DoubleNestedJSON | select FirstLevel)

We get the following back from Powershell:

FirstLevel    
---------- 

... it seems that Powershell doesn't resolve nested objects in its entirety? We get a similar results if we intentionally use something that doesn't exist:

($MyPSJson | select $_.DoubleNestedJSON | select Doesnotexist)

... also simply returns:

Doesnotexist  
------------ 

So - for now - it seems as if "Invoke-Expression" works most reliably (and most easily, as it's just a case of handing it a variable with the path'ed string).

I still can't explain the WHY of any of this so far (since I've used 'dotwalk'-ing with multiple variables through arrays quite happily), but at least there's a solution for now ... and that is Invoke-Expression !

The best (/least bad?) explanations for Invoke-Expression I've found so far are here (Microsoft's own description of the cmdlet doesn't really make a great job of hinting that it'd help in situations such as this):

2 Comments

Check out both my up to date and Joey's answers for an explanation of why.
Have done already. Awesome hat-tipping to both of you. Made it crystal clear where I fouled up & why. Also giving me a few options for how to attack this. I'll play around with it some more now, as I'm not certain how many levels of nested objects will be heading my way - so a function or filter method strikes me as the safer path forward. But yeah - very helpful on both counts. Thank you so much indeed for clearing up my "duh" for me :).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.