shell parsing json and loop output the combine variable

Question

Jusk like my previous thread, I know how to parse simple json with spaces.

Now I have another question is that if I have multiple module structures, their keys are the same, but the values are different, I want the output value to be a combination of the values in each module, but actually the value in the last module will overwrite the previous one.

My test sample JSON like:

{
  "WorkspaceName":"aaa bbb ccc ddd eee",
  "ReportFileName":"xxx yyy zzz",
  "StageName":"sit uat prod"
},
{
  "WorkspaceName":"1111 2222 3333 4444 5555",
  "ReportFileName":"6666 7777 8888",
  "StageName":"sit1 uat1 prod1"
}

And my tried shell script(s) mian.sh is as follows:

InitialFile=$WORKSPACE/deployment/configuration/Initial.json
eval $(sed -n -e 's/^.*"\(.*\)":\(".*"\).*$/\1=\2/p' $InitialFile)

ConfigFile="$WorkspaceName"_"$ReportFileName"

echo The Config File is_$ConfigFile

The result is always The Config File is_1111 2222 3333 4444 5555_6666 7777 8888, I want get both values: aaa bbb ccc ddd eee_xxx yyy zzz and 1111 2222 3333 4444 5555_6666 7777 8888.

How do I achieve this?

A little background to understand why I'm doing this and some of my limitations：

I am executing my pipeline on jenkins and it will execute my mian.sh. So the entry is mian.sh. In addition, the jenkins server is maintained by a separate team, and we cannot directly access the server, so we cannot run shell code directly on the server.

Another, I need to combine variables in order to use this variable to match the name of the corresponding configuration file. Different results need to match different files for subsequent testing.

Experts always advice to use tools like jq which understand json file well. Can you use jq here? Once you confirm you may get answer as per that, cheers. — RavinderSingh13
– RavinderSingh13, Commented Oct 20, 2022 at 6:47
@RavinderSingh13, Thanks for your comment, I could not allowed to use jq, because the security level of our server is relatively high. We cannot install jq. It cannot be installed until the end. If the installation is unavoidable, and the installation will to be a very long process of security approval and verification — Joy
– Joy, Commented Oct 20, 2022 at 7:02
Ok sure, thanks for confirming, have added an answer based on that, cheers. — RavinderSingh13
– RavinderSingh13, Commented Oct 20, 2022 at 7:03
IMHO the usage of eval will rather increase the security risk. — tshiono
– tshiono, Commented Oct 20, 2022 at 7:11
There are other tools that are commonly available on servers and also understand JSON, e.g. Python. This is going to be extremely flaky without a proper JSON parser. — shadowtalker
– shadowtalker, Commented Oct 20, 2022 at 10:05

RavinderSingh13 · Accepted Answer · 2022-10-20 10:03:22Z

5

Important points for this answer:

Since OP can't install and use jq so going with an awk approach here.
I have provided 3 solutions here, 1st: is GNU awk approach and 2nd is NON-GNU awk approach and 3rd one is running NON-GNU awk code from a shell script.
1st 2 codes are stand along awk codes to run on terminal OR in an awk script
Then as per OP's request since their code is running in Jenkins I have posted a shell script which accepts an argument which is a Input_file name to be passed to it.
To save output into a shell variable could be done in 3rd code of this answer by changing 1st line to StageName=$(awk -v RS= ' and changing last line of 3rd code to ' "$1").

1st solution: With your shown samples please try following GNU awk code. Using match function of GNU awk where I am using regex [[:space:]]+"WorkspaceName":"([^"]*)",\n[[:space:]]+"ReportFileName":"([^"]*) to get the required values and creating 2 capturing groups out of it which further stores values into an array named arr to be get values later on as pre reuiqrement.

awk -v RS= '
{
  while(match($0,/[[:space:]]+"WorkspaceName":"([^"]*)",\n[[:space:]]+"ReportFileName":"([^"]*)",/,arr)){
    print arr[1]"_"arr[2]
    $0=substr($0,RSTART+RLENGTH)
  }
}
'  Input_file

2nd solution: With your shown samples please try following code, should work in any POSIX awk. This solution also uses match function but it doesn't create array and doesn't have any capturing groups in it, since capturing group capability is part of GNU awk. So using split function here to split the matched values and get only required part out of it.

awk -v RS= '
{
  while(match($0,/[[:space:]]+"WorkspaceName":"[^"]*",\n[[:space:]]+"ReportFileName":"[^"]*",/)){
    val=substr($0,RSTART,RLENGTH)
    split(val,arr,"\"WorkspaceName\":\"|\"ReportFileName\":\"|,\n")
    sub(/"$/,"",arr[2])
    sub(/",$/,"",arr[4])
    print arr[2]"_"arr[4]
    $0=substr($0,RSTART+RLENGTH)
  }
}
'  Input_file

To run code from shell script try like:

#!/bin/bash
awk -v RS= '
{
  while(match($0,/[[:space:]]+"WorkspaceName":"[^"]*",\n[[:space:]]+"ReportFileName":"[^"]*",/)){
    val=substr($0,RSTART,RLENGTH)
    split(val,arr,"\"WorkspaceName\":\"|\"ReportFileName\":\"|,\n")
    sub(/"$/,"",arr[2])
    sub(/",$/,"",arr[4])
    print arr[2]"_"arr[4]
    $0=substr($0,RSTART+RLENGTH)
  }
}
'  "$1"

edited Oct 20, 2022 at 10:03

answered Oct 20, 2022 at 7:03

RavinderSingh13

135k14 gold badges61 silver badges100 bronze badges

Sign up to request clarification or add additional context in comments.

27 Comments

RavinderSingh13 Over a year ago

@Joy, If you are not having access to install GNU awk then you could request your root admins(yes we need root access OR sudo access given by root admins to install something) who have access(as per your company's process) and if it will take time then try my 2nd solution which doesn't need GNU awk and let me know how it goes, cheers.

RavinderSingh13 Over a year ago

@Joy, If you need to concatenate all values and save it into variable we can easily do it, once you confirm about running command on Input_file I can take care of it then.

RavinderSingh13 Over a year ago

@Joy, I have now updated my answer check section To run code from shell script try like: and save whole content into your script. This script assumes that you are passing file name as parameter to it, because that's how your current setup running I believe so, in my code "$1" stand for 1st argument value only.

Joy Over a year ago

Thanks a lot. I think we are close to the answer. As I test, I pass "$InitialFile" instead of 1, And the result output two value: aaa bbb ccc ddd eee xxx yyy zzz and 1111 2222 3333 4444 5555 6666 7777 8888. A little help is that 1. WorkspaceName and ReportFileName are joined together by underscore. 2. How do I assign a value to a variable so that I can use it to loop through it. 3. How to get StageName?

RavinderSingh13 Over a year ago

@Joy, its giving you 2 values as of now correct? If yes then please explain what's the problem by which you are not able to retrieve both?

|

Collectives™ on Stack Overflow

shell parsing json and loop output the combine variable

1 Answer 1

27 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

27 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related