parsing the json kind of file to find out particular values using unix shell script/Python

Question

I would like to extract few lines of given file using shell script. The problem is I have set of files which need to parsed and extract data. The below examples explains my scenario: I am looking to extract only "source" and "destination" values from the given files.


lists = [ 
{
name = "Account - UU ",
source = "1-account",
destination = "account-hhh",
other = "111111"
},
{
name = "Account - PP,
source = "2-account",
destination = "account-hhh12345",
other = "1212"
},
{
name = "Account - GG ",
source = "3-account",
destination = "account-gg567",
other = "44444"
},
{
name = "Account - QQ,
source = "4-account",
destination = "account-manager123456",
other = "23232323"
}
]

My expected output would be

source = "1-account" | destination = "account-hhh"
source = "2-account" | destination = "account-hhh12345"
source = "3-account" | destination = "account-gg567"
source = "4-account" | destination = "account-manager123456"

Please edit your question to show what you've tried. I'm pretty sure a quick "google" would show you how to do this. Do you have or can you install a json parser such as jq? — Ed Morton
– Ed Morton, Commented Jan 12, 2021 at 18:27

tshiono · Accepted Answer · 2021-01-15 02:06:16Z

1

If you can modify the list to a valid json format (as below) and python is your option, please try:

#!/usr/bin/python

lists = [
{
"name" : "Account - UU",
"source" : "1-account",
"destination" : "account-hhh",
"other" : 111111
},
{
"name" : "Account - PP",
"source" : "2-account",
"destination" : "account-hhh12345",
"other" : 1212
},
{
"name" : "Account - GG",
"source" : "3-account",
"destination" : "account-gg567",
"other" : 44444
},
{
"name" : "Account - QQ",
"source" : "4-account",
"destination" : "account-manager123456",
"other" : 23232323
}
]

for i in lists:
    print('source = "%s" | destination = "%s"' % (i["source"], i["destination"]))

Output:

source = "1-account" | destination = "account-hhh"
source = "2-account" | destination = "account-hhh12345"
source = "3-account" | destination = "account-gg567"
source = "4-account" | destination = "account-manager123456"

EDIT

Assuming you have a HOCON file file.conf which looks like:

lists = [
{
name = "Account - UU",
source = "1-account",
destination = "account-hhh",
other = 111111
},
{
name = "Account - PP",
source = "2-account",
destination = "account-hhh12345",
other = 1212
},
{
name = "Account - GG",
source = "3-account",
destination = "account-gg567",
other = 44444
},
{
name = "Account - QQ",
source = "4-account",
destination = "account-manager123456",
other = 23232323
}
]

Then try to execute the python script:

#!/usr/bin/python

from pyhocon import ConfigFactory

conf = ConfigFactory.parse_file('./file.conf')
for i in conf['lists']:
    print('source = "%s" | destination = "%s"' % (i['source'], i['destination']))

Output:

source = "1-account" | destination = "account-hhh"
source = "2-account" | destination = "account-hhh12345"
source = "3-account" | destination = "account-gg567"
source = "4-account" | destination = "account-manager123456"

EDIT2

If you want to parse multiple files in multiple directories recursively, save the following script as something like myscript.py:

#!/usr/bin/python

from pyhocon import ConfigFactory
import sys

conf = ConfigFactory.parse_file(sys.argv[1])
try:
    if type(conf['lists']) is list:
        for i in conf['lists']:
            print('source = "%s" | destination = "%s"' % (i['source'], i['destination']))
except Exception:
    pass

Then run:

find "$dir" -name "*.conf" -type f -exec python myscript.py {} \;

The command line above assumes the HOCON files have .conf extension and the variable $dir holds the top directory path of the conf files. Please modify them according to your environment.

EDIT3
The python script above may be altered by:

#!/usr/bin/python

from pyhocon import ConfigFactory
import sys

conf = ConfigFactory.parse_file(sys.argv[1])
if 'lists' in conf.keys():
    for i in conf['lists']:
        print('source = "%s" | destination = "%s"' % (i['source'], i['destination']))

which will be a bit smarter.

edited Jan 15, 2021 at 2:06

answered Jan 13, 2021 at 8:07

tshiono

22.3k2 gold badges18 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

16 Comments

BigD Over a year ago

Yea, I am looking for a solution if its available,e in python also.Basically am getting is it as conf file (Hocon). How do i call it as files rather than variables using python ? basically i am getting these kind of sample files in a directory and i need to parse these files and find source and destination

tshiono Over a year ago

Thank you for the feedback. I've updated my answer with an another version which reads the conf file as is, internally converts to json format, then outputs the same result. Would you please try it?

BigD Over a year ago

Thanks , as mentioned earlier my file is Hocon file. Looks like we need to use hocon parser - stackoverflow.com/questions/53188635/… Any idea how make it workable ?

tshiono Over a year ago

Thank you for the info. Would you provide some example files? Or is it still okay to use the provided one as an example?

BigD Over a year ago

Thanks, sample file is exactly same as given in the question

|

Collectives™ on Stack Overflow

parsing the json kind of file to find out particular values using unix shell script/Python

1 Answer 1

16 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

16 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related