JQ create json array using bash

Question

I am currently trying to get information about my file hosting accounts. As I keep a lot of my backup media on different accounts. I am using megatools to query information about the account, which I then parse into an array. The array is then flattened using \n for raw input.

The script works wonderfully but its not creating valid json. I am not sure what I am missing to make it valid. Thanks for the help in advance.

script

function join_by { local IFS="$1"; shift; echo "$*"; }

while IFS='' read -r line || [[ -n "$line" ]]; do
    IFS=, read -ra array <<< "$line"

    nickname=${array[0]}
    user=${array[1]}
    pass=${array[2]}
    data=($(megadf --username=$user --password=$pass))
    data[${#data[@]}]+="$nickname"

    stats=$(join_by $'\n' ${data[@]})

    echo $stats | jq --slurp --raw-input 'split("\n")[:-1] | map([ split(" ")[] ]) | map({
    nick: .[6],
    total: .[1],
    used: .[3],
    free: .[5]
    })' >> /opt/stats/json/accounts.json

done < .accounts

json output

[
  {
    "nick": "alt",
    "total": "53687091200",
    "used": "7885201595",
    "free": "45801889605"
  }
]
[
  {
    "nick": "main",
    "total": "214748364800",
    "used": "87240914483",
    "free": "127507450317"
  }
]

What should be

[
  {
    "nick": "alt",
    "total": "53687091200",
    "used": "7885201595",
    "free": "45801889605"
  },
  {
    "nick": "main",
    "total": "214748364800",
    "used": "87240914483",
    "free": "127507450317"
  }
]

.accounts

nickname,user,pass

You shouldn't call jq inside the loop. Pipe the output of the loop to jq, so it can create a JSON array of everything. — Barmar
– Barmar, Commented Aug 30, 2018 at 0:33
data[${#data[@]}]+="$nickname" would be more clearly (and more correctly, should an array be sparse) written as data+=( "$nickname" ). — Charles Duffy
– Charles Duffy, Commented Aug 30, 2018 at 0:43

Ryahn · Accepted Answer · 2018-08-30 00:40:04Z

1

@barmar found the simplest solution that I overlooked....

Needed to pipe the loop into jq

while IFS='' read -r line || [[ -n "$line" ]]; do
    IFS=, read -ra array <<< "$line"
    nickname=${array[0]}
    user=${array[1]}
    pass=${array[2]}
    data=($(megadf --username=$user --password=$pass))
    data[${#data[@]}]+="$nickname"
    stats=$(join_by $'\n' ${data[@]})
    echo $stats
done < .accounts  | jq --slurp --raw-input 'split("\n")[:-1] | map([ split(" ")[] ]) | map({
    nick: .[6],
    total: .[1],
    used: .[3],
    free: .[5]
    })' >> /opt/stats/json/accounts.json

answered Aug 30, 2018 at 0:40

Ryahn

5779 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Charles Duffy Over a year ago

Consider running your code through shellcheck.net -- there are a bunch of quoting errors in here. Look at what happens if a user has hi * world as their password -- the * will be replaced with a list of filenames in the current directory. Similarly, echo $stats is removes the newlines you just put in with join_by, by splitting on all whitespace, passing each word as a separate argument to echo, and letting echo re-join content with regular spaces.

Ryahn Over a year ago

@CharlesDuffy Thank you for that tool. Going to help fix all my own code, than having to pester all the wonderful people here <3

Collectives™ on Stack Overflow

JQ create json array using bash

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related