jq json object concatenation to bash string array

Question

I want to use jq (or anything else when it's the wrong tool) to concatenate a json object like this:

{
  "https://github.com": {
    "user-one": {
      "repository-one": "version-one",
      "repository-two": "version-two"
    },
    "user-two": {
      "repository-three": "version-three",
      "repository-four": "version-four"
    }
  },
  "https://gitlab.com": {
    "user-three": {
      "repository-five": "version-five",
      "repository-six": "version-six"
    },
    "user-four": {
      "repository-seven": "version-seven",
      "repository-eight": "version-eight"
    }
  }
}

recursively to a bash string array like this:

(
    "https://github.com/user-one/repository-one/archive/refs/heads/version-one.tar.gz"
    "https://github.com/user-one/repository-two/archive/refs/heads/version-two.tar.gz"
    "https://github.com/user-two/repository-three/archive/refs/heads/version-three.tar.gz"
    "https://github.com/user-two/repository-four/archive/refs/heads/version-four.tar.gz"
    "https://gitlab.com/user-three/repository-five/-/archive/version-five/repository-five-version-five.tar.gz"
    "https://gitlab.com/user-three/repository-six/-/archive/version-six/repository-six-version-six.tar.gz"
    "https://gitlab.com/user-four/repository-seven/-/archive/version-seven/repository-seven-version-seven.tar.gz"
    "https://gitlab.com/user-four/repository-eight/-/archive/version-eight/repository-eight-version-eight.tar.gz"
)

for subsequent use in a loop.

for i in "${arr[@]}"
do
   echo "$i"
done

Have no idea how to do that. As you can see, the values must be handled differently depending on the object name.

"https://github.com" + "/" + $user_name + "/" + $repository_name + "/archive/refs/heads/" + $version + ".tar.gz"

"https://gitlab.com" + "/" + $user_name + "/" + $repository_name + "/-/archive/" + $version + "/" + $repository_name + "-" + $version + ".tar.gz"

Could anyone help?

The same techniques in the recently-answered question stackoverflow.com/questions/66891879/traversing-with-jq-yq/… are what you need here. — Charles Duffy
– Charles Duffy, Commented Apr 2, 2021 at 17:46
...thus, ideally, I would be closing this question as a duplicate, unless it explicitly asked a question that wasn't covered there. — Charles Duffy
– Charles Duffy, Commented Apr 2, 2021 at 17:57

Charles Duffy · Accepted Answer · 2021-04-02 18:17:59Z

4

Easily done.

First, let's focus on the jq code alone:

to_entries[]                # split items into keys and values
| .key as $site             # store first key in $site
| .value                    # code below deals with the value
| to_entries[]              # split that value into keys and values
| .key as $user             # store the key in $user
| .value                    # code below deals with the value
| to_entries[]              # split that value into keys and values
| .key as $repository_name  # store the key in $repository_name
| .value as $version        # store the value in $version
| if $site == "https://github.com" then
    "\($site)/\($user)/\($repository_name)/archive/refs/heads/\($version).tar.gz"
  else
    "\($site)/\($user)/\($repository_name)/-/archive/\($version)/\($repository_name)-\($version).tar.gz"
  end

That generates a list of lines. Reading lines into a bash array looks like readarray -t arrayname < ...datasource...

Thus, using a process substitution to redirect jq's stdout as if it were a file:

readarray -t uris < <(jq -r '
  to_entries[]
  | .key as $site
  | .value
  | to_entries[]
  | .key as $user
  | .value
  | to_entries[]
  | .key as $repository_name
  | .value as $version
  | if $site == "https://github.com" then
      "\($site)/\($user)/\($repository_name)/archive/refs/heads/\($version).tar.gz"
    else
      "\($site)/\($user)/\($repository_name)/-/archive/\($version)/\($repository_name)-\($version).tar.gz"
    end
  ' <config.json
)

edited Apr 2, 2021 at 18:17

answered Apr 2, 2021 at 17:55

Charles Duffy

299k43 gold badges441 silver badges497 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Tamaro Skaljic Over a year ago

Wow, thanks! I would never have figured that out on my own.

Léa Gris Over a year ago

Filtering the various $user, $repository_name, $version with the @url filter would also ensure proper URL-Encoding.

peak Over a year ago

@LéaGris - @url filter?

peak Over a year ago

@CharlesDuffy - Glad to see the "Farewell to Trailing Pipes".

Léa Gris Over a year ago

@peak I mean @uri. Like ( .key | @uri ) as $user, ( .key | @uri ) as $repository_name and ( .value | @uri ) as $version. Then if one of these contains an URI prohibited character, it will be %coded. Example a a version of "version-eight&nine" will get encoded as version-eight%26nine

peak · Accepted Answer · 2021-04-03 01:41:50Z

1

The basic task of generating the strings can be done efficiently and generically (i.e., without any limits on the depths of the basenames) using the jq filter:

paths(strings) as $p | $p + [getpath($p)] | join("/")

There are several ways to populate a bash array accordingly, but if you merely wish to iterate through the values, you could use a bash while loop, like so:

< input.json jq -r '
  paths(strings) as $p | $p + [getpath($p)] | join("/")' | 
  while read -r line ; do
    echo "$line"
  done

You might also wish to consider using jq's @sh or @uri filter. For a jq urlencode function, see e.g. https://rosettacode.org/wiki/URL_encoding#jq

(If the strings contain newlines or tabs, then the above would need to be tweaked accordingly.)

edited Apr 3, 2021 at 1:41

answered Apr 2, 2021 at 19:39

peak

119k21 gold badges185 silver badges218 bronze badges

Collectives™ on Stack Overflow

jq json object concatenation to bash string array

2 Answers 2

5 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related