0

I'm trying to do something akin to this:

jq -r  '. | ."Time Series (Daily)"."2020-12-02" | ."1. open"' newdata.json

...but with the key coming from a variable, as in:

jq -r --arg key "$key" '. | ."Time Series (Daily)"."[$key]" | ."1. open"' newdata.json

The first one works just fine, but when I assign the date to a variable called key and then try to get the data, it fails.

I tried This answer and This answer. But did not work for me.

{
    "Meta Data": {
        "1. Information": "Daily Prices (open, high, low, close) and Volumes",
        "2. Symbol": "AB",
        "3. Last Refreshed": "2020-12-02",
        "4. Output Size": "Compact",
        "5. Time Zone": "US/Eastern"
    },
    "Time Series (Daily)": {
        "2020-12-02": {
            "1. open": "32.6700",
            "2. high": "33.3300",
            "3. low": "32.5000",
            "4. close": "33.1200",
            "5. volume": "273799"
        },
        "2020-12-01": {
            "1. open": "32.1500",
            "2. high": "32.8000",
            "3. low": "32.0000",
            "4. close": "32.6000",
            "5. volume": "265086"
        },
        "2020-11-30": {
            "1. open": "32.3800",
            "2. high": "32.4900",
            "3. low": "31.7500",
            "4. close": "31.8700",
            "5. volume": "251970"
        }
    }
}

The above is the newdata.json file. What I want to get is the "1. open" value. I am using a for loop to iterate over all the keys of "Time Series (Daily)" and the keys are generated correctly. There is no issue with that. I then want to use the $key variable in each iteration to get the data I need.

readarray keys <<< "$(jq  '."Time Series (Daily)" |  keys[]' newdata.json)"
for key in "${keys[@]}"; do
  jq -r --arg key "$key"  '. | ."Time Series (Daily)" | .[$key] | ."1. open"' newdata.json
done
Improve this question
7
  • Also, bash !== sh, please tag accordingly. Commented Dec 3, 2020 at 15:05
  • Now I have updated it. Commented Dec 3, 2020 at 15:13
  • @CharlesDuffy the thing is that I have tried all these different version which were asked by some other stackoverflow users. This was one of those version I tried. Commented Dec 3, 2020 at 15:14
  • 1
    So, the bug you have here is because you didn't use the -t argument to readarray, so your keys have newlines in them. And you didn't use -r on the first jq, so they have literal quotes in them too. Commented Dec 3, 2020 at 16:16
  • 1
    BTW, this is in the class of bugs that running set -x in your script to log the commands that it invokes would have found for you, since you'd see the --argstr key "$key" passing a different key value than intended. Commented Dec 3, 2020 at 16:18

2 Answers 2

Reset to default
1

Focusing On The Immediate Issue

The problem isn't how you're passing key to jq; the problem is how you're populating the key variable in the first place.

Change:

readarray keys <<< "$(jq  '."Time Series (Daily)" |  keys[]' newdata.json)"

...to:

readarray -t keys <<< "$(jq -r '."Time Series (Daily)" |  keys[]' newdata.json)"

There are two changes here:

  1. We added the -t argument to readarray, so it no longer includes the newline ending each line in the variable itself.
  2. We added the -r argument to jq, so it no longer adds literal quotes around the strings.

Sidebar: Retrieving both keys and values at the same time

There's no reason to do one pass to retrieve keys and another to retrieve values -- better to just get them all at once:

dates=( )
opening_prices=( )
while IFS=$'\t' read -r date opening_price; do
  dates+=( "$date" )
  opening_prices+=( "$opening_price" )
done < <(
  jq -r '
    ."Time Series (Daily)" | to_entries[] | [.key, .value."1. open"] | @tsv
  ' <newdata.json
)

...after which, declare -p dates opening_prices emits:

declare -a dates=([0]="2020-12-02" [1]="2020-12-01" [2]="2020-11-30")
declare -a opening_prices=([0]="32.6700" [1]="32.1500" [2]="32.3800")

Original response (before population of keys was shown)

Here's a different approach that only calls jq once, instead of once per item, while still getting your keys from an array. It does this by using -R to read raw strings as input; . is then used to address those inputs (which we rename to $key to make it clear how this lines up with the old code).

keys=("2020-12-02" "2020-12-01" "2020-11-30")
readarray -t openingPrices < <(
  jq -Rr --slurpfile indatas newdata.json '
    $indatas[0] as $indata | . as $key |
    $indata."Time Series (Daily)"[$key]["1. open"]
  ' < <(printf '%s\n' "${keys[@]}")
)

After running that, declare -p keys openingPrices (to show how both arrays are defined) emits:

declare -a keys=([0]="2020-12-02" [1]="2020-12-01" [2]="2020-11-30")
declare -a openingPrices=([0]="32.6700" [1]="32.1500" [2]="32.3800")

...so you have an output array that lines up with your input array (so long as the latter isn't sparse).

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

This worked for me when I use it in the exact way you gave it. but what I want is not doing this by hard coding the "keys" array. I tried this same exact code using the generated keys array. But that did not work at all. returned null values for the openingPrices. gist.github.com/smooth-felix/57a19c76f4bc4e64991b7e54756ee05b check this gist. I have the output I got.
As I told you in a comment on the question, there are two things you're doing wrong when you generate keys. You can't have literal quotes in the values, and need to tell readarray not to sure the newline characters.
(blegh, s/sure/store/; typed that above comment on a phone).
1

Use | .[$key] | to get the key from your $key variable;

key="2020-12-02"
jq -r --arg key "$key"  '."Time Series (Daily)" | .[$key] | ."1. open"' newdata.json
# output: 32.6700

Or, combined with the for() (hardcoded keys, since we're not sure how you get those)

keys=("2020-12-02" "2020-12-01" "2020-11-30")
for key in "${keys[@]}"; do
    jq -r --arg key "$key"  '."Time Series (Daily)" | .[$key] | ."1. open"' newdata.json
done

32.6700

32.1500

32.3800

Improve this answer

15 Comments

readarray keys <<< "$(jq '."Time Series (Daily)" | keys[]' newdata.json)" for key in "${keys[@]}"; do jq -r --arg key "$key" '. | ."Time Series (Daily)" ."\($key)" | ."1. open"' newdata.json done This was the code I wrote. But the result is just null values
We can't help you debug if you don't provide enough info. What is readarray, why did you remove the first . | ."Time...
Just [$key] works fine without the extra quotes; there's no reason for string interpolation here.
@0stone0, readarray is a bash builtin (added in version 4.0; it's a different name for mapfile, also added in that same release).
@RathinduWathsala, it absolutely does work when used correctly. If you want to claim it doesn't, you need to show how you're using it that doesn't work so we can see the problem ourselves.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.