Can't add json items to the end of json array. Out of array

Question

I wan to to add json items to the end of current array "issues".

I try this in bash:

#!/bin/bash
currentReportContent=`cat $FILE_REPORT`
for (( pageNumber=1; pageNumber<=$COUNT_PAGES; pageNumber++ ))        
do
    read -r currentIssuesArr < <(echo $(curl -s -XGET $BASE_URL/issues/search\?componentKeys\=$PROJECT_KEY\&p\=$pageNumber\&ps\=$PAGE_SIZE | jq '.issues'))
    echo  "before_currentIssuesArr = $currentIssuesArr"
    # remove first [ and last ]
    resultIssues=${currentIssuesArr:1:${#currentIssuesArr}-2}
    echo "resultIssues = $resultIssues"
    currentReportContent=$(jq -r ".issues[.issues| length] |= . + $resultIssues" <<< $currentReportContent) 
    echo "$currentReportContent" > "$FILE_REPORT"
    done

Here output

 before_currentIssuesArr = [ { "key": "AX0h7PXHENszTAxEiB7V", "rule": "java:S1871", "severity": "MAJOR", "component": "myProject", "project": "some_component", "line": 390, "hash": "129a49831e0f7ba156e0f3e4e4508a8c", "textRange": { "startLine": 390, "endLine": 393, "startOffset": 2, "endOffset": 9 }, "flows": [ { "locations": [ { "component": "myProject/server-services/services-cfg/cfg-impl/src/", "textRange": { "startLine": 386, "endLine": 389, "startOffset": 2, "endOffset": 9 }, "msg": "Original" } ] } ], "status": "OPEN", "message": "This case's code block is the same as the block for the case on line 386.", "effort": "10min", "debt": "10min", "author": "eugeniur", "tags": [ "design", "suspicious" ], "creationDate": "2021-11-15T06:40:16+0200", "updateDate": "2021-11-15T06:40:16+0200", "type": "CODE_SMELL", "scope": "MAIN" }, { "key": "AX0h7PXvENszTAxEiB7W", "rule": "java:S122", "severity": "MINOR", "component": "myProject/server-services", "project": "some_component", "line": 238, "hash": "e88a41c67f42388448824393203574dd", "textRange": { "startLine": 238, "endLine": 238, "startOffset": 0, "endOffset": 87 }, "flows": [], "status": "OPEN", "message": "At most one statement is allowed per line, but 2 statements were found on this line.", "effort": "1min", "debt": "1min", "author": "eugeniur", "tags": [ "convention" ], "creationDate": "2021-11-15T06:40:16+0200", "updateDate": "2021-11-15T06:40:16+0200", "type": "CODE_SMELL", "scope": "MAIN" } ]
    
    resultIssues =  { "key": "AX0h7PXHENszTAxEiB7V", "rule": "java:S1871", "severity": "MAJOR", "component": "myProject/server-services/services-cfg/cfg-impl/src/", "project": "some_component", "line": 390, "hash": "129a49831e0f7ba156e0f3e4e4508a8c", "textRange": { "startLine": 390, "endLine": 393, "startOffset": 2, "endOffset": 9 }, "flows": [ { "locations": [ { "component": "myProject/server-services/services-cfg/cfg-impl/src/", "textRange": { "startLine": 386, "endLine": 389, "startOffset": 2, "endOffset": 9 }, "msg": "Original" } ] } ], "status": "OPEN", "message": "This case's code block is the same as the block for the case on line 386.", "effort": "10min", "debt": "10min", "author": "eugeniur", "tags": [ "design", "suspicious" ], "creationDate": "2021-11-15T06:40:16+0200", "updateDate": "2021-11-15T06:40:16+0200", "type": "CODE_SMELL", "scope": "MAIN" }, { "key": "AX0h7PXvENszTAxEiB7W", "rule": "java:S122", "severity": "MINOR", "component": "myProject/server-service", "project": "some_component", "line": 238, "hash": "e88a41c67f42388448824393203574dd", "textRange": { "startLine": 238, "endLine": 238, "startOffset": 0, "endOffset": 87 }, "flows": [], "status": "OPEN", "message": "At most one statement is allowed per line, but 2 statements were found on this line.", "effort": "1min", "debt": "1min", "author": "eugeniur", "tags": [ "convention" ], "creationDate": "2021-11-15T06:40:16+0200", "updateDate": "2021-11-15T06:40:16+0200", "type": "CODE_SMELL", "scope": "MAIN" }

As you can see it successfully removes the first [ and last ].

Nice.

Now I try to add resultIssues to the end of array issues.

But here output in the file:

 "issues": [
    {
      "key": "AX0h7PXHENszTAxEiB7V",
      "rule": "java:S1871",
      "severity": "MAJOR",
      "component": "myProject",
      "project": "some_project",
      "line": 390,
      "hash": "129a49831e0f7ba156e0f3e4e4508a8c",
      "textRange": {
        "startLine": 390,
        "endLine": 393,
        "startOffset": 2,
        "endOffset": 9
      },
      "flows": [
        {
          "locations": [
            {
              "component": "myProject",
              "textRange": {
                "startLine": 386,
                "endLine": 389,
                "startOffset": 2,
                "endOffset": 9
              },
              "msg": "Original"
            }
          ]
        }
      ],
      "status": "OPEN",
      "message": "This case's code block is the same as the block for the case on line 386.",
      "effort": "10min",
      "debt": "10min",
      "author": "eugeniur",
      "tags": [
        "design",
        "suspicious"
      ],
      "creationDate": "2021-11-15T06:40:16+0200",
      "updateDate": "2021-11-15T06:40:16+0200",
      "type": "CODE_SMELL",
      "scope": "MAIN"
    }
  ]
}
{
  "key": "AX0h7PXvENszTAxEiB7W",
  "rule": "java:S122",
  "severity": "MINOR",
  "component": "some_",
  "project": "some_project",
  "line": 238,
  "hash": "e88a41c67f42388448824393203574dd",
  "textRange": {
    "startLine": 238,
    "endLine": 238,
    "startOffset": 0,
    "endOffset": 87
  },
  "flows": [],
  "status": "OPEN",
  "message": "At most one statement is allowed per line, but 2 statements were found on this line.",
  "effort": "1min",
  "debt": "1min",
  "author": "eugeniur",
  "tags": [
    "convention"
  ],
  "creationDate": "2021-11-15T06:40:16+0200",
  "updateDate": "2021-11-15T06:40:16+0200",
  "type": "CODE_SMELL",
  "scope": "MAIN"
}

Why some json items out of array "issues"?

BTW, ideally, all locally-defined variables should have at least one lower-case character: all-caps variable names are used by variables that change the behavior of the shell itself or other POSIX-defined tools. See the relevant standard document at pubs.opengroup.org/onlinepubs/9699919799/basedefs/…, keeping in mind that setting a shell variable overwrites any like-named environment variable; so the only way to avoid overwriting environment variables in reserved space is to also keep regular shell variables out of that space as well. — Charles Duffy
– Charles Duffy, Commented Nov 17, 2021 at 13:31

Botje · Accepted Answer · 2021-11-17 13:28:28Z

3

Why are you mangling JSON with sed if you already have jq?

I would simplify the loop to:

currentReportContent=`cat "$FILE_REPORT"`
for (( pageNumber=1; pageNumber<=$COUNT_PAGES; pageNumber++ ))        
do
    read -r resultIssues < <(curl -s -XGET "$BASE_URL/issues/search?componentKeys=$PROJECT_KEY&p=$pageNumber&ps=$PAGE_SIZE" | jq '.issues')
    currentReportContent=$(jq --argjson resultIssues "$resultIssues" '.issues |= . + $resultIssues' <<< "$currentReportContent") 
    echo "$currentReportContent" > "$FILE_REPORT"
done

and if you don't mind a few temporary files it becomes massively simpler yet:

allPages=$(mktemp allPages.XXXXXX)

for (( pageNumber=1; pageNumber<=$COUNT_PAGES; pageNumber++ ))        
do
    curl -s -XGET "$BASE_URL/issues/search?componentKeys=$PROJECT_KEY&p=$pageNumber&ps=$PAGE_SIZE" | jq '.issues[]' >> $allPages
done

jq --slurpfile resultIssues $allPages '.issues |= . + $resultIssues' "$FILE_REPORT" | sponge "$FILE_REPORT"

edited Nov 17, 2021 at 13:28

answered Nov 17, 2021 at 12:55

Botje

32.1k4 gold badges36 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Charles Duffy Over a year ago

Not sure you even need the temporary file; why not pass slurpfile a process substitution?

Charles Duffy Over a year ago

(also, you're using double quotes in the second code block when you want $resultIssues to be a jq variable rather than a bash variable -- that should be single quotes -- and there are missing double quotes around other shell expansions; safer to add them to avoid surprising behavior when variables don't have the values they should)

Charles Duffy Over a year ago

Also, what's the point to < <(echo $(curl ...) | jq) instead of just <(curl ... | jq)?

Botje Over a year ago

All good points, thanks. I did not touch the curl invocation in the first rewrite.

Alexei Over a year ago

@Botje Yes, with temp files no error "Argument list too long". And approach with temp files is VERY FAST. This approach is better.

|

Collectives™ on Stack Overflow

Can't add json items to the end of json array. Out of array

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related