2

Newbie trying to learn bash and jq. I'm on a Windows machine so I thought I'd test with git bash - I thought git bash was identical to bash running on Linux.

For my test, I'm trying to extract a list from JSON into a variable. However, I was getting (what I thought to be) odd results, so I tried the same code on Ubuntu in a Docker container. Ubuntu gave me the expected results.

The left side is Ubuntu running in Docker. The right side is git bash. Can someone explain why the two results are different?

bash example

Sorry, here's the text:

Ubuntu:

root@ac88bd1c736e:/usr/local/bin# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"
root@ac88bd1c736e:/usr/local/bin# jq --version
jq-1.6
root@ac88bd1c736e:/usr/local/bin# ls -la
total 20
drwxr-xr-x 1 root root 4096 Mar  8 16:48 .
drwxr-xr-x 1 root root 4096 Mar  1 02:03 ..
-rwxr-xr-x 1 root root   56 Jul 25  2022 service.py
-rwxr-xr-x 1 root root   96 Mar  7 18:21 test.json
root@ac88bd1c736e:/usr/local/bin# cat test.json
{
        "foo": "bar",
        "access": [
                "rangers",
                "avs"
        ],
        "blah": {
                "bizz": "buzz"
        }
}root@ac88bd1c736e:/usr/local/bin# foo=`cat test.json | jq .access`
root@ac88bd1c736e:/usr/local/bin# echo $foo
[ "rangers", "avs" ]
root@ac88bd1c736e:/usr/local/bin#

git bash:


leehenry@PW03T2JD MINGW64 ~/..../bash
$ git --version
git version 2.39.2.windows.1

leehenry@PW03T2JD MINGW64 ~/..../bash
$ jq --version
jq-1.6

leehenry@PW03T2JD MINGW64 ~/..../bash
$ cat test.json
{
        "foo": "bar",
        "access": [
                "rangers",
                "avs"
        ],
        "blah": {
                "bizz": "buzz"
        }
}
leehenry@PW03T2JD MINGW64 ~/..../bash
$ foo=`cat test.json | jq .access`

leehenry@PW03T2JD MINGW64 ~/..../bash
$ echo $foo
 ]avs"ers",

leehenry@PW03T2JD MINGW64 ~/..../bash
$
Improve this question
4
  • 1
    This, $ echo $foo → ]avs"ers", is typical of a Windows format file being processed on a UNIX/Linux machine Commented Mar 9, 2023 at 19:14
  • As roaima says, it seems like jq outputs DOS-formatted text on the latter system, while the shell expects Unix-formatted text. Where did you install jq from on that system? Commented Mar 9, 2023 at 19:15
  • @Kusalananda for git-bash, I got the executable from github (github.com/stedolan/jq/releases/download/jq-1.6/jq-win64.exe) and dropped it here: C:\Program Files\Git\usr\bin. Commented Mar 9, 2023 at 19:31
  • @Kusalananda now i'm wondering if i should've grabbed the Linux executable. I'll try that I'll also look at line endings. Thanks. Commented Mar 9, 2023 at 19:32

1 Answer 1

Reset to default
0

This is CRLF vs LF line terminator trouble.

I would expect

$ cat test.json | jq .access

to look fairly normal, with each output CR being invisible and then a LF takes us down to next line.

You can verify with

$ cat test.json | jq .access | hexdump -C

or more compactly by just piping through | hd. Note the 0x0d CRs and the 0x0a LFs.

After apt install dos2unix you can fix jq's output by piping it through | dos2unix, and verify with hexdump if you like.

The bash manpage is rather long. Here is the specific place where

$ foo=`cat test.json | jq .access`

was tripping you up.

Command Substitution ...

Bash performs the expansion by ... replacing the command substitution with the standard output of the command, with any trailing newlines deleted.

So think of it as a tr that deleted LFs, but preserved CRs.

Note also this variable:

IFS

The Internal Field Separator that is used for word splitting after expansion and to split lines into words with the read builtin command. The default value is <space><tab><newline>.

Grep'ing for angle newline reveals a few more places where that crops up.

So if you're going to ask bash to parse jq output that contains CRs, you might want to adjust your IFS setting.

Or, choose a compute context better suited to your needs.

hexdump usually comes standard with Ubuntu.

https://packages.ubuntu.com/jammy/amd64/bsdextrautils/filelist

If missing, you can use apt install bsdmainutils to pull it in.

Improve this answer
6
  • cat test.json | jq .access works fine in both environments, however assigning the results to a variable like this throws an error: foo=cat test.json | jq .access. Command substitution got around that, though I don't know the reason behind it. Seems like that's a separate question that I don't understand. Also, hexdump doesn't seem to available in either the Ubuntu image or git-bash. Think I have a lot of work to understand this. Commented Mar 9, 2023 at 20:14
  • 1
    @J_H, command substitution only removes trailing newlines, not all of them. I.e. it turns foo\nbar\n into foo\nbar. Then the echo prints one additional newline at the end on the next line. That has nothing to do with the carriage returns. Commented Mar 9, 2023 at 20:33
  • Yeah, I guess "with any trailing newlines" denotes just final singular newline. Commented Mar 9, 2023 at 20:36
  • 1
    @HenryLee, foo=cat test.json | jq .access, exactly like that means to assign the string cat to variable foo, and then run test.json | jq .access (probably giving a command not found error). If you want the output of a command assigned to a variable, then a command substitution is the thing for that. I don't understand why passing the output through a variable would change things (but then I don't know what weirdness that "git bash" thing involves). Commented Mar 9, 2023 at 20:37
  • 1
    @ilkkachu oh, duh..i see...ok, that makes sense. One mystery down...thanks! Commented Mar 9, 2023 at 20:42

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.