1

I'm trying to use Python to extract info from some JSON (on a system where I can't install jq). My current approach runs afoul of the syntax restrictions described in Why can't use semi-colon before for loop in Python?. How can I modify this code to still work in light of this limitation?


My current code looks like the following:

$ SHIFT=$(aws ec2 describe-images --region "$REGION" --filters "Name=tag:Release,Values=$RELEASE_CODE_1.2003.2")
$ echo "$SHIFT" | python -c "import sys, json; for image in json.load(sys.stdin)['Images']: print image['ImageId'];"
  File "<string>", line 1
    import sys, json; for image in json.load(sys.stdin)['Images']: print image['ImageId'];
                        ^

SyntaxError: invalid syntax

Since Python's syntax doesn't allow a for loop to be separated from a prior command with a semicolon, how can I work around this limitation?

7
  • 1
    BTW, personally, I'd use jq -r '.Images[].ImageId' <<<"$SHIFT" rather than embedding Python here. Commented Jan 18, 2018 at 21:55
  • @CharlesDuffy: File "<string>", line 1 import sys, json;\nfor img in json.load(sys.stdin)['Images']:\n\tprint img['ImageId'] ^ SyntaxError: unexpected character after line continuation character Commented Jan 18, 2018 at 21:57
  • 1
    also can't install jq due to server limitations atm Commented Jan 18, 2018 at 21:58
  • the __import__ strategy worked wonders. thank you Commented Jan 18, 2018 at 22:00
  • if you make an answer I'll gladly accept it Commented Jan 18, 2018 at 22:05

1 Answer 1

4

There are several options here:

  • Pass your code as a multi-line string. Note that " is used to delimit Python strings rather than the original ' here for the sake of simplicity: A POSIX-compatible mechanism to embed a literal ' in a single-quoted string is possible, but quite ugly.

    extractImageIds() {
      python -c '
    import sys, json
    for image in json.load(sys.stdin)["Images"]:
        print image["ImageId"]
    ' "$@"
    }
    
  • Use bash's C-style escaped string syntax ($'') to embed newlines, as with $'\n'. Note that the leading $ is critical, and that this doesn't work with /bin/sh. See the bash-hackers' wiki on ANSI C-like strings for details.

    extractImageIds() { python -c $'import sys, json\nfor image in json.load(sys.stdin)["Images"]:\n\tprint image["ImageId"]' "$@"; }
    
  • Use __import__() to avoid the need for a separate import command.

    extractImageIds() { python -c 'for image in __import__("json").load(__import__("sys").stdin)["Images"]: print image["ImageId"]' "$@"; }
    
  • Pass the code on stdin and move the input onto argv; note that this only works if the input doesn't overwhelm your operating system's allowed maximum command-line size. Consider the following example:

    extractImageIds() {
      # capture function's input to a variable
      local input=$(</dev/stdin) || return
      # ...and expand that variable on the Python interpreter's command line
      python - "$input" "$@" <<'EOF'
    import sys, json
    for image in json.loads(sys.argv[1])["Images"]:
        print image["ImageId"]
    EOF
    }
    

    Note that $(</dev/stdin) is a more efficient bash-only alternative to $(cat); due to shell builtin support, it works even on operating systems where /dev/stdin doesn't exist as a file.

All of these have been tested as follows:

extractImageIds <<<'{"Images": [{"ImageId": "one"}, {"ImageId": "two"}]}'

To efficiently provide stdin from a variable, one could run extractImageIds <<<"$variable" instead. Note that the "$@" elements in the wrapper are there to ensure that sys.argv is populated with arguments to the shell function -- where sys.argv isn't referenced by the Python code being run, this syntax is optional.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.