Bash variable gets butchered when being inserted into cURL command

Question

The first part of the script looks like:

curl -k -X POST -H "Content-Type: application/x-www-form-urlencoded" -H "Cache-Control: no-cache" -d 'username=username&password=password "https://get_token" >> token.txt
token=$(<'token.txt'$'\r')
echo "$token"

The output from echo $token looks fine(417 characters):

eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIyTVd0bGo4Y1VPYW8xNmdZTlV4QWlFS3RLcTc1b2JsSEN0c3hKWnlxeExCTkRKb1wvKzNSK1l5XC9YUFhoWExLaytJRVdKSE5CTE9MZmxPNksxemFjYXVlM0t1NnFIQjNRXC9QeUxzUTdVSmVOYk9pS3FDeHhcLzk1dz09IiwibmJmIjoxNDgzOTg5NjgwLCJpc3MiOiJwaHgtcmVtc3QtMDAzIiwiX2NhY2hlSWQiOjE0MzA0OCwiZXhwIjoxNDgzOTkzNDAwLCJpYXQiOjE0ODM5ODk4MDAsImp0aSI6IklER0FBNVYwR1pNU0pBTzk5QlVXTzhOMDY5VlRGTiJ9.H3vD-WfZBFXOII4k-cy7Ey2QM9YXytp31m9-Huj9vKw

Let's see what happens when I try to insert the token into the next part of the script.

echo curl -k -X POST -H "Content-Type: application/json" -H "Authorization: AR-JWT "$token"" -H "Cache-Control: no-cache" -d '{ 
  "values":{
      
    "First_Name": "John",
    "Last_Name": "Doe",
  }
}  ' "https://randomurl.com"

Here is the output:

curl -k -X POST -H Content-Type: application/json -H Authorization: AR-JWT eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIyTVd0bGo4Y1VPYW8xNmdZTlV4QWlFS3RLcTc1b2JsSEN0c3hKWnlxeExCTkRKb1wvKzNSK1l5XC9YUFhoWExLaytJRVdKSE5CTE9MZmxPNksxemFjYXVlM0t1NnFIQjNRXC9QeUxzUTdVSmVOYk9pS3FDeHhcLzk1dz0 -H Cache-Control: no-cache -d { OiJwaHgtcmVtc3QtMDAzIiwiX2NhY2hlSWQiOjE0MzA0OCwiZXhwIjoxNDgzOTkzNDAwLCJpYXQiOjE0ODM5ODk4MDAsImp0aSI6IklER0FBNVYwR1pNU0pBTzk5QlVXTzhOMDY5VlRGTiJ9.H3vD-WfZBFXOII4k-cy7Ey2QM9YXytp31m9-Huj9vKw
  "values":{
      
    "First_Name": "John",
    "Last_Name": "Doe",
  }
}   https://randomurl.com

2 Things happen when the token is passed in

"9IiwibmJmIjoxNDgzOTg5NjgwLCJpc3Mi" within the token gets deleted
The 2nd half of the token(After the deleted part) gets inserted into the body

What I've tried:

Using a different token
Use token without passing it into a file first
Moving the Authorization header before the URL

All of these results in the same behavior.

I'm using cygwin64 on Windows 7 When the second cURL command is typed with full token in the correct space into cygwin, it works no problem

Output when set-x is added to the script:

$ ./variable_testing.sh
+ ./variable_testing.sh
+ token=$'eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIyTVd0bGo4Y1VPYW8xNmdZTlV4QWlFS3RLcTc1b2JsSEN0c3hKWnlxeExCTkRKb1wvKzNSK1l5XC9YUFhoWExLaytJRVdKSE5CTE9MZmxPNksxemFjYXVlM0t1NnFIQjNRXC9QeUxzUTdVSmVOYk9pS3FDeHhcLzk1dz09IiwibmJmIjoxNDgzOTg5NjgwLCJpc3MiOiJwaHgtcmVtc3QtMDAzIiwiX2NhY2hlSWQiOjE0MzA0OCwiZXhwIjoxNDgzOTkzNDAwLCJpYXQiOjE0ODM5ODk4MDAsImp0aSI6IklER0FBNVYwR1pNU0pBTzk5QlVXTzhOMDY5VlRGTiJ9.H3vD-WfZBFXOII4k-cy7Ey2QM9YXytp31m9-Huj9vKw\r'
+ echo $'eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIyTVd0bGo4Y1VPYW8xNmdZTlV4QWlFS3RLcTc1b2JsSEN0c3hKWnlxeExCTkRKb1wvKzNSK1l5XC9YUFhoWExLaytJRVdKSE5CTE9MZmxPNksxemFjYXVlM0t1NnFIQjNRXC9QeUxzUTdVSmVOYk9pS3FDeHhcLzk1dz09IiwibmJmIjoxNDgzOTg5NjgwLCJpc3MiOiJwaHgtcmVtc3QtMDAzIiwiX2NhY2hlSWQiOjE0MzA0OCwiZXhwIjoxNDgzOTkzNDAwLCJpYXQiOjE0ODM5ODk4MDAsImp0aSI6IklER0FBNVYwR1pNU0pBTzk5QlVXTzhOMDY5VlRGTiJ9.H3vD-WfZBFXOII4k-cy7Ey2QM9YXytp31m9-Huj9vKw\r\r'
eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIyTVd0bGo4Y1VPYW8xNmdZTlV4QWlFS3RLcTc1b2JsSEN0c3hKWnlxeExCTkRKb1wvKzNSK1l5XC9YUFhoWExLaytJRVdKSE5CTE9MZmxPNksxemFjYXVlM0t1NnFIQjNRXC9QeUxzUTdVSmVOYk9pS3FDeHhcLzk1dz09IiwibmJmIjoxNDgzOTg5NjgwLCJpc3MiOiJwaHgtcmVtc3QtMDAzIiwiX2NhY2hlSWQiOjE0MzA0OCwiZXhwIjoxNDgzOTkzNDAwLCJpYXQiOjE0ODM5ODk4MDAsImp0aSI6IklER0FBNVYwR1pNU0pBTzk5QlVXTzhOMDY5VlRGTiJ9.H3vD-WfZBFXOII4k-cy7Ey2QM9YXytp31m9-Huj9vKw
+ echo curl -k -X POST -H '"Content-Type:' 'application/json"' -H '"Authorization:' AR-JWT 'eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIyTVd0bGo4Y1VPYW8xNmdZTlV4QWlFS3RLcTc1b2JsSEN0c3hKWnlxeExCTkRKb1wvKzNSK1l5XC9YUFhoWExLaytJRVdKSE5CTE9MZmxPNksxemFjYXVlM0t1NnFIQjNRXC9QeUxzUTdVSmVOYk"' -H '"Cache-Control:' 'no-cache"' -d '{ LCJpc3MiOiJwaHgtcmVtc3QtMDAzIiwiX2NhY2hlSWQiOjE0MzA0OCwiZXhwIjoxNDgzOTkzNDAwLCJpYXQiOjE0ODM5ODk4MDAsImp0aSI6IklER0FBNVYwR1pNU0pBTzk5QlVXTzhOMDY5VlRGTiJ9.H3vD-WfZBFXOII4k-cy7Ey2QM9YXytp31m9-Huj9vKw
  "values":{
      
    "First_Name": "John",
    "Last_Name": "Doe",
  }
}  ' $'https://randomurl.com\r'
curl -k -X POST -H "Content-Type: application/json" -H "Authorization: AR-JWT eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIyTVd0bGo4Y1VPYW8xNmdZTlV4QWlFS3RLcTc1b2JsSEN0c3hKWnlxeExCTkRKb1wvKzNSK1l5XC9YUFhoWExLaytJRVdKSE5CTE9MZmxPNksxemFjYXVlM0t1NnFIQjNRXC9QeUxzUTdVSmVOYk9pS3FDeHhcLzk1" -H "Cache-Control: no-cache" -d { OiJwaHgtcmVtc3QtMDAzIiwiX2NhY2hlSWQiOjE0MzA0OCwiZXhwIjoxNDgzOTkzNDAwLCJpYXQiOjE0ODM5ODk4MDAsImp0aSI6IklER0FBNVYwR1pNU0pBTzk5QlVXTzhOMDY5VlRGTiJ9.H3vD-WfZBFXOII4k-cy7Ey2QM9YXytp31m9-Huj9vKw
  "values":{
      
    "First_Name": "John",
    "Last_Name": "Doe",
  }
}   https://randomurl.com

Full, simplified script that that displays output above

#!/bin/bash
set -x logs
token="eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIyTVd0bGo4Y1VPYW8xNmdZTlV4QWlFS3RLcTc1b2JsSEN0c3hKWnlxeExCTkRKb1wvKzNSK1l5XC9YUFhoWExLaytJRVdKSE5CTE9MZmxPNksxemFjYXVlM0t1NnFIQjNRXC9QeUxzUTdVSmVOYk9pS3FDeHhcLzk1dz09IiwibmJmIjoxNDgzOTg5NjgwLCJpc3MiOiJwaHgtcmVtc3QtMDAzIiwiX2NhY2hlSWQiOjE0MzA0OCwiZXhwIjoxNDgzOTkzNDAwLCJpYXQiOjE0ODM5ODk4MDAsImp0aSI6IklER0FBNVYwR1pNU0pBTzk5QlVXTzhOMDY5VlRGTiJ9.H3vD-WfZBFXOII4k-cy7Ey2QM9YXytp31m9-Huj9vKw"
echo "$token"
echo curl -k -X POST -H \"Content-Type: application/json\" -H \"Authorization: AR-JWT $token\" -H \"Cache-Control: no-cache\" -d '{ 
  "values":{
      
    "First_Name": "John",
    "Last_Name": "Doe",
  }
}  ' "https://randomurl.com"

The quoting in -H "Authorization: AR-JWT "$token""... I mean, frankly, what do you intend it to mean? Do you want the internal quotes to be literal, vs syntactic? — Charles Duffy
– Charles Duffy, Commented Jan 9, 2017 at 20:28
BTW, use printf '%q\n' "$token", or declare -p token, not echo "$token", to display your value in an unambiguous form. — Charles Duffy
– Charles Duffy, Commented Jan 9, 2017 at 20:29
In general -- don't ever use echo to debug what the shell is actually doing. You can't tell the difference between echo touch "hello world.txt" and echo touch hello world.txt, and those two commands are utterly unlike each other. — Charles Duffy
– Charles Duffy, Commented Jan 9, 2017 at 20:31
Could you update your question to show set -x logs, so we can see what the shell is actually passing to curl? — Charles Duffy
– Charles Duffy, Commented Jan 9, 2017 at 20:35
Also, what's with token=$(<'token.txt'$'\r')? Do you really have a CR at the end of your filename? I would tend to expect that to be evidence of a bug, as opposed to something you'd want to keep. (If your script is in CRLF format but cygwin is reading it expecting UNIX lines... well, convert it to a UNIX text-file format, with lines properly terminated with LF only!) — Charles Duffy
– Charles Duffy, Commented Jan 9, 2017 at 20:41

Charles Duffy · Accepted Answer · 2017-01-09 21:25:48Z

8

First: Your token currently has literal CRs in it -- the apparent butchering is caused by those CRs being printed in your echo output, moving the cursor back to the beginning of the line so parts of your token get overwritten with later content. To eliminate them, you can either convert the file into UNIX text format (and ensure that there aren't CRs in any of the commands generating that file), or you can use:

token=$(<token.txt)    # read token.txt into "token" variable
token=${token//$'\r'/} # strip CRs from token variable

If you want to pass the Authorization header to curl as a single word, don't end and reinitialize your quotes; keep that single argument inside one set of syntactic quotes.

-H "Authorization: AR-JWT $token"

Finally, make sure your script itself is in UNIX format -- that your editor isn't saving it with DOS newlines.

edited Jan 9, 2017 at 21:25

answered Jan 9, 2017 at 20:36

Charles Duffy

299k43 gold badges441 silver badges497 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Philippe Over a year ago

I have spent 2 hours to find out why I had this problem. Stripping the CRs from the token variable did it. Thanks!!

Collectives™ on Stack Overflow

Bash variable gets butchered when being inserted into cURL command

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related