Use Variable in CURL GET

Question

I feel that I am just missing something very silly and not quoting/escaping something as I should, but I've been reading and testing for a solid amount of time now, and just can't get this to work.

I have a CSV file with data like the below:

TESTING.csv

17A3120UAXF-AA002771,9911017093
S150Y52157201,9911008933
17A3120UAXF-AA004545,9911016519
S170Y13084226,9911024365
S160Y45021270,9911018486

For the first part of my script, I need to read the second variable (ie. the items starting with "99110...")

I am parsing this data into a CURL while loop to sequence through all lines in the file. My bash script is like the below at this time:

#!/bin/bash

while IFS=, read -r device account; do
echo "device : $device"
echo "account : $account"
curl -X GET --header "Content-Type: application/json" --user username:password --verbose 'https://www.website.com:8444/api/subscriber?account=$account'
done < TESTING.csv

The issue that I'm running into is that while the "echo" statements are able to correctly pull/show the variable that I'm wanting to pass, this same information is not being passed into my CURL commands. When I run my script, the output is like the below:

device : 17A3120UAXF-AA002771
account : 9911017093
* About to connect() to www.website.com port 8444 (#0)
*   Trying 8.8.8.8...
* Connected to www.website.com (8.8.8.8) port 8444 (#0)
* Server auth using Basic with user 'username'
> GET /api/subscriber?account=$account HTTP/1.1
> Authorization: Basic madeUpJunkAndNumbers12345==
> User-Agent: curl/7.29.0
> Host: www.website.com:8444
> Accept: */*
> Content-Type: application/json
>
< HTTP/1.1 404 Not Found
< Content-Type: application/json
< Content-Length: 0
<
* Connection #0 to host www.website.com left intact
^C

I think it's here 'https://www.website.com:8444/api/subscriber?account=$account' change ' to " — Ivan
– Ivan, Commented Feb 6, 2020 at 6:04

Poshi · Accepted Answer · 2020-02-06 06:03:06Z

2

You are using single quotes int the URL, so the variables don't get expanded. Use double quotes:

curl -X GET --header "Content-Type: application/json" --user username:password --verbose "https://www.website.com:8444/api/subscriber?account=$account"

answered Feb 6, 2020 at 6:03

Poshi

5,8533 gold badges22 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Drew Over a year ago

Thanks for answering Poshi! I had tried that before, and just retested now, and it throws a curl error: device : 17A3120UAXF-AA002771 account : 9911017093 * Illegal characters found in URL * Closing connection -1 curl: (3) Illegal characters found in URL ^C

Yuri G. Over a year ago

@Drew this is definitely the answer. Try to check for non printable characters in csv file.Also you can update your question with the curl output with double quotes

Daniel Stenberg Over a year ago

"Illegal characters found in URL" meant that there are CR or LF characters in the URL. Modern curl has changed that error reporting though...

Poshi Over a year ago

As other people told you, you have something wrong with your URL. Check its characters, including non-printable, CR, LF...

Drew Over a year ago

Ah, it was indeed characters hiding. This was a CSV file directly from Windows Excel. I did the a "tr -d '\r' < TESTING.csv > TESTING_FIXED.csv", and then re-ran with the double quotes, and it passed through how I would expect it to. Thanks for all of the help!

Collectives™ on Stack Overflow

Use Variable in CURL GET

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related