How to use variable for URL inside cURL GET command

Question

I'm trying to get a specific json field from a cURL GET command in a bash script from an Elasticsearch index. To to this I have set up a script to pass variables into the cURL command. Hard coding the values gives the proper response so I just need to figure out how to use variables so I can get different responses for different purposes.

This is the desired outcome and the current working command:

docs=`curl -k -u admin:hunter1 -HContent-Type:application/json -X GET 'https://10.10.15.15:9200/logstash-linux-pci-2019.03.15/_stats' | jq '._all.primaries.docs.count'`
echo "The document count is "$docs

The document count is 183,283,999

I am importing line-by-line indexes from a text file that might look something like this:

list.txt

logstash-linux-pci-2019.03.15
logstash-linux-pci-2019.03.16
logstash-linux-pci-2019.03.17
logstash-linux-pci-2019.03.18

I need to place an ip variable along with these^ into a curl command as such:

reindex.sh

user='admin'
pass='hunter1'
ip='https://10.10.15.15:9200'
indexList=(`cat "list.txt"`)

for index in "${indexList[@]}"
do
    echo "Reindexing: "$index
    docs=`curl -k -u $user:$pass -HContent-Type:application/json -X GET '$ip/$index/_stats' | jq '._all.primaries.docs.count'`
    echo "index "$index" has a doc count of "$docs
done
echo "Complete!"

The only thing that doesn't work in the above example is ....GET '$ip/$index/_stats'....

I have tried hard coding the variables as plain text in the command like a normal curl GET and it works fine, I can get the doc count for the hard coded index.

I have tried:

....GET '$ip/$index/_stats'....

Throws error: curl: (52) Empty reply from server

.

....GET "${ip}/${index}/_stats"....

Throws error: curl: (52) Empty reply from server

.

....GET 'http://10.10.15.15/'"$index"'/_stats'....

Throws error: curl: (52) Empty reply from server

.

....GET '$ip/$index/_stats'....

Throws error: curl: (52) Empty reply from server

.

url="${ip}/${index}/_stats"
....GET $url ....

Throws error: curl: (52) Empty reply from server

.

I have searched online for hours but anything I find is related to curl PUT and putting variables as json onto servers. How can I curl with a variable URL and index?

Jason · Accepted Answer · 2019-10-11 13:00:30Z

Edit: For anyone who finds this in the future. The actual cause of the issue was using http instead of https.

You are using the quotes all wrong.

Variables inside '...' are not going to expand. However, they will expand just about anywhere else like within double quotes. For example this line:

echo "index "$index" has a doc count of "$docs

Here, the second double quote is terminating the first one. These cannot be nested. You could just write this as:

echo "index $index has a doc count of $docs"

So that means your variables are not going to expand in here:

....GET '$ip/$index/_stats'....

This is equivalent to actually sending the literal string $ip/$index/_stats. You want to use double quotes here: ....GET "$ip/$index/_stats".....

Another thing to watch out for is quoting the result of your variable assignment. If for whatever reason the result of this command:

curl -k -u $user:$pass -HContent-Type:application/json -X GET '$ip/$index/_stats' | jq '._all.primaries.docs.count'

was foo bar, then the variable assignment turns into docs=foo bar. This is the same as saying set a variable docs=foo and execute a program bar. So you should be quoting the entire assignment. However, take a look at it with a quoted assignment:

docs="`curl -k -u $user:$pass -HContent-Type:application/json -X GET "$ip/$index/_stats" | jq '._all.primaries.docs.count'`"

This is starting to look pretty ugly and confusing. This is one reason why it is recommended to use the syntax $(...) over the backticks. They are functionally equivalent, but it is easier to read what is going on here:

docs="$(curl -k -u $user:$pass -HContent-Type:application/json -X GET "$ip/$index/_stats" | jq '._all.primaries.docs.count')"

So, I went on a little tangent, but that is important stuff. In all reality, this solution that you tried should have worked: ....GET "${ip}/${index}/_stats"..... I think your problem may actually be more related to the fact that it is in a script since it worked fine outside of it. I don't see a shebang line and you may not even be running bash. You are using some features (like the array) that are specific to bash. Try adding #!/bin/bash to the top of the script. If this doesn't work, try running your script like this: bash -x reindex.sh. This is like a debug mode for bash, and it will show you the interpreted commands that are being executed.

I forgot to put the shebang in the post but it was in the program. My problem turned out to be that somewhere along the way of working on this I started using http instead of https. This causes curl: (52) Empty reply from server Lesson learned, always check error codes before assuming it's another problem.

Collectives™ on Stack Overflow

How to use variable for URL inside cURL GET command

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related