curl post JSON file to REST API while also using --data-urlencode to encode query paramater

Question

Setup

In a Bash script I'm Using curl to POST the contents of a JSON file to a RESTful API running on tomcat behind nginx.

This POST also requires basic auth with 3 different query parameters on the end of the URL.

while IFS= read -d '' -r file; do
 base=$(basename "$file")
 datetime=$(find $file -maxdepth 0 -printf "%TY/%Tm/%Td %TH:%TM:%.2TS")
 username="vangeeij"
 curl -vX POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" \
   -u username:password \
   -d @"$file" \
   "http://192.168.105.10/homeaccess/services/aCStats/uploadData?username=$username&filename=$base" \
 --data-urlencode datetime=$datetime

 #sudo mv "$file" /home/vangeeij/acserver/resultsOld
done < <(sudo find . -type f -print0)

Results/Problem

as can be seen in the output from curl, the URL does NOT have the datetime= parameter attached tot he end of the URL

* Connected to 192.168.105.10 (192.168.105.10) port 80 (#0)
* Server auth using Basic with user 'username'
> POST /homeaccess/services/aCStats/uploadData?username=username&filename=2017_3_1_8_50_RACE.json HTTP/1.1
> Host: 192.168.105.10

Question

What is the proper syntax to accomplish what I'm trying here. Curl POST of JSON file to URL with the attachment of parameters to the URL, at least one of which HAS to be URLEncoded?

Shouldn't the datetime=$datatime be part of the url query string as &datetime=$datetime if you want to pass it to the server? — hcheung
– hcheung, Commented Mar 12, 2017 at 4:37
Correct.. but as mentioned, $datetime needs to be URLEncoded other wise it causes problems do to spaces and special characters. Hence the attempts at trying to utilize curl function --data-urlencode. — DarbyM
– DarbyM, Commented Mar 12, 2017 at 6:29
well i suggest to use '--trace-ascii my-trace.txt' in the file you will see what curl sends on the wire. Please can you show us the my-trace.txt, thanks. — Aleksandar
– Aleksandar, Commented Mar 12, 2017 at 8:58
The query string is primarily a way of adding a query to a GET request. The various --data options place the parameters in the body of the request, which anything processing a POST request is certain to look for. You should even be able to add --data username="$username" and filename="$base" in place of ?username=$username&filename=$base and get the same result. — chepner
– chepner, Commented Mar 12, 2017 at 19:16
@chepner I appreciate your feed back.. you gave me a better understanding of -G functionality... sadly.. I'M the one that wrote this API.. and currently it would not look in the body data for these particular parameters. But to be fair, there is a valid reason for it... I think. lol — DarbyM
– DarbyM, Commented Mar 12, 2017 at 23:36

DarbyM · Accepted Answer · 2017-03-12 23:49:31Z

Solution

I found the simplest way to handle the process of URLEncoding a string and passing it as a parameter in the URL was to modify my script as following.

#!/bin/bash
cd /home/vangeeij/acserver/results

urlencode() {
   # urlencode <string>
   old_lc_collate=$LC_COLLATE
   LC_COLLATE=C

   local length="${#1}"
   for (( i = 0; i < length; i++ )); do
       local c="${1:i:1}"
       case $c in
           [a-zA-Z0-9.~_-]) printf "$c" ;;
           *) printf '%%%02X' "'$c" ;;
       esac
   done

   LC_COLLATE=$old_lc_collate
}


while IFS= read -d '' -r file; do
   base=$(basename "$file")
   datetime=$(find $file -maxdepth 0 -printf "%TY/%Tm/%Td %TH:%TM:%.2TS")
   datetimeEncoded=$(urlencode "$datetime")
username="username"
   curl -vX POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" \
      -u username:password \
      -d @"$file" \
       "http://192.168.105.10/homeaccess/services/aCStats/uploadData?username=$username&filename=$base&datetime=$datetimeEncoded"
   sudo mv "$file" /home/vangeeij/acserver/resultsOld
 done < <(sudo find . -type f -print0)

Short Answer

I added the function "urlencode()". Then pass the String variable $datetime through it to encode the date time as needed.. then pass the resulting String variable $datetimeEncoded to the URL parameter datetime=$datetimeEncoded

Problem solved.

URLEncode script found here

Collectives™ on Stack Overflow

curl post JSON file to REST API while also using --data-urlencode to encode query paramater

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related