1

I've created a script that accesses a website with the use of a datafile and outputs the sites responses (one line XML) to an output file. I would like the output to start with the query of the datafile and then the response of the website. When I echo the query one one line and write it to an output file and then write the site's response to the same output file it uses two lines but I only want one line because I would like to end up with a comma separated file that I can import in excel.

This works but with having two lines of data:

while read -r line || [[ -n $line ]]
do
datatogather="$line"
echo $datatogather >>outputfile.txt
curl http://login:[email protected]/application.php?$datatogather >>outputfile.txt
echo >>outputfile.txt
done < datafile.txt

This doesn't work (although it shows the comma in the output file, so that line is being processed):

while read -r line || [[ -n $line ]]
do
datatogather="$line"
echo $datatogather,>>outputfile.txt | curl http://login:[email protected]/application.php?$datatogather >>outputfile.txt
echo >>outputfile.txt
done < datafile.txt

Stripping the output file of it's garbage data with sed was a breeze to figure out, even reading the input file into the site was very easy compared to figuring out how to use a variable more than once in a single line. Hope you can help me.

Improve this question

1 Answer 1

Reset to default
1

Do one echo operation instead of two, and only one redirection instead of many, and use command substitution to capture the output of curl:

while read -r line && [[ -n "$line" ]]
do
    echo "$line,$(curl http://login:[email protected]/application.php?$line)"
done < datafile.txt  >>outputfile.txt

Note that the test is changed from || to &&, and personally I don't like unquoted variables, though [[ is less problematic in some respects than [ (but introduces other problems, IMO).

The only nasty feature there is that the double quotes mean that newlines in the website response are preserved in the output. If you like living dangerously, you could simply remove the double quotes. It would probably be better to revise it to map the newlines to spaces:

while read -r line && [[ -n "$line" ]]
do
    echo "$line,$(curl http://login:[email protected]/application.php?$line | tr '\n' ' ')"
done < datafile.txt  >>outputfile.txt

Note that you're liable to have problems if the output from the website includes any commas in its output, but you've not (yet) asked about that.

You do not want to put the tr operation outside the loop; you want a newline at the end of each echo, so this would be bad:

while read -r line && [[ -n "$line" ]]
do
    echo "$line,$(curl http://login:[email protected]/application.php?$line)"
done < datafile.txt | tr '\n' ' ' >>outputfile.txt

Improve this answer
Sign up to request clarification or add additional context in comments.

11 Comments

You changed read -r line || [[ $line ]] into read -r line && [[ $line ]]. This completely changes the meaning of the code.
The former case is a way to deal with incomplete lines—and hence files that are not text file (I really dislike the fact that more and more people think that it's smart and it looks cool to do read -r line || [[ $line ]] when they don't even know what a line is.
The latter case will break the loop as soon as there's an empty line (or on incomplete lines).
The quotes inside [[ are mandatory on the rhs of equalities. Here it's safe to use [[ -n $line ]] or even better [[ $line ]]. Now you're mention quotes: you should definitely quote the address http://login:[email protected]/application.php?$line in the curl statement! you clearly have a glob character ? and a variable expansion! this is where you need quotes_.
Thanks for the response. Only thing is that I get a line break in my output. it shows the line value with the comma on one line and the output on the other line. My input datafile.txt has a variable on each line, the output created from the website is also a variable (xml-string) on each line. The following doesn't work: echo "$(echo "$line" | tr '\n' ' '),$(curl login:[email protected]/application.php?$line)"
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.