0

Problem statement:

  1. Read a file line by line
  2. Execute a command on each file
  3. Print the line and output of command in another file

Input file (urls.txt):

www.google.com
www.yahoo.com
www.bing.com

Expected output file (output.csv):

www.google.com,200
www.yahoo.com,500
www.bing.com,404

Script I have so far

while IFS= read -r line
do
    curl -LI "$line" -o /dev/null -w '%{http_code}\n' -s
 >> output.csv
done < urls.txt

not an expert in bash scripting. just a beginner. any help is much appreciated. Thank you. :)

Improve this question
4
  • So what is your question, exactly? Commented Mar 21, 2020 at 18:04
  • What doesn't work? Commented Mar 21, 2020 at 18:05
  • I want the output mentioned in Expected output file. Currently output.csv would only contain the http codes i.e. only the output of the command. Output I want is for 'www.google.com','200' is the response Commented Mar 21, 2020 at 18:08
  • So please edit your question to tell us that. You've shown us what you've tried - that's great. But what does it do that you don't want it to do, and what doesn't it do that you do want it to do? Commented Mar 21, 2020 at 20:30

2 Answers 2

Reset to default
1

By using echo and command substitution formatting of each line can be done like that:

while IFS= read -r line
do
    echo "$line,$(curl -LI "$line" -o /dev/null -w '%{http_code}\n' -s)" >> output.csv
done < urls.txt
Improve this answer
1

Consider using %{url_effective} in the -w output fields:

awk -v OFS== -v out=/dev/null \
    '!/^(#|[[:blank:]]*$)/{print "url", $0; print "out", out}' urls.txt |

curl -K- -sSLI -w '%{num_redirects},%{url_effective},%{http_code}\n' >> output.csv
  • Because -L is used to follow redirects, there may be some discrepencies between the urls from urls.txt and %{url_effective}. If the first url gives a 3xx response, the first column from the output above will be greater than zero, the second column will contain last url followed (not the initial url from urls.txt)
  • Above, instead of looping over the file and calling curl on each line, awk is used to turn urls.txt into a "config" file for curl which is then piped to curl. The formatting for this is specific to curl – for more info, search for -K in man 1 curl

e.g.

$ cat urls.txt
https://unix.stackexchange.com/
#www.example.com
https://unix.stackexchange.com/question
http://unix.stackexchange.com/

$ awk -v OFS== '!/^(#|[[:blank:]]*$)/{print "url",$0; print "out","/dev/null"}' \
urls.txt | curl -K- -sSLI -w '%{num_redirects},%{url_effective},%{http_code}\n'
0,https://unix.stackexchange.com/,200
0,https://unix.stackexchange.com/question,404
1,https://unix.stackexchange.com/,200
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.