How to output first line of curl result in a file with bash script

Question

I want to read a file of urls, curl each url and only get the first line which contains the HTTP code. I am running under Windows 10 inside Cmder.

#!/bin/bash
input="urls.csv"
truncate -s 0 dest.csv
while IFS= read -r var
do
    result= `curl -I ${var%$'\r'} | grep HTTP $result`
echo "$var $result" >> dest.csv
done < "$input"

however the output file is empty, thank you

@thatotherguy no i don't see error, I am running inside Cmder under Windows 10 — Yvon Huynh
– Yvon Huynh, Commented Mar 22, 2019 at 21:27

oguz ismail · Accepted Answer · 2019-03-23 00:42:13Z

4

Assuming urls.csv is just a simple list of URLs and you're working on a linux system (or any system which has /dev/null), following command will send HEAD requests to each URL and output them next to HTTP response codes.

sed 's/^/url = /; s/\r\?$/\n-o \/dev\/null/' urls.csv |
curl -s -K- -w '%{http_code} %{url_effective}\n' -I >outfile

see curl man page for further information.

edited Mar 23, 2019 at 0:42

answered Mar 22, 2019 at 21:18

oguz ismail

51.8k16 gold badges64 silver badges83 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Yvon Huynh Over a year ago

Hi thank you, I didn't manage to make it run in Cmder Windows 10, but it ran ok on Linux, now can you tell me how to output the HTTP code + the url in a file? thanks

Bill Naylor · Accepted Answer · 2019-03-23 12:52:05Z

1

In reply to your query on outputting the HTTP code, this is given in the second line of the header. You may get this by:

export IFS=$; # This is required to stop everything coming out on one line!
curl -i <domain> | head -n 2

answered Mar 23, 2019 at 12:52

Bill Naylor

5021 gold badge10 silver badges21 bronze badges

Collectives™ on Stack Overflow

How to output first line of curl result in a file with bash script

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related