Bash Script - getting input from standard input or a file

Question

I have a bash script that prints columns by name taken from the command line. It works well if I give the script the file as one of the arguments. It does not work well if I pipe input to the script and use /dev/stdin as the file. Does anyone know how I can modify the script to accept standard input from a pipe correctly? Here is my script.

#!/bin/bash

insep=" "
outsep=" "
while [[ ${#} > 0 ]]
do
option="$1"
if [ -f $option ] || [ $option = /dev/stdin ]; 
    then
    break;
fi
case $option in
    -s|--in_separator)
    insep="$2"
    shift # past argument
    shift # past argument
    ;;
    -o|--out_separator)
    outsep="$2"
    shift # past argument
    shift # past argument
    ;;
    *)
         echo "unknown option $option"
         exit 1;
    ;;
esac
done

headers="${@:2}"
grep_headers=$(echo "${headers[@]}" | sed 's/ /|/g')
file=$1

columns=$(awk -F: 'NR==FNR{b[($2)]=tolower($1);next}{print $1,b[$1]}' \
<(head -1 $file | sed "s/$insep/\n/g" | egrep -iwn "$grep_headers" | awk    '{s=tolower($0);print s}') \
<(awk -F: -v header="$headers" 'BEGIN {n=split(tolower(header),a,"  ");for(i=1;i<=n;i++) print a[i]}' $file ) \
| awk '{print "$"$2}' ORS='OFS' | sed "s/OFS\$//")

awk -v insep="$insep" -v outsep="$outsep" "BEGIN{FS=insep;OFS=outsep}{print $columns}" $file

exit;

Sample Input:

col_1 col_2 col_3 col_4 col_5 col_6 col_7 col_8 col_9 col_10
10000 10010 10020 10030 10040 10050 10060 10070 10080 10090
10001 10011 10021 10031 10041 10051 10061 10071 10081 10091
10002 10012 10022 10032 10042 10052 10062 10072 10082 10092
10003 10013 10023 10033 10043 10053 10063 10073 10083 10093
10004 10014 10024 10034 10044 10054 10064 10074 10084 10094
10005 10015 10025 10035 10045 10055 10065 10075 10085 10095
10006 10016 10026 10036 10046 10056 10066 10076 10086 10096
10007 10017 10027 10037 10047 10057 10067 10077 10087 10097
10008 10018 10028 10038 10048 10058 10068 10078 10088 10098

Running with file as an argument (works as expected):

> ./shell_scripts/print_columns.sh file1.txt col_1 col_4 col_6 col_2 | head
col_1 col_4 col_6 col_2
10000 10030 10050 10010
10001 10031 10051 10011
10002 10032 10052 10012
10003 10033 10053 10013

Piping from standard in (does not work as expected):

> head file1.txt | ./shell_scripts/print_columns.sh /dev/stdin col_1 col_4 col_6 col_2 | head
0185 10215  10195
10136 10166 10186 10146
10137 10167 10187 10147
10138 10168 10188 10148
10139 10169 10189 10149

while interesting, do we really need all of this code to answer the Q "how I can modify the script to accept standard input from a pipe?" (I don't think so). Maybe by reducing your problem to just the issue at hand, you'll find your own answer. Also, your construction for columns=$(... seems extremely baroque. You should be able to reduce head,sed,egrep,etc to 1 awk. — shellter
– shellter, Commented Mar 9, 2016 at 20:52
What is the minimal case for reproducing the problem? Will it be the same wothout the while .. done parameter parsing? Do you have problems when you replace some awk's with echoes? Which statement is crucial for reproducing the error? — Walter A
– Walter A, Commented Mar 9, 2016 at 21:12
@WalterA the problem is in the last line. awk -v insep="$insep" -v outsep="$outsep" "BEGIN{FS=insep;OFS=outsep}{print $columns}" $file. For some reason, the firs 130 rows of the standard input are being removed. I only provided a small sample of the input file, but it is about 140 rows. — mmm
– mmm, Commented Mar 9, 2016 at 23:21
Perhaps stackoverflow.com/a/15330784/3220113 helps: you have stdin open and you open stdin again. Confusing! — Walter A
– Walter A, Commented Mar 10, 2016 at 22:33

Cyrus · Accepted Answer · 2016-03-09 21:13:27Z

2

An example:

script.sh:

#!/bin/bash

if [[ -f "$1" ]]; then
  file="$1"
  cat "$file"
  shift
 else
  while read -r file; do echo "$file"; done
fi
echo "${@}"

Test with:

./script.sh file1.txt abc 123 456

and with UUOC:

cat file1.txt | ./script.sh abc 123 456

answered Mar 9, 2016 at 21:13

Cyrus

90.2k15 gold badges112 silver badges173 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Bash Script - getting input from standard input or a file

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related