bash script for formatting piped output

Question

Okay, you asked for it - here is my whole code that I am trying to work with.

The help I need with is this: postprocauto.sh: a script that presents the data in a nice readable format.

I am supposed to take a cat of the autocsv file (sample below - full file is 160k) and then pipe it through various other scripts so in the end, you will have this command line:

cat autocsv | ./ prepprocauto.sh | ./BMW.sh | ./6cyl.sh | ./hwyfe.sh | ./postprocauto.sh

My problem is that I can get everything to run except the postprocauto.ch. I need to be able to do this:

Requirements for this script:
It must print header information after 20 lines of data. 

The breaks must have a least one empty line between them 
before printing another header. 

The last line of the output should print out the number of 
records that were processed.

An example of the output:

Year Eng. Disp. Cyl. City FE Hwy FE Model
2013 1.5 4     39 38       ILX
2013 2    4    24 35      ILX
2013 2.4 4     22 31      ILX
2013 2.4 4      22 31      TSX
2013 2.4 4      21 29      TSX
2013 3.5 6      19 28      TSX
2013 1.3 4      41 44      INSIGHT
2013 1.3 4      41 44      INSIGHT
2013 3.5 6      20 29     TL 2WD
2013 3.7 6      18 26     TL 4WD
2013 3.7    6      17 25     TL 4WD
There were 11 records processed.

the different scripts are below - if you want the full size autocsv, tell me where to put it - 520k file size

prepprocauto.sh

#!/bin/bash

while read x
do
    echo $x | awk -F',' ' { print $1":"$2":"$4":"$7":"$8":"$10":"$11":"$12":"$22":"$24}'

done

BMW.sh

#!/bin/bash

selection='BMW'

if [ ! $# -lt 1 ]; then
    selection=$1
fi


while read y; do

    model=$(echo $y | awk -F':'  '{print $2 }')

    if [ "$model" == "$selection" ]; then
            echo $y
    fi
done

6cyl.sh

#!/bin/bash

selection='6'

if [ ! $# -lt 1 ]; then
    selection=$1
fi


while read y; do

    model=$(echo $y | awk -F':'  '{print $5 }')

    if [ "$model" == "$selection" ]; then
            echo $y
    fi
done

hwyfe.sh

#!/bin/bash

selection='31'

if [ ! $# -lt 1 ]; then
    selection=$1
fi


while read y; do

    hwy=$(echo $y | awk -F':'  '{print $7 }')

    if [ "$hwy" -gt "$selection" ]; then
            echo $y
    fi
done

postprocauto.sh

output=$(awk -F ':' '{print $1 "\t" $4 "\t"$5"\t" $6 "\t" $7 "\t" $3}')
echo "Year  Eng. Disp Cyl  City FE  Hwy FE   Model"
echo "$output"

Contents of the autocsv file (cut down a lot)

2013,Audi,Audi,TT Roadster quattro,ADX,67,2,4,Auto(AM-S6),22,31,26,28.4068,42.25
79,33.3217,22.407,31.1674,25.6515,,TC,Turbocharged,AMS,Automated Manual- Selecta
ble (e.g. Automated Manual with paddles),6,Y,N,A,10,GP,Gasoline (Premium Unleade
d Recommended),MPG,N,,,,,,,2200,2200,2,2,1,Two Seaters,car,Vehicle Specific 5-cy
cle label,6/18/12,12113,,N,N,,,N,N,ENGINE CODE CDMA ONLY.,N,,Y,CONTINUOUS VARIAB
LE VALVE TIMING,N,,,,,,,,,,,,,,,,,,,,,,,,,,,GDI,Spark Ignition Direct Injection,
N,N,5W40,33.3,7,7,,DADXV02.03UA,5,600,
2013,BMW,BMW,Z4 sDrive28i,BMX,428,2,4,Auto(A8),22,33,26,27.9499,46.8923,34.1594,
21.9803,33.2305,25.9308,,TC,Turbocharged,A,Automatic,8,Y,N,R,10,GP,Gasoline (Pre
mium Unleaded Recommended),MPG,N,,,,,,,2200,2200,2,2,1,Two Seaters,car,Derived 5
-cycle label,7/24/12,11033,,N,N,,,N,N,,N,,Y,variable valve timing at inlet and o
utlet valves,Y,variable valve lift at inlet valves,,,,,,,,,,,,,,,,,,,,,,,,,,GDI,
Spark Ignition Direct Injection,,Y,0W30,34.3,7,7,,DBMXJ02.0N20,5,600,
2013,BMW,BMW,Z4 sDrive28i,BMX,429,2,4,Manual(M6),22,34,26,28.3664,48.0364,34.774
1,22.2841,34.0033,26.3746,,TC,Turbocharged,M,Manual,6,N,N,R,10,GP,Gasoline (Prem
ium Unleaded Recommended),MPG,N,,,,,,,2200,2200,2,2,1,Two Seaters,car,Derived 5-
cycle label,7/25/12,11092,,N,N,,,N,N,,N,,Y,variable valve timing at inlet and ou
tlet valves,Y,variable valve lift at inlet valves,,,,,,,,,,,,,,,,,,,,,,,,,,GDI,S
park Ignition Direct Injection,,Y,0W30,34.9,7,7,,DBMXJ02.0N20,5,600,
2013,BMW,BMW,Z4 sDrive35i,BMX,436,3,6,Auto(AM-S7),17,24,19,21.1097,32.8224,25.14
8,16.8973,23.5833,19.3682,,TC,Turbocharged,AMS,Automated Manual- Selectable (e.g
. Automated Manual with paddles),7,N,N,R,10,GP,Gasoline (Premium Unleaded Recomm
ended),MPG,N,,,,,,,3000,3000,2,2,1,Two Seaters,car,Derived 5-cycle label,8/7/12,
12492,,N,N,,,N,N,,N,,Y,variable valve timing at inlet and outlet valves,N,,,,,,,
,,,,,,,,,,,,,,,,,,,,GDI,Spark Ignition Direct Injection,,Y,0W30,25.3,4,4,,DBMXV0
3.054R,5,,3400
2013,BMW,BMW,Z4 sDrive35i,BMX,435,3,6,Manual(M6),19,26,21,23.3,36.6,27.855,18.54
45,26.1997,21.352,,TC,Turbocharged,M,Manual,6,N,N,R,10,GP,Gasoline (Premium Unle
aded Recommended),MPG,N,,,,,,,2700,2700,2,2,1,Two Seaters,car,Derived 5-cycle la
bel,8/7/12,11743,,N,N,,,N,N,,N,,Y,variable valve timing at inlet and outlet valv
es,N,,,,,,,,,,,,,,,,,,,,,,,,,,,GDI,Spark Ignition Direct Injection,,Y,0W30,28.1,
5,5,,DBMXV03.054R,5,,1900

everything you describe in your specification can be handled in 1 awk program. But first try echo "$output". An unquote variable passed to echo will reduce all whitespace to a single space between each words. To get your 20 line header, just use awk's NR (Record Number) variable like if (NR % 20) {print "header}. Good luck. — shellter
– shellter, Commented Mar 2, 2014 at 3:25

iruvar · Accepted Answer · 2014-03-02 02:41:28Z

1

Just quote the $output

echo "$output"

Additionally, you may want to get rid of redundant use of cat and do

awk -F ':' '{print $1 "\t" $4 "\t"$5"\t" $6 "\t" $7 "\t" $3}' ./temp.log

instead

answered Mar 2, 2014 at 2:41

iruvar

23.5k7 gold badges58 silver badges83 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

jaypal singh Over a year ago

... and set the OFS to make awk cleaner.

Amit · Accepted Answer · 2014-03-02 02:41:29Z

1

Try using column -t

echo $output | column -t

answered Mar 2, 2014 at 2:41

Amit

20.6k7 gold badges51 silver badges55 bronze badges

Comments

Ярослав Рахматуллин · Accepted Answer · 2014-03-02 02:50:18Z

1

You can use the modulo function to display a header every nth line like so:

$ seq 1 12 > moo 
$ awk < moo 'NR%4==0 { print "\nheading\n" } {print} 
             END{ print "total lines: " NR}'
1
2
3

heading

4
5
6
7

heading

8
9
10
11

heading

12
total lines: 12

answered Mar 2, 2014 at 2:50

Ярослав Рахматуллин

7,9681 gold badge44 silver badges65 bronze badges

Collectives™ on Stack Overflow

bash script for formatting piped output

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related