bash script, modifying a file

Question

Here's my script:

#!/bin/bash


for i in *.csv
do
        echo "i: $i"
        THE_FILE2="$i-2.csv"

        file_read()
        {
                lineno=0
                while read line
                do
                        echo $line | awk -F, '{print $1","$2",,,"$3","$4}'
                        ((lineno++))
                done
        } < $i > $THE_FILE2
        echo "the_file2: $THE_FILE2"
        echo "end"
done

file_read

Output:

i: 2992.csv
the_file2: 2992.csv-2.csv
end
i: 5415.csv
the_file2: 5415.csv-2.csv
end
i: csa.csv
the_file2: csa.csv-2.csv
end
i: loc.csv
the_file2: loc.csv-2.csv
end
i: visa.csv
the_file2: visa.csv-2.csv
end

$ ls
2992.csv       csa.csv        transform.sh   visa.csv-2.csv
5415.csv       loc.csv        visa.csv

Unfortunately, it only does it for the last file in the list. It's skipping over all the others. I have a feeling this has something to do with buffering or globbing or something. What am I missing?

Why are you reading the file line by line in your script? Awk does it by default for you. awk -F, '{print $1","$2",,,"$3","$4}' $i > $THE_FILE2 does what you want probably (untested) — Bernhard
– Bernhard, Commented Oct 2, 2012 at 6:52

John3136 · Accepted Answer · 2012-10-02 06:39:25Z

3

You have defined file_read inside the loop then called it from outside.

It should be the other way around!

You'll may want to pass some of the values through to file_read as parameters.

answered Oct 2, 2012 at 6:39

John3136

29.3k4 gold badges55 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Bernhard Over a year ago

I would get rid of the function altogether in this case.

John3136 Over a year ago

In the example given above, that is probably a good solution - it's small enough that it can remain inline without confusing the code too much.

Barmar · Accepted Answer · 2012-10-02 07:00:09Z

1

There's no need for the file_read function. Just put that code directly in the for-loop.

Also, why do you use the shell's read command and then echo that to a new awk command for each line? You can just redirect the entire file to awk, it reads each line automatically.

And you're incrementing lineno, but not using it for anything. If you need the line number for something you haven't shown, you can use awk's NR variable.

for i in *.csv
do
        echo "i: $i"
        THE_FILE2="$i-2.csv"
        awk -F, '{print $1","$2",,,"$3","$4}' < $i > $THE_FILE2
        echo "the_file2: $THE_FILE2"
        echo "end"
done

answered Oct 2, 2012 at 7:00

Barmar

789k57 gold badges555 silver badges669 bronze badges

Comments

cdarke · Accepted Answer · 2012-10-02 09:45:19Z

1

Unfortunately, it only does it for the last file in the list.

That's because you are only calling file_read once, after the loop. It only executes it for the last file because the variables i and THE_FILE2 remain from the last iteration of the loop.

Here is an awk solution:

awk -f - *.csv << 'EOD'
BEGIN { OFS = FS = "," }
{
    out = FILENAME "-2.csv"
    print $1,$2,",",$3,$4 >> out
}
EOD

edited Oct 2, 2012 at 9:45

answered Oct 2, 2012 at 9:19

cdarke

44.8k8 gold badges85 silver badges86 bronze badges

Collectives™ on Stack Overflow

bash script, modifying a file

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related