getting data from multiple csv files in linux

Question

I'm attempting to select certain data from numerous csv files using linux, making a new file containing just the data I require for processing.

I have started by using grep and awk commands to select the rows and columns I require from a file.

I then wish to add a column which gives the filename as an entry for all rows.

Finally I want to repeat this for a number of files in a folder and append the results into one file. Is there a way of doing this?

Thank you in advance for any help.

I'd just do it in a quick Perl script, personally. Loop in a file, spiit each line, and concat each column you want, plus the filename of the current file at the end of the line. Then loop over each file in the specified directory. — Mike
– Mike, Commented Feb 18, 2015 at 15:23
If you've already got the awk parts working (if you'd rather use another scripting language that's fine) for one file, you can use the shell for f in *.csv; do ... done or find ... | while read f; do ... done techniques to do many files; redirect the script output to get your one file out. — mpez0
– mpez0, Commented Feb 18, 2015 at 15:26
could you give me a quick example script so I can understand what you mean? thanks — energyengineer2000
– energyengineer2000, Commented Feb 18, 2015 at 15:52

John M. · Accepted Answer · 2015-02-24 16:46:12Z

1

Perhaps this will help?

grep gives you the filename of each file it finds a pattern.

You can pipe the output to sed and translate the ':' after each filename to a ',' as follows:

    grep FS *.awk  | sed 's/:/,/'

    empty.awk,BEGIN { FS = "\t" }
    join.awk,    OFS = sep = "\t"
    merge.awk,awk ' BEGIN { FS = "\t" }
    p11.awk,BEGIN { FS = "\t"   # make tab the field separator
    p42.awk,BEGIN                   { FS = OFS = "\t" }
    p43.awk,BEGIN   { FS = OFS = "\t" }
    p48.awk,BEGIN { FS = "\t" }
    p52.awk,BEGIN   { OFS = ":"; ORS = "\n\n" }
    p54.awk,BEGIN { FS = "\t" }
    smith.awk,BEGIN         { RS = ""; FS = "\n" }

answered Feb 24, 2015 at 16:46

John M.

313 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

getting data from multiple csv files in linux

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related