Read multiple files using bash script/shell script

Question

I am new in shell/bash scripting. I want to extract data from multiple netcdf files using the bash or shell script. Each file contains a time series of temperature values. For example:

FileA.nc contains 20 20 21 22 23 24
FileB.nc contains 23 24 25 26 27 24
FileC.nc contains 21 20 19 18 22 23

I want to extract the values per file and merge the results of the three files. The output should look like this saved as a csv file.

A 20 20 21 22 23 24
B 23 24 25 26 27 24
C 21 20 19 18 22 23

What is the easiest way to do this? Many thanks in advance for the help.

My awk script gives the expected output but it doesn't look like the valid csv file. — sjsam
– sjsam, Commented Jul 29, 2016 at 7:27
Expected output differs from the contents of the fileC.nc. — James Brown
– James Brown, Commented Jul 29, 2016 at 7:49

Chem-man17 · Accepted Answer · 2016-10-26 07:50:45Z

3

You can proceed as-

for i in {A..C}
do 
echo -n "$i " >> master_file
cat File"$i".nc >> master_file
done

Now you'll have a huge file with appended individual files. Now to make spaces into commas (if you want a .csv format)

sed -i 's/ /,/g' master_file

edited Oct 26, 2016 at 7:50

answered Jul 29, 2016 at 6:02

Chem-man17

1,7901 gold badge14 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Chem-man17 Over a year ago

Your question suggested that the files only had the temperature data. You need to be clearer about what your individual input files contain. Please edit your question.

Lyndz Over a year ago

Many Thanks for the help, I deleted my previous comment here. I was processing the wrong file a while ago.This is already working for my problem.Many thanks again

Chem-man17 Over a year ago

If it's a valid solution then please 'accept' it as an answer.

sjsam Over a year ago

The result should also contain a part of the file-name ie A in FileA.nc and so. This solution doesn't implement it.

Chem-man17 Over a year ago

@sjsam Yup, fair enough. Edited it to make it exactly what OP wanted.

|

sjsam · Accepted Answer · 2016-07-29 07:16:04Z

2

awk is your friend :

$ arry=( file{A..C}.nc ) # store all the filenames in an array
$ # Then feed all the files to awk like below
$ awk '{printf "%s %s\n",gensub(/file(.)\.nc/,"\\1","1",FILENAME),$0}' "${arry[@]}" >newfile
$ cat newfile 
A 20 20 21 22 23 24
B 23 24 25 26 27 24
C 21 20 19 18 22 23

Note

This requires you have [ gnu awk ] which I suppose you already have.

edited Jul 29, 2016 at 7:16

answered Jul 29, 2016 at 5:50

sjsam

22k6 gold badges63 silver badges114 bronze badges

Comments

James Brown · Accepted Answer · 2016-07-29 07:50:26Z

1

In AWK:

$ awk '{gsub(/^file|\.nc$/,"",FILENAME); print FILENAME,$0}' file*.nc
A 20 20 21 22 23 24
B 23 24 25 26 27 24
C 21 20 19 18 22 23

answered Jul 29, 2016 at 7:50

James Brown

37.7k8 gold badges52 silver badges64 bronze badges

Collectives™ on Stack Overflow

Read multiple files using bash script/shell script

3 Answers 3

6 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

6 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related