Extract data from several netcdf files using ncks in for loop

Question

I am trying to extract certain variables from netcdf files. The following code works if I apply it to a single file:

ncks -C -F -d nj_u,151,152,1 ni_u,234,235,1 -v vel_u 20091208000001.nc testU.nc

See also question: Hyperslab of a 4D netcdf variable using ncks for explanation. Now I want to use this code on several .nc files with following names:

20091208000001.nc
20091208000002.nc
20091208000003.nc

I tried the following loop:

# !bin/bash
for ((x=1;x<=3;x+=1))
do
ncks -C -F -d nj_u,151,152,1 ni_u,234,235,1 -v vel_u 2009120800000$x.nc testU.nc
done

I get the error

ncks: ERROR received 4 filenames; need no more than two

How do I get the loop to only extract from one file at a time and then append the extracted output from all the files into a single output file?

For every loop iteration, each of the files from 20091208000001.nc to ..3.nc is processed. So it basically is one file at a time. What is wrong with this — Inian
– Inian, Commented Feb 7, 2019 at 10:22
I don't know it should work. That's why I posted it. I get this error 4 times in my output. — Jellyse
– Jellyse, Commented Feb 7, 2019 at 10:28

Packard CPW · Accepted Answer · 2019-02-07 15:42:33Z

2

I believe the words ni_u,234,235,1 were mistaken as another filename. You would need another -d before that.

And if you are processing multiple nc files, you might want to rename testU.nc so that they don't overlap, or you could use ncrcat to concatenate into one single file. E.g.

ncrcat -C -F -d nj_u,151,152,1 -d ni_u,234,235,1 -v vel_u 2009120800000?.nc testU.nc

answered Feb 7, 2019 at 15:42

Packard CPW

3391 gold badge5 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Poshi Over a year ago

If this was the reason (which makes sense, BTW), then the claim made in the question that the code works fine out of the loop would be false :-(

Jellyse Over a year ago

@Poshi yes, I know, it worked fine the day before, I did not see the -d was missing. Lesson learned ...

Poshi Over a year ago

That's the reason I was asking you to rerun the command, and me writing the exact command you had to test, for you to just cut and paste and check the statement was true. BTW, it would be nice to correct the question so the answer makes sense.

Poshi · Accepted Answer · 2019-02-07 13:19:18Z

1

I see a couple errors in your script, but nothing that could lead to your actual error.

The shebang line should not contain space and the path should be absolute

There's a comma in the for condition that should be a semicolon

#!/bin/bash

for ((x=1;x<=3;x+=1))
do
    ncks -C -F -d nj_u,151,152,1 ni_u,234,235,1 -v vel_u 2009120800000$x.nc testU.nc
done

When I prepend echo to the command you want to run, I get this result:

ncks -C -F -d nj_u,151,152,1 ni_u,234,235,1 -v vel_u 20091208000001.nc testU.nc
ncks -C -F -d nj_u,151,152,1 ni_u,234,235,1 -v vel_u 20091208000002.nc testU.nc
ncks -C -F -d nj_u,151,152,1 ni_u,234,235,1 -v vel_u 20091208000003.nc testU.nc

Three invocations with a single file each. That code is working. It looks like there's something else. Are you simplifying your code or showing us the full code?

edited Feb 7, 2019 at 13:19

answered Feb 7, 2019 at 10:23

Poshi

5,8733 gold badges22 silver badges37 bronze badges

9 Comments

Jellyse Over a year ago

I corrected the mistakes, still the same error. I use bash namescript.sh to run it.

Poshi Over a year ago

@Jellyse I updated my answer with the result I'm obtaining.

Jellyse Over a year ago

This is litteraly my complete code in the file testing.sh The folder only has those three files and the testing.sh file. Is it possible the nco is not correctly installed?

Jellyse Over a year ago

Ok I reinstalled it, still the same error. The only thing I can think of is that somewhere on my computer there is another file with the same name and that's why it's confused.

Poshi Over a year ago

@Jellyse, prepend echo to your command (echo ncks -C......) and check the output. Having other files with the same name in the computer should not cause this error.

|

Charlie Zender · Accepted Answer · 2019-02-07 20:32:06Z

1

@Packard is right on both counts. Moreover, the stride of 1 is default and thus not needed. Hence

ncrcat -C -F -d nj_u,151,152 -d ni_u,234,235 -v vel_u 2009120800000${x}.nc testU${x}.nc

answered Feb 7, 2019 at 20:32

Charlie Zender

6,45217 silver badges24 bronze badges

Comments

user12920237 · Accepted Answer · 2020-02-18 16:08:42Z

0

I edited the code above according to the dimensions I wanted (lat, lon)

ncrcat -C -F -d nj_u,151,152,1 -d ni_u,234,235,1 -v vel_u 2009120800000?.nc testU.nc

and the feedback was this:

HINT: If operation fails, try multislabbing (http://nco.sf.net/nco.html#msa) wrapped dimension using ncks first, and then apply ncrcat to the resulting file

answered Feb 18, 2020 at 16:08

user12920237

1

Collectives™ on Stack Overflow

Extract data from several netcdf files using ncks in for loop

4 Answers 4

3 Comments

9 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

9 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related