Merge 70 netCDF files with xarray

Question

I want to merge 70 netCDF files into one. For that I use the .to_netcdf() function of xarray:

    ds = xarray.open_mfdataset('*.nc')
    ds.to_netcdf('SST_2021-10_timeseries.nc')

My problem is that my jupyter notebook always hangs up because the number of files is too high. Is there a more efficient way to merge the files?

Not an answer with netCDF, but another option would be to use a zarr store rather than netCDF (e.g. ds.to_zarr), as zarr has chunking support and parallel writes. — Michael Delgado
– Michael Delgado, Commented Dec 2, 2021 at 17:11

Robert Wilson · Accepted Answer · 2021-12-02 10:36:14Z

1

An alternative would be to use nctoolkit. Commands would be as follows:

import nctoolkit as nc
ds = nc.open_data('*.nc')
ds.merge("time")
ds.to_nc('SST_2021-10_timeseries.nc')

Or you could do it on the command line with CDO:

cdo -mergetime *.nc SST_2021-10_timeseries.nc

Those options should get around any RAM issues.

answered Dec 2, 2021 at 10:36

Robert Wilson

3,44714 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

ScienceNoob Over a year ago

Seems like I can't open all datasets with the asterix in this line: ds = nc.open_data('*.nc') Will get then the message that there is no dataset with the name *.nc

Robert Wilson Over a year ago

Which version of nctoolkit have you installed? Conda has a tendency to install an old version, so the version you have may not accept wild cards

ScienceNoob Over a year ago

nctoolkit-0.3.9, I think that's the latest, right?

Robert Wilson Over a year ago

OK. That should work. But try: ds = nc.open_data(nc.create_ensemble(".", recursive = False)). What OS are you using?

Robert Wilson Over a year ago

though the string "*.nc" should replaced be glob.glob("'.nc') by nctoolkit (line 436: github.com/pmlmodelling/nctoolkit/blob/master/nctoolkit/api.py). So I'm puzzled by what's going on

Collectives™ on Stack Overflow

Merge 70 netCDF files with xarray

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related