Assign new data to an existing xarray.DataArray

Question

I have 20 netCDF files from WRF model (different runs) and I'm want to get slp variable from all these files and put into only one xarray.DataArray.

After reading the first file, I used expand_dims to add a new dimension. So, after that I tried to assign values obtained from the other files. However, I got the following error:

ValueError: assignment destination is read-only

Below is the code:

arq = nc.Dataset( dataDir+lista[0] )
dados = wrf.getvar( arq, 'slp', wrf.ALL_TIMES )
arq.close()

pnmm = dados.expand_dims( dim={ 'membro':Narqs } )

for membro in np.arange(1,20):
    arq = nc.Dataset( dataDir+lista[ membro ] )
    dummy = wrf.getvar( arq, 'slp', wrf.ALL_TIMES )
    pnmm[ membro, :, :, :] = dummy
    del dummy
    arq.close()

here, Narqs is an integer variable with the number of netCDF files got from list lista which was obtained with os.listdir().

After expand_dims I'm getting the desirable array shape with an extra dimension for all data from all files. However, the line

pnmm[ membro, :, :, :] = dummy

is producing the error. I tried to assign just the values of dummy (with dummy.values), but didn't work because pnmm is read-only.

So,

1) Why pnmm DataArray is read-only? Is it truly read-only or I'm accessing its elements in a wrong way?

2) Is there a way to easily assign values to pnmm DataArray?

I've tried using xarray.concat and it works, but I really would like to understand why the above approach didn't work, because it seems much more intuitive. Also, with concat I have to create and delete more dummy variables to make it works.

UPDATE:

As I have written above, using xarray.concat solves my problem and I show the code below:

for membro in np.arange(0,20):
    if membro == 0:
        print( 'acessando arquivo 0' )
        arq = nc.Dataset( dataDir+lista[ membro ] )
        pnmm = wrf.getvar( arq, 'slp', wrf.ALL_TIMES )
        arq.close()
    else:
        print( 'acessando arquivo '+str( membro ) )
        arq = nc.Dataset( dataDir+lista[ membro ] )
        d2 = wrf.getvar( arq, 'slp', wrf.ALL_TIMES )
        arq.close()

        d3 = xr.concat( [pnmm,d2], dim='membro' )

        del pnmm, d2
        pnmm = d3.copy()
        del d3

However, even with this problem solved, I really would like to know how to do this in a simpler way, like using pnmm[ membro, :, :, :] = dummy. I also have tried other ways, like:

dados = wrf.getvar( arq, 'slp', wrf.ALL_TIMES )                                                                                                                                                                        
pnmm = dados.copy(  deep=True, data=None )                                                                                                                                                                                        
pnmm = dados.expand_dims( dim={ 'membro':Narqs } )

However, the read-only problem still rises.

In other words: I just want to get the structure of a DataArray from a netCDF file, add a new dimension to this structure and assign values to this new object.

Thank you.

Mateus

Ryan · Accepted Answer · 2019-08-04 17:44:08Z

2

1) Why pnmm DataArray is read-only? Is it truly read-only or I'm accessing its elements in a wrong way?

It's ready only at this stage because it points directly to the netCDF file on disk.

2) Is there a way to easily assign values to pnmm DataArray?

My recommendation would be to use a more xarray-centric approach, avoiding the low-level netCDF4 library. You may be able to do exactly what you want in one line with xarray.open_mfdataset. Something like this:

ds = xr.open_mfdataset('/path/to/files/*.nc', concat_dim='member')

If this fails, you may need to add a preprocessing function, or open the files individually and then manually call xarray.concat on the resulting datasets.

answered Aug 4, 2019 at 17:44

Ryan

8067 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Mateus da Silva Teixeira Over a year ago

Hi, I already tried to read all files, but using <wrf-python.readthedocs.io/en/latest/…>. The big problem is that when getting slp from all files with wrf.getvar the memory is completely used and the PC slows down. With rainfall (RAINNC and RAINC) it works very well. Now, I've tried to make a deep copy from dados to pnmm and after using expand_dims on _pnmm. However, I have same error: "ValueError: assignment destination is read-only".

Ryan Over a year ago

Can you bypass wrf and just work directly with the netcdf files? Xarray should use lazy loading by default, and you shouldn't have to worry about these memory / copy / deep copy questions. I'm not sure what wrf-python is doing, but it sounds like it is not optimal. What value does wrf-python add over just using xarray directly?

Mateus da Silva Teixeira Over a year ago

I think I can't bypass wrf-python. Wrf-python provides methods to extract variables from WRF model output, including variables derived from its output - post-processed variables. For example, your approach for open files with open_mfdataset doesn't work: "ValueError: arguments without labels along dimension 'member' cannot be aligned because they have different dimension sizes: {2, 3, 5}".

Ryan Over a year ago

Then it seems like a bug in wrf-python. Wrf-python needs to be refactored to take advantage of xarray's lazy loading. I recommend you open an issue at github.com/NCAR/wrf-python.

Mateus da Silva Teixeira Over a year ago

Yes, I believe so! I would do that. Thank you.

Collectives™ on Stack Overflow

Assign new data to an existing xarray.DataArray

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related