How to make an array loaded from netCDF (.nc) writeable?

Question

I am loading a few variables from a netcdf (.nc) file using python scipy.io.netcdf module. In the file that I am loading them from, missing data have a filler value of 99999, which I want to change to np.nan from numpy.

Once I have imported a variable and assigned it to a numpy array however, I can't clean up the 99999 filler values because I get:

    RuntimeError: array is not writeable

I have attempted to change the attribute of the array writeable to True but that returns:

    ValueError: cannot set WRITEABLE flag to True of this array

So I am at a loss. A simplified version of my code and a couple example outputs are below. Does anyone have any suggestions here? Originally I was using netCDF4 rather than scipy.io.netcdf and in that case I would set nc.set_auto_mask(False) however I haven't seen an analogous property in scipy.io so I have just left that out.

Sample code:

    import scipy.io as sio
    import numpy as np

    nc = sio.netcdf.netcdf_file('filename.nc','r') # open file
    arr = nc.variables['PARAMETER'][:] # load into np array
    output_arr = cleanarr(arr) # replace filler 99999 vals with np.nan
    nc.close()

Worth noting is it doesn't seem to matter if I do nc.close() before or after cleanarr(arr)

Sample outputs:

    type(arr)
        <type 'numpy.ndarray'>

    arr.flags
          C_CONTIGUOUS : True
          F_CONTIGUOUS : False
          OWNDATA : False
          WRITEABLE : False
          ALIGNED : True
          UPDATEIFCOPY : False

hpaulj · Accepted Answer · 2015-06-02 18:17:19Z

2

From the docs

http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.io.netcdf.netcdf_file.html

NetCDF files, when opened read-only, return arrays that refer directly to memory-mapped data on disk:

>>>
>>> data = time[:]
>>> data.base.base
<mmap.mmap object at 0x7fe753763180>
If the data is to be processed after the file is closed, it needs to be copied to main memory:

>>>
>>> data = time[:].copy()
>>> f.close()
>>> data.mean()

I think this has also been addressed in earlier SO questions, though I haven't taken the time to look those up.

answered Jun 2, 2015 at 18:17

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Chris Gordon Over a year ago

This worked perfectly - and I better understand what is happening in your solution than the one I posted below. Thanks.

N1B4 Over a year ago

The netCDF4 module has replaced the scipy.io.netcdf module. I recommend switching over to it, more details can be found here: github.com/Unidata/netcdf4-python

hpaulj Over a year ago

But does netCDF4 handle this 'read-only' issue any differently?

Chris Gordon · Accepted Answer · 2015-06-02 18:16:08Z

1

Solved by using np.require (http://docs.scipy.org/doc/numpy/reference/generated/numpy.require.html) to change the flags rather than the original method I was using. Code now looks like:

    import scipy.io as sio
    import numpy as np

    nc = sio.netcdf.netcdf_file('filename.nc','r') # open file
    arr = nc.variables['PARAMETER'][:] # load into np array
    arr = np.require(arr,dtype='f4',requirements=['O','W'])
    output_arr = cleanarr(arr) # replace filler 99999 vals with np.nan
    nc.close()

This seems to do the job but if anyone can comment on the validity of my answer in terms of whether should or shouldn't be done generally (I feel like I am forcing something here, and I have never used np.require before) I would appreciate it.

answered Jun 2, 2015 at 18:16

Chris Gordon

3014 silver badges14 bronze badges

1 Comment

hpaulj Over a year ago

Does a plain .copy also work? require is a copy with some flags parameters.

Collectives™ on Stack Overflow

How to make an array loaded from netCDF (.nc) writeable?

2 Answers 2

3 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related