Python numpy mask disappears after assigning array to another array

Question

So I am working with geodetical data and use masks to cover areas without data. Here I have just run into the problem of trying to design an array with different time series data of different spots on earth. The actual problem is only that when I assign my seperate spot arrays to the bigger array, the masks disappear without me being able to even reassign the masks.

I create an array and load the data (daily data of a 1° grid, excluding the southernmost 30°) and transpose it so that i can look at different spots easily:

minmax_0_10T = np.ma.zeros((2,360,150,365))
with np.load("Data/Pythonarrays/1DEG_SM_minmax_SWI_001_SM0_10cm_2015.npz") as npz:
    minmax_0_10T[0] = np.ma.MaskedArray(**npz).T
with np.load("Data/Pythonarrays/1DEG_SM_minmax_SWI_005_SM0_10cm_2015.npz") as npz:
    minmax_0_10T[1] = np.ma.MaskedArray(**npz).T

I have 2 datasets I want to plot and 4 different spots i want to look at

spots_1deg = [[41, 210], [29, 119], [108, 196], [117, 256]]
minmax_0_10_spots = np.ma.zeros((2,4,365))
"""Create array of minmax and gldas spotdata for choice of SWI layer"""
for i in range(len(spots_1deg)):
    for layer_gldas in range(len(gldas_smT)):
        gldas_spots[layer_gldas][i] = gldas_smT[layer_gldas][spots_1deg[i][1]][spots_1deg[i][0]]
        
    """now all different gldas layers of minmax fitting"""
    minmax_0_10_spots[0][i] = minmax_0_10T[0][spots_1deg[i][1]][spots_1deg[i][0]]
    minmax_0_10_spots[1][i] = minmax_0_10T[1][spots_1deg[i][1]][spots_1deg[i][0]]

however if I then compare minmax_0_10T[0][spots_1deg[i][1]][spots_1deg[i][0]] with minmax_0_10_spots[0][i] directly, i see that the minmax_0_10_spots array does not contain the mask anymore (as in the mask is False everywhere). How and why is this happening? Assigning the masks each also did not work, in the end I had to define minmax_0_10_spots as list and filling it as such, however also there cannot tranform it into an array, cause that makes the masks disappear.

I have tried to look into the np.ma documentation but I must have either overread something or not understood...

It all boils down to the following problem: If I have my 3 dimensional array the mask is not transferred to the array:

array = np.ma.zeros((2,4,4))
data = np.ma.array(data=[0,0,5.435,3.657], mask =[True,True,False,False],fill_value=1e+20)
array[0][0] = data
data2 = array[0][0]
now data2 does not contain the mask anymore.

However if i remove 1 dimension the mask is transferred:

array = np.ma.zeros((2,4))
data = np.ma.array(data=[0,0,5.435,3.657], mask =[True,True,False,False],fill_value=1e+20)
array[0] = data
data2 = array[0]

hpaulj · Accepted Answer · 2023-01-26 19:21:34Z

In [11]: array = np.ma.zeros((2,4,4))
    ...: data = np.ma.array(data=[0,0,5.435,3.657], mask =[True,True,False,False],fill_value=1e+20)

Your assignment:

In [12]: array[0][0]=data

In [13]: array
Out[13]: 
masked_array(
  data=[[[0.   , 0.   , 5.435, 3.657],
         [0.   , 0.   , 0.   , 0.   ],
         [0.   , 0.   , 0.   , 0.   ],
         [0.   , 0.   , 0.   , 0.   ]],

        [[0.   , 0.   , 0.   , 0.   ],
         [0.   , 0.   , 0.   , 0.   ],
         [0.   , 0.   , 0.   , 0.   ],
         [0.   , 0.   , 0.   , 0.   ]]],
  mask=False,
  fill_value=1e+20)

Better assignment:

In [14]: array[0,0]=data

In [15]: array
Out[15]: 
masked_array(
  data=[[[--, --, 5.435, 3.657],
         [0.0, 0.0, 0.0, 0.0],
         [0.0, 0.0, 0.0, 0.0],
         [0.0, 0.0, 0.0, 0.0]],

        [[0.0, 0.0, 0.0, 0.0],
         [0.0, 0.0, 0.0, 0.0],
         [0.0, 0.0, 0.0, 0.0],
         [0.0, 0.0, 0.0, 0.0]]],
  mask=[[[ True,  True, False, False],
         [False, False, False, False],
         [False, False, False, False],
         [False, False, False, False]],

        [[False, False, False, False],
         [False, False, False, False],
         [False, False, False, False],
         [False, False, False, False]]],
  fill_value=1e+20)

Normally a[0,0]=... and a[0][0]=... do the same thing. But apparently np.ma indexing has a flaw, and the mask doesn't carry through in the two stage indexing.

a[0][0]=... does temp=a[0]; temp[0]=..... That works if temp is a (normal) view. It does not work if temp is a copy (as with advanced indexing). For MaskedArrays, a[0] is not a view (as it would be for a regular ndarray).

In short, it's best to use a[0,0] style of indexing, especially when doing assignment.

hpaulj · Accepted Answer · 2023-01-25 17:04:28Z

0

With the arrays coming from load I can't test your code. It would be easier if you constructed something much small - that can be copy-n-pasted.

Here's a simple test on 1d arrays:

In [8]: m = np.ma.zeros(5)
In [9]: m
Out[9]: 
masked_array(data=[0., 0., 0., 0., 0.],
             mask=False,
       fill_value=1e+20)

In [11]: m1 = np.ma.masked_array(np.arange(5),[1,0,1,0,1])
In [12]: m1
Out[12]: 
masked_array(data=[--, 1, --, 3, --],
             mask=[ True, False,  True, False,  True],
       fill_value=999999)
In [13]: for i in range(5):m[i]=m1[i]
In [14]: m
Out[14]: 
masked_array(data=[--, 1.0, --, 3.0, --],
             mask=[ True, False,  True, False,  True],
       fill_value=1e+20)

answered Jan 25, 2023 at 17:04

hpaulj

233k14 gold badges260 silver badges392 bronze badges

1 Comment

GemGre Over a year ago

Ah of course my mistake i will put a test you can do in my question!

Collectives™ on Stack Overflow

Python numpy mask disappears after assigning array to another array

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related