6

I have data points that represent a coordinates for a 2D array (matrix). The points are regularly gridded, except that data points are missing from some grid positions.

For example, consider some XYZ data that fits on a regular 0.1 grid with shape (3, 4). There are gaps and missing points, so there are 5 points, and not 12:

import numpy as np
X = np.array([0.4, 0.5, 0.4, 0.4, 0.7])
Y = np.array([1.0, 1.0, 1.1, 1.2, 1.2])
Z = np.array([3.3, 2.5, 3.6, 3.8, 1.8])
# Evaluate the regular grid dimension values
Xr = np.linspace(X.min(), X.max(), np.round((X.max() - X.min()) / np.diff(np.unique(X)).min()) + 1)
Yr = np.linspace(Y.min(), Y.max(), np.round((Y.max() - Y.min()) / np.diff(np.unique(Y)).min()) + 1)
print('Xr={0}; Yr={1}'.format(Xr, Yr))
# Xr=[ 0.4  0.5  0.6  0.7]; Yr=[ 1.   1.1  1.2]

What I would like to see is shown in this image (backgrounds: black=base-0 index; grey=coordinate value; colour=matrix value; white=missing).

matrix

Here's what I have, which is intuitive with a for loop:

ar = np.ma.array(np.zeros((len(Yr), len(Xr)), dtype=Z.dtype), mask=True)
for x, y, z in zip(X, Y, Z):
    j = (np.abs(Xr -  x)).argmin()
    i = (np.abs(Yr -  y)).argmin()
    ar[i, j] = z
print(ar)
# [[3.3 2.5 -- --]
#  [3.6 -- -- --]
#  [3.8 -- -- 1.8]]    

Is there a more NumPythonic way of vectorising the approach to return a 2D array ar? Or is the for loop necessary?

4 Answers 4

8

You can do it on one line with np.histogram2d

data = np.histogram2d(Y, X, bins=[len(Yr),len(Xr)], weights=Z)
print(data[0])
[[ 3.3  2.5  0.   0. ]
 [ 3.6  0.   0.   0. ]
 [ 3.8  0.   0.   1.8]]
Sign up to request clarification or add additional context in comments.

Comments

2

You can use X and Y to create the X-Y coordinates on a 0.1 spaced grid extending from the min to max of X and min to max of Y and then inserting Z's into those specific positions. This would avoid using linspace to get Xr and Yr and as such must be quite efficient. Here's the implementation -

def indexing_based(X,Y,Z):
    # Convert X's and Y's to indices on a 0.1 spaced grid
    X_int = np.round((X*10)).astype(int)
    Y_int = np.round((Y*10)).astype(int)
    X_idx = X_int - X_int.min()
    Y_idx = Y_int - Y_int.min()

    # Setup output array and index it with X_idx & Y_idx to set those as Z
    out = np.zeros((Y_idx.max()+1,X_idx.max()+1))
    out[Y_idx,X_idx] = Z

    return out

Runtime tests -

This section compare the indexing-based approach against the other np.histogram2d based solution for performance -

In [132]: # Create unique couples X-Y (as needed to work with histogram2d)
     ...: data = np.random.randint(0,1000,(5000,2))
     ...: data1 = data[np.lexsort(data.T),:]
     ...: mask = ~np.all(np.diff(data1,axis=0)==0,axis=1)
     ...: data2 = data1[np.append([True],mask)]
     ...: 
     ...: X = (data2[:,0]).astype(float)/10
     ...: Y = (data2[:,1]).astype(float)/10
     ...: Z = np.random.randint(0,1000,(X.size))
     ...: 

In [133]: def histogram_based(X,Y,Z): # From other np.histogram2d based solution
     ...:   Xr = np.linspace(X.min(), X.max(), np.round((X.max() - X.min()) / np.diff(np.unique(X)).min()) + 1)
     ...:   Yr = np.linspace(Y.min(), Y.max(), np.round((Y.max() - Y.min()) / np.diff(np.unique(Y)).min()) + 1)
     ...:   data = np.histogram2d(Y, X, bins=[len(Yr),len(Xr)], weights=Z)
     ...:   return data[0]
     ...: 

In [134]: %timeit histogram_based(X,Y,Z)
10 loops, best of 3: 22.8 ms per loop

In [135]: %timeit indexing_based(X,Y,Z)
100 loops, best of 3: 2.11 ms per loop

Comments

1

You could use a scipy coo_matrix. It allows you to construct a sparse matrix from coordinates and data. See examples on the attached link.

http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.sparse.coo_matrix.html

Hope that helps.

Comments

1

The sparse matrix is the first solution that came to mind, but since X and Y are floats, it's a little messy:

In [624]: I=((X-.4)*10).round().astype(int)
In [625]: J=((Y-1)*10).round().astype(int)
In [626]: I,J
Out[626]: (array([0, 1, 0, 0, 3]), array([0, 0, 1, 2, 2]))

In [627]: sparse.coo_matrix((Z,(J,I))).A
Out[627]: 
array([[ 3.3,  2.5,  0. ,  0. ],
       [ 3.6,  0. ,  0. ,  0. ],
       [ 3.8,  0. ,  0. ,  1.8]])

It still needs, in one way or other, to match those coordinates with [0,1,2...] indexes. My quick cheat was to just scale the values linearly. Even so I had to take care when converting floats to ints.

sparse.coo_matrix works because a natural way of defining a sparse matrix is with (i, j, data) tuples, which of course can be translated to I, J, Data lists or arrays.

I rather like the historgram solution, even though I haven't had occasion to use it.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.