Faster method for iterating through a numpy array of numpy arrays

Question

I have a numpy array of numpy arrays like the following example:

data = [[0.4, 1.5, 2.6],
        [3.4, 0.2, 0.0],
        [null, 3.2, 1.0],
        [1.0, 4.6, null]]

I would like an efficient way of returning the row index, column index and value if the value meets a condition.

I need the row and column values because I feed them into func_which_returns_lat_long_based_on_row_and_column(column, row) which is applied if the value meets a condition.

Finally I would like to append the value, and outputs of the function to my_list.

I have solved my problem with the nested for loop solution shown below but it is slow. I believe I should be using np.where() however I cannot figure that out.

my_list = []
for ii, array in enumerate(data):
    for jj, value in enumerate(array):
        if value > 1:
            lon , lat = func_which_returns_lat_long_based_on_row_and_column(jj,ii)
            my_list.append([value, lon, lat])

I'm hoping there is a more efficient solution than the one I'm using above.

ComplicatedPhenomenon · Accepted Answer · 2019-08-14 14:04:15Z

2

import numpy as np
import warnings
warnings.filterwarnings('ignore')
data = [[0.4, 1.5, 2.6],
        [3.4, 0.2, 0.0],
        [np.nan, 3.2, 1.0],
        [1.0, 4.6, np.nan]]
x = np.array(data) 
i, j = np.where(x > 1 )
for a, b in zip(i, j):
    print('lon: {} lat: {} value: {}'.format(a, b, x[a,b]))

Output is

lon: 0 lat: 1 value: 1.5
lon: 0 lat: 2 value: 2.6
lon: 1 lat: 0 value: 3.4
lon: 2 lat: 1 value: 3.2
lon: 3 lat: 1 value: 4.6

As there is np.nan in comparison, there will be RuntimeWarning.

edited Aug 14, 2019 at 14:04

answered Aug 14, 2019 at 13:54

ComplicatedPhenomenon

4,2113 gold badges26 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Alex Ferguson · Accepted Answer · 2019-08-14 12:47:43Z

0

you can use

result = np.where(arr == 15)

it will return a np array of indices where element is in arr

answered Aug 14, 2019 at 12:47

Alex Ferguson

1072 silver badges10 bronze badges

Comments

some_name.py · Accepted Answer · 2019-08-14 13:27:07Z

try to build a function that works on arrays. For instance a function that adds to every element of the data the corresonding column and row index could look like:

import numpy as np

def func_which_returns_lat_long_based_on_row_and_column(data,indices):
    # returns element of data + columna and row index
    return data + indices[:,:,0] + indices[:,:,1]

data = np.array([[0.4, 1.5, 2.6],
                 [3.4, 0.2, 0.0],
                 [np.NaN, 3.2, 1.0],
                 [1.0, 4.6, np.NaN]])

# create a matrix of the same shape as data (plus an additional dim because they are two indices)
# with the corresponding indices of the element in it
x_range = np.arange(0,data.shape[0])
y_range = np.arange(0,data.shape[1])
grid = np.meshgrid(x_range,y_range, indexing = 'ij')
indice_matrix = np.concatenate((grid[0][:,:,None],grid[1][:,:,None]),axis=2)

# for instance:
# indice_matrix[0,0] = np.array([0,0])
# indice_matrix[1,0] = np.array([1,0])
# indice_matrix[1,3] = np.array([1,3])

# calculate the output 
out = func_which_returns_lat_long_based_on_row_and_column(data,indice_matrix)

data.shape
>> (4,3)

indice_matrix.shape
>> (4, 3, 2)

indice_matrix 
>>> array([[[0, 0],
        [0, 1],
        [0, 2]],

       [[1, 0],
        [1, 1],
        [1, 2]],

       [[2, 0],
        [2, 1],
        [2, 2]],

       [[3, 0],
        [3, 1],
        [3, 2]]])

indice_matrix[2,1]
>> array([2, 1])

Collectives™ on Stack Overflow

Faster method for iterating through a numpy array of numpy arrays

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related