Append Value to 3D array numpy

Question

I iterate over a 3D numpy array and want to append in every step a float value to the array in the 3rd dimension (axis =2). Something like (I know the code doesn't work as of now, latIndex, data and lonIndex for simplicity as randoms)

import numpy as np
import random

GridData = np.ones((121, 201, 1000)) 
data = np.random.rand(4800, 4800)
for row  in range(4800):
    for column in range(4800):
        latIndex = random.randrange(0, 121, 1)
        lonIndex = random.randrange(0, 201, 1)
        GridData = np.append(GridData[latIndex, lonIndex, :], data[column, row], axis = 2)

The 3rd dimension of GridData is arbitrary in this example of size 1000. How can I achieve this?

Addition: It might be possible without np.append but then I don't know how to do this since the 3rd index is different for every combination of latIndex and lonIndex.

Appending is slow. Do you know the end shape of GridData in advance? — user228395
– user228395, Commented Nov 18, 2019 at 10:05
@user228395 I know the first two dimensions (121 and 201) and the third is approximately 950 - 1000 (varies probably). — clearseplex
– clearseplex, Commented Nov 18, 2019 at 10:09
@user228395 I cannot compute the exact size in advance. I could just make it big enough, so that I probably don't exceed the indices. — clearseplex
– clearseplex, Commented Nov 18, 2019 at 10:13
Are you trying to append data values to GridData, so that you end up with an array of shape (121, 201, 1002) (in this example)? Or do you want to append GridData to data to end with an array of shape (4800, 4800, 1000)? — nheise
– nheise, Commented Nov 18, 2019 at 10:14

nheise · Accepted Answer · 2019-11-18 12:00:30Z

2

You can allocate extra space for your array grid_data, fill it with the NaN, and keep track of the next index to be filled in another array while iterating through and filling with values from data. If you completely fill the third dimension for some lat_idx, lon_idx with non-NaN values, then you just allocate more space. Since appending is expensive with numpy, it's best that this extra space is pretty large so you only do it once or twice (below I allocate twice the original space).

Once the array is filled, you can remove the added space that was unused with numpy.isnan(). This solution does what you want but is very slow (for the example values you gave it took about two minutes), but the slow execution comes from iterating rather than the numpy operations.

Here's the code:

import random
import numpy as np

grid_data = np.ones(shape=(121, 201, 1000))
data = np.random.rand(4800, 4800)

# keep track of next index to fill for all the arrays in axis 2
next_to_fill = np.full(shape=(grid_data.shape[0], grid_data.shape[1]),
                       fill_value=grid_data.shape[2],
                       dtype=np.int32)

# allocate more space
double_shape = (grid_data.shape[0], grid_data.shape[1], grid_data.shape[2] * 2)
extra_space = np.full(shape=double_shape, fill_value=np.nan)
grid_data = np.append(grid_data, extra_space, axis=2)

for row in range(4800):
    for col in range(4800):
        lat_idx = random.randint(0, 120)
        lon_idx = random.randint(0, 200)

        # allocate more space if needed
        if next_to_fill[lat_idx, lon_idx] >= grid_data.shape[2]:
            grid_data = np.append(grid_data, extra_space, axis=2)

        grid_data[lat_idx, lon_idx, next_to_fill[lat_idx, lon_idx]] = data[row,
                                                                           col]
        next_to_fill[lat_idx, lon_idx] += 1

# remove unnecessary nans that were appended
not_all_nan_idxs = ~np.isnan(grid_data).all(axis=(0, 1))
grid_data = grid_data[:, :, not_all_nan_idxs]

answered Nov 18, 2019 at 12:00

nheise

3111 silver badge8 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

clearseplex Over a year ago

Thanks! I think fill_value should be 0, not 1000 like in your code.

nheise Over a year ago

You mean when you create next_to_fill? It should be 1000, so that you start at the end of your array. If you start at 0 you will overwrite your previous data in grid_data.

Collectives™ on Stack Overflow

Append Value to 3D array numpy

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related