31

I've got a problem on creating a numpy array of numpy arrays. I would create it in a loop:

a=np.array([])
while(...):
   ...
   b= //a numpy array generated
   a=np.append(a,b)
   ...

Desired result:

[[1,5,3], [9,10,1], ..., [4,8,6]]

Real result:

[1,5,3,9,10,1,... 4,8,6]

Is it possible? I don't know the final dimension of the array, so I can't initialize it with a fixed dimension.

2 Answers 2

66

Never append to numpy arrays in a loop: it is the one operation that NumPy is very bad at compared with basic Python. This is because you are making a full copy of the data each append, which will cost you quadratic time.

Instead, just append your arrays to a Python list and convert it at the end; the result is simpler and faster:

a = []

while ...:
    b = ... # NumPy array
    a.append(b)
a = np.asarray(a)

As for why your code doesn't work: np.append doesn't behave like list.append at all. In particular, it won't create new dimensions when appending. You would have to create the initial array with two dimensions, then append with an explicit axis argument.

Sign up to request clarification or add additional context in comments.

4 Comments

Thanks for the reply. The big problem is that I need about 100 arrays each one with a very big dimension, each one has got 1440000 integers inside. With list my program is killed by OS. Any suggestions?
Can you guess how many elements you will have, and preallocate a big enough array? If not you might have to get very clever, like building the output array in chunks or something.
You should also be able to preallocate using a larger array, then use a view of that array at the end once you know how many elements are actually present.
Finally I followed your suggestions! Thanks!
6

we can try it also :

arr1 = np.arange(4)
arr2 = np.arange(5,7)
arr3 = np.arange(7,12)

array_of_arrays = np.array([arr1, arr2, arr3])
array_of_arrays
np.concatenate(array_of_arrays)

2 Comments

Yes the point is that each inside array should be numpy not list as this saves all the space. Then make an array from the list of arrays.
Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.