How to properly stack numpy arrays?

Question

I am having trouble understanding how data is being stacked in a numpy array and why I cannot match the last data that I added to an array with the last generated data. Here is a MWE:

import numpy as np
np.random.seed(1)

# build storage
container = []

# gen data
x = np.random.random((13, 1, 64, 768))

# add to container
container.append(x)

# gen data
x2 = np.random.random((13, 1, 64, 768))

# add to container
container.append(x2)

# convert to np array
container = np.asarray(container)

# reshape to [13, 2, 64, 768]
container = container.reshape(13, 2, 64, 768)

# check that the last generated data matches the last appended data
assert np.all(x2.flatten() == container[:, -1, :, :].flatten()), 'not a match'

Use np.concatenate with the desired axis value. np.array joins the arrayys on a leading axis, making (2,...) . reshape doesn't move axes around. — hpaulj
– hpaulj, Commented Jan 9, 2021 at 16:10

Stefan · Accepted Answer · 2021-01-09 16:29:35Z

3

Instead of stacking manually with appending to lists and then reshaping you could use the vstack or the concatenate function of numpy.

# gen data
x1 = np.random.random((13, 1, 64, 768))
x2 = np.random.random((13, 1, 64, 768))

container = np.vstack((x1,x2))
assert np.all(x2.flatten()) == np.all(container[:, -1, :, :].flatten()), 'not a match'

To answer your question: your code does work, just make sure to put np.all() at both sides of the comparison. It's always a good idea to make your input much smaller (say (2,1,2,2)) so you can see what actually happens.

edited Jan 9, 2021 at 16:29

answered Jan 9, 2021 at 15:48

Stefan

9575 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

John Stud Over a year ago

Interesting, but any idea why the reshape is not keeping the integrity of the batches?

hpaulj · Accepted Answer · 2021-01-09 17:06:56Z

In [152]: alist = []
In [154]: alist.append(np.random.random((2,1,3)))
In [155]: alist.append(np.random.random((2,1,3)))
In [156]: alist
Out[156]: 
[array([[[0.85221826, 0.56088315, 0.06232853]],
 
        [[0.0966469 , 0.89513922, 0.44814579]]]),
 array([[[0.86207845, 0.88895573, 0.62069196]],
 
        [[0.11475614, 0.29473531, 0.11179268]]])]

Using np.array to join the list elements produces a 4d array - it has joined them on a new leading dimension:

In [157]: arr = np.array(alist)
In [158]: arr.shape
Out[158]: (2, 2, 1, 3)
In [159]: arr[-1,]        # same as alist[-1]
Out[159]: 
array([[[0.86207845, 0.88895573, 0.62069196]],

       [[0.11475614, 0.29473531, 0.11179268]]])

If we concatenate on one of the dimensions:

In [160]: arr = np.concatenate(alist, axis=1)
In [161]: arr
Out[161]: 
array([[[0.85221826, 0.56088315, 0.06232853],
        [0.86207845, 0.88895573, 0.62069196]],

       [[0.0966469 , 0.89513922, 0.44814579],
        [0.11475614, 0.29473531, 0.11179268]]])
In [162]: arr.shape
Out[162]: (2, 2, 3)     # note the shape - that 2nd 2 is the join axis
In [163]: arr[:,-1]
Out[163]: 
array([[0.86207845, 0.88895573, 0.62069196],
       [0.11475614, 0.29473531, 0.11179268]])

[163] has the same numbers as [159], but a (2,3) shape.

reshape keeps the values, but may 'shuffle' them:

In [164]: np.array(alist).reshape(2,2,3)
Out[164]: 
array([[[0.85221826, 0.56088315, 0.06232853],
        [0.0966469 , 0.89513922, 0.44814579]],

       [[0.86207845, 0.88895573, 0.62069196],
        [0.11475614, 0.29473531, 0.11179268]]])

We have transpose the leading 2 axes before reshape to match [161]

In [165]: np.array(alist).transpose(1,0,2,3)
Out[165]: 
array([[[[0.85221826, 0.56088315, 0.06232853]],

        [[0.86207845, 0.88895573, 0.62069196]]],


       [[[0.0966469 , 0.89513922, 0.44814579]],

        [[0.11475614, 0.29473531, 0.11179268]]]])
In [166]: np.array(alist).transpose(1,0,2,3).reshape(2,2,3)
Out[166]: 
array([[[0.85221826, 0.56088315, 0.06232853],
        [0.86207845, 0.88895573, 0.62069196]],

       [[0.0966469 , 0.89513922, 0.44814579],
        [0.11475614, 0.29473531, 0.11179268]]])

Collectives™ on Stack Overflow

How to properly stack numpy arrays?

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related