How avoid changing numpy array's shape when appending?

Question

i am trying to read 30 images and create a large numpy array from it by appending the numpy array for each image to that one single array so that i can use it later in a flow function for keras.

I have an empty list to which I am appending numpy arrays after doing face detection in a loop, after which I am creating a large numpy array from this list. The problem is that when,I create a numpy array from this list, it changes the shape of my array, which was originally (1,139,139,3), to (30,1,139,139,3). It basically adds the total number of images at the start as I am appending, and I want to get back to original shape. I do not want to use reshape as that might affect the data.

Here is the code:

img_width, img_height = 139, 139
confidence = 0.8
#graph = K.get_session().graph
data1 = []
def get_face(path):
    with graph.as_default():
        img = io.imread(path)
        dets = detector(img, 1)
        output = None
        for i, d in enumerate(dets):
            img = img[d.top():d.bottom(), d.left():d.right()]
            img = resize(img, (img_width, img_height))
            output = np.expand_dims(img, axis=0)
            break
        return output

for row in df.itertuples():
    data1.append(get_face(row[1]))
data1 = np.array(data1)
print(data1)

You're (most probably) creating a list of 30 numpy arrays of shape (1,139,139,3), which gets converted to (30,1,139,139,3), just as it should be. I think you need to add more context on what you want to achieve. Also, don't be afraid to use reshape, it doesn't randomly affect data. As long as you understand what it's doing nothing bad will happen. — fulaphex
– fulaphex, Commented Feb 13, 2018 at 15:13
@Fulaphex i added some more context as i want to have a numpy with the shape of (139,139,3) having stored all the images which then i want to use later. i tried reshape but it did not work — Rehan
– Rehan, Commented Feb 13, 2018 at 15:36
A single image has 139x139x3 integers, when you want to store data from 30 images you'll have 30x139x139x3 integers. You cannot really overcome this unless you reduce your matrix over the first axis. (30, 139, 139, 3) array would be an array of 30 pictures you loaded. — fulaphex
– fulaphex, Commented Feb 13, 2018 at 16:01
@Fulaphex oh ok.. but i guess an array of (30,139,139,3) will solve my issue as my main problem was that i was getting an error of array can have rank 4 while my array has status (30,1,139,139,3). if i can somehow remove 1 that might solve the issue. — Rehan
– Rehan, Commented Feb 13, 2018 at 16:11

fulaphex · Accepted Answer · 2018-02-13 21:21:38Z

2

As @filippo pointed out you probably want to omit np.expand_dims.

img_width, img_height = 139, 139
confidence = 0.8
#graph = K.get_session().graph
data1 = []
def get_face(path):
    with graph.as_default():
        img = io.imread(path)
        dets = detector(img, 1)
        output = None
        for i, d in enumerate(dets):
            img = img[d.top():d.bottom(), d.left():d.right()]
            output = resize(img, (img_width, img_height))
            break
        return output

for row in df.itertuples():
    data1.append(get_face(row[1]))
data1 = np.array(data1)
print(data1)

This piece of code will produce a list of 30 numpy arrays with shapes (139, 139, 3). Calling np.array constructor on that will give you array with shape (30, 139, 139, 3). You should also read documentation for np.stack and np.concatenate. Using the second function you can actually get what you want keeping np.expand_dims if for whatever reason you might need this.

answered Feb 13, 2018 at 21:21

fulaphex

3,3393 gold badges21 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

filippo Over a year ago

Keras 2D layers work with 4D tensors with shape (samples, rows, cols, channels), if you're using channels_last ordering. When you're working with single image batches it is common to call expand_dims so you still have a 4D array. Guess that call is probably left there from some tutorial that expected a single sample.

hpaulj · Accepted Answer · 2018-02-13 17:54:57Z

1

np.array joins the elements of the list on a new front dimension:

In [141]: alist = []
In [142]: for i in range(2):
     ...:     arr = np.zeros((3,4))
     ...:     alist.append(arr)
     ...:     
In [143]: np.array(alist).shape
Out[143]: (2, 3, 4)

expand_dims adds a new dimension:

In [144]: alist = []
In [145]: for i in range(2):
     ...:     arr = np.zeros((3,4))
     ...:     arr = np.expand_dims(arr,0)
     ...:     alist.append(arr)
     ...:     
In [146]: np.array(alist).shape
Out[146]: (2, 1, 3, 4)

concatenate joins on an existing dimension:

In [149]: np.concatenate(alist, axis=0).shape
Out[149]: (2, 3, 4)

An alternative to np.array is np.stack.

answered Feb 13, 2018 at 17:54

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Collectives™ on Stack Overflow

How avoid changing numpy array's shape when appending?

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related