Reshape Python list to numpy array

Question

I have a use-case where I need to make predictions from a trained LSTM model (TensorFlow = 2.8 and Python = 3.9) where the dataset is large and results in OOM error on my GPU card. The turnaround I have is to predict for each batch and store the resulting tensor in a Python3 list:

# Create TF dataset to iterate over-
train_dataset = tf.data.Dataset.from_tensor_slices(X_train).batch(batch_size)
# batch_size = 256

# Python3 list to contain all model predictions-
preditcions = []

for x in train_dataset:
    y_pred = model_e2d2.predict(x)
    preditcions.append(y_pred)

# Sanity check-
len(preditcions)
# (2048)

# Each batch of prediction has the size: (256, 5, 11).

# Sanity check-
preditcions[0].shape, preditcions[-1].shape
# ((256, 5, 11), (222, 5, 11))

The aim is to convert precitions Python3 list to a numpy array of desired shape: (524254, 5, 11)

?

preditcions = np.vstack(preditcions) should already do it? — Rutger Kassies
– Rutger Kassies, Commented May 25, 2022 at 10:45
Numpy has a reshape function: numpy.org/doc/stable/reference/generated/numpy.reshape.html — Tzane
– Tzane, Commented May 25, 2022 at 11:06

MSS · Accepted Answer · 2022-05-26 17:25:57Z

1

In general you can do it as follows:

predictions = np.concatenate(predictions, axis=0)

This gives you control on which axis you want to concatenate.

Look at the example given here https://numpy.org/doc/stable/reference/generated/numpy.concatenate.html#numpy-concatenate

>>a = np.array([[1, 2], [3, 4]])
>>b = np.array([[5, 6]])
>>np.concatenate([a, b], axis=0) #slight difference from example.
  # instead of tuple I have passed a list as is OP's requirement
array([[1, 2],
       [3, 4],
       [5, 6]])

All thanks to @hpaulj for correcting my mistake.

edited May 26, 2022 at 17:25

answered May 25, 2022 at 11:05

MSS

3,6932 gold badges28 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

hpaulj Over a year ago

The arrays are already 3d; he doesn't want to add a dimension. Also they aren't all the same shape (the last is smaller). So stack won't work here. (Otherwise it is useful for turning an list or object array of arrays into one).

MSS Over a year ago

Thanks for making it clear. I made a mistake.

Collectives™ on Stack Overflow

Reshape Python list to numpy array

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related