Splitting pytorch dataloader into numpy arrays

Question

In principle I'd like to do the opposite of what was done here https://datascience.stackexchange.com/questions/45916/loading-own-train-data-and-labels-in-dataloader-using-pytorch.

I have a Pytorch dataloader train_dataloader with shape (2000,3). I want to store the 3 dataloader columns in 3 separate numpy arrays. (The first column of the dataloader contains the data, the second column contains the labels.)

I managed to do it for the last batch of the train_dataloader (see below), but unfortunately couldn't make it work for the whole train_dataloader.

for X, y, ind in train_dataloader:
    pass

train_X = np.asarray(X, dtype=np.float32)
train_y = np.asarray(y, dtype=np.float32)

Any help would be very much appreciated!

Shai · Accepted Answer · 2022-04-14 09:11:33Z

1

You can collect all the data:

all_X = []
all_y = []
for X, y, ind in train_dataloader:
  all_X.append(X)
  all_y.append(y)
train_X = torch.cat(all_X, dim=0).numpy()
train_y = torch.cat(all_y, dim=0).numpy()

answered Apr 14, 2022 at 9:11

Shai

115k39 gold badges259 silver badges398 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Splitting pytorch dataloader into numpy arrays

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related