Randomly split a numpy array

Question

I have a numpy array of size 46928x28x28 and I want to randomly split that array into two sub-matrices with sizes (41928x28x28) and (5000x28x28). Therefore, to randomly pick rows from the initial array. The code I tried so far (to calculate the indexes for the two sub-arrays) is the following:

ind = np.random.randint(input_matrix.shape[0], size=(5000,))
rest = np.array([i for i in range(0,input_matrix.shape[0]) if i not in ind])
rest = np.array(rest)

However, surprisingly the shapes of ind is (5000,) while the shape of the rest is (42192,). What am I doing wrong in that case?

Wouldn't train_test_split from sklearn work?

niraj
– niraj

2018-05-23 15:06:35 +00:00
Commented May 23, 2018 at 15:06 — niraj
– niraj, Commented May 23, 2018 at 15:06

Gianluca Micchi · Accepted Answer · 2018-05-23 16:35:05Z

20

The error is that randint is giving some repeated indices. You can test it by printing len(set(ind)) and you will see it is smaller than 5000.

To use the same idea, simply replace the first line with

ind = np.random.choice(range(input_matrix.shape[0]), size=(5000,), replace=False)

That being said, the second line of your code is pretty slow because of the iteration over the list. It would be much faster to define the indices you want with a vector of booleans, which would allow you to use the negation operator ~.

choice = np.random.choice(range(matrix.shape[0]), size=(5000,), replace=False)    
ind = np.zeros(matrix.shape[0], dtype=bool)
ind[choice] = True
rest = ~ind

On my machine, this method is exactly as fast as implementing scikit.learn's train_test_split, which makes me think that the two are doing exactly the same thing.

edited May 23, 2018 at 16:35

answered May 23, 2018 at 15:09

Gianluca Micchi

1,66316 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

niraj · Accepted Answer · 2018-05-23 15:09:58Z

9

One way may be to try using train_test_split from sklearn documentation:

import numpy as np
from sklearn.model_selection import train_test_split

# creating matrix
input_matrix = np.arange(46928*28*28).reshape((46928,28,28))
print('Input shape: ', input_matrix.shape)
# splitting into two matrices of second matrix by size
second_size = 5000/46928

X1, X2 = train_test_split(input_matrix, test_size=second_size)

print('X1 shape: ', X1.shape)
print('X2 shape: ', X2.shape)

Result:

Input shape:  (46928, 28, 28)
X1 shape:  (41928, 28, 28)
X2 shape:  (5000, 28, 28)

answered May 23, 2018 at 15:09

niraj

18.2k4 gold badges36 silver badges50 bronze badges

Comments

sacuL · Accepted Answer · 2018-05-23 15:22:23Z

I agree with the comment that train_test_split might be the way to go. However, since this is tagged numpy, here is a numpy way of doing things, which is pretty fast:

# recreate random array:
x = np.random.random((46928,28,28))

# pick your indices for sample 1 and sample 2:
s1 = np.random.choice(range(x.shape[0]), 41928, replace=False)
s2 = list(set(range(x.shape[0])) - set(s1))

# extract your samples:
sample1 = x[s1, :, :]
sample2 = x[s2, :, :]

Your output:

>>> sample1.shape
(41928, 28, 28)
>>> sample2.shape
(5000, 28, 28)

Timings:

Just out of curiosity, I timed this numpy method compared to sklearn.model_selection.train_test_split and got little difference. train_test_split is faster, but only by a tiny bit. In any case, I stand by train_test_split being the better option.

numpy method: 0.26082248413999876 seconds on average

train_test_split method: 0.22217219217000092 seconds on average

Grant · Accepted Answer · 2021-01-28 22:38:51Z

6

Just a quick update to say that this is readily solved using shuffle:

rng = np.random.default_rng()
rng.shuffle(data, axis = 0)
split1 = data[:41928]
split2 = data[41928:]

If you're using this for an ML application, this has the added benefit of randomizing the order of your train and test sets, which is often desirable. If you need to preserve the given ordering on the two split arrays, you can shuffle indices instead and re-sort:

idx = np.arange(data.shape[0])
rng.shuffle(idx)
idx1 = np.sort(idx[:41928])
idx2 = np.sort(idx[41928:])
split1 = data[idx1, ...]
split2 = data[idx2, ...]

answered Jan 28, 2021 at 22:38

Grant

611 silver badge3 bronze badges

Collectives™ on Stack Overflow

Randomly split a numpy array

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related