I would like to split my data into two random sets. I've done the first part:
ind = np.random.choice(df.shape[0], size=[int(df.shape[0]*0.7)], replace=False)
X_train = df.iloc[ind]
Now I would like to select all index' not in ind to create my test set. Please can you tell me how to do this?
I thought it would be
X_test = df.iloc[-ind]
but apparently it isn't
Check out scikit-learn test_train_split()
Example from the docs:
>>> import numpy as np
>>> from sklearn.model_selection import train_test_split
>>> X, y = np.arange(10).reshape((5, 2)), range(5)
>>> X
array([[0, 1],
[2, 3],
[4, 5],
[6, 7],
[8, 9]])
>>> list(y)
[0, 1, 2, 3, 4]
>>>
>>> X_train, X_test, y_train, y_test = train_test_split(
... X, y, test_size=0.33, random_state=42)
...
>>> X_train
array([[4, 5],
[0, 1],
[6, 7]])
>>> y_train
[2, 0, 3]
>>> X_test
array([[2, 3],
[8, 9]])
>>> y_test
[1, 4]
In your case you could do it like this:
larger, smaller = test_train_split(df, test_size=0.3)
Another way to get a 70 - 30 train test split would be to generate indices, shuffle them and then split it in 70 - 30 parts.
ind = np.arange(df.shape[0])
np.random.shuffle(ind)
X_train = df.iloc[ind[:int(0.7*df.shape[0])],:]
X_test = df.iloc[ind[int(0.7*df.shape[0]):],:]
I would suggest converting the pandas.dataframe to a numeric matrix and using scikit-learn's train_test_split to do the splitting unless you really want to do it this way.
train_test_split before (although I had forgotten about it), but I find the data easier to check and visualise in a dataframe.Try this pure-Python approach.
ind_inversed = list(set(range(df.shape[0])) - set(ind))
X_test = df.iloc[ind_inversed]
ind was calculated the same way as in the original question. ind_inversed represents all the other indecies not in ind.int() argument must be a string, a bytes-like object or a number, not 'set' with this technique