complementary slicing in a numpy array

Question

If I have a numpy array for example :

A = np.array([[3, 2], [2, -1], [2, 3], [5, 6], [7,-1] , [8, 9]])

I would like to separate the part of the array with the subarrays having -1 from the ones who don't. Keep in mind that I'm working on very big data set, so every operation can be very long so I try to have the most effective way memory and CPU-time wise.

What I am doing for the moment is :

 slicing1 = np.where(A[:, 1] == -1)
 with_ones = A[slicing1]
 slicing2 = np.setdiff1d(np.arange(A.shape[0]), slicing1, assume_unique=True)
 without_ones = A[slicing2]

Is there a way to not create the slicing2 list to decrease the memory consumption as it can be very big? Is there a better way to approach the problem?

ely · Accepted Answer · 2015-03-18 01:09:03Z

6

One way is to store the logical index needed and then in the second case index using its logical negation:

In [46]: indx = A[:, 1] != -1

In [47]: A[indx]
Out[47]: 
array([[3, 2],
       [2, 3],
       [5, 6],
       [8, 9]])

In [48]: A[~indx]
Out[48]: 
array([[ 2, -1],
       [ 7, -1]])

answered Mar 18, 2015 at 1:09

ely

77.8k36 gold badges158 silver badges234 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

senderle Over a year ago

This is certainly better than using setdiff1d for many reasons. And since boolean arrays only use a byte per item, even two copies of a boolean index array will be smaller than an integer index array and its complement. To save even a bit more memory, I believe this won't make a copy: numpy.logical_not(ix, out=ix).

S4M · Accepted Answer · 2015-03-18 01:04:12Z

1

I managed to create without_ones with:

filter(lambda x: x[1] != -1,A)

answered Mar 18, 2015 at 1:04

S4M

4,7016 gold badges40 silver badges49 bronze badges

Comments

Balint Domokos · Accepted Answer · 2015-03-18 01:31:32Z

1

Or you could use a generator function:

A = np.array([[3, 2], [2, -1], [2, 3], [5, 6], [7,-1] , [8, 9]])

def filt(arr):
    for item in arr:
        if item[1]!=-1:
            yield item

new_len = 0
for item in A:
    if item[1] != -1:
        new_len += 1

without_ones = np.empty([new_len, 2], dtype=int)
for i, item in enumerate(filt(A)): 
    without_ones[i] = item

answered Mar 18, 2015 at 1:31

Balint Domokos

1,0218 silver badges12 bronze badges

Collectives™ on Stack Overflow

complementary slicing in a numpy array

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related