Finding duplicate subarrays inside a multi dimensional array

Question

I'm saving images in a multi dimenisonal numpy array a in shape (100,128,128,1). I'd like to check whether there are duplicate images in array a. Other than some for loops implementation, what would be the pythonic way to do it?

@Divakar Images are in (128,128,1) size and there are 100 images in the array a. I'd like to see whether there are duplicate images in entire a. — colt.exe
– colt.exe, Commented Jun 14, 2020 at 15:56
And if there are duplicate images in array a, what's the expected output? — Divakar
– Divakar, Commented Jun 14, 2020 at 15:57
Think you should look into np.unique( ..axis, return_index=True). Check out return_inverse argument too. — Divakar
– Divakar, Commented Jun 14, 2020 at 16:00

Divakar · Accepted Answer · 2020-06-14 18:33:21Z

2

As stated in the comments, we need np.unique, but some extra steps are needed to gather duplicate indices. Here's the complete implementation that gather duplicate images in one tuple each, while non-duplicate ones would end up alone in a tuple each -

def gather_duplicate_indices(a):
    _,tags,count = np.unique(a, axis=0, return_inverse=True, return_counts=True)
    sidx = tags.argsort()
    return np.split(sidx, count.cumsum())[:-1]

Sample run -

In [43]: np.random.seed(0)
    ...: a = np.random.randint(0,5,(10,2,2,1))
    ...: a[5] = a[2]
    ...: a[7] = a[2]
    ...: a[6] = a[1]
    ...: a[9] = a[4]
# so the pairings are (2, 5, 7), (1, 6), (4, 9), while rest are singles.

In [44]: gather_duplicate_indices(a)
Out[44]: 
[array([4, 9]),
 array([8]),
 array([3]),
 array([1, 6]),
 array([2, 5, 7]),
 array([0])]

edited Jun 14, 2020 at 18:33

answered Jun 14, 2020 at 18:23

Divakar

222k19 gold badges273 silver badges374 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Finding duplicate subarrays inside a multi dimensional array

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related