Remove duplicate hash from perl array

Question

I have the perl array as below

my @arr = ({
  CONTEXTID => 1230,
  NAME => 'test8824',
  PROVIDERID => 163
}, {
  CONTEXTID => 8824,
  NAME => 'test8824',
  PROVIDERID => 77
}, {
  CONTEXTID => 8824,
  NAME => 'test8824',
  PROVIDERID => 779
}, {
  CONTEXTID => 8824,
  NAME => 'test8824',
  PROVIDERID => 141
}, {
  CONTEXTID => 1230,
  NAME => 'test8824',
  PROVIDERID => 163
})

I want to remove the duplicate hashes from the array , The output should be like this:

({
  CONTEXTID => 1230,
  NAME => 'test8824',
  PROVIDERID => 163
}, {
  CONTEXTID => 8824,
  NAME => 'test8824',
  PROVIDERID => 77
}, {
  CONTEXTID => 8824,
  NAME => 'test8824',
  PROVIDERID => 779
}, {
  CONTEXTID => 8824,
  NAME => 'test8824',
  PROVIDERID => 141
}
)

The duplicate will be identified only when all the keys of hash are matching else its not duplicate.

ikegami · Accepted Answer · 2020-01-17 09:13:07Z

4

The following is a common idiom for removing duplicates:

my %seen;    
my @unique = grep !$seen{$_}++, @strings;

That use a string comparisons to determine if two items are identical or not. That will not do in our case (as that would effectively compare the addresses of the hashes, finding them all unique).

But we can easily generalize the above as follows:

my %seen;    
my @unique = grep !$seen{key($_)}++, @items;

All we need now is a function key that produces a string such that the following conditions are true:

key($a) ne key($b) if $a is considered to be different than $b.
key($a) eq key($b) if $a is considered to be the same as $b.

In this case, we could use the following:

use feature qw( state );

use Cpanel::JSON::XS qw( );

sub key {
   state $encoder = Cpanel::JSON::XS->new->canonical;
   return $encoder->encode($_[0]);
}

edited Jan 17, 2020 at 9:13

answered Jan 17, 2020 at 8:16

ikegami

391k17 gold badges291 silver badges555 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Grinnz Over a year ago

An alternative form of the algorithm: use List::UtilsBy 'uniq_by'; my @unique = uniq_by { key($_) } @items;

ikegami Over a year ago

A lot of work just to replace something with which one should be familiar, but it is more self-documenting.

user3817378 Over a year ago

@ikegami thanks for the answer, But my existing code based doesn't allow to introduce the perl module Cpanel::JSON::XS

ikegami Over a year ago

The concept still applies whether you use that module or not. Adjust to your needs.

user3817378 · Accepted Answer · 2020-01-21 06:00:25Z

2

In one of the stackoverflow answer I got this solution which works for me , I don't remember the original post thought.

 my %seen;
 my @array;
 @array =  grep { my $e = $_; my $key = join '___', map { $e->{$_}; } sort keys %$_;!$seen{$key}++ } @array;

put your array of href inside the array variable and returned array will be having unique hash values.

edited Jan 21, 2020 at 6:00

answered Jan 21, 2020 at 5:34

user3817378

13112 bronze badges

Comments

Polar Bear · Accepted Answer · 2020-01-17 20:46:45Z

1

Please verify that following piece satisfies your requirements

#!/usr/bin/perl

use strict;
use warnings;
use feature 'say';

use Data::Dumper;

my @result;
my %seen;

my @arr = ({
  CONTEXTID => 1230,
  NAME => 'test8824',
  PROVIDERID => 163
}, {
  CONTEXTID => 8824,
  NAME => 'test8824',
  PROVIDERID => 77
}, {
  CONTEXTID => 8824,
  NAME => 'test8824',
  PROVIDERID => 779
}, {
  CONTEXTID => 8824,
  NAME => 'test8824',
  PROVIDERID => 141
}, {
  CONTEXTID => 1230,
  NAME => 'test8824',
  PROVIDERID => 163
});

foreach my $el ( @arr ) {
    my $k = join('|', @$el{qw/CONTEXTID NAME PROVIDERID/ });
    push @result, $el unless $seen{$k};
    $seen{$k} = 1;
}

print Dumper(\@result);

Output:

$VAR1 = [
          {
            'PROVIDERID' => 163,
            'CONTEXTID' => 1230,
            'NAME' => 'test8824'
          },
          {
            'NAME' => 'test8824',
            'CONTEXTID' => 8824,
            'PROVIDERID' => 77
          },
          {
            'CONTEXTID' => 8824,
            'PROVIDERID' => 779,
            'NAME' => 'test8824'
          },
          {
            'NAME' => 'test8824',
            'CONTEXTID' => 8824,
            'PROVIDERID' => 141
          }
        ];

answered Jan 17, 2020 at 20:46

Polar Bear

6,8061 gold badge8 silver badges13 bronze badges

1 Comment

ikegami Over a year ago

Note that this assumes that the every record has the same three keys, but that's probably ok.

Collectives™ on Stack Overflow

Remove duplicate hash from perl array

3 Answers 3

4 Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related