Find & remove duplicate strings via Unix shell script. How to?

Using awk :

$ printf '%s\n' cat dog cat bird fish bear dog | awk '!arr[$1]++'
cat
dog
bird
fish
bear

or

$ echo 'cat dog cat bird fish bear dog' | awk '!arr[$1]++' RS=" "

or

$ printf '%s\n' cat dog cat bird fish bear dog | sort -u

If it works in a shell, it will works in a script =)

Did you say Perl?

perl -e 'while($_=shift@ARGV){$seen{$_}++||print}print"\n" ' \
cat dog cat bird fish bear dog

Equivalently, dedupe.pl contains:

#!/usr/bin/perl
while ($w = shift @ARGV) {
    $seen{$w}++ || print "$w";
}
print "\n";

Now chmod u+x dedupe.pl and:

./dedupe.pl cat dog cat bird fish bear dog

Either way, output is as desired.

cat dog bird fish bear 

Ahh perl... the write-only language. :)

As long as you're calling out to another scripting language, might as well consider something readable. :)

#!/usr/bin/env ruby

puts ARGV.uniq.join(' ')

which means:

puts = "print whatever comes next"
ARGV = "input argument array"
uniq = "array method to perform the behavior you're looking for and remove duplicates"
join(' ') = "join with spaces instead of default of newline. Not necessarily needed if you're piping to something else"