How to get unique string from variable in bash

Question

Hello I would like to ask how can I get unique string from variable.

while read -r line      
    do
        route=$(echo $line | awk -F'[:]' '{print $2}') #get path from log file
        if [ "`dirname "$route"`" == "`xrealpath "$PWD"`" ]; then #compare path from log file with $PWD
            name=$(echo $line | awk -F'[:]' '{print $1}') #take name from 1st column in log file
        fi
        if ! [ "$name" == "$help_name" ]; then
            echo $name
            help_name=$name
            pom=$pom:$name
        fi
    done < $WEDI_RC

Sample logfile:

proj.sh:/Users/Tom/Documents/proj.sh:2015-03-21:1
proj1.sh:/Users/Tom/Documents/proj.sh:2015-03-21:1
proj.sh:/Users/Tom/Documents/proj.sh:2015-03-21:2
proj1.sh:/Users/Tom/Documents/proj.sh:2015-03-21:2
proj.sh:/Users/Tom/Documents/proj.sh:2015-03-21:3
proj1.sh:/Users/Tom/Documents/proj.sh:2015-03-21:3

How can I echo each unique just one time?

My output now looks something like this:

proj.sh
proj1.sh
proj.sh
proj1.sh
proj.sh
:proj.sh:proj1.sh:proj.sh:proj1.sh:proj.

Expecting output:

proj.sh
proj1.sh

I don't know how much files can be readed in while cycle. We cannot use any temporary files Thank you

If you want to avoid any duplicates then you need to keep a list/array of every value you've seen not just the most recent one (that only works for sorted input). — Etan Reisner
– Etan Reisner, Commented Mar 20, 2015 at 22:42

John1024 · Accepted Answer · 2015-03-21 06:25:53Z

1

Answer for Original Version of This Question

This uses the associative array seen to keep track of what names have been seen:

declare -A seen
while read -r line    
... blabla ...
do
    if [ -z "${seen[$name]}" ]; then
        echo $name
        seen["$name"]=1
        pom=$pom:$name
    fi
done < "$WEDI_RC"

Working Example (No `blabla`)

Let us start with this file:

$ cat file
proj.sh
proj1.sh
proj.sh
proj1.sh
proj.sh

We will run this code (note that ...blabla... has been removed and the loop now reads in name directly):

$ cat script.sh
declare -A seen
while read -r name
do
    if [ -z "${seen[$name]}" ]; then
        echo $name
        seen["$name"]=1
        pom=$pom:$name
    fi
done < file
declare -p pom

This is the result:

$ bash script.sh
proj.sh
proj1.sh
declare -- pom=":proj.sh:proj1.sh"

Answer for Revised Question

In the revised question, the following code appears:

    route=$(echo $line | awk -F'[:]' '{print $2}') #get path from log file
    if [ "`dirname "$route"`" == "`xrealpath "$PWD"`" ]; then #compare path from log file with $PWD
        name=$(echo $line | awk -F'[:]' '{print $1}') #take name from 1st column in log file

This means that, as the code runs, name may never be set depending on the current directory when the script is run. This would explain the error messages reported in the comments.

edited Mar 21, 2015 at 6:25

answered Mar 20, 2015 at 23:13

John1024

115k15 gold badges152 silver badges183 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Tom Over a year ago

no it didn't worked for me: Tom$ ./wedi -l ./wedi: line 16: declare: -A: invalid option declare: usage: declare [-afFirtx] [-p] [name[=value] ...] wedi ./wedi: line 150: proj1.sh: syntax error: invalid arithmetic operator (error token is ".sh")

John1024 Over a year ago

@tom That is the message that occurs if name is never defined. In the code as given in the question, the loop reads in line and then ... blabla ... occurs. It is necessary that ... blabla ... define the variable name in order for something useful to happen. Does your ... blabla ... define the variable name?

John1024 Over a year ago

@Tom Please see the updated answer for code that is analogous to yours but is complete and runs (no blabla).

Tom Over a year ago

I add whole while cycle with "blabla", we cannnot use any temporary files

John1024 Over a year ago

So, that would appear to mean that blabla did not define name. "we cannnot use any temporary files" Why did you mention this? Temporary files are not used in this code.

|

Collectives™ on Stack Overflow

How to get unique string from variable in bash

1 Answer 1

Answer for Original Version of This Question

Working Example (No `blabla`)

Answer for Revised Question

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Answer for Original Version of This Question

Working Example (No blabla)

Answer for Revised Question

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related

Working Example (No `blabla`)