bash: filter list of files, that can contain spaces" - associative array required?

Question

I want to create a list of filenames - that might have some filenames with spaces within.
This list shall be filtered in bash (not with 'find' itself - or alike).
The final list has to be processed somehow.
I can't make it work - unless using associative array.

Here's my solution.

test directory:

> find $HOME/test-dir/
/home/frank/test-dir/
/home/frank/test-dir/FileA
/home/frank/test-dir/File D
/home/frank/test-dir/FileC
/home/frank/test-dir/FileB

script #1 (works):

> cat test2.sh 
#!/bin/bash

mapfile -t Data < <(find $HOME/test-dir/ -type f)

for Key in ${!Data[@]}
do
    echo "$Key -> ${Data[$Key]}"
done
echo

# remove #1 element via variable
Del=2
unset 'Data[$Del]'

while read Value
do
    echo "$Value"
done < <(IFS=$'\n'; for Value in ${Data[@]}; do echo $Value; done)
echo

Note: The process substitution at the end of the script shall enable to handle the values only within the loop w/o need to know it's stored in an associative array (for old code).

output:

> ./test2.sh 
0 -> /home/frank/test-dir/FileA
1 -> /home/frank/test-dir/File D
2 -> /home/frank/test-dir/FileC
3 -> /home/frank/test-dir/FileB

/home/frank/test-dir/FileA
/home/frank/test-dir/File D
/home/frank/test-dir/FileB

Any attempt to use a pure array fails for the "File D". I can populate the array, but traversing or trying to remove an element breaks it again:

script #2 (does not work):

> cat test2.sh 
#!/bin/bash

OLDIFS="$IFS"
IFS=$'\n'
readarray -t Data < <(find $HOME/test-dir/ -type f)
IFS="$OLDIFS"  # works only if i drop this

for Value in ${Data[@]}
do
    echo "$Value"
done
echo

# remove #1 element via variable
Del=2
unset 'Data[$Del]'

for Value in ${Data[@]}
do
    echo "$Value"
done

output:

> ./test2.sh 
/home/frank/test-dir/FileA
/home/frank/test-dir/File
D
/home/frank/test-dir/FileC
/home/frank/test-dir/FileB

/home/frank/test-dir/FileA
/home/frank/test-dir/File
D
/home/frank/test-dir/FileB

Interestingly, removing the restoration of the IFS (see commented line above) results in

output:

> ./test2.sh 
/home/frank/test-dir/FileA
/home/frank/test-dir/File D
/home/frank/test-dir/FileC
/home/frank/test-dir/FileB

/home/frank/test-dir/FileA
/home/frank/test-dir/File D
/home/frank/test-dir/FileB

But i want to localize the setting of IFS to not interfere with old code, that relies on different IFS value.

What's the way to make it work with pure array (not associative array)?

Addendum:

This also works:

> cat test2b.sh 
#!/bin/bash

readarray -t Data < <(find $HOME/test-dir/ -type f)

while read Value
do
    echo "$Value"
done < <(IFS=$'\n'; for Value in ${Data[@]}; do echo $Value; done)
echo

# remove #1 element via variable
Del=2
unset 'Data[$Del]'

while read Value 
do
    echo "$Value"
done < <(IFS=$'\n'; for Value in ${Data[@]}; do echo $Value; done)
echo

But kind of strange to have to go like this. I used process substitution for my solution with associative array. But that was, because i introduced key->value myself and had to get back to the values only. Required to do so for the pure array feels strange.

array and associative array are based on the same implementation. You can access your array in the associative array manner like here: stackoverflow.com/questions/9084257/…. Or you can use process substituion. Any better way? — Frank Bergemann
– Frank Bergemann, Commented Jan 5, 2020 at 9:02
Have a look at this bash pitfall. It will point you in the right direction. — kvantour
– kvantour, Commented Jan 5, 2020 at 9:52

kvantour · Accepted Answer · 2020-01-05 22:05:45Z

1

My suggestion to you is the following:

use find to list the files and process them in a while loop
let find print the filesnames followed by a null-character instead of a newline character
do the selection in the loop.

This would look like this:

#!/usr/bin/env bash
del=2
counter=0
find $HOME/test-dir/ -type f -print0 | while read -d $'\0' file; do
   # ignore element
   (( counter++ == del )) && continue
   # perform action
   echo "$file"
done

edited Jan 5, 2020 at 22:05

answered Jan 5, 2020 at 21:41

kvantour

26.9k4 gold badges57 silver badges81 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

dash-o · Accepted Answer · 2020-01-05 10:54:59Z

0

Consider using 'printf' to convert the array into new-line separated values. More compact, and will not get tricked by file names starting with '-' (that looks like options).

while read Value 
do
    echo "$Value"
done < <(printf '%s\n' "${Data[@]}" )

answered Jan 5, 2020 at 10:54

dash-o

14.6k1 gold badge14 silver badges43 bronze badges

1 Comment

dash-o Over a year ago

@kvantour Yes, but in this case, OP specifically asked for a script that will handle filename with non-special characters + space. No need to deal with new lines. If such a requirement will exists, it will require changing the script to handle null terminated file names.

tshiono · Accepted Answer · 2020-01-06 00:32:54Z

The cause of the problem is you are not wrapping ${Data[@]} with double quotes in the for loop. Then the array ${Data[@]} is expanded to a list of IFS-splitted words against your will.
In addition you need not to temporarily assign IFS to a newline for readarray.

Then script2 will look like:

readarray -t Data < <(find $HOME/test-dir/ -type f)

for Value in "${Data[@]}"
do
    echo "$Value"
done
echo

# remove an element via variable
Del=2
unset 'Data[$Del]'

for Value in "${Data[@]}"
do
    echo "$Value"
done

BTW as @kvantour suggests, it is strongly recommended to use a NUL character as a filename delimiter instead of a newline. Then the readline sentence will look like:

while IFS= read -r -d "" f; do
    Data+=("$f")
done < <(find $HOME/test-dir/ -type f -print0)

If readarray supports -d option (bash >= 4.4) you can alternatively say:

readarray -t -d "" Data < <(find $HOME/test-dir/ -type f -print0)

Hope this helps.

Collectives™ on Stack Overflow

bash: filter list of files, that can contain spaces" - associative array required?

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related