81

I want to iterate through a list of files without caring about what characters the filenames might contain, so I use a list delimited by null characters. The code will explain things better.

# Set IFS to the null character to hopefully change the for..in
# delimiter from the space character (sadly does not appear to work).
IFS=$'\0'

# Get null delimited list of files
filelist="`find /some/path -type f -print0`"

# Iterate through list of files
for file in $filelist ; do
    # Arbitrary operations on $file here
done

The following code works when reading from a file, but I need to read from a variable containing text.

while read -d $'\0' line ; do
    # Code here
done < /path/to/inputfile
Improve this question
3
  • 9
    I don't think it's possible to store null characters in a bash variable. At least, I've never found a way to do it... Commented Dec 30, 2011 at 18:25
  • 2
    Confirmed, bash: warning: command substitution: ignored null byte in input. This is because bash is intended for posix derivative environments, in which env vars are internally stored in a null-terminated buffer, and bash vars are (in every case I've ever examined) host env vars. Commented Apr 16, 2020 at 15:58
  • You may be able to store a null char in a bash variable, but you can not get it out, so there is no way to tell. First example prove that assigning non-displayable chars works (as we all know), e.g. a tab in octal : test=$'a\011b';echo ${#test} ="${test}"= results in 3 =a b=. Then try an octal 0 : test=$'a\0b';echo ${#test} ="${test}"= results in 1 =a=; this reports the zero-terminated string length of $test as 1, but the 'b' and another zero may still be stored into the variable, we do not know. Commented Oct 15, 2022 at 10:59

6 Answers 6

Reset to default
128

The preferred way to do this is using process substitution:

while IFS= read -r -d $'\0' file <&3; do
    # Arbitrary operations on "$file" here
done 3< <(find /some/path -type f -print0)

If you were hell-bent on parsing a bash variable in a similar manner, you can do so as long as the list is not NUL-terminated.

Here is an example of bash var holding a tab-delimited string:

$ var="$(echo -ne 'foo\tbar\tbaz\t')"
$ while IFS= read -r -d $'\t' line <&3; do
    echo "#$line#"
  done 3<<<"$var"
#foo#
#bar#
#baz#
Improve this answer
Sign up to request clarification or add additional context in comments.

14 Comments

What is the use of the IFS since the -d flag is set?
@thisirs By setting IFS to the null string, leading and trailing whitespace characters will be preserved.
Fully agree with @joanpau. I don't know how stuff worked in 2011, but in 2016 with bash4 this method does not work. You can easily verify that this fails it if you assign var=$(find . -print0) prior to while loop. Process substitution works indeed, but variables not. Even if you built a variable on the fly like var=$(echo -e "some\0text") will fail to separate some from text. For variables you need to make a trick like this:stackoverflow.com/questions/6570531/…
This is not setting IFS to the null string. It is unsetting IFS, causing it to use the default values ($' \t\n').
@jeremysprofile actually, IFS= is disabling word-splitting. It is not setting IFS back to its default value like unset IFS does.
|
8

Pipe them to xargs -0:

files="$( find ./ -iname 'file*' -print0 | xargs -0 )"

xargs manual:

-0, --null
    Input items are terminated by a null character instead of
    by whitespace, and the quotes and backslash are not
    special (every character is taken literally).
Improve this answer

1 Comment

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
5

Use env -0 to output the assignments by the zero byte.

env -0 | while IFS='' read -d '' line ; do
    var=${line%%=*}
    value=${line#*=}
    echo "Variable '$var' has the value '$value'"
done
Improve this answer

1 Comment

This example is simpler than the accepted answer, but it's a bit misleading. env -0 isn't part of the solution - it's only what you're using to demonstrate the solution.
1

In terms of readability and maintainability a bash function might be cleaner:

An example that converts MOV files to MP4 using ffmpeg (works with files containing spaces and special characters):

#!/usr/bin/env bash

do_convert () { 
  new_file="${1/.mov/.mp4}"
  ffmpeg -i "$1" "$new_file" && rm "$1" 
}

export -f do_convert  # needed to make the function visible inside xargs

find . -iname '*.mov' -print0 | xargs -0 -I {} bash -c 'do_convert "{}"' _ {}

Does not apply to the OP's question but in case your input is generated by find then there is no need to pipe via xargs -0 as find is perfectly capable of handling non-ascii characters and spaces in file names. If you don't care about readability and maintainability then the command above can be simplified to:

find . -type f -iname "*.mov" -exec bash -c 'ffmpeg -i "${1}" "${1%.*}.mp4" && rm "${1}"' _ {} \;
Improve this answer

1 Comment

As noted in this answer, it doesn't handle arbitrary strings with null characters. However, it does answer a different, common question for performing actions on files without using the "-print0 | xargs -0" form, and without using IFS= overrides.
0

Here is the simplest solution based on choroba's answer, which more directly answers the OP's question (it still doesn't use a bash variable).

find /some/path -type f -print0 | while IFS='' read -d '' file ; do
    # Arbitrary operations on $file here
done

I prefer this form because it uses a single, easy-to remember pipe (the |) rather than the complex "broken-heart" <&3 and 3< pipes found in the top-voted answer.

Improve this answer

Comments

-6

I tried working with the bash examples above, and finally gave up, and used Python, which worked the first time. For me it turned out the problem was simpler outside the shell. I know this is possibly off topic of a bash solution, but I'm posting it here anyway in case others want an alternative.

import sh
import path
files = path.Path(".").files()
for x in files:
    sh.cp("--reflink=always", x, "UUU00::%s"%(x.basename(),))
    sh.cp("--reflink=always", x, "UUU01::%s"%(x.basename(),))
Improve this answer

1 Comment

Had a similar issue with a (sh-, not bash-) shell script. Decided to rewrite that whole damn script in Perl.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.