Using Bash, how can you traverse folders within specified folder, find all files of specified file type, and every time you find a file, get full file path with file name and full file path without file name as a variables and pass them to another Bash script, execute it, and continue searching for the next file?
5 Answers
Assuming a GNU find (which is not unreasonable) you can do this using just find:
find /path -type f -name '*.ext' -exec my_cool_script \{\} \;
5 Comments
ustun
What is the purpose of \{\} there? Any advantage over {} or '{}'?
unwind
That is a bit of extra safety against the shell. The manual page for find states: "Both of these constructions might need to be escaped (with a `\') or quoted to protect them from expansion by the shell."
neuro
the problem with that is that find with exec do not correctly handle long list of files. I've in something like "line too long". See my detailed answer ...
ephemient
@neuro: Really?
-exec ; executes one-at-a-time, and if that command line is too long, you're pretty screwed with either find or xargs. Now, GNU findutils has had -exec + bugs in the past, but they should be resolved now, and the difference between find -exec + versus find -print0 | xargs -0 is pretty minimal.neuro
@ephemient : Well I've had this sort of errors in the past, so I've used xargs since then. I probably hit some bug at the time. I used to use -exec but I found that using xargs is more readable than -exec and the escape sequences. The -print0 | xargs -0 is a magical way to handle the spaces in names particularly when you use your script in a windows/mingw/cygwin environment ...
find is the way. Using xargs handle long list of files/dirs. Moreover to handle correctly names with spaces and problem like that, the best find line command I've found is :
find ${directory} -name "${pattern}" -print0 | xargs -0 ${my_command}
The trick is the find -print0 that is compatible with the xargs -0 : It replace endlines by '\0' to correctly handle spaces and escape characters. Using xargs spares you some "line too long" message when your filelist is too long.
You can use xargs with --no-run-if-empty to handle empty lists and --replace to manage complex commands.
Comments
If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:
find . -name '*.ext' | parallel echo {} '`dirname {}`'
Substitute echo with your favorite bash command and ext with the file extension you are looking for.
Watch the intro video for GNU Parallel to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ
Comments
looks very much like homework.
find /path -type f -name "*.ext" -printf "%p:%h\n" | while IFS=: read a b
do
# execute your bash script here
done
read the man page of find for more printf options....
6 Comments
ephemient
That doesn't seem quite right... if
%p:%h has no spaces, read a b will put it all in a and nothing in b, unless you set IFS=:.
ghostdog74
decided to use back : since there might be files with spaces
ephemient
There can easily be files with colons too. I would use
'%p\n%h\n' and read a && read b, but that still fails if there are filenames with embedded newlines.ephemient
In UNIX, any byte sequence (possibly with a length restriction) not containing NUL ("\0") or '/' is acceptable for a file name, even if they are not printable characters.
neuro
Beware of escape characters. Use find -print0 with xargs -0. see my more precise answer ...
|
As some have already mentioned, Linux/UNIX has no mandatory file extensions. However, on UNIX based systems the type of many commonly encountered files can be determined by the file command. The following example employs the file command to pass the name and type or each file to my_script:
find /path -type f | xargs file | while read -r line ; do my_script $line; done
The xargs command has been used to minimise the number of exec's of the file command.
The type info will be quite extensive, as can be seen in the following sample output from the xargs file step:
/usr/bin/easy_install-3.8: Python script, ASCII text executable
/usr/bin/splain: Perl script text executable
/usr/bin/nvzoom: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=e2295b7d737246db5a2986c790452139793bdbfc, for GNU/Linux 3.2.0, stripped
The other requirements around extracting the path from the filename could be done inside my_script or if it has to be done prior to my_script, by using bash string extraction or sed/awk/perl to reformat the output.
Should you expect any filenames with spaces or other special characters in them, then as suggested in previous answers, you can use -print0:
find /path -type f -print0 | xargs -0 file | while read -r line ; do my_script "$line"; done
