1

I'm new to UNIX and have this really simple problem:

I have a text-file (input.txt) containing a string in each line. It looks like this:

House
Monkey
Car

And inside my shell script I need to read this input file line by line to get to a variable like this:

things="House,Monkey,Car"

I know this sounds easy, but I just couldnt find any simple solution for this. My closest attempt so far:

#!/bin/sh
things=""
addToString() {
    things="${things},$1"
}
while read line; do addToString $line ;done <input.txt
echo $things

But this won't work. Regarding to my google research I thought the while loop would create a new sub shell, but this I was wrong there (see the comment section). Nevertheless the variable "things" was still not available in the echo later on. (I cannot just write the echo inside the while loop, because I need to work with that string later on)

Could you please help me out here? Any help will be appreciated, thank you!

Improve this question
6
  • This should work, just make sure to use quotes around your variables. Commented Apr 14, 2015 at 14:11
  • Well it does not, it just echos an empty string... Commented Apr 14, 2015 at 14:16
  • 1
    No, a while block does not create a subshell. Commented Apr 14, 2015 at 14:59
  • @Charles Duffy thank you for letting me know, it seemed that I maybe just misread this, I will change it in the question in about a second and will refer to your comment, thx ! +1 Commented Apr 14, 2015 at 15:03
  • ...that said, if you're targeting bash 4.0 or newer, you might prefer to use readarray -- or, for an older version, an appropriate invocation of read -a. Both of those will read into arrays, not strings with comma-separated values, but transforming the former into the latter is trivial. Commented Apr 14, 2015 at 15:05

4 Answers 4

Reset to default
1

What you proposed works fine! I've only made two changes here: Adding missing quotes, and handling the empty-string case.

things=""
addToString() {
    if [ -n "$things" ]; then
      things="${things},$1"
    else
      things="$1"
    fi
}
while read -r line; do addToString "$line"; done <input.txt
echo "$things"

If you were piping into while read, this would create a subshell, and that would eat your variables. You aren't piping -- you're doing a <input.txt redirection. No subshell, code works without changes.

That said, there are better ways to read lists of items into shell variables. On any version of bash after 3.0:

IFS=$'\n' read -r -d '' -a things <input.txt  # read into an array
printf -v things_str '%s,' "${things[@]}"     # write array to a comma-separated string
echo "${things_str%,}"                        # print that string w/o trailing comma

...on bash 4, that first line can be:

readarray -t things <input.txt # read into an array
Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Thanks for your answer! You were right this works and also that only piping into a while creates a subshell (I mixed that up). In my headline of my script I used "#!/bin/sh" ... changing this to using the bash ("#!/bin/bash") the code I posted works fine. Also thank you for cleaning that code up adding missing quotes and handling the empty-string case, and also thanks for introducing alternative solutions on shells 3.0+
Changing [[ $things ]] to [ -n "$things" ] will make this work with #!/bin/sh. By the way, you might consider read -r line rather than read line on both shells -- otherwise, content read has backslash-escape sequences processed rather than handled literally.
1

This is not a shell solution, but the truth is that solutions in pure shell are often excessively long and verbose. So e.g. to do string processing it is better to use special tools that are part of the “default” Unix environment.

sed ':b;N;$!bb;s/\n/,/g' < input.txt

If you want to omit empty lines, then:

sed ':b;N;$!bb;s/\n\n*/,/g' < input.txt

Speaking about your solution, it should work, but you should really always use quotes where applicable. E.g. this works for me:

things=""
while read line; do things="$things,$line"; done < input.txt
echo "$things"

(Of course, there is an issue with this code, as it outputs a leading comma. If you want to skip empty lines, just add an if check.)

Improve this answer

Comments

0

This might/might not work, depending on the shell you are using. On my Ubuntu 14.04/x64, it works with both bash and dash.

To make it more reliable and independent from the shell's behavior, you can try to put the whole block into a subshell explicitly, using the (). For example:

(
things=""
addToString() {
    things="${things},$1"
}
while read line; do addToString $line ;done 
echo $things
) < input.txt

P.S. You can use something like this to avoid the initial comma. Without bash extensions (using short-circuit logical operators instead of the if for shortness):

test -z "$things" && things="$1" || things="${things},${1}"

Or with bash extensions:

things="${things}${things:+,}${1}"

P.P.S. How I would have done it:

tr '\n' ',' < input.txt | sed 's!,$!\n!'
Improve this answer

4 Comments

Why in the world are you creating a subshell, and thus throwing away the constructed variable? If you want to use grouping to scope the redirection, just use { ...; }, not ( ).
"Why in the world are you creating a subshell?" - Because it is simple enough to expain. And because I can do it by adding only two characters. And as a brute force tool, subshell is cleaner and straightforward.
{ and } are also two characters, and they allow "$things" to be used after the entire code segment is complete. Though why you'd redirect into the whole block rather than only the while loop is something I'm unclear on.
"why you'd redirect into the whole block" - why not? Inless I optimize for performance or I really need the value as variable, I rarely put data into variables. Sometimes I simply pipe things from one subshell to another, like ( ... ) | ( ... ) | ( ... ) which is a convenient paradigm for scalable batch data handling. People these days put too much data into the shell variables, instead of lettimg them flow through the pipes.
0

You can do this too:

#!/bin/bash
while read -r i
do
[[ $things == "" ]] && things="$i" || things="$things","$i"
done < <(grep . input.txt)
echo "$things"

Output:

House,Monkey,Car

N.B:

Used grep to tackle with empty lines and the probability of not having a new line at the end of file. (Normal while read will fail to read the last line if there is no newline at the end of file.)

Improve this answer

5 Comments

arr=$(cat input.txt) is misleading -- that's a string variable, not an array (and the more efficient way to read it would be arr=$(<input.txt), if you were going to read it as a string in that manner). Also, see mywiki.wooledge.org/DontReadLinesWithFor
I'd also suggest checking what this code does when input.txt contains a line with only a literal *.
BTW, the grep is overkill. while read -r i || [[ $i ]]; do is a much more efficient way to solve handling of incomplete lines. (To be clear, a file lacking a trailing newline on its last line is not a valid UNIX text file, so failing to read that last line is not necessarily an error).
It's just a precaution, even though it's a overkill. Not all the time the text file ends up with a trailing new line.
Sure. But if you're going to take the precaution, it's still better to take the version of that precaution that doesn't require an extra fork/exec and pipeline element (thus, extra read() and write() calls for all content processed).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.