Sending the output of a while loop to a bash function

Question

I've created a .bashrc file where I have two functions. One loops through the lines of a file in a while loop. I'm attempting to save the content of the lines if they match certain conditions and then pass all three matches to a second function that will then echo them. However, I've tried exporting variables and I've also tried piping to the second function, but neither works. Piping also acts very strangely as I'll try to illustrate in my code example.

readAndPipe() {
  while read -r line || [[ -n "$line" ]]; do
  (   
  if [[ $line == FIRSTNAME=BOB ]]; then
     echo $line;
  fi; 
  if [[ $line == LASTNAME=SMITH ]]; then
     echo $line;
  fi; 
  if [[ $line == BIRTHMONTH=AUGUST ]]; then
     echo $line;
  fi; 
  ); done < "file.txt" | printArguments $1 #pass the original command line argument;
}

printArguments() {
    #This is where the weirdness happens
    echo $@                   #Prints: only the original command line argument
    echo $#                   #Prints: 1
    echo $2 $3 $4             #Prints nothing
    varName=$(cat $2 $3 $4)
    echo $varName             #Prints: FIRSTNAME=BOB
                              #        LASTNAME=SMITH
                              #        BIRTHMONTH=AUGUST
    cat $2 $3 $4              #Prints nothing
    echo $(cat $2 $3 $4)      #Prints nothing
    cat $2 $3 $4 | tr "\n" '' #Prints tr: empty string2
}

Obviously I'm not a bash expert so I'm sure there's a lot of mistakes here, but what I'm wondering is

What are these seemingly magical $2 $3 $4 arguments that are not printed by echo but can be used by cat exactly once.
What is the correct way to save content during a while loop and pass it to another function so that I can echo it?

how are u calling the first function ???

Inder
– Inder

2018-09-20 20:57:59 +00:00
Commented Sep 20, 2018 at 20:57 — Inder
– Inder, Commented Sep 20, 2018 at 20:57
readAndPipe Hello

andrewpw
– andrewpw

2018-09-20 20:59:56 +00:00
Commented Sep 20, 2018 at 20:59 — andrewpw
– andrewpw, Commented Sep 20, 2018 at 20:59
I'm calling it from the directory that contains file.txt

andrewpw
– andrewpw

2018-09-20 21:00:55 +00:00
Commented Sep 20, 2018 at 21:00 — andrewpw
– andrewpw, Commented Sep 20, 2018 at 21:00

glenn jackman · Accepted Answer · 2018-09-20 21:34:57Z

1

$@, $*, $1, $2, etc are the arguments that are passed to the function. For example, in myfunc foo bar baz, we have $1 == foo, $2 == bar and $3 == baz.

When you pipe data to your function, you have to retrieve it from from stdin:

myfunc() {
    data=$(cat)
    echo "I received: >$data<"
}
for n in {1..5}; do echo "x=$n"; done | myfunc

produces

I received: >x=1
x=2
x=3
x=4
x=5<

varName=$(cat $2 $3 $4) works because $2 $3 and $4 are empty, so the shell sees this:
varName=$(cat )

The reason cat "only works once" is because you are consuming a stream. Once you consume it, it's gone. "you can't eat your cake and have it too."

The printArguments function can use the readarray command to grab the incoming lines into an array, instead of using cat to grab all the incoming text into a variable:

printArguments() {
    readarray -t lines
    echo "I have ${#lines[@]} lines"
    echo "they are:"
    printf ">>%s\n" "${lines[@]}"
}
{ echo foo; echo bar; echo baz; } | printArguments

outputs

I have 3 lines
they are:
>>foo
>>bar
>>baz

Learn more by typing help readarray at an interactive bash prompt.

edited Sep 20, 2018 at 21:34

answered Sep 20, 2018 at 21:02

glenn jackman

249k42 gold badges233 silver badges362 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

andrewpw Over a year ago

Thank you! That is super helpful for my understanding. So is cat the "correct" way to consume the stream? Sometimes saving cat and the dereferencing the variable later produces unexpected results. In some cases I've tried echo "Hello $varName, how are you" and it prints ", how are you" if $varName is empty, losing the Hello

andrewpw Over a year ago

Also, do you know how I could save these into 3 variables instead of echoing and piping them? Exporting them didn't work for me. I'm thinking there might be something using the > for stdout

glenn jackman Over a year ago

If $varName ends with a carriage return (\r), you'll see that effect of overwriting the text from the first column -- this typically happens with text files created on Windows (using CRLF (\r\n) line endings) and processed in *nix (expecting LF (\n) line endings).

glenn jackman Over a year ago

Answered your 2nd question in my answer. This is getting into some more advanced bash stuff (arrays). You'll want to look at a bash tutorial. Check out the "Learn more" link on the bash tag page.

KamilCuk · Accepted Answer · 2018-09-20 21:11:33Z

Imagine a script:

 func() {
       echo $#  # will print 2, func were executed with 2 arguments
       echo "$@"  # will print `arg1 arg2`, ie. the function arguments
       in=$(cat)   # will pass stdin to `cat` function and save cat's stdout into a variable
       echo "$in" # will print `1 2 3`
 }
 echo 1 2 3 | func arg1 arg2
 #                  ^^^^^^ function `func` arguments
 #          ^ passed one command stdout to other command stdin
 # ^^^ outputs `1 2 3` on process stdout

cat invoked without any arguments, reads stdin and outputs it on stdout.
Invoking a command in command substitution passes stdin along (ie. in=$(cat) will read stdin as normal cat just saves the output (ie. the cat's stdout) into a variable)

To your script:

readAndPipe() {
  # the while read line does not matter, but it outputs something on stdout
  while read -r line || [[ -n "$line" ]]; do
         echo print something 
  # the content of `file.txt` is passed as while read input
  done < "file.txt" | printArguments $1 # the `print something` (the while loop stdout output) is passed as stdin to the printArguments function
}

printArguments() {
    # here $# is equal to 1
    # $1 is equal to passed $1 (unless expanded, will get to that)
    # $2 $3 $4 expand to nothing
    varName=$(cat $2 $3 $4) # this executes varName=$(cat) as $2 $3 $4 expand to nothing
    # now after this point stdin has been read (it can be read once, it's a stream or pipe
    # is you execute `cat` again it will block (waiting for more input) or fail (will receive EOF - end of file)
    echo $varName             #Prints: `print something` as it was passed on stdin to this function
}

If the file file.txt contains only just:

FIRSTNAME=BOB
LASTNAME=SMITH
BIRTHMONTH=AUGUST

you can just load the file . file.txt or source file.txt. This will "load" the file, ie. make it part of your script, syntax is bash. So you can:

. file.txt
echo "$FIRSTNAME"
echo "$LASTNAME" 
echo "$BIRTHMONTH"

This is a common way of creating configuration files in /etc/ and then they are loaded by scripts. That's why in many /etc/ files comments start with #.

Notes:

Always enclose your variables. echo "$1" printArguments "$1" echo "$@" echo "$#" cat "$2" "$3" "$4" [ "$line" == ... ], a good read is here
Remove newlines with tr -d '\n'
A ( ) creates a subshell, which creates a new shell which has new variables and does not share variable with the parent, see here.

That's super cool! I didn't know about sourcing a file until you mentioned it. Thanks, that will work much better than my approach

Collectives™ on Stack Overflow

Sending the output of a while loop to a bash function

2 Answers 2

4 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related