Writing a Unix filter using bash

Question

If a Unix/Linux command accepts its input data from the standard input and produces its output (result) on standard output is known as a filter.

The trivial filter is cat. It just copies stdin to stdout without any modification whatsoever.

How do I implement cat in bash? (neglecting the case that the command gets command line arguments)

I came up with

#! /bin/bash

while IFS="" read -r line
do
  echo -E "$line"
done

That seems to work in most cases, also for text files containing some binary bytes as long as they are not null bytes. However, if the last line does not end in a newline character, it will be missing from the output.

How can that be fixed?

I'm nearly sure this must have been answered before, but my searching skills don't seem to be good enough.

Obviously I don't want to re-implement cat in bash: It wouldn't work anyway because of the null byte problem. But I want to extend the basic loop to do some custom processing to certain lines of a text file. However, we have all seen text files without the final line feed, so I'd prefer if that case could be handled.

You're probably better off chaining existing filters (like sed and awk) in a pipeline rather than trying to implement something like this in bash. — larsks
– larsks, Commented Sep 18, 2017 at 14:41
It won't work for a binary file that contains null bytes; a shell variable simply cannot store null bytes. — chepner
– chepner, Commented Sep 18, 2017 at 14:46
@chepner: You are correct, I noted that. It's not a problem for my use case, because I know there will be no null bytes. Will edit the question to make that clear. — Uwe Geuder
– Uwe Geuder, Commented Sep 18, 2017 at 14:49
@larsks Yes, I'm quite sure the problem can be solved using awk. However the custom processing I need to do is rather complicated and a bash script would be easier to maintain. — Uwe Geuder
– Uwe Geuder, Commented Sep 18, 2017 at 14:59

chepner · Accepted Answer · 2017-09-18 14:51:22Z

5

Assuming you don't need to work with arbitrary binary files (since the shell cannot store null bytes in a variable), you can handle a file that isn't terminated by a newline by checking if line is not empty once the loop exits.

while IFS= read -r line; do
    printf '%s\n' "$line"
done
if [ -n "$line" ]; then
    printf '%s' "$line"
fi

In the loop, we output the newline that read stripped off. In the final if, we don't output a newline because $line would be empty if the last line read by the while loop had ended with one.

answered Sep 18, 2017 at 14:51

chepner

538k77 gold badges594 silver badges746 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Writing a Unix filter using bash

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related