How to track bash script - exit status and why it failed

First step, encapsulate that script in a file rather than run it directly at the terminal (and lose the UUOC award while we're at it).

#!/bin/bash

{
awk 'NR%4 == 2 { print $1 }' file.txt | sort | uniq -c | sort -gr >>output.txt
} 2>error.log

This captures all the error messages in the file error.log. Then you can add diagnostic information.

#!/bin/bash

{
date >&2
set -x
awk 'NR%4 == 2 { print $1 }' file.txt | sort | uniq -c | sort -gr >>output.txt
date >&2
} 2>error.log

Now you've got the when information — when it was started and when it finished. Since you're in bash, you could arrange to capture the exit statuses of each process in the pipeline if you wanted to, so you'd know exactly which commands exited with which status. You may or may not get messages about which process was killed (if a process was killed by an external signal), but if the process dies of its own accord, it should print a message on standard error (that's what it is there for, and why it is crucial that errors are printed to standard error, not standard output).

With this script, you have standard output going to output.txt and standard error going to error.log; the script does not use standard input (the data comes from file.txt). So, you can run this with nohup or simply in the background with & without any qualms.

You may prefer to make the name file.txt into command line parameters; you may want to make the output and log files configurable. You might prefer a different format for the date output. All of that is tweakable. But the key point is to put it into a shell script so that you can handle such things straight-forwardly and systematically.

Use screen, pv and tee to capture all errors, have a progress bar, and allow restarting from the last successful command rather than from scratch upon error.

You can use screen (the multiplexer) rather than backgrounding your process. That way, you can always review its status and not miss error messages. Just type screen, run your command without the &, and hit CTRL-a,d. You can then logout. To later review the output, login (even remotely) and type screen -r.

Furhtermore, if you replace the initial cat with pv (the Pipe Viewer), you will have a progress bar telling you how much has been processed already:

pv -cN file.txt

and you will see something like

611MB 0:00:11 [58.3MB/s] [=>      ] 15% ETA 0:00:59

whenever you reattach the screen process.

Alternatively/complementarily, you can insert tee after a command to copy its output to a file before it is propagated to the next one.

uniq -c | tee afteruniq.tmpfile | sort -gr

The file afteruniq.tmpfile will contain the result of uniq -c. So you know what worked and what failed. You furthermore can resume the chain after the last successful step from the tee'd file.

You need to redirect both stderr and stdout to your output file.

Make your command like this:

( awk 'NR%4 == 2{print $1}' file.txt| sort | uniq -c | sort -gr ) >> output.txt 2>&1 &

Take note of 2>&1 to redirect stderr to wherever stdout is going.