0

I'm doing something along these lines:

declare -Ft handle_format_output &>/dev/null && exit 1   # test if this function name is already present in this scope
handle_format_output() {
    case $1 in
    domain) ./scripts/idn_to_punycode.pl >>"${2}_${1}.txt" ;;
    ipv4)
        # read doesn't bother w/ pipe input as its undefined: guessing it's leading to pipe breaks
        # therefore open stdin as a separate file and read from it to close the previous pipe
        while IFS= read -r line <&3; do
            case $line in
            */*) printf "%s\n" "$line" >>"${2}_${1}_cidr.txt" ;; # cidr block
            *-*) ipcalc "$line" >>"${2}_${1}_cidr.txt" ;;        # deaggregate ip range
            *) printf "%s\n" "$line" >>"${2}_${1}.txt" ;;        # ip address
            esac
        done 3< /dev/stdin
        ;;
    ipv6) cat -s >>"${2}_${1}.txt" ;;
    esac
}

...  |
  mawk '!seen[$0]++' | # filter duplicates and blank lines
  handle_format_output "$format" "$color"

Where the input is linear text that's either a web domain, IPv4 address, IPv4 CIDR block, or IPv6 address. The format is either "domain," "ipv4," or "ipv6." The color is either "white" or "black."

This error keeps being thrown no matter what I'm trying:

mawk: write failure (Broken pipe)
mawk: close failed on file /dev/stdout (Broken pipe)
Error: Process completed with exit code 2.

What am I doing wrong?

Improve this question
4
  • What are the values of $format and $color? (please double check these and mention them in the question). Commented Jul 3, 2021 at 15:36
  • @Kusalananda I've added those details in the question Commented Jul 3, 2021 at 15:39
  • Could you please confirm that $format is actually exactly the string ipv4 and nothing else (no trailing spaces or newline etc.)? You seem to think there's something wrong with your read call, so the first step is to make sure that the code actually gets there. If $format is a string that is not one of the three that you test for, the code will fall through that case statement and you'll get the behavior that you describe. You may therefore want to add a *) case that just prints error! or something, for debugging. Commented Jul 3, 2021 at 16:36
  • @Kusalananda If anything were different or more relevant it would've been documented. Commented Jul 3, 2021 at 17:23

1 Answer 1

Reset to default
0

I figured it out! Some of the format data fields were "ivp4" instead of "ipv4." I'm now wondering why there have been successful builds this long then, but after fixing them regular piping now works.

Improve this answer
7
  • There is literally no difference between the code in your answer and the code in you question. In both instances, your function reads the data from its standard input. The input is not "completely processed first" in your answer's code. It's redirected from a named pipe, which is effectively the same as piping it into the function. You obviously changed something else in the code or in the input data that made it work. Commented Jul 3, 2021 at 17:18
  • @Kusalananda I've changed nothing other than this line, and no error is thrown. From what I understand, this version waits until all the text is processed from the other commands, and passes the final output into the function. Before, there was still some processing going on, which is why the pipe broke when processing the output. Commented Jul 3, 2021 at 17:22
  • I don't really buy that as a reason unless there were some timeout while reading the data. Your code shows nothing that should be sensitive to timing out or reading lines with slight delays. Commented Jul 3, 2021 at 19:57
  • @Kusalananda Here's literally all I've changed if you want the full context: github.com/T145/the-blacklist/commit/… After that change everything built perfectly, so if there's any way to do things better I'd love to learn it. Commented Jul 3, 2021 at 20:13
  • Try comparing { while true; do echo hello; sleep 1; done; } | while read; do echo "$REPLY"; done and while read; do echo "$REPLY"; done < <( while true; do echo hello; sleep 1; done ) and note how the second of these does not finish processing before sending the data to the read (the output is the same in both example, the string hello is repeatedly printed with a delay). It is not clear that you are providing the new code with the same input as the old code. Commented Jul 3, 2021 at 20:50

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.