Streaming commands output progress from Goroutine

Question

Streaming commands output progress question addresses the problem of printing progress of a long running command.

I tried to put the printing code within a goroutine but the scanner claims to have already hit the EOF immediately and the for block is never executed.

The bufio.scan code that gets executed on the first execution of the Scan() method is:

    // We cannot generate a token with what we are holding.
    // If we've already hit EOF or an I/O error, we are done.
    if s.err != nil {
        // Shut it down.
        s.start = 0
        s.end = 0
        return false
    }

And if I print s.err the output is EOF.

The code I'm trying to run is:

cmd := exec.Command("some", "command")
c := make(chan int, 1)

go func(cmd *exec.Cmd, c chan int) {
    stdout, _ := cmd.StdoutPipe()

    <-c

    scanner := bufio.NewScanner(stdout)
    for scanner.Scan() {
        m := scanner.Text()
        fmt.Println(m)
    }
}(cmd, c)

cmd.Start()

c <- 1

cmd.Wait()

The idea is to start the Goroutine, get a hold of the cmd.stdout, wait that the cmd is started, and start processing its output.

The result is that the long command gets executed and the program waits for its completion, but nothing is printed to terminal.

Any idea why by the time scanner.Scan() is invoked for the first time the stdout has already reached EOF?

score 6 · Accepted Answer · 2016-08-10 11:54:23Z

There are some problems:

The pipe is being closed before reading all data.
Always check for errors
Start cmd.Start() after c <- struct{}{} and use unbuffered channel c := make(chan struct{})

Two working sample codes:

1: Wait using channel then close the pipe after EOF using defer func() { c <- struct{}{} }(), like this working sample code:

package main

import (
    "bufio"
    "fmt"
    "os/exec"
)

func main() {
    cmd := exec.Command("Streamer")
    c := make(chan struct{})

    go run(cmd, c)

    c <- struct{}{}
    cmd.Start()

    <-c
    if err := cmd.Wait(); err != nil {
        fmt.Println(err)
    }
    fmt.Println("done.")
}

func run(cmd *exec.Cmd, c chan struct{}) {
    defer func() { c <- struct{}{} }()
    stdout, err := cmd.StdoutPipe()
    if err != nil {
        panic(err)
    }
    <-c
    scanner := bufio.NewScanner(stdout)
    for scanner.Scan() {
        m := scanner.Text()
        fmt.Println(m)
    }
    fmt.Println("EOF")
}

2: Also you may Wait using sync.WaitGroup, like this working sample code:

package main

import (
    "bufio"
    "fmt"
    "os/exec"
    "sync"
)

var wg sync.WaitGroup

func main() {
    cmd := exec.Command("Streamer")
    c := make(chan struct{})
    wg.Add(1)
    go func(cmd *exec.Cmd, c chan struct{}) {
        defer wg.Done()
        stdout, err := cmd.StdoutPipe()
        if err != nil {
            panic(err)
        }
        <-c
        scanner := bufio.NewScanner(stdout)
        for scanner.Scan() {
            m := scanner.Text()
            fmt.Println(m)
        }
    }(cmd, c)

    c <- struct{}{}
    cmd.Start()

    wg.Wait()
    fmt.Println("done.")
}

And Streamer sample code (just for testing):

package main

import "fmt"
import "time"

func main() {
    for i := 0; i < 10; i++ {
        time.Sleep(1 * time.Second)
        fmt.Println(i, ":", time.Now().UTC())
    }
}

And see func (c *Cmd) StdoutPipe() (io.ReadCloser, error) Docs:

StdoutPipe returns a pipe that will be connected to the command's standard output when the command starts.

Wait will close the pipe after seeing the command exit, so most callers need not close the pipe themselves; however, an implication is that it is incorrect to call Wait before all reads from the pipe have completed. For the same reason, it is incorrect to call Run when using StdoutPipe. See the example for idiomatic usage.

Thanks, 1) works fine. If I want to process Stdout and Stderr at the same time, I guess I can just use 2 channels and 2 go routines (one per pipe type).

jussius · Accepted Answer · 2016-08-10 10:50:44Z

From godocs:

StdoutPipe returns a pipe that will be connected to the command's standard output when the command starts.

Wait will close the pipe after seeing the command exit, so most callers need not close the pipe themselves; however, an implication is that it is incorrect to call Wait before all reads from the pipe have completed.

You are calling Wait() immediately after starting the command. So the pipe gets closed as soon as the command completes, before making sure you have read all the data from the pipe. Try moving Wait() to your go routine after the scan loop.

go func(cmd *exec.Cmd, c chan int) {
    stdout, _ := cmd.StdoutPipe()

    <-c

    scanner := bufio.NewScanner(stdout)
    for scanner.Scan() {
        m := scanner.Text()
        fmt.Println(m)
    }

    cmd.Wait()
    c <- 1
}(cmd, c)

cmd.Start()
c <- 1

// This is here so we don't exit the program early,
<-c

There's also a simpler way to do things, which is to just assign os.stdout as the cmd's stdout, causing the command to directly write to the os.stdout:

cmd := exec.Command("some", "command")
cmd.Stdout = os.Stdout
cmd.Run()

Collectives™ on Stack Overflow

Streaming commands output progress from Goroutine

2 Answers 2

There are some problems:

Two working sample codes:

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

There are some problems:

Two working sample codes:

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related