Golang - Copy Exec output to Log

Question

I would like to redirect the output of a process to log in a timely manner. I can do it if I wait for the process to finish like this:

cmd := exec.Command("yes", "Go is awesome") // Prints "Go is awesome", forever 
out, err := cmd.CombinedOutput()
log.Printf("%s", out)

However, if the process takes a long time or doesn't finish this is less useful. I know I can write to stdout in real time like this:

cmd := exec.Command("yes")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.Run()

This doesn't really help me though, because I am writing a service that isn't writing to a terminal. I'm looking for something that will let me do something like:

cmd := exec.Command("yes")
cmd.Stdout = log.Stdout
cmd.Stderr = log.Stdout
cmd.Run()

log doesn't give direct access to its writer so this is not possible. Surely I'm not the only with this problem, how is this typically done?

This is caused by buffering in the command you are running. You might find the answers to this question useful. — Nick Craig-Wood
– Nick Craig-Wood, Commented Aug 7, 2014 at 19:50
@NickCraig-Wood While that is indeed a good question for explaining buffering, I'm fairly sure this is not a buffering problem. CombinedOutput() waits for the program to finish and returns the output and cmd.Stdout = os.Stdout does not exhibit the problem but writes to an undesired output. Thanks for the comment though... — Rick Smith
– Rick Smith, Commented Aug 7, 2014 at 20:03

SirDarius · Accepted Answer · 2014-08-08 08:17:35Z

37

You should use a pipe here, for example:

stdout, err := cmd.StdoutPipe()
if err != nil {
    return 0, err
}

// start the command after having set up the pipe
if err := cmd.Start(); err != nil {
    return 0, err
}

// read command's stdout line by line
in := bufio.NewScanner(stdout)

for in.Scan() {
    log.Printf(in.Text()) // write each line to your log, or anything you need
}
if err := in.Err(); err != nil {
    log.Printf("error: %s", err)
}

I have only handled Stdout here, but it is possible to handle Stderr at the same time, for example by using a goroutine.

edited Aug 8, 2014 at 8:17

answered Aug 7, 2014 at 20:19

SirDarius

43.2k8 gold badges92 silver badges106 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Rick Smith Over a year ago

Couple of suggestions: Get the StderrPipe() then do multi := io.MultiReader(stdout, stderr). Then do in := bufio.NewScanner(multi). This allows a single scanner and no goroutine which I think is quite nice. I think scanner.Err() should be in.Err(). I took out the return 0, err and just made it return err. Great answer, thanks!

Arcite Over a year ago

When I use this solution combined with the solution provided by @RickSmith the output is in reverse order (last line first, first line last). I think the for loop might need modified?

SirDarius Over a year ago

@Arcite there is no reason why a pipe might reverse the order of lines.This would mean that the program being piped runs backwards in time, which is pretty unlikely given our current knowledge in time travel technologies. More seriously, please ask a new question with your exact reproducible issue.

Elwinar · Accepted Answer · 2014-08-08 08:52:38Z

5

Both exec.Command and log.Logger are based on an io.Writer. You don't have access to those of the second, but you don't need to because either you didn't changed it, and you're using os.Stderr or you changed it and you have the io.Writer at hand when creating the logger.

So you just have to adapt you second example by using the right io.Writer…

Edit

After some though on it, I think you may be good to go by just embedding a log.Logger into a struct that will implement the io.Writer interface…

Example:

type LogWriter struct {
    logger *log.Logger
}

func NewLogWriter(l *log.Logger) *LogWriter {
    lw := &LogWriter{}
    lw.logger = l
    return lw
}

func (lw LogWriter) Write (p []byte) (n int, err error) {
    lw.logger.Println(p)
    return len(p), nil
}

And then pass it to your exec.Command output…

cmd := exec.Command(…)
cmd.Stdout = NewLogWriter(log.New(…))

You can use it for both standard or error output, or create new object for each one.

Edit2

As I'm going to use this trick for one of my next projects, I put it into a package on github. Feel free to give feedback, etc.

edited Aug 8, 2014 at 8:52

answered Aug 7, 2014 at 20:26

Elwinar

9,5791 gold badge35 silver badges40 bronze badges

4 Comments

SirDarius Over a year ago

Using the logger's Writer directly while using the Logger itself is not a good idea, because a Logger allows access from multiple goroutines without mangling its output, which a pure Writer does not. Also, in contrast to a Writer, a Logger normally decorates the output, which the OP probably desires, for example to have the exact date of the output in the log file.

Elwinar Over a year ago

@SirDarius: Updated my answer with an idea that popped while reading your comment. What do you think about that ?

SirDarius Over a year ago

Nice idea too. Encapsulating a logger into a Writer should be part of the standard library imho !

Elwinar Over a year ago

As it is not and as I'm thinking of using it in my next project (involve commands, happy coincidence), I put it into a package of my own on github. Feel free to use it, to give me feedback, etc. I will put it on godoc too when I will have the documentation done. Not that it's hard, thought…

Collectives™ on Stack Overflow

Golang - Copy Exec output to Log

2 Answers 2

3 Comments

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related