Trying to pipe result to both log file and stdout

[Theeraphol Wattanavekin]

Hi,

I am trying to run command execution and pipe its output to stdout and log file. I want to make sure that the command execution won't finish before writer write to log file and stdout by using sync.WaitGroup. The problem is I got "fatal error: all goroutines are asleep - deadlock!" and following is my code:

package main

import (

"bufio"

"fmt"

"io"

"os"

"os/exec"

"sync"

"time"

)

func main() {

layout := "20060102_1504"

err := os.MkdirAll("/tmp/test", 0755)

if err != nil {

fmt.Println("Error making folder")

fmt.Println(err)

return

}

logFile, err := os.Create("/tmp/test/" + time.Now().Format(layout) + ".log")

if err != nil {

fmt.Println("Cannot create logfile")

fmt.Println(err)

return

}

// declare writer for logfile

wLogFile := bufio.NewWriter(logFile)

fmt.Println("Starting to run echo")

cmd := exec.Command("echo", "HELLO")

// create pipe to use reader to write to stdout and logfile in another goroutine

r, w := io.Pipe()

cmd.Stderr = w

cmd.Stdout = w

if err := cmd.Start(); err != nil {

fmt.Println(err)

return

}

var wgWriting sync.WaitGroup

wgWriting.Add(1)

go func(reader io.Reader) {

scanner := bufio.NewScanner(reader)

for scanner.Scan() {

// writing to stdout

fmt.Println(scanner.Text())

// writing to logfile

if _, err := wLogFile.WriteString(scanner.Text() + "\n"); err != nil {

fmt.Println(err)

}

if err = wLogFile.Flush(); err != nil {

fmt.Println(err)

}

}

if err := scanner.Err(); err != nil {

fmt.Println(err)

}

logFile.Close()

wgWriting.Done()

fmt.Println("END")

}(r)

cmd.Wait()

wgWriting.Wait()

fmt.Println("Finished running")

}

Also, just in case, I also put in Go Playground as follow:

http://play.golang.org/p/W022pCxkNX

Thanks,

Theeraphol

[Rui Ueyama]

scanner.Scan() would block to wait for io.EOF. If you directly assign values to cmd.Stdout or cmd.Stderr, the call of cmd.Wait does *not* seem to close them. In order to send an io.EOF to the other end of the pipe, you need to close w by yourself after cmd.Wait.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Theeraphol Wattanavekin]

I have tried again by deleting all writing to file and stdout and left only goroutine and pipe. Also, changed from sync.Waitgroup to channel.  It still give me the same error. Please see my updated code below:

package main

import (

"fmt"

"io"

"os/exec"

)

func main() {

fmt.Println("Starting to run echo")

cmd := exec.Command("echo", "HELLO")

// create pipe to use reader to write to stdout and logfile in another goroutine

r, w := io.Pipe()

cmd.Stderr = w

cmd.Stdout = w

if err := cmd.Start(); err != nil {

fmt.Println(err)

return

}

done := make(chan error, 1)

go func(reader io.Reader) {

<-done

fmt.Println("END")

}(r)

done <- cmd.Wait()

fmt.Println("Finished running")

}

[Rui Ueyama]

You have made a different issue in the stripped down version.

Now nobody reads from r, so echo command would block when it tries to write the string "HELLO" to stdout, which is connected to r. As a result Wait does not return.

If you add ioutil.ReadAll(r) before <-done, it'll unblock cmd.Wait.

The error message that the runtime prints when it detects deadlock contains line numbers where your program stopped. Take a look at them carefully.

[Theeraphol Wattanavekin]

Thank you for your reply.

I did what you said and it solved the error. 
Follow is my updated code:

package main

import (

"fmt"

"io"

"os/exec"

"sync"

)

func main() {

fmt.Println("Starting to run echo")

cmd := exec.Command("echo", "HELLO")

// create pipe to use reader to write to stdout and logfile in another goroutine

_, w := io.Pipe()

cmd.Stderr = w

cmd.Stdout = w

if err := cmd.Start(); err != nil {

fmt.Println(err)

return

}

var wg sync.WaitGroup

wg.Add(1)

go func() {

fmt.Println("END")

defer func() {

wg.Done()

w.Close()

}()

}()

cmd.Wait()

wg.Wait()

fmt.Println("Finished running")

}

http://play.golang.org/p/ZcL5avQpcR

I am still wonder why don't cmd.Wait close them?

Thanks,

Theeraphol

[Theeraphol Wattanavekin]

The update on my gmail is too slow. So, I didn't catch up your 1st and 2nd replies before I sent the updates.

Anyway, I think you are right. I think it should be resolved. I will look at it carefully.

[Theeraphol Wattanavekin]

The changes you told me works good.

Btw, any reason why cmd.Wait does not close them if I directly assign values to cmd.Stdout and cmd.Stderr?

On Friday, September 12, 2014 12:55:58 PM UTC+9, Rui Ueyama wrote:

[Carlos Castillo]

On Thursday, September 11, 2014 9:55:26 PM UTC-7, Theeraphol Wattanavekin wrote:

The changes you told me works good.

Btw, any reason why cmd.Wait does not close them if I directly assign values to cmd.Stdout and cmd.Stderr?

Programs generally don't close stdout & stderr in regular use. Also, if you wanted to write the contents of multiple commands to the same streams you couldn't. 

Generally it's bad form in an API if a function accepts a reader / writer and closes it behind the caller's back.

[roger peppe]

You don't need a pipe or an extra goroutine at all.
The exec package can do all the work for you:

http://play.golang.org/p/fCLCzeV4oN


On 12 September 2014 04:40, Theeraphol Wattanavekin

[Theeraphol Wattanavekin]

@Carlos 

I think that makes sense.

@Rog 

Thank you very much! This is exactly what I am looking for!