Fwd: cmd.Wait & io.Pipe & SIGKILL

[Salvatore Domenick Desiano]

I'm hoping y'all can help me get the correct semantics for code that sets up an external process, replaces Stdin, Stderr, Stdout, runs the command, calls exec.Command.Wait() and handles SIGKILL correctly.

I've read more than two dozen threads, PRs, and bug reports about how exec.Command interacts with io.Pipe and SIGKILL/cmd.Kill(). I've seen comments from Ian that some problems can't be solved and a few regrets about the current design but nothing that describes a working approach.

What I want is a function that does this:

ctx, cancelCmd := context.WithCancel(context.Background())

cmd := exec.CommandContext(ctx, cmdName, cmdArgs...)

cmd.Stdin = stdinPipe

cmd.Stdout = stdoutPipe

cmd.Stderr = stderrPipe

if err := cmd.Run(); err != nil {

g.log.Warnf("Run error: %w", err)

}

return nil

and that returns when the command exits, gets an (externally issued) SIGKILL, or gets a cancelCmd(). Every combination I've tried hangs because the pipes aren't closed. Although the pipes are monitored and drained (not shown), that doesn't help because the command doesn't get a chance to close them.

What am I missing? Do I need a goroutine to watch the process table?

Thank you!

-- Salvatore

smile.

[Ian Lance Taylor]

First I'll ask what is keeping the pipes open? if the child process
is killed, the child's copy of the pipes will be closed. Is the child
passing them along to some other process? Or is the parent process
keeping them open? Can either of those be changed?

That said, Go 1.20 added new Cancel and WaitDelay fields to
os/exec.Cmd. These fields may be used to control how the parent
process should behave when waiting for a child with open pipes.

Ian

[Salvatore Domenick Desiano]

I hope this isn't a red herring but in constructing a self-contained example I triggered a deadlock. Perhaps the deadlock is the answer to my question or perhaps it is another issue. Either way this code does not seem malformed:

func main() {

cmdName := "/bin/bash"

cmdArgs := []string{

"-c",

"while true; do echo step; sleep 1; done",

}

cmdStdinR, cmdStdinW := io.Pipe()

cmdStdoutR, cmdStdoutW := io.Pipe()

cmdStderrR, cmdStderrW := io.Pipe()

ctx, cancelCmd := context.WithCancel(context.Background())

cmd := exec.CommandContext(ctx, cmdName, cmdArgs...)

cmd.Stdin = cmdStdinR

cmd.Stdout = cmdStdoutW

cmd.Stderr = cmdStderrW

if err := cmd.Start(); err != nil {

fmt.Println(err)

}

cmdStdinW.Close()
if err := cmd.Wait(); err != nil {

fmt.Println(err)

}

fmt.Println("DONE")

cancelCmd = cancelCmd

cmdStdoutR = cmdStdoutR

cmdStderrR = cmdStderrR

}

If you run this code and SIGKILL the bash process, go flags it as a deadlock and panics. FWIW, this also happens if there are goroutines monitoring cmdStdoutR and cmdStderrR.

What am I missing?

-- Salvatore

smile.

[Salvatore Domenick Desiano]

Simpler self-contained example, same result:

func main() {

cmdName := "/bin/bash"

cmdArgs := []string{"-c", "while true; do echo step; sleep 1; done"}

cmdStdoutR, cmdStdoutW := io.Pipe()

cmd := exec.Command(cmdName, cmdArgs...)

cmd.Stdout = cmdStdoutW

if err := cmd.Run(); err != nil {

fmt.Println(err)

}

fmt.Println("DONE")

cmdStdoutR = cmdStdoutR

}

For the sake of completeness I'll mention that this runs fine if cmd.Stdout is not assigned, which points to exec.awaitGoroutines() as being the culprit.

FWIW, I have no direct evidence that the pipes aren't being closed on the SIGKILL. That said, the (exec-internal) goroutines are hung on io.Copy(). I expect that if the process's output pipe was actually closed the io.Copy() would return, the goroutine would complete, and the deadlock would not happen.

-- Salvatore

smile.

[Robert Engels]

It looks like you don’t have anything reading from the pipe so it’s going to hang. 

On Jun 16, 2024, at 8:54 PM, Salvatore Domenick Desiano <near...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEQs7S-Gwuvh8MJ48Nv1Cwd7miHiUX4RdwKjqmeLqMBcWYYgJQ%40mail.gmail.com.

[Robert Engels]

It hangs because cmd.Run() waits for the process to complete - and with nothing reading the pipe it will never complete. 

On Jun 16, 2024, at 9:33 PM, Robert Engels <ren...@ix.netcom.com> wrote:



[Salvatore Domenick Desiano]

Ah!

I apologize... in reducing it to a compact example I didn't notice that I changed the semantics of what the code was doing. You are, of course, right about the cause of the deadlock in this code.

That said, working backward from the self-contained example to my full code clarified the actual issue I'm debugging. It is sufficiently different from what I said here that I will start a new thread if I can't figure it out myself.

Thank you!

-- Salvatore

smile.

[Harri L]

I’m wondering if there’s a specific reason you’re using io.Pipe instead of the Cmd.StdoutPipe helper? When using io.Pipe you need to Close the pipe by yourself, but when using the Cmd.StdoutPipe, the Cmd.Wait will close the pipe for you.

Here’s a sample following your previous example:

func main() { defer err2.Catch() cmdName := "/bin/sh" cmdArgs := []string{ "-c", "for i in 1 2 3 4 5 6 7 8 9 10; do echo step $i; sleep 0.1; done", } cmd := exec.CommandContext(context.Background(), cmdName, cmdArgs...) stdoutPipe := try.To1(cmd.StdoutPipe()) var wg sync.WaitGroup go readPipe(stdoutPipe, &wg) try.To(cmd.Start()) wg.Wait() try.To(cmd.Wait()) fmt.Println("DONE") } func readPipe(src io.Reader, wg *sync.WaitGroup) { defer err2.Catch() wg.Add(1) defer wg.Done() buf := make([]byte, 256) for eof, n := try.IsEOF1(src.Read(buf)); !eof; eof, n = try.IsEOF1(src.Read(buf)) { try.To1(os.Stdout.Write(buf[:n])) } }

And here’s a sample using io.Pipe; please see the noted line:

func main() { defer err2.Catch() cmdName := "/bin/sh" cmdArgs := []string{ "-c", "for i in 1 2 3 4 5 6 7 8 9 10; do echo step $i; sleep 0.1; done", } cmdStdoutR, cmdStdoutW := io.Pipe() cmd := exec.CommandContext(context.Background(), cmdName, cmdArgs...) cmd.Stdout = cmdStdoutW var wg sync.WaitGroup go readPipe(cmdStdoutR, &wg) try.To(cmd.Run()) cmdStdoutW.Close() // NOTE: needed for io.Pipe ↓ fmt.Println("DONE") wg.Wait() }

​

[Salvatore Domenick Desiano]

There might be. Or I might be holding it wrong. I'm about to post another thread with a different angle on this. This thread was mis-framed.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/782a2013-703d-451c-9567-7be585d17c5fn%40googlegroups.com.