Re: [go-nuts] NonBlocking IO on STDOUT Streams

[Ian Lance Taylor]

On Thu, Jul 19, 2012 at 11:05 AM, Matthew Kanwisher <ma...@errplane.com> wrote:
> Having a situation where I can't easily do non-blocking io on a bufferedio
> object coming from STDOUT of another program. All the non blocking methods
> don't see to work properly with STDIO streams. Any suggestions?
>
> stdout, err := cmd.StdoutPipe()
> if err != nil {
> log.Fatal(err)
> }
> err = nil
> reader := bufio.NewReader(stdout)
> sbuffer := ""
> lines := 0
> for ; err == nil; {
> s,err := reader.ReadString('\n') //This line is always blocking
> ...

Use a goroutine that does blocking reads and sends the data to a channel.

Ian

[Matthew Kanwisher]

Yeah so I do that right now, however that goroutine can never die if there is never any future io. The problem I run into is I have these abandoned  go-routines that I have no way of killing and they won't end. There seems like there is no safe way to kill a goroutine.

[Matt Kane's Brain]

On Thu, Jul 19, 2012 at 2:24 PM, Matthew Kanwisher <ma...@errplane.com> wrote:
> Yeah so I do that right now, however that goroutine can never die if there
> is never any future io. The problem I run into is I have these abandoned
> go-routines that I have no way of killing and they won't end. There seems
> like there is no safe way to kill a goroutine.

Give each goroutine a channel on which you can write a quit message?

for {
select {
case byte := <- stdout:
doStuff()
case <- quit:
return
}
}

--
matt kane's brain
http://hydrogenproject.com

[André Moraes]

On Thu, Jul 19, 2012 at 3:24 PM, Matthew Kanwisher <ma...@errplane.com> wrote:
> Yeah so I do that right now, however that goroutine can never die if there
> is never any future io. The problem I run into is I have these abandoned
> go-routines that I have no way of killing and they won't end. There seems
> like there is no safe way to kill a goroutine.

Do you want to stop reading stdin when there is no data in it?

The goroutine solution works, since you can check for erros and end your loop

buff := make([]byte,1024)
for {
n, err := reader.Read(buff)
if err != nil {
break
}
outChan <- buff
}

But I don't know how reading from stdin would cause a error.

--
André Moraes
http://amoraes.info

[Matthew Kanwisher]

Problem is even a read blocks on an STDOUT stream it seems, conceptually I want to do something like this

  cmd := exec.Command("tail", "-f", "/tmp/matt.txt")

  stdout, err := cmd.StdoutPipe()

  if err != nil {

      log.Fatal(err)

  }

  if err := cmd.Start(); err != nil {

      log.Fatal(err)

  }

  err  = nil

  

  buff := make([]byte,1024)

  for {

    select {

    case n, _ stdout.Read(buff):             This is the line that I don't get how to handle, is there a way I can do a blocking call like this on a select?

      if n < 1  {

        fmt.Printf(" ")

        break

      } else {

        fmt.Printf("0")

        fmt.Sprintf("found data -%s", n)

      }

    case <- quit:

        return

[Ian Lance Taylor]

On Thu, Jul 19, 2012 at 1:55 PM, Matthew Kanwisher <ma...@errplane.com> wrote:
> Problem is even a read blocks on an STDOUT stream it seems, conceptually I
> want to do something like this
>
> cmd := exec.Command("tail", "-f", "/tmp/matt.txt")
> stdout, err := cmd.StdoutPipe()
> if err != nil {
> log.Fatal(err)
> }
> if err := cmd.Start(); err != nil {
> log.Fatal(err)
> }
>
> err = nil
>
> buff := make([]byte,1024)
> for {
> select {
> case n, _ stdout.Read(buff): This is the line that I don't
> get how to handle, is there a way I can do a blocking call like this on a
> select?
> if n < 1 {
> fmt.Printf(" ")
> break
> } else {
> fmt.Printf("0")
> fmt.Sprintf("found data -%s", n)
> }
> case <- quit:
> return
> }
> }
>

Reading from stdout is kind of odd.

In any case, no. The select statement works on channels. Using a
goroutine to move data from the file to the channel is the way to go.
Do you expect to have many of these goroutines? Do you expect them to
be reading from an input sources that is never closed? If so, you may
need to use a timeout and a quit channel, or something like that. In
typical cases this is not an issue.

Ian

[Matthew Campbell]

So basically we are writing a server similar to nagios/monit in go. So we spawn a lot of sub processes to gather system stats or allow plugins. In unix fashion we string together utilities using unix pipes. Reading from STDOUT is not really an odd behavior, it seems golang is a bit immature in areas like spawning subprocesses and monitoring their output. I'll try digging into the source and see why it blocks. 

Matthew Kanwisher

http://errplane.com

[Ian Lance Taylor]

On Thu, Jul 19, 2012 at 2:05 PM, Matthew Campbell <ma...@errplane.com> wrote:
> So basically we are writing a server similar to nagios/monit in go. So we
> spawn a lot of sub processes to gather system stats or allow plugins. In
> unix fashion we string together utilities using unix pipes. Reading from
> STDOUT is not really an odd behavior, it seems golang is a bit immature in
> areas like spawning subprocesses and monitoring their output. I'll try
> digging into the source and see why it blocks.

It's no mystery why it blocks. All I/O in the Go library is blocking.
That's because goroutines and channels provide another approach to
permitting a program to run while waiting for I/O, an approach that is
normally easier to work with.

If you are stringing together programs with pipes, then it is cheap to
have goroutines reading from standard input and feeding the results
into a channel. The goroutine will quit when the standard input is
closed.

(Reading from stdout still seems quite odd to me, pipes or not. Pipe
descriptors are unidirectional.)

Ian

[Matthew Kanwisher]

We'll if you read from common unix utils, top, tail, etc your going to need to read from stdout. Yeah most of the time this behavior works fine, the one case we were having problems was with "tail -f" which will block forever. We were using the unix tail since their are no implementations of tail in go, and its not a simple library to create considering you need to be handle a lot of corner cases with files getting truncated and rotated.

~Matt
http://errplane.com

[Cole Mickens]

If you're worried about it blocking, you could kill the underlying process. Once it's running something else could do cmd.Process.Kill(). That would presumably stop the read and prevent a forever block in the case of tail -f.

[Matthew Kanwisher]

Haha awesome, I was thinking that earlier, I thought that was a bit ghetto but it seems like its a very workable solution. Thanks a bunch!

[DisposaBoy]

On Friday, July 20, 2012 12:29:45 AM UTC+1, Matthew Kanwisher wrote:

Haha awesome, I was thinking that earlier, I thought that was a bit ghetto but it seems like its a very workable solution. Thanks a bunch

You could probably also just close the pipe thus causing both sides to end gracefully. 

 

[Dmitry Maluka]

On 07/19/2012 11:59 PM, Ian Lance Taylor wrote:
> Reading from stdout is kind of odd.

It's just a terminology confusion: Matthew means stdout of the program
on the other end of the pipe, which is stdin of his program.

[Uriel]

On Thu, Jul 19, 2012 at 11:05 PM, Matthew Campbell <ma...@errplane.com> wrote:
> So basically we are writing a server similar to nagios/monit in go. So we
> spawn a lot of sub processes to gather system stats or allow plugins. In
> unix fashion we string together utilities using unix pipes.

Blocking is the Unix way of doing IO. Go's model fits with it very nicely.

People have to re-learn that blocking is good. Code that blocks is
easy to reason about and understand.

Uriel

[Matthew Kanwisher]

Uriel you seem to make a comment without reading the code in the post. I'm spawning child processes of tail, that block forever. I'm trying to find a way to cleanly kill the goroutine. If you have a blocking call in a goroutine it can be stuck forever is my problem. 

Dmitry this isn't a piped program, so its not my stdin, its the stdout of a sub process I am spawning.

~Matt

[Dmitry Maluka]

On 07/20/2012 04:54 PM, Matthew Kanwisher wrote:
> Dmitry this isn't a piped program, so its not my stdin, its the stdout
> of a sub process I am spawning.

Yep, you are right. Sorry, I meant not stdin but the descriptor returned
by os.Pipe(). I just pointed out that your program is not reading its
own stdout.

[Jesse McNelis]

On Fri, Jul 20, 2012 at 11:54 PM, Matthew Kanwisher <ma...@errplane.com> wrote:
> Uriel you seem to make a comment without reading the code in the post. I'm
> spawning child processes of tail, that block forever. I'm trying to find a
> way to cleanly kill the goroutine. If you have a blocking call in a
> goroutine it can be stuck forever is my problem.

Killing the goroutine is wrong since the process sending it data
hasn't stopped sending it data. It's very strange to stop receiving
data when there is more being sent.

If you stop the sending, then the receiver stops automatically since
there is nothing more to receive.






--
=====================
http://jessta.id.au

[Matthew Kanwisher]

I'm not killing the go routine, I'm killing the underlying "tail -f" process that I spawned. Basically the problem is when reading from an STDOUT stream of a spawned process there is no way to timeout so there is no way for the goroutine to get new messages to exit or do other things.

Matt

http://errplane.com

[Mike Rosset]

Matt, if you are stopping the process that should produce a io.EOF.
which you can check for.

n, err := stdout.Read(buff)
if err == io.EOF {
break, or w/e stop reading logic here.
}

Mike