Using the select statement more like sys/select(3) for named pipe I/O?

[Daniel Bryan]

I have a go program that opens a large collection of named pipes at their read end. It has to read data from all of these, do some internal processing, and output a single stream of data.

Since reading from (or opening) a named pipe blocks the thread until it's written to (or opened at the other end), I have a goroutine for each named pipe. The goroutines read lines from their corresponding named pipe, do a bit of buffering, and eventually send data on a shared channel.

The main goroutine reads data from this channel, processes it and outputs it until all the goroutines signal that their pipes have closed.

It's a reasonable design, but the code to set up this infrastructure is quite verbose and there's a compromise between:

• not wanting to block the main goroutine on any operations on a named pipe, and
• wanting to effectively handle errors found when trying to operate on these named pipes

Is there a different approach available? What I really want conceptually is a select loop, but with reads from an io.ReadCloser, instead of channel receives / sends. Is this possible at all, without having to have a goroutine handling the work on each io.ReadCloser, so that the main select loop can work with channels? Or is my current design the go way?

Thanks,

[Kyle Lemons]

On Thu, Feb 21, 2013 at 3:11 PM, Daniel Bryan <danb...@gmail.com> wrote:

I have a go program that opens a large collection of named pipes at their read end. It has to read data from all of these, do some internal processing, and output a single stream of data.

Since reading from (or opening) a named pipe blocks the thread until it's written to (or opened at the other end), I have a goroutine for each named pipe. The goroutines read lines from their corresponding named pipe, do a bit of buffering, and eventually send data on a shared channel.

Sounds good.

 

The main goroutine reads data from this channel, processes it and outputs it until all the goroutines signal that their pipes have closed.

Using a WaitGroup, presumably.

 

It's a reasonable design, but the code to set up this infrastructure is quite verbose and there's a compromise between:

• not wanting to block the main goroutine on any operations on a named pipe, and
• wanting to effectively handle errors found when trying to operate on these named pipes

I guess I'm not seeing what is particularly verbose about this.  It actually seems like it would be quite concise.

// warning: pseudocode; types and functions used for illustration purposes only

func ManagePipe(file, outchan) {

  p := openPipe(file)

  defer p.Close()

  for {

    line := readLine(p)

    outChan <- line

  }

}

func ManagePipes([]file) {

  var wg WaitGroup

  ch := make(chan data)

  for _, f := range files {

    wg.Add(1)

    go func(f file) {

      defer wg.Done()

      go ManagePipe(f, ch)

    }(f)

  }

  go func() {

    wg.Wait()

    close(ch)

  }()

  for data := range ch {

  }

}

Instead of the range, you could have a select against the data channel and an error channel and/or a timeout and/or a "stop" channel if you need those things.

 

Is there a different approach available? What I really want conceptually is a select loop, but with reads from an io.ReadCloser, instead of channel receives / sends. Is this possible at all, without having to have a goroutine handling the work on each io.ReadCloser, so that the main select loop can work with channels? Or is my current design the go way?

Thanks,

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.