how to organise code with channels and error handling?

[Jochen Voss]

Dear all,

I am writing a program which processes data in stages.  To structure the code cleanly, I would like implement each stage as an individual function and use channels to pass data between stages.

My questions:

- How to structure the code?  Is my approach described below reasonable?

- How to deal with errors?

Here is a sketch of I currently have (runnable version at https://go.dev/play/p/7Rrq-OLARl_R ):

func getInputs() <-chan t1 {

    c := make(chan t1)

    go func() {

        for ... {

            ...

            c <- x

        }

        close(c)

    }()

    return c

}

func process(in <-chan t1) <-chan t2 {

    c := make(chan t2)

    go func() {

    for i := range in {

        x, err := ...

        _ = err // what to do with the error?

        c <- x

    }

    close(c)

    }()

    return c

}

func main() {

    c1 := getInputs()

    c2 := process(c1)

    summarise(c2)

}

There are some things I like about this approach:

• the main function looks nice and tidy
• the stages of work are cleanly separated

But there are also things I don't like:

• I can't see an obvious way to handle errors in the stages.  In my case the first stage is locating input files on the file system, and there may be read errors etc.  The second stage processes the files, and files may have invalid contents.
• The code for the individual stages looks a bit strange with the double indentation from the outer func(){} and the go func() {}().

So my questions are:

• What would be a good way to incorporate error handling into my code?  I assume this will require one or more additional channels, just for errors.  Or is there a better way?
• Is there a better way to structure my code?

Many thanks,

Jochen

[Bryan C. Mills]

On Friday, September 23, 2022 at 10:24:36 AM UTC-4 joche...@gmail.com wrote:

Dear all,

I am writing a program which processes data in stages.  To structure the code cleanly, I would like implement each stage as an individual function and use channels to pass data between stages.

I would suggest that you reconsider whether a channel architecture is really cleaner after all.

It is usually much easier to reason about functions that accept and return values than about long-running goroutines.

For example, you could restructure your example as (https://go.dev/play/p/R0eDbFKnKgE):

• an iterator function (instead of a channel) that returns the next value when called

• a synchronous “process” function that returns a result and error, and

• a synchronous “add” method that accumulates the result.

That results in a somewhat larger “func main”, but makes the error behavior much clearer and substantially reduces the risk of things like deadlocks and goroutine leaks.

For (much) more detail, see the first section of my talk on Rethinking Classical Concurrency Patterns from GopherCon '18.