Channels vs Callbacks in API

[dc0d]

The statement being made here: a receive only channel, as a return value of a function, in an API, is not bad.

Reasoning:

• can not be closed
  
• can not be set to nil
  
• it's more clear than callbacks
  
• if signaling is the desired behavior, callbacks for sure are the wrong choice
• internals of the API might be affected by unknown behavior of callbacks 
• lets the consumer decides how many goroutines is needed to handle it

The last item especially is important since creating one goroutine per result item might not be the most efficient way. And if the API internals have no idea how to schedule the callbacks (panics aside) to make most of the machine resource (IO/CPU).

I would appreciate any help with the proposed (problem and) solution (reasoning).

[Axel Wagner]

Some counterpoints:

* In the question of "Channels vs. Callbacks", the best choice is "neither". Write a blocking API with a fixed number of items and a Scanner-like Iterator for an unknown number of items.

* It requires spawning a Goroutine even if none is necessary

* It adds syntactic overhead to the caller even if none is necessary

* If the caller in any way wants to customize the channel (e.g. add buffering) they have to spawn yet another Goroutine

* It requires specifying how the API behaves if the channel isn't read/quickly enough/whether it needs to be drained or not/when it will be drained - while an io.Closer already is the common idiom for resources to be closed

• Internals of the API might be affected by unknown behavior of callbacks 

Like what? A panic should just crash the program, no affected behavior. The package providing the API shouldn't have to worry about it. Anything else is isomorphic to unknown behavior of the channel consumer.

That being said, again, the correct choice is neither, which also doesn't suffer any of these problems. Encapsulate the concurrency.

• lets the consumer decides how many goroutines is needed to handle it

Just like a Scanner-like iterator.

[Kaveh Shahbazian]

1 - I should emphasize that the API in question is concurrent by nature (inbounds at any time which might need replies).

2 - All points made are good valid points in general. Thanks!

3 - With 1 in mind, still IMHO, channels should be preferred. Draining on either side, needs goroutines. But consumer must have better control - not just one goroutine per inbound. But yes, it's a concern.

(I'm on mobile now, but I'll give it more thought).

[thepud...@gmail.com]

On a semi-related note, I think there is a fair amount of wisdom in the "Synchronous Functions" section of the Code Review Comments golang wiki page (including it is not written as an absolute and starts with the word "Prefer").

https://github.com/golang/go/wiki/CodeReviewComments#synchronous-functions

 

----------------------------------------------------------------------

Synchronous Functions

----------------------------------------------------------------------

Prefer synchronous functions - functions which return their results directly or finish any callbacks or channel ops before returning - over asynchronous ones.

Synchronous functions keep goroutines localized within a call, making it easier to reason about their lifetimes and avoid leaks and data races. They're also easier to test: the caller can pass an input and check the output without the need for polling or synchronization.

If callers need more concurrency, they can add it easily by calling the function from a separate goroutine. But it is quite difficult - sometimes impossible - to remove unnecessary concurrency at the caller side.

----------------------------------------------------------------------

--thepudds

[Bryan Mills]

In some cases, a synchronous callback can be a fine choice: consider filepath.Walk.

A synchronous callback does not require any extra goroutines, and if the caller needs to adapt it to work with a channel (or perform longer-duration processing in parallel) that's easy enough to do at the call site: then the information about blocking/buffering/closing/draining behavior is localized at the call side instead of split across multiple packages.

As Axel notes, a channel-based API adds a lot of complexity due to goroutines: you have to be careful not to leak them, particularly if the caller stops reading early (e.g. due to encountering an error).

However, goroutines are not the only issue: in particular, you should also consider the interaction with error reporting. What happens if the function encounters an error? How will you inform the caller, will they remember to check for it, and will forgotten error-checks be obvious to future readers of the code? With an iterator or synchronous-callback API, the function call can return the error directly. With channels, you have to either return a channel of value-or-error or return a separate `<-chan error`, and either alternative makes the channel-draining invariants even more complex (and goroutine leaks more likely).

For example, with a separate error channel, if one channel is closed the caller needs to remember whether to check/drain the other one. With a single channel of value-and-errors, the caller needs to know whether they should keep reading the channel after the first error.

[Jason E. Aten]

Either is fine and the advantages of one over the other can be subtle and context dependent.

I typically think that for the utmost flexibility and performance at scale, I prefer callbacks, so as not to require the client use additional goroutines if they do not wish to.  The client can then send on their own channel, should they prefer. It leaves the choice to them. But it may depend on how adversarial you expect clients to be. The price for more guarantees is typically lower performance. Every channel operations requires synchronization under the covers, with all the cache coherency protocol slowness that implies, and at scale, that can be a drawback.