Opinion on core.async vs callbacks in abstract APIs?

[Christopher Small]

Greetings

I imagine most of us here would rather use core.async channels over callbacks in their application code, particularly with more complicated applications. But is it okay/preferable for Clojure libraries to force their users to use core.async channels as part of an API (an event channel, for example)?

As much as I love core.async, I can't help but wonder whether sticking with callbacks for an API isn't a simpler/better design strategy. It's easy enough to drop messages on a channel in a callback, and this let's users opt-in. But if one expects core.async channels are what most would prefer anyway, is it okay to foist them upon everyone?

As a follow up, does your opinion on the matter change if implementations of an API become simpler using core.async channels?

Looking forward to your thoughts :-)

Chris Small

PS I'm asking because I'm working on a physical computing API (https://github.com/clj-bots/pin-ctrl) and debating between using channels vs callbacks for the edge detection functionality (if you're not familiar, edge detection let's you asynchronously handle changes in pin state, such as button pushes). If you're interested in this question as it applies specifically to this application, feel free to join the discussion on our gitter channel: https://gitter.im/clj-bots/chat

[Alejandro Ciniglio]

Zach Tellman talks about exactly this in his conj talk from last year https://www.youtube.com/watch?v=3oQTSP4FngY

He built a library around this that essentially gives the library user a choice of either option: https://github.com/ztellman/manifold

[Eldar Gabdullin]

I would implement everything sticking to just callbacks, then create separately requirable core.async version of API if that matters.

Ideally this should be a separate lib, but practically, it seems better to just have a separate file. 

понедельник, 1 июня 2015 г., 22:18:19 UTC+3 пользователь Christopher Small написал:

[Eldar Gabdullin]

I'd like to add that core.async is quite a big thing, it has lots of parts that could be implemented differently.

[Gary Trakhman]

I think this is one of those cases where the rules are different for libraries than applications.  

Does your lib need to pull in core.async, and does it need to be coupled to a specific version?  If it's a building block sort of lib that clojure and the community prefers, I think the answer is no.  Is this more of an opinionated 'framework' or set of batteries (luminus might fall in this category)?  In that case, yes.

As an application-writer having to write some glue code feels less frustrating than when libs make assumptions that I don't like.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrey Antukh]

Hi!

Personally I think that manifold has the same problem that core.async. So if you are exposing your api using manifold you are forcing to someone to use manifold. It is not bad, but is the same problem as with core.async. 

And the same problem with callbacks. If you are using callbacks you are force to people to use callbacks or adapt it to whatever other abstraction.

So, independently of the chosen abstraction, you are always forcing the user to use the chosen abstraction or adapt their code to another abstraction.

About the original question, I think it depends that you really wants. In some projects I expose api using inter operable with jvm abstractions like (reactive-streams) or promises (completable future in jdk8), in other I just use core.async. 

There is no single solution I think!

My two cents!

Andrey

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Andrey Antukh - Андрей Антух - <ni...@niwi.nz>

http://www.niwi.nz

https://github.com/niwinz

[Alejandro Ciniglio]

That’s a fair point. Although, I think manifold does have going for it that it’s designed to interoperate with the other abstractions we’re discussing, so it shouldn’t be as binding as building your API around core.async would be.

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/nuy2CAA89sI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Timothy Baldridge]

I'd say stick with callbacks simply because they're harder to mess up. There's many different ways to interface with core.async. You can return a channel, you can require a channel as an argument. You can spin up go blocks that park, you can have pipelines, you can create a go block per call to an api, or one that lives for the lifetime of the app. Some APIs can have cancelable operations, some don't. Knowing how to build an API that fits everyones needs is really hard. 

Sadly, I don't trust many people (including myself) to implement a core.async API correctly for the needs of every application. But callbacks are pretty hard to mess up. Just document where the callbacks are run (on a single thread, in a thread pool, etc.) and I can easily adapt your library to fit my application. 

Timothy

“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Stanislav Yurin]

I think returning futures for asynchronous calls is a good tradition in Clojure world.

Better than callbacks because you are leaving the threading model choice in the hands of the caller, which is a good thing.

[Stanislav Yurin]

As for the core.async, I think it is too personal and has too much raw power, to be all that restricted in some logical bottleneck upon results return from the third-party lib. 

Not counting the fact it is a (a) dependency that (b) changes fast.

On Monday, June 1, 2015 at 10:18:19 PM UTC+3, Christopher Small wrote:

[Timothy Baldridge]

The problem with futures is that you can't attach callbacks to them, you can only block a thread waiting on them. So futures interface quite poorly with async libraries, hence the reason core.async was created in the first place. 

Core.async is a dependency, but it's hardly one that changes fast. The last breaking change was about a year and a half ago (Jan 2014). Besides that, all changes are additional "opt-in" features. That's a lot less change than most libraries in the Clojure ecosystem.  

Timothy

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stanislav Yurin]

Why so? With a callback, someone should be waiting somewhere too, until callback is fired. 

Why not expose this choice to user. E.g. I am often waiting for the future in the (thread ..) and returning the result to the channel,

but again, I like this to be my choice, because there are so much ways of doing this.

[Leonardo Borges]

For people interested in using the 'futures' approach, this might be of interest: https://github.com/leonardoborges/imminent

It's a library that implements composable futures for Clojure on top of JVM's ForkJoin framework. It allows you to attach callbacks as well as apply combinators such as map etc...

[Christopher Small]

Lots of great thoughts here; thanks so much for the feedback!

It seems the general opinion so far leans towards keeping things simple. I definitely resonate with this statement: "having to write some glue code feels less frustrating than when libs make assumptions that I don't like."

Unfortunately, the abstractness of my question is allowing things to drift a little further than I intended. So to ground things a bit, in my particular use case promises wouldn't work because I need to allow for a stream of events. Promises only let you capture a single event.

Additionally, I feel it worth responding to some of Timothy's comments about the difficulty of getting a core.async centric API "right". While I agree that in general (and particular with more complicated APIs) getting things right can be tricky, the situation I'm dealing with (I think) can be dealt with quite simply. Imagine a stream of events coming in from The Real World (TM), and that we'd like to process somehow. The solution I've been chewing on is a function that let's us specify a single channel on which events should be placed. The nice thing about this is that it assumes almost nothing about how you'll use core.async, only that you'll be using it. You can decide whether you want a buffer or not, whether it should drop or slide or block/park, whether it should be tapped, etc.

The potentially nice thing about core.async for this use case over callbacks (as far as the implementation is concerned) is that should we decide to stop listening to events, we can just close! the channel. This can then serve as a signal to the underlying code/process actually reading in events that it can stop. With callbacks, it will probably be necessary to have something that manages these processes and listens for kill signals, potentially complicating things quite a bit. (It's possible there is an elegant solution here, and I haven't thought about it too much yet, but this is my current intuition...)

I should also mention that Manifold did come to mind for this project, and it's something I'll probably look at and consider again. Thanks for bringing it up.

Please continue to share your thoughts given all this :-)

With gratitude

Chris

[Gary Verhaegen]

Have you considered returning a lazy seq of events?

[Erik Price]

On Tue, Jun 2, 2015 at 3:17 AM, Christopher Small <metas...@gmail.com> wrote:

Additionally, I feel it worth responding to some of Timothy's comments about the difficulty of getting a core.async centric API "right". While I agree that in general (and particular with more complicated APIs) getting things right can be tricky, the situation I'm dealing with (I think) can be dealt with quite simply. Imagine a stream of events coming in from The Real World (TM), and that we'd like to process somehow. The solution I've been chewing on is a function that let's us specify a single channel on which events should be placed. The nice thing about this is that it assumes almost nothing about how you'll use core.async, only that you'll be using it. You can decide whether you want a buffer or not, whether it should drop or slide or block/park, whether it should be tapped, etc.

I like and use core.async, but this sounds like something that I might want to handle with a compositional event system like RxJava. A simple callback-based API would suffice for that.

The potentially nice thing about core.async for this use case over callbacks (as far as the implementation is concerned) is that should we decide to stop listening to events, we can just close! the channel. This can then serve as a signal to the underlying code/process actually reading in events that it can stop. With callbacks, it will probably be necessary to have something that manages these processes and listens for kill signals, potentially complicating things quite a bit. (It's possible there is an elegant solution here, and I haven't thought about it too much yet, but this is my current intuition...)

Good point. CES systems that I’ve used have a first-class notions of “stream complete” and “stream errored”, which is even more expressive than core.async‘s “send nil“. If your API offered three separate callbacks (one for each of those conditions and one for values, with the invariant that if either the “stream complete” or “stream errored” callback fires for a stream, no further callbacks will be invoked), it would be pretty easy for a user to map them to their CES equivalents, and if someone prefers to work in core.async, they can send whatever value they wish on the channel, followed by nil.

e

​

[Christopher Small]

Lazy seqs by themselves wouldn't work, since it needs to be asynchronous, and it's always possible that when you ask for the next element, that there won't have been any new events yet. I suppose that you could return a lazy sequence of futures, which is interesting, but not particularly idiomatic. At that point I'd rather just use core.async, because that's what it starts to look like.

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/nuy2CAA89sI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Christopher Small]

@Erik: I should clarify in this situation, the _user_ of the API would decide whether they want to stop listening to events. So there's not so much that _they_ would have to specify in terms of shutdown routines. I'm more concerned about how API implementations get notified that they don't need to deal with new events.

[Erik Price]

Oh, so the events are for all intents and purposes infinite?

On Tue, Jun 2, 2015 at 11:10 AM, Christopher Small <metas...@gmail.com> wrote:

@Erik: I should clarify in this situation, the _user_ of the API would decide whether they want to stop listening to events. So there's not so much that _they_ would have to specify in terms of shutdown routines. I'm more concerned about how API implementations get notified that they don't need to deal with new events.

[Christopher Small]

Yes, potentially. And their temporal distribution could be any imaginable.

The idea is that we're dealing with things like button presses, which could happen as frequently or infrequently as a physical device might need, and for which an application would have to be prepared for the duration of the device being in operation. I imagine most "production" applications probably wouldn't need to "stop listening" to events (though they could...). Mostly, this is for interactive REPL development; If you're tinkering around with things and decide you want to shut off or change the edge direction of a given pin, you don't want the old callbacks to keep firing.

Chris

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/nuy2CAA89sI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Gary Verhaegen]

I'm not sure I follow why lazyness cannot be asynchronous, as long as reads are blocking (which they are by default in core.async, I believe). Your description of the problem made me think of this blog post I read some years ago:

http://oobaloo.co.uk/clojure-abstraction-and-zeromq

which explains how to expose a stream of zeromq messages as a lazy seq, so that consuming code does not need to know about zeromq. It sounded similar, but maybe it's not.

[Timothy Baldridge]

I think the problem is more centered around push vs pull based APIs. Callbacks, and core.async work quite well for push based apis (e.g. event streams). Lazy seqs work on a pull basis. ZeroMQ has a pull API, so lazy seqs work quite well there. 

Also lazy-seqs aren't async. If you call (next ..) and the computation of the next cell makes a network request, you've now blocked the current thread waiting for a response. 

Timothy

[Gary Verhaegen]

That makes sense. Thanks for the explanation.

[Andrey Antukh]

On Tue, Jun 2, 2015 at 7:18 AM, Leonardo Borges <leonardo...@gmail.com> wrote:

For people interested in using the 'futures' approach, this might be of interest: https://github.com/leonardoborges/imminent

It's a library that implements composable futures for Clojure on top of JVM's ForkJoin framework. It allows you to attach callbacks as well as apply combinators such as map etc...

Great library, thanks for sharing.  I have done something similar, only focused on promise abstraction and with focus on using existing jvm implementation instead of creating a new one: https://funcool.github.io/futura/latest/#_promises

--

[Andrey Antukh]

On Mon, Jun 1, 2015 at 11:12 PM, Gary Trakhman <gary.t...@gmail.com> wrote:

I think this is one of those cases where the rules are different for libraries than applications.  

Does your lib need to pull in core.async, and does it need to be coupled to a specific version?  If it's a building block sort of lib that clojure and the community prefers, I think the answer is no.  Is this more of an opinionated 'framework' or set of batteries (luminus might fall in this category)?  In that case, yes.

As an application-writer having to write some glue code feels less frustrating than when libs make assumptions that I don't like.

Completely agree with that! 

[Zach Tellman]

The problem with using bare callbacks is that there's no way for the invoked callback to exert backpressure, except by blocking or passing in a CPS-style callback to the callback, neither of which is ideal.  If you're looking for a "neutral" choice, I'd suggest an infinite lazy-seq over callbacks.  If you absolutely need it to be async, something like Manifold might be warranted, in that it allows people to consume it however they like (though they will have to be at least a little familiar with Manifold to do so).

Zach

[Ivan L]

Is there a detail of core.async that is needed in a third party usage?  If not, you should't have to require core.async.

Be liberal in what you accept and conservative in what you produce - Art of Unix Programming paraprhased.