core.async top use cases

[Matan Safriel]

Hi,

It's very easy to see how core.async solves callback hell for front-end development with clojurescript.

In what use cases would you use it for server-side? we already have non-blocking IO from Java, and we have clojure agents. So what's a bunch of salient use cases?

Are there prominent clojure http server implementations which rely on it for transcending the threaded web service paradigm?

Thanks,

Matan

[Rangel Spasov]

http://aleph.io/aleph/literate.html

"Alternately, we can use a core.async goroutine to create our response, and convert the channel it returns using manifold.deferred/->source, and then take the first message from it. This is entirely equivalent to the previous implementation."

"Returns a streamed HTTP response, consisting of newline-delimited numbers every 100 milliseconds. While this would typically be represented by a lazy sequence, instead we use a Manifold stream. Similar to the use of the deferred above, this means we don't need to allocate a thread per-request."

I have tried the examples, it all works. Haven't done any benchmarks against using thread per-request though - you should think if your use-case can really benefit from this approach.

Rangel

[Mond Ray]

Pushing asynchrony further into the stack is useful for reliability and fault tolerance. We can also use it as a basis for Complex Event Processing using time series windows. 

I wrote up a few examples in my blog if you have the time to check out a longer explanation with code.

I recently wrote a small set of functions to enable HTML5 Server Sent Events from any Kafka topic which also uses core.async (with an example using Aleph and Compojure). You might like to check that repo out too.

Ray

[Matan Safriel]

Thanks, and I put the blog post on my reading list. 

Although I can't avoid thinking that we already have asynchronous idioms in the core language itself, like agents. I think the crux for server-side is more about the convenient piping, rather than the mere asynchronism itself, but I might be wrong in any of this.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/peJXvE0nBZs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Leon Grapenthin]

It is not just convenience. For example agents don't provide functionality like buffering, backpressure and select aka alts. If you send an action to an agent you don't get to know when it's done or to choose what to do if it is currently busy. 

On Monday, September 19, 2016 at 11:49:13 AM UTC+2, Matan Safriel wrote:

Thanks, and I put the blog post on my reading list. 

Although I can't avoid thinking that we already have asynchronous idioms in the core language itself, like agents. I think the crux for server-side is more about the convenient piping, rather than the mere asynchronism itself, but I might be wrong in any of this.

On Mon, Sep 19, 2016 at 9:14 AM, Mond Ray <mondr...@gmail.com> wrote:

Pushing asynchrony further into the stack is useful for reliability and fault tolerance. We can also use it as a basis for Complex Event Processing using time series windows. 

I wrote up a few examples in my blog if you have the time to check out a longer explanation with code.

I recently wrote a small set of functions to enable HTML5 Server Sent Events from any Kafka topic which also uses core.async (with an example using Aleph and Compojure). You might like to check that repo out too.

Ray

On Sunday, 18 September 2016 08:37:38 UTC+2, Matan Safriel wrote:

Hi,

It's very easy to see how core.async solves callback hell for front-end development with clojurescript.

In what use cases would you use it for server-side? we already have non-blocking IO from Java, and we have clojure agents. So what's a bunch of salient use cases?

Are there prominent clojure http server implementations which rely on it for transcending the threaded web service paradigm?

Thanks,

Matan

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/peJXvE0nBZs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Matan Safriel]

Right!

[William la Forge]

The really nice thing to me is that async handles side-effects while agents do not.

[Ken Restivo]

Decoupling components within a system, similar to what one would use queues or a messaging architecture for.

Used it extensively in two projects (one open source, one for a client) some years ago.

Rather than coupling the components via namespace dependencies and a call API, I just had the components passing data via async channels (queues), including pub/sub and topic subscription.

Worked great for its intended purpose.

-ken

[Matan Safriel]

Thanks but I'm not entirely sure about this. I could use agents for side effects too, or at least I thought so. Care to further clarify?

-------- Original Message --------
From:William la Forge
Sent:Tue, 20 Sep 2016 02:37:20 +0300
To:Clojure
Subject:Re: core.async top use cases

--

[William la Forge]

My bad. I was thinking of atomic. Swap! doesn't work with side effects, but send does.

[Beau Fabry]

I'm no expert on this, but the Actor model and the CSP model seem to be two different ways to model a concurrent system. Clojure supports them both. Personally I find the CSP model a simpler and easier to understand one than Actors, and so pretty much default to it. You might find non-clojure related sources comparing the tradeoffs between the two though?

[Matan Safriel]

Actually, I am not sure clojure implements the actor model, which I can only construe as the Erlang actor model here. I am almost certain the core language explicitly does not: http://clojure.org/about/state

It can be shoehorned somehow (see okku) but I would probably not venture saying clojure supports the actor model.

Sent from my mobile

[Beau Fabry]

You're probably right, I was confusing actors with agents.

[Derek Troy-West]

Fine grained control of parallelism is a superb aspect of core.async.

Cassandra is a distributed database, often a query requires you resolve-on-read denormalised data partitioned multiple ways (semantically, by time, etc). You can think of it like a grid I guess.

Lets say I have a query that I want to execute against Cassandra that covers a week of data, where the data is partitioned chronologically by the hour, and by modulo 24 to spread data around the cluster, that's going to require 4032 queries. Judicious use of core.async/pipeline-async allows me to execute those queries in parallel with granular control of the chronological / modulo parallelism independent of one another, meaning I can play with that parallelism in the REPL by just adjusting a couple of numbers and reviewing the results. Twiddling knobs, measuring output.

That allows me to execute those 4032 queries in parallel, sub-second, with non-blocking requiring none of the overhead of thread-per-request.

My services are Netty end-to-end (server and driver interface to Cassandra) so there are always a fixed number of threads. If I wanted to tune the buffers or thread-pool sizes I could probably get an even higher degree of throughput / lower-latency on each monolithic cassandra query, the constraints are known, quantifiable, and easily configurable.

[larry google groups]

> It is not just convenience. For example agents don't provide functionality like buffering, back pressure 

> and select aka alts. If you send an action to an agent you don't get to know when it's done or 

> to choose what to do if it is currently busy. 

So when to use agents? I've looked through Clojure repos on Github, looking for uses of agents, and I found very few. (I was writing a blog post about concurrency in Clojure, and I found that agents are among the least used tools for concurrency). I found a lot of uses of futures and promises and channels and core.async, and certainly atoms, but I didn't find many uses of agents. When are agents best used? 

[William la Forge]

On Thursday, October 13, 2016 at 3:38:16 PM UTC-4, larry google groups wrote:

So when to use agents? I've looked through Clojure repos on Github, looking for uses of agents, and I found very few. (I was writing a blog post about concurrency in Clojure, and I found that agents are among the least used tools for concurrency). I found a lot of uses of futures and promises and channels and core.async, and certainly atoms, but I didn't find many uses of agents. When are agents best used? 

When using clojurescript, adding async really increases the load time. That's one place where you might want to use agents when you can.

 

[Timothy Baldridge]

>> When using clojurescript, adding async really increases the load time. That's one place where you might want to use agents when you can.

But Clojurescript doesn't support agents.

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Timothy Baldridge]

Agents combine two things 1) a queue of functions, 2) mutable state. The key thing about agents is that they still respect Clojure's concept of "instant deref". That is to say, you can always deref an agent even if the queue is backlogged. This is one of the key differences between agents and actors. You have to send a message to an actor to deref it, an agent is always deref-able. 

Now it's true that you could build an agent via a core.async channel and an atom, but there's really no need since agents already exist in the runtime. Agents also support the lesser known features of Clojure's mutable refs, namely, validators and watchers. But the short answer is: agents existed before core.async. I've used them once in the past 2 years, but that's not to say they aren't useful sometimes. And it's nice being able to have a queue attached to a mutable reference without having to import core.async. 

[Alex Miller]

The other other special feature of agents is that the stm knows about them so it's a safe way to have a side effect occur in an stm transaction (all agent sends are delayed till the txn succeeds). I've found that to be pretty handy in advanced usage.

[Mark Engelberg]

My primary use case for agents has always been when I want to coordinate multiple threads writing to a log file.  The agent effectively serializes all the write requests with a minimum of fuss.

[Mark Engelberg]

I always found it a bit ironic that my main use case for agents doesn't really at all make use of the "mutable ref" aspect of the agent, only the queue piece.  I usually hold the name of the log file in the mutable ref to emphasize that the agent is "guarding" this particular log file, but I don't actually mutate it, so the mutability doesn't really matter for this purpose.

[Timothy Baldridge]

Yeah, I used to do that, but once core.async came out I started to desire the back pressure aspects of channels. I don't think I've used agents for logging since. You always run the risk of something backing up the queue of the agent and causing your thread to crash when it runs out of memory. 

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gordon Syme]

I've used agents to wrap thread-unsafe mutable Java objects with a defined life cycle, so that they could be used from multiple threads whilst respecting the life cycle.

My particular case was server-side gRPC StreamObservers for long lived client connections.
These are either usable, closed, or errored and an exception is thrown if you try to send on a closed or errored observer.

Using the agent state to model the life cycle, and the natural queueing of agent actions gave me thread-safe fire-and-forget semantics (which was appropriate for the messages involved).

To be honest it was the first time I've found a good use for agents, but they fit really well.

-Gordon