Pattern for service with multiple clients using two-way channels?

[Vic Putz]

I love the idea of core.async channels as generic inboxes for "in-program microservices" for lack of a better term.  They're a great decoupling mechanism.

But I get a little confused when you get beyond a producer-consumer model.  A service with an inbox (consumer) works well for multiple clients (producers) because they can all just dump data into the same inbox.  But it gets a little trickier if you want any data back (toy example: a jukebox microservice with an input channel: drop the song name in as a string and it starts playing, and doesn't care if A, B, or C dropped the song name in, but if A wants a list of available songs you need a return channel and the jukebox can't just drop A's list of songs in a single outbox.

Either it feels like you need separate return channels for each client (which complects transport with processing, but would let each client pretend they had a private connection) or you have to use core.async publish-subscribe, maybe pulling "sender" out of the topic for the return address and have clients subscribe to the service's output channel based on their own identifier, but that forces messages to carry extra information about routing... which may be fine.

Is there an established pattern or mechanism for this, or is pub/sub probably the best way to go?

[William la Forge]

Let each request carry the callback channel. It is then the requestor's option to reuse a channel for multiple requests or to open one for each request or some combination.

For example, some requests might get a lot of responses and the requestor may need to close the channel before getting them all. Obviously in this case you would want the request to have its own dedicated channel.

In general I'd start with one channel per request and then optimize as needed. :-)

[Timothy Baldridge]

Most would perhaps start with the classic approach of having each request contain a :reply-to channel that the responses go into, but I think this is a mistake. 

IMO, there are two main ways of building systems with core.async 1) request/response. 2) dataflow graphs. 

I prefer the second approach which is to view services as pipeline stages. For example you might write something like this in clojure:

(-> (read-file-list)
      (map load-image)
      (map resize-image)
      (map save-image))

You could easily then write this sort of thing as a pipeline:

(let [file-chan (chan 1024)
       image-chan (chan 1024)
       resized-chan (chan 1024)]
  (read-file-list file-chan)
  (load-images file-chan image-chan)
  (resize-images image-chan resized-chan)
  (save-images resized-chan))

In addition, you can have these pipelines stages use async/pipeline-* functions to give you very clean knobs with which to tune your system. 

And finally, when it comes time to distribute processing it's fairly trivial to replace the channels in the second example with a durable message queue, and suddenly you can go from a system running on one machine to one running on multiple machines (granted with different failure characteristics, but I digress).

So try to think of your system as a pipeline and not so much as request/response calls. Even things like HTTP servers with their stack of middleware can be viewed as a processing pipeline, and in fact this is way some libraries, like Pedestal, model them. 

Timothy

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Vic Putz]

Thanks to both of you!

On Wednesday, January 6, 2016 at 10:11:39 PM UTC, tbc++ wrote:

Most would perhaps start with the classic approach of having each request contain a :reply-to channel that the responses go into, but I think this is a mistake. 

IMO, there are two main ways of building systems with core.async 1) request/response. 2) dataflow graphs. 

I like the dataflow-graph approach, because I don't like the idea of conflating routing with messages that a "sender" or "reply-to" within the message requires (though William, I hadn't thought of just putting the channel itself as reply-to... great idea if I go that route and much simpler than dropping into pub/sub).  So I'd rather the service just have "in" and "out" channels and have the wiring be done externally (probably through initialization via "component").

And if the components were wired together 1-to-1, that would be dead easy.  I'm just not sure how to deal with a component/service which should be able to handle multiple in-out connections ... my naïve approach would probably just be for services to have a vector of in/out connection pairs that got wired up at initialization and maybe go-loops over all of them, which would probably work but seems mildly inelegant somehow.

(again, using the toy application of a jukebox service and three clients... the clients can't request a song until they know what songs are available, so bidirectional communication seems necessary somehow; I can't see how to design a wiring diagram without it)

[William la Forge]

So use a mixed approach.

Queries are likely best handled by including a return channel with the request.

Updates are likely best handled with a flow approach.

Complex queries may involve passing a return channel deeper into the process. Especially notification requests.

No one answer is best for all, as usual. But keep simple things simple.

Bill

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/_8fHJ3J-MYg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.