core.async over websocket + cljs + clojure

[t x]

Hi,

  I apologize for my vague question.

  Does anyone have a good example / blog / library for using the core.async abstraction across a websocket.

  * one side of the channel is in clojure land

  * other side of the channel is in cljs land

  * I promise that all messages can be encoded via pr-str and read via clojure.edn/read-string

  What I'm struggling with are matters of:

  * how to ensure data is not lost even when websocket disconects / reconnects

  * "naming" on client/server side to ensure messages go to right channels on both sides

  * issues I haven't even begun to imagine.

  Good news:

  * I control both sides: both the clj and cljs side, so any workable design is fine.

Thanks!

[Sean Corfield]

Ah, what good timing!

David Pollak's project Plugh does this, essentially as an implementation detail. I spent some time with him today discussing this, as I want to use exactly this functionality in a project I'm building.

The plan is for me to create a standalone project, based on the kernel of David's code, and enhance it to address a number of issues that he identified as needing work before the project could be used in production code, and then - hopefully - people will use it and provide additional integrations (such as core.async over a message hub to provide server-to-server operation, or core.async between browser client and server using more transmission methods than just web sockets).

David's code addresses naming using a registry of channels, identified by GUIDs, on both sides. The web socket reconnection issue is one of the specific enhancements he identified that I plan to figure out and address. There are several others (including actually "GC'ing" closed channels on the other side of the address space divide).

Sean

 

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

[t x]

For anyone else interested, http://www.youtube.com/watch?v=Nb3ztFeFfdw is the talk.

[Sean Corfield]

On Jan 23, 2014, at 12:18 AM, t x <txre...@gmail.com> wrote:

For anyone else interested, http://www.youtube.com/watch?v=Nb3ztFeFfdw is the talk.

That's David Pollak's talk at Clojure/conj. You'll probably also want to watch his Strange Loop talk first, where he introduced the concepts:

http://www.infoq.com/presentations/pushy-data

[ckirkendall]

I recently gave a talk about doing this.  You can find the code for my talk here:  https://github.com/ckirkendall/chatter-box,  Its a group chat application that has the feel of twitter.  It uses core.async and websockets.  The core.async code is bit dated but the websocket stuff should be good to go.  If your interested in seeing the talk: http://www.youtube.com/watch?v=jj6eQieGWqo&list=TLaY6nSMlp35yusQFm7HAZPE607i_re1Cv

CK

[Logan Linn]

I've been working on a game in my spare time that does this.

The Clojure backend and ClojureScript client communicate with core.async over WebSocket carrying edn data

Game: https://github.com/loganlinn/ji

Client WebSocket using: https://github.com/loganlinn/cljs-websockets-async

Server WebSocket using: https://github.com/lynaghk/jetty7-websockets-async

I had hoped to have writeup about this technique by now, but coding seems to take higher priority when I get free time :P

[Mark Mandel]

I'm sure Plugh probably does similar things, but my "learning clojure app" has my own custom RPC mechanism over websockets that I wrote (because it was fun) that is all based around core.async and uses edn to transfer data back and forth.

https://github.com/markmandel/chaperone

Server Side:

https://github.com/markmandel/chaperone/blob/master/src/chaperone/web/websocket.clj

https://github.com/markmandel/chaperone/blob/master/src/chaperone/web/rpc.clj

Client Side:

https://github.com/markmandel/chaperone/blob/master/src-cljs/chaperone/websocket.cljs

Hope that helps.  The project isn't hugely far along, but the RPC mechanism works.

Mark

--
E: mark....@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

2 Devs from Down Under Podcast

http://www.2ddu.com/

[t x]

Fantastic! I will study these solutions and spam questions later.

[Alexander Hudek]

We've had something like this working for a while now, though our system uses browserchannel rather than websockets since we need to support older browsers. On the plus side, browserchannel handles some issues already such as managing the state of the connection to the server and retrying when it fails. We also have it connected up to rabbitmq on the server side for server-to-server communication. It's possible for a backend service to broadcast messages right up to multiple connected browsers (e.g. service -> web-server-nodes -> clients).

Not all parts are as generic as we'd like for an open source release, but seeing as people might be interested we'll try to get some pre-release code out somewhat soon.

[Patrick Logan]

  * one side of the channel is in clojure land

  * other side of the channel is in cljs land

Are you implementing coordination across the wire, as if the two channels are "the same virtual channel"? If so, read on... otherwise, n/m, sorry if I misinterpreted...

CSP-like channels aren't a good across-the-wire abstraction. Their blocking semantics are intended to coordinate concurrency within a single runtime.

To be reliable you'd have to introduce addition machinery to account for the hazards of distributed systems, so you're probably better off starting with an abstraction that has those hazards in mind already.

I'm unsure what the arguments would be in favor of CSP-like behavior across distances, especially between a server (clj) and a browser (cljs)?

[Logan Linn]

To be reliable you'd have to introduce addition machinery to account for the hazards of distributed systems, so you're probably better off starting with an abstraction that has those hazards in mind already.

The WebSocket (protocol) is the machinery that's handling this. We aren't discussing using CSP to implement the actual network communication. We aren't concerned what happens outside of the edges of the system.

Unlike when using actors, with CSP you don't care where a value you took from a channel came from, and similarly you don't care where the value you put onto a channel goes. The channel is only a primitive for conveyance. The application knows nothing of a WebSocket other than what does the put!s and take!s.

CSP-like channels aren't a good across-the-wire abstraction. Their blocking semantics are intended to coordinate concurrency within a single runtime.

Coordination is possible through the blocking semantics of CSP, but isn't the only mechanism it provides. CSP also facilitates buffered, asynchronous operations. I don't see why it would imply anything about a single runtime.

[Patrick Logan]

In CSP you might have a limited size buffer, but then block on the next Put. That's not something you want to casually attempt over a distance. It seems you want an interface like Channels that deal in fully formed objects, but you don't want CSP blocking semantics.

[Cedric Greevey]

What about a hybrid blocking/timeout approach? At the sending end of the network bridge you have the "taker" and at the receiving end the "putter". The "putter", when it sees an object on the wire, puts it on a local channel (blocking put with timeout). If it succeeds it sends an ack up the wire. If the timeout elapses it sends a heartbeat up the wire instead, then tries again. The "taker", meanwhile, takes from a channel and sends the object on the wire, then waits for an ack or a heartbeat with a longer timeout. If neither arrives within this timeout, it uses another channel to signal the application that the network connection to the "putter" was lost. If a heartbeat arrives it waits again, with the clock restarted. If an ack arrives, it takes another object from a channel and sends it on the wire.

This should cause backpressure to go over the network to the sending end and block the sender, but with the caveat that if the network connection is actually lost, the sender's business logic will discover that shortly, and can distinguish it from the receiver just being slow to process each item. In particular, it can listen for a network drop event on a control channel, and separately use a timeout when putting onto the "taker" channel to react if the receiver is too slow but still connected.

On Sat, Jan 25, 2014 at 9:11 AM, Patrick Logan <patric...@gmail.com> wrote:

In CSP you might have a limited size buffer, but then block on the next Put. That's not something you want to casually attempt over a distance. It seems you want an interface like Channels that deal in fully formed objects, but you don't want CSP blocking semantics.

[Patrick Logan]

This seems like more trouble than it is worth. There are almost certainly suitable but more established protocols and implementations for the problem at hand. Anyway, maybe it's worth exploring. To me it seems to muddy the waters for what core.async seems intended to provide, which seems to me to be fairly straight-forward CSP.

[Michał Marczyk]

If the application is structured in such a way that messages from the
other side of the client/server divide are put on channels upon
arrival, which seems to be a reasonable idea, why not encapsulate the
connection-handling logic in a black box exposing a (bunch of)
channel(s)?

On 26 January 2014 00:01, Patrick Logan <patric...@gmail.com> wrote:
> This seems like more trouble than it is worth. There are almost certainly suitable but more established protocols and implementations for the problem at hand. Anyway, maybe it's worth exploring. To me it seems to muddy the waters for what core.async seems intended to provide, which seems to me to be fairly straight-forward CSP.
>

[t x]

I believe the semantics can be simplified as this:

  * the server (clj) can disconnect at any time

  * the client (cljs) has the responsibility of

    (1) informing the user

    (2) reconnecting

    (3) figuring out what went wrong

    in such events

I believe this is both necessary and sufficient.

It's sufficient in that a dropped network connection can be viewed as "server decided to disconnect."

It's also necessary in that -- if we want defensive programming and to avoid DOS attacks by malicious clients, it's important for the server to, at some point, say "F U client, you're out of tokens; I'm disconnecting you" or to say "F U client, I can't send data to you; I'm disconnecting you."

This is not CSP semantics, but I believe it's both necessary and sufficient for having "core.async" across the network -- the semantics being

  * the server can, at any time, arbitrary disconnect a client

  * it's the client's job to inform the user / try to reconnect

[Sean Corfield]

On Jan 25, 2014, at 3:17 PM, Michał Marczyk <michal....@gmail.com> wrote:
> If the application is structured in such a way that messages from the
> other side of the client/server divide are put on channels upon
> arrival, which seems to be a reasonable idea, why not encapsulate the
> connection-handling logic in a black box exposing a (bunch of)
> channel(s)?

This is more or less what the core of David's code does:

* the code uses regular core.async channels (on both sides)
* both side have the concept of a remote channel (which is associated with a GUID in a registry atom)
* a "black box" on each side takes from remote channels and sends data across the wire (including the channel ID)
* that "black box" also receives data over the wire and puts data onto the appropriate remote channel

As far as the application is concerned, it's all just channels - the black box performs relaying of data across the client/server gap for each remote channel.

All of the solutions folks have posted so far seem to assume a single channel routed over the client/server transport so I feel comfortable going ahead with library-izing David's code to support multiple channels and, ultimately, multiple transports.

[James Henderson]

Hi folks, sorry I'm a bit late to the party, but Chord might be a good base for what you want to do? It probably falls a bit short at the moment on your disconnection semantics, but could be worth a shot.

https://github.com/james-henderson/chord

James

[ton...@gmail.com]

Anyone come up with anything? 

I'm also wondering ways to abstract this problem. Here's my current impl https://gist.github.com/whodidthis/9126971, you can imagine chord instead of the input and output channels. Any ideas and suggestions would be cool as im new to clojure.

[Sean Corfield]

My priority to work on turning David's codebase into a library changed as I was originally planning to do it for conference presentations in May and June, but for personal reasons I've had to pull out of both conferences. So this is still on my radar but it's moved down my list some way.

Sean