[ANN] Sente - Clojure(Script) + core.async + WebSockets/Ajax

[Peter Taoussanis]

Hi folks,

Quick post to announce a new lib release: https://github.com/ptaoussanis/sente

From the README:

Sente is small client+server library that makes it easy to build reliable, high-performance realtime web applications with Clojure.

* Bidirectional a/sync comms over both WebSockets and Ajax (auto-selecting).

* Robust: auto keep-alives, buffering, mode fallback, reconnects.

* edn rocks. So send edn, get edn: no json here.

* Tiny, simple API: make-channel-socket! and you're good to go.

* Automatic, sensible support for users connected with multiple clients and/or devices simultaneously.

* Flexible model: use it anywhere you'd use WebSockets or Ajax.

* Fully documented, with examples (more forthcoming).

* Small: less than 600 lines of code for the entire client+server implementation.

* Supported servers: currently only http-kit, but easily extended.

---

Have been using something like this in production since a little after core.async came out, and wouldn't want to go back. Note that I tweaked a few things for the public release so there may be some rough edges initially. 

An example project's included (new as of today) and there's a Leiningen alias configured to handle all the fiddly bits like getting the Cljx and Cljs to compile: just download, `lein start-dev` at a terminal, and you're good to go.

Any questions/problems/whatever, you can reach me here or on GitHub.

That's it! Happy hacking, cheers! :-)

- Peter Taoussanis

https://twitter.com/ptaoussanis

https://www.taoensso.com/clojure-libraries

[Base]

Peter,

This looks great!  can't wait to try it out. 

Well done.

[edbond]

Thanks. Can you compare sente with chord?

[Peter Taoussanis]

@Base

This looks great!  can't wait to try it out. 
Well done.

Much appreciated, hope it's useful :-)

 @edbond

Thanks. Can you compare sente with chord?

Sure, I can try - have never actually used Chord though (it wasn't around when I wrote Sente's predecessor) - so I can only comment superficially on what I've gleaned from the README now (would appreciate any corrections!).

Similarities:

* Both seem to offer edn WebSocket messages over core.async channels.

* Both seem to require http-kit for the moment.

Primary difference seems to be that Chord supports only WebSockets (?) whereas Sente will fall back to Ajax when WebSockets are unavailable. This has ramifications for the API where Chord seems to prefer working with the channel directly and Sente uses a send fn to wrap away some of the protocol differences.

Sente's wrapping extends to allow things like optional callbacks over WebSockets (something not natively available) and broadcasting to multiple clients (e.g. when a particular user has multiple browser tabs and/or devices connected simultaneously, etc.) which I've found important in practice. Some web apps don't do this very well (even Google seems to struggle with it), so having a sensible solution baked in can be helpful.

Am not sure what Chord's doing in the way of reconnects, etc.

tl;dr Chord seems be a relatively close-to-the-metal mapping of WebSockets to edn channels where Sente is more like Socket.io with edn channels.

I'd probably select between the two based on use-case though, again, I'm not at all familiar with Chord or other alternatives so I may well be off here.

Hope that helps a little, cheers! :-)

[Peter Taoussanis]

Quick addendum: I've had a few folks ask about various security implementations.

So first up - as I mention in the REAME, I have had to make some changes to the codebase for the public release so I wouldn't quite trust anything pre-1.0 in production just yet (there's likely at least minor bugs). Having said that, the general model has played out well in practice elsewhere.

A few specifics that have come up:

 * Tools.reader's read-string is used server-side, so arbitrary code evaluation isn't an issue.

 * It _is_ possible for the server to receive malformed or hostile requests (as it would be via any other protocol). The implementation itself doesn't depend on the accuracy of any information provided by the client, but the usual security measures will apply server-side for code you write: only respond to well-formed requests, ensure users are authenticated+authorised where appropriate, etc. Auth stuff is beyond the scope of Sente, but it won't do anything to get in your way.

 * Ring session information _is_ made available server-side for all requests (including WebSocket requests), so the commonest session-based auth methods basically work out-the-box.

 * The client-side user-id *can* be forged, but it isn't used for anything that's security sensitive. It's there only as an optimisation to disable Ajax long-polling when a user clearly isn't logged in. If someone were to fudge the id, the only negative effect would be an the creation of an unnecessary long-polling connection that is never used (the server will never acknowledge it).

 * HTTPS works seamlessly, even for WebSockets. When the page is served over HTTPS, Sente will work over HTTPS.

 * CSRF protection is baked in, but you need to do a few things on your end to enable it. A client-side warning will be printed when no CSRF code is provided. Recommend checking out the example project for details (it includes CSRF protection).

So tl;dr - there's a bit going on, but nothing that's fundamentally different from the usual Ring security model. Auth and CSRF are the responsibility of the app as usual, but Sente provides hooks in the appropriate places to plug in whatever you're using.

Will try update the README later today with the info above.

Hope that helps, happy Friday! :-)

- Peter

[Mark]

Hi Peter -

I'm trying to learn Sente using the LightTable browser connected repl.  I try to evaluate the following code:

(let [{:keys [chsk ch-recv send-fn]}

      (sente/make-channel-socket! "http://localhost:6100/chsk" ; Note the same URL as before

         {})]

  (def chsk       chsk)

  (def ch-chsk    ch-recv)

  (def chsk-send! send-fn))

(put! ch-chsk "From client!")

From the JS console, I see repeated, failed attempts to connect the server.  The server is running and responding to other http requests.  

Taking a very quick look at the source for the CLJS make-channel-socket!, it seems that it tries to resolve the url using the CLJS chsk-url function which, in turn, uses encore/get-window-location.  Since my browser window location is file://..., I think this is going to generate a bad url.  Am I reading the source correctly? Is there some way to make a channel socket by providing an absolute url?

[Peter Taoussanis]

Hi Mark,

I'm trying to learn Sente using the LightTable browser connected repl.  

Ahh, nice. Haven't tried LightTable yet myself, but I'd think it should work okay.

 

From the JS console, I see repeated, failed attempts to connect the server.

The URL is what's tripping you up here. Is there a reason you specifically want to provide an absolute URL? At the moment `make-channel-socket!` just takes a path (like "/chsk"). The host and port, etc. will all be determined for you automatically. "/chsk" is a reasonable choice btw - it's not user-facing so doesn't need to be particularly meaningful to anyone but you.

Taking a very quick look at the source for the CLJS make-channel-socket!, it seems that it tries to resolve the url using the CLJS chsk-url function which, in turn, uses encore/get-window-location.

That's correct. It just grabs the host and protocol of whatever page is serving the JavaScript. HTTPS protocols will be converted to WSS, everything else goes to WS. You should be fine there, I'd think.

> (put! ch-chsk "From client!")

Note that you'll never be putting to the `ch-chsk` yourself, it's for receiving only. To send events you'll use the `chsk-send!` fn (the API is described at https://github.com/ptaoussanis/sente#now-what).

I also updated the repo a couple days ago to include a working example project that you could pull into LightTable to hack on. Otherwise feel free to open a GitHub issue if you have any other questions - am happy to help.

Cheers :-)

- Peter

[Eduard Bondarenko]

You are probably hit by CORS, try to open webpage at the same host http://localhost:6100/, see http://enable-cors.org/index.html

Check Network tab in Chrome dev tools.

Best regards,
Eduard

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/5J4L8pbGwGU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Mark]

Yep, I was running into the two problems you pointed out:  First, I needed to use a relative URL in make-channel-socket! and that necessitated delivering my HTML through http-kit rather than the shortcut of reading it from the file system using file://...  Once I did that, an assertion nicely documented the use of chsk-send! (thanks for that!).  One small thing that tripped me up: It seems the event id must be a namespaced keyword.  I haven't thoroughly rtfm yet so if that's doc'd, bad on me.  

Thanks for the pointers!

[Peter Taoussanis]

and that necessitated delivering my HTML through http-kit rather than the shortcut of reading it from the file system using file://...  

Yes, good catch - sorry, forgot to mention that.

 

Once I did that, an assertion nicely documented the use of chsk-send! (thanks for that!).  One small thing that tripped me up: It seems the event id must be a namespaced keyword.  I haven't thoroughly rtfm yet so if that's doc'd, bad on me.

It's documented but easy to miss - I'll add an extra assertion to print on malformed event ids (thanks for the feedback!).

Cheers :-)

[Sam Ritchie]

Hey Peter,

I like the UUID feature, but it doesn't look like the list of all connected users is available in the API. How would you do a global broadcast to all connected clients using Sente? I'm having trouble figuring out how to write a chat demo using Sente.

Thanks for your work!

[Peter Taoussanis]

Hey Sam!

it doesn't look like the list of all connected users is available in the API.

Yeah, that's right. Have been punting on this...

For the moment have left it up to applications to decide who they're interested in broadcasting to. This might be everyone that's connected, everyone that's connected with certain credentials, everyone subscribed to certain topics, etc.

Basically the application's expected to keep an appropriate index of event-types -> user-ids.

Haven't thought much about what kind of help the core API could offer. Probably a good starting point would be a simple set of all currently connected uids. Then folks could intersect that against their own subscriptions, etc.

Would that be helpful in your case? Otherwise, like I say - haven't thought much about this so ideas would be very welcome.

Cheers! :-)

[Sam Ritchie]

I think the two granularities I'd be interested in are channels and individual users.

If we exposed a global list of uids, and a list of UIDs per channel, I think we'd have enough to build up a nice set of broadcast operations. For example:

- Broadcast to this channel for users that pass this predicate
- hit these multiple channels except for these users
- re-broadcast this user's message to everyone in the channel except this user

So yeah, I think that exposing a list will get us pretty far. The missing piece, then, would be the ability for a a client to send a connection request for a specific channel.

Peter Taoussanis

April 15, 2014 10:18 AM

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Sam Ritchie

April 15, 2014 9:05 AM

Hey Peter,

I like the UUID feature, but it doesn't look like the list of all connected users is available in the API. How would you do a global broadcast to all connected clients using Sente? I'm having trouble figuring out how to write a chat demo using Sente.

Thanks for your work!

On Wednesday, February 26, 2014 6:57:24 AM UTC-7, Peter Taoussanis wrote:

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Peter Taoussanis

February 26, 2014 6:57 AM

Hi folks,

Quick post to announce a new lib release: https://github.com/ptaoussanis/sente

From the README:

Sente is small client+server library that makes it easy to build reliable, high-performance realtime web applications with Clojure.

* Bidirectional a/sync comms over both WebSockets and Ajax (auto-selecting).

* Robust: auto keep-alives, buffering, mode fallback, reconnects.

* edn rocks. So send edn, get edn: no json here.

* Tiny, simple API: make-channel-socket! and you're good to go.

* Automatic, sensible support for users connected with multiple clients and/or devices simultaneously.

* Flexible model: use it anywhere you'd use WebSockets or Ajax.

* Fully documented, with examples (more forthcoming).

* Small: less than 600 lines of code for the entire client+server implementation.

* Supported servers: currently only http-kit, but easily extended.

---

Have been using something like this in production since a little after core.async came out, and wouldn't want to go back. Note that I tweaked a few things for the public release so there may be some rough edges initially. 

An example project's included (new as of today) and there's a Leiningen alias configured to handle all the fiddly bits like getting the Cljx and Cljs to compile: just download, `lein start-dev` at a terminal, and you're good to go.

Any questions/problems/whatever, you can reach me here or on GitHub.

That's it! Happy hacking, cheers! :-)

- Peter Taoussanis

https://twitter.com/ptaoussanis

https://www.taoensso.com/clojure-libraries

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Sam Ritchie (@sritchie)

Paddleguru Co-Founder
703.863.8561
www.paddleguru.com
Twitter // Facebook

[Peter Taoussanis]

So yeah, I think that exposing a list will get us pretty far. The missing piece, then, would be the ability for a a client to send a connection request for a specific channel.

I'll be honest I'm a little hesitant to add any kind of room/subscription facilities to Sente itself...

My thinking currently goes:
* Far as I can tell (?), this is always _very_ easy to do application-side.
* Doing it application-side gives a lot more flexibility. For example, what if you've got multiple servers and want a distributed/db-backed subscription index?
* For general hygiene I prefer keeping state (like subscriptions) separate from the comms mechanism itself. Keeping subscription info in Sente makes it tricky to get to if you want to do something unexpected with it. When you control the shape+location of the relevant data/atom(s), you're free to use it and bash on it however you like.

Instead, I'd propose to just expose a set of currently-connected uids. You can then intersect that set against any subscription/channel logic you may have.

Having said all that, I'm not sure what Socket.IO's rationale was when they chose to bundle subscription semantics into the core API so I might well be missing something...

Does that make sense? What do you think? Is your concern more that maintaining your own subscription data will be a nuisance, or that it's difficult to do? Am definitely open to ideas I may not have thought of.

[Peter Taoussanis]

For those that might be following, have pushed `v0.10.0` so long: https://github.com/ptaoussanis/sente/releases/tag/v0.10.0

The server-side `make-channel-socket!` fn's return value now includes a `:connected-uids` atom key. You can deref that to get a realtime snapshot of all connected user-ids, or you can attach a watch to get notified of any changes as they happen.

This makes something like a "who's online now" trivial, and is a good foundation for higher-order stuff like subscriptions, etc.

Also updated the reference example to show how one would use this for a simple broadcast.

[keeds]

Peter,

Just wanted to say thanks for Sente. Just dropped it into an existing application. Going to make improving the UI alongside Om a breeze.

Thanks,

Andrew

[Peter Taoussanis]

You're very welcome Andrew, thanks for saying so!

Cheers! :-)

--

Peter Taoussanis

taoensso.com

[Henrik Eneroth]

Yes, thanks for Sente, Peter!

Will Sente eventually use/support Transit? :-)

[Peter Taoussanis]

Hey Henrik,

 

Yes, thanks for Sente, Peter!
Will Sente eventually use/support Transit? :-)

Good question. Haven't actually taken a proper look at Transit yet, but definitely planning to when I can find some time. Issue / PR welcome till then if you or anyone else has some specific ideas in mind :-)

Cheers!