Re: [2.x] Broadcast via WebSocket for "real time" live sports app

[James Roper]

Hi Ariel,

Play is perfect for your use case, in fact I would be very interested in hearing how you go, so please keep the questions coming.

The utilities to do what you want to do can be found in the play.api.libs.iteratee.Concurrent object.  I'll give you sample code to follow.

At 1 million users, 1 server is probably not going to be enough, so you'll need some sort of messaging system.  Here are some options (there are many more):

* RabbitMQ, here are some instructions for using RabbitMQ with Play 2: http://java.dzone.com/articles/connect-rabbitmq-using-scala

* Akka remoting

* Plain HTTP with reactive streams from Play nodes to a Play hub (in which case, the Play hub and the Play nodes have the same code for handling client connections, but the Play nodes connect back to the hub and fan out its messages to its clients)

So as for the code, lets say a client comes in requesting to receive updates for game A.  First you check if a broadcaster exists for game A, if it doesn't create it.  This might look something like this:

val (enumerator, channel) = Concurrent.broadcast[GameUpdates]

Presumably when there are no longer any clients subscribed to that game, you want to clean up the broadcaster, so there's another method with the same name, this wraps an enumerator to broadcast into something that let's you monitor whether clients are connected, and it invokes a callback when there are none, so you can do your clean up (remove it from the map, unsubscribe from upstream notifications, whatever).  So then we can do this:

val (broadcastEnumerator, broadcaster) = Concurrent.broadcast(enumerator, broadcaster => /* clean up code here */)

Ok, so you've got your broadcastEnumerator, now aside from maybe mapping it with an enumeratee to transform the messages into something that the client can parse (which actually you should do in the code above when creating it, otherwise the mapping will be done for each client), you can now send that enumerator to the client using the websockets documentation here:

http://www.playframework.org/documentation/2.0.3/ScalaWebSockets

You might want to checkout Concurrent.patchPanel, you could send this to the client instead, and patch the games enumerator into that, and that would also let you multiplex multiple enumerators (games) into the one stream, or stop one, or whatever.  Also, the broadcast enumerator will wait for all iteratees to consume each input before sending the next one.  You might not want to do this, so theres also a Concurrent.buffer enumeratee, and a Concurrent.dropInputIfNotReady enumeratee, which can be used to wrap your iteratee so that it will consume (and buffer) the input immediately, so a slow client won't hold up everyone else from receiving messages.

When you receive an update to broadcast, you just look up the enumerator for that game, and call channel.push() with the update.

With this solution, there's no looping on your part (though underneath there will be loops, but since there's no blocking, it will generally be very fast, and a lot of stuff will be done in parallel.

So, not sure if that's basic enough for you, but feel free to ask more questions.  But I can tell you one thing, this will scale, you'll be able to implement this using Play using far less servers than other frameworks, and far less code too.

On Wednesday, 19 December 2012 06:46:45 UTC+11, Ariel Scarpinelli wrote:

Hi all,

I'm evaluating using Play 2.1 to dev an app that has to broadcast data updates from about 100k to 1m users. The app is to follow a live sports match.

My question is: How well Play could handle this kind of load?

I've been looking at the websocket-chat example. It seems to be using a for loop to walk thru every connected user to broadcast a message. Doesn't seem to be very performant (just an impression). Is there any better option for broadcasting?.

In the case that the volume of users requiere to run the app in different servers, how should I configure these to communicate each other in order to get updates and broadcast them to their connected users?

As a side note, as all the traffic is unidirectional, my other option is to "cook" updated JSON files on each new event and upload them to S3. Then use polling to S3 on the client side. I'm open to suggestions about this option or any other you think could work.

Thanks!

[Will Sargent]

There are two projects that I know of that focus on a consistent
fallback model for real time applications -- socket.io, and vert.x. I
don't know how well they integrate with Play, but there has been some
discussion of integrations in the forums.

Will.

On Fri, Jan 4, 2013 at 3:28 PM, Grant Beaty <gbe...@gmail.com> wrote:
> James & Ariel,
>
> I'm also developing a real-time sports-scoring app using play2. I suspect I
> won't have anywhere near 1m users, though that would be nice :) The app is
> Akka based. Data is gathered from an applet at the location and sent to my
> server via Akka remoting. The data is then persisted to a database, compiled
> into client-side updates and sent to client handler actors (which might be
> remote), which handle the iteratees.
>
> However I'm kind of stuck on the best way to update the client browsers in
> real-time. There are websockets, which require some logic to handle
> reconnections and lost updates. There is EventSource, which handles
> everything I need but doesn't work on IE10 (sigh). There is Lift, but it
> presents some of its own challenges (play2 is much more performant in my
> testing) and I really prefer working with play2's statically-typed
> templates. Then there are various javascript means to substitute
> long-polling in place of EventSource or Websockets where the latters are not
> supported.
>
> I suspect Websockets would be the most performant solution, given that they
> do not have the HTTP overhead of comet or EventSource?
>
> Anyway, any advice on the most painless way to implement this is great
> appreciated!
>
> /Grant

> --
>
>

[James Roper]

If you only need one way communication, server sent events are fine, and there is no overhead here as compared to websockets (in fact, sse might even be lighterweight).  And the nice thing about SSE is automatic reconnections.  But, it's not exactly hard to implement automatic reconnections yourself, just add an onerror and onclose handler that reconnects after a timeout.

Although Play itself doesn't, out of the box, support any fallbacks if these aren't supported, it's really not hard to implement the fallbacks yourself.  In your javascript, you'd have a single callback that accepts a parsed message, and call this both from your websocket/sse callback and your comet callback.  A good solution here might be long polling with jsonp, since I assume the sports updates are not sensitive data.  Here's docs on that:

http://www.playframework.org/documentation/api/2.0.4/scala/play/api/libs/Jsonp.html

On Sat, Jan 5, 2013 at 10:28 AM, Grant Beaty <gbe...@gmail.com> wrote:

James & Ariel,

I'm also developing a real-time sports-scoring app using play2. I suspect I won't have anywhere near 1m users, though that would be nice :) The app is Akka based. Data is gathered from an applet at the location and sent to my server via Akka remoting. The data is then persisted to a database, compiled into client-side updates and sent to client handler actors (which might be remote), which handle the iteratees.

However I'm kind of stuck on the best way to update the client browsers in real-time. There are websockets, which require some logic to handle reconnections and lost updates. There is EventSource, which handles everything I need but doesn't work on IE10 (sigh). There is Lift, but it presents some of its own challenges (play2 is much more performant in my testing) and I really prefer working with play2's statically-typed templates. Then there are various javascript means to substitute long-polling in place of EventSource or Websockets where the latters are not supported.

I suspect Websockets would be the most performant solution, given that they do not have the HTTP overhead of comet or EventSource?

Anyway, any advice on the most painless way to implement this is great appreciated!

/Grant

On Tuesday, December 18, 2012 9:28:37 PM UTC-5, James Roper wrote:

--
 
 

--
James Roper
Software Engineer

Typesafe - The software stack for applications that scale
Twitter: @jroper

[Grant Beaty]

James,

I thought SSE used HTTP, and thus incurred the header overhead, while WebSockets were nearly TCP? This link is one datapoint in favor of WebSockets, though the difference is small.

In any case I'm using normal WebSockets with an automatic reconnect, as IE10 has no SSE support (neither does play2, I guess?). Upon connection, the server synchs the full state (of an arbitrary ServerStateful element on the page) with the client. Afterwards, the client is only sent incremental updates. Since TCP ensures order, the state will always be consistent unless the connection is broken. If the client reconnects it gets the full-server side state again. I hope to make seamlessly render ServerStatefuls as HTML or WebSocket-updated values, depending on whether or not they're expected to change.

Thanks for the help & link.

/Grant

[Yann Simon]

Hi Grant,

2013/1/27 Grant Beaty <gbe...@gmail.com>:

> James,
>
> I thought SSE used HTTP, and thus incurred the header overhead, while
> WebSockets were nearly TCP? This link is one datapoint in favor of
> WebSockets, though the difference is small.
>
> In any case I'm using normal WebSockets with an automatic reconnect, as IE10
> has no SSE support (neither does play2, I guess?).

play2 has support for Server Sent Events.
(I wrote about it:
http://yannexpress.blogspot.de/2012/08/handling-data-streams-with-play2-and.html
demo: http://wiki-growth.herokuapp.com/)

> --
>
>

[Guillaume Bort]

Grant,

Avoid WebSockets if you can. WebSockets work well on local network and for niche projects when you need high performance 2 ways communication. But on the internet, it's a mess. It's not standard HTTP and many proxy will block it.

ServerSent event is really the best way to send real time notification to a browser. There is no overhead at all compared to Websocket, although it supports only one way communication.

Also, it's easy to fallback to plain Comet on older browser. At the Play code level it's just an Enumeratee switch (Comet() vs EventSource()).

--
 
 

--
Guillaume Bort, http://guillaume.bort.fr

[Alex Jarvis]

This is why I'd recommend only using secure WebSockets (wss://).

In fact I first noticed http header manipulation (or incompatibility with the handshake mechanism in http proxies) 2 years ago with a Play 1 app when I was testing an application on my mobile phone operator's network. They just didn't work at all.

Encrypting the communication mitigates against this problem, but of course you need to observe the requirements for your project and decide if WebSockets are indeed the right choice as Guillaume suggests.

[ngocdaothanh]

You can try Xitrum:
http://ngocdaothanh.github.com/xitrum/

Demo:
http://cntt.tv:8000/sockjs_chat

It has many built-in features that you need:
* SockJS for WebSocket and fallbacks (see https://github.com/sockjs/sockjs-client)
* HTTPS for WebSocket clients behind broken proxies
* Clustering for distributing load to many servers


On 2012年12月19日, 午前4:46, Ariel Scarpinelli <triforc...@gmail.com>

[Alex Jarvis]

Hi James,

I've also been doing some research with regards to a pub/sub + unicast type of design for WebSockets and distributed nodes and notice that you mention RabbitMQ. Have you got any experience using this? It would be appreciated if you could give any advice.

So far I've considered the following technologies to achieve multi-node scalability, each with their own advantages and differences:

- Mongodb capped collections

- Redis pubsub

- Akka remotes & Akka cluster (to use discoverability instead of configuration of addresses)

- Eventsourced Akka actors to ensure deliverability even on remotes and also improved recovery via journaling.

- ZeroMQ pubsub with Akka

Personally, I would prefer to move into a more distributed type of design where there isn't a single point of failure / bottleneck and so have been leaning towards ZeroMQ pub sub or Akka cluster. If anyone has any tips / pointers before I embark on this quest of prototyping that would be extremely helpful. One thing that I've just learnt is that ZeroMQ can be configured to use tcp multicast which could be interesting..

On another note, I think that the current WebSocket chat example probably isn't tuned for the maximum performance possible, given that there is only one actor processing all of the messages for all of the clients? Would you advise creating a balancing dispatcher that used multiple processing actors instead (and moving the connected clients map out into the singleton, but ensuring it's safe for concurrent access)?

Cheers!

Alex

[Grant Beaty]

Alright, SSE it is.

James,

Akka clustering is what I'd hoped to use, though I shouldn't need anything distributed for some time so I haven't done much research here.

I would think most of the performance gains would be had in multicasting SSEs to clients. It appears Kaazing supports something like this, though obviously its not open source.

/Grant