Design Patterns for Sleepy Clients

[Michael Karliner]

Hi,

For the last few days I've been working away with socketstream
pub/sub on a little app that is mainly targeted on mobile devices,
and I've come across a design problem that I thought I'd ask
the group for advice on.

Fundamentally, mobile browsers have a habit of going to sleep
when idle for a while, or rather the device does.
This means that pub/sub becomes unreliable as events can
be lost during sleep periods. Obviously, I can use a couple of
different strategies to get round this:

1) Ensure that messages contain all useful state on each transfer,
so the next message in the pipe will update the client to current
state.
2) Provide an rpc call that returns current state, (although this
might be subject to timing holes).
3) Provide a 'catchup' channel that periodically issues the current
state.

I wonder if anyone has any better patterns?
Also, is this going to be a problem with Real Time Models?

cheers
Mike

[Leandro Ostera Villalva]

Hi Michael,

What about a pingback from the device that won't be made while sleepy? So you can stack the next states (if any) and periodically re send a message to the device or wait until the device pings the server back (if there is any state or event client-side that would allow you to ping the server on awaking then there's when you could start publishing again).

Hope it's of any help!

--

Regards,

Leandro Ostera Villalva,

Read my technical blog
Check my latest projects

[Michael Karliner]

Hi Leandro,

    Thanks for that. That's a pretty good suggestion.
   
    I found this idea
    http://stackoverflow.com/questions/6543325/what-happens-to-javascript-execution-settimeout-etc-when-iphone-android-goes

    on StackOverflow for detecting client sleeps so I can trigger a
    call back.

    Mike.

[Leandro Ostera Villalva]

Hacky, but seems to work. Making the callbacks fire only once instead of two or more times is a matter of adding a flag inside the process being executed (or a nasty global flag outside it).

It would definitely be more elegant to have an on-awake event to hook to. For that matters, this SO question looks better.

Also here they suggest to use phonegap. No idea if that would be of any help.

[Skipster]

This is where durable reliable state change technologies like ShareJS and operational transform come in. pub/sub is transient in nature, whereas the problem you are actually solving is synchronization, which needs to work if the device is waking up after 1 month. In a RequireJS/Derby app, your server would be storing a JSON diff of every update to a collection, alongside the collection itself. And ShareJS calls that the "ops" or JSON-diffs table, whereas your usual collection is called the "snapshot". So then what your client should do in response to a pubsub event is trigger its "fast sync" process, which is simply doing an RPC to ask the server for all the JSON diffs (or "ops" documents) since your last sync, which is a watermark in the form of a MongoDB ObjectID of the last ops document.

The great thing about SocketStream is that you can do one of these fast-syncs in 2ms. So whenever you get notified of anything, all you really need is the collection name. Plus, you put your fast sync logic in "ss.server.on(connected)", so that your PhoneGap app just automatically starts getting the updates whenever its WAN link comes back online.

It goes without saying that your PhoneGap app should be built against a local database, so that it is not just another one of these fake mobile "tethered" apps. So then what you're left with is doing the data upload and conflict resolution part of your PhoneGap app, which is by nature very business domain specific.

You could use a service like Simperium if you wanted to outsource all of this, but you give up hosting your own data, and it's also too expensive for my taste.

David

[Michael Karliner]

David,

    thanks very much for that introduction to state change technologies.

    I wonder if you could point out any problems you'd foresee in using ShareJS with
    SocketStream?

    Mike

[Skipster]

The problems with using ShareJS with SocketStream are the same for using ShareJS with Derby. ShareJS is not ready for use, it does too much and can't be pulled apart (for example, to use SocketStream's transport instead of its own bindings to SockJS). It also has an open showstopper bug relating to the JSON OT API being flawed in relation to array operations, and everything after the array insertion going wrong. Proof beyond all, the DerbyJS sample apps DON'T WORK, as several posters and myself have observed. Rearrange a few scrabble letters, and the state tree has already fallen apart. The ShareJS author, by his own account, has not worked on his project for a long time, to take it further to completion. He is writing games instead.

So ShareJS is useful on a conceptual level only. I think its MongoDB and CouchDB adapters, plus the node JSON diff-patch-merge module, are the important reference points for doing a mobile synchronization solution for SocketStream. It's what I'm doing, also with PhoneGap and SocketStream. I'm using an Ember Data style of API in the client, which is a good place to produce a JSON diff during a save, because you have the before and after state of a model there.

David

[Carsten Kraus]

Hey guys,

just getting into SocketStream, maybe you could help me find out: is synchronisation/conflict resolving on the roadmap?

I have been watching Owen's talk: http://www.youtube.com/watch?v=LOS1lpWXphs. 

At the end of the talk, on a slide titled 'Roadmap to 1.0'(around here), there's a feature 'Connection Handling' - is that pointing in the direction?

Or is the whole sync business too closely associated with the notion of 'models', which as Owen states in his talk are out of scope for SocketStream?

Thanks for a pointer!

Carsten

[Owen B]

Hi Carsten

Yes I can confirm that having rock solid connection handing, especially for mobile clients, is something I do want to see as a core part of the framework.

This comes under the 'essentials' category for me.

I'd like to see what we can do with 0.4 in this regard over the coming months.

Owen

[Skipster]

Connection handling is a little weak right now (because I'm with Heroku). Long polling works great, except for one particular version of IE 9, that stalls for 45 seconds before downloading the CSS & JS "beach-head".

So in the "essentials" category, I would add SockJS or Pusher or PubNub alternative transports, so that there is at least a baseline of SocketStream working on all browsers, where native end-to-end WebSockets aren't possible.

David

[Carsten Kraus]

@Owen: thanks for the prompt response!

Could you elaborate a bit on the term 'connection handling'?

Is it rather meant in regard to the actual transport issues like (re-)connecting/falling back to other transports? 

Does it also contain strategies for data synchronization for situations the OP was describing?

Thanks again!

[Skipster]

It looks like Heroku is also to blame for the long-polling problems, which also don't work through their proxies. SockJS has apparently dealt with this, but Socket.IO has not:

  https://github.com/sockjs/sockjs-node/issues/57#issuecomment-5242187

David

[Owen B]

Hi David

Yeah, another good reason why it's good to ensure your app is independent from the transport layer!

I'm hoping 0.4 will standardise the API for transport layers so others can make transports for PubNub, Pusher, etc.

Carsten: My view is that anything that can be handled by the websocket transport (e.g. transport negotiation) should be. Anything we have to do to ensure reconnection works well (esp over mobile clients) should go in the framework. Anything to do with model and data synching should not be supported by the core, but possible to build using the Request Responder API.

Owen

[Carsten Kraus]

Hi Owen,

Anything to do with model and data synching should not be supported by the core, but possible to build using the Request Responder API.

uhu, thats how I understood the talk you held. I also think the actual model, object/document mapping would be nice to have in a 3rd party form.

But what about message queuing, for example when a client temporarily lost network, but still should be able to work with the app and its state, trigger rpc calls, etc?

IMHO issues like this are essential in the realtime web and should mostly be taken of the shoulders of framework users, just like auth/session handling are mostly trivial for Django/RoR/etc users.

Is a story for these things on the socketstream roadmap?

Cheers, 

Carsten

[Carsten Kraus]

I just took the default chat app for a test ride in Firefox, playing with FF's 'work offline' feature.

It seems message queuing is already implemented on the client side, at least for rpc calls? : D

[Owen B]

Hi Carsten 

Yes indeed. Requests to Socket.IO are automatically buffered; however that's not *always* what you want (e.g. when bidding on something in realtime).

When things settle down with the new API I'm hoping we'll see several modules which implement different type of messaging patterns.

Owen