Javascript EventStore using LocalStorage?

[Adam Mills]

I am looking at doing an occasionally connected app that is entirely hosted in browser, backed by an EventStore that uses LocalStorage (events are also published to a central server, merged and distributed)

Has anyone done anything like this?

Any code that is shareable? Especially a LocalStorage backed EventStore?

Thanks

Adam

[Greg Young]

You can grab some of the code from the Event Store sample chat app
(github/com/eventstore/eventstore). As a backend it already has js
libraries to handle subscriptions to browser (over atompub and you can
host projections in the browser if you want).

As for the localstorage I would just store in json. I can help you
with this if you want as it is something that would be nice to drop
into our client API as well.

--
Le doute n'est pas une condition agréable, mais la certitude est absurde.

[Adam Mills]

Thanks I'll have a look.
Not sure what you mean by "store in JSON"? I want persisted storage across
browser restarts client side, so had planned to store the events in the
browser LocalStorage, as JSON

[Jonathan George]

I'm doing exactly the same thing at the moment and am storing my events in the HTML5 IndexedDB (obviously this restricts the browsers that can be used, but I'm ok with that at the moment.) My starting point was a Javascript port of Greg's "simplest possible thing" sample - I understand that this is not exactly a full blown event store, but I have a fairly tight deadline for my first phase which is primarily a proof of concept - assuming all goes well I will be revisiting the infrastructure bits in a month or so.

My first pass at porting Greg's C# to JS can be found here - https://bitbucket.org/jon_george1/javascriptmvc-cqrs. It's a bit heavier than it might have been because I'm using the JavascriptMVC framework (http://javascriptmvc.com/) and it comes with a number of health warnings - the biggest being that because interactions with IndexedDB are async you can run into some major concurrency issues with the code as-is. In the codebase for the app I'm working on, I ended up using a plugin (http://www.bennadel.com/blog/2326-jQuery-whenSync-Plugin-For-Chaining-Asynchronous-Callbacks-Using-Deferred-Objects.htm) to force things into a more synchronous model. There may well be a better way of doing this.

Obviously I can't share my client's codebase with you, but I can pull out bits of the code to share if it would be useful for you and I'm more than happy to get into more detail on this - it would be good to have someone else attempting the same thing as me to bounce ideas off :-)

Cheers

Jon

[Greg Young]

Thanks for sharing that! The main point of an indexdb store imho is in occasionally connected scenarios eg: wAiting to sync.

[Adam Mills]

Thanks, it’s good to see people taking up Promises (deferred objects).

I have also been toying with the idea of retrieving AR’s from the server and storing them in localstorage as JSON. Then having them process new events and storing the events for sync.

However this requires domain duplication, a domain on the server for hydration.

If I go down the client side ES path I should be able to keep one copy of the domain on the client, the server is just a pure ES. Merging would be done on the client.

 

@jon - In your example where are you doing your event sync and merge? Or do you not push the events of the client?

 

Adam

[Greg Young]

Node.js server? You can run projections at all places (db,node,browser)

[Adam Mills]

Had a quick look at the projections in ES, very cool.

During my offline mode, do you have a mechanism that allows me to push events through the defined projections in the browser? Without going to the server?

When offline my domain would keep processing commands and producing events. I would need those to be processed by the projections to keep the read model up to date.

 

Does ES have any in-built store and forward capabilities if the server is not online?

 

Thanks

[Jonathan George]

I'm intending the same as you - at least to start with, the server will be a pure event store. To start with it will just be persisting the events but I expect that in the future I will need to publish them as well to do things like analytics, MI reporting, etc that will be surfaced in a totally different interface to the main tools I'm building.

What I've done so far is:

- built a client side event store that uses the IndexedDb (IndexedDbEventStore.js).

- built a basic service to allow the client to push/pull events. When you pull, you can specify a starting revision.

- built a client side class to talks to this (ServerEventStore.js).

- built a class that wraps both IndexedDbEventStore.js and ServerEventStore.js, and exposes the same interface as IndexedDbEventStore - i.e. Decorator pattern. This is my "sync manager" class, and this is what the repository uses to save/load events.

Whenever the repository saves events, the sync manager stores them locally using the IndexedDbEventStore class then attempts to push them remotely using the ServerEventStore class. At this point, one of two things will happen.

- The server will accept the new events and persist them

- The server will reject the new events because it has received new events since the last time the client pulled. In this case, the client is responsible for resolving the problem, a sequence that will normally involve getting the new events from the server, rewinding the client position to the point where these new server events can be applied, then attempting to re-apply the client's new events on top of this (may be automatic, but conflicts will require user intervention). This will result in events that can be pushed back to the server.

I've been keeping this as analagous to the way Git works with remotes as I can. The conflict scenario I've described above is essentially working the same way as rebasing a local branch on it's corresponding remote.

The sync manager will also poll on a regular basis for new server events and will handle pulling any new events and persisting them locally (which will also result in them being published for consumption by my denormalizers.)

I've got quite a long way with this but it's not yet complete. The biggest question I have yet to answer is how, in the conflict scenario I described, I can keep the read models correct without having to totally rebuild them from the full event stream - potentially expensive, especially on slower machines. However, I'm expecting that I'll need to introduce snapshots at some point which might help alleviate this problem - if I can do something whereby the denormalizers can deal with snapshots as well as standard events.

Hope that all makes sense... is this broadly in line with your thinking  or have you taken a different approach?

Jon

[Adam Mills]

That was exactly my vision. In my domain the order of UnitsOfWork don’t really matter, so I don’t think I will need to rebuild the read model. I can just apply the new events.

This method saves duplication of domain on the server, but requires an EventStore.js (which you have written) on the client and some simple “service bus”, I had planned to use Rx.js

 

However Greg has got me thinking (whether intentionally or not J) of using his EventStore and its client projections.

I would host my domain in node.js and provide a thin service layer to retrieve my AR’s, which would all be preloaded into localstorage when the app started for the first time.

Then commands would be applied to my local domain and the events pushed to the server and into my local ES projections.

 

It just depends how much the EventStore provides for offline use (I haven’t had time to dig yet). Ideally it would handle the caching of events and posting to the server once a connection was made (with some hook for conflict resolution) as well as allowing for events to be pushed through the local projections (rather than a server round-trip).

 

I’ll go for the option that requires me to write the least infrastructure code J

 

Thanks for sharing, it’s nice to know the ideas you come up with aren’t entirely of the wall.

Adam

[Adam Mills]

Are you able to pull out any of this infrastructure code from your current code based?

Very interested in the IndexedDBEventStore and whatever does your merging.

 

How do you handle version numbers? Do you rewrite the client stream? So nothing is “committed” till it hits the server?

 

Thanks

Adam

 

From: ddd...@googlegroups.com [mailto:ddd...@googlegroups.com] On Behalf Of Jonathan George

Sent: 14 November 2012 12:53

[Jonathan George]

Will have a go but the merging code is not complete - it currently doesn't handle anything other than golden paths of pushing events to the server that are not rejected and doing a pull of the full event stream.

My intention is to have a merging process that's analagous to a git rebase - so once I have pulled changes from the server, I will attempt to "replay" the new client events on top of them, modifying their version numbers as I go. If any of the events conflict the user will have some kind of UI to resolve the conflict, potentially resulting in new events. Once all new client events are dealt with they will then be pushed back to the server which will hopefully accept them.

In case that's not obvious.

• Server has events 1, 2, 3. Clients a and b pull these.
• Client a creates 2 new events 4(a) and 5(a) and pushes back. Server now has events 1, 2, 3, 4(a), 5(a).
• Client b creates 2 new events 4(b) and 5(b), with no knowledge of the fact that the server has moved on. 
• Once these have been persisted locally, we attempt to push. 
• The push is rejected, so we request any server events with version number > 3. 
• Server passes us 4(a) and 4(b).
• We (somehow) pull 4(b) and 5(b) out of our local event store and replace with 4(a) and 4(b). 
• Our local store now looks exactly the same as the server. 
• We then try and apply 4(b) on top of this. 
• Our business rules tell us there are no conflicts so we change the version number of 4(b) to 6(b) and add it to our local event store. 
• Our business rules tell us that 5(b) conflicts with 4(a) so we ask the user, who elects to drop their change. 
• Our local event store now has 1, 2, 3, 4(a), 5(a), 6(b). 
• We push 6(b) to the server which accepts it. 

As far as the system goes, the server is the system of record - the Javascript client is responsible for keeping itself aligned with the server, handling and resolving any conflicts.

Will have a look at what I can extract from my codebase for you. Feel free to nag if I don't get back to you in the next couple of days.

Regards

Jon

[Greg Young]

Have you seen my talk on occasionally connected systems? It discusses exactly what you are discussing here and how to do it (hour long talk). The trick is that in the client you store original commands for transient items.

[Jonathan George]

Yes, have watched this. I remember in your talk you store commands on the client and send these back to the server, which then attempts to apply them and can reject them at that stage. Part of what made this possible was being able to host your domain on both client and server which allows you to know what events each command will produce, and therefore how to update your local read models when offline.

This sounds like a good way to do it - the only thing that put me off was the question of hosting the same domain both client and server side - as far as I know the only sensible way for me to do this (given my domain is 100% in Javascript) would be to use node.js on the server - not entirely a deal breaker in itself but not something I can do now as I felt my initial timescales were too tight (never used node.js before). I haven't ruled out going down that path in the future, but for now it feels like the approach of storing and attempting to synch events rather than commands will be enough.

Would be interested in any drawbacks you think this approach might encounter. As I'm getting a bit further down the road I'm starting to get a feel for what's probably the main one - when a single command generates multiple events, I'm going to get to a point where I struggle to present conflicting events to the user for resolution in a form they understand - e.g. the user knows they took action A, they won't understand when they are asked to resolve conflicts with resulting events X, Y and Z.

Thanks

Jon

[Greg Young]

Move command processing into the client and then post the events... The commands that are transient have been applied in memory. When I get new events I just append them to my committed events the replay my transient commands (assuming no conflict). This makes it quite simple as the local store only ever contains committed events.

[Jonathan George]

Not sure I understand...

"Move command processing into the client and then post the events"

- This is what I am doing, but not what I thought your talk was suggesting. I thought in your talk you processed commands client side when offline, but then posted the commands to the server when online, which then resulted in "official" events from the server (as opposed to the "unofficial" events you'd generated on the client side.

"This makes it quite simple as the local store only ever contains committed events"

- So the local event store only ever contains events that have come from the server, whilst the events you generate client side are held outside this store, in memory? This kind of makes sense - just not sure what the point is in differentiating, as you still have the issue of how to apply these events to your read model in such a way that they can be removed if there are conflicts?

[Greg Young]

Yes but there is nothing that says you need to have two domain models.. The one at server verifies. If you trust your clients just do it in the client.

[Yevhen Bobrov]

The following be a stupid idea, but as I understood one of the issues that you have mentioned is that you dont want to fully replay (wipeout and rebuild) read models every time and those transient events if conflicted are forcing you to either rebuild full read model or find a way to revert them somehow.

But what if you just host two versions of the read model - one built from official source, keeping it as a shadow copy and the one (the clone of master read model) to which transient events are applied. 

If there problems you just throw away the transient model that you're using, make a clone of master, do an incremental update with new events you got from official source, resolve conflicts and then replay your transient event on top of it.

17 нояб. 2012, в 00:36, Jonathan George <j...@jonathangeorge.co.uk> написал(а):

[Tom Janssens]

I would not take this approach: exposing events from the server might expose secure information, and it might require to much bandwidth; just process it on both server and client....

Op zaterdag 17 november 2012 00:01:08 UTC+1 schreef Yevhen Bobrov het volgende:

[Greg Young]

That's pretty domain specific though no? I wouldn't put as a blanket rule that it's a bad idea.