Dexie.Syncable fails to initiate a sync?

[do...@beulahelectronics.co.uk]

Hi,

I've just started using Dexie, I especially want to make use of it's syncing capabilities.

I have implemented a basic sync protocol, and I'm fairly happy that it can fetch data and keep things up to date across clients and the server.

The one big issue I have is trying to figure out *when* Dexie wants to initiate syncing.

My simplest use-case is to sync user settings/preferences between clients. To this end, I've set up a 'usersettings' schema, and implemented a protocol, finally calling db.syncable.connect() with appropriate arguments. Unfortunately, I am unable to share any specific code with you.

If I clear out the browser's databases, and load my app (like a user would for the first time) - Dexie happily creates everything, syncs, and writes data into the database.

The issues start when I reload the page. As far as I understand, before calling db.syncable.connect() I do need to know the db.syncable.getStatus() for the url. In my case, I am only calling db.syncable.connect() if the status is -1 or 0. In other cases, I believe Dexie knows the syncer exists and should start the polling.

Most of the time, Dexie never starts polling.

I've dug through the code, and established (via adding copious amounts of logging), that Dexie.Observable has an internal polling loop which repeatedly calls cleanup(). At some point, on some client, it should establish whether that client becomes the sync master, and the cleanup handler in Dexie.Syncable will then initiate the db.syncable.connect() call. I understand why this is done - we only need one active client to perform the syncing.

What I have observed though, is that even with a database which has only ever had one client connect (i.e. start from empty database, load, then reload in the same tab) - the Dexie.Observable cleanup() polling never decides that this tab is the master, and hence no syncing ever starts.

Other times, if I do have 2 tabs open, one of them will eventually claim to be master and start the syncing - however the time interval to reach this decision far exceeds any of the sync protocols' intervals. It can take minutes for any syncing to happen.

Am I maybe doing something wrong? Anyone else seen this behaviour?
I'm developing against Firefox 38.4.0 on Centos 6.6.

Regards,
Doug Hammond

[David Fahlander]

You have understood exactly how it is supposed to work. Hope you're using the latest version from github and not the npm because some issues have been adressed lately with the addons. Still, we have the problem with detecting which node to be master after a reload of the page, it can take minutes before it is detected that the previous master is dead. However I don't recognize the other issue - that it would never get in sync again.

I think that these addons would benefit from being rewritten and restrutured. For example, the use of localStorage for inter-tab communication could be extracted as a separate framework that didnt nescessarily rely on localStorage, but be able to fallback to simple polling or using service workers.

For me personally, the addons are important because I use them in my own applications. So far however, they behave okay in regards to hooking up with sync as needed. However, if exceptions occur in the sync provider (as happened for me one time), the "never recap" problem could occur.

I use the addons on a WebBrowser component (running IE11 engine) in a windows application. One of the unit tests runs fast on IE but slow on Chrome, so there might be some issue to look into further.

You seem to be understanding the ideas behind and if I were you, I would consider rewriting the addons from scratch unless you could wait and hope for a solution.

My plan is to fix and release a stable synching mechanism in the coming months. Probably by rewriting the addons and separate them into smaller and testable

modules. Until then, I'm thankful for all types of contributions on the addons.

David

[David Fahlander]

Also, to answer your questions, yes, it's ok to call connect() on every startup, even when a remote end point is already known. A sync connection is persistent by design, so if you call db.syncable.connect() and then reload the page, you don't "have to" call connect again, but its ok to do so.

The addon will try connected on db.open() but as you also mentioned, it can take up to minutes until the previous master node (other tab that has been closed) is detected gone. There's code that should speed this process up by listening to the beforeunload event and set a localStorage entry: 'Dexie.Observable/deadnode:<nodeId>/dbName' to the value "dead", which should make a new node know that the master is dead and take over the leadership. Unfortunately, this does not seem to work at all times, and then the old master isn't detected dead until it has stopped responding to heart beats for about a minute. Also, if a page crashes, it wont call onbeforeunload.

[do...@beulahelectronics.co.uk]

Hi,

Thanks David, it's nice to know I wasn't completely off the mark.

What I've ended up doing is to ditch Dexie.Syncable and write a simpler syncer implementation which can call into the same ISyncProtocol interface.

I also did a little work with observing the 'cleanup' event and now get an instant notification in all tabs when the master changes. With the help of 2 extra booleans, I can reliably fire an event to let the system know if it should start or stop the syncs.

In my world I store no state about the syncer, but let the Observable master/slave state change do what you would call the 'connect'.

One other quirk I noticed which didn't appear to be documented was that Observable will not fire change events on tables which start with underscore (e.g. '_syncerdata') ? That took a while to figure out.

Cheers,
Doug.

[David Fahlander]

Sounds interesting!

Would it be possible for you to share your code?

ps. sorry about the lowercase- (or $-) prefix quirk ;(... .

[Doug Hammond]

Hi,

Unfortunately my code is part of a proprietary application used internally by the company I work for, so I won't be able to share any code. I haven't made any modifications to Dexie itself either.

Regards,
Doug

[David Fahlander]

I recently detected a serious bug in a commit to Dexie.Syncable for 21 days ago. This could be an issue. Resolved in a recent commit right now: https://github.com/dfahlander/Dexie.js/commit/e7f8c2c5b0a40daffac327c03c791b6e3394a511.

[TREHOUT Nathan]

This behavior still here in 2021, the workaround is to dig around the _syncNodes and manually set the master when it's the right moment.