According to the discussion thread below, the current approach to data
integrity in Weave is wiping server data every time there is a
mismatch between Weave version that wrote the data and currently
installed Weave version. In future this should be changed to warn
instead of wiping data if the installed version is lower than the
version the created the data. I guess this solution is good enough -
but only if all data comes from the Weave extension itself. As soon as
other extensions start registering their engines there will be
trouble.

In particular, Adblock Plus 1.1 will most likely be able to
synchronize through Weave. However, for the list of filters it will
ignore positioning of the filters because tracking position changes
isn't currently possible in Adblock Plus (it was never necessary). A
future version might fix this - but if a computer with this version
and another with Adblock Plus 1.1 are synced there will be data loss.

I think the best solution here is versioning the data format per
engine. Each engine should have a number representing its data format
version, it should increase on each format change. So it would be
possible to wipe the server only for the engines where it is
necessary, and it would allow Weave to disable a single engine if it
is outdated (as well as inform the user about it). There should
probably still be a global data format version that should change
whenever some format detail changes that is common to all engines.

Does that sound like something Weave would implement? And as a side
question - any hints on how to deal with this issue in Adblock Plus as
long as it is not implemented?

Wladimir

On Sat, May 16, 2009 at 10:12 AM, Wladimir Palant
Right now for these early versions of Weave, we're perhaps wiping more
often than necessary because we bump up the version and cause
mismatches. But eventually we'll want to bump the weave version
without changing the server version number.

Part of the reason for wiping is that base record structure might
change -- independent of the engines. For example, we changed how
deleted records were stored on the server, and that would be
incompatible for all engines.

Engines could make use of their own versioning and decide to wipe on
their own. But actually, engines can put an-opaque-to-weave version
field in each record that it uploads, and the engine can decide what
to do with different version numbers.

But currently none of the built-in engines do that, so perhaps we
might incorrectly decide to bump the weave server version just because
we changed a built-in engine. (I don't think that line of reasoning
has happened before, but that's only because we bump up  the version
every time.)

Dan: Down the line, what types of changes would warrant a weave server
version bump?

Either way, it might be a good idea to incorporate versioning into the
built-in engines.

Ed

On May 17, 4:30 am, Edward Lee <edi...@gmail.com> wrote:

Putting a version number into each record sounds pretty wasteful. I
guess, simply ignoring records with wrong version number will lead to
inconsistent data. Disabling engine or forcing a server wipe would be
a better course of action then. I'll have to check whether this can be
done reliably.

Wladimir

but regardless of how it is stored and how it behaves - there should
be a warning - "will wipe all server data" [OK, Cancel].

On May 17, 6:27 pm, disaster123 <ste...@priebe.ws> wrote:

Yes. And it is nuances like this one why I would prefer to leave it to
Weave rather than having to worry about it myself.

On May 16, 2009, at 8:12 AM, Wladimir Palant wrote:

I don't think this is quite correct--if an extension adds a sync  
engine, Weave would ask that engine to re-sync at the same time that  
it asks other built-in engines to resync.  So it really is no different.

The problem is if Weave gets used to put data on the server that is  
not for synchronization (it's data that lives only on the server).  
Then a server wipe would be very bad.  Currently we don't have any  
data like that, but we may in the future.

Versioning specific engine data does make sense anyway, for cases when  
a specific engine's data changes but the overall server data format is  
the same.  I wonder how to best implement that though.  Maybe just an  
extra metadata record somewhere?

There is other metadata we need to upload per-engine, currently we  
sync keys with set names ('keys/pubkey' for example), but I think we  
should use a guid and save a record pointing to where the pubkey/
privkey, etc are.  That way if you ever delete the key it would be  
obvious when data was encrypted with a non-existant key.  We could put  
this metadata in the same place as the above version.

Dan