OI Cloud Sync Sequence Diagram

[Vettukal Vincent Mathew]

hi,

This is my first attempt to make any UML diagram to try to show how I plan to implement the Synchronization process. The lastTS is the time of the last sync done by an application using OI Sync it is stored inside the OI Sync. Below is the diagram..

And should I make a just different java Class or a separate package in OI Notepad for synchronization process.

Thanks

[Friedger Müffke]

Can you add the process what you want to do about the possible time difference between server and client?

Please use a new package in OI Notepad.
Cheers
Friedger

2012/5/28 Vettukal Vincent Mathew <vett...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "OpenIntents" group.
To view this discussion on the web visit https://groups.google.com/d/msg/openintents/-/JJ-G5a6DFogJ.
To post to this group, send email to openi...@googlegroups.com.
To unsubscribe from this group, send email to openintents...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/openintents?hl=en.

--
OpenIntents UG (haftungsbeschränkt)
Suarezstraße 41
14057 Berlin
tel:+49 30 60982220
mailto:in...@openintents.biz
enum:+493060982220

Vertretungsberechtigter Geschäftsführer: Friedger Müffke
Registergericht: Amtsgericht Berlin (Charlottenburg)
Registernummer: HRB 118597
Ust-IdNr: DE265992701

[Vettukal Vincent Mathew]

To resolve the possible time difference, we could only assign Timestamps on local Client Apps on incrementation. That is when a new Entity is created it is assigned a TimeStamp of 0. Then when it is sent to the OI Sync, the OI Sync assigns it current Time from the server. And then it is sent to the server and back to the Client App with new assigned TimeStamp.

When the Entity is modified instead of assigning it a new TimeStamp based on the time of Client. The TimeStamp(this TimeStamp was assigned by server on previous sync) is incremented by some constant value. When the next time Sync is done all the entities greater than the lastTS is sent to OI Sync, then assigned a current time from server to the modified Entities. Sent for persistence to server and sent back to the client.

Every time the sync is done LastTS in the main OI Sync is updated from the server. Below are the Sequence Diagram to show it.

                                                              When a new Entity is created.  

When an Entity is modified

 
On Tuesday, May 29, 2012 3:55:53 AM UTC+5:30, Friedger Müffke wrote:

Can you add the process what you want to do about the possible time difference between server and client?

Please use a new package in OI Notepad.
Cheers
Friedger

2012/5/28 Vettukal Vincent Mathew <vett...@gmail.com>

hi,

This is my first attempt to make any UML diagram to try to show how I plan to implement the Synchronization process. The lastTS is the time of the last sync done by an application using OI Sync it is stored inside the OI Sync. Below is the diagram..

And should I make a just different java Class or a separate package in OI Notepad for synchronization process.

Thanks

--
You received this message because you are subscribed to the Google Groups "OpenIntents" group.
To view this discussion on the web visit https://groups.google.com/d/msg/openintents/-/JJ-G5a6DFogJ.
To post to this group, send email to openi...@googlegroups.com.

To unsubscribe from this group, send email to openintents+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/openintents?hl=en.

[aap]

Vettukal,

If you are using timestamps, how do you detect conflicts-- when an item has been modified both in the app and differently on the server since last sync?

Also, will a system based on timestamps be extensible in the future to support syncing with multiple sources?

-- Aaron

[Vettukal Vincent Mathew]

Making it extensible to allow sync from multiple sources, is the reason the design is a challenge :-)

Taking some clues from your previous feedback, i.e

I'm not sure it is worthwhile to maintain separate sets of UIDs for local and remote identification of the same object. 

I came with this design..

_id is the primary key on Local Client DB like "Notes._ID + " INTEGER PRIMARY KEY," and ModifiedDate is "Notes.MODIFIED_DATE + " INTEGER,"  

The first DB is the DB at the client App and second DB is used at the Main OI Sync. It contains two columns i.e _id and AppEngID.

Some random state just after a Sync

LastSyncTime=2400

                                                       DB at Client App                                                                      DB at main OI Sync

A new Entity is added to the list

Now the sync is again called. At this all the Entities are checked whose ModifiedDate!=2400 and hence some modification done.

From above condition _id=3 row is selected. It is checked with the OI Sync DB. No row exists with that _id value, means its newly created.

It sends the Entity for persistence to the appEngine and gets back the AppEngID.

Time Difference of the client does not matter. Whether its behind or ahead of server.

After sync LastSyncTime is updated to 2600 and also modifiedDate is set to 2600

Modification of a Note. Say note with _id=2 is modified.

At the time of next sync the rows whose value is not equal to 2600(LastSyncTime) is selected and sent to the server.

After the sync

At the time of sync the new entities from the server is fetched, by checking which all entities are greater than LastSyncTime. They are checked with Main OI Sync DB to check whether they already exist on local Client DB, to decide whether modification is needed or creation of a new row on the local DB.

The conflict will be detected when the same entity(with same AppEngineId) is taken from local Client for updation to the server and also it has been fetched from AppEngine.

 

[aap]

I am confused about a couple points in your response.

It sounds like you want to change last modified date of all records, even the ones which did not change on the server. Why would you do that?

Also, if last modified date is such a critical point in the sync logic, that seems to make it less likely that syncing with multiple servers will work.

It seems like you need to remember something about each record from the last time it was synced with each server. So, at last sync with server X, record 3's last modified date was 2400. If the last modified date on the server for that record is still 2400 on that server, then you know it hasn't changed on the server since last sync. Meanwhile at last sync with server Y record 3'd last modified date was 2200, so next time we sync with server Y we know we have to propagate the local change to the server.

-- Aaron

[aap]

One way to avoid timestamp problems is to take a cue from git and use a content-derived hash value as a change id. Each record could have an immutable UUID as the record ID, plus a hash value of all the other synced fields of the record as the current change ID. Then you could use the change ID in place of what you currently call a timestamp. Still would need to remember the last change ID for each record for each sync source.

[Friedger Müffke]

I like the of the content hash value. It allows to detect same changes on different servers as well as "Undo"s

Maybe you just need to add a field for a global version number, increase the local verison number on changes and decrease on "Undo"s. Only the server can increase the global version number...

Cheers

Friedger

2012/6/3 aap <aa...@peromsik.net>

To view this discussion on the web visit https://groups.google.com/d/msg/openintents/-/Kf02ASSPSdoJ.

To post to this group, send email to openi...@googlegroups.com.

To unsubscribe from this group, send email to openintents...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/openintents?hl=en.

[Peli]

If only the server can increase a "global version number", this would probably not solve the problem raised by Aaron previously if you want to synchronize with multiple servers (if two servers independently increase the "global version number", everything gets messed up).

Questions:

1) Do we really want to synchronize with multiple servers? I would be more than happy if several clients can synchronize with the same server?

2) Whether or not one takes multiple servers into account, if servers or client modify an entry independently, there is still the question how to "merge" the information. In order to do so, the system would have to track the differences from one entry to the next, rather than the current state of the entries themselves.

Example:

We start from an item with status "not bought" and without note.

* User A adds a note to an item. (not bought + note)

* User B marks the same item as bought. (bought + no note)

=> In the ideal case, the server would be clever enough to merge the information and add the note to the item as well as mark the item as bought.

BUT: Since only the final state is stored, the server can not know what really has happened. Looking at the differences in the current versions, it may be that:

the situation started actually with an item with status "bought" and with a note (who knows, the old state is never recorded)

* User A marked the item as "not bought" (not bought + note -> same final state as above!)

* User B deleted the note of the item. (bought + no note -> same final state as above!)

in which case the correct merge would be to set the state of the item to "not bought" and to delete the note of the item.

So, properly merging items is not possible unless one tracks the history of the items.

Therefore:

I think it is beyond the scope of this project to create a revision control system. The simplest way out of this is to decide for a simpler strategy:

1) Either the person with later timestamp gets selected (one user marks it as bought, another person bought it - > use whoever set the status later).

2) Or prompt the user whenever an item has been edited by 2 different people concurrently and let them manually merge -> more complicated to implement, and may annoy users more often that it is worth it.

The problem with different time zones can be solved by using UTC throughout as timestamp, and the problem with slight time differences (up to a few minutes) could be simply ignored. One could also synchronize the clocks prior to synchronizing the data (or at least determine the offset that could be corrected when syncing data).

Conclusion:

I think it is enough to just store the timestamp. This solution works with several servers and several clients if one always uses the data with the most recent time stamp (i.e. a data set with more recent time stamp always overwrites a data set with older time stamp).

In order to avoid the situation that 2 data sets differ because they were generated in exactly the same millisecond on 2 different devices (with millions of users, this may happen sooner or later), the server and client could keep an additional variable of "last synchronization" for each client or server they connect to. Data with same time stamp > last synchronization are compared literally, and if they differ, you randomly overwrite one by the other and set a new timestamp on both based on the current time. This guarantees that this record is newer and will overwrite over time all other records on other servers or clients.

(one could also use a hash here, but if one chooses a too short hash, there is the possibility of unwanted collision, and if one chooses a long hash, one could probably just as well simply send the data? Also, a suitable hash function would have to be implemented consistently on the server and on various mobile operating systems.)

(alternatively, instead of the hash, one could simply generate a random number and check timestamp and the random number? Only if both agree, are the items identical.)

At the next update, one only needs to compare items that have been modified after the "last synchronization" time, because all older entries are guaranteed to be identical. (As mentioned, the "last synchronization" time has to be stored for each client or each server separately, but that may just be a handful of entries.)

Conclusions of the conclusions:

Since anyway a proper merging is not possible without recording the history, and recording the history is beyond the scope of this project, a simple synchronization strategy based simply on the timestamp works probably well enough in most cases.

(anything more sophisticated would probably have to be *much* more sophisticated - there's not much point in implementing something half-way sophisticated, as it will not work better than the simple solution)

Peli

[aap]

I have sometimes referred to "multiple servers" but I try to correct myself to "multiple sync sources".

Multiple servers might even be multiple areas within the same server. I might have some lists which I want to share with my wife and others she wouldn't care about but that I might want to edit online myself. Or I might want some list to sync with a local backup on SD card when I am offline but also sync with a cloud server when I am online. Or I might have some lists with which I would trust Google's servers but others which I would only want to sync with my own server. For work-related lists an employer might have a policy that does not allow data outside the corporate firewall. 

Most of those use cases still only involve one sync source per list but some involve more than one. I certainly am not insisting that we need to support all that this summer, but I hope we can choose a design that will be extensible in that direction.

-- Aaron

[aap]

Regarding merge... Our databases are not so large. Perhaps it would be reasonable for the sync app to keep a full copy of the relevant data for each sync source at the time of last sync with that source? (That's what "unison" does.) Full history of interim changes would not be required to make a somewhat intelligent merge. In your example, the sync app would know that the state at last sync was bought with no note, so it will be clear what has changed on each side since last sync. (Not sure I understood your comments about "same final state" though.)

I am not so sure our users will be happy with choosing one of the values at random without merging. If I add a note to a shopping list item and my wife changes the quantity because she wants more for a new recipe she found, the choose-record-at-random strategy is going to guarantee that at least one of us will be unhappy with the sync results.

-- Aaron

[Peli (OpenIntents)]

The unison trick sounds clever. That is something that we didn't have in the discussion yet.

But it also sounds like something that can be implemented later, on top of a basic merging solution with the timestamp, simply by implementing a more clever client. The server wouldn't have to change and would not need to keep the old state.

(Alternatively the client would stay simple and the server would do the clever merging, but the way data would be stored on the server speaks against this.)

To view this discussion on the web visit https://groups.google.com/d/msg/openintents/-/5iO3zEWNRVEJ.

To post to this group, send email to openi...@googlegroups.com.

To unsubscribe from this group, send email to openintents...@googlegroups.com.