CloudSync Entity Class

[Vettukal Vincent Mathew]

Hi,

An introduction to the project can be found on this wiki page.

Entity class is the class used for data persistence on Google AppEngine. Below are the fields that I am planning

to add in the Entity class.

Structure of 1 record or 1 Entiry

The Entity class may look similar to this. The fields that it may have.

 long locallyGeneratedUID;
 long appEngineUID;
 long timestamp;
 char tag;
 int/String appPackageName;
 String jsonStringData;

Explanation of each field

locallyGeneratedUID

Desc: Its an Id generated in the clientApp before any sync happens and without network connection.
Purpose: To uniquely identify every record.
Issue: How to generate a truly unique ID locally? Lets say we have a phone and tablet. locallyGeneratedUID should be unique for every record without getting to know which UID was used on other device.

appEngineUID

Desc: An auto-generated ID on GoogleAppEngine

timestamp

Desc: A field to show when was it last modified. It is milliseconds passed from 1 Jan 1970.
Purpose: Help to decide which record is the latest in case of updations and also for conflict resolution.
Issue: If it is locally generated and if time on the users phone can be inaccurate, read -10 mins or more. It may label a new record on the users phone which is behind 10 mins to be shown as old when compared to record which were created on phone with correct time.A logical constraint that will be there is any modification to this field should lead to higher value instead of a low value.

tag

Desc: Field to identify which operation of done to record most recently. Example Updation will have 'U', deletion will have 'D' and creation will be 'C' 

appPackageName

Desc: It will be used to know which record belongs to which App.
Issue: An int could be used instead of String by using hash() function of String class. Will be space saver and may be also good for efficiency. Not calculated the hash function collision that may occur, yet.

jsonStringData

Desc: All data of one record will be made into one String and stored in this field.
'Example: From OI Notes

 _ID, // 0
 TITLE, // 1
 NOTE,//2
 CREATED_DATE,
 MODIFIED_DATE,//4
 TAGS,
 ENCRYPTED,//6
 THEME,
 SELECTION_START,//8
 SELECTION_END,
 SCROLL_POSITION//10 

Data of all fields will be put in one String using Json.

How will metadata be stored

Will the fields like appEngineUID,tag,locallyGeneratedUID be added to the table where _ID,TITLE,NOTE,THEME etc are stored or a different table will be created where these will be stored and values of two tables will be mapped together using single primary key?

Any suggestions for additions or modifications of fields and also how to approach some issues mentioned above will be greatly appreciated.

Thanks.

[Al Sutton]

Hi Vettukal,

Is there a reason you've chosen to use Json as the storage format for the string?

Have you considered having a table per application and instead of combining the data fields into one item having the fields as separate database columns so they could be queried more easily?

Al.

-- 
Al Sutton - Funky Android Ltd.
(www.funkyandroid.com)

T: @alsutton  
G+: http://goo.gl/ymi9b

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries. 

--
You received this message because you are subscribed to the Google Groups "OpenIntents" group.
To view this discussion on the web visit https://groups.google.com/d/msg/openintents/-/oM0yVUNYj7MJ.
To post to this group, send email to openi...@googlegroups.com.
To unsubscribe from this group, send email to openintents...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/openintents?hl=en.

[aap]

Vettukal,

I'm not sure it is worthwhile to maintain separate sets of UIDs for local and remote identification of the same object. Doesn't that add extra complexity to the sync logic? If the point of a UID is to be universally unique, then any given item should only need one. 

Maybe you don't mean "locallyGeneratedUID", but rather, just the local native ID already used in the database tables for each application?

-- Aaron

[Vettukal Vincent Mathew]

Just to be sure we all are at same page.The data will be stored as Json string only in servers and not in local apps.

Is there a reason you've chosen to use Json as the storage format for the string?

I thought of using JSON by combining the data fields to one, so as to generalize the Entity Class. By this all applications will have just one kind of Entity class. I am not a big fan of querying data on the server because I have heard operations on server side are expensive. I think a design that has minimum server based ops is good. And thinking about the syncing process, I could not come up with a situation where query based on a particular column will be used. Though a good way to efficiently combine the data to one field has to made, to keep minimum overhead. 

To unsubscribe from this group, send email to openintents+unsubscribe@googlegroups.com.

[Vettukal Vincent Mathew]

I'm not sure it is worthwhile to maintain separate sets of UIDs for local and remote identification of the same object. 

I also think two UIDs are not useful. I think appEngineUID can be dropped. But included it as I am not sure whether there will be any problems from AppEngine if we do not have their auto-generated ID.

Though I mean "locallyGeneratedUID" to be unique universally. Part of the reason is, when a record is created there is no surety the network will be accessible as to assign an ID given by the server.

A possible solution can be,

When record are created, local Apps do not assign any ID to them. Later when user instantiates the sync process, these newly created record are pushed to AppEngine and given autogenerated IDs. Then these records with new IDs are fetched back by CloudSync. CloudSync then provides these new records with IDs to local Apps where these Apps replace old "IDless" records with new records.

It sounds a bit complicated, so I wanted to avoid that. But this approach can be realized.

[Vettukal Vincent Mathew]

One more field that has to be added it

String userEmailId;

For authentication.

[Al Sutton]

Yup, we're on the same page :).

Although a deserialisation action on the server would be slightly more expensive, I can see an advantage to having the data available on the server side so we could develop some server based apps.

What do other people think?

Al.

-- 
Al Sutton - Funky Android Ltd.
(www.funkyandroid.com)

T: @alsutton  
G+: http://goo.gl/ymi9b

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries. 

To view this discussion on the web visit https://groups.google.com/d/msg/openintents/-/FLScTofabtEJ.

To post to this group, send email to openi...@googlegroups.com.

To unsubscribe from this group, send email to openintents...@googlegroups.com.

[Al Sutton]

How AppEngine specific are we looking to make it? If we want to tie it to AppEngine and Googles authentication there's a User object we can use which handles tracking of users.

Al.

-- 
Al Sutton - Funky Android Ltd.
(www.funkyandroid.com)

T: @alsutton  
G+: http://goo.gl/ymi9b

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries. 

--
You received this message because you are subscribed to the Google Groups "OpenIntents" group.

To view this discussion on the web visit https://groups.google.com/d/msg/openintents/-/9AYuNVCerIwJ.

[aap]

I thought we were looking to come up with a generalized sync framework, for which App Engine would be perhaps the first sync plug-in, but we would structure things such that others could write additional sources for other services or local sync of some sort.

To unsubscribe from this group, send email to openintents+unsubscribe@googlegroups.com.

[Al Sutton]

That's my understanding too, as I understand it the sync mechanics should be adaptable to any app. 

My question here is more about the specific implementation for AppEngine. Using a JSON string to store all the data for one column is an option, but, personally, I can see the benefit of developing customised tables per app.

Al.

-- 

Al Sutton - Funky Android Ltd.
(www.funkyandroid.com)

T: @alsutton  
G+: http://goo.gl/ymi9b

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries. 

To view this discussion on the web visit https://groups.google.com/d/msg/openintents/-/F-yQmG7vslgJ.

To post to this group, send email to openi...@googlegroups.com.

To unsubscribe from this group, send email to openintents...@googlegroups.com.

[Peli]

It is good to think about the general strategy, but this should not lead to unnecessarily complex design structures early in the process. Even if we start with the simplest possible solution with most straightforward implementation now, it can always be expanded to something more general later on.

To accelerate the process, since we had 3 working demos during the application process already, I would suggest

1) to upload those to the OI repositories and start from there.

2) Next step is to unify the structures so that the solutions can work together,

3) and then one can build and expand on top of something working.

Details like how to store data or whether to use AppEngine specific variables can always be modified later on.

Or are there other suggestions for strategies for a quick start?

Peli

[Peli]

Details like how to store data or whether to use AppEngine specific variables can always be modified later on.

(*) Especially, since we are still in the pre-alpha-stage and don't have to care about backward compatibility.

[Vettukal Vincent Mathew]

It is good to think about the general strategy, but this should not lead to unnecessarily complex design structures early in the process. Even if we start with the simplest possible solution with most straightforward implementation now, it can always be expanded to something more general later on. 

I also think that would be the best approach, because that would help in accelerated development and we can make modifications later..

1) to upload those to the OI repositories and start from there. 

I cannot start working from that code. Will have to start from scratch because I have thought of a different design(much more efficient one) thanks to discussion/questions that I had with mentors during proposal phase. I believe it will not take too long even if I start from scratch because the underlying implementation would be same.

Should I still upload the demo code to CloudSync repo so that others can see it or should I just give a link on wiki page to the Google Code demo project?

[Peli (OpenIntents)]

If there is other code coming soon then just commit that :-)

Peli

To view this discussion on the web visit https://groups.google.com/d/msg/openintents/-/W2MY4-JkQHEJ.

To post to this group, send email to openi...@googlegroups.com.

To unsubscribe from this group, send email to openintents...@googlegroups.com.

[Randy McEoin]

If we don't have customized tables per app in AppEngine then how will OI Web Interface efficiently work with the data?

The transfer of data between OI Sync and AppEngine can be a single string, but it seems to me that we're going to need to go through the effort of creating customized tables on the AppEngine side for efficient queries and updating by the web client.

Do we want to take a stab at Access Control Lists for the data?  Or save that for another round later?   ACL's would allow the end user to specify other people who could access the data.  I like the way Google Docs works on that with Sharing.   You can have a single owner, other people are either read/write, or read.   Personally I think we should save it for later and I don't see how building it in up front is a requirement.   So for now we'd just be happy to have the data synced to the cloud for backup purposes and multiple devices associated with your Google account.

Randy

[Al Sutton]

That's pretty much my concern as well with regard to not having custom tables, but that may be the next step in the development after sync is working for generic string blobs.

I'd say ACLs are important, but it's also something which can be added after the basic mechanics are in place.

Al.

-- 
Al Sutton - Funky Android Ltd.
(www.funkyandroid.com)

T: @alsutton  
G+: http://goo.gl/ymi9b

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries. 

To view this discussion on the web visit https://groups.google.com/d/msg/openintents/-/k8PBW6DH-UQJ.

To post to this group, send email to openi...@googlegroups.com.

To unsubscribe from this group, send email to openintents...@googlegroups.com.