Re: [mesh4x] questions on sync scenarios
0 views
Skip to first unread message
Eduardo Jezierski
Dec 22, 2008, 5:45:06 PM12/22/08
to mes...@googlegroups.com
Daniel, this has been sitting in my drafts forever SORRY!!!
SqlLite
We mostly use hibernate because it solves the connection and xml mapping aspects with no additional code.
Writing an adapter, as I mentioned, is not complex at all – we would ask that you add a table to the sqllite db to keep track of versioning metadata; and then we would implement the IContentAdapter interface I showed before which mostly looks like a crud class – insert delete update an a couple of get methods.
That is about being able to expose the data and the versioning metadata apropriately. That is the start for everything else.
Discovery and automatic startup of sync opportunities
We haven’t run into concrete scenarios yet where we had to implement this ‘automatic discovery’ of sync opportunities in mesh4x, but I think it would be good if we started researching it as it would be a great simplifier. I don’t know how dropbox does the discovery, but I suggested we all look at zeroconf as a way to publish/seek the existence of the mesh4x service.
So there is no out of the box support for this yet. I suggested zeroconf to not get into the whole business of publishing udp broadcasts to say ‘ im here, is anyone out there’ while keeping track.
Dropping connection in the middle of a http sync
Hm, we did this work on geochat to support http streaming (you start processing the data on the receiver end and if the connection dies halfway at least you get to proces the 1st half of the data) for the SMS gateways.... But I don’t think we did that work on the http receivers of the mesh4x service yet; on both the process of doing POST and getting a response.
ondat: can you add this to the issue list? It’s ironic but predictable that one would want the reliability characteristics of the sms-based protocol, over http!
USB Stick
Syncing to a usb Stick is bsically a matter of writing on side A a file that has the data and versioning, and then running/driving over to B, and then reading the file and performing a merge locally.
The file would be a flat atom/rss file with the data and versioning metadata of the dataset in question.
Your UI would probably have a ‘ sync with USB’ option and specify a folder in the gizmo, the app would see if files are there, if they are do a merge, then write new files w the latest versions, --> and then the same thing would happen everytime the stick gets passed around. I guess one could build a monitor app to autdetect that this has mesh files and do the sync-on-plug...but again, that woud have to be written still.
We need your help with a decision as soon as we progress on the server and mapping xforms/rdf the questions you pose above become priority; I feel a lot of it goes beyond the mesh ‘protocols’ but into having a suite of utilities and apps that does all this ‘moving around’ of the data easily. Do you think we should start investing in, say a java client that implements these things? Or should we continue to have more building blocks that are useful but still push the complexity of integration over to adopters? Would love your thoughts on strategy here so we better support exiting and future projects you may have.
ej
From: Daniel Myung <dmy...@dimagi.com>
Reply-To: <mes...@googlegroups.com>
Date: Wed, 17 Dec 2008 14:00:02 -0500
To: <mes...@googlegroups.com>
Subject: [mesh4x] questions on sync scenarios
Hi Ed,
In response to your question:
"If you tell us the schema for the table you want to use as the example we can work it off of that! Reminder we have a milestone coming up Friday, do we have until then? Does this outline I just created look appropriate for your needs?
http://code.google.com/p/mesh4x/wiki/HowToSyncADatabase"
I wanted to provide a bit of feedback and then tell you a bit more about the specific use case we're trying to support – in addition to providing a schema for you if you do want to use it as an example (see end of message)
This looks like a good outline of what we need to do. Two decision points we are having which maybe you can help us address are as follows:
Out of the box, django uses sqlite via pysqlite. Small, local, simple config. However it appears to not have full-blown hibernate support. So:
In particular, we have some additional questions regarding the HTTP transport.
Not surprisingly, we expect to encounter potentially very bad connections, in terms of being up and down frequently and potentially high drop/timeout rates and long periods of offline time. We're looking at a "on demand, seldom on" GPRS modem connection for a laptop.
* Is there out-of-the box support for a scenario like this - possibly with opportunistic calls to the http sync methods?
* If not, what would be your recommended solution for working with mesh in an environment like this?
* Are there issues that come up when a connection drops in the middle of a sync?
* if there is a huge amount of data that needs to be synced...and the sync never finishes (due to continual drops and timeouts), is there a chance it'll be in sort of stuck state?
These concerns on HTTP has had us curious about the USB file key transport mechanism. How are changes propagated via a USB stick? Is it assuming that the entire database (or excel file) need to be present such that the sync engine can touch Left and Right to do the sync? As such, it's just a choice of a certain adapter and storage?
This is partly why we've been looking hard at the Dropbox/WLM option. It to us, represents a USB stick file transfer minus the cars/walking and USB stick. When a remote update finishes, only then will a file will actually appear as a file in your shared folder. Presumably then, it'll be ready to synchronize. But maybe I'm overanalyzing the situation. If HTTP can handle the poor connectivity, then this conversation is moot :)
For info on our database schema, take a look at this link: http://code.dimagi.com/ec3/wiki/SynchronizationThoughts for a high-level description of the project for some additional context. This also includes the tables you can use as a model for the documentation if you would like.
Tangentially, I think if the sqlite case is addressed, then googlegears + mesh4x would be a pretty freaking sweet setup for future apps :)
Dan
--
Daniel Myung
Senior Engineer, Dimagi, Inc
529 Main St, Suite 606 | Charlestown, MA 02129
office: (617) 580-3100 | mobile: (617) 645-3236 | fax: (617) 580-3103
http://www.dimagi.com/
SqlLite
We mostly use hibernate because it solves the connection and xml mapping aspects with no additional code.
Writing an adapter, as I mentioned, is not complex at all – we would ask that you add a table to the sqllite db to keep track of versioning metadata; and then we would implement the IContentAdapter interface I showed before which mostly looks like a crud class – insert delete update an a couple of get methods.
That is about being able to expose the data and the versioning metadata apropriately. That is the start for everything else.
Discovery and automatic startup of sync opportunities
We haven’t run into concrete scenarios yet where we had to implement this ‘automatic discovery’ of sync opportunities in mesh4x, but I think it would be good if we started researching it as it would be a great simplifier. I don’t know how dropbox does the discovery, but I suggested we all look at zeroconf as a way to publish/seek the existence of the mesh4x service.
So there is no out of the box support for this yet. I suggested zeroconf to not get into the whole business of publishing udp broadcasts to say ‘ im here, is anyone out there’ while keeping track.
Dropping connection in the middle of a http sync
Hm, we did this work on geochat to support http streaming (you start processing the data on the receiver end and if the connection dies halfway at least you get to proces the 1st half of the data) for the SMS gateways.... But I don’t think we did that work on the http receivers of the mesh4x service yet; on both the process of doing POST and getting a response.
ondat: can you add this to the issue list? It’s ironic but predictable that one would want the reliability characteristics of the sms-based protocol, over http!
USB Stick
Syncing to a usb Stick is bsically a matter of writing on side A a file that has the data and versioning, and then running/driving over to B, and then reading the file and performing a merge locally.
The file would be a flat atom/rss file with the data and versioning metadata of the dataset in question.
Your UI would probably have a ‘ sync with USB’ option and specify a folder in the gizmo, the app would see if files are there, if they are do a merge, then write new files w the latest versions, --> and then the same thing would happen everytime the stick gets passed around. I guess one could build a monitor app to autdetect that this has mesh files and do the sync-on-plug...but again, that woud have to be written still.
We need your help with a decision as soon as we progress on the server and mapping xforms/rdf the questions you pose above become priority; I feel a lot of it goes beyond the mesh ‘protocols’ but into having a suite of utilities and apps that does all this ‘moving around’ of the data easily. Do you think we should start investing in, say a java client that implements these things? Or should we continue to have more building blocks that are useful but still push the complexity of integration over to adopters? Would love your thoughts on strategy here so we better support exiting and future projects you may have.
ej
From: Daniel Myung <dmy...@dimagi.com>
Reply-To: <mes...@googlegroups.com>
Date: Wed, 17 Dec 2008 14:00:02 -0500
To: <mes...@googlegroups.com>
Subject: [mesh4x] questions on sync scenarios
Hi Ed,
In response to your question:
"If you tell us the schema for the table you want to use as the example we can work it off of that! Reminder we have a milestone coming up Friday, do we have until then? Does this outline I just created look appropriate for your needs?
http://code.google.com/p/mesh4x/wiki/HowToSyncADatabase"
I wanted to provide a bit of feedback and then tell you a bit more about the specific use case we're trying to support – in addition to providing a schema for you if you do want to use it as an example (see end of message)
This looks like a good outline of what we need to do. Two decision points we are having which maybe you can help us address are as follows:
Out of the box, django uses sqlite via pysqlite. Small, local, simple config. However it appears to not have full-blown hibernate support. So:
- Has anyone had experience syncing a Sqlite DB in mesh4x? I assume that getting hibernate on top of it solves this problem, but we're not sure how hard that is.
- If we had to use Sqlite and hibernate was not an option, what would writing an adapter for it entail? Is one already written?
- Assuming that switching to MySql was an option that introduced some (but not a ton of) complexity – would that be your recommended solution over trying to wrap Sqlite?
- In summary: We use sqlite, but have the option of mysql. What's the best way to proceed w/ mesh4x?
In particular, we have some additional questions regarding the HTTP transport.
Not surprisingly, we expect to encounter potentially very bad connections, in terms of being up and down frequently and potentially high drop/timeout rates and long periods of offline time. We're looking at a "on demand, seldom on" GPRS modem connection for a laptop.
* Is there out-of-the box support for a scenario like this - possibly with opportunistic calls to the http sync methods?
* If not, what would be your recommended solution for working with mesh in an environment like this?
* Are there issues that come up when a connection drops in the middle of a sync?
* if there is a huge amount of data that needs to be synced...and the sync never finishes (due to continual drops and timeouts), is there a chance it'll be in sort of stuck state?
These concerns on HTTP has had us curious about the USB file key transport mechanism. How are changes propagated via a USB stick? Is it assuming that the entire database (or excel file) need to be present such that the sync engine can touch Left and Right to do the sync? As such, it's just a choice of a certain adapter and storage?
This is partly why we've been looking hard at the Dropbox/WLM option. It to us, represents a USB stick file transfer minus the cars/walking and USB stick. When a remote update finishes, only then will a file will actually appear as a file in your shared folder. Presumably then, it'll be ready to synchronize. But maybe I'm overanalyzing the situation. If HTTP can handle the poor connectivity, then this conversation is moot :)
For info on our database schema, take a look at this link: http://code.dimagi.com/ec3/wiki/SynchronizationThoughts for a high-level description of the project for some additional context. This also includes the tables you can use as a model for the documentation if you would like.
Tangentially, I think if the sqlite case is addressed, then googlegears + mesh4x would be a pretty freaking sweet setup for future apps :)
Dan
--
Daniel Myung
Senior Engineer, Dimagi, Inc
529 Main St, Suite 606 | Charlestown, MA 02129
office: (617) 580-3100 | mobile: (617) 645-3236 | fax: (617) 580-3103
http://www.dimagi.com/
Daniel Myung
Dec 22, 2008, 10:23:58 PM12/22/08
to mes...@googlegroups.com
Hi Ed,
For that reason, we've switched our django instance to mysql. It's only a few connection strings to change and it's a seamless transition. So that was an easy for us.
Dan
Thanks for the update, here are my replies:
re: SqlLite.
Long term I think for future projects, the implications for sqlite seem very interesting for the near universality of the db file on whatever system it resides on. But the lack of hibernate support for sqlite (only on nhibernate) seemed to be a drawback for our entry.
For that reason, we've switched our django instance to mysql. It's only a few connection strings to change and it's a seamless transition. So that was an easy for us.
since you guys have already have something going with hibernate, we figure'd we'd help to refine/test what you've already got. I've connected hibernate to our db and got the helper classes generated. is there a place in the codebase i could use to test mesh4x off our data model? You mentioned the unit tests you guys have already built for java. Is there a good entry point to get me started to apply them to our DB?
re: automated sync discovery...and usb stick
zeroconf is a bit lower level than we originally imagined. We're currently not expecting a system that allows for advertisement/discovery. I guess that's where we figured the Dropbox/WLM file system based synchronization would be tolerant of discovery. Let those services (basically RSync clones) run and let something like zeroconf do the connectivity and discovery. I had envisioned that there was some sort of "magic" file that could sync with all the other adapters.
re: http tolerance
sounds good. I guess the operative phrase to use here is "we want to use the modern protocols, but bust them up good" :)
Our thoughts:
We were thinking of building up on the mysql hibernate front...and perhaps going the HTTP route first to make it work out ok. I think ultimately GUI frontend developments probably will only be useful for testing and simulation. Our ultimate implementation will want to leverage the above mentioned adapters and just be headless, spawned by some thread in django spawning the mesh4x jars periodically. We want one less moving button/dialog that our users will need to actuate.
So in that perspective, I think working on the building blocks and a compelling "environment" will make things more interesting for the community to build upon. Since I'm in a django state of mind, I've been thinking of their startup development environment that really roped me in and hooked me. Download, run one script, edit one file, and i have a database i can view and enter data on in a browser.
actually, having just said that, I'm a bit torn myself with what actually helps in the near term. A "reference" tool, vs. building up other components of the development experience? I guess I'm still voting for the latter.
Cheers,
Dan
Eduardo Jezierski
Dec 23, 2008, 8:59:43 PM12/23/08
to mes...@googlegroups.com
Daniel, want to add SqlLite support to http://mesh4x.uservoice.com? Be the first brave soul!
Im thinking because of the hibernate support, that might be a good example for the ‘ how to build your adapter’
Im thinking because of the hibernate support, that might be a good example for the ‘ how to build your adapter’
From: Daniel Myung <dmy...@dimagi.com>
Reply-To: <mes...@googlegroups.com>
Date: Mon, 22 Dec 2008 22:23:58 -0500
To: <mes...@googlegroups.com>
Subject: [mesh4x] Re: questions on sync scenarios
Hi Ed,
To: <mes...@googlegroups.com>
Subject: [mesh4x] Re: questions on sync scenarios
Hi Ed,
Thanks for the update, here are my replies:
re: SqlLite.
Long term I think for future projects, the implications for sqlite seem very interesting for the near universality of the db file on whatever system it resides on. But the lack of hibernate support for sqlite (only on nhibernate) seemed to be a drawback for our entry.
For that reason, we've switched our django instance to mysql. It's only a few connection strings to change and it's a seamless transition. So that was an easy for us.
since you guys have already have something going with hibernate, we figure'd we'd help to refine/test what you've already got. I've connected hibernate to our db and got the helper classes generated. is there a place in the codebase i could use to test mesh4x off our data model? You mentioned the unit tests you guys have already built for java. Is there a good entry point to get me started to apply them to our DB?
---> Tondat, let me know if you have a better idea, but maybe the EpiInfo mesh4x tool is the best to start with?
Or the Mesh4j swingUI spike?
http://code.google.com/p/mesh4x/source/browse/Mesh4j/spikes/
Or the Mesh4j swingUI spike?
http://code.google.com/p/mesh4x/source/browse/Mesh4j/spikes/
re: automated sync discovery...and usb stick
zeroconf is a bit lower level than we originally imagined. We're currently not expecting a system that allows for advertisement/discovery. I guess that's where we figured the Dropbox/WLM file system based synchronization would be tolerant of discovery. Let those services (basically RSync clones) run and let something like zeroconf do the connectivity and discovery. I had envisioned that there was some sort of "magic" file that could sync with all the other adapters.
--> the vision of the file is correct, actually it is the exact same RSS XML the server would produce (example: http://sync.instedd.org/Service.svc/feeds/A10 )
From: Daniel Myung <dmy...@dimagi.com <http://dmy...@dimagi.com> >
Reply-To: <mes...@googlegroups.com <http://mes...@googlegroups.com> >
Date: Wed, 17 Dec 2008 14:00:02 -0500
Reply all
Reply to author
Forward
0 new messages