Re: [mesh4x] Re: Documentation Proposal

[Eduardo Jezierski]

Hmm – I think it is simpler than that

Imagine the core of mesh4x is a library that can ‘ sync’ the data in A and B.
A and B are reached through adapters, so you can make all sorts of combinations of local db / excel / cloud if you have the right adapters.
Excel to db? Db to cloud? Cloud to cloud? Db to db? It’s mix and match, and the options are many.

The purpose of the scenarios is to highlight common use patterns and provide a ‘ oh, that’s me!’  entry point to the docs, and help folks with a 1.2.3.. of setting it up.
So if I understood your scenario correctly:

• you will have roaming laptops (entering and exiting networks)
• You will have database (sql lite)  and some photos you want to share  ‘everything with everyone’  (you don’t have constraints which laptops see which filtered subsets of data from what I understod)
• You want the sync to be triggered by ‘ walking in the room’  (getting network visibility)
  

I also got feedback from th CDC the scnenario titles are misleading. Back to the drawing board!

---

From: Daniel Myung <dmy...@dimagi.com>
Reply-To: <mes...@googlegroups.com>
Date: Tue, 16 Dec 2008 12:37:57 -0500
To: <mes...@googlegroups.com>
Subject: [mesh4x] Re: Documentation Proposal

Hi all,

Just have a few questions regarding the scenarios listed.

Some of the howtos in that section seem to make an assumption of syncing client app to some more "heavier" server app, like the ec2 instance or the cloud services.  If I recall your web demo from the call a few weeks back, you demo'ed a mysql and excel sync.  

Was something similar to that demo your intent behind the "sync heterogeneous applications" scenario?  Is it a "how to sync excel to mysql to sqlite" sort of scenario, or is it gonig to be a  "how to sync epi-info to openmrs using the cloud services" sort of thing?  

I guess this is a question of syncing in the absence of these servers. A question on the need to define service endpoints and discovery, something you guys highlight as a pro/con of this sync.   One needing a high degree of planning ahead (which I'm all for), but it might present itself as being daunting to the implementor to try to address the myriad of sync scenarios that might prop up.  

In other words, is there anything special that needs to be done to support a diversity of adapters/sync methods in a given application?

Dan

On Tue, Dec 16, 2008 at 4:53 AM, Eduardo Jezierski <ed...@instedd.org> wrote:

Hello everyone, how do you feel about the following doc skeleton? Any gaps?
If you have specific suggestions, let me know for example

' take me from just having a sqllite db schema for a table to syncing it
over the internet with your service'


Thanks!
~ ej
=========================


Guidance Index
Application Scenario

    * Scenario: Phone to Tethered PC Sync
    * Scenario: Building Maps Collaboratively using KML
    * Scenario: Web-based UI for desktop-based data
    * Scenario: Creating an updateable store for for data-collection tools
    * Scenario: Sync Heterogenous Applications
    * Scenario: Tethered Phone to Tethered Phone sync
    * Scenario: Cloud Sync of single-user databases



Checklist Item

    * Checklist - Making sure Tethered Phone to Tethered Phone Sync will
work
    * Checklist - Setting up your development environment



Code Example

    * Code Example: Making a CSV file adapter




Guideline

    * Roadmap - Joining the Mesh4x Community
    * Roadmap - Getting to try out Mesh4x Technolgies



How To

    * How To Sync a MySQL Table to the Cloud
    * How to start your own EC2 Instance of the Mesh4x Server on S3
    * How to edit data in a mesh server with JavaROSA
    * How to create an adapter for an existing applications store
    * How to create your own adapter
    * How To Set up your own Mesh4x Server on MySQL
    * How To Sync an Access Table



Info

    * Info - List of Mesh4x adapters
    * Info - List of applications using Mesh4x
    * Info - FeedSync Reference
    * Info - RDF and RDFs primer
    * Info - Mesh4x Support for binary files
    * Info - How does the Mesh4x over SMS protocol work


Principle

    * Allow data augmentation
    * Conflicts are interesting
    * Keep Schemas small
    * Default to standard payload representations

--
Daniel Myung
Senior Engineer, Dimagi, Inc  
529 Main St, Suite 606 | Charlestown, MA 02129
office: (617) 580-3100  |  mobile: (617) 645-3236 | fax: (617) 580-3103
http://www.dimagi.com/

[Daniel Myung]

• you will have roaming laptops (entering and exiting networks)

yeah, but right now we're expecting that the laptops will be well known.  They'll just be flakey in connectivity.

 

• You will have database (sql lite)  and some photos you want to share  'everything with everyone'  (you don't have constraints which laptops see which filtered subsets of data from what I understod)

see prior email, we just jumped to mysql.  For our use case, an edge(remote) laptop will share everything.  But in reality, it's a subset linked to that laptop/facility.  We're envisioning a star topology where central DB will divy partition up images based on an identifier unique to the edge node. 

• You want the sync to be triggered by ' walking in the room'  (getting network visibility)
  

Our effective "walk in the room" is the successful synchronization of a Dropbox shared directory.  This actually simplifies the sync process somewhat in terms of needing the app have to do an opportunisitc connectivity check over HTTP.  Let dropbox's smarter daemon handle the byte pushing over the wide area link, and treat the dropbox directory as a USB drive.  Put a sniffer on the shared directory.  If its contents change, we know that something's different and can sync.

[Eduardo Jezierski]

Would the dropbox be shared across all pcs or is it just between the roaming pc and the stationed pc?
Because im thinking with what you suggest, a file sniffer could see dropbox changed a local file, and then ‘ sync’ happens with that file (incoprorating changes from the file in the local store, - and viceversa, or you only need changes to flow one-way?)

ej

---

From: Daniel Myung <dmy...@dimagi.com>
Reply-To: <mes...@googlegroups.com>

Date: Mon, 22 Dec 2008 22:29:05 -0500

To: <mes...@googlegroups.com>
Subject: [mesh4x] Re: Documentation Proposal

• you will have roaming laptops (entering and exiting networks)
  

yeah, but right now we're expecting that the laptops will be well known.  They'll just be flakey in connectivity.
 

• You will have database (sql lite)  and some photos you want to share  'everything with everyone'  (you don't have constraints which laptops see which filtered subsets of data from what I understod)
  

see prior email, we just jumped to mysql.  For our use case, an edge(remote) laptop will share everything.  But in reality, it's a subset linked to that laptop/facility.  We're envisioning a star topology where central DB will divy partition up images based on an identifier unique to the edge node.

• You want the sync to be triggered by ' walking in the room'  (getting network visibility)
  

Our effective "walk in the room" is the successful synchronization of a Dropbox shared directory.  This actually simplifies the sync process somewhat in terms of needing the app have to do an opportunisitc connectivity check over HTTP.  Let dropbox's smarter daemon handle the byte pushing over the wide area link, and treat the dropbox directory as a USB drive.  Put a sniffer on the shared directory.  If its contents change, we know that something's different and can sync.

 

I also got feedback from th CDC the scnenario titles are misleading. Back to the drawing board!

---

From: Daniel Myung <dmy...@dimagi.com <http://dmy...@dimagi.com> >
Reply-To: <mes...@googlegroups.com <http://mes...@googlegroups.com> >

Date: Tue, 16 Dec 2008 12:37:57 -0500

To: <mes...@googlegroups.com <http://mes...@googlegroups.com> >

Subject: [mesh4x] Re: Documentation Proposal


Hi all,

Just have a few questions regarding the scenarios listed.

Some of the howtos in that section seem to make an assumption of syncing client app to some more "heavier" server app, like the ec2 instance or the cloud services.  If I recall your web demo from the call a few weeks back, you demo'ed a mysql and excel sync.  

Was something similar to that demo your intent behind the "sync heterogeneous applications" scenario?  Is it a "how to sync excel to mysql to sqlite" sort of scenario, or is it gonig to be a  "how to sync epi-info to openmrs using the cloud services" sort of thing?  

I guess this is a question of syncing in the absence of these servers. A question on the need to define service endpoints and discovery, something you guys highlight as a pro/con of this sync.   One needing a high degree of planning ahead (which I'm all for), but it might present itself as being daunting to the implementor to try to address the myriad of sync scenarios that might prop up.  

In other words, is there anything special that needs to be done to support a diversity of adapters/sync methods in a given application?

Dan

[Daniel Myung]

Hey all,

Quick answer, yeah.  It's exactly as you say it.  With Dropbox (or WLM or Rsync for that matter), we're seeing that with a file sniffer, we think 2 way meshing is possible.

Our scenario we are hoping to support:

CentralServer in their dropbox has shared folders:

/Dropbox/FacilityA

/Dropbox/FacilityB
/Dropbox/FacilityC

Out in the clinic, the LaptopA will have in their dropbox

/Dropbox/FacilityA

Only data in the LaptopA's /FacilityA folder will be synced with the CentralServer.

CentralServer is initiating the shares with individual users, as each CentralServer and all the laptops will log into dropobox with different IDs.

Dropbox will also let you specify multiple shares for multiple machines.

Let's say FacilityB is actually made up of 3 laptops X,Y,Z, then they could conceivably have their own accounts and sync together as well as with CentralServer.

Drawback of FacilityB having 3 laptops X,Y,Z is that in order to sync "dropbox style" you need connectivity to the Dropbox servers.  You can't do peer to peer.

It should be noted with Dropbox, you can have the same account on any number of machines you want running the Dropbox process.  WLM works in a similar way with a bit more control.  We're going with Dropbox right now due to its support of Linux.

But just as you said, sniffer listens for file changes in the Dropbox folder and mesh4x's to it.  Any local changes also could change that file in Dropbox and alert dropbox to update other remote machines, so it's truly 2 way.

On CentralServer;

Change-> /Dropbox/FacilityA/dataset.xml

<dropbox magic>

FacilityA laptop:

Dropbox delta-> /Dropbox/FacilityA/dataset.xml Updated!

Mesh4x updates mysql database, nurse sees changes.

Nurse reads and does whatever.

Nurse adds photos and new interactions

Mesh4x detects delta between more up to date mysql database and "stale" dropbox dataset.xml

dataset.xml updated

Dropbox magic - dataset.xml changed + sees photos in FacilityA, so dropbox starts to upload all the dataset changes AND the photos

all are pushed back to CentralServer.

[Daniel Myung]

Hi all,

Happy new year everyone from an icy but warming (> 32 degrees F) boston :)

Our partners in Zambia are getting up to speed with regard to our implementation, and as a result our schedule for doing a synchronization framework is rapidly formalizing.  

We were wondering if you guys would have an estimate for when your documentation project would be ready for 3rd party implementors like us.  Are you guys blocking on us to provide more scenario description?  Or are the other voices on uservoice pushing additional development elsewhere?  Some sort of timeline to allow us to set the expectations for our partners and ourselves would be great.  They're in the know that 'tis better to be right than to be rushed :)  

So yeah, let us know if there's anything more you need from us to flesh out our particular scenario...whether we need to tweak our parameters to fit what's already been implemented, or anything else that could help us on the way.

Thanks, 

Dan

On Tue, Dec 23, 2008 at 9:40 PM, Eduardo Jezierski <ed...@instedd.org> wrote: