Django App DB replicas - call for suggestions

[Blazor]

Hi all,

I was asked to design a new django app by a client whose needs are a little bit context-specific.

The purpose of this app is to introduce computer-assisted technologies in a paper-based environment, in the e-Health domain. This client would like to access a central server with a few mobile clients with the constraint of a non-permanent internet connection, because the location of the mobile clients is continuously changing and an internet connection might not always be available.

My current idea is based on a portable small server (a nettop, or even a raspberry pi, as the computing power is not a problem for this app) with a django app on it and a wireless access point for providing mobile clients with an always-on portable server reachability. At the same time, I'd like other clients around the world (a very little number, though) to still be able to access a central remote version of the same service, so I'd still deploy the same app (with its own DB replica) within a remote server.

I hope I made my client's requirements clear: how would you manage the synchronization between the two DBs, given that both DBs need a write access and that the only relaxation of this constraint is that when the portable server is on, the remote DB might work in a read-only mode (but this is not required).

Thanks in advance for any suggestions you'll provide me with.

B.

[Phang Mulianto]

Hi,

There is database do the offline and sync to main db server when connection available. See couchdb.

or maybe for manual sync, in local db use sqlite, and your client app will sync when online to main db.

Regards,

mulianto

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/bb65e66c-4788-4010-9bfe-b8f20e41e5d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lane Campbell]

To accomplish what you've explained you could move forward with the app on the PI for the clients.  Each client would need to sync with a database somewhere.  To achieve this I would explore how to individualize and track each machine in the field.  Then setup another django app in the cloud somewhere that has an api these clients are designed to access.  Set the sync interval at a low enough point that the user experience is good, but set it high enough that your server doesn't get overloaded.  

Sent from my iPhone

--

[Derek]

I recently came across this article:

https://www.clairereynaud.net/blog/adding-offline-mode-to-your-mobile-app/

May be of interest as it deals with versions and revisions to track conflict between changes...

[Cal Leeming]

From what I can tell, you want to run a copy of the database on
multiple mini devices and have replication between them? If so, this
isn't really a good use case for replication, and adds several
unnecessary layers of complexity. So lets look at the end goal, you
want multiple clients to have read/write access to the same data, with
the constraint that the internet connection will be unreliable, but
you still want the data to be accessible, and possibly, editable.

This can be easily achieved by implementing good development
practises, similar to that of SPAs [1] or any "properly written"
mobile application. The application would maintain its own database,
for SPAs you'd probably use local storage [2] (assuming you didn't
need to support ancient browsers), and for mobile apps you can use an
embedded copy of SQLite3 [4] for relational structures, or couchbase
mobile [3] for document store. These databases would sync using your
"central" server, and your API design would depend on exactly what you
are trying to do, for example for simple requirements you could use
REST, but in others you may want to consider an RPC style, or you
might want to have the best of both worlds. There are some frameworks
out there which handle things like this, emberjs, angularjs etc,
though personally I am yet to find a decent, simple and clean
implementation of such a structure.

To achieve this, you also have to consider conflict handling, for
example if both users are offline and they edit a model, how would you
handle the conflict? If you automatically handle merge conflicts then
you also have to notify the affected users, and reflecting this in
your UX is *hard*. The theory of distributed systems are actually
quite simple, at least once you get past all the academic stuff, the
hard part is exposing it to the user in a friendly way.

You can get around many of these problems by disabling edits unless
there is an internet connection, allowing only read access in the
event of a connection drop out, then using model caching on the client
side (along with a bulk retrieve request). However both approaches
come with the same problem of delta changes, if a model changes then
each app will need to be notified about this, ideally via a push
mechanism. Designing an events system around this can be quite
challenging, for even the most seasoned developers.

If you have enough time/resources then it could be a fantastic
learning experience for you, but if this is a quick/dirty project
primarily lead by billable hours, then you probably don't want to
follow this route. If someone sent me a spec like this, I'd say it
would *easily* be 300+ hours work.

Hope this helps, if I have misunderstood, let me know.

Cal

[1]: http://en.wikipedia.org/wiki/Single-page_application
[2]: http://diveintohtml5.info/storage.html
[3]: http://www.couchbase.com/nosql-databases/couchbase-mobile
[4]: http://www.appcoda.com/sqlite-database-ios-app-tutorial/

[Blazor]

Hi Cal (and all others),

From what I can tell, you want to run a copy of the database on
multiple mini devices and have replication between them?

I think I've actually made myself not perfectly clear. My plan would be to have a local portable django server for letting all the tablets in a LAN (the only clients existing for it) access this local central information repository even when an outside internet connection is not available and let this portable server sync to the central remote storage whenever it can reach it. At the same time someone from the outside might edit the very same models directly in the remote repository. An automatic mechanism would be in place to compute deltas and merge changes from both: once over, both DBs would have the same data inside.

 

If so, this
isn't really a good use case for replication, and adds several
unnecessary layers of complexity.

Even though starting from preconditions not totally matching with the real use case, this conclusion equally holds true ;-)

 

So lets look at the end goal, you
want multiple clients to have read/write access to the same data, with
the constraint that the internet connection will be unreliable, but
you still want the data to be accessible, and possibly, editable.

Exactly.

 

This can be easily achieved by implementing good development
practises,

[...]

simple and clean
implementation of such a structure.

I'm not sure I see the connection here. Native mobile apps and/or html5 clients can leverage local storage strategies to be properly synchronized as soon as the connection is available. This would totally make sense in a context in which each CUD (not considering read here, since it's not much of a problem) operation is related to "personal non-shared" data only. The problem, here, is that more than one client will almost always perform some modifications to the same models, so your next point is not an exception, it's the rule.

 

To achieve this, you also have to consider conflict handling, for

[...] 

quite simple, at least once you get past all the academic stuff, the
hard part is exposing it to the user in a friendly way.

And this is exactly the problem I'm trying to face here. My original purpose was to understand whether there's any out-of-the-box solution managing a "dual-master" synchronization in django.

I found this [1], but besides requiring a patch to the Postgres 9.4 code base, is not yet considered production-ready.

 

You can get around many of these problems by disabling edits unless
there is an internet connection, allowing only read access in the
event of a connection drop out, then using model caching on the client
side (along with a bulk retrieve request).

By using a local django repository (local meaning  within the local wireless network among the tablets) I would get totally rid of the synchronization and notifications problems among all the native and/or html5 clients, moving the synchronization problem between the two databases only (the local and the remote one). Whether this would be an actual improvement or not, I'd have some more experienced user to decide.

 

However both approaches
come with the same problem of delta changes, if a model changes then
each app will need to be notified about this, ideally via a push
mechanism.

I am not an expert in this field and I'm afraid it would require a lot of time to design a proper merging mechanism.

 

Designing an events system around this can be quite
challenging, for even the most seasoned developers.

I get that.

 

If you have enough time/resources then it could be a fantastic
learning experience for you, but if this is a quick/dirty project
primarily lead by billable hours, then you probably don't want to
follow this route. If someone sent me a spec like this, I'd say it
would *easily* be 300+ hours work.

I'd like a quick and clean solution here ;-), since I have not enough time to follow all the required learning curve but I will be required to maintain and improve this project, so I'd rather pick a scalable solution from the beginning.

 

Hope this helps, if I have misunderstood, let me know.

It helps a lot, thanks for your time!

B.

[1] https://wiki.postgresql.org/wiki/BDR_User_Guide