Local proxies in Myanmar ?

[Charles Flèche]

Hi everyone,


Context
=======

We are implementing an MNCH project in rural Myanmar : community health
workers and midwives are provided a smartphone with a diagnosis helper /
patient registry / medical files application. CommCare is our framework,
mainly thanks to its case sharing feature.

This mHealth project is a part of a bigger initiative that aims to build
capacity and support these CHW. Our main activity consists of monthly
refresher trainings.

We operate in Dala, a peri-urban zone very close to Rangoon. We'll scale
the pilot project up to the Kayin (Karen) State at the end of the year,
a much more rural and politically sensitive region.


Connectivity issues
===================

We are facing a connectivity problem. In the villages, connections are
unreliable at best and often unusable most of the time. My guess is the
network backbone doesn't support the increased traffic incurred by the
mobile phone boom of the last 2/3 years. You can actually connect to the
mobile network, but the backbone seems so congested it becomes unusable.
We are talking pings to 8.8.8.8 > 30000ms here (no typo).

However, during the monthly refresher training meetings, we can sync the
phones using the ADSL connexion of our Dala office. It is slow, painful
and unreliable, but at least we can update and sync the devices. To give
you an idea, last time it took me an entire day to update 18 phones…

Obviously the situation will be much worse in Kayin State. We do hope
that the state of network connexions will improve in Myanmar with the
arrival of two new operators (Ooredoo from Qatar and Telenor from
Norway), either by building new networks or by forcing the current
operator MPT to invest and upgrade its current systems. However for the
time being connecting to a CommCare server over the internet will prove
to be a little complicated.


Local servers
=============

I was thinking about deploying our local installs of CommCare servers.
Each base would have a small, portable server coupled with a WIFI access
point (a laptop / small computer like a BeagleBone or Raspberry Pi and a
WIFI dongle configured as Infrastructure Access Point should do it).

During the monthly trainings, this server would be carried to the
training location. Users will directly connect through the LAN WIFI and
update / sync their devices. Back at the base, M&E staff could get their
reports directly from the server on the LAN. This portable server could
also be carried during the field visits that our monitors do on a
regular basis, hence syncing the devices more often.

With these portable CommCare servers, we wouldn't rely on dodgy Internet
connexions, everything would be done over LANs.


In fact, local proxies
======================

Apart from the obvious extra cost of buying / installing / maintaining /
backuping these servers, the main issue will be to keep all these
CommCare instances in sync in term of forms / users management /
application building. The last thing I'd want would be to manually
manage a cluster of unrelated CommCare installations : users connecting
to a local, LAN server should have access to the same data as if they
were connecting to a unique central instance. The distributed nature of
this architecture should be hidden from the user (granted, minus sync
delays between servers).

With this configuration, local servers would initiate connections and
sync to a central server overnight when connexion are more reliable.
They would push their forms gathered form the field and pull application
/ users / configuration / forms updates.


Implementation
==============

So I was wondering :

1. Am I over-engineering this stuff ? Is a simpler, more straightforward
workflow possible ?

2. Did someone already implement such an infrastructure ? How did it go ?

3. Does CommCare support such a feature already : local proxies ?

4. As a first step, even if the users management was totally
decentralized and not consolidated between servers, at least I'd like to
have reports and applications updated to local proxies on a nightly
basis. I see a cron job running a python script getting reports /
applications through the API from a central server then pushing these to
a local instance. Do you think it could work ?


My apologies for this long email…

Many thanks,


--
Charles Flèche
mHealth advisor
Télécoms Sans Frontières http://www.tsfi.org
Première Urgence - Aide Médicale Internationale http://www.pu-ami.org
+95 9 431 978 25
Skype: charles.fleche

[Clayton Sims]

Charles,

Before jumping into too many details, I have one question for you.

Have you tried configuring a test app against our server in india at "india.commcarehq.org"? It's significantly closer geographically and may have better connectivity.

We have some tools to help mitigate long-term offline connections, but we don't currently have a micro-node setup for handling the process you're describing currently. We can chat about some potential options once it's clear whether the india server is a better fit.

-Clayton

On Mon, Aug 11, 2014 at 3:34 AM, Charles Flèche <mhealth...@tsfi.org> wrote:

Hi everyone,


Context
=======

We are implementing an MNCH project in rural Myanmar : community health workers and midwives are provided a smartphone with a diagnosis helper / patient registry / medical files application. CommCare is our framework, mainly thanks to its case sharing feature.

This mHealth project is a part of a bigger initiative that aims to build capacity and support these CHW. Our main activity consists of monthly refresher trainings.

We operate in Dala, a peri-urban zone very close to Rangoon. We'll scale the pilot project up to the Kayin (Karen) State at the end of the year, a much more rural and politically sensitive region.


Connectivity issues
===================

We are facing a connectivity problem. In the villages, connections are unreliable at best and often unusable most of the time. My guess is the network backbone doesn't support the increased traffic incurred by the mobile phone boom of the last 2/3 years. You can actually connect to the mobile network, but the backbone seems so congested it becomes unusable. We are talking pings to 8.8.8.8 > 30000ms here (no typo).

However, during the monthly refresher training meetings, we can sync the phones using the ADSL connexion of our Dala office. It is slow, painful and unreliable, but at least we can update and sync the devices. To give you an idea, last time it took me an entire day to update 18 phones...

Obviously the situation will be much worse in Kayin State. We do hope that the state of network connexions will improve in Myanmar with the arrival of two new operators (Ooredoo from Qatar and Telenor from Norway), either by building new networks or by forcing the current operator MPT to invest and upgrade its current systems. However for the time being connecting to a CommCare server over the internet will prove to be a little complicated.


Local servers
=============

I was thinking about deploying our local installs of CommCare servers. Each base would have a small, portable server coupled with a WIFI access point (a laptop / small computer like a BeagleBone or Raspberry Pi and a WIFI dongle configured as Infrastructure Access Point should do it).

During the monthly trainings, this server would be carried to the training location. Users will directly connect through the LAN WIFI and update / sync their devices. Back at the base, M&E staff could get their reports directly from the server on the LAN. This portable server could also be carried during the field visits that our monitors do on a regular basis, hence syncing the devices more often.

With these portable CommCare servers, we wouldn't rely on dodgy Internet connexions, everything would be done over LANs.


In fact, local proxies
======================

Apart from the obvious extra cost of buying / installing / maintaining / backuping these servers, the main issue will be to keep all these CommCare instances in sync in term of forms / users management / application building. The last thing I'd want would be to manually manage a cluster of unrelated CommCare installations : users connecting to a local, LAN server should have access to the same data as if they were connecting to a unique central instance. The distributed nature of this architecture should be hidden from the user (granted, minus sync delays between servers).

With this configuration, local servers would initiate connections and sync to a central server overnight when connexion are more reliable. They would push their forms gathered form the field and pull application / users / configuration / forms updates.


Implementation
==============

So I was wondering :

1. Am I over-engineering this stuff ? Is a simpler, more straightforward workflow possible ?

2. Did someone already implement such an infrastructure ? How did it go ?

3. Does CommCare support such a feature already : local proxies ?

4. As a first step, even if the users management was totally decentralized and not consolidated between servers, at least I'd like to have reports and applications updated to local proxies on a nightly basis. I see a cron job running a python script getting reports / applications through the API from a central server then pushing these to a local instance. Do you think it could work ?

My apologies for this long email...

Many thanks,

--
Charles Flèche
mHealth advisor
Télécoms Sans Frontières http://www.tsfi.org
Première Urgence - Aide Médicale Internationale http://www.pu-ami.org
+95 9 431 978 25
Skype: charles.fleche

--
You received this message because you are subscribed to the Google Groups "commcare-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commcare-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cory Zue]

Hi Charles,

What you're describing is an architecture that the project that was predecessor to CommCare HQ was actually designed to solve. As a result a lot of what you're looking for could be supported relatively cleanly by the underlying technology. Right now the primary data store of CommCare is CouchDB which is designed to handle bidirectional replication across a number of separate nodes via its replication engine. If you do end up going with the network of servers approach I'd recommend you look into using that as your underlying sync mechanism. There will be some problems with it, since not all CommCare data is stored in CouchDB anymore, but a lot of things (including specifically reports) should just work. We had good success with this architecture for a 40-clinic deployment in Zambia with very bad network (where each clinic worked fully offline and then synced with couchdb replication in the background).

All that said, this will be a bit flaky and the devil is in the details. Your API based approach will make a lot of that cleaner and allow your central server to be cloud HQ (we don't support database-level access so replication wouldn't be an option) but at the expense of having to explicitly do work for each category of data that you want to push over.

When you think you have a plan I'm happy to provide input via the developers list.

Cory

To unsubscribe from this group and stop receiving emails from it, send an email to commcare-user...@googlegroups.com.

[Charles Flèche]

Hi Clayton,

On 11/08/2014 22:21, Clayton Sims wrote:
> Have you tried configuring a test app against our server in india at

> "india.commcarehq.org <http://india.commcarehq.org>"?


Many thanks for the suggestion. It seems like india.commcare.org domain
cannot be resolved at the moment :

$ nslookup india.commcare.org commcare.org
Server: commcare.org
Address: 198.1.112.104#53

** server can't find india.commcare.org: NXDOMAIN


I'll try using the Indian server when it is available again. However,
hitting a closer machine may not solve my connectivity issues as in some
area I just cannot connect at all or the network backbone seems so
congested that if you can actually connect to a mobile network BTS, you
can't do anything with it. Even SMS are extremely unreliable in these areas…

Thanks for the india.commcare.org tip anyway.

Charles

[Charles Flèche]

Hi Cory,

Many thanks for your swift answer.

On 12/08/2014 02:29, Cory Zue wrote:
> There will be some problems with it, since not all CommCare
> data is stored in CouchDB anymore

Is there a documentation (I'm thinking code comments) that states what
is stored in CouchDB and what is in PostgreSQL ? For my geeky curiosity,
what is the rational behind using two types of DB ?

> but a lot of things (including specifically reports) should
> just work. We had good success with this architecture for a 40-clinic
> deployment in Zambia with very bad network (where each clinic worked
> fully offline and then synced with couchdb replication in the background).

Seems like a challenging deployment, congratulation for that ! How did
you do the sync if the clinics were fully offline : shuffling USB keys
around ?

> All that said, this will be a bit flaky and the devil is in the details.
> Your API based approach will make a lot of that cleaner and allow your
> central server to be cloud HQ (we don't support database-level access so
> replication wouldn't be an option) but at the expense of having to
> explicitly do work for each category of data that you want to push over.

Yeah, I'm not totally confident with brute-force DB storage replication,
especially in a mixed PostgreSQL / CouchDB environment, but as I have no
experience with such procedures so it is probably just me being too
defensive.

> When you think you have a plan I'm happy to provide input via the
> developers list.

My contract here in Myanmar ends pretty soon and it is still unclear if
I'll stay longer in order to implement this distributed architecture or
if it will be my successor's duty. However I intend to leave with at
least a few ideas on how to solve our current issues.

Context:
- All servers are LAN hosted on LANs, behind our firewalls
- Master server (M) is hosted in our main office, with a descent (well,
Myanmar-descent…) internet connection and is always online.
- Proxy servers (P) are hosted in remote bases, with an unreliable and
slow internet connection (3G), are mostly offline but can be connected
overnight for data sync

Note : it is unclear if our ISP allows static IPs for our main office
connection. If not, the main office server will be another P, while M
would be hosted in the cloud HQ. For simplicity, let's consider M to be
hosted behind our firewall for the time being.

In a ideal world, every P is the exact copy of M. However, a minimal
system allows an application to be built on M, then pulled from P on a
regular basis. P pushes its forms to M overnight.

User management is decentralized : M does not know anything about the
users. In fact, M has just a few test mobile users configured. It
doesn't know anything about cases too. It just stores forms from every
bases. We assume that formid and caseid are random enough not to
collide, coming from different P.

1. Does the API allow to pull / push applications ?

2. Does the API allow to push forms to another server ?

3. What would be the best strategy to adopt for pushing new forms from P
to M ? Keeping a list on P of already pushed forms (in the DB itself
maybe) ? Querying for existing formids on M before pushing the ones not
already on M ?

4. Would reporting break if forms state unknown caseids (as M would
known nothing about the mobiles users / cases on P) ?


Many thanks everyone,

[bmcd...@proximitydesigns.org]

Hi Charles,

You will run into trouble trying to get a fixed ipaddress here in Myanmar. Its very expensive and most of the companies that say they offer fixed ipaddresses dont really, theyre are just fixed ipaddresses on the isp's subnet.

Brian

[Charles Flèche]

On 12/08/2014 10:31, bmcd...@proximitydesigns.org wrote:
> You will run into trouble trying to get a fixed ipaddress here in
> Myanmar.

Hum, good to know, thanks for that Brian. What about IPv6 ? Would
CommCare work on an IPv6 stack ? Would ISPs provide an IPv6 address block ?

So a Master cloud server may be the best option here.

[Craig A.]

Hi Charles,

Your nslookup uses india.commcare.org and it should be india.commcarehq.org.

Sincerely,

Craig

[Charles Flèche]

On 12/08/2014 11:21, Craig A. wrote:
> Your nslookup uses india.commcare.org and it should be india.commcarehq.org.

Whoopsie… Thanks for that Craig.

[Cory Zue]

Hey Charles,

Very quick answers (dropping cc-users via bcc).

On 12/08/2014 02:29, Cory Zue wrote:

There will be some  problems with it, since not all CommCare

> data is stored in CouchDB anymore

Is there a documentation (I'm thinking code comments) that states what is stored in CouchDB and what is in PostgreSQL ?

Not formal documentation. You can look for subclasses of a django Model as opposed to a couchdbkit Document.

 

For my geeky curiosity, what is the rational behind using two types of DB ?

We chose couchdb as our primary data store and then ran into some problems with it (mostly around performance and reporting) so started moving some things to SQL.

but a lot of things (including specifically reports) should
just work. We had good success with this architecture for a 40-clinic
deployment in Zambia with very bad network (where each clinic worked
fully offline and then synced with couchdb replication in the background).

Seems like a challenging deployment, congratulation for that ! How did you do the sync if the clinics were fully offline : shuffling USB keys around ?

Sorry, they functioned fully offline via a LAN, but had netsticks that could eventually sync via the cell network.

All that said, this will be a bit flaky and the devil is in the details.
Your API based approach will make a lot of that cleaner and allow your
central server to be cloud HQ (we don't support database-level access so
replication wouldn't be an option) but at the expense of having to
explicitly do work for each category of data that you want to push over.

Yeah, I'm not totally confident with brute-force DB storage replication, especially in a mixed PostgreSQL / CouchDB environment, but as I have no experience with such procedures so it is probably just me being too defensive.

When you think you have a plan I'm happy to provide input via the
developers list.

My contract here in Myanmar ends pretty soon and it is still unclear if I'll stay longer in order to implement this distributed architecture or if it will be my successor's duty. However I intend to leave with at least a few ideas on how to solve our current issues.

Context:
- All servers are LAN hosted on LANs, behind our firewalls
- Master server (M) is hosted in our main office, with a descent (well, Myanmar-descent…) internet connection and is always online.
- Proxy servers (P) are hosted in remote bases, with an unreliable and slow internet connection (3G), are mostly offline but can be connected overnight for data sync

Note : it is unclear if our ISP allows static IPs for our main office connection. If not, the main office server will be another P, while M would be hosted in the cloud HQ. For simplicity, let's consider M to be hosted behind our firewall for the time being.

In a ideal world, every P is the exact copy of M. However, a minimal system allows an application to be built on M, then pulled from P on a regular basis. P pushes its forms to M overnight.

User management is decentralized : M does not know anything about the users. In fact, M has just a few test mobile users configured. It doesn't know anything about cases too. It just stores forms from every bases. We assume that formid and caseid are random enough not to collide, coming from different P.

(this should be a fine assumption)

1. Does the API allow to pull / push applications ?

(Assume you are referring to the HQ APIs in this section. The couch replication would support all of these things out of the box.)

We currently only support pull, but adding support for push is something that could be done relatively easily.

2. Does the API allow to push forms to another server ?

Yes. You would have to use the submission API.

 

3. What would be the best strategy to adopt for pushing new forms from P to M ? Keeping a list on P of already pushed forms (in the DB itself maybe) ? Querying for existing formids on M before pushing the ones not already on M ?

Simplest would likely be keeping a list of whether the form had been pushed, which only gets set once you get a 2XX response from the submission API. You could also try using the built in form forwarding (which basically does that), although the backoff it has might be too aggressive for your setup.

4. Would reporting break if forms state unknown caseids (as M would known nothing about the mobiles users / cases on P) ?

If you are able to forward the forms in order you can rely on case processing to recreate the cases on M, so this theoretically could work (you would also need to sync users to have the reports work).

API docs are here in case you hadn't seen them: https://confluence.dimagi.com/display/commcarepublic/CommCare+HQ+APIs

Cory

Many thanks everyone,

--
Charles Flèche
mHealth advisor
Télécoms Sans Frontières http://www.tsfi.org
Première Urgence - Aide Médicale Internationale http://www.pu-ami.org
+95 9 431 978 25
Skype: charles.fleche