Looking for a missing link in mobile based field survey data visualization / analysis

[Nikhil VJ]

Hi Friends,

This is regarding mobile-based field surveys. I'm using the "ODK
Collect" app on mobile, and a free account on "ona.io" as the
backend/dashboard/server.

I want to make a direct connection between our inputs (forms filled on
mobile phones) and the output (a map showing the collected data, and
maybe some data visualizations and tables). For that, one crucial link
in the chain is evading me. Need help from people who might already
know something about this. Please see this question on stackoverflow
and see if you can help:

http://stackoverflow.com/questions/32631442/bamboo-io-integration-with-xlsform-hosted-on-ona-io-like-formhub-was-providing

-----

For those interested in exploring possibilities to reduce data entry
headache in field surveys and even capture precise locations, photos
and audio in your surveys, you can get started by going on
http://ona.io , create an account, and read whatever comes on screen.
It's deceptively simple.

And if your org has a decent budget, there's a great bunch of people
offering to do this with much more features, with many social sector
projects already under the belt. Check out http://dhwaniris.com/


--
--
Cheers,
Nikhil
+91-966-583-1250
Pune, India
Self-designed learner at Swaraj University <http://www.swarajuniversity.org>
http://nikhilsheth.blogspot.in

[Devdatta Tengshe]

Hi Nikhil,

I'm assuming that you are looking for a free, hosted solution that does the entire thing, from data collection, to aggregation to maps completely for you. I'm afraid that something like this doesn't really exist. There are several commercial solutions which will do this, but they don't come cheap.

If on the other hand, you are willing to get your hands dirty, then this is very simple to do with something like ODK Aggregate. I had done this for an organisation in my previous job.

You need to setup ODK Aggregate with a postgres/mysql database, and you should push your data to this instance of ODK Aggregate. This data will be written to a set of tables, and then the only missing link is to write up a service which will Query this Database, and return your data back as JSON.

Let me know if you require any further details.

Regards,

Dev

--
Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org
---
You received this message because you are subscribed to the Google Groups "datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to datameet+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Regards,

Devdatta

[Nikhil VJ]

Hi Devdatta,

/* friends, going into gory details here, apologies if not understood */

Clarification: ona.io is already doing for me what you're saying about ODK Aggregate, without me having to set up my own server. They're basically hosting my Aggregate server for me. Do check it out! This was a big relevation and something that ate up many days of headbanging on my end, primarily because the folks on ODK website didn't bother sufficiently sharing about this option and were pushing me to roll my own ODK Aggregate instance instead. I stumbled upon it while looking for guidance on Aggregate. Link to relevant article: http://craigappl.github.io/Decisions-Decisions/

In its predecessor, formhub.org , there is a "bamboo.io" integrating button which generates a URL for me.

Proof-in-pudding : Please click this link

See some JSON there? The sources for it, are:

Form (go ahead, fill it!)
Form dashboard

To access this form from the ODK collect app on mobile, Hit Options > General Settings > Configure Platform Settings > URL : set to http://www.formhub.org/nikhilvj

>> and then you can fill the form from the mobile.

Go ahead and fill the form, either from web or from ODK Collect app (do remember to Send Finalized Forms so it's sent to server). You'll instantly see your submitted data collect, in JSON format, here.

I can import that (getJSON) into my javascript, and do my funky stuff with it.

So here I do have a free, hosted solution that handles the process from data collection to aggregation, and gives me everything in one JSON feed at a handy set URL. Which I will then process at my own end to do my mapping, visualization, etc.

The trouble comes with this : formhub.org isn't being maintained anymore; the people who created it aren't bothering to upgrade the servers to be able to handle more users & data. It's retiring. Already at certain hours when I go to the site, it doesn't load and I have to reload a couple of times (annoying!). The people who worked on it have now started ona.io as its clone and successor. So I moved there.. everything loads much faster there. If my field survey includes photos, audio and video then I'm certainly not going to risk working on formhub.org.

The only thing the wonderful ona.io guys left out is the easy bamboo.io integration that formhub.org had. So I'm not having a handy URL of a latest JSON feed for my forms on ona.io . That's the missing link (literally!). There IS functionality provided (REST service as well as API), but I'm at sea regarding exactly what to do to make things work here.. the documentations seem to be made for people who have some Linux or SSH terminal operating. I only have an HTML page and FileZillaFTP. . Really appreciated the direct no-torture linkage that formhub.org was giving. Looking for guidance on what to do, or any alternative path, to achieve the same ease-of-business here on ona.io which in all other aspects is meeting my needs perfectly.
(There are many ways described, but presently beyond my understanding levels)

So, stackoverflow question : http://stackoverflow.com/questions/32631442/bamboo-io-integration-with-xlsform-hosted-on-ona-io-like-formhub-was-providing

/* end of gory details */

[Ma-roof M]

Hi,

Recently, Nafundi has made their ready to deploy ODK Aggregate VM free of cost (pay as you wish).
Nafundi.com/products.

Kind regards
Mahroof

[Nikhil VJ]

Hi Friends,

So after taking one more shot at ona.io's API, I found the missing link:

This is where I can start:
https://ona.io/api/v1/data?owner=nikhilvj

This gave me the URLs to all my forms' collected data.

https://ona.io/api/v1/data/81012

This is giving me the live collecting data for one of my forms.
(Live as in I press Refresh and it's refreshed with the latest. I have
simple needs, ra)

https://ona.io/api/v1/data/81012?format=json
This is my data in JSON format.

So YAY! I now have a JSON source that I can do funky things with while
our team collects data on the streets :)

Insight : I think open source programmers could use some guidance on
how to communicate about their work in a way that the people who will
apply their creation, can understand. In ona.io's case, all they
needed to say was:

Start by going here:
https://ona.io/api/v1/data?owner=[your usename]
>> That's it! :P


@Maroof, thanks for the heads up! Had seen that too, but self-hosted
servers simply don't fit the use case scenario.

And I think same would be true for any social sector org out there
that just wants a simple substitute to collecting data on paper.
Seriously, we can do things quite simply.

I'm going to lay out what works, and no, it's not the proper way. But
it works, and it's about as difficult as setting up a facebook
account.

## GETTING STARTED WITH MOBILE BASED FIELD SURVEYS, FOR DUMMIES ##

1. Go to http://ona.io and sign up, create an account.

2. Start by duplicating existing stuff. Go to https://ona.io/forms/,
search for "nikhil", grab one of those excels.

3. Login on http://ona.io, upload one of the excels which is an XLSForm.

4. On your phone, install ODK Collect :
https://play.google.com/store/apps/details?id=org.odk.collect.android

5. On your profile / dashboard at ona.io (ie, the first place you see
after logging in), read and follow the instructions at the bottom to
hook your ODK Collect app with your stuff. Basically it's about
setting the server as https://ona.io/[your usename]

6. in ODK Collect app, "Get Blank Form", you'll see the form you
uploaded in step 3.

7. Now "Fill Blank Form", fill it up, and after completion, "Send
Finalized Form"

8. From the computer, on your dashboard, https://ona.io/[your
usename], you'll see the form has gotten an entry. You can see the
data in a table, if you geo-tagged it then you can see it on a map.
You can even see any photo you submitted in the form.

9. To use the data collected in your own programming, like a map or a
data visualization, go to this URL :
https://ona.io/api/v1/data?owner=[your usename]
..and follow your way through. (you might need to set the form's data
as public by going into the form's profile page)

10. And then many things you can do depending on how you want to apply
the data. HERE you would need a programmer who understands Javascript,
JSON and HTML. Till this point, you didn't really need any
programmer.. it was all point and click.

11. Open up the excels you downloaded, understand the way it's
structured, look up on http://xlsform.org/ for full documentation, and
reverse-engineer it to create your own surveys.

--
--
Cheers,
Nikhil
+91-966-583-1250
Pune, India
Self-designed learner at Swaraj University <http://www.swarajuniversity.org>
http://nikhilsheth.blogspot.in

On 9/18/15, Ma-roof M <mahr...@gmail.com> wrote:
> Hi,
>
> Recently, Nafundi has made their ready to deploy ODK Aggregate VM free of
> cost (pay as you wish).
> Nafundi.com/products.
>
> Kind regards
> Mahroof
> On 17 Sep 2015 22:25, "Devdatta Tengshe" <devd...@tengshe.in> wrote:
>
>> Hi Nikhil,
>>

>> I'm assuming that you are looking for a *free, hosted solution* that does