KoboCollect/KoboSync vs droidSURVEY

[DataMax]

Greetings everyone!
Just curious...
Has anyone tried using the Kobo tools to do a survey out in the field on numerous Android phones WITH sim cards AND then upload the results to a WEB SERVER running some sort of aggregator in order to generate an SPSS or csv data file??
Is it RELIABLE & easy to set up?

Also, how come no one has written a PHP server script that would do the aggregating since Java servlet hosting is EXPENSIVE compared to CHEAP shared Linux hosting where PHP5 and Ruby are already included?

Thanks!

[Neil Hendrick]

Greetings Max,

It has been done using ODK Aggregate for the server side. Aggregate will export CSV files after upload.

Is it easy to set up and reliable? While we are big supporters of the Open Data Kit community and a big believer in their tools, I don't think that Aggregate is all that easy to set up. As we say, "It's easy if you know how." And while older versions of Aggregate were a little wonky, the newest version is pretty stable. You have to watch out for large scale deployments because hosting Aggregate can incur charges on the Google App Engine platform.

The cheapest and easiest way to aggregate is to use KoBoSync or another offline aggregation tool. If it is important that you sync your data to a server (for example, your enumerators are geographically spread out) then there are other solutions. We use an Android DropBox client called "DropSync" that syncs your completed data to a DropBox account in the background.

We would be very interested to hear more about your ideas on a PHP/Ruby solution for aggregation.

thanks for you excellent questions and ideas,

~Neil

--
You received this message because you are subscribed to the Google Groups "Kobo Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/kobo-users/-/QvO-FZ9z0G4J.
To post to this group, send email to kobo-...@googlegroups.com.
To unsubscribe from this group, send email to kobo-users+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/kobo-users?hl=en.

[lizsa...@gmail.com]

I created a form with kobo form copied it to my android filled the form but it but am not able to aggregate it to my app account it takes so long and does not upload IV also tried sync but I don't get the form the way it should be maybe am not creating questions the way they should be please help.

Neil Hendrick <neil.h...@kobotoolbox.org> wrote:

>Greetings Max,
>
>It has been done using ODK Aggregate

><http://opendatakit.org/use/aggregate/>for the server side. Aggregate

>will export CSV files after upload.
>
>Is it easy to set up and reliable? While we are big supporters of the Open
>Data Kit community and a big believer in their tools, I don't think that
>Aggregate is all that easy to set up. As we say, "It's easy if you know
>how." And while older versions of Aggregate were a little wonky, the newest
>version is pretty stable. You have to watch out for large scale deployments
>because hosting Aggregate can incur charges on the Google App Engine
>platform.
>
>The cheapest and easiest way to aggregate is to use KoBoSync or another
>offline aggregation tool. If it is important that you sync your data to a
>server (for example, your enumerators are geographically spread out) then
>there are other solutions. We use an Android DropBox client called

>"DropSync<https://play.google.com/store/apps/details?id=com.ttxapps.dropsync>"

[DataMax]

Hi Neil,
Thanks for the reply! Well, what got me thinking about a PHP implementation was this text I saw on the Aggregate link:
**Looking for other servers? Try ODK Briefcase (for local offline storage), FormHub (for free data hosting on the cloud), DataHQ (like Aggregate but in Python), RapidSMS (for SMS integration) or OpenMRS (for medical records). **

Essentially, the Python code can be translated into PHP (or Ruby) -- if a coder was comfortable in both languages. So, since it's been done in Python, a PHP version shouldn't be that much of a challenge. Java servlet hosting is very expensive because of the memory footprint. :-) But PHP is available everywhere -- even on $3 per month Linux hosting plans. :-)
Thanks for the other tips!! :-)

Max
----------

On Friday, March 30, 2012 12:49:23 AM UTC+3, Liz Samuels, wrote:

I created a form with kobo form copied it to my android filled the form but it but am not able to aggregate it to my app account it takes so long and does not upload IV also tried sync but I don't get the form the way it should be maybe am not creating questions the way they should be please help.

Neil Hendrick  wrote:

>Greetings Max,
>
>It has been done using ODK Aggregate
><http://opendatakit.org/use/aggregate/>for the server side. Aggregate
>will export CSV files after upload.
>
>Is it easy to set up and reliable? While we are big supporters of the Open
>Data Kit community and a big believer in their tools, I don't think that
>Aggregate is all that easy to set up. As we say, "It's easy if you know
>how." And while older versions of Aggregate were a little wonky, the newest
>version is pretty stable. You have to watch out for large scale deployments
>because hosting Aggregate can incur charges on the Google App Engine
>platform.
>
>The cheapest and easiest way to aggregate is to use KoBoSync or another
>offline aggregation tool. If it is important that you sync your data to a
>server (for example, your enumerators are geographically spread out) then
>there are other solutions. We use an Android DropBox client called
>"DropSync<https://play.google.com/store/apps/details?id=com.ttxapps.dropsync>"
>that syncs your completed data to a DropBox account in the background.
>
>We would be very interested to hear more about your ideas on a PHP/Ruby
>solution for aggregation.
>
>thanks for you excellent questions and ideas,
>
>~Neil
>

[ニコノコ]

Neil,

You mentioned you're using DropBox, and this is a fairly recent message. 

I'm trying out that particular option right now but stuck with an error from Collect RC2: "Error Getting Form List:

Form listing failed. ContentType: text/html; http://192.168.2.102:8080/ formList is not text/xml. This is often caused a network proxy. Do you need to login to your network?"

Have you ever encountered this and found a solution?

Regards,

Nik

To unsubscribe from this group, send email to kobo-users+unsubscribe@googlegroups.com.

[ニコノコ]

Geesh.

I thought I read "DropBox" (like in the DropBox component of ODK) but reading Neil's message one more time and clicking on the link he provided, it was obviously something else.

Sorry for the bother. 

P.S.

Any comments about the question is welcome, though.

On Tuesday, June 19, 2012 8:38:54 AM UTC+8, ニコノコ wrote:

Neil,

You mentioned you're using DropBox, and this is a fairly recent message. 

I'm trying out that particular option right now but stuck with an error from Collect RC2: "Error Getting Form List:

Form listing failed. ContentType: text/html; http://192.168.2.102:8080/ formList is not text/xml. This is often caused a network proxy. Do you need to login to your network?"

Have you ever encountered this and found a solution?

Regards,

Nik

On Thursday, March 29, 2012 10:47:51 PM UTC+8, Neil Hendrick wrote:

Greetings Max,

It has been done using ODK Aggregate for the server side. Aggregate will export CSV files after upload.

Is it easy to set up and reliable? While we are big supporters of the Open Data Kit community and a big believer in their tools, I don't think that Aggregate is all that easy to set up. As we say, "It's easy if you know how." And while older versions of Aggregate were a little wonky, the newest version is pretty stable. You have to watch out for large scale deployments because hosting Aggregate can incur charges on the Google App Engine platform.

The cheapest and easiest way to aggregate is to use KoBoSync or another offline aggregation tool. If it is important that you sync your data to a server (for example, your enumerators are geographically spread out) then there are other solutions. We use an Android DropBox client called "DropSync" that syncs your completed data to a DropBox account in the background.

We would be very interested to hear more about your ideas on a PHP/Ruby solution for aggregation.

thanks for you excellent questions and ideas,

~Neil

[Vinck, Patrick]

Hi Nik,

 

Dropsync works with your existing DropBox account, but the difference is that the dropbox app on your phone does not let you specify what folder you would like to add (i.e. your “ODK” folder on the phone.). Dropsync does…

 

Sometime we would like to look further into this and build a KoBocollect that includes a dropsync function.

 

Best,

 

Patrick

--

You received this message because you are subscribed to the Google Groups "Kobo Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/kobo-users/-/maqFFmv__gcJ.

To post to this group, send email to kobo-...@googlegroups.com.

To unsubscribe from this group, send email to kobo-users+...@googlegroups.com.

[Neil Hendrick]

To add a little further to Patrick's comments; Dropsync is a good solution to keep your data synchronized to a remote server in a way that is not difficult to set up, does not require a special server, and is reliable.
It does require you to have some level of connectivity. You can configure dropsync to run in the background all the time, or to run on a schedule, like once an hour or once a day.
 It has the limitation that it will upload your completed surveys, but it will not download new forms, for example if you had made changes to your survey and put them in a dropbox folder. So, it's good for synchronizing data, not so good for updating your survey.

~Neil Hendrick

[ゴー・ニコライ]

Thank you Patrick, Neil for your responses.

Yes, pretty nice, only if you have some "level of connectivity" which is my concern. I wonder if KoBo is working on non-Internet alternatives to KoBo Sync, without fiddling with USB connections (i.e. adhoc WiFi on notebook?)

[Neil Hendrick]

Nik,

You might try Bluetooth File Transfer. That would give you a local wireless connection. Establishing the BT connection, however, would be no quicker than plugging in a USB cable. 

There are some other options, you could use Android Debug Bridge, which still requires you to plug in a cable but allows you to use the 'pull' command to sync your data without having to mount your SD card. It's makes it fast, if you are comfortable on the command line. 

If you don't like command line, or don't want to go to the trouble of using ADB, you can save yourself a couple of clicks and swipes by using an app to auto-mount your SDcard when you plug in the cable. 

Those are some ideas, but keep in mind that no matter what you have to push your data in some way, and it can be a little tedious, but it is the most important part of the whole process, so if you have to put some effort, a method that is tedious and reliable is better than a method that is easy and unreliable. 

~Neil

--

[ゴー・ニコライ]

Thanks again Neil.

If that doesn't happen, the options are:

• Run a local ODK dropbox server, ask each enumerator to sync with it while debriefing. (multi-tasking, meet while synching in the background)
• Direct USB hookup for every handset and sync using ADB (or Automount app) on USB cable - (surefire method, need to spend time on each handset)
• local FTP server and FTP clients run on android (somewhat tedious) and connects using local wifi - slow, and would require too much fiddling on the handset. (I just tried this, and though wireless works, it's just too slow)

I'm okay with CLI, that's not going to be any hassle. And I agree that reliability tops convenience. I'm hoping to learn from your experience here for an optimal way of handling 40-50 handsets (for the pilot, may double in other pilot tests) with the least amount of pain and heartache o_0. 

I'm scouring the archives for comments and feedback about your experience on this.

On Tuesday, June 19, 2012, Neil Hendrick wrote:

Nik,

You might try Bluetooth File Transfer. That would give you a local wireless connection. Establishing the BT connection, however, would be no quicker than plugging in a USB cable. 

There are some other options, you could use Android Debug Bridge, which still requires you to plug in a cable but allows you to use the 'pull' command to sync your data without having to mount your SD card. It's makes it fast, if you are comfortable on the command line. 

If you don't like command line, or don't want to go to the trouble of using ADB, you can save yourself a couple of clicks and swipes by using an apcollectsp to auto-mount your SDccollectsard when you plug in the cable. 

--

You received this message because you are subscribed to the Google Groups "Kobo Users" group.
To post to this group, send email to kobo-...@googlegroups.com.
To unsubscribe from this group, send email to kobo-users+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/kobo-users?hl=en.

[Neil Hendrick]

Nik,

One more thought is that if you are using a multi-port hub, you can plug in a number of phones at the same time. If you are using Android Debug Bridge, you can create a bat script with the Device ID # of every phone in your arsenal, and a 'pull' command for each one. ADB can talk to more than one phone at a time. 

Here's a little more about how to use ADB. 

To get the ID for each phone, plug them in, open a command line, and type "adb devices". This will give you the serial number for each device.

 To use 'pull', you normally just do it like this:

adb pull <remote> <local>

But, if you are using multiple devices, you have to specify the device serial number. For each device, you need a line in the bat with the switch "-s". 

For example, my phone tells me it's serial number is SH06EPB01400 when I use the command 'adb devices'.

So, If I want to make sure I am talking to that phone when I pull my data, I specify it like this:

adb -s SH06EPB01400 pull /sdcard/odk/instances/  C:\Users\Neil\Sync\

To make it clear, this command is saying "Android Debug Bridge, check the connected phone with the serial number SH06EPB01400, get everything out of the /sdcard/odk/instances/ directory and copy it over to my  C:\Users\Neil\Sync\  directory. 

I can make a bat command with a line like that for every device in my arsenal. 

adb -s SH06EPB01400 pull /sdcard/odk/instances/  C:\Users\Neil\Sync\ 

adb -s QL06EPB03294 pull /sdcard/odk/instances/  C:\Users\Neil\Sync\ 

adb -s PR06EPB40243 pull /sdcard/odk/instances/  C:\Users\Neil\Sync\ 

adb -s ZZZ06EPB76403 pull /sdcard/odk/instances/  C:\Users\Neil\Sync\ 

If you have 50 phones in your arsenal, you can't plug them all in to the same hub (unless you have the world's most magnificent USB hub), but it doesn't matter. Your BAT file has all the phone's serial numbers in it, if only ten of them are connected, the rest of the lines nothing happens, you sync your 10 connected phones anyway. Then you connect 10 new phones and run the same BAT. 

A word of caution. If you are collecting enough data you can run into a conflicting name. By default, KoBoCollect names each completed survey instance with the name of the survey plus the date and time to the second. 

"KoBo_Demo_2012-05-30_23-13-10.xml"

If you have a lot of  people doing surveys at the same time, it is possible that two phones will create a survey at the same second. If that happens, you will end up with two surveys with the same name. If that happens, using the above method one survey will over-write the other survey and you will LOSE DATA! This is the worst thing that can happen. This may seem far fetched, but it has happened to us in the field when we had a very large sample size (4500) and 50 phones in the field collecting data every day for six weeks. Out of that group, there were about 8 sets of survey instances that happened on the same day, at the same time to the second.

In order to avoid this, KoBoSync changes the name of the survey when it synchronizes the data by adding the device ID to the name of the instance, so that it is impossible to have two surveys with the same name because it is impossible for the same phone to make two surveys at the same second. You get a name like:

351771046714334_KoBo_Demo_2012-05-30_23-13-10.xml

~Neil

☞§※☼✈☂☻

~Neil

[ゴー・ニコライ]

Very good tips Neil. Thanks.

--