Fwd: next gen data collection

[Yaw Anokwa]

hey all,

a few of us have been having discussions about what the next generation of data collection might look like. much of this is driven by a dissatisfaction with xforms and a desire to move to technologies that more forward looking.

it'd be great to get some feedback from this workgroup about the ideas, as well as some use cases we could explore together.

yaw

Forwarded conversation
Subject: next gen data collection
------------------------

From: Yaw Anokwa <yan...@gmail.com>
Date: Fri, May 27, 2011 at 18:53
To: mikey...@gmail.com
Cc: Eduardo Jezierski <ed...@instedd.org>, Nicolás di Tada <ndi...@instedd.org>, Jeff Wishnie <jwis...@thoughtworks.com>, Matt Berg <mlb...@gmail.com>, Gaetano Borriello <gae...@cs.washington.edu>


great chatting everyone. looking forward to the next generation of
data collection...

----------
From: Mike McKay <mikey...@gmail.com>
Date: Sat, May 28, 2011 at 02:11
To: Yaw Anokwa <yan...@gmail.com>
Cc: Eduardo Jezierski <ed...@instedd.org>, Nicolás di Tada <ndi...@instedd.org>, Jeff Wishnie <jwis...@thoughtworks.com>, Matt Berg <mlb...@gmail.com>, Gaetano Borriello <gae...@cs.washington.edu>


Guys,

Thanks for the call this morning. Before talking about the next generation I think it's worth pointing out how great ODK, OpenROSA, EpiSurveyor, etc etc - all are. I think one of the  key success factors has been collaboration and coordination (perhaps competition?) across many organization. Quite honestly, it's that community of success that appeals so much to me.

At Baobab Health, we created a system called the Touchscreen Toolkit. It was a touchscreen forms engine aimed at developer use that we built from scratch in Malawi using plain javascript (not even JQuery!) with everything rendering in the browser. We used plain HTML forms and then created about a dozen different attributes that we would add to form elements as needed. Our javascript library would search for those attributes and then apply the logic. So imagine an input element:

<input type='text' name='weight'></input>

to add validation to it you just add some attributes:
<input absoluteMax='300' type='text' name='weight'></input>

or skip logic:
<input type='text' id='gender' name='gender'></input>
<input condition='$("#gender")!="female"' type='text' name='weight'></input>

In the above case the condition attribute get's eval'd to determine if that field should be displayed (never ask a woman her weight!).

or autocomplete (appends the current value to an AJAX request and provides a dropdown to select from)
<input type='text' ajaxURL='/country/list?search=' name='country'></input>

There are more of these attributes, which we created organically as needed and it has worked quite well for us for the past 5 years (Partners In Health uses it a bit as well). We never added repeatable form fields or sections, but we found we could fake that feature by using conditions to determine when an additional field should be shown.

Here is the documentation that we still maintain and use in Malawi:
http://docs.google.com/View?id=dcw6g5mq_2fprpj5

At one point, we got some funding to make "Formtastic.us", which was an attempt to generalize these forms beyond the touchscreen and to integrate them tighter with the OpenMRS concept dictionary. We did this, but never managed to deploy it to a project (we switched from innovation mode to scale mode and there were a lot of fires to fight!). I created a short video many years ago to demonstrate formtastic.us. It's worth watching (6 mins) to see where we were heading as I think it can inform where ODK is heading:

http://youtu.be/iGb5HUBVt1w

Finally, I have been working with colleagues on a research & development proposal within RTI (http://www.ictedge.org/) that would aim to mash up some of the technologies and ideas/successes of the following:

* ODK (Build/Create/Aggregate)
* ZCore
* JQuery Mobile
* CouchDB
* Touchscreen Toolkit

We all know what ODK is, so I can skip that one. ZCore is an RTI developed system that has been used to build a number of different data collection and decision support systems in Africa (http://www.ictedge.org/zcore). It is mostly java and javascript, renders in the browser, and has some decent syncing capabilities that we have developed.  JQuery Mobile is a javascript UI tool that renders forms across an impressively wide array of smartphones and tablets:

• Apple iOS (3.1-4.2): tested on iPhone, iPod Touch, iPad
• Android (1.6-2.3): all devices, tested on the HTC Incredible, Motorola Droid, Google G1 and Nook Color
• Blackberry 6: tested on Torch and Style
• Windows Phone 7: tested on HTC Surround
• Palm WebOS (1.4): tested on Pre, Pixi
• Opera Mobile (10.1): Android
• Opera Mini (5.02): iOS, Android
• Firefox Mobile (beta): Android

CouchDB is a NOSQL database that can be embedded onto both iphone and android (or a regular cloud server). It has sync built-in (databases on phones can sync with each other or to the cloud for instance) and as Matt mentioned you can serve applications directly from it, so you don't even need a separate web server for your html, javascript and css. All database requests are just HTTP request. Matt and Jeff are working on an app that uses couchdb, we have been developing a couchapp for education (pure js/html/couch uses JQuery Mobile, no python/java/ruby) at RTI, and I know Cory at Dimagi has been doing some great stuff with CouchDB as well.

So all that said my vision would look something like:

* ODK Build-like UI for creating "basic" forms (built as couchapp so you could create forms in the cloud or offline on your smartphone)
* Forms would be stored as HTML5 forms (probably using the "data-" attributes of HTML5 to define form logic (a la touchscreen toolkit), or perhaps JSON), but not XForms
* Data collection would happen in a browser, with all assets being served and saved from a CouchDB instance (probably a local one so that it would all work offline) (tangential thought/ feature: consider using HTML5 offline for when CouchDB is not possible).
* Simple data analysis would be available locally in the browser, with logic served via a couchapp (using flot for charting). Data could also sync to a cloud couchdb instance to merge data sets and run "queries" that require lots of horsepower (couchdb can shard and distribute map/view logic).
* (/me waves hands vigorously) encryption, ability to link to existing records, dashboard functionality that queries other records for decision support

Wow, well that is a much longer email than I had planned to write (I wonder how many of you nerds actually read this far on a Friday night), but I ended up laying out my vision for the future of data collection. I look forward to hearing some of your own visions!

Mike

----------
From: Matt Berg <mlb...@gmail.com>
Date: Tue, May 31, 2011 at 03:02
To: Mike McKay <mikey...@gmail.com>
Cc: Yaw Anokwa <yan...@gmail.com>, Eduardo Jezierski <ed...@instedd.org>, Nicolás di Tada <ndi...@instedd.org>, Jeff Wishnie <jwis...@thoughtworks.com>, Gaetano Borriello <gae...@cs.washington.edu>


Mike,

Thanks this is really great and I apologize for the delay in writing back.  I did enjoy reading it Friday night though.

Agree strongly with most of what you propose.

Couple of comments/questions:

• need a bit of understanding of how zcore fits in to the picture. It seems like a java base for rendering javascript on a number of different interfaces?  Needing to know more, my gut says we might want to keep the use of java to a minimum and try and do it all with javascript/html5 and maybe the webserver couch provides.
• I think we will still want a way (most likely in json) to represent a form.  We will want to ensure we can support an ecosystem of form authoring tools.  I also think having a form representation is import to serve as a transport layer.  There will be times you want to store and transfer completed forms as files and not always rely on couchdb sync.

I'm really excited to see this move forward and am hoping our team can help.

Having a simple initial app that can serve as a starting point I think will be a tremendous help.  Do you think the learning tool you described could serve that purpose or should we start with something basic and just iterate from there?  

We might just want to start with html5 forms as you describe and see what time of form abstraction comes out of it based on our need.

Thanks again.  Really look forward to seeing where this leads us.

Matt

----------
From: Matt Berg <mlb...@gmail.com>
Date: Tue, May 31, 2011 at 05:25
To: Mike McKay <mikey...@gmail.com>
Cc: Yaw Anokwa <yan...@gmail.com>, Eduardo Jezierski <ed...@instedd.org>, Nicolás di Tada <ndi...@instedd.org>, Jeff Wishnie <jwis...@thoughtworks.com>, Gaetano Borriello <gae...@cs.washington.edu>


You've probably seen this before but this provides some nice overview of what a couchapp is:

http://couchapp.org/page/what-is-couchapp

----------
From: Gaetano Borriello <gae...@cs.washington.edu>
Date: Tue, May 31, 2011 at 11:23
To: Matt Berg <mlb...@gmail.com>
Cc: Mike McKay <mikey...@gmail.com>, Yaw Anokwa <yan...@gmail.com>, Eduardo Jezierski <ed...@instedd.org>, Nicolás di Tada <ndi...@instedd.org>, Jeff Wishnie <jwis...@thoughtworks.com>


The direction this conversation is taking is very positive.  Let me add some things we learned from ODK:

1. Keep tools as simple as possible - do not OVER-integrate - allow people to change system components easily.

2. Interfaces between tools need to be simple and use standard formats.

3. All operations must support running asynchronously.

4. Never lose data even if means sometimes replicating it and having to clean it later.

This leads to a few things being necessary:

1. Forms need to be user-readable - this is what makes XForms so frustrating - and need to be packagable units that can be send around and shared.

2. Same goes for databases (CouchDB provides a nice model for this - but maybe too much?).

3. As Mike says, simple forms have to have simple syntax and can be made incrementally more complex - there is a tension between declarative and procedural semantics (as many times as I want to use 'ask question 2 if x is true' I also want to say 'question 3 follows question 2 only if x is true' - the latter is much easier for non-CS users - but this is in the form and not necessarily in the UI for creating the form).

4. Some of the hardest things in forms is figuring out the right choices to provide in selects.  Getting these from a URL is not a general solution, but computing these or extracting them on the fly from a db makes a lot of forms simpler and easier to write.

5. An internal model of a form needs to be as simple as possible so that forms can get big (some forms we've encountered top 400 questions) - in fact, the only model should be instance values - everything else is best computed dynamically - not as in JavaRosa.

6. As Mike's also says, it should be possible to extend with new attributes so that innovation isn't stifled.  Of course, it is also important to not stray too far without agreement from the community.  JavaRosa's approach is probably too restirictive.

How about we get down to the business of proposing a form language to replace Xforms?  Important to consider a local DB in this - to hold forms, select options, longitudinal data, etc. - couchDB can then sync all or parts of these across devices.  Does this make sense?

Gaetano

----------
From: Matt Berg <mlb...@gmail.com>
Date: Tue, May 31, 2011 at 15:12
To: Gaetano Borriello <gae...@cs.washington.edu>
Cc: Mike McKay <mikey...@gmail.com>, Yaw Anokwa <yan...@gmail.com>, Eduardo Jezierski <ed...@instedd.org>, Nicolás di Tada <ndi...@instedd.org>, Jeff Wishnie <jwis...@thoughtworks.com>, Andrew Marder <andrew.e...@gmail.com>


Following up on a couple of points from Gaetano some of which we discussed on the last call.  

I think one route could look like this:

• Main form representation stored in json
• Advanced form functionality done through javascript based scripting (json is derived from javascript)
• For form authoring, we might want to support something like yaml which is very easy to read.  The symfony php project uses yaml quite a bit (http://www.symfony-project.org/book/1_2/10-Forms)
• I understand the concern of perhaps couchdb being to big in terms of a local db for storage.  The advantage of couch is each form type can be represented as a unique document type with the raw json submission stored in couch. It would be very easy to then access the data from earlier submissions and even use that to build some of your selects.  Another option would be something like sqlite.  For forms why not just store all the data in json hash/arrays which can be included in separate shared files as necessary.  This would allow the submissions to be much smaller.

Each form might be stored in a directory structure looking something like this. A lot of this will obviously need to be thought through through a lot more.

• / - forms directory.  All the forms in this directory can share the common js and data structures 
• /baseform.yml - yaml representation of the base form
• /baseform.json - base form in json (can have multiple forms share
• /js/ - js helper functions.  This includes some functions to help json objects in data.  We will probably need to write a core js api that all this can plug into
• functions.js - js helper functions. These can include lookup functions for some of the properties stored in json
• /data/countries.json  - data used in the forms. These can be included in the forms themselves but doing includes can make them more manageable to edit.
• /submissions/ - the baseform.json with submitted values.  Submissions can be submitted as a tar of the entire directory to be able to track back to the data options, etc
• Accessing submitted data locally - we can store submissions locally in either couchdb, sqlite to be accessibly by some of the JS helper / native functions we build into this.  If we store data in sqllite we look as using couchdb as an optional syncing transport layer.  It would be nice if we wrote an api where you could just use couchdb directly too (ie we support both).
• Note: we probably want to provide strong support for metadata / concept dictionaries

Also - adding Andrew in the mix who's been thinking about this type of stuff a lot of late.

Maybe we can start by just listing out the types of form functions we know we need to support in a google doc? Not sure if openrosa has anything that could be a good starting point?

Thanks,

Matt

----------
From: Mike McKay <mikey...@gmail.com>
Date: Thu, Jun 2, 2011 at 00:09
To: Matt Berg <mlb...@gmail.com>
Cc: Yaw Anokwa <yan...@gmail.com>, Eduardo Jezierski <ed...@instedd.org>, Nicolás di Tada <ndi...@instedd.org>, Jeff Wishnie <jwis...@thoughtworks.com>, Gaetano Borriello <gae...@cs.washington.edu>

zCore fits in as a reference point - I don't expect that we would use any code, except perhaps some javascript. I will not be advocating for any new java development! Practically speaking, I want to cast this work as fitting into the same general vision or logical progression of zCore, because that makes it easier for me to sell the idea to RTI. I do think that zCore can add some good real world usage around synchronization and decision support for more sophisticated data collection, but that would mostly come from my colleague Chris.

I think a JSON representation is probably reasonable. I can imagine that a deployed html form might have ids and names specific to a deployment or instance, while the json abstraction would be cleaner without such things. On the other hand, a close to the metal pure html form might save us from complexity creep (cough xforms cough). They can be transported/saved as easily as json (ajax), have a built in renderer (any browser) as well as a transport layer (browser assembled http POST). That said for saving form data into couchdb (which uses json natively) I tend to use a javascript function from stackoverflow to serialize my html form data into a json object and then post that json to couchdb: https://github.com/mikeymckay/egra/blob/release%2Falpha1/_attachments/models/jqueryMobilePage.js#L196-211

It's an interesting and important discussion...

Yeah I don't know. My tendency would be to tinker around and spike out some little prototypes of various ideas before deciding a way forward, but I am not sure.

Mike

----------
From: Jeff Wishnie <jwis...@thoughtworks.com>
Date: Thu, Jun 2, 2011 at 04:35
To: Gaetano Borriello <gae...@cs.washington.edu>
Cc: Matt Berg <mlb...@gmail.com>, Mike McKay <mikey...@gmail.com>, Yaw Anokwa <yan...@gmail.com>, Eduardo Jezierski <ed...@instedd.org>, Nicolás di Tada <ndi...@instedd.org>


I too like the direction this conversation is going. 

A few comments/thoughts sparked by Gaetano's (in line below)

best,

Jeff

There is a tension between 'human readable' and 'sufficient representational power'. And of course 'readable' is relative to the reader and dependent on the innate complexity of the Form. 

A 400 question form, in any representation is going to be hard to read.

That said, format decisions can make a HUGE difference. For example, I find MS 'document' style SOAP pretty readable—it's like JSON. Sun RPC style SOAP is impossible. Both are XML, but where Document SOAP tries to represent concrete 'objects', Sun RPC is a declarative description of procedural calls and just nasty.

I think a good design would be highly-readable or basic forms, with progressive complexity that could be ignored. For example, the basic structure and questions of form that contains complicated skip-logic should be understandable/readable even by someone who can't understand the embedded logic.

I whole heartedly agree with both points: Human Readable, Packagable/Transportable—the first point is going to be tough and need some good design.

 

I like Matt's suggestion on the 'packaging' that if things get complex and a 'Form' is best represented by multiple files, we can package them in a directory structure and probably zip the whole thing up—the 'JAR' approach.

Agreed. And there is also a correlation between the representational power of any logic syntax (declarative or procedural) and the complexity of the needed interpretor. 

One of the drawbacks of XForms (what I think may have doomed the format) is the complexity of interpreting them—which in part comes from various degrees of freedom of how things like constraints and logic are represented.

My gut is to aim to keep things simple and declarative (including basic logic constructs for skips/decisions), human readable/writable and easy for a machine to interpret.

I'm imagining a model not unlike the early DOM where the basic data structures are simple and declarative and complexity is added as an _external_ system manipulating the structure.

Could a URL (maybe following REST structures/semantics) be a good standard interface for passing parameters to a DB and retrieving generated choices? 

Can you explain more? This sounds right, but I'm not sure what you mean by the contrast with JavaRosa? Are you saying that you would remove the computed fields in form submissions?

Yes!

----------
From: Yaw Anokwa <yan...@gmail.com>
Date: Thu, Jun 2, 2011 at 21:37
To: Jeff Wishnie <jwis...@thoughtworks.com>
Cc: Gaetano Borriello <gae...@cs.washington.edu>, Matt Berg <mlb...@gmail.com>, Mike McKay <mikey...@gmail.com>, Eduardo Jezierski <ed...@instedd.org>, Nicolás di Tada <ndi...@instedd.org>


i think we should move the discussion to http://groups.google.com/group/openrosa-workgroup.

from there, we can maybe tease out two or three specific use cases that we could build a demo app or two around. seems like we agree on the core tech and the vision.

that sound reasonable to all?

[Anton de Winter]

Hi Yaw,

Thanks for the heads up.  It's always good to keep an eye on up and coming tech that would make our lives easier.  That said, I have a few questions (sorry if they've been answered before):

1)  What are the primary issues people are having with the XForms standard?

2)  Are there limitations in terms of what one can do with XForms that you're running into?

3)  What kind of form spec, would you say, are people leaning towards?  Pure programming language (e.g. everything written in JS), some other form standard, something else?

4)  Will this new tech be more complex than XForms or something more basic (but easier to use)?

5)  Will you be migrating all the ODK tools to this new spec by modifying the existing tools/backends or re-write from scratch?

Lots of questions, take your time in answering (though please answer if you can!)

Cheers,

Anton

[Eduardo Jezierski]

In general for basics you need a standard for

1- representing the data model (types, validations, cardinalities, vocabularies, completeness etc)

2- representing instances of the data model above

3- representing basic controller (e.g. flow, skip logic, etc) behavior;

4- representing basic view behavior (look & feel binding, validation, etc)

One of the issues of Xforms since v1 is that it's tried to do all #1, #2, #3, #4. As is common, the biggest strength is the biggest weakness. Composite standards have a tougher time evolving. #3 has great variability across devices and form factors too

There's great learnings from the rest of the industry that has been doing dynamic forms and flows for decades. XForms has taken a role as a down-level interop and sharing standard, (when used at all) but not as a 'core model' on which whole frameworks are based.

[Ian Lawrence]

Hi,

I think if we can specify exactly (like in a document) how the Open
Rosa specification differs from the XForms one that would be a good
step forward for us. We can then extract this diff and just apply it
to the new model if/when this comes around.
Great discussions b.t.w - love the couchapp idea and moving to dynamic
for creation on devices

Regards

--
http://ianlawrence.info

[Yaw Anokwa]

sorry about the delay, anton. responses are inline.

> 1) What are the primary issues people are having with the XForms standard?

i'll let others speak for themselves, but i'd like to be on the right technology trend, and pretty much no one uses xforms (except for orbeon). it's huge and hard to understand and it seems like we are always forcing our ideas to make everything fit that model. as people take different approaches, we have also quickly diverged from each other, making the whole notion of a standard useless.

> 2) Are there limitations in terms of what one can do with XForms that you're running into?

some of my complaints are tied to javarosa. i'd like to try gotos, instead of relevancy. i'd like to support gigantic forms thru lazy loading. i'd like to pull values from a database and filter on them. i'd like to support custom ui per form and per widget. and i'd like the form design to be more lightweight and human-understandable. there are more complaints in the thread i attached. are you guys at dimagi happy with xforms?

> 3) What kind of form spec, would you say, are people leaning towards? Pure programming language (e.g. everything written in JS), some other form standard, something else?

the consensus seems to be forming around the html5 form spec with additions through javascript for the base case. mike mckay's touchscreen toolkit has a good pass at this. jquery mobile also has some of this functionality as well. as far the complex forms, no idea. my gut says javascript. code that runs natively in a browser feels like the right approach.

> 4) Will this new tech be more complex than XForms or something more basic (but easier to use)?

i think we want to optimize for ease of use for developers and users. we have a good sense of the common case of forms and we just want to make that super easy. at the same time, we want to make complex things like eimci possible. i dunno if you've looked at that form, but it's a mess. there has to be a better way...

> 5) Will you be migrating all the ODK tools to this new spec by modifying the existing tools/backends or re-write from scratch?

this is more gaetano's call and we haven't talked about it in detail. personally, i'd like to see us switch to maintenance mode on the existing tools and start fresh with this new spec.

[Anton de Winter]

Hi Yaw,

 I think there are definitely some shortfalls to the XForm spec (more in terms of useability than real functionality) and agree with your major points. So this sounds pretty good, I'm excited to see what people come up with.   Where will discussion for this new spec be held?  We could do it on this mailing list, it seems like this type of discussion is in line with the goals of OpenRosa...

Cheers,

Anton

[Yaw Anokwa]

yup. i was thinking the list would be exactly the place. our team will have a rough pass at our approach in a few days...stay tuned :)

[Mike McKay]

I would like to revive this thread and hopefully get some very practical direction from the community. Over the next 2-3 months myself and Chris Kelley will be dedicating the majority of our time to working on many of the ideas that have been discussed for the future of ODK. A lot of our motivation has come out of being super impressed with how well the existing system works (Chris has recently used it in Zimbabwe, I am using it in the Philippines). I was particularly impressed with how easy it was to train others in how to create forms, deploy them and aggregate data. Clearly there is a lot of collective experience around, so any feedback is much appreciated.

There are two major directions that we want to go: web browser, couchdb.

Firstly, we would like to be able to collect data in the browser. Android is still our basic use case, but being browser focused means that we could use the same forms on a laptop or non-android smartphones & tablets. I've recently used http://JQueryMobile.com to easily create some reasonably nice web apps that are totally cross platform (iPad, IE, Android, firefox) and we would like to be able to use some kind of JQuery Mobile UI for rendering the forms.

Secondly, we would really like to use CouchDB as our backend. There are many reasons for this but chief among them is Couch's replication abilities. CouchDB runs natively on both Android and Iphone (as well as more traditional database servers like Linux/Mac/Windows) and the recommended deployment strategy is to simply deploy the database to your client and then use replication (whenever network connectivity is available) to keep all clients and central data servers in sync. Dimagi blogged about how they write their log messages to Couch, and then after a sync their main DB has all of the logged data from all of their deployments - sweet!

These two directions have a number of implications. Firstly, all code will be Javascript to enable all of the logic to run in the browser on the client (I may use my coffeescript crutch to help). Secondly all of the data will be stored as schemaless json documents (CouchDB is a NoSQL db). This includes the form definitions and the results. I'm not a huge fan of XForms and it sounds like neither are many of you. So I was going to start by implementing the form logic that we used in Malawi at Baobab Health as part of our Touchscreen Toolkit. It began as a bit of a hack, but has turned out to be a very pragmatic approach to handling form logic. You can see the documentation here: https://docs.google.com/View?docID=0AUuZsxdTjqLmZGN3Nmc1bXFfMmZwcnBqNQ&revision=_latest

So that's about it. I would love to hear any "violent reactions" as they say in the Philippines or pretty much any feedback. Also, please get in touch if you want to help code. We have been ramping up on the technology and are starting right away... github repo details to follow shortly!

Cheers,

Mike McKay

@mikeymckay

[Jonathan Jackson]

Hi Mike,

I am violently a fan of this.  I'll speak from Dimagi's (selfish) perspective, and advocate for building on the core technologies below.  Depending on how far along you are already, some of this may not be terribly feasible.  Anton and Cory - please interject if you disagree.

1)  Anything we build needs to be backward compatible with XForms, so if there is a new form player for Jquerymobile, we need to be able to go from xform -> new form definition -> mobile browser trivially.  

2)  CouchForms (https://github.com/dimagi/couchforms) is what we use as the back-end couch repository for forms.  This is currently the back-end for two production systems: both the system Cory blogged about in Zambia with local replication, as well as CommCareHQ in use in over 10 countries and 20 organizations.  
3)  Anton is just about to release an alpha of the pure javascript/Jquery (X)form authoring tool he has been working on.  We should evaluate the Javascript and JSON object representation of his code, make adjustments as necessary, and use that as our basis for shared form definitions.  One could then export just as JSON, convert to XForms, or whatever.

thoughts?  

[Jeff Wishnie]

I 'violently' love the idea as well.

The Mangrove/DW team (https://github.com/mangroveorg/mangrove) has been developing a couchDB based datastore for form content and a simplistic (currently) JSON form representation.

Currently the forms are 'played' only via an SMS channel, but the intent is to add Web form and Xform support (whereby the same internal JSON rep is presented in a web page or translated to an XForm for consumption by mobile clients).

The teams focus right now is on a couple of user-level applications, so what's implemented internally is pretty simple with just enough work done to support those apps.

That said, we'd love to 'upgrade' our JSON form definition to something more robust that you may be developing.

best,

jeff wishnie
director of social impact/principal consultant
thoughtworks
mail: jwis...@thoughtworks.com
mobile: +1 503 893 9193
skype/yahoo/aim: jwishnie

[Matt Adams]

Hi Mike,

I'd like to chime in here and say that I'm excited about the direction
that this and previous discussions are headed. Our company has a data
collection solution based on ODK Collect and CouchDB (groupcomplete.com)
that focuses on making data gathering a collaborative experience. Like
Collect, our app is native to Android but we are experimenting with web
frontends.

I cannot stress enough, as others have already done, how important
CouchDB is to the future of data collection. It's replication and
ability to operate on mobile platforms are key. As far as I know we
were the first company that based a product on Couch residing on the
mobile device (I know Dimagi has done similar work with Couch but I'm
not sure that it was on the smartphone per-se). We've been intimately
involved with helping to bring Couch to Android and at no point have we
regretted the decision to base our solution on it.

With respect to XForms, compatibility moving forwards would be of
interest to us and no doubt to others as well. I know that people have
expressed concerns about its limitations and trouble extending it as
time goes on but I am hesitant to throw the baby out with the bathwater.
JavaRosa XForms is an important part of a lot of implementations,
toolkits and processes (some publicly known and others internal to
various organizations and businesses that have adopted it) and I think
that it makes sense to consider the impact of transitioning away from a
technology that has fairly wide use. Some kind of one-way or
bi-directional bridge between XForms and next-gen would be useful. I'm
sure that the guys at Dimagi are thinking about this.

Radical would like to commit resources either in the form of active
development, testing and/or bugfixes to this and similar endeavours by
other teams. I think that it will help the community immensely to focus
efforts on one or two solutions much like JavaRosa/ODK have done. I
recall Yaw mentioning an approach that the ODK team is working on but I
don't know what the status of it is.

Cheers,

Matt
--
Matt Adams
Radical Dynamic
www.radicaldynamic.com

[Mike McKay]

3)  Anton is just about to release an alpha of the pure javascript/Jquery (X)form authoring tool he has been working on.  We should evaluate the Javascript and JSON object representation of his code, make adjustments as necessary, and use that as our basis for shared form definitions.  One could then export just as JSON, convert to XForms, or whatever.

We'd love to help out with this tool as some kind of a pure javascript form editor is on our todo list. Anton - is this in a state where you could use some help? Github it up!

...

I read everybody loud and clear on needing some kind of compatibility layer with xforms. I suggest that we take some kind of an iterative approach and gradually expand how much of xforms (as used by present day openrosa systems) is convertible. I certainly don't have all of this worked out and I expect plenty of dragons and snakeholes. I know that a lot of solutions we used with the touchscreen toolkit made use of scripted/imperative approaches vs the straight declarative approach of xforms. My hunch is that there is some sort of a usability sweetspot in there (I am aiming for the power/complexity/learning curve of an excel spreadsheet that uses formulas and pivot tables) , but would probably need to sacrifice elegance over pragmatism.

Mike

[Mike McKay]

 

2)  CouchForms (https://github.com/dimagi/couchforms) is what we use as the back-end couch repository for forms.  This is currently the back-end for two production systems: both the system Cory blogged about in Zambia with local replication, as well as CommCareHQ in use in over 10 countries and 20 organizations.  

CouchForms sounds great. It would be awesome if we could get have that functionality entirely encapsulated in a CouchDB instance and nothing else. Removing the python dependency would mean that creating a repository would be as simple as setting up a free account at http://www.iriscouch.com/ or https://cloudant.com/#!/solutions and then replicating the CouchApp into it. Crazy?

[Mike McKay]

On Fri, Jul 22, 2011 at 6:12 PM, Matt Adams <matt....@radicaldynamic.com> wrote:

 

 We've been intimately
involved with helping to bring Couch to Android and at no point have we
regretted the decision to base our solution on it.

I've been posting issues on github (https://github.com/couchbaselabs/couch-android-launcher/issues) and @daleharvey from couchbase has been really responsive. The latest version of CouchDB for android, CouchApp (here's a link to the Android market: https://market.android.com/details?id=com.arandomurl.couchapp) seems to have been built as a direct response to a bunch of issues that I created on Github. I have been using it for another project and it seems to work fine. I have not managed to build it from source yet, and my hope is that I won't have to. It comes with a couchapp called mobilefuton for doing database management on the phone and it seems pretty good but would certainly benefit from some more love (https://github.com/daleharvey/mobilefuton/issues)

 

Radical would like to commit resources either in the form of active
development, testing and/or bugfixes to this and similar endeavours by
other teams.  I think that it will help the community immensely to focus
efforts on one or two solutions much like JavaRosa/ODK have done.  I
recall Yaw mentioning an approach that the ODK team is working on but I
don't know what the status of it is.

It is awesome to have your experience in the community. Any wisdom that you could share from going through a similar process? Pitfalls to watch out for? I am particularly interested to hear about your experiences with running couch on android. How well does replication work - do ever do phone to phone replication? 

I am trying to get a good dev environment setup that will allow for easy collaboration. You guys will hear about it when it is good to go and then we can all dive in.

Mike

[Clint Tseng]

For what it's worth, ODK Build actually stores everything in an intermediate format that has a flexible schema and which isn't tied down to XForms. It's not until a user hits the "export" button that I downconvert everything to XForm XML. (I say downconvert more because it's so difficult to re-extract the author's original intent from an XForm rather than because it is less featureful).

I've been thinking about how best to do an ODK Webcollect for some time now, and one of the options that seemed most appealing to me was to make ODK Build just export its intermediate format to whatever Webcollect application I would build, and use that rather than try to decrypt XForms without the benefit of Javarosa.

I think that this approach probably makes the most sense; use a format that is much more easily understood as a form as we know it than XForm, and backconvert when necessary. Forwardconverting from XForm to whatever new thing we come up with seems like an errorprone process that's going to fail in ways that are cryptic to nontechnical users, and technical users' forms are likely going to be complex enough that they'll want to do the process themselves anyway.

Just my $0.02.

Best,

Clint Tseng

[Jonathan Jackson]

Clint - Absolutely.

And, you would just punt webcollect as an option for anything that is too hard and say you need to use something that has the JR engine in it (ODK, JavaRosa, Touch/Type Forms). 

What is the object model / flexible schema in?  

Cheers,

Jonathan

[Anton de Winter]

On Mon, Jul 25, 2011 at 3:42 PM, Mike McKay <mi...@vdomck.org> wrote:

3)  Anton is just about to release an alpha of the pure javascript/Jquery (X)form authoring tool he has been working on.  We should evaluate the Javascript and JSON object representation of his code, make adjustments as necessary, and use that as our basis for shared form definitions.  One could then export just as JSON, convert to XForms, or whatever.

We'd love to help out with this tool as some kind of a pure javascript form editor is on our todo list. Anton - is this in a state where you could use some help? Github it up!

Hey Mike,

Yes, it's on github! Check it out here.  I'm making a big code push today/tonight and tomorrow for an internal demo that's coming up.  The goal is to have the FD be able to do end to end form creation i.e. (parse an existing xform)->edit form->xform output. 

Quick tech description:

The aim of the project was to have an FD that could run offline (i.e. when you're somewhere where internet connectivity is poor to non-existent at best).  To that end, the FD has been written entirely in JavaScript.   The UI frontent makes use of CSS, HTML and Jquery libraries to do the heavy lifting.  The model code was written with the aim that there'll be as little as possible UI hardcoding that needs to be done while still keeping the code readable/maintainable.  This means, for example, that question properties are automatically enumerated and treated correctly in the UI. 

Another big win (I feel) is the internal validation framework.  This framework keeps track of what kind of Question/Data object is being dealt with and allows one to alert the user as soon as an issue becomes apparent.  For example, if it is known that a certain question type MUST have a label and the user has neglected to enter one, the validation framework will bubble up the error to the UI (allowing it to automatically highlight the given problem), which presumably will help to keep operator error at a minimum.

Once the requirements for Form Editing are met, there's a lot of room for interesting features like XLS importing (via the clipboard), visual analysis, question set templates, and more.  This should help ease the pain of the end-user form developer who is in the business of making more complex forms.

The internal structure is a little different from what you'd expect for an xform (we designed the architecture with possible big spec changes in mind).  I'd be happy to give a full rundown of how things are happening internally to anyone who is interested, but at the moment I should get back to coding ;)  At some point I'll be writing up some docs on how everything works (due to Bus Factor), so that should prove helpful too.

As for help, it is more than welcome :) The code is entirely open source. After tomorrow I'd be happy to work with others on new features or things that could be added.

Cheers,

Anton 

[Gaetano Borriello]

I am also very positive about this direction.

A couple of things.  

I think it would be best to keep the issues of a forms language and CouchDB separate.  I think you all are, but the two keep landing in the same messages and I would like to see them better separated.  CouchDB is a great solution but there are going to be many reason to still have a file-based form.  

As for the forms language, we need to differentiate between what a user will write and what will be the executable representation.  I am all for JSON and JS for the executable.  I do not think they are appropriate as the user language.  I see that as being largely defined by the authoring tools - graphic, spreadsheet, or otherwise - there should be many of these.  Having JS underneath means that a more experienced author could reach down and change things at a low level of details, but for most people, requiring JS capability will just be too much.  In the past, the details of the representation have crept in to the authoring tools (e.g., how skip logic and constraints are defined).  Just retargeting tools developed with Xforms in mind may not lead to the end result we want.

About the user language.  It needs to have a basic model of a form, widgets on the form, and how they are arranged for presentation (e.g., multiple prompts on a single screen/page, multi-column screens/pages, etc.).

That would be the basics.  Then, we can add constraints and navigation.  I would argue for moving away from the declarative semantics of Xforms and the concept of an "internal model" of the form and "relevance" concepts and move toward reactive procedural semantics of specifying what needs to be done when different events occur.  This should make complex forms easier to deal with and dynamic form prompt a real possibility.  

Another issue is that although I think it is important to support the form language in a browser on any platform, I would like to have abstractions that can still permit an app model that renders the form more appropriately on a specific platform.

Here is the general shape of my proposal:

First, there are two types of things: data and forms.  The data may come packaged with the form, but it should be considered separate.

Data can be a collection of database tables or a list of JSON key-value pairs that can be turned into a table when they are first read.

A form contains the following:

1. a chunk of data (to download from a server or embedded in the form - to transfer to a local database)

2. variables to hold data (static input vars, intermediate dynamic vars, output vars that will make up the data submission)

2. a series of screens each defining a set of prompts along with code for each screen to determine the next one

The general structure would be that the rendering application (as opposed to the one that navigates the form description) starts up and provides an object handle for the JS to use.  For example, as webview's addJavascriptInterface on Android.  The JS then calls that rendering object with methods for the different form elements.  If this were done entirely in JS, then the rendering code would be in the JS itself.  Anyway, the rendering code is pretty simple - one procedure for each widget type that takes a JSON description as an input parameter and returns a JSON description of the data gathered.  That makes it easy to add new widgets as long as the JS knows how to handle the data returned and the format in which to provide the data input - e.g., question text, hint text, choices for selects, along with multi-media representations.  

When rendering a prompt on the screen, the rendering object should also be able to call back into the javascript to a constraint checker whenever an input value changes.  When the user advances to the next screen, the rendering object returns the data to the JS, which can then analyze it to determine if the data is correct, put it away in variables or database, and determine what is to be the next screen (e.g., based on a switch statement on a variable value).

One JS script can call another.  That is how repeats should be handled - essentially a repeat iteration should be an instance of a subform.  A call stack can keep track of nesting and determine which basic linear instances are to be composed into a repeat collection.  Screen for dialogue with the user (to start or continue a repeat, prepare a submission, or just show some static information) are all just other widgets.

I see ODK developing this way.  A lean rendering engine that uses WebView to startup a JS form that calls into the ODK renderer.  On a browser, the ODK renderer would be replaced by a browser renderer that could be appended to the form.

Could we grab a sample form and start showing each other how it would look according the different proposals being voiced?  

Does what I'm describing make any sense at all? 

Gaetano

[Clint Tseng]

Jonathan:

It's all in JSON. The irony of ODK Build is that I wrote an internal form definer in order to build a tool that allows others to define forms, so I have a set of configuration that defines what each type of control takes as fields, which then turn into the data associated with those fields. Then, when everything gets exported from my internal representation to XForm, I just have a bunch of little codeblocks that each understand some particular field and modify the XML result as necessary.

Here's an example of what a form looks like to me: http://pastie.org/2270694

Best,

Clint

[Clint Tseng]

I strongly agree with separating the implementation details (couchdb) from whatever forms definition spec that we all agree on ("XForms next"). We shouldn't marry ourselves to any particular backend store.

I'm on the fence about JS vs JSON.

On the one hand, I think forms should be authored in a pure data format like JSON, and allowing arbitrary JS execution is going to be a source of both support headache and potential security vulnerabilities. In addition, allowing JS brings us back to what I most fundamentally dislike about XForms as someone who builds an XForm-authoring tool, which is that it's extremely difficult to reconstruct original intent from an XForm, just as it'll be extremely difficult to reverse-engineer intent out of arbitrary JS.

On the other hand, it does allow for a very simple (from our perspective building the system), flexible way for us to resolve the complexity of relevance as it stands today. In fact, in thinking about webcollect how I was going to deal with relevance was something resembling the following:

1. Construct a sandbox in JS that contains a clone of the instance data.

2. Parse the XForm relevance expression into an AST (rely heavily on JR's implementation as a reference for this).

3. Construct a JS statement from the AST.

4. Run the JS statement within the sandbox directly against the data. Record the result and act accordingly.

It seems like it's only a couple steps fewer than this to just allow arbitrary JS to execute in a sandbox against instance data.

Ideally, I'd like to see some sort of solution that compromises between these two worlds. As much as we can possibly make it should be in JSON/YML/XML/some other pure-data format, but we should offer something that is as flexible and strongly-typed as possible for relevance. I also don't want to end up with something resembling what we have for filter expressions at Socrata (my day job), which is strongly-typed, but overly verbose: http://dev.socrata.com/querying-datasets (scroll down a bit)

Best,

Clint

[Cory Zue]

Hey Guys, 

Sounds like we're mostly in agreement on everything here which is great. I am a strong proponent of a JSON form definition standard (hopefully shared between ODK's tools and Anton's Form Designer) with no dependencies on couch. 

Quick note to Mike below.

On Tue, Jul 26, 2011 at 1:19 AM, Mike McKay <mi...@vdomck.org> wrote:

2)  CouchForms (https://github.com/dimagi/couchforms) is what we use as the back-end couch repository for forms.  This is currently the back-end for two production systems: both the system Cory blogged about in Zambia with local replication, as well as CommCareHQ in use in over 10 countries and 20 organizations.  

CouchForms sounds great. It would be awesome if we could get have that functionality entirely encapsulated in a CouchDB instance and nothing else. Removing the python dependency would mean that creating a repository would be as simple as setting up a free account at http://www.iriscouch.com/ or https://cloudant.com/#!/solutions and then replicating the CouchApp into it. Crazy?

We actually built CouchForms' core as an update handler[1]  in couch with the explicit idea that it could be used without python. The python bit is UI and extra metadata on top of what the core provides and could be entirely stripped out with almost no detriment to the underlying form representations. The dynamic export functionality we've built (to re-convert couchforms back to a tabular structure) is more heavily dependant on python, though.

Cory

[1] http://wiki.apache.org/couchdb/Document_Update_Handlers

[Matt Adams]

On 27/07/11 05:47 AM, Cory Zue wrote:

> Sounds like we're mostly in agreement on everything here which is great.
> I am a strong proponent of a JSON form definition standard (hopefully
> shared between ODK's tools and Anton's Form Designer) with no
> dependencies on couch.

For what it's worth I agree with these ideas, although I haven't had
time to look at Anton's Form Designer yet.

[Mike McKay]

I might be misunderstanding or oversimplifying here, but I suggest that forms could be stored as json, with snippets of javascript embedded as appropriate.

For example, here's a form asking for the gender and whether or not the person is pregnant, but we only want the pregnancy question if the gender is female:

{
   "title": "gender_pregnancy_form",
   "controls": [
       {
           "name": "gender",
           "label": {
               "eng": "Gender",
               "fra": "Le Gender"
           },
           "required": true,
           "type": "inputText"
       },

       {
           "name": "pregnant",
           "label": {
               "eng": "Pregnant",
               "fra": "Le Pregnant"
           },
           "required": false,
           "type": "inputText",
           "condition": "$('#gender').val() == 'Female'"

       }]

The way the above might work would be that after any update to the form, all of the condition properties are executed. If false is returned then the control would be disabled. Totally untested example code:

$('form#gender_pregnancy_form').change(function(){

  _.each(json_form_definition["controls"], function(control){

    if eval(control["condition"]) != true;

      control.disable();

  });

});

(backbone.js has an elegant way of handling this sort of stuff - probably better than the above)

Expecting a novice user to write $('#gender').val() == 'Female' is not perfect, but a GUI could help or we could develop some kind of DSL to make the syntax cleaner: control("gender").equals("female") for example. I think similar things could be done for repeatable elements.

Strapping on my flame retardant suit... ;-)

Mike

On Mon, Jul 25, 2011 at 6:19 PM, Clint Tseng <cl...@dontexplain.com> wrote:

a

[Yaw Anokwa]

no flame retardant suit needed here. curious how will you imagine
doing the following...

link controls to data (local stuff like defaults and maybe a remote db)

logic for branching (basic jumps and e-imci style)

multiple questions per screen (android tablets are getting cheap)

do calculation across various nodes (if i enter a birthdate, i wanna
calculate age in months)

looping across a node n times (i enter that i have five kids, and i
want five child repeats)

totally custom controls (say a select one that looks like
http://opendatakit.org/wp-content/uploads/2011/07/plotType.png)

[Daniel Kayiwa]

It will also be very very useful that for each of the suggestions, a comparison is given as to how it makes life easier compared with the current xforms and its JR extensions.
This will ensure that we do not just simply end up with a new way of doing the same things as before without any added advantage when compared to the existing xforms code bases that we have already invested in!!!

--
If we keep uppermost in our minds the unkind and unjust acts of others, we shall find it impossible to love them as Christ has loved us; but if our thoughts dwell upon the wondrous love and pity of Christ for us, the same spirit will flow out to others.

[Mike McKay]

On Wed, Jul 27, 2011 at 7:43 PM, Yaw Anokwa <yan...@gmail.com> wrote:

no flame retardant suit needed here. curious how will you imagine
doing the following...

I can tell you how we did this sort of thing in Malawi with the touchscreen toolkit.

link controls to data (local stuff like defaults and maybe a remote db)

        {
            "name": "first_name",

            "required": true,
            "type": "inputText",

            "ajaxURL": "/getFirstNames?s="
        },

 

The ajaxURL property takes the content of the input field and appends it to the url, does a GET request and then creates a suggestion box below the control with the contents. This worked awesome for names because the URL performed a soundex search (the team implemented Bantu soundex!) sorted by frequency, so the most common matching names always showed up first. I find these sorts of features critical when developing UIs for non keyboards. 

I suggest implementing the same (probably a different property name). Ideally all of the requests will just go to CouchDB server, where you could implement similar logic as map/reduce queries.

logic for branching (basic jumps and e-imci style)

We got pretty far with using the condition property to enable/disable controls to get the appearance of simplistic branching. You can basically put all of your controls in one form and then write the conditions for each control to simulate branching. Obviously this gets unwieldy if you have anything particularly complicated (e-imci for example!).

The other approach would be to implement each branch as its own form and then link them together with the branching logic. We called this linking of forms a workflow. Since everything is served up out of a local database it will be fast and probably transparent to the user. JQuery mobile does some cool stuff with Ajax to dynamically load pages as you need them but it still feels like everything was pre-loaded. 

 

multiple questions per screen (android tablets are getting cheap)

I am curious as to what others think. As much as possible I would like to separate the form and form logic from the presentation layer. So in a perfect world, we would just have different stylesheets. When an element gets disabled that could mean that it gets skipped on a one question per screen scenario, or that it disappears or gets grayed out on a multiple questions per screen scenario. JQuery Mobile is a lot more than just css though, hence the imperfect world will probably require some css + javascript depending on the device or the type of layout that is selected. All of the required css/javascript can be in the couchdb, so you should just be able to select a presentation layer and the form should display without issue. 

 

do calculation across various nodes (if i enter a birthdate, i wanna
calculate age in months)

We have an onLoad attribute that allows you to run arbitrary javascript as soon as the control is entered. We've used this to do a lot of these sorts of calculations and GUI updates. For instance, after the height and weight are entered into two separate controls we have an empty control that we use to show the body mass index (BMI) which was necessary to determine malnutrition. Later we made this even more sophisticated by doing an AJAX request to get the BMI history, appending the new BMI and then displaying it in a chart with flot (really nice javascript based charting tool).

 

looping across a node n times (i enter that i have five kids, and i
want five child repeats)

Our original touchscreen toolkit solution for this was a hack. We would just put in like 20 controls and enable them one by one if the user indicated they needed more. We need something better. As long as we have a good standard for the name field of the html element, I don't think this will be hard to dynamically add form elements as needed. 

<input name='child.name[]'/>

<input name='child.name[]'/>

<input name='child.name[]'/>

Serializing the results into json would return an array that could be as long as we need it to be.

 

totally custom controls (say a select one that looks like
http://opendatakit.org/wp-content/uploads/2011/07/plotType.png)

Nice control! Clearly we are mostly just limited by what html & js can do - and we can make a picture based selector. The challenge is abstracting the concept into something that could be reusable by others (ideally with an Excel level of expertise).

 

Tough ones Yaw! I am curious to hear what others think. 

Mike

[Yaw Anokwa]

clint and i (mostly clint) are hammering out a document that lays out
some of the concerns with xforms. it'll keep us from re-inventing the
spec. thanks to daniel for suggesting this and clint for doing it.

some other questions i've had...

huge forms (and thus huge form defs) is something javarosa struggles
with. it'd be great if we had some discussion on how to load forms
sequentially or how to link forms together.

any concern that we are not separating out the data and the logic on
the json? everyone ok with all that stuff sitting in one place?

thoughts on having a standard list of prompt types and data types and
attributes? if not, how do we ensure that forms work across platforms?

is it reasonable to ask for hidden nodes -- variables that can store
temporary data that are either globally or locally available?

if we are going to build this new form processor in backbone.js (or
some other equally hip framework), how will all this get to native
apis on android? how do i scan a barcode? or get the imei number? or
pull from a sqlite db on the phone?

[Anton de Winter]

if we are going to build this new form processor in backbone.js (or
some other equally hip framework), how will all this get to native
apis on android? how do i scan a barcode? or get the imei number? or
pull from a sqlite db on the phone?

Looks like you'll be able to execute JS in your android app using WebViews, you content can be dynamically generated by your app, so you may be able to build some sort of a bridge between hardware/sqlitedb and the js-web-app thing.

--
Anton de Winter

Dimagi Inc

(617) 692 0786

529 Main St

Charlestown, MA

02129

[Ian Lawrence]

Hi,

> Looks like you'll be able to execute JS in your android app using WebViews,
> you content can be dynamically generated by your app, so you may be able to
> build some sort of a bridge between hardware/sqlitedb and the js-web-app
> thing.

You can use WebViews together with phonegap which makes it possible to
call Java methods from Javascript on Android

For example I call the android camera in this example jquery mobile
and sqlite app (about finding and rating cachaça :) )

http://ianlawrence.info/downloads/bafo-de-tigre/index.html

Regards

--
http://ianlawrence.info

[Drew Roos]

On Wed, Aug 3, 2011 at 12:02 PM, Yaw Anokwa <yan...@gmail.com> wrote:

any concern that we are not separating out the data and the logic on
the json? everyone ok with all that stuff sitting in one place?

i'm not ok with the relevancy model of attaching conditions to individual questions that turns them on or off (if that's what you're referring to). my experience with xforms has shown me that model is COMPLETELY inadequate for any moderately branchy form. in this model it is also very hard to do 'if X go to end of form'. it has a place for simple forms (and simpler sub-parts of complex forms), and i don't think we should do away with it. but any new spec we come up with that hopes to improve upon xforms must have an innate concept of branching. (and probably other control flow constructs as well, such as 'loop until condition is satisfied').

[Yaw Anokwa]

is there an existing spec or framework that does this reasonably well?
or a set of control flow constructs you want to see?

[Jonathan Jackson]

>> any concern that we are not separating out the data and the logic on
>> the json? everyone ok with all that stuff sitting in one place?
>>
>
> i'm not ok with the relevancy model of attaching conditions to individual
> questions that turns them on or off (if that's what you're referring to). my
> experience with xforms has shown me that model is COMPLETELY inadequate for
> any moderately branchy form. in this model it is also very hard to do 'if X
> go to end of form'. it has a place for simple forms (and simpler sub-parts
> of complex forms), and i don't think we should do away with it. but any new
> spec we come up with that hopes to improve upon xforms must have an innate
> concept of branching. (and probably other control flow constructs as well,
> such as 'loop until condition is satisfied').

Would this still be backward compatible or would this cause a fundamental break between the capabilities of the new language and xforms?  

[Drew Roos]

we already do complex branching in xforms, but it's a nightmare

[Clint Tseng]

It would be if anything a downconvert, so there's no reason we couldn't apply the same in reverse. The plan for branching in ODK Build essentially involves the same abstraction on top of the primitives XForms provides.

[Drew Roos]

not of the top of my head, but i haven't researched very thoroughly.

i did wire up touchforms to an alternative engine where forms are written as a javascript function and questions to ask are emitted via 'yield'. the control flow path taken through the function is the path you take from the form. in this way, the "form" is not in a declarative format but actually behaves more like a DSL.

concept example:

ask A

ask B

if B > 3 {

  ask C

  ask D

} else {

  ask E

}

the approach taken with this alternate engine is not without its own problems, but i have found it quite pleasant for writing complex interactions. it's something to think about.

[Yaw Anokwa]

spent some time chewing this over with the odk gang. consensus seems
to be that we are dangerously close to re-inventing the xforms spec,
but with json instead of xml and with javascript instead of java.

we are also likely conflating weaknesses in javarosa-core with
weaknesses in xforms (which can do some pretty cool stuff...even in a
browser. see http://www.orbeon.com/forms/compare). can we be explicit
on this? have people who've read most of the xforms spec really think
we can do better? if xforms is terrible, why is this new thing json
thing better?

if i had to make a decision today, i'd say that xforms is likely fine,
javarosa-core is not, and we'd be better served taking a deeper look
at xforms and maybe building a better core. if the problem isn't
xforms, but more of the weakness of a declarative form description,
then we need to reframe much of this discussion to the procedural (or
hybrid) approach that gaetano and drew are suggesting...

[Gaetano Borriello]

I want to underscore the difference in the style of the description.
We can agree later about what the syntax should be. I have no problem
with JSON or XML or anything else - as long as it easy to read - yes,
by a human. At some level we will always need a human-readable form
specification.

My problems with XForms are the following:

1. Scattering of question properties into different parts of the form
(instance, bind, and body) making it difficult to cut/paste sections
and easily reason about form logic flow.

2. Difficulty in extending with new functionality (read arbitrary
code, when needed by experienced users).

3. Declarative vs. procedural form logic flow which I believe to be
more difficult for the average person to grasp (Drew's example is much
more readable, for example).

There are even more problems with Javarosa-core, but those could be
surmounted more easily than the XForm issues.

I'd like to propose something much more object-oriented where forms
and questions are objects.

A form has a collection of variables and a collection of questions.
The questions are arranged into pages, pages have a sequential default order.
A page is another object, and it can determine what the next page is
(basing its determination on the value of variables).
A question has a type, constraints, and calculations.
When a form is loaded, it is asked for its first page to display.
Each question on the page is asked its type (default behavior) and
asked to display itself - question text can be determined on the fly
although default behavior will be to have a string specified for this,
similarly, for any default values.
Whenever the user changes a value in an entry on a page, that
question's calculation method is invoked (default is null) so that any
variables can be updated. A page's display is refreshed if any
question on that page "watches" a variable that is updated.
Variables are updated through a method invocation as well and
questions/pages can register to "watch" a variable.
Whenever the user attempts to advance to the next page, all questions'
constraints are checked (default is a null constraint that is always
satisfied) and the question returns a warning to display if the
constraint is violated preventing moving to the next page.
Also, when the user attempts to advance to the next page, a method is
invoked on the form to determine the next page. The default method
advances through pages sequentially while a specialized method may
include a flow (such as Drew's).
There are no "repeats", rather a repeated series of questions is
simply another form. Thus, a form's next page, may be another form.
A stack is maintained to by the form renderer to keep track of which
form/subform it is currently in.

Pages are not just for question collections. Some pages may be used
to present data to the user, e.g., display a graph of data, summarize
other data, etc. Also, a submission is a type of page where variable
values are collected and packaged for shipping - over a data channel,
to storage/db, or to SMS.

Most question renderings will be built-in, others will be used to
invoke external applications for special data gathering (e.g., bar
code, photo, sensor data, etc.).

The idea is that, most of the time, a form and its pages and questions
will use default methods. However, these methods can be overridden
with specialized arbitrary code for easy extension and customization.

Here is a simple, three page form with a question depending on the
answer to another:

Form:
variables va, vb, vc, vd
pages pa, pb, pc
questions qa, qb, qc, qd, qe

Pages:
pa: qa
pa/next: if va < 10 pb else pb2
pb: qb, qc, qd
pb2: qc, qd
pc: qe

Questions:
qa: number
qa/text: enter a number
qa/constraint: if number <= 5 warn "enter a number > 5"
qa/variable: va

qb: text
qb/text: enter a string
qb/variable: vb

qc: photo
qc/text: take a photo
qc/variable: vc

qd: location
qd/text: where did you take this picture?
qd/variable: vd

qe: text
qe/text: "your previous answer was " + vb + ", please amend it"
qe/default: vb
qe/variable: vb

I can't vouch for the efficiency of doing it this way, but it seems
like a well-structured approach (at least to me, but isn't that always
the case with one's own ideas?). Of course, the syntax could be your
favorite - JSON, XML, etc.

Thoughts?

Gaetano