Where to store instanceID to avoid data duplication

[chan...@cell-life.org]

Hi All,

My name is Chantell, I work at cell-life. I am currently working on the following issue((see link) in ODK collect: https://trac.openxdata.org/ticket/1019.

Here  is a link to a previous discussion on the issue : https://groups.google.com/forum/#!topic/opendatakit-developers/f4y9jm4QE4Y.

We managed to get the instance ID back from the server, but we are not sure where to store it. We are considering doing the folowing:

1. Store the instance ID with the form ID  in a text file on the server. This will cause problems with backwards compatibility and updating the server.

2. Fix the problem in ODK collect which will be fairly easy, but this means that a customized version of odk collect should always be used. 

3. Ragu suggested that we don't store the instance ID in the form data xml.

What do you think? Does anyone else have any other suggestions? 

Thank you.

Kind Regards

Chantell

[Ronald Kayondo]

Hi Chantell,

Thanx for working on this.

Option one seems like the most non intrusive. I do not think the backward incompatibility issues will be such a big problem if the text file is managed by the protocol and does not change often. But I do agree it will make managing oxd a little harder. e.g if your are shifting to another server you have to transfer the text file + the database + warfile.

 And if you have multiple oxd instances you have to figure which instance points to which file.

Options three: did you mean store the id in the data xml? If yes... I think this would be ok. The Only problem would be searching for a specific id to make sure it does not exist in the db. MySql will have to go through alot of XML(which might also have base64 videos).

However on another note, would writing a liquibase change-set to add a column to the form_data table be a bad idea?

Oxd generally has a problem of duplicates not only for forms from odk but also from mforms, the webclient and E-Collect especially in areas with poor network connectivity where connections abort many times. This would be a the first step towards addressing this duplication problem. All other client protocols/bridges that wish to implement this feature(de-duplication) would have a uniform way of accessing these instanceIds.

Regards,

Ronald.

--
You received this message because you are subscribed to the Google Groups "openXdata Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openxdata-de...@googlegroups.com.
To post to this group, send email to openxd...@googlegroups.com.
Visit this group at http://groups.google.com/group/openxdata-dev.
For more options, visit https://groups.google.com/d/optout.

--
Ronald. K

[Chantell Witbooi]

Hi Ronald,

Thank you for the feedback.

Kind Regards

[Jørn Klungsøyr]

Hi,

 

I support Ronalds suggestion on adding a new column to store unique client side generated id’s and dedupe on that.

That approach would also allow updating mForms and other clients to handle it similarly.

 

Requirement would have to be that the client can generate a unique random id like UUID/GUID – ideally I would prefer a server generated UUID on submission and send that back to the client – but nothing is perfect….

 

Jørn

 

__________________________________________________________
Jørn Klungsøyr – www.zegeba.com - mobile: +4791365731, chat: jornklung
--------------------

[Dagmar Timler]

Thanks everyone for your responses.

I'd also prefer a server generated UUID on submission, but without changing the ODK client, that isn't possible.

The second best solution would be to save it somewhere obvious in the database (like a new column). There are unfortunately a major snag with this approach. The first is that making a server change will result in a new protocol-api and server dependencies. This will result in a solution that is not backwards compatible. From Cell-Life's perspective making a change like this will make roll out difficult. The problem for us is that the latest version of the form designer doesn't allow question bindings with uppercase letters - since we have existing forms, this is an extremely blocking change for us. Also we haven't had an official release for ages.....

I agree that in the XML is not possible (and for one more reason - the RDMS exporter throws an Exception because there's no meta/instanceID question in the form definition).

This leaves me with the text file on the server. I don't entirely like this approach because (as Ronald pointed out) it makes moving the server a little more difficult, but I think we've already gone down this road with our text files for protomap etc.

So.... thoughts on that?

[Jørn Klungsøyr]

Hi Dagmar,

 

I’m worried about doing a filebased “database” of value pairs (oxd ID <--> instanceID) – or did I misunderstand something?

If we can save it in a file, we should be able to save it properly in the database together with the record?

 

As long as we only add new elements to the protocol-api, the «old» plugins should still work, right?

 

A mix of server/client generated UUID could be done with the OXD-API creating a UUID if not supplied by the client through the protocol-API. Thereby retaining backwards compatibility.

 

mForms / mForms proto could later be improved to also support client side generated instanceID (without breaking existing clients).

 

I’ve not looked at the details of this for quite some time, so I might be completely off….

 

The purcforms nightmare needs to be tackled, I’ll followup with a separate thread on that beast.

 

And yes, would be an idea to do a new public release…..

 

Best

Jørn

 

 

 

 

__________________________________________________________
Jørn Klungsøyr – www.zegeba.com - mobile: +4791365731, chat: jornklung
--------------------

 

[Dagmar Timler]

Hi Jørn

You are right about just adding elements (not deleting) will ensure backwards compatibility. What I meant was that forwards compatibility will be affected. Meaning with our old oXd server we will not be able to use new protocol plugins due to plugin API and server modifications. Because we are unable to upgrade our servers (the reason is mentioned in our previous email), this is unfortunately a real concern.

I think your idea of using the client generated UID if it's there and generating a new UID if it's not, is the correct approach.

However, Chantell is going to continue with a file approach as an interim solution (due to our issues with being unable to upgrade our existing servers). This will just be a text file mapping form_data ids to instanceIDs for the ODK collect data. It will be located close to the other text files used for configuring the ODK collect proto (i.e. in the protocol-jars folder).

Once we solve the problem with the form bindings, we will consider completing the implementation as you suggested and then submitting a patch.

If you would like us to submit a patch with the file solution, we are also happy to do that.

Thanks

Dagmar

[Jørn Klungsøyr]

Hi Dagmar,

I missed the part on you not being in a position to upgrade your servers.

I don’t have other suggestions.

Please do upload the changes you make, maybe they can be used as a basis for a general API/protocol improvement.