Upload problem

[Alex Ewing]

We are trying to test sana with openMRS.

Sana 1.1.2

openMRS 1.9.3 Build f535e9

We have validated the connection between the MDS and the openMRS server.

We have also successfully retrieved patient information from the openMRS server. However, we have not been able to upload or create new patient records on the server from SANA. We have checked that the concepts of the procedure are defined correctly, but it still doesn't work.

Here is the message we get from the Server Log:

WARN - ExtensionPointTag.doStartTag(136) |2013-05-28 11:08:29,699| Extensions at this point (org.openmrs.gutter.tools) are required to be of class org.openmrs.module.web.extension.LinkExt or a subclass. class org.moca.extension.GutterExtension is not. Can anyone assist us?

[Eric Winkler]

To use the Sana module with the 1.9.3 version of OpenMRS, you will need to use the 2.0 snapshot module and mds 1.2 beta. Both of those are available on our git repo's downloads page, http://code.google.com/p/sana/downloads/list. There are additional instructions for upgrading in the mds archive. Both should be reasonably stable at this point and work with the version of the client you are using. Otherwise, you would need to downgrade the OpenMRS version you have installed to the one available in the packages on our repository, http://demo.sana.csail.mit.edu/packages/ubuntu/.

Best,

Eric

Sana Development

[Alex Ewing]

So I elected to downgrade to Sana 1.6.1. I am no longer receiving the error message I posted earlier. However, the uploads are still failing.

From the Server log:

DEBUG - Extension.initialize(71) |2013-06-04 15:11:06,613| Initializing extension for point: org.openmrs.admin.list

The test procedure involves first getting the patient's information, then filling in some basic demographic information and a "test word" which can be anything, then uploading the data to the server.

The MDS log looks like this:

06/04/2013 01:38:25	/mds/json/procedure/submit/	0.0715s
06/04/2013 01:38:25	/mds/json/procedure/submit/	0.0281s
06/04/2013 01:38:25	/mds/json/procedure/submit/	0.0455s
06/04/2013 01:38:25	/mds/json/procedure/submit/	0.0433s
06/04/2013 01:38:25	/mds/json/procedure/submit/	0.0289s
06/04/2013 01:38:15	/mds/json/patient/1234/	0.1007s

The bottom one (06/04/2013 01:38:15/mds/json/patient/1234/0.1007s) is the log for retrieving the patient record, which was successful. Clicking on any of the others yields this:

+0.000s json.py:205  Received saved procedure submission.
+0.002s api.py:88  Registering saved procedure SPcfEkOqaILEANuAzDyxKp
+0.002s api.py:90  sp_guid -> SPcfEkOqaILEANuAzDyxKp, client_id -> {"patientId":{"answer":"1234","type":"PATIENT_ID","concept":"PATIENT ID","question":"???????????:"},"patientLastName":{"answer":"Test","type":"ENTRY","concept":"LAST NAME","question":"????????:"},"1":{"answer":"q","type":"ENTRY","concept":"GEORGE TEST","question":"ENTER A WORD"},"patientBirthdateYear":{"answer":"1979","type":"ENTRY","concept":"BIRTHDATE YEAR","question":"??:"},"patientBirthdateMonth":{"answer":"??","type":"SELECT","concept":"BIRTHDATE MONTH","question":"???????????:"},"patientGender":{"answer":"?","type":"RADIO","concept":"GENDER","question":"?????????:"},"procedureTitle":{"answer":"Test Upload","id":"procedureTitle","type":"HIDDEN"},"patientEnrolled":{"answer":"Yes","id":"patientEnrolled","type":"RADIO","question":"Does the patient already have an ID card?"},"patientBirthdateDay":{"answer":"01","type":"SELECT","concept":"BIRTHDATE DAY","question":"??:"},"patientFirstName":{"answer":"Test","type":"ENTRY","concept":"FIRST NAME","question":"?????????:"}}, responses -> +8613241419670
+0.005s util.py:44  (0.001) SELECT `mrs_client`.`id`, `mrs_client`.`name`, `mrs_client`.`last_seen` FROM `mrs_client` WHERE `mrs_client`.`name` = +8613241419670 ; args=(u'+8613241419670',)
+0.008s util.py:44  (0.000) SELECT (1) AS `a` FROM `mrs_client` WHERE `mrs_client`.`id` = 1 LIMIT 1; args=(1,)
+0.010s util.py:44  (0.000) UPDATE `mrs_client` SET `name` = +8613241419670, `last_seen` = 2013-06-04 01:38:25 WHERE `mrs_client`.`id` = 1 ; args=(u'+8613241419670', u'2013-06-04 01:38:25', 1)
+0.015s util.py:44  (0.001) SELECT `mrs_savedprocedure`.`id`, `mrs_savedprocedure`.`guid`, `mrs_savedprocedure`.`procedure_guid`, `mrs_savedprocedure`.`client_id`, `mrs_savedprocedure`.`responses`, `mrs_savedprocedure`.`upload_username`, `mrs_savedprocedure`.`upload_password`, `mrs_savedprocedure`.`uploaded`, `mrs_savedprocedure`.`created`, `mrs_savedprocedure`.`modified`, `mrs_savedprocedure`.`encounter` FROM `mrs_savedprocedure` WHERE (`mrs_savedprocedure`.`guid` = SPcfEkOqaILEANuAzDyxKp AND `mrs_savedprocedure`.`client_id` = 1 ); args=(u'SPcfEkOqaILEANuAzDyxKp', 1)
+0.016s api.py:100  SavedProcedure -> SPcfEkOqaILEANuAzDyxKp exists. unstable connection?
+0.018s util.py:44  (0.000) SELECT (1) AS `a` FROM `mrs_savedprocedure` WHERE `mrs_savedprocedure`.`id` = 12 LIMIT 1; args=(12,)
+0.021s util.py:44  (0.001) UPDATE `mrs_savedprocedure` SET `guid` = SPcfEkOqaILEANuAzDyxKp, `procedure_guid` = 0, `client_id` = 1, `responses` = {"patientId":{"answer":"1234","type":"PATIENT_ID","concept":"PATIENT ID","question":"???????????:"},"patientLastName":{"answer":"Test","type":"ENTRY","concept":"LAST NAME","question":"????????:"},"1":{"answer":"q","type":"ENTRY","concept":"GEORGE TEST","question":"ENTER A WORD"},"patientBirthdateYear":{"answer":"1979","type":"ENTRY","concept":"BIRTHDATE YEAR","question":"??:"},"patientBirthdateMonth":{"answer":"??","type":"SELECT","concept":"BIRTHDATE MONTH","question":"???????????:"},"patientGender":{"answer":"?","type":"RADIO","concept":"GENDER","question":"?????????:"},"procedureTitle":{"answer":"Test Upload","id":"procedureTitle","type":"HIDDEN"},"patientEnrolled":{"answer":"Yes","id":"patientEnrolled","type":"RADIO","question":"Does the patient already have an ID card?"},"patientBirthdateDay":{"answer":"01","type":"SELECT","concept":"BIRTHDATE DAY","question":"??:"},"patientFirstName":{"answer":"Test","type":"ENTRY","concept":"FIRST NAME","question":"?????????:"}}, `upload_username` = admin, `upload_password` = Admin123, `uploaded` = False, `created` = 2013-06-04 01:38:25, `modified` = 2013-06-04 01:38:25, `encounter` = -1 WHERE `mrs_savedprocedure`.`id` = 12 ; args=(u'SPcfEkOqaILEANuAzDyxKp', u'0', 1, u'{"patientId":{"answer":"1234","type":"PATIENT_ID","concept":"PATIENT ID","question":"???????????:"},"patientLastName":{"answer":"Test","type":"ENTRY","concept":"LAST NAME","question":"????????:"},"1":{"answer":"q","type":"ENTRY","concept":"GEORGE TEST","question":"ENTER A WORD"},"patientBirthdateYear":{"answer":"1979","type":"ENTRY","concept":"BIRTHDATE YEAR","question":"??:"},"patientBirthdateMonth":{"answer":"??","type":"SELECT","concept":"BIRTHDATE MONTH","question":"???????????:"},"patientGender":{"answer":"?","type":"RADIO","concept":"GENDER","question":"?????????:"},"procedureTitle":{"answer":"Test Upload","id":"procedureTitle","type":"HIDDEN"},"patientEnrolled":{"answer":"Yes","id":"patientEnrolled","type":"RADIO","question":"Does the patient already have an ID card?"},"patientBirthdateDay":{"answer":"01","type":"SELECT","concept":"BIRTHDATE DAY","question":"??:"},"patientFirstName":{"answer":"Test","type":"ENTRY","concept":"FIRST NAME","question":"?????????:"}}', u'admin', u'Admin123', False, u'2013-06-04 01:38:25', u'2013-06-04 01:38:25', u'-1', 12)
+0.022s api.py:112  patientLastName : {u'answer': u'Test', u'question': u'????????:', u'concept': u'LAST NAME', u'type': u'ENTRY'}
+0.022s api.py:112  patientEnrolled : {u'answer': u'Yes', u'question': u'Does the patient already have an ID card?', u'type': u'RADIO', u'id': u'patientEnrolled'}
+0.022s api.py:112  patientFirstName : {u'answer': u'Test', u'question': u'?????????:', u'concept': u'FIRST NAME', u'type': u'ENTRY'}
+0.023s api.py:112  patientId : {u'answer': u'1234', u'question': u'???????????:', u'concept': u'PATIENT ID', u'type': u'PATIENT_ID'}
+0.023s api.py:112  procedureTitle : {u'answer': u'Test Upload', u'type': u'HIDDEN', u'id': u'procedureTitle'}
+0.023s api.py:112  1 : {u'answer': u'q', u'question': u'ENTER A WORD', u'concept': u'GEORGE TEST', u'type': u'ENTRY'}
+0.024s api.py:112  patientBirthdateMonth : {u'answer': u'??', u'question': u'???????????:', u'concept': u'BIRTHDATE MONTH', u'type': u'SELECT'}
+0.024s api.py:112  patientGender : {u'answer': u'?', u'question': u'?????????:', u'concept': u'GENDER', u'type': u'RADIO'}
+0.024s api.py:112  patientBirthdateYear : {u'answer': u'1979', u'question': u'??:', u'concept': u'BIRTHDATE YEAR', u'type': u'ENTRY'}
+0.024s api.py:112  patientBirthdateDay : {u'answer': u'01', u'question': u'??:', u'concept': u'BIRTHDATE DAY', u'type': u'SELECT'}
+0.025s api.py:151  Should I upload SPcfEkOqaILEANuAzDyxKp to the MRS?
+0.027s util.py:44  (0.000) SELECT `mrs_binaryresource`.`id`, `mrs_binaryresource`.`procedure_id`, `mrs_binaryresource`.`element_id`, `mrs_binaryresource`.`guid`, `mrs_binaryresource`.`content_type`, `mrs_binaryresource`.`data`, `mrs_binaryresource`.`upload_progress`, `mrs_binaryresource`.`total_size`, `mrs_binaryresource`.`uploaded`, `mrs_binaryresource`.`created`, `mrs_binaryresource`.`modified`, `mrs_binaryresource`.`convert_before_upload`, `mrs_binaryresource`.`conversion_complete` FROM `mrs_binaryresource` WHERE `mrs_binaryresource`.`procedure_id` = 12 ; args=(12,)
+0.027s api.py:172  Encounter: SPcfEkOqaILEANuAzDyxKp has [] READY and [] WAITING
+0.027s api.py:186  Uploading Encounter -> SPcfEkOqaILEANuAzDyxKp, Binaries to upload = []
+0.028s json.py:241   File "\/opt\/sana\/mrs\/json.py", line 222, in procedure_submit password) 
+0.028s json.py:241   File "\/opt\/sana\/mrs\/api.py", line 139, in register_saved_procedure return maybe_upload_procedure(sp) 
+0.028s json.py:241   File "\/opt\/sana\/mrs\/api.py", line 258, in maybe_upload_procedure patient_birthdate = '%s\/%s\/%s' % (months_dict[patient_month], patient_day, 

These are the modules I'm using.

Sana Module     1.1.2     

Logic Module     0.4         

Serialization Xstream      0.2.2

Data Entry Statistics        1.2

Media Viewer Module  1.1         

HTML Form Entry             1.6.2

Reporting Compatibility                1.5.0.3

REST Module     1.4.2

FormEntry           4.4.1

I cannot tell where the error is coming from here, as there's nothing in these logs that clearly shows up as an error. Any thoughts?

[Alex Ewing]

Also, the question marks in the "questions" are Chinese characters. As this may be an issue we are going to translate those questions into English first to see if it works, but it will eventually have to be in Chinese.

[Alex Ewing]

We have determined that the server set up is in fact functioning correctly now that we've downgraded, and the problem lies on the software side. We're working that out now, thanks for the assistance!

[Eric Winkler]

I am glad to hear you made some headway but there definitely is an encoding issue going on. Referring back to something in the logs:

patientBirthdateMonth":{"answer":"??","type":"SELECT","concept":"BIRTHDATE MONTH","question":"???????????:"}

 

The answer is used to get a month value based on a dict in the mds code. It is something that needs to be worked on because it is English language centric. From past experience working with a Portuguese translation, we ended up having to add the portuguese month string to two-digit month to the dictionary which is located in the api.py file of mds. As I mentioned, it is a known problem which will be corrected in the next version. In the meantime, you may need to make one or more modifications similar to what was mentioned above.

Bet,

Eric 

[Alex Ewing]

Thank you for the response, that does seem to be the problem. Also, we will need to use Chinese characters in the final version (not just for the month, but for every concept), but when we try to upload Chinese characters, the server does not seem to recognize them, hence all the question marks. I was under the impression that Chinese would be supported using Unicode. Do you have any idea why this is happening?

[Alex Ewing]

We were thinking that perhaps the current Sana program doesn't allow Chinese characters to be uploaded. Where in the Sana code is the language dealing with the upload process? The code is pretty long!

[Eric Winkler]

The question marks appearing in the log output are purely related to how the log strings are formatted and not necessarily indicative that it won't work with the character set you are using or other upload related issues. While I don't have much direct experience working with the Chinese character set, I can tell you from anecdotal evidence I have received from others that it is possible to do so. Furthermore, you are correct that the underlying platform is Unicode based so there isn't any reason that it shouldn't work.

As far as where to start, the first thing I would suggest would be to look at the OpenMRS settings in the admin and make sure that you have the locale you need included in the list of allowed locales as well as in the list of allowed locales for the user account that is uploading. You can also set a default locale in the admin if it is something you want to apply for all user accounts. On the Sana end for OpenMRS, the data gets uploaded through the UploadServlet so that may be the place in the Sana module you need to look if there is something that needs to be corrected. From my personal experience working with Portuguese and Spanish-two locales that require non ASCII characters, the adjustment to the months dictionary in the mds code previously mentioned and the OpenMRS admin settings seemed to do the trick.

Best,

Eric  

[Alex Ewing]

Hi Eric, 

Thanks for your response. We have now confirmed that we can perform an upload using Chinese characters as long as we use the English months. However, when uploading Chinese characters, we are still having the problem that question marks show up in the OpenMRS. We have included the english and Chinese locales in the allowed locales. We have also tried making the default locale Chinese. However, it still does not seem to be working. Not only do question marks show up in OpenMRS, but if one goes into OpenMRS and creates a new patient record with a Chinese name using the admin account on the computer, the mobile device will be unable to retrieve this record. If the patient record has an English name, the mobile device can find the record.

You mentioned that you don't have any experience yourself using Chinese characters and Sana/OpenMRS. I'm wondering if you know of anyone who has, who may have run into similar issues. Perhaps we could try to contact them.

Thanks,

Alex

[Eric Winkler]

This may be a couple of separate but related issues.

1) OpenMRS interface. If the ? marks are occurring within the OpenMRS web page content it is likely that OpenMRS or one of the modules needs to be rebuilt after translating a messages.properties file. If the question marks show up in the patient data it may be a missing character set on the server or some component on the server(This page may help). I have seen similar issues to what you described on a server where the primary locale was Portuguese. We were able to resolve the issues by adding the missing locale on the server and installing a translated OpenMRS version.

2) Patient Fetching. It sounds like something is using a string as opposed to unicode function. There are a couple of places where that might happen on either the client, mds, or the OpenMRS REST module. I will need to look into it a bit further and get back to you.

Best,

Eric

[Alex Ewing]

Hi Eric, I think the first issue has been resolved. We can create a patient record on the server and it will display the Chinese in OpenMRS without a problem. It does seem to me that it must be the second, that for some reason, a string is being sent. Still trying to figure this out. We are trying to deploy this thing in a couple of weeks, so we are worried!

[Eric Winkler]

Hi Alex,

Assuming it is the patient fetching, here are a couple of thoughts:

1. OpenMRS. I don't think this is where the problem is. If you are interested, the relevant code that returns the REST requests is in the src/org/openmrs/module/restmodule/web/ directory (The link goes to the online code viewer). 

2. MDS. 

a. Source code. There may be a couple of lines in the code that could be related. I am attaching a couple of modified versions of the source files from the 'mrs' directory. Primarily the changes amount to additional logging lines that may help shed some light on where things are going wrong. The modifications are summarized below

(1) api.py - Added some notes on how to add month and gender translations. Changes: lines 52 -110, 311, 313

(2) json.py - the relevant code is in the parseOne and parseAll functions. I added a line towards the end of each that looks like:

logging.debug(u'Fetched patient: %s' % patient) 

There is also a similar line at the top of the parseOne which logs the raw data returned from OpenMRS. WARNING! Those lines should probably be removed before deployed. 

b. Settings. Not sure to what degree this is effecting the patient fetch but may be related. In the top-level settings.py file for mds-i.e. /opt/sana/settings.py  you may want to look at the LANGUAGE_CODE (default is 'en-us')  Django docs references: Settings and Internationalization and localization 

Let me know if any of that helps.

Best,

Eric

[Alex Ewing]

Hi Eric, thanks for the response. We are going to get on that problem soon. We actually don't have anyone in-house with much experience on the server side of things and have to get someone to come in and help with that stuff. Unfortunately, we seem to be also having a problem which may or may not be related. While we did get a simple test upload procedure to work upload, we haven't been able to get our own follow-up procedure to upload to the server.

This results in a whole bunch of stuff in the log, but I think the important part is the end:

+0.196s openmrs.py:256  Validating permissions to manage sana queue
+0.198s openmrs.py:260  Got result {"status":"OK","code":"unspecified","message":"User has Manage Sana Queue privileges"}
+0.198s openmrs.py:267  Uploading procedure
+0.199s openmrs.py:279  Encoded parameters, checking files.
+0.200s openmrs.py:292  About to post to http:\/\/192.168.120.5:8080\/openmrs\/moduleServlet\/moca\/uploadServlet
+0.242s openmrs.py:294  Got result {"status":"FAIL","code":"unspecified","message":"Concept error: Concept Undefined: (INCRE3CHECK, ??????)"}
+0.242s openmrs.py:300  Done with upload
+0.242s api.py:297  API: RESULT = False
+0.243s json.py:229  Failed to register procedure: sp.pk -> 109, Concept error: Concept Undefined: (INCRE3CHECK, ??????)

One idea I had was that this is also a problem with the MDS/Chinese. In the test upload, the hard-coded name, birthdate concepts are asked using Chinese, but the user defined concept is in English. In this follow-up procedure, all the questions are asked using English. Is it possible that because of the MDS issue, the procedures aren't matching their definitions in the dictionary?

[Alex Ewing]

Sorry, that last part contains an error. What I meant to say was that in the follow up procedure, all of our questions (user defined) are asked using Chinese, i.e. everything in the description field is Chinese. While the hard-coded procedures work regardless of what is put in the description field, in the user defined procedures, they only work if they are in English.

[Maoyi Tian]

Thank you Eric for the reply. I am working with Alex on this project. 

May I clarify the current issue we had?

I created a procedure designed for the project, most of the 'questions' in the procedure were written in Chinese language. I saved the procedure in the res/raw folder. When we were trying to upload from this procedure, the upload failed. We checked the MDS/LOG, all Chinese characters were shown as ? (as illustrated in Alex's last response). 

I did another test, create another new procedure, and the "questions" in this procedure were written in English. The upload was successful from this procedure.

Could you please help us to figure out what is the problem is? Thank you.

Maoyi

[Eric Winkler]

Apologies for the slow response. I have been trying to reproduce and think I may finally have an answer and I believe it is a combination of:

OpenMRS

1) User default locale. Set the default locale in OpenMRS for the user to be Chinese, etc. The profile access should be visible as a button labeled "My Profile"-or whatever the translation is for other languages-in the upper right corner of the browser window. You can also set it globally in the administration global settings for when you add new users. 

2) Localized concept definition. In the dictionary, if you search for a concept and select it to view/edit, you should see a list of the locales that are enabled in the admin. You will need to add the translated question/description field for the locale.

MDS

1) Check the character sets of the database. Please refer to the link here here. Basically if either the request log or saved procedure tables aren't getting opened with utf-8 character encoding you will may see what are considered  unrecognized characters set getting replaced with ?'s. 

Hope some of that helps.

Best,

Eric