Localization issues in CPRS. A question about non-US formatting.

[Kevin Toppenberg]

We recently have had reason to look into an issue with dates in CPRS and have a better understanding of some localization issues in CPRS.

First some background.  In Delphi, one can get the date from the system with a "Date" function.  This is a number of fractional days from yr 1800 or something.  So this will be consistent in all locations.  But then, to display to the user, the code calls DateToStr which converts this into something readable, e.g. "3/2/1972".  Because Delphi tries to be somewhat localization friendly, is uses, by default, the localization settings specified by Windows.    So the above date could, in another country, show up as "1972/2/3".

So we encountered a situation with CPRS use in another country, where the standard formatting of dates is "dd/mm/yyyy" instead of "mm/dd/yyyy" like in the US.    This non-US external form of a string was then being passed to the server and generating errors.  With a date like "3/2/1980", the server would accept the date.  But with "28/5/2000" it would balk.

We were able to fix the specific problem by using a custom date formatting string when generating dates to talk back to the server. 

So doing a quick code search in CPRS, we were able to find a variety of places where the creating of external forms of dates were being hard coded to US standard.   And our first thought was to change all those such that they use localization-specific formats.  But Eddie brought up the concern of changing some of the places, but then perhaps missing some others.  At least the way it is, a user can know that the entire program is in US format and understand all such dates accordingly.  Perhaps some dates will be sent from the server in external (US) format in some RPC call, rather than as internal format dates for CPRS to convert.  Or perhaps we will miss some area of the code that needs to be changed.

So question for the message board: for TMG-CPRS, would it be better for us to do our best and try to get dates to display in local-specific format--and perhaps miss something somewhere?  Or should we just leave it in US format, where we can be sure that it is all consistent?

Thanks
Kevin

[Chris Farley]

Check out this tool:

http://devtools.korzh.com/localization-delphi/

I was involved in previous projects where it did an excellent job translating software to other languages, and it might be able to help you find all the date instances.  I'm not a Delphi guy, but the team-member that used the tool was able to translate CPRS to Spanish and Portuguese in about two weeks, and it seemed fairly easy to use.

C. B. Farley

HIT Man

--
--
http://groups.google.com/group/Hardhats
To unsubscribe, send email to Hardhats+u...@googlegroups.com
 
---
You received this message because you are subscribed to the Google Groups "Hardhats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hardhats+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Sam Habiel]

Kevin,

US Dates displayed to non-US users are sure to cause medical errors
sooner or later. A point I make now and again.

VISTA can handle non-US dates if the user has a different DUZ("LANG")
and the LANGUAGE file entry for the field 10.21 DATE/TIME FORMAT
(FMTE) (K), [FMTE;E1,245] is filled out properly. DUZ("LANG") can be
either set globally in the Kernel or per User.

Fileman (MSC and V22.2) as a whole obeys DUZ("LANG") for the dates. If
any application does it's own date formatting, it should use Fileman
or the Kernel instead.

What you should do in TMG CPRS is up to you. I would vote for full
i18n and UTF-8 support (the latter older versions of Delphi [before
2009] cannot deliver).

One point you do not mention but is related is the fact that CPRS
interprets Fileman dates coming from the server (3130228.091131) as a
number. In Germany and all surrounding countries (Scandinavia, Poland,
etc) the delimiter for decimals is a comma, not a period (full stop).
If you run CPRS in one of these locales, it crashes trying to
interpret Fileman dates as a number. What it should rather do is parse
by the period so that it will work across locales.

I believe Lars Malmgren has already done both changes to CPRS that you
are contemplating and he has released the source code for his changes.
I can't find it right now, but give me a call some time at night and I
will dig for you.

Sam

[Smith, Todd]

When I was reading this article I thought to myself; I wonder how well that M might handle this situation and from what I know, perhaps pretty well. Please take a look and see if this seems to be a good fit for M. Of course, the two languages mentioned, Erlang and Clojure are pretty obscure. Someone willing to learn one of those might be interested in learning M as well.

Todd Smith
Network Analyst
Charleston Area Medical Center

http://blog.paralleluniverse.co/post/44146699200/spaceships



Confidentiality Note: The information contained in this message
may be privileged and confidential. If this e-mail contains
protected health information, you are hereby notified that any
dissemination, distribution or copying of this communication is
strictly prohibited,except as permitted by law. If you have
received this communication in error, please notify the sender
immediately by replying to this message and deleting it from your
computer. Thank you.

[Bhaskar, K.S]

Here is an Erlang binding to GT.M: https://github.com/ztmr/egtm

Regards
-- Bhaskar

-- 
GT.M - Rock solid. Lightning fast. Secure. No compromises.

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

[Smith, Todd]

Thanks Bhaskar,

I was more thinking that M might be a good fit by itself but I am not as versed in how well M might handle concurrency. From what I know, it should do a good job with that but it would be nice for some of the list experts to confirm that.

Todd

--
--
http://groups.google.com/group/Hardhats
To unsubscribe, send email to Hardhats+u...@googlegroups.com<mailto:Hardhats+u...@googlegroups.com>

---
You received this message because you are subscribed to the Google Groups "Hardhats" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hardhats+u...@googlegroups.com<mailto:hardhats+u...@googlegroups.com>.

For more options, visit https://groups.google.com/groups/opt_out.

[Sergey Gavrilov]

On Thursday, February 28, 2013 9:15:41 PM UTC+4, Sam Habiel wrote:

Kevin,

Fileman (MSC and V22.2) as a whole obeys DUZ("LANG") for the dates. If
any application does it's own date formatting, it should use Fileman
or the Kernel instead.

Unfortunately, localization of the FileMan has some flaws. In my opinion, one of the significant ones is that that there is no separation between user preferences and the system locale.

For example (my personal experience), let us populate a record of the LANGUAGE file (#.85) for RUSSIAN and assign correct values to the UPPERCASE CONVERSION (#10.4) and LOWERCASE CONVERSION (#10.5) fields. Then, if there are users who prefer English roll-and-scroll interface (but still work with textual data in Russian!), we will get a big and unpleasant surprise: all their work will produce incorrect data in regards to case conversions! There will be no conversion of Russian letters at all because the M-code from the corresponding fields of the ENGLISH record in the LANGUAGE file do not affect characters with codes greater than 127! Because of this, those fields have to be modified in all records of the LANGUAGE file.

In my opinion, the LANGUAGE file should not be used to set user preferences; in its current configuration it defines parameters for all site/enterprise/country users!

Because date conversion functions are used both for user-readable output and intermediate conversions in the code, they make the problem even worse. I had to modify the $$FMTE^XLFDT to accept an additional flag ('*'), which overrides the DUZ("LANG") and makes the functions work in "English" mode. Then, I analyzed all (I hope ;-) ) date conversions in FileMan, Kernel, KIDS, Toolkit, Broker, MailMan, and HL7 (these are the packages that I use) and used this function with the new flag in those places, where MM/DD/YYYY results were expected.

In the client/server data exchange I use the system locale independent YYYY/MM/DD@HH:MM:SS format (see the FormatSettings parameter of the StrToDateTime and StrToDateTimeDef). But probably, the YYYY-MM-DD (ISO 8601) would be better.

Speaking of the CPRS, I would not use the MM/DD/YYYY format in those countries that use DD/MM/YYYY or DD.MM.YYYY. This is definitely very dangerous! I know from my own experience how difficult is to constantly switch from one format to another and not to make a mistake.

Sergey

[Bhaskar, K.S]

On 03/01/2013 06:26 AM (US Eastern Time), Sergey Gavrilov wrote:

On Thursday, February 28, 2013 9:15:41 PM UTC+4, Sam Habiel wrote:

Kevin,

Fileman (MSC and V22.2) as a whole obeys DUZ("LANG") for the dates. If
any application does it's own date formatting, it should use Fileman
or the Kernel instead.

Unfortunately, localization of the FileMan has some flaws. In my opinion, one of the significant ones is that that there is no separation between user preferences and the system locale.

For example (my personal experience), let us populate a record of the LANGUAGE file (#.85) for RUSSIAN and assign correct values to the UPPERCASE CONVERSION (#10.4) and LOWERCASE CONVERSION (#10.5) fields. Then, if there are users who prefer English roll-and-scroll interface (but still work with textual data in Russian!), we will get a big and unpleasant surprise: all their work will produce incorrect data in regards to case conversions! There will be no conversion of Russian letters at all because the M-code from the corresponding fields of the ENGLISH record in the LANGUAGE file do not affect characters with codes greater than 127! Because of this, those fields have to be modified in all records of the LANGUAGE file.

In my opinion, the LANGUAGE file should not be used to set user preferences; in its current configuration it defines parameters for all site/enterprise/country users!

[KSB] I will take a slightly more extreme position.  Internationalization of VistA should only be done with Unicode and not with ISO-8859 variants.  Use of ISO-8859 variants for multiple languages inevitably leads to ambiguities in the general case (of course, in specific cases it may not).

Because date conversion functions are used both for user-readable output and intermediate conversions in the code, they make the problem even worse. I had to modify the $$FMTE^XLFDT to accept an additional flag ('*'), which overrides the DUZ("LANG") and makes the functions work in "English" mode. Then, I analyzed all (I hope ;-) ) date conversions in FileMan, Kernel, KIDS, Toolkit, Broker, MailMan, and HL7 (these are the packages that I use) and used this function with the new flag in those places, where MM/DD/YYYY results were expected.

In the client/server data exchange I use the system locale independent YYYY/MM/DD@HH:MM:SS format (see the FormatSettings parameter of the StrToDateTime and StrToDateTimeDef). But probably, the YYYY-MM-DD (ISO 8601) would be better.

Speaking of the CPRS, I would not use the MM/DD/YYYY format in those countries that use DD/MM/YYYY or DD.MM.YYYY. This is definitely very dangerous! I know from my own experience how difficult is to constantly switch from one format to another and not to make a mistake.

[KSB] Good points.  Having been burned personally, I use mmm dd, yyyy (where mmm is text) for dates in normal correspondence, or ISO-8601 when I want to be formal (e.g., for material that is to be both computer and human readable).

-- Bhaskar

Sergey

--
--
http://groups.google.com/group/Hardhats
To unsubscribe, send email to Hardhats+u...@googlegroups.com
 
---
You received this message because you are subscribed to the Google Groups "Hardhats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hardhats+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
GT.M - Rock solid. Lightning fast. Secure. No compromises.

[Sam Habiel]

"Because date conversion functions are used both for user-readable
output and intermediate conversions in the code, they make the problem
even worse."

Can you give me an example of "intermediate conversions"?

Sam

[Kevin Toppenberg]

Sam,

Let me echo back what I am hearing you say:

On Thursday, February 28, 2013 12:15:41 PM UTC-5, Sam Habiel wrote:

Kevin,

US Dates displayed to non-US users are sure to cause medical errors
sooner or later. A point I make now and again.

Yes we should try to convert the dates in CPRS to local-specific

 

VISTA can handle non-US dates if the user has a different DUZ("LANG")
and the LANGUAGE file entry for the field 10.21     DATE/TIME FORMAT
(FMTE) (K), [FMTE;E1,245] is filled out properly. DUZ("LANG") can be
either set globally in the Kernel or per User.

If we configure the server, then we don't need to worry about it sending out incorrect date formats to the client.

 

Fileman (MSC and V22.2) as a whole obeys DUZ("LANG") for the dates. If
any application does it's own date formatting, it should use Fileman
or the Kernel instead.

Have the server do the date conversion instead of the client.  I'm of the opinion that both the server and the client will need to be local-configured (i.e. so the embedded controls display proper unicode characters etc), so either one should be able to do the conversion appropriately.

 

What you should do in TMG CPRS is up to you. I would vote for full
i18n and UTF-8 support (the latter older versions of Delphi [before
2009] cannot deliver).

I don't know the details of these formats, so I can't really comment on what this will take to achieve.  I will make a note that Eddie recently made a change to one of the VistA clients (I can't recall which one) that was not properly displaying foreign characters.  He ended up having to change a TMemo (which is the traditional way in Delphi to display text) to a TRichEdit, which does support unicode etc.

 

One point you do not mention but is related is the fact that CPRS
interprets Fileman dates coming from the server (3130228.091131) as a
number. In Germany and all surrounding countries (Scandinavia, Poland,
etc) the delimiter for decimals is a comma, not a period (full stop).
If you run CPRS in one of these locales, it crashes trying to
interpret Fileman dates as a number. What it should rather do is parse
by the period so that it will work across locales.

I would have to look into this.  There is a function in CPRS called FMDateToDateTime (or something similar).  I would assume that the code would deal with the FMDate as a string, as you describe.  But perhaps the do just port it to a floating point number (with a FloatToStr function) and then adjust from there.

If/when we do this for TMG-CPRS, I'll check this. 

 

I believe Lars Malmgren has already done both changes to CPRS that you
are contemplating and he has released the source code for his changes.
I can't find it right now, but give me a call some time at night and I
will dig for you.

Thanks for the feedback, Sam.  If you do find the changes from Lars, that would be great.

Kevin

[Marty Mendelson]

After spending 8 weeks in France in a research lab, I have become acutely aware of the dd/mm/yyyy versus mm/dd/yyyy issue. I suspect that we all are going to have to bite the bullet some day and proclaim that we shall adhere to ISO 8601. It's rather like the metric vs english problem. Some folks just won't give up feet and pounds in favor of meters and kilos, but increasingly we are being dragged (possibly kicking and screaming) into a standard worldwide system. Maybe we should get out in front on this one too.

Marty

================================================================

[r...@rcresearch.us]

Marty;

There is a lot more to it than just this aspect. There are time zones and
observer point of view that needs to be taken up as well. I have written
a couple of papers on the subject and have proposed extending VistA to
handle this. There is the issue of the patient aboard an air craft, being
relocated. First of al, on the air craft all time is in Zulu time frame.
If a soldier being evacuated from Landstuhl to Walter Reed, it would be
nice to tell when the last time the soldier got morphine. The VA is still
struggling with this when they sre supporting multiple time zones at one
data center. Contact me for more information about this.

Chris

> After spending 8 weeks in France in a research lab, I have become acutely
> aware of the dd/mm/yyyy versus mm/dd/yyyy issue. I suspect that we all are
> going to have to bite the bullet some day and proclaim that we

> *shall*adhere to ISO 8601. It's rather like the metric vs english problem.

[Kevin Toppenberg]

Just an observation.

This thread has mentioned:

iso-8859
iso-8601
Unicode
i18n
UTF-8

I am sure that these formats can be quickly reviewed on the web.  But I can't be the only one that sees these different methods/standards and wants to throw up hands in frustration.  It all gives me a sinking feeling of hopelessness.

Kevin

[Bhaskar, KS]

Why the concern, Kevin?

• Unicode is just a standard for encoding characters in all languages; UTF-8 is just a representation of Unicode.
• ISO-8601 is just a standard for encoding dates.
• i18n is just an abbreviation for internationalization.
  
• ISO-8859 is a family of standards for encoding characters in different languages; Unicode just happens to be a better one.

You deal with greater complexity every time you see a patient.

Regards
-- Bhaskar

--
GT.M - Rock solid. Lightning fast. Secure. No compromises.

---

From: hard...@googlegroups.com [hard...@googlegroups.com] on behalf of Kevin Toppenberg [kdt...@gmail.com]
Sent: Saturday, March 02, 2013 8:38 PM
To: hard...@googlegroups.com
Subject: [Hardhats] Re: Localization issues in CPRS. A question about non-US formatting.

Just an observation.

This thread has mentioned:

iso-8859
iso-8601
Unicode
i18n
UTF-8

I am sure that these formats can be quickly reviewed on the web.  But I can't be the only one that sees these different methods/standards and wants to throw up hands in frustration.  It all gives me a sinking feeling of hopelessness.

Kevin

[George Timson]

As has been said many times, "The nice thing about standards is that there are so many to choose from".

--George

[r...@rcresearch.us]

We worked on this for a long time in the MDC (MUMPS Development Committee)
over the years. We were building utility libraries for the different ways
of displaying money, geo-location, values, and dates. It takes money to
get these efforts started again to support the whole community. If this
is an important issue to you please consider patronage support of
WorldVistA and ear-mark funds and resources to representation of text
libraries. True, there are external means of doing this representation of
text, but the APIs with MUMPS can be very valuable. Also, this is another
opportunity for the point of care people to give back their solutions for
these specific problems so we can include these into the Copy-Left
versions of VistA. Get exactly what you need, not what is going to be
handed to you without hope of customization without great cost.

Chris Richardson, one of the founders of WorldVistA and former MDC member

> As has been said many times, "The nice thing about standards is that there
> are so many to choose from".
>
> --George
>

[Sergey Gavrilov]

Sam,

Below are a couple of examples (HL7 and Mailman) but I am pretty sure that there are many more in the clinical packages (e.g. to extract month and year from the date).

Example 1:

Function $$DH4PRT^HLCSFMN1 converts $H to external format and then converts the result to DD-MM-YY @ HH:MM:SS. If the user's language is switched to one that is configured for non-English dates (e.g. DD/MM/YYYY or DD.MM.YYYY), the function returns incorrect results.

Example 2:

The function $$INDT^XMXUTIL1 has the same problem.

Sergey

[Sam Habiel]

Thanks Sergey,

I believe these packages should be using the Kernel or Fileman for
date conversion. All of this will be part of the pain of making sure
VISTA works outside of the US. I think we are all agreed that VISTA as
a whole has a long way to go.

This article details the long and difficult conversion Unix and Linux
had to do to be UTF-8 compliant. Nowadays UTF-8 is the default
encoding in Linux. The same process has to take place in VISTA.
http://www.cl.cam.ac.uk/~mgk25/unicode.html

[Kevin Toppenberg]

Bhaskar,

You are right of course.  I guess it is that I find it easier to tell "hypertension" apart from "diabetes" than I can "ISO-8601" vs "ISO-8859".  I think this is because programming is not my day job.  I'll try to keep up.  :-)

Kevin

[Bhaskar, K.S]

I too have trouble keeping everything straight in my mind.  Since my memory isn't what I'd like it to be, I rely on a friend called Google who helps me a lot.

Regards
-- Bhaskar

--
--
http://groups.google.com/group/Hardhats
To unsubscribe, send email to Hardhats+u...@googlegroups.com
 
---
You received this message because you are subscribed to the Google Groups "Hardhats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hardhats+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
GT.M - Rock solid. Lightning fast. Secure. No compromises.

[Jari Friis Jørgensen]

Kevin

A while ago (2-3 years) we had the issues with this is in our Danish VistA Venerea project. We found all the places that the date was hardcoded and added our own code. There were also issues with the CPRS crashing...I can send you our version of TMG-CPRS if interested. That version works with dd-mm-yy format.

Jari 

[Kevin Toppenberg]

If you marked your changes with some unique comment, such that I can search for it, then yes, I would like the code back.  But if not, TMG-CPRS has moved on and I don't know if I could find your changes with a simply diff exam.

Thanks

Kevin