qewd-up in Unicode

[Michael Reach]

We have been using qewd-up for a while now, with good result. Yesterday I noticed an issue which I assume is connected to https://groups.google.com/g/enterprise-web-developer-community/c/5yXC166Qh78/m/CMvz0BIgDQAJ - non-ASCII data.

My database Cache for Windows (x86-64) 2018.1.4 (Build 505_1_20258U) Thu Sep 10 2020 10:22:2 2 EDT

Had a field containing an apostrophe like "didn’t" 

SMILE>r !,x

didn’t

SMILE>w $a($e(x,5))

8217

When the field was set to the web page via qewd-up/qewd-apps, nothing arrived. The log  output.log shows

{"actionsNRootCause":"",

i.e., didn't get that far.

When I noticed the apostrophe and replaced it with ASCII 39, it worked:

{"actionsNRootCause":"The staff found that colony morphology didn't look like penicillium spp. \r\nbecause the colony was rather gray-black while Penicillium is mostly \r\ngreen colony. So he answered Paecilomyces spp. by microscopy examination.",

I saw on that other thread that Rob said he could explain wha happened and how to set this up properly. Is that explained somewhere, or could you do it now?

Thanks,

Michael Reach

[Chris Munt]

This is an issue with the module underpinning qewd-up - mg-dbx.  Current this module offers native support for 8-bit ASCII and UTF8 encoded Unicode.  Problems arise when mg-dbx comes across raw Unicode data in the DB Server.

For example, consider the following two global nodes.  The two strings are actually the same but one (utf16) is in raw 16-bit Unicode whereas the second (utf8) is the same but encoded as UTF8.

USER>s ^CM("utf16")="abc"_$c(1488,1489,1490)_"abc"

 

USER>s ^CM("utf8")=$zcvt("abc"_$c(1488,1489,1490)_"abc","o","utf8")

 

Turning to JavaScript/mg-dbx using the UTF8 character encoding.

var result = db.get("^CM", "utf8");

db.set("^CM", "new", result);

Because mg-dbx is trading in UTF8, nodes ^CM("utf8") and ^CM("new") will be the same.  However, the same will not be true for the following code.

var result = db.get("^CM", "utf16");

db.set("^CM", "new", result);

The upshot is that mg-dbx will need a bit of work in order to natively support Unicode data held in the DB Server.

I recognize that this needs to be done so it will definitely go on the TODO list ...

Chris.

[Michael Reach]

Thanks much, Chris. Maybe you can explain what Cache' actually is using in my "U" implementation? I see from my back-and-forth emails with Rob (which I hope you saw?) that I am dealing with two different kinds of data, but I don't understand what I'm seeing exactly, which is which, and how they got that way. [Should I paste in my last couple of emails showing my test results from mg-dbx?]

Michael

[Josef Zvonicek]

Hi Chris, 

I'm also pleading for this functionality. Support from ISC and NODE.JS is not ideal.

Josef.

Dne úterý 10. srpna 2021 v 16:47:36 UTC+2 uživatel Chris Munt napsal:

[Chris Munt]

Support for Unicode (using InterSystems Cache and IRIS) will be in the next release of mg-dbx and automatically available to qewd-up.

Chris.

[Chris Munt]

For those interested in native Unicode support for InterSystems DB Servers, a new release of mg-dbx is now available with this functionality -  v2.4.26.

This is a fairly hefty change so I would only recommend that those who really need this functionality should upgrade at this time.

With regard to comments about InterSystems Node.js support, I think you have to concede that cache/iris.node isn't too bad.  After all, I wrote this module back in 2011!  However, this module was deprecated in favor of a later solution more aligned to Cache Objects and Java, so cache/iris.node fell by wayside.  Anyway, mg-dbx was developed along similar lines but is better performing and Open Source too.