Helma NG on app engine and UTF-8

[ccwu]

Hi,

I'm trying to depoly Helma NG on app engine at http://novelplayer.appspot.com/

I have changed function index(req) in actions.js to the following:

function index(req) {
return new SkinnedResponse('skins/welcome.txt', {title: '歡迎光臨
Helma NG'});
}

The chinese characters are correctly displayed on http://localhost:8080,
but cannot be shown on app engine.

Any ideas?

Cheng-Chang

[Hannes Wallnoefer]

Hi Cheng-Chang,

I've made sure that the HTTP part and the app engine storage part work
with multi-byte Unicode characters, for example here:
http://hensotest.appspot.com/Ng/

What I haven't tested (and what's obviously causing the problems) is
having multi-byte characters in the application code. There are two
possible reasons I can think of: Either the the appcfg utility messes
the characters up during the deployment, or Helma NG uses the wrong
encoding to read the code.

After some testing, it looks like app engine just doesn't support
Unicode for deployed source code. The app engine nodes report
ANSI_X3.4-1968, which according to Wikipedia seems to be another name
for plain ASCII, and all my attempts to force UTF-8 encoding (like
setting the java "file.encoding" system property to "UTF-8") didn't
change anything.

What you could do as a workaround is to encode your strings using the
"\uxxxx" syntax, with xxxx being the hexadecimal UTF-16 character
code. The following code snippet shows how to escape any string to
that format:

helma> var str = "歡迎光臨 Helma NG";
helma> Array.map(str, function(c) {
> var cc = c.charCodeAt(0);
> return cc < 128 ? c : '\\u' + c.charCodeAt(0).toString(16);
> }).join('');
\u6b61\u8fce\u5149\u81e8 Helma NG
helma> "\u6b61\u8fce\u5149\u81e8 Helma NG"
歡迎光臨 Helma NG
helma> "\u6b61\u8fce\u5149\u81e8 Helma NG" === str
true

Hannes

2009/7/14 ccwu <ccwu6...@gmail.com>:

[ccwu]

Hi Hannes,

I searched the internet and found some discussions about it:

http://code.google.com/p/googleappengine/issues/detail?id=1349

Because Java programs with embedded Chinese Unicode in Java string do
not have such a problem, can this problem with Helma NG occurs during
the javascript compilation stage?

Cheng-Chang

[Hannes Wallnoefer]

Java class files and jar files are binary files, so the text encoding
is not dependent on the host platform file encoding. Since even
explicitly reading the source files as UTF-8 still produced garbled
characters, I think the problem must be in the deployment stage.

Hannes

> Cheng-Chang

[sleepnova]

I found some thing strange with helma-ng on AppEngine.

1. It's ok to read in source file and output UTF8 character with new
java.io.InputStreamReader(url.openStream(), 'UTF8'); and eval it with
Rhino both on local and AppEngine host.
2. It's ok to read and output UTF8 character with helma-ng on the
local, but fail on the AppEngine.

I test it with simple jack app.

function handler(env) {
return [200, {"Content-Type": "text/plain"}, ["中文測試!"]];
}

I can't even make helma-ng demo start at AppEngine, it drop me with
an

Uncaught exception from servlet
javax.servlet.UnavailableException: Initialization failed.
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler
(AppVersionHandlerMap.java:199)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler
(AppVersionHandlerMap.java:167)
at
com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest
(JettyServletEngineAdapter.java:127)
at com.google.apphosting.runtime.JavaRuntime.handleRequest
(JavaRuntime.java:235)
at com.google.apphosting.base.RuntimePb$EvaluationRuntime
$6.handleBlockingRequest(RuntimePb.java:4823)
at com.google.apphosting.base.RuntimePb$EvaluationRuntime
$6.handleBlockingRequest(RuntimePb.java:4821)
at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest
(BlockingApplicationHandler.java:24)
at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:
359)
at com.google.net.rpc.impl.Server$2.run(Server.java:820)
at com.google.tracing.LocalTraceSpanRunnable.run
(LocalTraceSpanRunnable.java:56)
at com.google.tracing.LocalTraceSpanBuilder.internalContinueSpan
(LocalTraceSpanBuilder.java:516)
at com.google.net.rpc.impl.Server.startRpc(Server.java:775)
at com.google.net.rpc.impl.Server.processRequest(Server.java:348)
at com.google.net.rpc.impl.ServerConnection.messageReceived
(ServerConnection.java:436)
at com.google.net.rpc.impl.RpcConnection.parseMessages
(RpcConnection.java:319)
at com.google.net.rpc.impl.RpcConnection.dataReceived
(RpcConnection.java:290)
at com.google.net.async.Connection.handleReadEvent(Connection.java:
428)
at com.google.net.async.EventDispatcher.processNetworkEvents
(EventDispatcher.java:762)
at com.google.net.async.EventDispatcher.internalLoop
(EventDispatcher.java:207)
at com.google.net.async.EventDispatcher.loop(EventDispatcher.java:
101)
at com.google.net.rpc.RpcService.runUntilServerShutdown
(RpcService.java:251)
at com.google.apphosting.runtime.JavaRuntime$RpcRunnable.run
(JavaRuntime.java:374)
at java.lang.Thread.run(Unknown Source)

sleepy

[sleepnova]

Forgot to mention, I also tried following modification,

AbstractResource:

public Reader getReader() throws IOException {
return new InputStreamReader(getInputStream(), "UTF8");
}

public String getContent() throws IOException {
return getContent("UTF8");
}

and

SkinParser:

public void parse(Resource res) throws IOException,
UnbalancedTagException {
parse(new InputStreamReader(res.getInputStream(), "UTF8"));
}

still fail to deal with UTF8 characters.