Issue 131023 in chromium: strings are garbled in flash animations using 'the legacy' way of character represenation

[chro...@googlecode.com]

Status: Untriaged
Owner: ----
CC: bre...@chromium.org, yus...@chromium.org
Labels: Type-Bug Pri-2 Feature-Flash Feature-I18N OS-Chrome

New issue 131023 by js...@chromium.org: strings are garbled in flash
animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

Moved from chromium-os:2804 ( http://crosbug.com/2804 )

How to reproduce:

1. Change the UI language to Korean (on ChromeOS)
2. Go to
http://openimage.interpark.com/_nip/flash/nwel/spotlight/ticket_spot3.swf?xml_url=http://ticket.interpark.com/Community/WelcomeSpot/WelcomeSpot.xml

What's expected:

Korean characters are shown in the flash animation

Actual: Garbled (non-sense) sequences of Latin letters are shown

Additional info: do the same on Windows/Mac/Linux with UI language of the
OS set to Korean and you don't have this problem.


The issue here is that the flash animation does not use Unicode but uses a
locale-specific legacy character encoding. They can get away with this
because the majority of people visiting their site have their OS UI
language set in such a way that the legacy character encoding used by Flash
matches the default legacy encoding in that UI language.

What we can do:

We can at least match the Windows behavior. Flash can assume the legacy
codepage/encoding corresponding to the current UI language of ChromeOS.

Korean => windows-949
Japanese => windows-936
Simp Chinese => windows-932
Trad. Chinese => windows-950
Thai => Windows-874
Russian => Windows-1251
Arabic => Windows-1256
Hebrew => Windows-1255
.....


@brettw: I thought that you had implemented this, but I still this problem
in M20.


See http://crosbug.com/2804 for more details.

[chro...@googlecode.com]

Comment #1 on issue 131023 by js...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

bug 130979 may be about the same issue.

[chro...@googlecode.com]

Comment #2 on issue 131023 by js...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

The version I tried this is 20.0.1132.15 dev (ChromeOS)

[chro...@googlecode.com]

Updates:
Cc: ray...@chromium.org

Comment #3 on issue 131023 by jeff...@google.com: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

Related to the work Raymes is doing?

[chro...@googlecode.com]

Comment #4 on issue 131023 by js...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

To clarify: To make NPAPI flash plugin behave the same way on CrOS as on
Windows/Linux/Mac, we need to tweak the way CrOS sets locale-related
environment variable. Adobe's NPAPI flash plugin on Linux (and CrOS) refers
to LC_ALL, LANG, etc, but none of them is set on CrOS.

However, we don't have to worry about NPAPI flash any more on CrOS. CrOS is
only shipped with PPAPI flash plugin.

So, what needs to do is to make PPAPI flash plugin derive the legacy
character encoding to use from the current OS UI language.

Although the number of flash animations using the legacy char encoding is
dwindling (and ones using Unicode are on the rise), it'll be quite a while
before they go completely go away.

So, it's important to have this implemented for non-Western-European
languages (currently, PPAPI flash plugin assumes windows-1252 that is for
Western European languages).

If we want to take a step further (as a follow-up bug), we can do better by
character encoding autodetection than what Adobe's NPAPI plugin does on
Windows/Mac/Linux. That way, regardless of the UI language, in most cases,
we can view characters ungarbled in flash animations using a legacy
encoding.

[chro...@googlecode.com]

Updates:
Cc: ko...@chromium.org

Comment #5 on issue 131023 by js...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

> So, what needs to do is to make PPAPI flash plugin derive the legacy
> character encoding to use from the
> current OS UI language.

BTW, it'd be even better if PPAPI flash plugin can use Chrome's preference
for the default charset ( intl.charset_default ). That is set to the
locale-specific legacy encoding out of box, but some users may change that.
So, respecting that makes flash plugin's behavior identical to the
rendering of HTML pages without charset label.

[chro...@googlecode.com]

Comment #6 on issue 131023 by js...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

FYI: On Linux, you can start Chrome with LC_ALL=ko_KR.UTF-8 and visit

http://openimage.interpark.com/_nip/flash/nwel/spotlight/ticket_spot3.swf?xml_url=http://ticket.interpark.com/Community/WelcomeSpot/WelcomeSpot.xml

Korean characters are NOT garbled with that. OTOH, setting the UI language
to Korean does not help.

Likewise, you can set LC_ALL to ja_JP.UTF-8 before starting Chrome and go
to http://www.theb-hotels.com/the-b-roppongi/jp/ . Japanese characters
won't be garbled (see the screenshot attached to bug 130979).

[chro...@googlecode.com]

Updates:
Cc: js...@chromium.org kenji...@chromium.org t...@chromium.org

Comment #7 on issue 131023 by js...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

Issue 130979 has been merged into this issue.

[chro...@googlecode.com]

Comment #8 on issue 131023 by bre...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

I thought I had hooked this up. It uses GetDefaultCharSet in
http://src.chromium.org/viewvc/chrome/trunk/src/ppapi/c/dev/ppb_char_set_dev.h?revision=116490&view=markup
So I take it this isn't working? It calls through eventually to
GetContentClient()->renderer()->GetDefaultEncoding()

[chro...@googlecode.com]

Comment #12 on issue 131023 by kin...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

> Still, it's kinda mystery why that does not work on CrOS while it works
> on Linux.

On Linux, it works when LANG is "correctly" set. By works I mean
IDS_DEFAULT_ENCODING is used and resolved to the lang's default charset;
preference seems not used on any platform. (I've found an issue filed on
this: http://crbug.com/56615)

On CrOS,
> On ChromeOS, neither LC_ALL nor LANG is set so that the default
> windows-1252 (iso-8859-1) is used.
I guess this is the situation.

[chro...@googlecode.com]

Updates:
Cc: mu...@chromium.org kin...@chromium.org

Comment #13 on issue 131023 by kin...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

Issue 126945 has been merged into this issue.

[chro...@googlecode.com]

Comment #15 on issue 131023 by bre...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

I'm having a bit of trouble following all this... But keep in mind we have
some general Flash text rendering problems on ChromeOS regardless of
encoding. I'd be sure that the text renders properly in a Unicode mode
before worrying about whether the proper encoding is used.

[chro...@googlecode.com]

Comment #16 on issue 131023 by kin...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

@jshin:
> the behavior on CrOS and Linux should be identical, but it's different
> for an unknown reason.

I guess is it the same reason as
http://src.chromium.org/viewvc/chrome/trunk/src/chrome/renderer/renderer_main.cc?r1=77118&r2=77117&pathrev=77118
? That is, sandbox process is starting earlier before UI language set.

@brettw:
I believe this is an encoding issue, rather than rendering issue. Since the
garbled text is revealing its byte-sequence, we can check how it happened:
1. open the reported flash in correct encoding by the steps in #6,
2. copy&paste the correctly rendered text to an editor, and save it in the
legacy encoding (euc-kr for Korean and shift_jis for Japanese),
3. open the text with Chrome, and manually set the encoding to iso-8859-1
from wrench menu.
Then I've seen exactly the same sequence of broken text. So the situation
is, flash is trying to decode a legacy encoding text, using an wrong
decoder (in this case, iso-8859-1).

[chro...@googlecode.com]

Updates:
Cc: to...@chromium.org

Comment #17 on issue 131023 by js...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

@brettw: The problem you mentioned is kinda orthogonal to this issue. This
issue is about getting byte-strings converted correctly to Unicode string
for processing (rendering or others). The issue you mentioned is how to
render a Unicode string correctly (however we gets it, Unicode from the
beginning or legacy-byte-encoding converted to Unicode).

@kinaba : Thank you for digging that up.

http://src.chromium.org/viewvc/chrome/trunk/src/chrome/renderer/chrome_content_renderer_client.cc?view=markup
has
this:

std::string ChromeContentRendererClient::GetDefaultEncoding() {
return l10n_util::GetStringUTF8(IDS_DEFAULT_ENCODING);
}

So, on CrOS, when a sandbox process reaches the above code, its locale is
still not set properly and ends up getting ISO-8859-1. One way to fix the
above is to change the above function to return the current value of a
preference 'intl.charset' instead of IDS_DEFAULT_ENCODING. Hmm...
actually, it may not work because the value of 'intl.charset' will end up
being retrieved from IDS_DEFAULT_ENCODING unless a user changes it
explicitly in Settings.

Hmm... in that case, how could we get localized strings in renderer process
(e.g. Webkit strings) on ChromeOS?

http://src.chromium.org/viewvc/chrome/trunk/src/chrome/renderer/renderer_main.cc?r1=77118&r2=77117&pathrev=77118

may have to call InitSharedInstanceWith*() as well?

ChromeMainDelegate::PreSandboxStartup() does this, too but it may not work
as intended on CrOS due to the issue mentioned in the comment in the
aforementioned code.

[chro...@googlecode.com]

Comment #18 on issue 131023 by to...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

Most processes call ResourceBundle::InitSharedInstanceWithLocale in
http://git.chromium.org/gitweb/?p=chromium/chromium.git;a=blob;f=chrome/app/chrome_main_delegate.cc;h=ab0729b1ebe17732a0baeaa56b9ff4f97cd5dc8d;hb=HEAD#l594 .
The full list of processes is here:
http://git.chromium.org/gitweb/?p=chromium/chromium.git;a=blob;f=chrome/app/chrome_main_delegate.cc;h=ab0729b1ebe17732a0baeaa56b9ff4f97cd5dc8d;hb=HEAD#l258

The zygote has it set (since we need to load them before enabling the
sandbox), but are you saying a different process type needs it as well?
l10n_util::GetStringUTF8(IDS_DEFAULT_ENCODING) should return the value from
the pack file or DCHECK if nothing is loaded. If we're not hitting a
DCHECK, maybe we're loading the wrong locale pak file?

[chro...@googlecode.com]

Comment #19 on issue 131023 by js...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

Tony,

http://src.chromium.org/viewvc/chrome/trunk/src/chrome/renderer/renderer_main.cc?r1=77118&r2=77117&pathrev=77118
has
this for CrOS in RendererMain. Could that indicate that we have to do
something similar about the resource bundle loading here?

#if defined(OS_CHROMEOS)
221 // As Zygote process starts up earlier than browser process gets
its own
222 // locale (at login time for Chrome OS), we have to set the ICU
default
223 // locale for renderer process here.
224 // ICU locale will be used for fallback font selection etc.
225 if (parsed_command_line.HasSwitch(switches::kLang)) {
226 const std::string locale =
227 parsed_command_line.GetSwitchValueASCII(switches::kLang);
228 base::i18n::SetICUDefaultLocale(locale);
229 }
230 #endif

[chro...@googlecode.com]

Comment #21 on issue 131023 by kin...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

> how could we get localized strings in renderer process (e.g. Webkit
> strings) on ChromeOS?

Ahh, by the way, I've been overlooking it but on ChromeOS those strings
seem never get localized (e.g., <input type="file"/> rendered in English
regardless of UI lang.) Hmm...

[chro...@googlecode.com]

Updates:
Cc: kar...@google.com viettrun...@chromium.org yzs...@chromium.org
jeffr...@chromium.org fal...@chromium.org i...@chromium.org

Comment #28 on issue 131023 by h...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

Issue 140627 has been merged into this issue.

[chro...@googlecode.com]

Comment #30 on issue 131023 by h...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

I agree with kinaba@ (comment #16) that this is an encoding problem,
however I have yet to track down where the incorrect MBCS-to-UTF-8
conversion (using Latin code page) takes place inside pepper flash.

[chro...@googlecode.com]

Updates:
Cc: sh...@chromium.org

Comment #33 on issue 131023 by h...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

The comment #6 from jshin@ is very interesting:

"you can set LC_ALL to ja_JP.UTF-8 before starting Chrome and go to
http://www.theb-hotels.com/the-b-roppongi/jp/ . Japanese characters won't
be garbled"

This indicates that Flash MBCS-to-UTF decoding has a hidden dependency on
the LC_xxx environment variables. Adding shess@ as it is related to several
outstanding flash localization problems.

Unfortunately we can't set LC_ALL to ja_JP.UTF-8 on CrOS because the
/usr/lib/locale/locale-archive is stripped off of all non-English locales.

[chro...@googlecode.com]

Comment #34 on issue 131023 by h...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

On Chrome OS, it seems that setting LC_ALL would not solve the problem. I
have had the following experiement:

(1) set CrOS owner language to Japanese;
(2) modify session_manager_setup.sh to set LC_ALL to ja_JP.utf8;
(3) add ja_JP.UTF-8 to /usr/lib/locale/locale-archive;
(4) modify ZygoteHostImpl to pass "--lang=ja" in the command line

After all of the above 4 changes, I have ensured that the renderer process
uses the Japanese locale, and all ICU- and glibc- based locales are in
Japanese. However, the website "www.gaitame.com" still shows garbled text
like before.

[chro...@googlecode.com]

Comment #35 on issue 131023 by kin...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

The
(A) http://www.gaitame.com
looks exhibiting a different issue than the other previously reported sites
like
(B) http://neutral.x0.com/home/sushida/play1.html, or
(C)
http://openimage.interpark.com/_nip/flash/nwel/spotlight/ticket_spot3.swf?xml_url=http://ticket.interpark.com/Community/WelcomeSpot/WelcomeSpot.xml

The sites (B) and (C) are already fixed by
(x) http://crbug.com/126945 : use GetDefaultCharSet() for MBCS conversion
in Pepper Flash, and
(y) http://crbug.com/141110 : make GetDefaultCharSet() to reflect the
font/encoding setting of Chrome.
Note that, before the change (y), ChromeOS GetDefaultCharSet() always
returned "iso-8859-1" due to not-yet-fixed bug
(z) http://crbug.com/131781
and hence the sites (B) and (C) had been garbled. But now the fix (y) is
functioning also on ChromeOS.

On the other hand, (A) www.gaitame.com is garbled on all platforms and on
all configurations (if Pepper Flash is used).
This is different--this is not OS specific trouble. Apparently, my fix (x)
has not been fully covering all the cases.

[chro...@googlecode.com]

Updates:
Status: Verified

Comment #37 on issue 131023 by h...@chromium.org: strings are garbled in

flash animations using 'the legacy' way of character represenation
http://code.google.com/p/chromium/issues/detail?id=131023

(No comment was entered for this change.)