Properties files, UTF-8 and Mozilla Translator (warning)
Ricardo Palomares Martinez
Some time ago, Axel published a small patch for Mozilla Translator to
write properties files using UTF-8 encoding instead of Unicode.
In my patched version of MT, and I think that also in the one
published by Giacomo, that patch is in use, but I've realized that
the patch just _writes_ using UTF-8 and doesn't care about _reading_
UTF-8.
I've browsed the J2SE documentation and it turns that
java.util.Properties, the Java class for dealing with Properties
files, doesn't read anything other than ISO-8859-1 and requires to
escape Unicode characters:
http://java.sun.com/j2se/1.5.0/docs/api/java/util/Properties.html#encoding
And, despite many enhancement requests to Sun in order to allow the
class to be able to read files encoded with UTF-8, no real progress
has been made to date.
Funnily enough, it looks that the motivation for requesting UTF-8
support in general, and in Mozilla environment in particular, is that
people can write files using directly UTF-8 characters in their text
editors instead of \u escape sequences. So, the patch from Axel
probably solves the part of the problem that is not really a problem
for people using MT. :-)
The way I see it, there are two possible solutions: revert back Axel's
patch, or extend Properties class in MT to make it read also UTF-8
encoded files.
It happens to be that MT centralizes all .properties persistance in
org.mozillatranslator.io.file.PropertiesAccess.java (the file to which
Axel's patch is applied to), and there Properties class is extended to
SortedProperties, which is the class used while actually reading the
file. So, I've gone with the second solution (just for fun, see below)
and I've copied and pasted the .load() method source code directly
from java.util.Properties, just changing "ISO-8859-1" to "UTF-8". It
should still be able to parse \u escaped files, and it definitely
works with UTF-8 files.
Now, as I think that at least Robert and Giacomo are using the MT
patched version, be warned, guys. The way you have it right now, you
can't export a .properties file and then import it again. I've noticed
that when trying to reuse translation of files from MOZILLA_1_8_BRANCH
to Sunbird 0.3 alfa 2.
In the long term, I'm not too sure that my hack is the better way. The
Properties files specification comes from Sun, and as of today, it
states that only ISO-8859-1 encoding must be used. Sourceforge.net
anonymous CVS is again online, and I checked that Serhiy has committed
the patch for OS/2 install.js, but not Axel's patch, my changes to
Edit Phrase dialog, nor Giacomo's patches.
I wrote to Serhiy yesterday regarding that, and suggesting him the
possibility of granting me access to the CVS as a fellow developer
(still no answer). There are a number of small things I would like to
touch and I really like that it would be better if we keep things
under control.
Any comments?
--
If it's true that we are here to help others,
then what exactly are the OTHERS here for?
Axel Hecht
It is a fact that the properties parser in mozilla uses utf-8 encoding
and nothing else, and we're not likely to change that.
I suggest that MT follows that, because, spec or not, it's targeting
mozilla development.
Good catch though, thanks.
Axel
Benjamin Smedberg
> It is a fact that the properties parser in mozilla uses utf-8 encoding
> and nothing else, and we're not likely to change that.
>
> I suggest that MT follows that, because, spec or not, it's targeting
> mozilla development.
So, you want MT to re-implement a .properties parser, instead of using the
"native" Java properties parser?
I'm not sure that officially diverging the charset defaults between Java and
Mozilla serves much good purpose.
--BDS
Axel Hecht
We did. That's done ages ago and I don't think anyone should fix it
inside Mozilla. So the question is only if MT should be able to cope
with the data that Mozilla uses or not.
Axel
David Fraser
anything that is not in ISO-8859-1 to be escaped as \uNNNN makes files
unreadable and uneditable by people trying to use a normal text editor.
I have had one idea though: we could make a simple DTD parser that
serves the DTD entity-string mapping in the same way as a properties
file, and then suddenly we would
a) not have to worry about whether our properties files are valid
b) have one main format for translators instead of two
c) have a headache persuading people to convert all their files
But this is maybe an idea worth considering
David
Benjamin Smedberg
> I have had one idea though: we could make a simple DTD parser that
> serves the DTD entity-string mapping in the same way as a properties
> file, and then suddenly we would
> a) not have to worry about whether our properties files are valid
> b) have one main format for translators instead of two
> c) have a headache persuading people to convert all their files
> But this is maybe an idea worth considering
Bug 253669, and I'll look at implementing this.
--BDS
Tsahi Asher
> Benjamin Smedberg wrote:
>
>> Axel Hecht wrote:
>>
>>> It is a fact that the properties parser in mozilla uses utf-8
>>> encoding and nothing else, and we're not likely to change that.
>>>
>>> I suggest that MT follows that, because, spec or not, it's targeting
>>> mozilla development.
>>
>>
>> So, you want MT to re-implement a .properties parser, instead of using
>> the "native" Java properties parser?
>>
>> I'm not sure that officially diverging the charset defaults between
>> Java and Mozilla serves much good purpose.
>
> It serves the purpose of making the files human-editable. Requiring
> anything that is not in ISO-8859-1 to be escaped as \uNNNN makes files
> unreadable and uneditable by people trying to use a normal text editor.
>
Unipad (http://www.unipad.org) can edit escaped unicode transparently,
hiding the escape sequences from the user. only problem is, the free
version is limited to 1000 characters. and it costs a freaking $199 to
buy (a text editor!). i still have the last beta version, that works
with no limitations, for anyone interested, but it works until april
14th 2002, so you have to set your system clock back (i've written a
small utility that does that and then launches unipad).
then again, maybe there is some other editor that can do that too out there.
--
Tsahi Asher
Hebrew L10n Team
http://www.mozilla.org.il
Michael Wolf
>> It serves the purpose of making the files human-editable. Requiring
>> anything that is not in ISO-8859-1 to be escaped as \uNNNN makes files
>> unreadable and uneditable by people trying to use a normal text editor.
>>
> Unipad (http://www.unipad.org) can edit escaped unicode transparently,
> hiding the escape sequences from the user. only problem is, the free
> version is limited to 1000 characters. and it costs a freaking $199 to
> buy (a text editor!). i still have the last beta version, that works
> with no limitations, for anyone interested, but it works until april
> 14th 2002, so you have to set your system clock back (i've written a
> small utility that does that and then launches unipad).
>
> then again, maybe there is some other editor that can do that too out
> there.
I use Unired. You can define *.dtd and *.properties files as file types
under Options-->File settings and assign them a default charset and a
Unicode representation, i.e. \uxxxx for *.properties files. For dtd
files you don't need assign a Unicode representation. You can choose
charset and unicode representation in the file and save dialog, too, but
they will then be valid for the current file only.
Daniel B.
> Hi,
>
> Some time ago, Axel published a small patch for Mozilla Translator to
> write properties files using UTF-8 encoding instead of Unicode.
What do you mean? Saying "UTF-8 encoding _instead of_ Unicode" doesn't
make sense, since UTF-8 is an encoding of Unicode and Unicode is not an
encoding of Unicode.
Did you mean instead of UTF-16 or whatever the 32-bit simple encoding is
called (UCS-32?)?
Daniel