Inform and character sets
Carl Muckenhoupt
I have a project in mind that would involve the use of characters
outside the ISO 8859-1 character set. I wish to write it in Inform.
Now, Inform knows about character sets. It even has a command-line
switch for easy selection of any of the nine ISO Latin character sets.
But what effect does this switch have?
Z-code didn't originally support more than a subset of Latin-1, but the
Z-machine Standards Document 1.0 contains an extension for mapping
ZSCII characters 155-251 into regions of Unicode. This extension,
however, is not universally supported - in fact, I have yet to find a
single interpreter that supports it! The closest I've seen is Frotz,
which recognizes Unicode characters and substitutes a question mark for
everything outside of Latin-1. Other interpreters, which don't
recognize the Unicode extension at all, tend to output consistent but
arbitrary values - even if the interpreter is using the right font, the
attempt at Unicode translation messes things up somehow. The only
character set that passes through such interpreters intact is, of
course, Latin-1.
So what can be done about this? Well, if you let the compiler default
to Latin-1 encoding, any character will pass through unscathed (at
least, once you've added it to the zcaracters table if necessary). The
player can then select a font for the intended character set.
In short, the only way to display non-Latin-1 characters in Inform is
to specifically avoid using Inform's support for them.
Sent via Deja.com http://www.deja.com/
Before you buy.
Andrew Plotkin
> Z-code didn't originally support more than a subset of Latin-1, but the
> Z-machine Standards Document 1.0 contains an extension for mapping
> ZSCII characters 155-251 into regions of Unicode. This extension,
> however, is not universally supported - in fact, I have yet to find a
> single interpreter that supports it!
Zip2000 on the Acorn might.
> In short, the only way to display non-Latin-1 characters in Inform is
> to specifically avoid using Inform's support for them.
Well, you're right. This is a purely hypothetical feature on the
Z-machine.
I'd say use Glk, but it's purely hypothetical there too. :-) My only moral
victory is that I refuse to add a feature to the Glk spec until I've
implemented it, at least experimentally. That's the lesson of the
Z-machine Unicode extension, I'm afraid...
--Z
"And Aholibamah bare Jeush, and Jaalam, and Korah: these were the
borogoves..."
Matthew T. Russotto
Carl Muckenhoupt <ca...@wurb.com> wrote:
}I have some irony to report.
}
}I have a project in mind that would involve the use of characters
}outside the ISO 8859-1 character set. I wish to write it in Inform.
}In short, the only way to display non-Latin-1 characters in Inform is
}to specifically avoid using Inform's support for them.
Or to get interpreter authors to support the non-Latin-1 stuff.
There's a problem there in that even those of us who have Unicode
support in the OS (Mac and Java both have it, though the Mac's API is
IMO cumbersome) probably don't have the Unicode-enabled fonts and thus
no way to see whether any such support is working
--
Matthew T. Russotto russ...@pond.com
"Extremism in defense of liberty is no vice, and moderation in pursuit
of justice is no virtue."
Iain Merrick
> Carl Muckenhoupt <ca...@wurb.com> wrote:
> > Z-code didn't originally support more than a subset of Latin-1, but the
> > Z-machine Standards Document 1.0 contains an extension for mapping
> > ZSCII characters 155-251 into regions of Unicode. This extension,
> > however, is not universally supported - in fact, I have yet to find a
> > single interpreter that supports it!
[...]
> I'd say use Glk, but it's purely hypothetical there too. :-) My only moral
> victory is that I refuse to add a feature to the Glk spec until I've
> implemented it, at least experimentally. That's the lesson of the
> Z-machine Unicode extension, I'm afraid...
I didn't realise until recently that Latin-1 doesn't include 'curly'
quotes, em-dashes and the like, which are the main reason I'd want to
use an extended character set. I had thought Glk would do this sort of
thing, but now the stuff in the spec about using Latin-1 and 'keeping it
holy' just makes me gnash my teeth.
So some sort of Unicode extension to Glk would be really good, is what
I'm saying. Hint hint.
(Not being Kevin Forchione, I won't bother pointing out that TADS
supports Unicode.)
(Hmmm... what's the charset situation in Hugo?)
--
Iain Merrick
i...@cs.york.ac.uk
David Kinder
> to specifically avoid using Inform's support for them.
It's one of those nasty situations where no interpreter author has
implemented it because no game has it, and no game author has used it
because ...
One thing to do would be to write an Inform program which should (if
the interpreter implemented it) output all the characters in the various
character sets, and upload it to GMD. This would at least give
interpreter authors something to work against and might encourage support
to appear.
I'll also add my vote for some sort of Unicode addition to Glk (though
implementing it will doubtless cause me some grief :-)
David
Carl Muckenhoupt
"David Kinder" <D.Ki...@btinter-remove-to-reply-net.com> wrote:
> > In short, the only way to display non-Latin-1 characters in Inform
is
> > to specifically avoid using Inform's support for them.
>
> It's one of those nasty situations where no interpreter author has
> implemented it because no game has it, and no game author has used it
> because ...
Hmm. Perhaps the best thing, then, would be to compile two versions of
my code: a "Latin-1" version that would actually be playable on current
interpreters with the right font, and a Unicode version to provide
interpreter writers with an incentive to add support for the Unicode
extension (and a game with which to test that support).
Carl Muckenhoupt
russ...@wanda.vf.pond.com (Matthew T. Russotto) wrote:
> Or to get interpreter authors to support the non-Latin-1 stuff.
> There's a problem there in that even those of us who have Unicode
> support in the OS (Mac and Java both have it, though the Mac's API is
> IMO cumbersome) probably don't have the Unicode-enabled fonts and thus
> no way to see whether any such support is working
Does Zplet currently attempt to recognize the Unicode extension?
Carl Muckenhoupt
Iain Merrick <i...@cs.york.ac.uk> wrote:
> So some sort of Unicode extension to Glk would be really good, is what
> I'm saying. Hint hint.
Or, alternately, an optional extension that allows the program to
specify an 8-bit character encoding. This would be much easier to
implement than full-blown Unicode support, and (unlike the Z-machine
Unicode extension) would be completely compatible with the current
(unofficial) method of selecting an encoding manually by loading an
alternate font.
> (Not being Kevin Forchione, I won't bother pointing out that TADS
> supports Unicode.)
Obviously not Glk-TADS, though. And anyway, Inform still seems like
the language of choice for languages other than English.
mjr_
>Iain Merrick <i...@cs.york.ac.uk> wrote:
>> (Not being Kevin Forchione, I won't bother pointing out that
>> TADS supports Unicode.)
>
>Obviously not Glk-TADS, though.
Actually, it does. How can this be, when glk itself doesn't
support unicode? Let me explain...
First off, I should clarify that Unicode support is a TADS 3
feature - TADS 2 supports any Latin-X character set, but only
one in a given game, whereas TADS 3 uses exclusively Unicode.
TADS 3 has an architectural boundary between the VM and the I/O
layer. Internally, everything is in Unicode: a compiled game
file on disk has all of its strings stored in Unicode, and all
string manipulation in memory is done in Unicode (which is cool
because it means that things like upper/lower case conversions
work correctly for every character from every alphabet). When
text is written to the display, however, the I/O layer converts
from Unicode to the local character set; likewise, the I/O layer
converts characters read from the keyboard or from external text
files to Unicode.
The "virtual OS" layer, which is responsible for the actual
device operations (such as showing text on the display or
reading it from the keyboard), sees whatever local character set
it wants - it tells the I/O layer what character set to use in
all OS-level calls, and the I/O layer maps accordingly. In glk-
tads, glk is part of (actually, most of) the OS layer; so if glk
wants Latin-1, glk gets Latin-1, and is never bothered with
Unicode characters. If in the future glk is upgraded for
unicode, it would tell the I/O layer to send it unicode (in
whatever representation - another thing the I/O layer can do is
map to the various different unicode encodings, such as UTF-8 or
UTF-16).
So, even though glk itself doesn't support unicode, glk-tads 3
does, in the same way that all tads 3 implementations do:
everything is done with unicode internally, and text is
converted to and from the local character set on input and
output.
>And anyway, Inform still seems like the language of choice for
>languages other than English.
I hope I'm not prosthyletizing too much, but I think tads 3 will
prove exceedingly flexible for building non-english libraries.
The ability to build the entire grammar declaratively is hugely
beneficial, and unicode makes all of the traditional nasty
character set issues practically disappear. I am also making an
effort to build the library in such a way that the english parts
are thoroughly isolated for easy replacement.
Of course, until I get the english version of the library done,
tads 3 isn't even very useful for building english games, let
alone games in other languages; so I'd best get back to it
rather than making all of these outrageous claims...
--Mike
-----------------------------------------------------------
Got questions? Get answers over the phone at Keen.com.
Up to 100 minutes free!
http://www.keen.com
mjr_
>> [about the tads 3 unicode<->local character mapping]
>I have one concern, though: What happens when the I/O layer
>gets a character that isn't part of the local character set?
>Will it try to normalize it by substituting the best equivalent
>(e.g., 'a' for 'a-ogonek'), or will it do something more
>anonymous, like the question marks I was complaining about in
>Frotz? The former would be preferable, but it may be an
>unreasonable thing to expect.
The mapping is directed by an external file (the "mapping
file"). Suitable mapping files are supplied with the binary
distribution, so the existence of the mapping file is meant to
be transparent to players; but the method for creating a mapping
file is documented and fairly easy, so people porting to new
systems with proprietary character sets can create their own
mappings, and players are also free to do so if they don't like
the supplied ones.
At any rate, the mapping file specifies what to do with each
Unicode character to turn it into a local character, and what to
do to each local character to turn it into a unicode character.
As a last resort, there's a "default" character in the mapping
file that gives the local character to use for each otherwise
unmapped character; you'd define this to a question mark or a
rectangle or whatever, so you'd get Frotz-like behavior for a
character with no mapping defined in the mapping file. However,
you can also explicitly include mappings for approximations,
such as 'a-ogonek' to 'a'. (In fact, you can map a single
unicode character to a series of local characters; the ASCII 7
mapping, for example, maps the copyright symbol to the
string "(c)".)
Note that several unicode characters could be mapped to the same
local character in this fashion, so various accented versions
of 'a' can all be mapped to an unaccented 'a'. This could
obviously lead to irreversible mappings, but the intention is
merely to provide graceful degradation on output; for input
purposes, the mapping file specifies a unique to-unicode mapping
for each local character, so there is no ambiguity.
Sorry for the complicated answer to a simple question. The
short answer is that what happens to a given unicode character
on output is up to the mapping file, and isn't hard-wired into
the system. So, ultimately it's under the control of players,
since they can replace their local mapping file if it doesn't
produce pleasing results.
Carl Muckenhoupt
features without a potential audience.
I have one concern, though: What happens when the I/O layer gets a
character that isn't part of the local character set? Will it try to
normalize it by substituting the best equivalent (e.g., 'a' for 'a-
ogonek'), or will it do something more anonymous, like the question
marks I was complaining about in Frotz? The former would be
preferable, but it may be an unreasonable thing to expect.
Kevin Bracey
Andrew Plotkin <erky...@eblong.com> wrote:
> Carl Muckenhoupt <ca...@wurb.com> wrote:
> > Z-code didn't originally support more than a subset of Latin-1, but the
> > Z-machine Standards Document 1.0 contains an extension for mapping
> > ZSCII characters 155-251 into regions of Unicode. This extension,
> > however, is not universally supported - in fact, I have yet to find a
> > single interpreter that supports it!
>
> Zip2000 on the Acorn might.
Nay, it does. It also handles Unicode, although only to the extent that
you get to choose an eight-bit system alphabet (code page) for display from
the menu, and any text from the game is mapped to that alphabet as best it
can.
Later versions of RISC OS support Unicode directly, but I haven't updated
Zip 2000 to take advantage of that.
>
> > In short, the only way to display non-Latin-1 characters in Inform is
> > to specifically avoid using Inform's support for them.
>
> Well, you're right. This is a purely hypothetical feature on the
> Z-machine.
It's not hypothetical, I've tested it thoroughly, and it works. I'm not sure
why other interpreters haven't caught up. It really isn't very hard. Zip 2000
uses 16-bit Unicode internally, but that isn't necessary for minimal support.
The Zip 2000 sources are publically available if anyone wants
inspiration.
--
Kevin Bracey, Principal Software Engineer
Pace Micro Technology plc Tel: +44 (0) 1223 518566
645 Newmarket Road Fax: +44 (0) 1223 518526
Cambridge, CB5 8PB, United Kingdom WWW: http://www.pace.co.uk/
Mark Musante - Sun Microsystems
> such as 'a-ogonek' to 'a'. (In fact, you can map a single
> unicode character to a series of local characters; the ASCII 7
> mapping, for example, maps the copyright symbol to the
> string "(c)".)
Unfortunately, (c) is not a recognized legal symbol for copyright,
so you may be doing a mild disservice to your users. "Copyright" and
"Copr." are legal, however.
http://www.templetons.com/brad/copymyths.html
-=- Mark -=-
Matthew T. Russotto
I don't think so. I remember I tried to do it, then realized that
without the fonts I'd never be able to see the results.
mjr_
>mjr_ (mjroberts...@yahoo.com.invalid) wrote:
>> (In fact, you can map a single unicode character to a series
>> of local characters; the ASCII 7 mapping, for example, maps
>> the copyright symbol to the string "(c)".)
>
>Unfortunately, (c) is not a recognized legal symbol for
>copyright, so you may be doing a mild disservice to your
>users. "Copyright" and "Copr." are legal, however.
Interesting point, although I'm not convinced the mapping would
ever be harmful, in that most countries' modern copyright laws
confer copyright automatically when a work is fixed in a medium
whether or not copyright is explicitly claimed anywhere in the
work. In addition, the mapping to "(c)" is merely a matter of
presentation; even if a circle-C symbol were still required to
be attached to a document in order for copyright to be
effective, one could still reasonably claim that the circle-C
symbol is in fact attached to the *document* itself, in that the
unicode character code for a circle-C is stored in the compiled
image file, and the "(c)" is merely an approximation that the
presentation software uses for display purposes. This claim is
further strenghtened by the openness of the mapping system - a
user could, for example, use the mapping file to manipulate the
presentation software into substituting the string "public
domain" for the unicode copyright symbol, and I doubt anyone
would seriously consider this to prevail over the author's claim
of copyright or the author's inclusion of the unicode circle-C
symbol in the original text.
Nonetheless, authors are probably well advised to write out the
word "copyright" in their copyright notices when creating works
in tads 3 or any other electronic medium, since the form of
presentation of a work in an electronic medium is almost never
completely under the control of the author, particularly when it
comes to characters not in basic 7-bit ASCII.
David Kinder
> my code: a "Latin-1" version that would actually be playable on current
> interpreters with the right font, and a Unicode version to provide
> interpreter writers with an incentive to add support for the Unicode
> extension (and a game with which to test that support).
That would definitely be a good idea. We could also do with a proper
test program. I seem to recall a couple of years back there was an
interpreter test program called Terpetude (by Zarf, I think) which perhaps
could be extended.
David
Dan Knapp
>presentation; even if a circle-C symbol were still required to
>be attached to a document in order for copyright to be
>effective, one could still reasonably claim that the circle-C
>symbol is in fact attached to the *document* itself, in that the
>unicode character code for a circle-C is stored in the compiled
>image file, and the "(c)" is merely an approximation that the
>presentation software uses for display purposes. This claim is
I find this interesting... I doubt it's been tested in court,
and it would be nice if it were, but I'm not a laywer; perhaps
it has.
Of course, the optimal solution would be a section of the law
to deal with such cases, but... it's an international treaty;
it would take forever to amend, and probably not come out in a
particularly meaningful way.
>Nonetheless, authors are probably well advised to write out the
>word "copyright" in their copyright notices when creating works
>in tads 3 or any other electronic medium, since the form of
>presentation of a work in an electronic medium is almost never
>completely under the control of the author, particularly when it
>comes to characters not in basic 7-bit ASCII.
Just quoting this to acknowledge that you weren't recommending
a questionable practice. :)
--
| Dan Knapp, Knight of the Random Seed
| http://www.angband.org/~dankna/
| ONES WHO DOES NOT HAVE TRIFORCE CAN'T GO IN.