Scheme v. Unicode draft/thoughts
Tom Lord
Unicode Meets Scheme Meets C
----------------------------
Specifying Unicode string and character support for Pika is an
exercise in language design (because standard Scheme needs to be
modified and extended), Scheme interface design, C interface
design, and implementation design.
Additionally, we would like a subset of the Pika C interface to
characters and strings to be something that _any_ C-compatible
Scheme implementation can provide (including Schemes that support
only ISO8859-* character sets or other subsets of Unicode).
Additionally, we would like the new and modified Scheme interfaces
to strings and characters to be such that other implementors may
choose to provide them and future revisions to the Scheme standard
may incorporate them.
This document provides some background on the problem, specifies the
three interfaces, identifies the (hopefully) portable subset of the
C interface, recommends changes to the Scheme standard, and finally,
describes how all of these will be implemented in Pika.
The reader is presumed to be somewhat familiar with the Unicode
standard.
Contents
* Scheme Characters Meet Unicode
* Scheme Strings Meet Unicode
* The Scheme Interface to Unicode Characters and Strings
* The Portable FFI Interface to Unicode Characters and Strings
* The Pika FFI Interface to Unicode Characters and Strings
Non-contents (unresolved issues)
* Unicode and Ports (what does READ-CHAR mean?)
* Scheme Characters
** Required and Expected Characters
R5RS requires that all members of the character set used to specify
the syntax of Scheme must be representable as Scheme characters.
These include various punctuation marks, space, newline, digits, and
the lowercase letters `a..z'.
The specifications of CHAR-UPCASE, CHAR-DOWNCASE, and
CHAR-ALPHABETIC? strongly imply that `A..Z' must also be present.
The only way to not read this as a requirement is to imagine an
implementation in which, for example, `a' is not an alphabetic
character.
By reasonable expectation, tab and form-feed should also be present.
Implementations are possible which do not provide these but would be
rather exceptional in that regard.
/=== R6RS Recommendation:
R6RS should explicitly require that tab, form-feed, and both `a..z'
and `A..Z' are present, that the letters are alphabetic characters,
and that they are case-mapped as expected.
|=== Portable FFI Recommendation:
A portable FFI should assume that that requirement is in place.
|=== Pika Design:
Pika assumes that that requirement is in place.
\========
Perhaps more controversially: all members of the character set
comprising the first half (U+0000..U+007F) of iso8859-1 should be
representable as Scheme characters. These characters are found in
ASCII, all iso8859-* character sets, and Unicode itself. They are
used almost universally and are likely to remain so indefinitely.
/=== R6RS Recommendation:
R6RS should explicitly and strongly encourage the presence of all
ASCII characters in the CHAR? type.
|=== Portable FFI Recommendation:
A portable FFI should require that all characters representable in
portable C as non-octal-escaped character constants are
representable in Scheme. It should strongly encourage that all
ASCII characters are representable in Scheme.
|=== Pika Design:
The Pika CHAR? type will include representations for all of ASCII.
\========
By somewhat reasonable expectation, there must be at least 256
distinct Scheme characters and INTEGER->CHAR must be defined for all
integers in the range `0..255'. There are many circumstances in
which conversions between octets and characters are desirable and
the requirements of this expectation say that such conversion is
always possible. It is quite possible to imagine implementations in
which this is not the case: in which, for example, a (fully general)
octet stream can not be read and written using READ-CHAR and DISPLAY
(applied to characters). Such an implementation might introduce
non-standard procedures for reading and writing octets and
representing arrays of octets. While such non-standard extensions
may be desirable for independent reasons, I see no good reason not
to define at least a subset of Scheme characters which is mapped to
the set of octet values.
/=== R6RS Recommendation:
R6RS should explicitly and strongly encourage the presence of
at least 256 characters and that INTEGER->CHAR is defined for
the entire range 0..255 (at least).
|=== Portable FFI Recommendation:
A portable FFI should require that the intersection of the
set of integers -128..127 and the set of values representable
as a `char' value are representable in Scheme as CHAR? values.
The preferred 8-bit character representation in the FFI should be
`unsigned char' and the scheme representation (if any) for any
unsigned character `uc' should be the same as that for `(char)uc'.
Note: stating these requirements is greatly simplified if the FFI
simply requires that `char' and `unsigned char' are 8-bit types.
|=== Pika Design:
Pika will satisfy the FFI requirement and require that `char' is an
8-bit integer.
\========
*** Remaining Degrees of Freedom for the CHAR? Type
Scheme implementations consistent with our proposed requirements
so far are likely to partition into four broad classes:
~ Those providing exactly 256 distinct (under CHAR=?) CHAR? values
~ Those providing approximately 2^16 CHAR? values
~ Those providing approximately 2^21 CHAR? values
~ Those providing an infinite set of CHAR? values
/=== Pika Design:
Pika is of the "approximately 2^21 characters" variety.
Specifically, the Pika CHAR? type will in effect be a _superset_ of
the set of Unicode codepoints. Each 21-bit codepoint will
correspond to a Pika character. For each such character, there
will be (2^4-1) (15) additional related characters representing the
basic code point modified by a combination of any of four
"buckybits".
For example, the Unicode codepoint U+0041 can be written in Pika
as:
#\A
and by applying buckybits (shift, meta, alt, hyper) an additional
15 characters can be formed giving the total set of 16 "A"
characters:
#\A
#\S-A #\M-A #\H-A #\A-A
#\S-M-A #\S-H-A #\S-A-A #\M-H-A #\M-A-A #\H-A-A
#\S-M-H-A #\S-M-A-A #\S-H-A-A #\M-H-A-A
#\S-M-H-A-A
\========
** Case Mappings
Strictly speaking, R5RS does not require that the character set
contain any upper or lower case letters. For example, it must
contain `a' but it does not require that CHAR-LOWER-CASE? of `a' is
true. However, in an implementation in which `a' is not lower
case, `a' must also not be alphabetic.
/=== R6RS Recommendation:
R6RS should explicitly require that `a..z' and `A..Z' are
alphabetic and cased in the expected way.
|=== Pika Design:
Pika will satisfy the proposed requirement.
\========
R5RS requires a partial ordering of characters in which upper and
lower case variants of "the same character" are treated as equal.
Most problematically: R5RS requires that every alphabetic character
have both an upper and lower case variant. This is a problem
because Unicode defines abstract characters which, at least
intuitively, are alphabetic -- but which lack such case mappings.
We'll explore the topic further, later, but briefly: it does not
appear that "good Unicode support" and "R5RS requirements for case
mappings" are compatible -- at least not in a simple way.
/=== R6RS Recommendation:
R6RS should drop the requirement from the closing sentence of
section 6.3.4 which says:
In addition, if char is alphabetic, then the result of
char-upcase is upper case and the result of char-downcase is
lower case.
Instead, it should say:
There is no requirement that all alphabetic characters have
an upper and lowercase mappings and no requirement that all
alphabetic characters return true for one of CHAR-UPPER-CASE?
or CHAR-LOWER-CASE?. There is no requirement that if a
character is upper or lower case that it has a case mapping
which is itself a character. However, it is required that
the characters `a..z' are lowercase, `A..Z' are uppercase,
and that CHAR-UPCASE and CHAR-DOWNCASE converts between those
two ranges.
And:
The case mapping procedures and predicates _must_ be
consistent with the case mappings that determine equivalence
of identifiers and symbol names. In many environments
they are usable for case mappings in a broader linguistic
sense but programmers are cautioned that, in general, they are
not appropriate for such uses in portable applications: some
alphabets lack the concept of case entirely; others have the
concept of case but lack a 1:1 mapping between upper and
lowercase characters. Different case mapping procedures
should be used in portable linguistically-oriented
applications.
|=== Pika Design:
Pika will include CHAR? values such as Unicode's eszett
(U+00DF) with the properties that can't satisfy R5RS' requirements
as in the examples:
(char-alphabetic? eszett) => #t
(char-lower-case? eszett) => #t
(char-lower-case? (char-upcase eszett)) => #t
(char=? eszett (char-upcase eszett)) => #t
\========
** Character Ordering and Enumeration
R5RS requires a bijective mapping between characters and a set of
integers (the CHAR->INTEGER and INTEGER->CHAR procedures). Let's
call this mapping the "enumeration of characters".
R5RS requires a total ordering of characters and requires that that
enumeration is isomorphic to the ordering of the enumeration.
R5RS underspecifies the total ordering of characters. It requires
that alphabetic characters `a..z' and (if present) `A..Z' be
(respectively) lexically ordered. It requires that decimal digits
be decimally ordered. It requires that digits either all proceed or
all follow all uppercase letters.
The ordering requirements of R5RS are problematic for Unicode.
While iso8859-* digits `0..9' easily satisfy the requirement,
Unicode defines additional decimal digits which do not.
Intuitively, it seems that either CHAR-NUMERIC? must not behave as
one would like on some Unicode abstract characters or the ordering
requirement will have to change in R6RS.
/=== R6RS Recommendation:
R6RS should modify the requirement from section 6.3.4 which says:
These procedures impose a total ordering on the set of
characters. It is guaranteed that under this ordering:
* The upper case characters are in order. For example,
(char<? #\A #\B) returns #t.
* The lower case characters are in order. For example,
(char<? #\a #\b) returns #t.
* The digits are in order. For example,
(char<? #\0 #\9) returns #t.
* Either all the digits precede all the upper case letters,
or vice versa.
* Either all the digits precede all the lower case letters,
or vice versa.
Instead, it should say:
These procedures impose a total ordering on the set of
characters. It is guaranteed that under this ordering:
* The characters `A..Z' are in order. For example,
(char<? #\A #\B) returns #t.
* The characters `a..z' are in order. For example,
(char<? #\a #\b) returns #t.
* The digits `0..9' are in order. For example,
(char<? #\0 #\9) returns #t.
Programmers are cautioned that the ordering of characters
by these procedures is not expected to have linguistic
significance suitable for portable applications. At the
same time, implementors are strongly encouraged to define the
ordering in such a way that a list of strings, sorted lexically by
the character ordering, is at least likely to be in an order that
is suitable for presentation to users -- even though the sorting
may not be culturally optimal.
|=== Pika Design:
Pika CHAR? values will map to their Unicode codepoint equivalents,
with bucky-bits added added as additional high-order bits.
This satisfies the proposed modified requirements and results in
a situation where, for example, all characters with the META bit
set will sort after all characters with no buckybits set.
\========
** Character Classes
R5RS defines 5 classes of characters by introducing 5 predicates:
char-alphabetic?
char-numeric?
char-whitespace?
char-upper-case?
char-lower-case?
In the context of Unicode, there is a three-way ambiguity between
linguistic categorization of characters, categorizations defined by
the Unicode standard, and categorizations as they apply to Scheme
syntax.
/=== R6RS Recommendation:
Section 6.3.4 should be modified. Instead of saying:
These procedures return #t if their arguments are alphabetic,
numeric, whitespace, upper case, or lower case characters,
respectively, otherwise they return #f. The following remarks,
which are specific to the ASCII character set, are intended only
as a guide: The alphabetic characters are the 52 upper and lower
case letters. The numeric characters are the ten decimal
digits. The whitespace characters are space, tab, line feed, form
feed, and carriage return.
it should say:
These procedures return #t if their arguments are alphabetic,
numeric, whitespace, upper case, or lower case characters,
respectively, otherwise they return #f. These procedures
_must_ be consistent with the procedure READ provided by the
implementation. For example, if a character is
CHAR-ALPHABETIC?, then it must also be suitable for use as the
first character of an identifier.
`a..z' and `A..Z' _must_ be alphabetic and _must_ be respectively
lower and upper case.
#\space, #\tab, and #\formfeed _must_ be CHAR-WHITESPACE?.
`0..9' _must_ be CHAR-NUMERIC?.
No character may cause more than one the procedures
CHAR-ALPHABETIC?, CHAR-NUMERIC? and CHAR-WHITESPACE? to return
#t.
No character may cause more than one of the procedures
CHAR-UPPER-CASE? and CHAR-LOWER-CASE? to return #t.
Programmer's are advised that these procedures are unlikely to be
suitable for linguistic programming in portable code while
implementors are strongly encouraged to define them in ways that
make them a reasonable approximation of their linguistic
counterparts.
|=== Pika Design:
Pika will return #t from CHAR-NUMERIC? _only_ for `0..9'.
Pika will return #t from CHAR-WHITESPACE? _only_ for space, tab,
newline, carriage return, and formfeed.
The set of characters for which Pika will return #t from
CHAR-ALPHABETIC? is undetermined at this time.
Pika will return #t from CHAR-UPPER-CASE? for those which have
the Unicode general category property Lu or Lt; from
CHAR-LOWER-CASE? for those which have the general category
property Ll.
\========
** What _is_ a Character, Anyway
We have observed the need for various changes to the Scheme
standard for the CHAR? type but have done so (deliberately) without
"nailing down" exactly what a character is.
The good reason for preserving this ambiguity is that there are
several reasonable choices for the CHAR? type -- implementations
should not be required to all be the same in this regard.
In particular, these strategies seem to me to be the most viable:
~ A CHAR? value roughly corresponds to a Unicode codepoint.
~ A CHAR? value roughly corresponds to a member of exactly one of
the iso8859-* family of character sets. (An implementation in
this class would not have full Unicode support, of course.
Nevertheless, the iso8859-* character sets are (abstractly, not
by numeric character values) a subset of Unicode -- they are
consistent with Unicode. It would seem gratuitous to rule an
implementation non-standard simply because it doesn't support a
huge character set.)
~ A CHAR? value corresponds to an 8-bit integer, with characters
0..127 corresponding to ASCII characters, and the meaning of
characters 128..255 depending on the host environment (such as a
LOCALE setting). (This is a second class of implementations
without full Unicode support but consistent with Unicode.
Implementations of this class would be consistent with a common
form of internationalization practices used in C programming.)
~ A CHAR? value corresponds to a "combining sequence" of an
arbitrary number of Unicode codepoints. (At this time, such
implementations are mostly theoretic. This is the kind of
implementation Ray Dillenger has been working on and he makes
an interesting case for it.)
Related to these possibilities is an important question of
_syntax_. A Scheme program can contain a lexeme denoting a
character constant:
#\X
and the question is "What can X be in a portable program?"
/=== R6RS Recommendation:
R6RS should explicitly define a _portable_character_set_
containing the characters mentioned earlier: `a..z', `A..Z',
space, formfeed, newline, carriage return, and the punctuation
required by Scheme syntax.
Additionally, R6RS should define an _optional_ syntax for
Unicode codepoints. I propose:
#\U+XXXXX
and in strings:
\U+XXXXX.
where XXXXX is an (arbitrary length) string of hexadecimal digits.
There is some "touchiness" with this optional syntax.
Conceivably, for example, implementations may support Unicode to a
degree but not support codepoints which are not assigned
characters, or not support all possible codepoints, or not permit
strings which are not in one of the (several available)
canonical forms. In my opinion, it is too early to try to
differentiate these various forms of optional conformance. At
this stage in history, the optional syntaxes should be permitted
-- but applications providing them should not be constrained to
support arbitrary codepoint characters or arbitrary Unicode
strings.
|=== Pika Design:
Pika's character set will include all Unicode codepoints with
all combinations of buckybits. Some characters will only be
writable using hexadecimal escapes and/or buckybit prefixes.
\========
* Scheme Strings Meet Unicode
R5RS defines Scheme strings as sequences of scheme characters.
They have a length measured in characters and may be indexed
(to retrieve or set a given character) by character offsets.
R5RS defines two lexical orderings of strings: once by the
definitions of the ordering of characters; again by the definitions
of the case-independent ordering of characters.
There is a strong expectation that strings are "random access"
meaning that any character within them can be accessed or set in
O(1) time. The _intuitive_ model of a Scheme string is a uniform
array of characters.
/=== R6RS Recommendation:
R6RS should strongly encourage implementations to make the
expected-case complexity of STRING-REF and STRING-SET! O(1).
\========
There is an additional problem when designing portable APIs: the
matter of string indexes.
Most of the possible answers to "what is a Scheme character" are
consistent with the view that characters correspond to (possibly a
subset of) Unicode codepoints.
One of the possible answers to that question has the CHAR? type
correspond to a _sequence_ of Unicode code points.
The difference between those two possibilities impacts on string
indexes. An index to the beginning of some substring has one
numeric value if characters are codepoints, another if they are
combined sequences of codepoints.
That difference is, unfortunately, intolerable for portable code.
Consider, for example, a string constant written:
"XYZ"
where X, Y, and Z are arbitrary Unicode substrings. What is the
index of the beginning of substring Y? The question becomes
especially pressing for portable programs that want to encode that
string index as an integer constant, for programs exchanging Scheme
data which includes string indexes with other Scheme
implementations, and for an FFI (in which FFI-using code may wish to
compute and return to Scheme a string index -- or to interpret a
string-index passed to C from Scheme).
So, an ambiguity about string indexes should not be allowed to
stand.
/=== R6RS Recommendation:
While R6RS should not require that CHAR? be a subset of Unicode,
it should specify the semantics of string indexes for strings
which _are_ subsets of Unicode.
Specifically, if a Scheme string consists of nothing but Unicode
codepoints (including substrings which form combining sequences),
string indexes _must_ be Unicode codepoint offsets.
\========
That proposed modification to R6RS presents a (hopefully small)
problem for Ray Dillinger. He would like (for quite plausible
reasons) to have CHAR? values which correspond to a _sequence_ of
Unicode codepoints. While I have some ideas about how to
_partially_ reconcile his ideas with this proposal, I'd like to hear
his thoughts on the matter.
* The Scheme Interface to Unicode Characters and Strings
In the preceding, we have added no new procedures or types to
Scheme -- only modified the requirements for existing procedures.
We have added new syntax for characters and strings.
Certainly, further extensions to Scheme are desirable -- for
example, to provide linguistically sensitive string processing.
I suggest that such extensions are best regarded as a separate topic
-- to be taken up "elsewhere".
* The Portable FFI Interface to Unicode Characters and Strings
This section is written in the style of Pika naming and calling
conventions.
** Basic Types
~ typedef <unspecified unsigned integer type> t_unicode;
An unsigned integer type sufficiently large to hold a Unicode
codepoint.
~ enum uni_encoding_scheme;
Valid values include but are not limited to:
uni_utf8
uni_utf16
uni_utf16be
uni_utf16le
uni_utf32
uni_iso8859_1
...
uni_iso8859_16
uni_ascii
** Character Conversions
~ t_scm_error scm_character_to_codepoint (t_unicode * answer,
t_scm_arena instance,
t_scm_word * chr)
Normally, return (via `*answer') the Unicode codepoint value
corresponding to Scheme value `*chr'. Return 0.
If `*chr' is not a character or is not representable as a
Unicode codepoint, set `*answer' to U+FFFD and return an
error code.
~ t_scm_error scm_character_to_ascii (char * answer,
t_scm_arena instance,
t_scm_word * chr)
Normally, return (via `*answer') the ASCII codepoint value
corresponding to Scheme value `*chr'. Return 0.
If `*chr' is not an ASCII character or is not representable as a
Unicode codepoint, set `*answer' to 0 and return an
error code.
~ t_scm_error scm_codepoint_to_character (t_scm_word * answer,
t_scm_arena instance,
t_unicode codepoint)
Normally, return (via `*answer') the Scheme character
corresponding to `codepoint' which must be in the range
0..(2^21-1). Return 0.
If `codepoint' is not representable as a Scheme character,
set `*answer' to an unspecified Scheme value and return an
error code.
~ void scm_ascii_to_character (t_scm_word * answer,
t_scm_arena instance,
char chr)
Return (via `*answer') the Scheme character corresponding
to `chr' which must be representable as an ASCII character.
** String Conversions
~ t_scm_error scm_extract_string8 (t_uchar * answer,
size_t * answer_len,
enum uni_encoding_scheme enc,
t_scm_arena instance,
t_scm_word * str)
Normally, convert `str' to the indicated encoding (which must be
one of `uni_utf8', `uni_iso8859_*', or `uni_ascii') storing the
result in the memory addressed by `answer' and the number of
bytes stored in `*answer_len'. Return 0.
On input, `*answer_len' should indicate the amount of storage
available at the address `answer'. If there is insuffiencient
memory available, `*answer_len' will be set to the number of bytes
needed and the value `scm_err_too_short' returned.
~ t_scm_error scm_extract_string16 (t_uint16 * answer,
size_t * answer_len,
enum uni_encoding_scheme enc,
t_scm_arena instance,
t_scm_word * str)
Normally, convert `str' to the indicated encoding (which must be
one of `uni_utf16', `uni_utf16be', or `uni_utf16le') storing the
result in the memory addressed by `answer' and the number of
16-bit values stored in `*answer_len'. Return 0.
On input, `*answer_len' should indicate the amount of storage
available at the address `answer' (measured in 16-bit values). If
there is insuffiencient memory available, `*answer_len' will be
set to the number of 16-bit values needed and the value
`scm_err_too_short' returned.
~ t_scm_error scm_extract_string32 (t_uint32 * answer,
size_t * answer_len,
t_scm_arena instance,
t_scm_word * str)
Normally, convert `str' to UTF-32, storing the result in the
memory addressed by `answer' and the number of 32-bit values
stored in `*answer_len'. Return 0.
On input, `*answer_len' should indicate the amount of storage
available at the address `answer' (measured in 32-bit values). If
there is insuffiencient memory available, `*answer_len' will be
set to the number of 32-bit values needed and the value
`scm_err_too_short' returned.
~ t_scm_error scm_make_string8_n (t_scm_word uchar * answer,
t_scm_arena instance,
t_uchar * str,
enum uni_encoding_scheme enc,
size_t str_len)
Convert `str' of length `str_len' and in encoding to `enc' to
a Scheme value (returned in `*answer') if possible, and return 0.
Otherwise, return an error code.
~ t_scm_error scm_make_string16_n (t_scm_word uchar * answer,
t_scm_arena instance,
t_uint16 * str,
enum uni_encoding_scheme enc,
size_t str_len)
Convert `str' of length `str_len' and in encoding to `enc' to
a Scheme value (returned in `*answer') if possible, and return 0.
Otherwise, return an error code.
~ t_scm_error scm_make_string32_n (t_scm_word uchar * answer,
t_scm_arena instance,
t_uint32 * str,
enum uni_encoding_scheme enc,
size_t str_len)
Convert `str' of length `str_len' and in encoding to `enc' to
a Scheme value (returned in `*answer') if possible, and return 0.
Otherwise, return an error code.
** Other FFI Functions
Various standard Scheme procedures (e.g., `STRING-REF') ought to
be present in the FFI as well. Their mappings into C are
straightforward given the proposed requirement that string indexes
refer to codepoint offsets.
* The Pika FFI Interface to Unicode Characters and Strings
The standard Scheme string procedures map into the native Pika FFI
in the expected way (their C function prototypes and semantics can
be inferred in a trivial way from their standard Scheme
specifications). As such, they are not specified here.
** Pika FFI Access to String Data
(Note: In earlier discussions with some people I had suggested
that Pika strings might be represented by a tree structure. I
have since decided that that should not be the case: I want Pika
to live up to the expectation that STRING? values are represented
as arrays. The tree-structured-string-like type will be present
in Pika, but it will be distinct from STRING?.)
There is a touchy design issue regarding strings. On the one hand,
for efficiency's sake, C code needs fairly direct access (for both
reading and writing) to string data. On the other hand, that
necessarily means exposing to C the otherwise internal details of
value representation and of addressing the possibility of a
multi-threaded Pika.
~ t_scm_error scm_lock_string_data (t_udstr * answer,
t_scm_arena instance,
t_scm_word * str)
Obtain the `t_udstr' (see libhackerlab) corresponding to `str'.
Return 0. If `*str' is not a string, set `*answer' to 0 and
return an error code.
A string obtained by this call must be released by
`scm_release_string_data' (see below).
It is critical that between calls to `scm_lock_string_data' and
`scm_release_string_data' no other calls to Pika FFI functions
be made, and that the number of instructions between those calls
is sharply bounded.
~ t_scm_error scm_lock_string_data2 (t_udstr * answer1,
t_udstr * answer2,
t_scm_arena instance,
t_scm_word * str1
t_scm_word * str2)
Like `scm_lock_string_data' but return the `t_udstr's for two
strings instead of one.
~ t_scm_error scm_release_string_data (t_scm_arena instance
t_scm_word * str)
Release a string locked by `scm_lock_string_data'.
~ t_scm_error scm_release_string_data (t_scm_arena instance
t_scm_word * str1
t_scm_word * str2)
Release strings locked by `scm_lock_string_data2'.
** Pika String Internals
libhackerlab contains a string-like type, `t_udstr' defined
(internally) as a pointer to a structure of the form:
struct udstr_handle
{
enum uni_encoding_scheme enc;
size_t length;
uni_string data;
alloc_limits limits;
int refs;
};
The intention here is to represent a string in an aribitray
Unicode encoding, with an explicitly recorded length measured in
coding units.
libhackerlab contains a (currently small but growing) number of
"string primitives" for operating on `t_udstr' values. For example,
one can concatenate two strings without regard to what encoding
scheme each is in.
In Pika, strings shall be represented as `t_udstr' values, pointed
to by an scm_vtable object.
Further restrictions apply, however:
Pika strings will have the property that:
~ any string including a character in the range U+10000..U+1FFFFF
will be stored in a t_udstr using the uni_utf32 encoding.
~ any other string including a character in the range U+0100..U+FFFF
will be stored in a t_udstr using the uni_utf16 encoding.
~ all remaining strings will be stored in a t_udstr using
the uni_iso8859_1 encoding.
In that way, the length of the t_udstr (measured in encoding units)
will always be equal to the length of the Scheme string (measured
in codepoints). Nearly all Scheme string operations involving
string indexes can find the referenced characters in O(1) time.
The primary exception is STRING-SET! if the character being stored
requires the string it is being stored into to change encoding.
At the same time, Scheme strings will have a space-efficient
representation.
Alex Shinn
>
> Unicode Meets Scheme Meets C
> ----------------------------
Thank you for this thoughtful draft about making Scheme more Unicode
friendly. I certainly hope R6RS takes these ideas into account. I have
a few comments below.
> By reasonable expectation, tab and form-feed should also be present.
> Implementations are possible which do not provide these but would be
> rather exceptional in that regard.
In an ideal world these probably don't belong as characters, since they
are formatting rather than linguistic elements. However, because of
backwards compatibility we may never be rid of them, and in either case
it's certainly the more important goal for Scheme to tackle Unicode
properly first, and worry about possible reform at a future date.
> Specifically, the Pika CHAR? type will in effect be a _superset_ of
> the set of Unicode codepoints. Each 21-bit codepoint will
> correspond to a Pika character. For each such character, there
> will be (2^4-1) (15) additional related characters representing the
> basic code point modified by a combination of any of four
> "buckybits".
I realize this is your own idea for Pika, and not an R6RS proposal, but
for the record I very strongly disagree on using buckybits. They are
mixing text data with "keystroke" data, which is more properly handled
by an event input loop. The event loop itself is only slightly more
complicated if you separate modifiers from the keys, and this is
required in a full GUI these days anyway. You can't program many video
games without handling holding down the shift key as a separate,
independent event. You also need this separation for fully chorded
input, such as described in _The_Humane_Interface_ by Raskin.
Buckybits also complicate all text-handling routines, which have to
account for this additional data in individual characters. Granted
characters become more complicated in a full Unicode environment, but
this is no reason to make them even more complicated.
Note you can still provide (define-key ...) type procedures that use
modifiers, and that even allow string notation such as "\M-C-x" without
requiring the chars themselves to contain this meta-info.
> Most problematically: R5RS requires that every alphabetic character
> have both an upper and lower case variant. This is a problem
> because Unicode defines abstract characters which, at least
> intuitively, are alphabetic -- but which lack such case mappings.
Note the inverse doesn't hold either. Unicode defines characters that
do not have the alphabetic property (such as the circled latin
characters), yet for which it does define case mappings.
> The case mapping procedures and predicates _must_ be
> consistent with the case mappings that determine equivalence
> of identifiers and symbol names.
You can't do this without taking locale into account. For example, in
Turkish the lowercase of I is not i. It would be very bad if the same
script behaved differently for a user in one locale from a user in a
different locale.
Note that this library:
http://synthcode.com/scheme/scheme-unicode-0.1.tar.bz2
has full definitions of Unicode character properties as SRFI-14
char-sets, and also provides the following case mapping procedures:
;; Return either a single char or string if the mapping does not fit
;; in a char.
(char-upcase* c)
(char-downcase* c)
(char-titlecase* c)
;; Returns appropriately case mapped string. SRC may be a string or
;; input port. LOCALE is an optional locale string as used in a Unix
;; env (e.g. "en" or "ja_JP.eucJP"). Handles all special cases,
;; including eszett, trailing Greek sigmas, and Turkish & Azeri I's..
(upcase src [locale])
(downcase src [locale])
(titlecase src [locale])
It's Gauche specific. You can use a Unicode terminal to play with this,
or use run-scheme in Emacs and add the following to your .emacs:
(add-hook 'inferior-scheme-mode-hook
(lambda ()
(set-process-coding-system (get-process "scheme") 'utf-8 'utf-8))
t)
> it should say:
>
> These procedures return #t if their arguments are alphabetic,
> numeric, whitespace, upper case, or lower case characters,
> respectively, otherwise they return #f. These procedures
> _must_ be consistent with the procedure READ provided by the
> implementation. For example, if a character is
> CHAR-ALPHABETIC?, then it must also be suitable for use as the
> first character of an identifier.
How about char-numeric? and extended numeric characters? Should read &
string->number allow these?
> In particular, these strategies seem to me to be the most viable:
[snip]
You left out CJKV and quite a number of other encodings that I wouldn't
really consider "inviable" :)
> Additionally, R6RS should define an _optional_ syntax for
> Unicode codepoints. I propose:
>
> #\U+XXXXX
Do we need the +?
> and in strings:
>
> \U+XXXXX.
>
> where XXXXX is an (arbitrary length) string of hexadecimal digits.
How do you write a Unicode codepoint followed by a hexadecimal digit?
You need to either require a \ escape for the trailing hex digit, or use
a fixed size for the initial string escape. Gauche uses 2 fixed sizes:
\uXXXX for 4-byte Unicode escapes (most chars)
\UXXXXXXXX for 8-byte Unicode escapes
> R6RS should strongly encourage implementations to make the
> expected-case complexity of STRING-REF and STRING-SET! O(1).
This is biased against both rope-like string implementations, and
implementations who use variable width character encodings. However, by
specifically saying STRING-REF and STRING-SET! may be O(n), you
encourage programmers to use more general string routines such as string
ports, string-fold, string-map, etc., which can easily be guaranteed to
be efficient for any string representation. The result is more elegant
and more portable code. I write huge amounts of text processing code
(including a full text editor), and *never* need to use either of
STRING-REF or STRING-SET!.
--
Alex
Joe Marshall
> /=== Pika Design:
>
> Pika is of the "approximately 2^21 characters" variety.
>
> Specifically, the Pika CHAR? type will in effect be a _superset_ of
> the set of Unicode codepoints. Each 21-bit codepoint will
> correspond to a Pika character. For each such character, there
> will be (2^4-1) (15) additional related characters representing the
> basic code point modified by a combination of any of four
> "buckybits".
>
> For example, the Unicode codepoint U+0041 can be written in Pika
> as:
>
> #\A
>
> and by applying buckybits (shift, meta, alt, hyper) an additional
> 15 characters can be formed giving the total set of 16 "A"
> characters:
>
> #\A
> #\S-A #\M-A #\H-A #\A-A
> #\S-M-A #\S-H-A #\S-A-A #\M-H-A #\M-A-A #\H-A-A
> #\S-M-H-A #\S-M-A-A #\S-H-A-A #\M-H-A-A
> #\S-M-H-A-A
May I suggest that this is a bad idea?
What is #\S-a, and how does it differ from #\A ?
I think you need to distinguish keystrokes from characters.
Tom Lord
Tom> Unicode Meets Scheme Meets C
Tom> ----------------------------
Alex> Thank you for this thoughtful draft about making Scheme more
Alex> Unicode friendly. I certainly hope R6RS takes these ideas into
Alex> account.
Thanks. Me too. Thanks for your reply. The Turkish issue is
particularly fun.
Tom> Specifically, the Pika CHAR? type will in effect be a _superset_
Tom> of the set of Unicode codepoints. [because characters can have
Tom> buckybits]
Alex> I realize this is your own idea for Pika, and not an R6RS proposal, but
Alex> for the record I very strongly disagree on using buckybits. They are
Alex> mixing text data with "keystroke" data, which is more properly handled
Alex> by an event input loop. [....]
Alex> Buckybits also complicate all text-handling routines, which have
Alex> to account for this additional data in individual characters.
Alex> Granted characters become more complicated in a full Unicode
Alex> environment, but this is no reason to make them even more
Alex> complicated.
I have to answer in two parts: one about Pika and the other about the
strings draft.
In the strings draft, I think it's useful to talk about buckybits just
because they ensure that the case-study implementation of the proposed
requirements is one that extends CHAR? and STRING? beyond what will be
required and one which provides some non-Unicode characters.
In Pika, I happen to disagree with you (and Joe Marshall) about
buckybit usefulness but in my view this is an area of pretty
subjective trade-offs -- I won't try to convince you at this time. If
you like, you could just think of them as a flashy retro feature added
just for flair: the Scheme equivalent (if there could be one) of
tailfins on a car.
Tom> [proposed R6RS requirement:] The case mapping procedures and
Tom> predicates _must_ be consistent with the case mappings that
Tom> determine equivalence of identifiers and symbol names.
Alex> You can't do this without taking locale into account. For
Alex> example, in Turkish the lowercase of I is not i. It would be
Alex> very bad if the same script behaved differently for a user in
Alex> one locale from a user in a different locale.
(Incidentally, it would be helpful to me if, when you cite issues like
this and when you can, please give unicode codepoint numbers and other
references if you have them. For example, it took me a little while
to figure out that you mean that Turkish "LATIN CAPITAL LETTER I"
(U+0049) has lowercase "LATIN SMALL LETTER DOTLESS I" (U+0131) rather
than U+0049 ("LATIN SMALL LETTER I"). It would also have been helpful
to point to the (superseded but handy) UTR21 ("Unicode Technical
Report #21: Case Mappings") or to the chapter of the 4.0 spec that
replaces it. (And people should generally be aware that such
references can be looked up at http://www.unicode.org))
That's an interesting problem that you mention. The Turkish case
conversion appears to be the (current?) sole exception among the
ASCII-equivalent `a..z' and `A..Z'.
The problem, if there is one, with the draft, to spell it out, is that
on the one hand it proposes that R6RS say:
The case mapping procedures and predicates _must_ be consistent
with the case mappings that determine equivalence of identifiers
and symbol names.
and on the other hand proposes that it also say:
it is required that the characters `a..z' are lowercase, `A..Z' are
uppercase, and that CHAR-UPCASE and CHAR-DOWNCASE converts between
those two ranges.
If I momentarilly ignore the Turkish issue, these seem to me to be
essential requirements. Without the former, it is impossible to write
portable Scheme programs which manipulate source texts. Without the
latter, it is impossible to write portable Scheme programs unless they
are written with the assumption that identifiers are _case_sensative_.
For example, is this Scheme fragment portable?:
(let ((i 3)) I) => 3
or this one?:
(list 'a 'b 'c) => (a b c)
or what about a procedure like:
(define (pretty-print-name-for-identifier s1)
(string-downcase s1))
and if your portable library defines:
FROB-LIST
may mine call:
frob-list
?
I think there is some wiggle room here. I see nothing that would
prohibit:
(char-upcase #\i) => #\I
(char-upcase dotless-i) => #\I
(char-downcase #\I) => #\i
(char-downcase dotless-i) => #\i
which, though linguistically peculiar for Turkish users, would solve
all of the problems.
Tom> it should say:
Tom> These procedures return #t if their arguments are
Tom> alphabetic, numeric, whitespace, upper case, or lower case
Tom> characters, respectively, otherwise they return #f. These
Tom> procedures _must_ be consistent with the procedure READ
Tom> provided by the implementation. For example, if a character
Tom> is CHAR-ALPHABETIC?, then it must also be suitable for use
Tom> as the first character of an identifier.
Alex> How about char-numeric? and extended numeric characters? Should
Alex> read & string->number allow these?
I think the requirements I've proposed add up to this:
0..9 (the ASCII-equivalents) _must_ be numeric and are part of
the portable character set
portable Scheme programs _must_ use those characters and no others
to write decimal digits in numbers
implementations _may_ make other characters count as numeric?
and _may_ extend the syntax of numbers beyond the portable
requirements
that a number is CHAR-NUMERIC? does _not_ imply that it is part of
the syntax of numbers. It does, however, have implications for
use of that character in identifiers.
Tom> In particular, these strategies seem to me to be the most viable:
Alex> You left out CJKV and quite a number of other encodings that I
Alex> wouldn't really consider "inviable" :)
Actually, I don't think I did though I didn't make it all that clear
where I think they fit in. I think they will most often fit in to the
"a CHAR? value is _roughly_ a Unicode codepoint" class. (_Very_
roughly in some cases, of course -- but not in any way that
contradicts the proposed requirements, I think.)
Tom> Additionally, R6RS should define an _optional_ syntax for
Tom> Unicode codepoints. I propose:
Tom> #\U+XXXXX
Alex> Do we need the +?
Happens to be what Pika uses just because it looks more like what you
see in Unicode documentation. I think it's a good choice but I don't
feel too strongly about it one way or the other.
Tom> and in strings:
Tom> \U+XXXXX.
Tom> where XXXXX is an (arbitrary length) string of hexadecimal
Tom> digits.
Alex> How do you write a Unicode codepoint followed by a hexadecimal
Alex> digit?
That's why the "." is there. ";" might be a better choice.
Alex> You need to either require a \ escape for the trailing hex
Alex> digit, or use a fixed size for the initial string escape.
Alex> Gauche uses 2 fixed sizes:
Alex> \uXXXX for 4-byte Unicode escapes (most chars)
Alex> \UXXXXXXXX for 8-byte Unicode escapes
I'd like all the variations of STRING-APPEND to work correctly where
the strings being appended are Scheme string constant syntax.
Guache's approach doesn't have that property.
Tom> R6RS should strongly encourage implementations to make the
Tom> expected-case complexity of STRING-REF and STRING-SET! O(1).
Alex> This is biased against both rope-like string implementations,
Alex> and implementations who use variable width character encodings.
I disagree. Rope-like implementations can claim conformance with this
_recommendation_ about _expected_case_ complexity in two ways:
1) On small->medium-sized strings, they should conform for free.
In that regard, they'll being doing better than, say, a naively
UTF-8-based implementation which will conform on small but not
medium-sized strings.
2) They can use self-balancing trees for their ropes.
In that regard, they'll conform even on huge strings at least for
programs that display a degree of locality in their string
accesses.
-t
Tom Lord
Joe> [re buckybits in characters]
Joe> May I suggest that this is a bad idea?
I think you just did :-)
Joe> What is #\S-a, and how does it differ from #\A ?
Joe> I think you need to distinguish keystrokes from characters.
The difference between #\S-a and #\A depends on how you use them,
of course.
Just because you can't type #\S-a on your VT-100 doesn't mean it
shouldn't exist.
One way to look at buckybits is, as I said to Alex, just to think of
them as a retro feature. They're a way of saying "Hey, I've got some
extra bits here in these immediate values. Let's give them
(suggestive) names."
-t
Joe Marshall
> One way to look at buckybits is, as I said to Alex, just to think of
> them as a retro feature.
You need Top, Front, Super, Hyper, and Greek, then, too.
> Just because you can't type #\S-a on your VT-100 doesn't mean it
> shouldn't exist.
I *can* type the A key while holding down the SHIFT key on a VT-100.
Is that #\A, #\S-a, or #\S-A? What if I *really* was holding down
caps lock? Does the control bit map things onto control codes?
Tom Lord
Tom> Just because you can't type #\S-a on your VT-100 doesn't mean it
Tom> shouldn't exist.
Joe> I *can* type the A key while holding down the SHIFT key on a
Joe> VT-100. Is that #\A, #\S-a, or #\S-A? What if I *really* was
Joe> holding down caps lock? Does the control bit map things onto
Joe> control codes?
Flippent answers aside:
Holding down shift while pressing the a key on your vt100 will xmit
a 0x41 to the host.
Assuming that this goes directly to your application without any
other translation, then most likely:
A binding for "A" has highest priority.
It _might_ be useful, if "A" is unbound, to fallback to the binding
for #\S-a. It depends a little on the application and on what the
other input methods commonly used with that application are. I can
imagine wanting this feature if I am automatically computing a larger
set of bindings from a smaller one.
I doubt that the binding for #\S-A should be used or that users should
typically want to bind that chord. On the other hand, I can imagine
it being used to distinguish _some_ instances of having caps-lock on,
holding shift, and typing A (other instances will just xmit "a" or "A"
in that case).
I notice that the version of Emacs I use simply barfs at characters
that are ambiguous in that way when they are passed to keymap-related
functions:
(global-set-key "\S-a" 'frob) => <error>
but still recognizes them as character constants:
?\S-a => 33554529
This seems to me to be a perfectly consistent and reasonable
approach. It makes the distinction you want me to make between
keystrokes and characters -- it just doesn't happen to make it in the
type system.
In general, there's a pristine Platonic ideal input device that no
actual input system implements. And at the same time, there's no
abstraction of input that captures really well all the actually
available input systems.
The approach that I've seen produce the most practical results is one
in which you build the keymap system data structures around the
platonic ideal, and then you make sure that there's enough hooks and
exceptions in RIUL (read-key, interpret, update loop) that you can
make very flexible defaults with a few very common non-default
alternatives, and customize for any outliers.
Hey, at least I didn't attach _fonts_ to characters.
-t
Alex Shinn
>
> In Pika, I happen to disagree with you (and Joe Marshall) about
> buckybit usefulness but in my view this is an area of pretty
> subjective trade-offs -- I won't try to convince you at this time. If
> you like, you could just think of them as a flashy retro feature added
> just for flair: the Scheme equivalent (if there could be one) of
> tailfins on a car.
But it's not just a feature - it also makes both text handling and event
handling more difficult in the general case. The only advantage is
slightly easier event handling in VT100 terminals (retro indeed), and if
you abstract your event library properly normal programs will never see
a difference anyway (speaking from experience, I wrote an event handling
library that transparently works with both VT100 and SDL backends,
though of course you can't express all chords in VT100).
I won't try to convince you either, but I also don't like the apparent
trend in Schemes to use buckeybits. I know only two new forthcoming
Scheme implementations that intend to support Unicode (Pika and Bear's)
and they both seem to have taken the attitude "our implementation can
handle it, so let's throw in buckeybits too" which could become
semi-standard (it's bad enough that mit-scheme uses these).
So to all forthcoming Schemes I reiterate, just because you can doesn't
mean you should, and do you really want to trade elegance, flexibility &
a clean path to future GUI's in exchange for a few quick hacks on old
VT100 terminals?
> I think there is some wiggle room here. I see nothing that would
> prohibit:
>
> (char-upcase #\i) => #\I
> (char-upcase dotless-i) => #\I
> (char-downcase #\I) => #\i
> (char-downcase dotless-i) => #\i
>
> which, though linguistically peculiar for Turkish users, would solve
> all of the problems.
You mean linguistically wrong for Turkish users. An analogy would be
suggesting the uppercase of e was E with a French accent ague (U+00C9),
and that the following code:
(let ((e 3)) E)
was an error, or that a case-insensitive keyword search for "english"
would fail to match on "English".
Of course, this depends on how the case-{mapping/folding} procedures are
intended to work wrt locale. If they require an explicit locale
argument, the standard can say that no locale is passed in source code
case-mapping (as in your examples above), which for Turkish & Azeri
users would be incorrect but livable since it's just the source code and
they can still pass the locale explicitly in normal user code. If the
case-{mapping/folding} procedures use some concept like (current-locale)
implicitly, then the standard could say that when reading Scheme source
code the (current-locale) is set to a null/neutral value.
Lithuanian is similar to Turkish & Azeri but if I has other accent marks
the lowercase i retains its dot. Currently these three locales are the
only locales that have special case-mappings in Unicode, and only with
respect to the dotted and dotless i variations.
This opens up one more alternative which is to map all i variations in
source code to the same value.
> Tom> and in strings:
>
> Tom> \U+XXXXX.
>
> Tom> where XXXXX is an (arbitrary length) string of hexadecimal
> Tom> digits.
>
> Alex> How do you write a Unicode codepoint followed by a hexadecimal
> Alex> digit?
>
> That's why the "." is there. ";" might be a better choice.
Ah, sorry, I didn't register the ".". A possibly more readable
alternative would be
\u{XXXX}
which is similar to what I've seen many C programs using:
\x{XXXX}
> Tom> R6RS should strongly encourage implementations to make the
> Tom> expected-case complexity of STRING-REF and STRING-SET! O(1).
>
> Alex> This is biased against both rope-like string implementations,
> Alex> and implementations who use variable width character encodings.
>
> I disagree. Rope-like implementations can claim conformance with this
> _recommendation_ about _expected_case_ complexity in two ways:
>
> 1) On small->medium-sized strings, they should conform for free.
Sure, and on 32-bit systems all operations are O(1), where the constants
involved may be up to O(2^32) :)
> 2) They can use self-balancing trees for their ropes.
>
> In that regard, they'll conform even on huge strings at least for
> programs that display a degree of locality in their string
> accesses.
Random access in a self-balancing tree is O(log(n)). If you want to
loosen the requirement to be O(1) given locality of reference then that
should be the R6RS recommendation. However this is still an
implementation restriction, as it implicitly requires strings to contain
an explicit "last address" index+pointer. This is adds memory and speed
overhead to all strings, and effectively prevents you from allowing
strings to be implicitly composed of shared substrings, since then every
chord in the string would have to be updated with every access!
An approach that is both more efficient and more flexible is to provide
an explicit, external string-pointer type. This is especially useful
because you can keep multiple pointers to the same string when needed
(useful for backtracking algorithms like regexp matching). Although my
text-buffer implementation makes no use of STRING-REF or STRING-SET!, it
has to rely on string-pointers for the Boyer-Moore search
implementation.
--
Alex
Alex Shinn
but that's really a general R6RS wishlist issue so it belongs here.
I'll sum up and encourage discussion to continue here.
The problem is that R5RS defines the Scheme reader to be
case-insensitive[1], and provides procedures for handling this, but the
meaning of case-insensitive changes in an extended character set
environment[2]. Specifically:
* Case-mapping is complicated on single chars, limiting the utility of
char-upcase, etc.
+ German eszett and other characters upcase to 2 characters
+ some mappings such as Greek sigma are context sensitive
* Case-mapping in general is a locale-sensitive operation. In certain
locales, the uppercase of Latin "i" is not "I".
Solutions to the general Scheme API include:
(1) Drop all case procedures from R6RS.
(2) Leave case procedures as they are, and define them only in terms
of ASCII characters, because this is very useful for programming
languages and many standards and protocols. Libraries could
provide optional full linguistic case-mapping procedures (which
would inherit the issues in (3) below).
(3) Extend case procedures generally to support Unicode. They must
take a locale parameter (which could optionally default to a
(current-locale)), and handle locale and all other special cases.
For char procedures we have the following options:
3a) drop char mapping procedures, only provide string-upcase, etc.
3b) allow char mapping procedures, but leave them incorrect for
some inputs, returning a single char
3c) allow char mapping procedures, but have them return strings
3d) allow char mapping procedures, but have them return chars
normally, and strings when there is overflow or underflow (0 or
2+ chars result from the mapping)
In any of the cases, string mapping cannot be defined in terms of
successive char mapping.
Then we have to decide how the Scheme reader itself will behave, with
the following options:
(i) R6RS is case-sensitive
(ii) R6RS is case-insensitive in the limited ASCII sense only
(iii) R6RS is case-insensitive in a general Unicode sense. This
implies that the linguistic folding be done in a locale-neutral
environment, making just a few locales seem unnatural.
(iv) R6RS uses an extended folding of symbols, which handles not only
case but some or all accent marks. Likely divisions are:
(ivA) Full case-folding plus Latin "i" in all cases with and
without a dot folds to the same character.
(ivB) Above plus all accent marks are folded out.
(ivC) Above plus half-width/full-width characters are folded to
the same value.
(ivD) All symbols fold to one of I, K or S :)
Note (ii) imples (2) or (3), and (iii) and up imply (2) with *required*
linguistic libraries or (3).
Because of a general desire to keep the Scheme core small, and because
linguistic case operations are of necessity much more complex (and
slower), a prudent choice would seem to be (2) & (ii). This is what
Gauche does currently (case-insensitivity is off by default but can be
enabled with a command-line option).
The next issue is that if the Scheme allows multiple ways to encode the
same character (e.g. with combining plus pre-combined characters) do we
ignore this, assume all source is in a normalized form, or normalize
when we read it in? By the same arguments it seems best to ignore this.
--
Alex
Footnotes:
[1] Symbols are actually case-sensitive but source code is implicitly
folded, see Lexical Conventions in R5RS.
[2] R6RS doesn't have to be Unicode-centric in general, though for
portability certain things such as escaped character syntax
(#\U+XXXX) and possibly conversion routines may be defined in terms
of Unicode. Arguably portable Scheme source should be defined as
utf-8.
Ray Dillinger
>
> I won't try to convince you either, but I also don't like the apparent
> trend in Schemes to use buckeybits. I know only two new forthcoming
> Scheme implementations that intend to support Unicode (Pika and Bear's)
> and they both seem to have taken the attitude "our implementation can
> handle it, so let's throw in buckeybits too" which could become
> semi-standard (it's bad enough that mit-scheme uses these).
Eh. I'm not terribly attached to them. I figured that since there are
some systems that use them, I should probably at least be able to represent
them.
But I'm not using them in any interface or to handle keystrokes, and don't
intend to. They're just combining codepoints in the private-use area, and
the codepoints can be mapped to something else if the application requires
it.
What I am not doing is using them to control the "upper bits" between
the 21-bit unicode size and the 32-bit representation size. That would
require 11 buckybits, which is, um, amusing to think about but silly.
(control, meta, alter, super, hyper, top, front, greek, hebrew, cyrillic ...
dang, that's all the shift keys I've ever seen on keyboards and I'm still
two short. I suppose we could distinguish right-alt and right-control if
we wanted to round out the set.)
> So to all forthcoming Schemes I reiterate, just because you can doesn't
> mean you should, and do you really want to trade elegance, flexibility &
> a clean path to future GUI's in exchange for a few quick hacks on old
> VT100 terminals?
I'm not using them in a hacky way - I just decided to be able to
represent them. Definitely in the "chrome" category, not the "how
it works" category, and nothing depends on them.
> Of course, this depends on how the case-{mapping/folding} procedures are
> intended to work wrt locale. If they require an explicit locale
> argument, the standard can say that no locale is passed in source code
> case-mapping (as in your examples above), which for Turkish & Azeri
> users would be incorrect but livable since it's just the source code and
> they can still pass the locale explicitly in normal user code.
Does this require values of type "locale" be added to scheme, or are
we content to represent locales as strings?
Bear
Marcin 'Qrczak' Kowalczyk
> Solutions to the general Scheme API include:
>
> (1) Drop all case procedures from R6RS.
>
> (2) Leave case procedures as they are, and define them only in terms
> of ASCII characters, [...]
> (3) Extend case procedures generally to support Unicode. They must
> take a locale parameter [...]
(4) Use the default locale-independent casing specified by Unicode,
and leave locale-dependent casing and other character handling stuff
(e.g. language-dependent ordering) for a separate library.
--
__("< Marcin Kowalczyk
\__/ qrc...@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/
Joe Marshall
> A lot of the discussion on case-mapping was done on the srfi-50 list,
> but that's really a general R6RS wishlist issue so it belongs here.
> I'll sum up and encourage discussion to continue here.
>
> The problem is that R5RS defines the Scheme reader to be
> case-insensitive[1],
One reason that Scheme is case insensitive is because case changes
preserve meaning in English: a rose is A ROSE is a RoSe. Since case
mapping is locale sensitive, a different sort of canonicalization
ought to be used.
Internally, *any* string should be convertable to a symbol, so some
escape sequence needs to exist for non-standard characters. It is
reasonable to prohibit certain characters from symbols in unescaped
form if including them would cause ambiguity. It is also reasonable
to be non-universal, for instance, although Turkic languages do not
map the Latin "i" into the Latin "I", it still may be reasonable to
perform the mapping. This may mean extra work for Turkish programmers
who would then be required to use #LATIN CAPITAL LETTER I WITH DOT
ABOVE explicitly (leadIng to code sImIlar to thIs), but code is not
prose.
> Solutions to the general Scheme API include:
> (3) Extend case procedures generally to support Unicode.
Don't forget option 4: provide *everything*.
char-fold-case-simple char->char
char-fold-case-full char->(char|string)
char-remove-diacritic
char-fold-width
char-fold-superscript
etc. etc.
> Then we have to decide how the Scheme reader itself will behave, with
> the following options:
>
> (iv) R6RS uses an extended folding of symbols, which handles not only
> case but some or all accent marks. Likely divisions are:
>
> (ivA) Full case-folding plus Latin "i" in all cases with and
> without a dot folds to the same character.
>
> (ivB) Above plus all accent marks are folded out.
>
> (ivC) Above plus half-width/full-width characters are folded to
> the same value.
These all seem reasonable.
> The next issue is that if the Scheme allows multiple ways to encode the
> same character (e.g. with combining plus pre-combined characters) do we
> ignore this, assume all source is in a normalized form, or normalize
> when we read it in? By the same arguments it seems best to ignore this.
It would be bad to normalize arbitrary input (you'd want strings to be
left alone), but more than reasonable to normalize identifiers and
such after some parsing has been done.
Feuer
> (control, meta, alter, super, hyper, top, front, greek, hebrew, cyrillic ...
> dang, that's all the shift keys I've ever seen on keyboards and I'm still
> two short. I suppose we could distinguish right-alt and right-control if
> we wanted to round out the set.)
Open-apple (anyone know the real name of this?), Option (on Apples)
Fn (Dell laptops)
> Does this require values of type "locale" be added to scheme, or are
> we content to represent locales as strings?
I should hope a new type.
David
Bradd W. Szonye
> On Fri, 23 Jan 2004 15:14:27 +0900, Alex Shinn wrote:
>
>> Solutions to the general Scheme API include:
>>
>> (1) Drop all case procedures from R6RS.
>>
>> (2) Leave case procedures as they are, and define them only in terms
>> of ASCII characters, [...]
>>
>> (3) Extend case procedures generally to support Unicode. They must
>> take a locale parameter [...]
> (4) Use the default locale-independent casing specified by Unicode,
> and leave locale-dependent casing and other character handling stuff
> (e.g. language-dependent ordering) for a separate library.
I suggested something like this on the SRFI-50 mailing list, with the
added suggestion that readers might want to permit "bootstrapping" of
other encodings and languages similar to the way XML does it: All files
must begin in one of a few, easily-recognizable encodings, but you can
include a "metadata" tag that changes the encoding or language. For
example, #,(locale UTF-8 EN-US (...)) would instruct the reader to
interpret ... according to US English conventions, in the UTF-8
encoding. This doesn't allow all possible encodings, but it could help
to make source code and program data more portable.
Unfortunately, I don't have enough experience to say how well the XML
approach works in practice. Anybody have comments regarding the prior
art?
--
Bradd W. Szonye
http://www.szonye.com/bradd
Alex Shinn
>
> Does this require values of type "locale" be added to scheme, or are
> we content to represent locales as strings?
Or lists? Locale is generally broken up into language, region, and
sometimes output CES. So it might be useful to picture a locale as:
(<LANG> [<REGION> [<ENCODING>]])
for example
("en")
("fr" "FR")
("tr" "CX" "BIG5")
However, it can also be useful in many cases to allow multiple locales,
as a fallback when a given translation is not available. Even the most
"professional" commercial software falls behind in its translations, and
in the broader world of the net you can come across any number of sites
which may or may not support your preferred locale. Thus web browsers
use the Accept-Language header which can be a list multiple of locales.
Likewise for my own gettext library I extended it to allow a list as the
locale.
However, Scheme does have prior work in this area in the form of
SRFI-29, which breaks locale into separate procedures:
current-language
current-country
effectively preventing you from having multiple locales. I'd like to
see Scheme in the future move to
current-locale
which at least leaves the option of multiple locales open.
--
Alex