case mangling and binary strings
Dan Sugalski
Assuming I have a parrot string which is explicitly marked as a
binary string...
What should happen when it's told to upcase/downcase/titlecase
itself? (You may assume that we have strings which are explicitly
marked at least Unicode, so there is a difference between STRING*
which are text and binary)
--
Dan
--------------------------------------it's like this-------------------
Dan Sugalski even samurai
d...@sidhe.org have teddy bears and even
teddy bears get drunk
Sam Ruby
> Okay, here's a question for everyone to hash out.
>
> Assuming I have a parrot string which is explicitly marked as a binary
> string...
>
> What should happen when it's told to upcase/downcase/titlecase itself?
> (You may assume that we have strings which are explicitly marked at
> least Unicode, so there is a difference between STRING* which are text
> and binary)
Not necessarily a good example to follow, but take it for what it is worth:
Python has two data types: str and unicode. The unicode type can be
unambiguously viewed as a sequence of characters. The string type,
however, is a sequence of bytes. Taking a unicode string and calling
the encode() method on it returns a str, reinforcing the notion that str
can be viewed as binary.
However, str has an upper() method defined on it. The way it operates
is to take the range of bytes that correspond to us-ascii and perform a
us-ascii uppercase on them. The remaining bytes are left alone.
Example output:
>>> u'\u0061\u00e1'.upper()
u'A\xc1'
>>> '\x61\xe1'.upper()
'A\xe1'
>>> u'\u0061\u00e1'.encode('iso-8859-1').upper()
'A\xe1'
- Sam Ruby
Matt Fowles
I vote for doing nothing in the up/down case options as those are
frequently just used to get a cannonical form for comparison.
Although I could understand an argument for throwing an exception...
Matt
On Tue, 2 Nov 2004 11:53:08 -0500, Dan Sugalski <d...@sidhe.org> wrote:
> Okay, here's a question for everyone to hash out.
>
> Assuming I have a parrot string which is explicitly marked as a
> binary string...
>
> What should happen when it's told to upcase/downcase/titlecase
> itself? (You may assume that we have strings which are explicitly
> marked at least Unicode, so there is a difference between STRING*
> which are text and binary)
> --
> Dan
>
> --------------------------------------it's like this-------------------
> Dan Sugalski even samurai
> d...@sidhe.org have teddy bears and even
> teddy bears get drunk
>
--
"Computer Science is merely the post-Turing Decline of Formal Systems Theory."
-???
Nicholas Clark
> However, str has an upper() method defined on it. The way it operates
> is to take the range of bytes that correspond to us-ascii and perform a
> us-ascii uppercase on them. The remaining bytes are left alone.
I'd prefer parrot not to do that, on the basis that Perl 5 supports EBCDIC
platforms, and I feel parrot should too, by being completely character
encoding agnostic. [Well, maybe better described as "atheist" :-) ]
Nicholas Clark
James Mastros
> Okay, here's a question for everyone to hash out.
>
> Assuming I have a parrot string which is explicitly marked as a binary
> string...
>
> What should happen when it's told to upcase/downcase/titlecase itself?
> (You may assume that we have strings which are explicitly marked at
> least Unicode, so there is a difference between STRING* which are text
> and binary)
exception can then be handled in whatever way the language that compiled
the code in question wants. (I'd want perl6 to do nothing and leave the
string as-is, with an optional, default-on, warning, but that's a matter
for p6l.)
Mechanism, not policy, on this issue.
-=- James Mastros
Nicholas Clark
> Okay, here's a question for everyone to hash out.
>
> Assuming I have a parrot string which is explicitly marked as a
> binary string...
>
> What should happen when it's told to upcase/downcase/titlecase
> itself? (You may assume that we have strings which are explicitly
> marked at least Unicode, so there is a difference between STRING*
> which are text and binary)
I'd see 2 possibilities as logical
Either
a: It throws an exception
or
b: It remains unchanged
Which means that I could think it nice if I could choose which behaviour I
wanted.
Nicholas Clark
Dan Sugalski
>Dan~
>
>I vote for doing nothing in the up/down case options as those are
>frequently just used to get a cannonical form for comparison.
>Although I could understand an argument for throwing an exception...
People better not be using binary data as a canonical form for
comparison. That's really, really a bad idea.
I think Nick's right -- a conditional exception's in order.
Leopold Toetsch
> Okay, here's a question for everyone to hash out.
> Assuming I have a parrot string which is explicitly marked as a
> binary string...
> What should happen when it's told to upcase/downcase/titlecase
> itself?
If it's pure binary pitch a fit.
If it has an encoding attached, continue,
leo
Dan Sugalski
I think I'd agree. Besides, it also means that we'd mis-mangle Leo's
name if we upcased a binary version of it and that just doesn't seem
right.
I'll make mangling on binary data throw an optional exception and
otherwise leave the string alone. And I'll also make sure we have at
least a binary and ASCII charset checked in to start with. I might do
Latin-1 as well if I'm feeling adventuresome and it's easy enough.
(It's a good thing I don't have the Ora CJKV book easily at hand or I
might take a shot at Shift-JIS and that'd blow a day or two... :)
Dan Sugalski
>Dan Sugalski wrote:
>
>>At 5:43 PM +0000 11/2/04, Nicholas Clark wrote:
>>
>>>On Tue, Nov 02, 2004 at 12:35:26PM -0500, Sam Ruby wrote:
>>>
>>>> However, str has an upper() method defined on it. The way it operates
>>>> is to take the range of bytes that correspond to us-ascii and perform a
>>>> us-ascii uppercase on them. The remaining bytes are left alone.
>>>
>>>I'd prefer parrot not to do that, on the basis that Perl 5 supports EBCDIC
>>>platforms, and I feel parrot should too, by being completely character
>>>encoding agnostic. [Well, maybe better described as "atheist" :-) ]
>>
>>I think I'd agree. Besides, it also means that we'd mis-mangle
>>Leo's name if we upcased a binary version of it and that just
>>doesn't seem right.
>>
>>I'll make mangling on binary data throw an optional exception and
>>otherwise leave the string alone. And I'll also make sure we have
>>at least a binary and ASCII charset checked in to start with. I
>>might do Latin-1 as well if I'm feeling adventuresome and it's easy
>>enough. (It's a good thing I don't have the Ora CJKV book easily at
>>hand or I might take a shot at Shift-JIS and that'd blow a day or
>>two... :)
>
>take it that it involves ditching ICU. *shrug*
Making ICU optional, at least. It's too problematic on too many
platforms, and just turns into a big headache. It seemed like a good
idea at the time, and while it's still better than most of the
alternatives that doesn't, unfortunately, make it good.
I expect I'll put together a Unicode charset that uses ICU to do its
thing, and go from there. We certainly need Unicode support, so it's
not like we can't do it. (And we still don't have a better option,
unfortunately)
Dan Sugalski
Yeah, that's the plan.
I'd like to add another entry to the internal API:
OPTIONAL_INTERNAL_EXCEPTION
which works like INTERNAL_EXCEPTION only it checks first to see if
the exception should get thrown before throwing it. If the
exception's throwable, it gets thrown, otherwise things continue on.
I'd say lets alter the current internal_exception code, except
everything that uses it expects them to be fully fatal exceptions and
if we continued, well... I expect the result would be bad.
Dan Sugalski
>
>I don't care if Parrot uses ICU on any platform.
>
>I do care that Parrot supports utf-8 on every platform.
Ah, OK. Yes, we will support all the unicode encodings, as well as
the unicode character set, on all platforms.
Ben Morrow
Quoth d...@sidhe.org (Dan Sugalski):
> Okay, here's a question for everyone to hash out.
>
> Assuming I have a parrot string which is explicitly marked as a
> binary string...
>
> What should happen when it's told to upcase/downcase/titlecase
> itself? (You may assume that we have strings which are explicitly
> marked at least Unicode, so there is a difference between STRING*
> which are text and binary)
Error. Case is a property of characters, and a binary string doesn't
consist of characters, so the operation is meaningless.
If you want to change the case, you have to decode it (specifying the
encoding) into a text string, and case-change that.
Ben
--
For the last month, a large number of PSNs in the Arpa[Inter-]net have been
reporting symptoms of congestion ... These reports have been accompanied by an
increasing number of user complaints ... As of June,... the Arpanet contained
47 nodes and 63 links. [ftp://rtfm.mit.edu/pub/arpaprob.txt] * b...@morrow.me.uk
Jeff Clites
I'll point out, for the record, that it make no sense to have strings
holding binary data--the observation that fundamental string API make
no sense for such strings is an indication that this is an attempt to
use the wrong data type for the wrong purpose. "Just thrown an
exception" is not a magic cure--it's an indication that things are
being modeled incorrectly. It shouldn't be necessary.
In terms of ICU specifically, I don't much care what underlying library
we use. That said, if we want to "support all the unicode encodings",
we're going to have to provide all of the relevant functionality that
ICU gives us. Trying to re-implement this, separately, from scratch, is
a mistake--a waste of resources. A better approach (for Parrot, and for
the computing community in general) would be to put our efforts toward
getting ICU working on the platforms on which it is having problems. Or
start with some other well-established library--but ICU seems to be the
best around.
JEff
Adam Thomason
> At 1:16 PM -0500 11/2/04, Sam Ruby wrote:
>
> Making ICU optional, at least. It's too problematic on too many
> platforms, and just turns into a big headache. It seemed like a good
> idea at the time, and while it's still better than most of the
> alternatives that doesn't, unfortunately, make it good.
>
Ah, hooray. Please announce when this is done... the porting problem
will become much more interesting and much less irritating when this
happens once more. Porting parrot and porting ICU need to be separate
issues.
Adam
Leopold Toetsch
> At 6:16 PM +0100 11/2/04, Leopold Toetsch wrote:
>>
>>If it's pure binary pitch a fit.
>>If it has an encoding attached, continue,
> Yeah, that's the plan.
> I'd like to add another entry to the internal API:
> OPTIONAL_INTERNAL_EXCEPTION
> which works like INTERNAL_EXCEPTION only it checks first to see if
> the exception should get thrown before throwing it.
We are doing that already in the find_global opcode. Depending on a bit
in the interpreter->ctx.errors flag, we through a real exception or not.
So I'd propose:
- make that a real_exception instead
- new API is then real_exception_if_error(..., ERROR_foo_bit, ...)
leo
Leopold Toetsch
> I expect I'll put together a Unicode charset that uses ICU to do its
> thing, and go from there. We certainly need Unicode support, so it's
> not like we can't do it. (And we still don't have a better option,
> unfortunately)
ICU 3.0 should be out AFAIK. This is supposed to fix a lot of build
issues.
leo
Leopold Toetsch
> Python has two data types: str and unicode.
Python's unicode "features" are probably not really good examples
generally. Ongoing discussion in Python lists seem to indicate that
there a rather rough edges still.
> - Sam Ruby
leo