Fat String Class
Le Chaud Lapin
interest in custom C++ string classes, indicating that std::basic_string may be
lacking, and that a discussion about the desired functionality of string classes
may serve to guide e.g. proposals & other future such classes. -mod/aps }
Hi All,
First, I would like to thank those of you who steered me away from the
notion of a "caseless string class" several months ago." I have since
gained an intuitive appreciation of the foolishness of that idea. :D
Now I must revisit lass String<> one final time, and I am worried that
I will make a critical omission from the internal representation of
string objects that will come back to haunt later. In particular, I am
worried about the semantics of international representation.
While not an expert in the subject, I do have knowledge of foreign
languages, and I know that UNICODE is not a cure-all. There are deep,
fundamental problems with the semantics of strings and and their
operations that might or might not have been solved as of 2008.
Perhaps a sensible solution exists in fragmented for in standard
libraries and encodings. Perhaps computational linguists have created
ad-hoc solutions that we are not yet aware of. Whatever the state of
the art, I would like to capture as much possible now to minimize
regret later. I do know that the locale will be included in each and
every object of my String<>.
Though feature richness is a requirement, I am not interested in
operators like:
operator ==
operator !=
operator >
operator <
These are obvious. They already exist in my String<>, and those that
are absent will be shamelessly copied from open-source projects.
Instead I would like to find a terminal or near-terminal
representation of a string object that facilitates sensibility between
international string operations.
Let's take a concrete example:
String<char> s1 "exasperation"; // English
String<char> s2 "exasp�ration"; // fran�ais
In your opinion,
1. Should [s1 == s2] be true?
2. What should be the sort order of s1 versus s2?
3. What should be the difference in representation for s1 and s2?
Note that s1 and s2 might be in a container of my choosing where
comparison operation is operator == for class String (Yes, I know this
is not the way std:: does it).
I am not concerned with efficiency in space. If it turns out that
sizeof(String<>) must be == 16 to preempt later grief, so shall it
be.
And finally, to the polyglots of the group whose native language(s) is
not English, how would you design a string class, with and without
consideration for ASCII?
-Le Chaud Lapin-
--
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
Alberto Ganesh Barbati
> While not an expert in the subject, I do have knowledge of foreign
> languages, and I know that UNICODE is not a cure-all. There are deep,
> fundamental problems with the semantics of strings and and their
> operations that might or might not have been solved as of 2008.
> Perhaps a sensible solution exists in fragmented for in standard
> libraries and encodings. Perhaps computational linguists have created
> ad-hoc solutions that we are not yet aware of. Whatever the state of
> the art, I would like to capture as much possible now to minimize
> regret later. I do know that the locale will be included in each and
> every object of my String<>.
If you had ever read the UNICODE book you would know that UNICODE is not
just an encoding but also a collection of data tables and algorithms to
solve a large number of issues related to localization. Yes, I do
believe that UNICODE is a cure-all, because I know that UNICODE is so
much more than what most people think it is.
> Though feature richness is a requirement, I am not interested in
> operators like:
>
> operator ==
> operator !=
> operator >
> operator <
>
> These are obvious. They already exist in my String<>, and those that
> are absent will be shamelessly copied from open-source projects.
There are far from obvious, in my opinion. See below.
> Instead I would like to find a terminal or near-terminal
> representation of a string object that facilitates sensibility between
> international string operations.
UNICODE is your best choice here.
>
> Let's take a concrete example:
>
> String<char> s1 "exasperation"; // English
> String<char> s2 "exaspération"; // français
>
> In your opinion,
>
> 1. Should [s1 == s2] be true?
> 2. What should be the sort order of s1 versus s2?
> 3. What should be the difference in representation for s1 and s2?
You are missing the point. It doesn't make any sense to compare an
English string with a French string! A string is a string. If I put an
English word in a French dictionary, the reader expects the dictionary
to be in French order and the English word should collate as-if it were
French. The locale is not attached to each single string, but rather to
the context in which the string is considered.
All string comparison operations therefore require contextual
information to be known: i.e. the locale. Once you have clear the locale
you are working into, the result of those operations is also clear and
can be determined using data tables.
The problem is that those operations should *not* be performed by
calling operator== or operator< because the interface of a binary
operator does not allow passing the contextual information! We could use
a global locale object (as in C setlocale()) but that isn't a good
solution for at least two reasons:
1) you have to be careful with multi-threading
2) if you enter strings as keys in a map or set with one locale and then
change the locale, the map or set is totally screwed up
That's why set and map allows stateful comparison objects! Because you
want to have an English map holding strings, rather than a (non-local
aware) map holding English strings.
> Note that s1 and s2 might be in a container of my choosing where
> comparison operation is operator == for class String (Yes, I know this
> is not the way std:: does it).
As I said, a locale-aware comparison should not be performed with ==.
> I am not concerned with efficiency in space. If it turns out that
> sizeof(String<>) must be == 16 to preempt later grief, so shall it
> be.
Storing locale in String<> objects is useless, because the locale is not
an attribute of the string. Therefore I don't think String<> should ever
need to be larger than a pointer and a size_t, unless you are using a
short-string optimization technique.
> And finally, to the polyglots of the group whose native language(s) is
> not English, how would you design a string class, with and without
> consideration for ASCII?
In my opinion, a string should model sequence of UNICODE characters. A
UNICODE character can be encoded with one or more code-units according
to some encoding, which ought to be hidden by the interface. The problem
is that most string implementations model a sequence of code-units
instead, thus the encoding "leaks" through the interface. This is bad.
The work made by Python community is exemplary. Python 2.x has two
classes, namely str and unicode. The former is a sequence of bytes and
the latter a sequence of UNICODE characters. The problem is that a
sequence of bytes can also be interpreted as a sequence of character in
some yet-to-be-specified code-page. A mess. Python 3.x fixes the
problem: the unicode class has been removed, str now models a sequence
of UNICODE characters and a new class bytes models a sequence of bytes,
which cannot be interpreted as a sequence of characters. In order to
interpret a bytes object as a string you need to apply the function
decode() and you get a str object. The encode() function performs the
opposite transformation.
Back to my idea of what a string should be, binary operations should
perform non-local-aware lexicographical comparison for convenience only.
Any other form of comparison should be provided through the use of
locale objects, whose interface should make them suitable to be used as
comparison function in containers like set and map.
Anyway, have a look at http://www.icu-project.org/
HTH,
Ganesh
Erik Wikström
> { Accepted (even though not C++-specific) because there seems to be a wide
> interest in custom C++ string classes, indicating that std::basic_string may be
> lacking, and that a discussion about the desired functionality of string classes
> may serve to guide e.g. proposals & other future such classes. -mod/aps }
>
> Hi All,
>
> First, I would like to thank those of you who steered me away from the
> notion of a "caseless string class" several months ago." I have since
> gained an intuitive appreciation of the foolishness of that idea. :D
>
> Now I must revisit lass String<> one final time, and I am worried that
> I will make a critical omission from the internal representation of
> string objects that will come back to haunt later. In particular, I am
> worried about the semantics of international representation.
>
> While not an expert in the subject, I do have knowledge of foreign
> languages, and I know that UNICODE is not a cure-all.
While Unicode might not solve all problems it is currently the best
we've got, able to represent all written characters in use today (and a
whole lot not in used, plus some other characters). Your best bet for a
fat string is probably to use UTF-16 or UTF-32 as internal
representation and allow conversions to/from other representations.
> There are deep,
> fundamental problems with the semantics of strings and and their
> operations that might or might not have been solved as of 2008.
> Perhaps a sensible solution exists in fragmented for in standard
> libraries and encodings. Perhaps computational linguists have created
> ad-hoc solutions that we are not yet aware of. Whatever the state of
> the art, I would like to capture as much possible now to minimize
> regret later. I do know that the locale will be included in each and
> every object of my String<>.
>
> Though feature richness is a requirement, I am not interested in
> operators like:
>
> operator ==
> operator !=
> operator >
> operator <
>
> These are obvious. They already exist in my String<>, and those that
> are absent will be shamelessly copied from open-source projects.
>
> Instead I would like to find a terminal or near-terminal
> representation of a string object that facilitates sensibility between
> international string operations.
>
> Let's take a concrete example:
>
> String<char> s1 "exasperation"; // English
> String<char> s2 "exaspération"; // français
>
> In your opinion,
>
> 1. Should [s1 == s2] be true?
In most cases if I ask if two strings are equal or not I'm interested in
equality in the mathematical sense, I do not want to pay the price for
costly operations which try to determine if the meaning of the words are
the same.
Another way to look at it is that while in English "naïve" and "naive"
might be the same thing, in some other language (or for some other word)
the addition of a diacritical mark might radically change the meaning of
a word. This means that if you allow ["naïve" == "naive"] you must
provide code for languages/words where it is not true.
In short, locale dependant comparisons are not part of the string class
(or, at the very least, it should not be the default behaviour).
> 2. What should be the sort order of s1 versus s2?
Irrelevant, I just want the operation to be fast (so that I effectively
can use it as a key in a map), so using simple comparison of the
individual characters in the string should suffice. Again, locale
dependent comparisons should not be part of the string.
> 3. What should be the difference in representation for s1 and s2?
Well, the 6th character is different, and they have different locales,
so I guess the pointers to the actual text differs, and the pointers/
members holding the locale should be different too. :-)
> Note that s1 and s2 might be in a container of my choosing where
> comparison operation is operator == for class String (Yes, I know this
> is not the way std:: does it).
>
> I am not concerned with efficiency in space. If it turns out that
> sizeof(String<>) must be == 16 to preempt later grief, so shall it
> be.
I do not think that the since of the actual string-class will be the
biggest problem, rather it is the representation of the text, unless you
want to use UTF-8, but there are reasons why UTF-16 and UTF-32 are
commonly used as internal representation.
> And finally, to the polyglots of the group whose native language(s) is
> not English, how would you design a string class, with and without
> consideration for ASCII?
Keep it simple, extra functionality can be provides outside the class,
what a string class should do is to hold a series of characters and
allow operations on them.
--
Erik Wikström
Oncaphillis
> Hi All,
>
> Let's take a concrete example:
>
> String<char> s1 "exasperation"; // English
> String<char> s2 "exaspération"; // français
>
> In your opinion,
>
> 1. Should [s1 == s2] be true?
Definitely NOT. They should always be considered identical
if they represent the same sequence of char. One could think about
a method like:
String<char> String::reduce2ascii
which tries to eliminate all diacritics etc. and do a comparison on
the results.
> 2. What should be the sort order of s1 versus s2?
It depends on your locale and other special settings. As a german
speaker I only know about (and always stumble over ) the 'Ä','Ö','Ü' rule:
<cite from="wikipedia">
In German dictionaries, the letter (Ä) is collated together with A, while in
German phonebooks the letter is collated as AE
</cite>
But the Swedish:
<cite from="wikipedia">
(Ä) is placed in the Swedish alphabet after Z and Å but before Ö.
</cite>
> 3. What should be the difference in representation for s1 and s2?
Why would you want to represent them differently ? They may be
just a sequence of char with all the additional magic coming from
std::local(), codecvt() etc..
> Note that s1 and s2 might be in a container of my choosing where
> comparison operation is operator == for class String (Yes, I know this
> is not the way std:: does it).
>
> I am not concerned with efficiency in space. If it turns out that
> sizeof(String<>) must be == 16 to preempt later grief, so shall it
> be.
>
> And finally, to the polyglots of the group whose native language(s) is
> not English, how would you design a string class, with and without
> consideration for ASCII?
hmm -- as I stated before, may be a method which tries to
map all char/wchar_t to ASCII as sensible as it can be done. May be
even a mapping 'hiragana/katakana/hangul' ==> Ascii. There also
should be a couple of other phonetic symbols in asian languages
which could be used for it (Vietnamese anyone ? ).
But -- I mean ASCII is really the most crippled representation of
text. Dating back from 7bit Telex. One should at least stick with
ISO-8859-1 or (-15).
I have some 10 year old books about internationalization catching
dust in my drawer which recommend that every internal text/string
representation should be done in wchar_t anyway. Did that ever happen ?
I mean consistently ?
Hope I understood your initial question
O.
Oncaphillis
> On 2008-07-08 01:45, Le Chaud Lapin wrote:
>> In your opinion,
>>
>> 1. Should [s1 == s2] be true?
>
> In most cases if I ask if two strings are equal or not I'm interested in
> equality in the mathematical sense, I do not want to pay the price for
> costly operations which try to determine if the meaning of the words are
> the same.
Fully Ack
> Another way to look at it is that while in English "na??ve" and "naive"
> might be the same thing, in some other language (or for some other word)
> the addition of a diacritical mark might radically change the meaning of
> a word. This means that if you allow ["na??ve" == "naive"] you must
> provide code for languages/words where it is not true.
Hmm -- if you're really after "meaning" you're heading for big trouble
and dwell deeply in the field of computer linguistics and automatic
translation. As an example the word "mail" has been imported
into the german language (even as a verb: Ich maile, du mailst, er/sie/es
mailed), but it is only applicable to e-mail.
so
conext<"english">("mail")!=conext<"de">("mail")
But -- a phonetic comparison makes sense for me. At least I'm
quite happy with google coming up with "Crème" when I lookup
"creme" because I'm not aware where all these accents have to
go (and is it grave, circumflex or accute ?).
So:
String("egg")!=String("tamago")
Should be true:
but a comparison like:
to_phonetics("tamago") ==
to_phonetics(<ta><ma><go>[hiragana]) ==
to_phonetics(<ta><ma><go>[katakana]) ==
to_phonetics(<tamaga>[kanji/hanzi]);
Makes sense in special cases, and it is provided by some SQL engines.
Although you're once again asking for trouble when it comes to
multiple pronouncation for the same character within the current
context because to_phonetics([kanji/hanzi]) would have to return a ref to
container of different phonetics .... etc...
> Keep it simple, extra functionality can be provides outside the class,
> what a string class should do is to hold a series of characters and
> allow operations on them.
>
Fully Ack
--
Harald Luessen
>interest in custom C++ string classes, indicating that std::basic_string may be
>lacking, and that a discussion about the desired functionality of string classes
>may serve to guide e.g. proposals & other future such classes. -mod/aps }
May be my reply is more directed to this than to the original post.
I am aware of the typical Unicode string classes described in the
other answers. A composition of std::string and UTF-8 or something
is way better than char* strings. But is it possible to create a
string class with much more information? The problem could be to
agree on the right additional information, its format and where
should this end. I think about infos like this:
In "Harald Lüßen" there could be the additional info
- language is german
- local is de_DE
- string[8] could be written as ue
- string[9] could be written as ss
- string[9] must be written as SS in uppercase
In "HARALD LUESSEN" there could be the additional info
- language is german
- local is de_DE
- string[8..9] could be written as Ü
- string[10..11] could be written as ss or ß in lowercase
This allows upper and lower operations without losing infos.
In "This sentence has a deutsches word." there could be the info
- language[0..19] is english
- language[20..29] is german
- language[30..35] is english
- local is en_EN
Or there could be a phonetic information assigned to the string.
With a specified phonetic language.
Or there could be a Transliteration assigned to the string.
A cyrillic string with a latin transliteration.
We have this kind of problems at work all the time. We work
with huge international databases. As an example think about
navigation system input strings.
May be most of this information is represented today in complex
data structures, trees, connected graphs and so on. Everybody is
doing something else. Could there ever be an agreement on a
really rich string class with a defined set of operations
working on it? This would be the next step from char* over
std::string towards artificial intelligent (Star Trek) strings.
:-)
Harald
Alberto Ganesh Barbati
>> { Accepted (even though not C++-specific) because there seems to be a wide
>> interest in custom C++ string classes, indicating that std::basic_string may be
>> lacking, and that a discussion about the desired functionality of string classes
>> may serve to guide e.g. proposals & other future such classes. -mod/aps }
>
> May be my reply is more directed to this than to the original post.
>
> I am aware of the typical Unicode string classes described in the
> other answers. A composition of std::string and UTF-8 or something
> is way better than char* strings. But is it possible to create a
> string class with much more information? The problem could be to
> agree on the right additional information, its format and where
> should this end. I think about infos like this:
There's no limit on the additional data (or metadata or contextual data)
that you can add to a string. No matter how much data you add, you will
encounter people who would want to add more data and people who would
want to have less data. The use cases are so many and so diverse that
it's difficult for a general purpose library to match all user expectations.
On the other hand, there will always be a desperate need for a plain
string without additional data. So I say a library should first try to
provide that. And once you have it, let the user build her own
SuperString(tm) with her own metadata.
Just my opinion,
Ganesh
Mathias Gaunard
> In your opinion,
>
> 1. Should [s1 == s2] be true?
Depends on the collation used.
I would probably assume the general Unicode collation there.
> 2. What should be the sort order of s1 versus s2?
Depends on the collation used.
I would probably assume the general Unicode collation there.
> 3. What should be the difference in representation for s1 and s2?
Just use a normalized form adequate for representing your data.
Unicode Normalized C form (encoded as whatever UTF-x) is a popular
one.
> And finally, to the polyglots of the group whose native language(s) is
> not English, how would you design a string class, with and without
> consideration for ASCII?
Iterate over real characters, not code points. Grapheme clusters.
You may want to provide iterators over code points too.
Maybe even bytes/code units.
Allowing iteration over words, sentences, lines etc. might also be
interesting, but that's becoming difficult because of Thai.
Oncaphillis
> - language is german
> - local is de_DE
> - string[8] could be written as ue
> - string[9] could be written as ss
> - string[9] must be written as SS in uppercase
Oh !! but this is context dependent because there isn't
any upper case 'ß' -- or at least it really looks
like the lower case 'ß' (and I guess it has the same UNICODE)
so it's 'SS' whenever the char is embedded into an
upper case word.
> In "HARALD LUESSEN" there could be the additional info
> - language is german
> - local is de_DE
> - string[8..9] could be written as Ü
> - string[10..11] could be written as ss or ß in lowercase
>
> This allows upper and lower operations without losing infos.
And what happens to all these poor Muellers with there name
actually written with 'ue' ?
> In "This sentence has a deutsches word." there could be the info
> - language[0..19] is english
> - language[20..29] is german
> - language[30..35] is english
> - local is en_EN
>
> Or there could be a phonetic information assigned to the string.
> With a specified phonetic language.
This, as far as I can see, is the most sensible thing to do.
Applying language/local rules when translating into a phonetic
alphabet. But the other way around ? I don't think so.
> Or there could be a Transliteration assigned to the string.
> A cyrillic string with a latin transliteration.
>
> We have this kind of problems at work all the time. We work
> with huge international databases. As an example think about
> navigation system input strings.
Yes, the possibilities are endless, and it's a really interesting
field. But then you write something like.
std::cerr << FatString("Hello World") << std::endl;
and C++ blows up your program to 50GB because it attaches a
whole KI system to your code. The overhead of iostream already
hurts sometimes.
> May be most of this information is represented today in complex
> data structures, trees, connected graphs and so on. Everybody is
> doing something else. Could there ever be an agreement on a
> really rich string class with a defined set of operations
> working on it? This would be the next step from char* over
> std::string towards artificial intelligent (Star Trek) strings.
> :-)
So the only question remaining: is it TOS or TNG ? :-)
O.
Pavel Minaev
> Let's take a concrete example:
>
> String<char> s1 "exasperation"; // English
> String<char> s2 "exasp�ration"; // fran�ais
>
> In your opinion,
>
> 1. Should [s1 == s2] be true?
If you are really shooting for safety, ditch operator== (and other
comparison operators) completely. Instead, provide comparison
functions, each of which clearly indicates whether the comparison is
codepoint-based, uses Unicode collation algorithm, or uses a specific
locale. E.g.:
template <class T>
bool is_equal_codepoint(const String<T>& s1, const String<T>& s2);
template <class T>
bool is_equal_uca(const String<T>& s1, const String<T>& s2);
template <class T>
bool is_equal(const String<T>& s1, const String<T>& s2, const
std::locale& locale);
Then, every comparison will have to be explicit with regard to what
kind it is, and every container would have to specify the comparison
predicate explicitly, too. This is really the only sane way, because
there is no general way to compare strings - it all depends on where
they come from, and how they are presented to the user.
If your String objects carry their locales with them, then comparison
doesn't make sense at all. There can be only one collation used in a
comparison, and each locale defines one. Frankly, there is no reason
why a string should have an associated locale (it may have an
encoding, sure, but that's not the same).
> 2. What should be the sort order of s1 versus s2?
The same approach as for equality.
> 3. What should be the difference in representation for s1 and s2?
Why should the language of the string have any relationship to
representation of its character data?
> And finally, to the polyglots of the group whose native language(s) is
> not English, how would you design a string class, with and without
> consideration for ASCII?
I prefer my strings to be plain containers of Unicode codepoints, and
everything collation-related to come from the locale which is
explicitly specified for every collation-aware operation. This also
seems to be the dominant model today - see Java, .NET etc (though they
also have the notion of "thread current locale", and can use that as
well).
Bart van Ingen Schenau
>
> Instead I would like to find a terminal or near-terminal
> representation of a string object that facilitates sensibility between
> international string operations.
>
> Let's take a concrete example:
>
> String<char> s1 "exasperation"; // English
> String<char> s2 "exaspération"; // français
>
> In your opinion,
>
> 1. Should [s1 == s2] be true?
With operator== a resounding NO.
You can have a separate comparator that can ignore accents and/or case
differences.
> 2. What should be the sort order of s1 versus s2?
That depends on the locale and the context where you need the ordering.
Different languages have different rules for the order, and sometimes
even different rules for dictionaries and phonebooks.
If you want the comparison to be locale-independent, take a look at the
UCA (UNICODE Collation Algorithm), which specifies an ordering for all
UNICODE codepoints.
> 3. What should be the difference in representation for s1 and s2?
I would let the difference appear in the codepoint values used for e and
é.
>
> Note that s1 and s2 might be in a container of my choosing where
> comparison operation is operator == for class String (Yes, I know this
> is not the way std:: does it).
>
> I am not concerned with efficiency in space. If it turns out that
> sizeof(String<>) must be == 16 to preempt later grief, so shall it
> be.
>
> And finally, to the polyglots of the group whose native language(s) is
> not English, how would you design a string class, with and without
> consideration for ASCII?
For a string that must hold international characters, I would use one of
the UNICODE codings for storage.
Which coding to use depends on the expected usage of the string class.
For the most generic class, I would probably use UCS4 coding, with
interfaces to convert to/from UTF-8 (and possibly UTF-16).
>
> -Le Chaud Lapin-
>
Bart v Ingen Schenau
--
a.c.l.l.c-c++ FAQ: http://www.comeaucomputing.com/learn/faq
c.l.c FAQ: http://c-faq.com/
c.l.c++ FAQ: http://www.parashift.com/c++-faq-lite/
Harald Luessen
>> In "Harald L?there could be the additional info
>> - language is german
>> - local is de_DE
>> - string[8] could be written as ue
>> - string[9] could be written as ss
>> - string[9] must be written as SS in uppercase
>
> Oh !! but this is context dependent because there isn't
> any upper case '? -- or at least it really looks
> like the lower case '? (and I guess it has the same UNICODE)
> so it's 'SS' whenever the char is embedded into an
> upper case word.
In "Lüßen" there is an L, an U+00FC (lower u umlaut), U+00DF (eszet),
an e and an n. These german letters are always a problem in old
software and in transforming to uppercase and lowercase and back.
"Masse" and "Maße" are different words, both are "MASSE" in uppercase,
and who knows how to make them lowercase again? I am sure
other languages have different problems. perhaps some of them can
be solved with metadata in each string. Of cource these FatStrings
should be everywhere from input devices, in operating systems and
in output devices. We want to keep the metadata.
>> In "HARALD LUESSEN" there could be the additional info
>> - language is german
>> - local is de_DE
>> - string[8..9] could be written as ?
>> - string[10..11] could be written as ss or ?in lowercase
>>
>> This allows upper and lower operations without losing infos.
>
> And what happens to all these poor Muellers with there name
> actually written with 'ue' ?
That is why the info is in the object and not in the class.
In "Mueller" there could be
- string[1..2] must not be changed to ü (U+00FC)
>> In "This sentence has a deutsches word." there could be the info
>> - language[0..19] is english
>> - language[20..29] is german
>> - language[30..35] is english
>> - local is en_EN
>>
>> Or there could be a phonetic information assigned to the string.
>> With a specified phonetic language.
>
> This, as far as I can see, is the most sensible thing to do.
> Applying language/local rules when translating into a phonetic
> alphabet. But the other way around ? I don't think so.
It was just an idea.
>> Or there could be a Transliteration assigned to the string.
>> A cyrillic string with a latin transliteration.
>>
>> We have this kind of problems at work all the time. We work
>> with huge international databases. As an example think about
>> navigation system input strings.
>
> Yes, the possibilities are endless, and it's a really interesting
> field. But then you write something like.
>
> std::cerr << FatString("Hello World") << std::endl;
>
> and C++ blows up your program to 50GB because it attaches a
> whole KI system to your code. The overhead of iostream already
> hurts sometimes.
Start now to design the FatString class and in 10 to 20 years
when it is ready that will not hurt. The 50 GByte will just be
in some cristal storage 1mm^3 in size, somewhere in your
cell phone navigation wikipedia holographic personal device
in your wedding ring or wrist watch.
>> May be most of this information is represented today in complex
>> data structures, trees, connected graphs and so on. Everybody is
>> doing something else. Could there ever be an agreement on a
>> really rich string class with a defined set of operations
>> working on it? This would be the next step from char* over
>> std::string towards artificial intelligent (Star Trek) strings.
>> :-)
>
> So the only question remaining: is it TOS or TNG ? :-)
TNG or DS9 or VOY.
This whole FatString idea is not meant to be very serious.
But it is a test to see if people could imagine such a beast.
There is a limit of what unicode strings can do. How do we
cross it and have a really international world wide text
representation? Is it possible? Do we need it? Would
anyone use it?
>O.
Harald
Nominal Pro
> Instead I would like to find a terminal or near-terminal
> representation of a string object that facilitates sensibility between
> international string operations.
>
> Let's take a concrete example:
>
> String<char> s1 "exasperation"; // English
>
> In your opinion,
>
> 1. Should [s1 == s2] be true?
> 2. What should be the sort order of s1 versus s2?
> 3. What should be the difference in representation for s1 and s2?
1. In what sense are strings s1 and s2 equal or equivalent? I think
you are confusing the concept of a "string" with the concept of a
"word". A string is not the same thing as a word. A string may contain
a word, or it may not (it could contain numbers, or a completely
meaningless sequence of octets). Thus, I would argue that the only
meaningful way that two strings, s1 == s2 would be true in C++ is if
they contain the exact same sequence of characters.
If you want to capture meaning of the words, then I would suggest a
different approach than making some kind of "smart" string class.
2. Without respect to a specific language or locale, the only
meaningful sort order is the order of the Unicode characters. Since
you are comparing two strings from different languages, you might be
implying that one language or the other's sort order applies. But can
you really pick one and have it make any sense? Suppose that, instead
of comparing an English and French word, you compare English and
Hebrew words? What is the order of the Hebrew characters in relation
to the English characters? Or, what if you had to compare two
meaningless strings of random Unicode characters? Would any locale
apply?
3. I am not certain what you mean by "difference in representation."
Again, I ask, are you talking about the representation of strings or
words? If all you seek to represent are strings, I believe the
std::wstring is sufficient representation in C++.
> And finally, to the polyglots of the group whose native language(s) is
> not English, how would you design a string class, with and without
> consideration for ASCII?
It would look exactly like std::wstring :)
Francis Glassborow
> On Wed, 9 Jul 2008 Oncaphillis wrote:
>
>>> In "Harald L?there could be the additional info
>>> - language is german
>>> - local is de_DE
>>> - string[8] could be written as ue
>>> - string[9] could be written as ss
>>> - string[9] must be written as SS in uppercase
>> Oh !! but this is context dependent because there isn't
>> any upper case '? -- or at least it really looks
>> like the lower case '? (and I guess it has the same UNICODE)
>> so it's 'SS' whenever the char is embedded into an
>> upper case word.
>
> In "Lüßen" there is an L, an U+00FC (lower u umlaut), U+00DF (eszet),
> an e and an n. These german letters are always a problem in old
> software and in transforming to uppercase and lowercase and back.
> "Masse" and "Maße" are different words, both are "MASSE" in uppercase,
> and who knows how to make them lowercase again? I am sure
> other languages have different problems. perhaps some of them can
> be solved with metadata in each string. Of cource these FatStrings
> should be everywhere from input devices, in operating systems and
> in output devices. We want to keep the metadata.
>
representation of ß is 'SS' but ISTM that in order not to loose
information we need a code point for 'SS' when it is the upper-case form
of ß as distinct from the upper-case form of 'ss'. I suspect that
similar considerations should be given to other places where this kind
of problem occurs.
I know that there are already places in Unicode where the same glyph has
more than one code point and those code points are intentionally
distinct because the glyph has distinct meanings for different languages.
Anyway this now has drifted way of topic for CLC++M, though it
demonstrates how difficult the real world is to computerise.
Mathias Gaunard
> It would look exactly like std::wstring :)
It's still fairly scary that people not only use wide characters but
also think they're the perfect solution to handling Unicode.
1) Wide characters are not Unicode. They're simply characters bigger
than char. What they are depends not only on paper but also in fact of
the implementation.
2) Even in UTF-32 (which only a few variants on Unix use as wchar_t),
Unicode characters are still variable-length. basic_string is totally
unable to work with that.
3) Even if you're crazy enough to use 16 bytes per character so that
you can use basic_string, there are still different ways to express
the same thing in Unicode that are exactly equivalent. The
representation should be normalized.
Eugene Gershnik
> Now I must revisit lass String<> one final time
A "string" means different things for different problem domains. Some
people need the "container of storage units" approach, others a
"container of Unicode codepoints", and you, it seems, a "container of
English words". A single string class cannot possibly satisfy these
goals. Even worse, as others pointed out, some information about how
to deal with a string is necessarily contextual and cannot possibly be
a part of a string object itself.
> Let's take a concrete example:
>
> String<char> s1 "exasperation"; // English
> String<char> s2 "exasp�ration"; // fran�ais
>
The way I see it, by saying
String<char>
rather than
String<utf8, normalization_form_c, ...>
or even more ambitiously
String<en_US>
you have already fixed you intention of creating a low level class
which is more or less a sequential container of storage units.
IOW if you want a "fat" string class the type of the underlying
storage unit quickly becomes an internal implementation detail of
little interest to the class user.
> And finally, to the polyglots of the group whose native language(s) is
> not English, how would you design a string class, with and without
> consideration for ASCII?
IMHO having a "fat" string class is a bad idea to begin with. I would
start with something much simpler than std::string to model an
*immutable* sequential container of code points. This gives you a
single binary interface to pass strings around modules with different
needs. With this class available, all higher level concepts should be
implemented by separate functions/classes external to the string
itself. Do you want to iterate over Unicode codepoints? Fine, create a
stream-like object that produces them out of String<char>. Do you need
to compare English and French words as equivalent if they have the
same meaning (for the given value of 'same')? Fine, plug in some AI
translation engine and create a comparator that does just this. ;-)
--
Eugene
ManicQin
> String<char> s1 "exasperation"; // English
> String<char> s2 "exasp�ration"; // fran�ais
>
> In your opinion,
>
> 1. Should [s1 == s2] be true?
That would be great but as oters said a string should be a string so I
vote for !=
What about languages such as Hebrew , Arabic , Chinese and other none
Latin or even none "left to right" languages?
Le Chaud Lapin
> On Jul 8, 1:45 am, Le Chaud Lapin <jaibudu...@gmail.com> wrote:
>
> > String<char> s1 "exasperation"; // English
> > String<char> s2 "exasp�ration"; // fran�ais
>
> > In your opinion,
>
> > 1. Should [s1 == s2] be true?
>
> That would be great but as oters said a string should be a string so I
> vote for !=
>
> What about languages such as Hebrew , Arabic , Chinese and other none
> Latin or even none "left to right" languages?
Hi All,
I am responding to ManicQuin's post only because his is
chronologically last.
First, I have read everyone's replies, most several times. I would
like to make a retraction regarding what I said about Unicode, related
to what Albert wrote. There does seem to be much that has been
solved, but there is still work to be done, IMHO, not with Unicode,
but with the incorporation of Unicode into a useable string class.
Second, the example I used comparing French "exasperation" to English
"exasperation" was poor. I was probably tired. I am again tired, so
no good examples come to mind right now, but my gut feeling is that
the class should have at least what I have been calling "locale", even
though that might not be the correct term. All of you have warned
against putting intelligence in the string class. I wonder if this bit
of extra information would count as too much intelligence. While I
have not read enough about Unicode to know the path I will follow, I
will probably include this bit of information anyway. My motivation is
the tedious misery that might result if not. I think about all my
code, where operator == and operator !=, etc. are used in contexts
where programmer fingers are not allowed, and would very much like to
continue doing this.
So my strategy is to find out exactly what extra elements should be
included in a String<> class based on work of Unicode to allowed
operator == () and its friends can remain as they are. If it happens
that two strings meet each other on both sides of an operator as
mutual foreigners:
String<> s1 = "mein"; // German for English "mine"
String<> s2 = "mein"; // English stolen from Chinese for type of
noodle.
s1 == s2;
I can let operator == decide through consultation of external state.
The comparison for same-locale strings should be fast enough, and if
the locales difference, then they will differ no matter which scheme
is used.
As for the multi-threading issue that will be encountered such state
is consulted, I have solution for that.;)
-Le Chaud Lapin-
Bo Persson
> On Jul 15, 11:28 am, ManicQin <Manic...@gmail.com> wrote:
>> On Jul 8, 1:45 am, Le Chaud Lapin <jaibudu...@gmail.com> wrote:
>>
>>> String<char> s1 "exasperation"; // English
You will have to store a lot of external state to determine that
English "mine" can also mean "Mine" (noun) in German, or "verminen"
(verb) if it blows up. It can also mean "Grube", or "Mine"(!) if you
dig in the ground.
The still other English "mine", like in "belongs to me" needs even
more context.
http://www.dict.cc/englisch-deutsch/mine.html
So, there is a lot of difference between "It's mine" and "It's a
mine". :-)
>
> As for the multi-threading issue that will be encountered such state
> is consulted, I have solution for that.;)
>
That's the easy part! :-)
Bo Persson
Alberto Ganesh Barbati
>
> "exasperation" was poor. I was probably tired. I am again tired, so
> no good examples come to mind right now, but my gut feeling is that
> the class should have at least what I have been calling "locale", even
I guess "language" is a more appropriate term than locale here, unless
with "locale" you mean other kind of contextual metadata, which is bound
to be more complex.
> though that might not be the correct term. All of you have warned
> against putting intelligence in the string class. I wonder if this bit
> of extra information would count as too much intelligence. While I
> have not read enough about Unicode to know the path I will follow, I
> will probably include this bit of information anyway.
> <snip>
> String<> s1 = "mein"; // German for English "mine"
> String<> s2 = "mein"; // English stolen from Chinese for type of
> noodle.
Ok, so... what about this:
"Let's eat spaghetti, a bratwurst and a crème brûlée"
is this English, Italian, German or French? You can't simply attach
metadata to the *whole* string, you have to consider substrings too. You
have two choices, either you can store metadata in a struct separate
from the textual data, or you store them in the textual data itself by
effectively introducing some form of tagging. XML (with xml:lang) and
Unicode language tags (see 16.9 in
http://www.unicode.org/versions/Unicode5.0.0/ch16.pdf) follow in the
latter category. Despite the obvious added complexity in parsing and
traversing the string, the tag approach has a lot of advantages.
However, according to Unicode terminology, attached metadata are
responsibility of a "higher level protocol" and so, IMHO, should not be
addressed by the basic container.
Just my opinion,
Ganesh
Le Chaud Lapin
wrote:
> Le Chaud Lapin ha scritto:
> > Second, the example I used comparing French "exasperation" to English
> > "exasperation" was poor. I was probably tired. I am again tired, so
> > no good examples come to mind right now, but my gut feeling is that
> > the class should have at least what I have been calling "locale", even
>
> I guess "language" is a more appropriate term than locale here, unless
> with "locale" you mean other kind of contextual metadata, which is bound
> to be more complex.
I finally had a chance today to look more at your ICU link:
http://www.icu-project.org/userguide/intro.html
It seems that locale includes language/country/script, and probably
others. I would probably grab as much information as possible.
> > though that might not be the correct term. All of you have warned
> > against putting intelligence in the string class. I wonder if this bit
> > of extra information would count as too much intelligence. While I
> > have not read enough about Unicode to know the path I will follow, I
> > will probably include this bit of information anyway.
> > <snip>
> > String<> s1 = "mein"; // German for English "mine"
> > String<> s2 = "mein"; // English stolen from Chinese for type of
> > noodle.
>
> Ok, so... what about this:
>
> "Let's eat spaghetti, a bratwurst and a crème brûlée"
>
> is this English, Italian, German or French? You can't simply attach
> metadata to the *whole* string, you have to consider substrings too. You
> have two choices, either you can store metadata in a struct separate
> from the textual data, or you store them in the textual data itself by
> effectively introducing some form of tagging. XML (with xml:lang) and
> Unicode language tags (see 16.9 inhttp://www.unicode.org/versions/Unicode5.0.0/ch16.pdf) follow in the
> latter category. Despite the obvious added complexity in parsing and
> traversing the string, the tag approach has a lot of advantages.
> However, according to Unicode terminology, attached metadata are
> responsibility of a "higher level protocol" and so, IMHO, should not be
> addressed by the basic container.
This is a good example. My answer: I don't know. :) I have to learn
more about Unicode.
If there is some super-code that will include all Latin scripts, I
would use that for this example to represent the string. Then all
would depend on what needed to be done. As everyone mentioned, this
will depend on the context in which the operations are to be
performed.
I will most likely fatten my String<> class sufficiently to allow the
programmer to insert semantic indicators into the string objects
themselves, at run-time, so that operator == will know what to do when
it is applied to two string objects.
-Le Chaud Lapin-