Unicode Support
Chew...@austarnet.com.au
Something that has been bugging me, since I started on my own
compiler/assembler is unicode support. Not the API's or libraries, or
how unicode works (it's quite simple once you get your head around it),
but the fact that most assemblers (if not all assemblers), and most HLL
Compilers (that I've used) still require the source code to be 8bit
ASCII. (and the use of code-pages).
So for Randy, Rene, etc, are there any plans to allow source code in
UTF-16 format for your compilers/assemblers? eg to allow symbols/labels
to contain non-ASCII characters, and allow easier unicode string
support from within the source file itself?
eg to be able to support source code like this:
<code = FASM>
org 100h
старт:
mov ax, 9
mov dx, шнур
int 21h
ret
шнур du "여보세요 세계"
db "$"
шнур2 du "Γειάσου κόσμος"
db "$"
</code>
Obvisously all directives and operands should remain as they are (in
english as defined by Intel/AMD), but would be nice to have true
support for userdefined labels and strings.
<mini rant>
Since we are now in 2005, most modern OS's support unicode, why do the
base tools we use, are still insisting on ASCII source code? We all
want the "viva asm revolution" to happen, but one thing IMHO that we
are lacking is UTF-16 support for sourcecode. Would it give a one-up on
common HLL's. Well I don't know, but it will make asm more accessible
to more global users around the world.
</rant>
PS. If your assembler already supports UTF-16 based source code, I
would be deeply interested in hearing about some of the challenges in
implementing unicode support. In particular, did you limit numbers to
the western 0..9 figures, or did you allow other numbers to be
included, eg arabic, many of the asian sets, etc. Did you limit to
valid range of characters to the BMP (the first 64K characters only),
or did you allow for the full range of characters (1024K characters)
for labels. How did you handle compatible encodings, and combining
characters? What about UTF-8 vs UTF-16 vs UTF-32?
PPS. I know the DOS API doesn't support unicode strings, but just used
it for the example.
PPPS. The full Unicode 4.1 spec can be downloaded as PDF's from
www.unicode.org.
PPPPS. I use jEdit as my preferred text editor. (It's pure java so
should run on any java enabled platform, and supports UTF-16 natively).
Betov
@f14g2000cwb.googlegroups.com:
> So for Randy, Rene, etc, are there any plans to allow source code in
> UTF-16 format for your compilers/assemblers? eg to allow symbols/labels
> to contain non-ASCII characters, and allow easier unicode string
> support from within the source file itself?
First, addressing both, at a time, "Randy, Rene" is insulting
for me, but, well... here we go.
There is no plan for implementing Unicode in RosAsm,
neither in the Assembler, nor in the Sources Editor.
There is actually a Developement for the Internationalization
of the RosAsm Strings. We have done most of the job for
the Errors Messages, and there is a chineese version.
At the end, all of the RosAsm Messages Strings will be
Internationalized. This is done through external Files,
and, of course, the output Routines are also implemented
for Unicode.
There is also a Plan (not started), to enable the
"Unicode Users" to store Unicode Strings, in the
Sources Data. Actually, the only implemented thing
is that the RosAsm Data can be direct Unicode, like
this:
[MyUnicodeString: U§ 'BlaBla', 0]
that is same as:
[MyUnicodeString: B§ 'B', 0 'l', 0 'a', 0, 'B', 0, 'l', 0, 'a', 0, 0, 0]
I must apologioze that i have no idea, at all,
about how this is usable or not usable for the real
"Unicode Users". Maybe, this is enough for them
to create Aplications with Unicode Strings. I
really do not know, but i suspect the answer is
NO, because of the Unicode KeyStroke holding, that
has been implemented by a oriental user, a long
time ago, and about which i never got any information.
So, if i am told that this does not work, i will
implement a Mechanism, through the so helful "Tag"
feature, to run an external Unicode Editor, and to
paste the content of this Editor, into the Source,
under the form of a List of Bytes, or something
like this (back and forth, of course). In fact
i well hope that an oriental user will take care
of all of this instead of me.
As for writing, as you suggest, say, the labels
Names, in Unicode, directly in a RosAsm Source,
this will never be implemented. I do not see any
reason for doing so.
The Assembler with the best Unicode support is
GoAsm. Unfortunately, it is closed Sources, but
its Author is a very nice guy. If you get in touch
with him, by Mail, i am am quite sure he could give
you way better answers than me.
Betov.
webs...@gmail.com
> Something that has been bugging me, since I started on my own
> compiler/assembler is unicode support. Not the API's or libraries, or
> how unicode works (it's quite simple once you get your head around
> it),
Are you sure about that?
> but the fact that most assemblers (if not all assemblers), and most
> HLL Compilers (that I've used) still require the source code to be
> 8bit ASCII. (and the use of code-pages).
>
> So for Randy, Rene, etc, are there any plans to allow source code in
> UTF-16 format for your compilers/assemblers? eg to allow
> symbols/labels to contain non-ASCII characters, and allow easier
> unicode string support from within the source file itself?
I would *HIGHLY* recommend staying away from UTF-16 for raw source
data. Its properties are worse than those of UTF-8. UTF-8 gracefully
directly supports 7-bit ASCII as a proper subset.
OTOH, supporting string constants that are specified in UTF-16 using
some special mode like:
dw UTF16"Some data",0
or whatever you feel like (is there a useful standard here?) The point
being that the string would be stored in Windows compatible UTF-16.
UTF-16 is basically the "american" approach to Unicode (because it was
backed by Sun, Microsoft and IBM, but insanely, they thought 16 bits
was good enough for all characters -- which of course came as a bit of
a surprise to the Chinese/Japanese/Koreans; surrogates were added in
*afterwards* to deal with this), while UTF-8 is the more international
approach (although it was actually invented by Ken Thompson -- the
point of it is that it naturally allows for a much larger encoding
range than the original american Unicode, which better supports the
original ISO10646 standard which eventually just got folded into the
Unicode standard).
- UTF-16 requires a "BOM" at the beginning of any string to distinguish
endianness, which means that if you split a string into two pieces,
then transmit them, you end up gaining an additional "BOM" character
for the second piece.
- UTF-8 directly supports ASCII encoding as a sub-mode of its encoding.
I.e., normal ASCII encoded text is *already* UTF-8 compatible.
Certain ASCII functions like changing english text case, or searching
for ASCII characters can be done directly on UTF-8 data. So '\0'
termination, tabs, or things like CR and LF don't have strange
embodiments or representations, even when viewed with ASCII tools.
UTF-8 encodings are also easy to learn to recognize on sight, even with
ASCII tools.
- UTF-8 can be "resynched" even if a transfer channel is corrupted, or
if you start from the middle of the string after only a very short (I
think at most 5) character scan. Compare this to UTF-16, where if you
suddenly become offset by one character due to some flaw/error, you
will have no idea that your data is all corrupted for an unbounded
length of time.
Not surprisingly, Microsoft supports UTF-16 pervasively.
> <mini rant>
> Since we are now in 2005, most modern OS's support unicode, why
> do the base tools we use, are still insisting on ASCII source code?
> We all want the "viva asm revolution" to happen, but one thing IMHO
> that we are lacking is UTF-16 support for sourcecode. Would it give
> a one-up on common HLL's. Well I don't know, but it will make asm
> more accessible to more global users around the world.
> </rant>
Because Unicode support is harder than you think.
Read the documentation on normalization. This is important, because
comparison of two unicode strings does *NOT* reduce to simply comparing
the raw byte data. The Unicode encoding is actually redundant. I.e.,
obviously you have certain obvious uniqueness requirements for things
like labels and variable names (you can't declare the same label twice
in the same scope right?) So if someone renders the same unicode
string using two different code point sequences, then you have to have
correctly functioning equality testing (which is a little bit
complicated, and changes with each update of the Unidata.txt file from
the Unicode standard (which usually changes, if even slightly, with
every Unicode update).)
All this being said, Java, I believe, just supports UTF-16 based
Unicode right out of the box.
> PS. If your assembler already supports UTF-16 based source code, I
> would be deeply interested in hearing about some of the challenges
> in implementing unicode support. In particular, did you limit
> numbers to the western 0..9 figures, or did you allow other
> numbers to be included, eg arabic, many of the asian sets, etc.
> Did you limit to valid range of characters to the BMP (the first
> 64K characters only), or did you allow for the full range of
> characters (1024K characters) for labels. How did you handle
> compatible encodings, and combining characters? What about UTF-8
> vs UTF-16 vs UTF-32?
UTF-32 is mostly useful from a programming internal format. I.e., I
don't think supporting it for source code encodings is worth while at
all (since its so inefficient.) But for data encodings, I would
recommend supporting all three of them (since programmers may want to
use any of the modes in their programs).
> PPS. I know the DOS API doesn't support unicode strings, but just
> used it for the example.
Well that's actually a kind of non-trivial point. If you support
Unicode as datatype (no reason why you couldn't) there is the question
of what APIs do you intend to pass this data around in?
> PPPS. The full Unicode 4.1 spec can be downloaded as PDF's from
> www.unicode.org.
Yeah, so is version 4.0, 3.1, 3.0, ... etc. Taking a step back, one of
the real problems with Unicode is that its rate of evolution is
unusually high for such an important and universal standard.
--
Paul Hsieh
http://www.pobox.com/~qed/
http://bstring.sf.net/
rand...@earthlink.net
Chew...@austarnet.com.au wrote:
> Hi Everyone,
>
> Something that has been bugging me, since I started on my own
> compiler/assembler is unicode support. Not the API's or libraries, or
> how unicode works (it's quite simple once you get your head around
it),
> but the fact that most assemblers (if not all assemblers), and most
HLL
> Compilers (that I've used) still require the source code to be 8bit
> ASCII. (and the use of code-pages).
For good reason. Most popular *editing* tools support this data format.
>
> So for Randy, Rene, etc, are there any plans to allow source code in
> UTF-16 format for your compilers/assemblers? eg to allow
symbols/labels
> to contain non-ASCII characters, and allow easier unicode string
> support from within the source file itself?
Again, the availability of editing tools (i.e., the lack thereof)
creates a problem here. Sure, RosAsm could do this as RosAsm users are
*tied* to that editor, but for generic tools (most other assemblers),
you're limited by the editing and other text processing tools available
to you.
And even if you find a decent editor, all of a sudden all of the other
text processing tools you're using don't work.
Is there really a need to put unicode characters into identifiers?
Sure, non-English (non-Roman) characters in identifiers might be cool
in some countries, but I don't see that dramatically improving the
usability of an assembler (or other programming language).
BTW, keep in mind that GoAsm supports Unicode in this manner today. So
people who *absolutely* need this facility (e.g., Edgar, aka "Donkey")
have a tool they can use already. Why steal GoAsm's thunder?
>
> eg to be able to support source code like this:
>
> <code = FASM>
> org 100h
>
> старт:
> mov ax, 9
> mov dx, шнур
> int 21h
> ret
>
> шнур du "여보세요 세계"
> db "$"
> шнур2 du "Γειάσου κόσμος"
> db "$"
> </code>
>
> Obvisously all directives and operands should remain as they are (in
> english as defined by Intel/AMD), but would be nice to have true
> support for userdefined labels and strings.
You can do the above with resources, easy enough, in any existing
assembler. Granted, it's not as convenient, but it works.
>
> <mini rant>
> Since we are now in 2005, most modern OS's support unicode, why do
the
> base tools we use, are still insisting on ASCII source code?
Because those base tools cooperate with a lot of secondary tools, that
also use ASCII. Getting them all to change overnight isn't going to
happen.
And there is the issue of *all* programming languages, not just
assembly. When all the major HLLs support Unicode in a manner you
describe, when all editors support it, etc., etc., I think you'll find
that assemblers are going to support it as well.
Just keep in mind, there are some *significant* costs associated with
using Unicode that dramatically affect compiler performance. Beyond the
obivious "source files are larger" issue, things like case
insensitivity can get real nasty in Unicode. Issues like "does this
character belong in a particular set of characters" gets ugly. And, of
course, there are issues like hashing functions and what-not that have
to be re-though-out.
> We all
> want the "viva asm revolution" to happen, but one thing IMHO that we
> are lacking is UTF-16 support for sourcecode.
GoAsm has Unicode support already. And does a *good* job of it. If
Unicode were all that was holding the "viva asm revolution" back, it
would be happening today, with GoAsm leading the way.
> Would it give a one-up on
> common HLL's. Well I don't know, but it will make asm more accessible
> to more global users around the world.
> </rant>
Again, it's not just the assembler. It's all the supporting tools.
GoAsm succeeds in this area because Jeremy provides a complete suite of
tools.
>
> PS. If your assembler already supports UTF-16 based source code, I
> would be deeply interested in hearing about some of the challenges in
> implementing unicode support. In particular, did you limit numbers to
> the western 0..9 figures, or did you allow other numbers to be
> included, eg arabic, many of the asian sets, etc. Did you limit to
> valid range of characters to the BMP (the first 64K characters only),
> or did you allow for the full range of characters (1024K characters)
> for labels. How did you handle compatible encodings, and combining
> characters? What about UTF-8 vs UTF-16 vs UTF-32?
I think you're starting to get the idea. :-)
Think of the bugs that are going to wind up in your code because you
didn't consider certain character combinations.
> PPPPS. I use jEdit as my preferred text editor. (It's pure java so
> should run on any java enabled platform, and supports UTF-16
natively).
And what of people who want to use a different editor, that is not
Unicode enabled? Therein lies the big problem.
Cheers,
Randy Hyde
Chew...@austarnet.com.au
> Chew...@austarnet.com.au écrivait:
>
> > So for Randy, Rene, etc, are there any plans to allow source code
in
> > UTF-16 format for your compilers/assemblers? eg to allow
symbols/labels
> > to contain non-ASCII characters, and allow easier unicode string
> > support from within the source file itself?
>
>
> First, addressing both, at a time, "Randy, Rene" is insulting
> for me, but, well... here we go.
Ooh, please. Just because I mention 2 people on the NG that actively
develop assemblers together your offended! In no way, do I imply that
you are both of the same quality character or even ability in producing
assemblers, just the fact that you both actively develop assemblers.
> There is no plan for implementing Unicode in RosAsm,
> neither in the Assembler, nor in the Sources Editor.
>
> There is actually a Developement for the Internationalization
> of the RosAsm Strings. We have done most of the job for
> the Errors Messages, and there is a chineese version.
> At the end, all of the RosAsm Messages Strings will be
> Internationalized. This is done through external Files,
> and, of course, the output Routines are also implemented
> for Unicode.
>
> There is also a Plan (not started), to enable the
> "Unicode Users" to store Unicode Strings, in the
> Sources Data. Actually, the only implemented thing
> is that the RosAsm Data can be direct Unicode, like
> this:
Well, that was somewhat the point of my post. That any user should be
able to store any string from any language/script within a source file.
> [MyUnicodeString: U§ 'BlaBla', 0]
>
> that is same as:
>
> [MyUnicodeString: B§ 'B', 0 'l', 0 'a', 0, 'B', 0, 'l', 0, 'a', 0,
0, 0]
>
> I must apologioze that i have no idea, at all,
> about how this is usable or not usable for the real
> "Unicode Users".
Would you prefer:
"Hello world"
or
48h, 65h, 6ch, 6ch, 6fh, 020h, 77h, 6fh, 72h, 6ch, 64h
within your source file.
I would prefer the first form (as anyone would). Now take the view of
the Hindi student in India who doesn't know english. Are you going to
force the second option on him, so he can use your assembler?
(I know this is a poor example, but think about other languages, eg
hebrew, most asian, etc).
> As for writing, as you suggest, say, the labels
> Names, in Unicode, directly in a RosAsm Source,
> this will never be implemented. I do not see any
> reason for doing so.
So your source code will forever be ASCII?
Even cutting back on on where were the full unicode char set can be
used, and limiting unicode to strings and comments would be a good step
forward.
> The Assembler with the best Unicode support is
> GoAsm. Unfortunately, it is closed Sources, but
> its Author is a very nice guy. If you get in touch
> with him, by Mail, i am am quite sure he could give
> you way better answers than me.
I'll take a look at GoAsm.
Thanks for the input Rene! :D
wolfgang kern
"Chewy509" asked about UTF-source-code:
Even the idea may not find many friends, ..
*tool size may explode if several UTF16-sets needs support
*loss on source-portability and general readability
*Net/NG-bandwidth
(my news-reader use 'Courier New, text-only', so I'd see only garbage)
..my new disassembler got separated decoder/interpreter routines,
so it is already prepared to output any syntax with any character-set.
At the moment it just output ASCII, UTF could be added easy.
But I think English is the proper language for technicians and
programmers. Without a global communication standard we had to
rely on Babel-fish, and its name include 'Babylonian confusion' ;)
__
wolfgang
Chew...@austarnet.com.au
> Chew...@austarnet.com.au wrote:
> > Something that has been bugging me, since I started on my own
> > compiler/assembler is unicode support. Not the API's or libraries,
or
> > how unicode works (it's quite simple once you get your head around
> > it),
>
> Are you sure about that?
Having read the spec a few times, it doesn't seem that hard?
(combinations and normalisations do seem to be the most complex issues
though).
All encoding methods, whether that be UTF-8, UTF-16 or UTF-32 handle
the same number of encodings, (1024K), just the way that they are held
in memory/disk is a little different. I wouldn't call UTF-8 "the more
international approach", but rather "the lets add backwards
compatiblity for the those that still believe ASCII is the only real
encoding format approach".
> - UTF-16 requires a "BOM" at the beginning of any string to
distinguish
> endianness, which means that if you split a string into two pieces,
> then transmit them, you end up gaining an additional "BOM" character
> for the second piece.
A BOM is not necessary, when the encoding is known or dictated by
implementation. Since assemblers generally operate on the target CPU (I
know there are exceptions, such as cross-compilers and
cross-assemblers), it can be fair to say that the format can be
dictated by the assembler, whether that be UTF-16BE or UTF-16LE (LE for
x86).
> - UTF-8 directly supports ASCII encoding as a sub-mode of its
encoding.
> I.e., normal ASCII encoded text is *already* UTF-8 compatible.
> Certain ASCII functions like changing english text case, or searching
> for ASCII characters can be done directly on UTF-8 data. So '\0'
> termination, tabs, or things like CR and LF don't have strange
> embodiments or representations, even when viewed with ASCII tools.
> UTF-8 encodings are also easy to learn to recognize on sight, even
with
> ASCII tools.
Granted.
> - UTF-8 can be "resynched" even if a transfer channel is corrupted,
or
> if you start from the middle of the string after only a very short (I
> think at most 5) character scan. Compare this to UTF-16, where if
you
> suddenly become offset by one character due to some flaw/error, you
> will have no idea that your data is all corrupted for an unbounded
> length of time.
Are you sure?
UTF-16 is easier than UTF-8 (and only requires 1 backstep at most), and
this doesn't apply to UTF-32 at all. The way surrogate pairs work in
UTF-16, it's just a quick test to see if a bit is 1 or 0 to determine
first or second character.
UTF-16 encodes 0-FFFF into 1 word, and 10000-10FFFF into 2 words as:
110110wwwwxxxxxx 110111xxxxxxxxxx
where wwww = top 4 bits - 1
and xxxx = lower 16 bits.
so if the first 6 bits are 110110 then we have the first of a pair,
else if the 6bits are 110111 then we have a second unit of a pair. Any
other encoding on the 6bits tells me I have a normal character.
The two ranges covered by the surrogate pairing used in UTF-16 are
reserved ranges within the character set, so there can be no overlap
with assigned encodings.
> Not surprisingly, Microsoft supports UTF-16 pervasively.
Surprising MS adopted Unicode *before* Unicode 1.0 was finalised.
(Which is what lead to some of the quirks of the MS Unicode
implementation).
<snip>
> > PS. If your assembler already supports UTF-16 based source code, I
> > would be deeply interested in hearing about some of the challenges
> > in implementing unicode support. In particular, did you limit
> > numbers to the western 0..9 figures, or did you allow other
> > numbers to be included, eg arabic, many of the asian sets, etc.
> > Did you limit to valid range of characters to the BMP (the first
> > 64K characters only), or did you allow for the full range of
> > characters (1024K characters) for labels. How did you handle
> > compatible encodings, and combining characters? What about UTF-8
> > vs UTF-16 vs UTF-32?
>
> UTF-32 is mostly useful from a programming internal format. I.e., I
> don't think supporting it for source code encodings is worth while at
> all (since its so inefficient.) But for data encodings, I would
> recommend supporting all three of them (since programmers may want to
> use any of the modes in their programs).
I agree. UTF-32 for source would be a waste. Internally, it would help
somewhat, but I don't think the overhead for the <1% of cases would be
worth it. The Unicode standard does allow for implementations only to
handle the BMP (U+0000 .. U+FFFF), and still be conformant. Maybe
that's an option?
> > PPS. I know the DOS API doesn't support unicode strings, but just
> > used it for the example.
>
> Well that's actually a kind of non-trivial point. If you support
> Unicode as datatype (no reason why you couldn't) there is the
question
> of what APIs do you intend to pass this data around in?
>
> > PPPS. The full Unicode 4.1 spec can be downloaded as PDF's from
> > www.unicode.org.
>
> Yeah, so is version 4.0, 3.1, 3.0, ... etc. Taking a step back, one
of
> the real problems with Unicode is that its rate of evolution is
> unusually high for such an important and universal standard.
However the standard does make references for further feature
compatibility. Eg any implementation that is Unicode 3.0 conformant,
will also be Unicode 4.x conformant. While it is an issue, I don't
believe there is enough risk evolved to warrant too much time on it.
Darran (aka Chewy509).
Beth
> Something that has been bugging me, since I started on my own
> compiler/assembler is unicode support. Not the API's or libraries, or
> how unicode works (it's quite simple once you get your head around it),
> but the fact that most assemblers (if not all assemblers), and most HLL
> Compilers (that I've used) still require the source code to be 8bit
> ASCII. (and the use of code-pages).
>
> So for Randy, Rene, etc, are there any plans to allow source code in
> UTF-16 format for your compilers/assemblers? eg to allow symbols/labels
> to contain non-ASCII characters, and allow easier unicode string
> support from within the source file itself?
>
> <code = FASM>
> org 100h
>
> старт:
> mov ax, 9
> mov dx, шнур
> int 21h
> ret
>
> шнур du "여보세요 세계"
> db "$"
> шнур2 du "Γειάσου κόσμος"
> db "$"
> </code>
> Obvisously all directives and operands should remain as they are (in
> english as defined by Intel/AMD), but would be nice to have true
> support for userdefined labels and strings.
> <mini rant>
> Since we are now in 2005, most modern OS's support unicode, why do the
> base tools we use, are still insisting on ASCII source code? We all
> want the "viva asm revolution" to happen, but one thing IMHO that we
> are lacking is UTF-16 support for sourcecode. Would it give a one-up on
> common HLL's. Well I don't know, but it will make asm more accessible
> to more global users around the world.
> </rant>
> PS. If your assembler already supports UTF-16 based source code, I
> would be deeply interested in hearing about some of the challenges in
> implementing unicode support.
> the western 0..9 figures, or did you allow other numbers to be
> included, eg arabic, many of the asian sets, etc. Did you limit to
> valid range of characters to the BMP (the first 64K characters only),
> or did you allow for the full range of characters (1024K characters)
> for labels. How did you handle compatible encodings, and combining
> characters? What about UTF-8 vs UTF-16 vs UTF-32?
║ And don't forget the "box drawing" characters!! ║
╚═════════════════════════════════════════════════╝
> it for the example.
> PPPS. The full Unicode 4.1 spec can be downloaded as PDF's from
> PPPPS. I use jEdit as my preferred text editor. (It's pure java so
> should run on any java enabled platform, and supports UTF-16 natively).
Chew...@austarnet.com.au
> "Chewy509" asked about UTF-source-code:
>
> Even the idea may not find many friends, ..
Hi Wolfgang,
That seems to be what's happening... Great in theory, but trying to
make it happen, well that's another story.
> *tool size may explode if several UTF16-sets needs support
Memory and disk usage, yes. (keeping related to assemblers/compilers),
the only real parts may be the tokeniser, parsing routines, and hashing
algorithm.
Even limiting the allowed areas of where the full range of unicode
characters, to say strings and comments, will make a transition a lot
easier. eg make the restriction that labels must only contain
'a..z,A..Z,_,0..9' and numbers only '0..9', directives/operands remain
as Intel/AMD defined, strings can contain the full unicode set.
Would doing this make a great difference to the actual internals of any
assembler or compiler?
> *loss on source-portability and general readability
Unicode support:
MacOS - yes
Linux and most modern UNIX - yes (UTF-8), even GNU Emacs supports
UTF-8.
Windows - yes
Even notepad.exe in Windows XP supports unicode! (both UTF-8 and
UTF-16)
> *Net/NG-bandwidth
> (my news-reader use 'Courier New, text-only', so I'd see only
garbage)
I'll give you that one.
> ..my new disassembler got separated decoder/interpreter routines,
> so it is already prepared to output any syntax with any
character-set.
> At the moment it just output ASCII, UTF could be added easy.
>
> But I think English is the proper language for technicians and
> programmers. Without a global communication standard we had to
> rely on Babel-fish, and its name include 'Babylonian confusion' ;)
:D. I can definately see the point on the last one!
Darran (aka Chewy509).
Beth
when sending HTML, it duplicates the post in plain text and in HTML...so,
it's included twice...what's the idea there exactly?
Using UTF-8 encoding directly should, of course, work...but Alexei tried
that on alt.os.development and it didn't work...some of the "propogation"
software out there seems to screw up UTF-8 encodings (trimming off or
rejecting the 8th bit or something)...so, if you miss those, it'll
work...if you don't, then it won't...
Just to test whether I needed or didn't need to send HTML in future:
???????
??ee?
Hiragana - ????
Ideographs:
????
Mathematical operators:
?????Ø????????
Dingbats:
??????????????
Thai:
????????
Tamil:
?????????
Hebraic:
?????????
Miscellaneous:
???
What does and doesn't show up here?
Beth :)
Beth
> What does and doesn't show up here?
None of them...great...just great...
Beth :)
Chew...@austarnet.com.au
> Chew...@austarnet.com.au wrote:
> > Hi Everyone,
> >
> > Something that has been bugging me, since I started on my own
> > compiler/assembler is unicode support. Not the API's or libraries,
or
> > how unicode works (it's quite simple once you get your head around
> it),
> > but the fact that most assemblers (if not all assemblers), and most
> HLL
> > Compilers (that I've used) still require the source code to be 8bit
> > ASCII. (and the use of code-pages).
>
> For good reason. Most popular *editing* tools support this data
format.
However don't most Windows based editors also support unicode, GNU
Emacs supports unicode (UTF-8 encoding)?
> >
> > So for Randy, Rene, etc, are there any plans to allow source code
in
> > UTF-16 format for your compilers/assemblers? eg to allow
> symbols/labels
> > to contain non-ASCII characters, and allow easier unicode string
> > support from within the source file itself?
>
> Again, the availability of editing tools (i.e., the lack thereof)
> creates a problem here. Sure, RosAsm could do this as RosAsm users
are
> *tied* to that editor, but for generic tools (most other assemblers),
> you're limited by the editing and other text processing tools
available
> to you.
Well, as you read from Rene's post, he doesn't think he'll support it.
(Message for Rene: Want to get one up on Randy. Support Unicode source
files! He doesn't seem to be interested.)
(Just stirring the pot :P).
> And even if you find a decent editor, all of a sudden all of the
other
> text processing tools you're using don't work.
>
> Is there really a need to put unicode characters into identifiers?
> Sure, non-English (non-Roman) characters in identifiers might be cool
> in some countries, but I don't see that dramatically improving the
> usability of an assembler (or other programming language).
Fair enough, but even if you limit full support to strings and
comments, wouldn't that be a step in the right direction. (As I posted
to reply to Wolfgang, leave labels/identifiers, etc as is, but allow
unicode in strings and comments).
<snip>
> >
> > <mini rant>
> > Since we are now in 2005, most modern OS's support unicode, why do
> the
> > base tools we use, are still insisting on ASCII source code?
>
> Because those base tools cooperate with a lot of secondary tools,
that
> also use ASCII. Getting them all to change overnight isn't going to
> happen.
But let's get the ball rolling. :)
> And there is the issue of *all* programming languages, not just
> assembly. When all the major HLLs support Unicode in a manner you
> describe, when all editors support it, etc., etc., I think you'll
find
> that assemblers are going to support it as well.
But why should the assemblers play second fiddle to the HLL compilers.
Why not have the assemblers lead the way!
<snip>
> > PS. If your assembler already supports UTF-16 based source code, I
> > would be deeply interested in hearing about some of the challenges
in
> > implementing unicode support. In particular, did you limit numbers
to
> > the western 0..9 figures, or did you allow other numbers to be
> > included, eg arabic, many of the asian sets, etc. Did you limit to
> > valid range of characters to the BMP (the first 64K characters
only),
> > or did you allow for the full range of characters (1024K
characters)
> > for labels. How did you handle compatible encodings, and combining
> > characters? What about UTF-8 vs UTF-16 vs UTF-32?
>
> I think you're starting to get the idea. :-)
> Think of the bugs that are going to wind up in your code because you
> didn't consider certain character combinations.
>
I realise my initial post, sounded a little newbish. However I have
given some thought it, and was wondering why others hadn't implemented
support yet.
> > PPPPS. I use jEdit as my preferred text editor. (It's pure java so
> > should run on any java enabled platform, and supports UTF-16
> natively).
>
> And what of people who want to use a different editor, that is not
> Unicode enabled? Therein lies the big problem.
:(
Darran (aka Chewy509).
rand...@earthlink.net
Chewy...@austarnet.com.au wrote:
> rand...@earthlink.net wrote:
> > For good reason. Most popular *editing* tools support this data
> format.
>
> However don't most Windows based editors also support unicode, GNU
> Emacs supports unicode (UTF-8 encoding)?
Not that I'm aware of. No doubt that EMACS does, though.
>
> Well, as you read from Rene's post, he doesn't think he'll support
it.
> (Message for Rene: Want to get one up on Randy. Support Unicode
source
> files! He doesn't seem to be interested.)
>
> (Just stirring the pot :P).
He can have it. :-)
Seriously, "pioneers take the arrows, settlers take the land." I'd
rather be a settler and add UTF-x support once it's proven to be
needed.
BTW, HLA (1 & 2) both support Unicode string constants. Though neither
compiler will support source code that is not seven-bit ASCII.
>
> > And even if you find a decent editor, all of a sudden all of the
> other
> > text processing tools you're using don't work.
> >
> > Is there really a need to put unicode characters into identifiers?
> > Sure, non-English (non-Roman) characters in identifiers might be
cool
> > in some countries, but I don't see that dramatically improving the
> > usability of an assembler (or other programming language).
>
> Fair enough, but even if you limit full support to strings and
> comments, wouldn't that be a step in the right direction. (As I
posted
> to reply to Wolfgang, leave labels/identifiers, etc as is, but allow
> unicode in strings and comments).
As I said, HLA does support 16-bit characters, much like the L"string"
format in MSVC. Granted, if you want to put a non-ASCII character into
that string you have to supply the numeric code for it, but both HLA
v1.x and v2.0 support wide characters today.
> > Getting them all to change overnight isn't going to
> > happen.
>
> But let's get the ball rolling. :)
Like I said, I prefer not to be the pioneer on this one :-) But when
the market demands it, I'm sure I'll be forced to add it. It's just
that the market isn't demanding it today (and those who *do* need it do
have GoAsm).
>
> But why should the assemblers play second fiddle to the HLL
compilers.
> Why not have the assemblers lead the way!
Because there are so many other things, of higher priority, that
assemblers still need. In my personal case, for example, I need to get
HLA v2.0 functioning. Beyond that, I'd like to explore adding an
optimizer to assembly language to help address the fact that certain
optimizations (e.g., instruction scheduling) are *much* more easily
done by machine than by hand. That's something that I believe will help
attract a lot of newcomers to assembly language. Currently, the belief
is that a HLL optimizer produces much better code than a beginning
assembly language programmer. Therefore, beginners aren't willing to
put in any effort into learning assembly because it will take a long
time before they can beat the compiler. However, if they could believe
that they could produce code that is *at least* comparable to that put
out by a compiler in a short period of time, they might be more willing
to take a look at assembly. To me, that's a higher priority than
Unicode source support. But, if the world changes overnight and Unicode
becomes highly desired by a large number of people, I'll certainly
revisit this issue.
Cheers,
Randy Hyde
arargh5...@now.at.arargh.com
<BethS...@hotmail.NOSPICEDHAM.com> wrote:
>Crap, I ain't sending nothing like that again...for no particular reason,
>when sending HTML, it duplicates the post in plain text and in HTML...so,
>it's included twice...what's the idea there exactly?
It's because you use: "Microsoft Outlook Express" and that's the way
they do it.
--
Arargh504 at [drop the 'http://www.' from ->] http://www.arargh.com
BCET Basic Compiler Page: http://www.arargh.com/basic/index.html
To reply by email, remove the garbage from the reply address.
Chew...@austarnet.com.au
<snip>
Hi Beth,
Thanks for the entertaining post.
I had thought about many of the issues, including directionality of
text, but as you said, an assembler/compiler should just see the
logical order. Some of the other issues, such as non-english labels and
numbers, can be a pain in the arse. Numbers not so much (AFIAK most of
the world recognises the 0..9 numerical set), but labels. The biggest
issue for me, would be combinations or equivalence. How should those be
handled? Well, it appears that the answer (for LuxAsm) is don't. Just
have labels/identifiers in the normal 7bit ASCII range as already
defined by most other assemblers/HLLs.
Darran (aka Chewy509).
webs...@gmail.com
> webs...@gmail.com wrote:
> > Chew...@austarnet.com.au wrote:
> > > Something that has been bugging me, since I started on my
> > > own compiler/assembler is unicode support. Not the API's
> > > or libraries, or how unicode works (it's quite simple
> > > once you get your head around it),
> >
> > Are you sure about that?
>
> Having read the spec a few times, it doesn't seem that hard?
> (combinations and normalisations do seem to be the most
> complex issues though).
Yeah see ... you kind of HAVE to implement that stuff (actually you
probably just need normalization). Otherwise how are you going to know
if two Unicode strings are the same? (As I said, just the ordinary
action of symbol matching boils down to writing a string comparison
function for Unicode strings -- did you know that you can write e with
an accent grave on it in two different ways?)
> All encoding methods, whether that be UTF-8, UTF-16 or UTF-32
> handle the same number of encodings, (1024K), just the way
> that they are held in memory/disk is a little different.
Actually its slightly more than 1024K. But currently only about 100K
of them are assigned.
> [...] I wouldn't call UTF-8 "the more
> international approach", but rather "the lets add backwards
> compatiblity for the those that still believe ASCII is the
> only real encoding format approach".
Right -- the point is that even internationally, ASCII is used
somewhat. UTF-16 is basically telling everyone "ok we all got to start
from scratch all over again!" and was pushed, mainly by an american
consortium. UTF-16 is a generalization of the original UCS-2, which
only encoded code points up to 0xffff (the BMP? Who cares, its
obsolete) and which was fine for Microsoft, et al's first attempt at
internationalization. This worked great for americans, and europeans,
but kind of left the asians with a bad taste in their mouth. UTF-8 is
not tied to a stunted encoding -- it was right from the very beginning.
> > - UTF-16 requires a "BOM" at the beginning of any string to
> > distinguish endianness, which means that if you split a string
> > into two pieces, then transmit them, you end up gaining an
> > additional "BOM" character for the second piece.
>
> A BOM is not necessary, when the encoding is known or dictated
> by implementation. Since assemblers generally operate on the
> target CPU (I know there are exceptions, such as cross-compilers
> and cross-assemblers), it can be fair to say that the format can
> be dictated by the assembler, whether that be UTF-16BE or
> UTF-16LE (LE for x86).
Ok, first of all, cross assemblers exist just as much as cross
compilers. Second of all, this is about what your *text editor*
supports, not your OS. The text editors will likely output a BOM
character, and will likely support *both* endians.
> > - UTF-8 directly supports ASCII encoding as a sub-mode of
> > its encoding. I.e., normal ASCII encoded text is *already*
> > UTF-8 compatible. Certain ASCII functions like changing
> > english text case, or searching for ASCII characters can
> > be done directly on UTF-8 data. So '\0' termination,
> > tabs, or things like CR and LF don't have strange
> > embodiments or representations, even when viewed with
> > ASCII tools. UTF-8 encodings are also easy to learn to
> > recognize on sight, even with ASCII tools.
>
> Granted.
>
> > - UTF-8 can be "resynched" even if a transfer channel is
> > corrupted, or if you start from the middle of the string
> > after only a very short (I think at most 5) character
> > scan. Compare this to UTF-16, where if you suddenly
> > become offset by one character due to some flaw/error,
> > you will have no idea that your data is all corrupted
> > for an unbounded length of time.
>
> Are you sure?
Yeah, that's how it was designed; let me check. If the top bit is 0,
then the byte is a character in the range 0x00-0x7f (i.e., its ASCII).
If the top two bits are 11, then the byte is the *start* of a multibyte
encoding of a UTF-8 character. If the top two bits are 10 then it is a
continuation character at there are at most *2* continuation characters
following it (for the legal Unicode range.) So, in fact, its at most 3
characters for a resynch of a unidirectional channel, or a at most two
characters in each direction in additional to the current character
(that's 5 total) for a bidirectional channel.
> UTF-16 is easier than UTF-8 (and only requires 1 backstep
> at most), and this doesn't apply to UTF-32 at all. The
> way surrogate pairs work in UTF-16, it's just a quick
> test to see if a bit is 1 or 0 to determine first or
> second character.
No, no, no. A channel is usually encoded as bytes. If you get offset
by a single byte, then you will just read different data as UTF-16.
Since most of the encodings of UTF-16 are in fact completely legal and
since it does not have any kind of self-integrity encoding features
like UTF-8 does, you will likely have no inkling that an error has
occurred until far far later that the original error actually occurred.
You cannot assume that "errors" will just nicely cause you to skip an
even number of bytes all the time.
> > Not surprisingly, Microsoft supports UTF-16 pervasively.
>
> Surprising MS adopted Unicode *before* Unicode 1.0 was
> finalised. (Which is what lead to some of the quirks of
> the MS Unicode implementation).
Yes, as I said, Microsoft, and a number of other american companies
really got on the bandwagon of Unicode waaaaaaay too soon. In fact I
would consider the *current* standard, the equivalent of a Beta release
(whereas each earlier release was an Alpha, or prototype.) ISO 10646
was clearly the superior evolution path.
No its not. That's the *mistake* Microsoft and Sun started with. Its
just an insult to the asians to ignore the encodings beyond U+FFFD
(U+FFFE and U+FFFF are illegal.) Remember, that Unicode people are
writing things in a way to try to try to cover up their mistakes from
the past. Just do it the right way, and let Microsoft deal with the
legacy problems they've made for themselves. At least UTF-16 is
backward compatible with the old Unicode standard.
> > > PPS. I know the DOS API doesn't support unicode
> > > strings, but just used it for the example.
> >
> > Well that's actually a kind of non-trivial point. If you
> > support Unicode as datatype (no reason why you couldn't)
> > there is the question of what APIs do you intend to pass
> > this data around in?
> >
> > > PPPS. The full Unicode 4.1 spec can be downloaded as
> > > PDF's from www.unicode.org.
> >
> > Yeah, so is version 4.0, 3.1, 3.0, ... etc. Taking a step
> > back, one of the real problems with Unicode is that its
> > rate of evolution is unusually high for such an important
> > and universal standard.
>
> However the standard does make references for further
> feature compatibility. Eg any implementation that is Unicode
> 3.0 conformant, will also be Unicode 4.x conformant. While
> it is an issue, I don't believe there is enough risk evolved
> to warrant too much time on it.
There are a number of proposed character sets still pending for
approval for the next update. (One includes an interesting alphabet
designed by some University of Toronto researchers to help retarded
people.) The standard does not say anything about whether or not any
future encodings will include other redundant encodings that further
complicates normalization.
wolfgang kern
Hi Beth,
| > What does and doesn't show up here?
|
| None of them...great...just great...
You expected something else? :)
I see every news-text with 'my' "Courier New 14pt" setting,
regardless of the character-set you may have typed it.
btw: You just filled my HD with Kenji/Russian-patterns?
Don't care, I have the whole UTF-book already.
__
wolfgang
wolfgang kern
Hi Darran,
| That seems to be what's happening... Great in theory, but trying to
| make it happen, well that's another story.
:) As usual, don't give up too early.
| > *tool size may explode if several UTF16-sets needs support
| Memory and disk usage, yes. (keeping related to assemblers/compilers),
| the only real parts may be the tokeniser, parsing routines, and hashing
| algorithm.
The whole source-code would double its size?
| Even limiting the allowed areas of where the full range of unicode
| characters, to say strings and comments, will make a transition a lot
| easier. eg make the restriction that labels must only contain
| 'a..z,A..Z,_,0..9' and numbers only '0..9', directives/operands remain
| as Intel/AMD defined, strings can contain the full unicode set.
IIRC, one of the RosAsm guys tried to add Chinese pattern support
onto the RosAsm-source. Rene may know more about it.
| Would doing this make a great difference to the actual internals of any
| assembler or compiler?
May not be too many work to implement it, but I'm afraid it will need
many good arguments to convince assembler authors to go for a change.
Especially as compiling speed (a competitive factor these days)
will be affected ;)
| > *loss on source-portability and general readability
| Unicode support:
| MacOS - yes
| Linux and most modern UNIX - yes (UTF-8), even GNU Emacs supports
| UTF-8.
| Windows - yes
| Even notepad.exe in Windows XP supports unicode! (both UTF-8 and
| UTF-16)
| > *Net/NG-bandwidth
| > (my news-reader use 'Courier New, text-only', so I'd see only garbage)
| I'll give you that one.
Ok :)
But I mean the readabilty from existing tools, not all tools use
OS-features (perhaps seen as bloated overhead) in their editors.
| > But I think English is the proper language for technicians and
| > programmers. Without a global communication standard we had to
| > rely on Babel-fish, and its name include 'Babylonian confusion' ;)
|
| :D. I can definately see the point on the last one!
Either way, I think if one assembler will start with UTF-support
all others will be forced to follow.
Even the use of UTF for programming tools seems not to be required,
I think about to change from 'my personal' ASCII(>7Fh)-set to a few
common readable UTF-sets (Math-symbols/Greek/Logic/Bool).
But I may also decide to just distribute my current character set.
__
wolfgang
Annie
On 2005-04-20 BethS...@hotmail.NOSPICEDHAM.com wrote:
> 2279 lines
Crikey. It's ANOTHER NEW RECORD!
Except it doesn't count. That HTML crap is cheating...
not to mention a violation of every existing 'Net
standard. Hehehe!
> ...I've tried sending UTF-8 plain text and such before but it
> didn't work (some stupid propogation software stripping the
> 8th bit when no-one asked it to, no doubt ;)...
_____
Duhhh! Usenet is a ((( `\
7-bit medium, Beth. _ _`\ )
It worked exactly (^ ) )
as it's supposed to. ~-( )
_'((,,,)))
And 173 trees were ,-' \_/ `\
killed to produce the ( , |
power necessary to `-.-'`-.-'/|_|
propagate this spam-ish \ / | |
message of yours. =()=: / ,' aa
Shame!
BTW, the whole idea of
assemblers/compilers
supporting 'Unicode'
is totally nutty, prima
facie...IMNSHO.
Beth
> Beth wrote:
> >Crap, I ain't sending nothing like that again...for no particular
reason,
> >when sending HTML, it duplicates the post in plain text and in
HTML...so,
> >it's included twice...what's the idea there exactly?
>
> It's because you use: "Microsoft Outlook Express" and that's the way
> they do it.
True, true...it's my own fault for using their crap, isn't it? Mea Culpa!
;)
Beth :)
Beth
> wolfgang kern wrote:
> > "Chewy509" asked about UTF-source-code:
> >
> > Even the idea may not find many friends, ..
>
> Hi Wolfgang,
>
> That seems to be what's happening... Great in theory, but trying to
> make it happen, well that's another story.
Ah, but much of it is "imagined problems" rather than actual ones...more a
case of "I like things how they are", if you know what I mean ;)...
> > *tool size may explode if several UTF16-sets needs support
>
> Memory and disk usage, yes. (keeping related to assemblers/compilers),
> the only real parts may be the tokeniser, parsing routines, and hashing
> algorithm.
>
> Even limiting the allowed areas of where the full range of unicode
> characters, to say strings and comments, will make a transition a lot
> easier. eg make the restriction that labels must only contain
> 'a..z,A..Z,_,0..9' and numbers only '0..9', directives/operands remain
> as Intel/AMD defined, strings can contain the full unicode set.
>
> Would doing this make a great difference to the actual internals of any
> assembler or compiler?
LuxAsm intends to support UTF-8 directly...and if we don't include symbols,
then the processing is actually very simple indeed...commments: they are
ignored, anyway...carry on until we reach "newline" (or "close comment" for
multi-line comments :)...strings: pass it through "as is" (possibly a
simple UTF-8 to UTF-16 or uTF-32, if "dw" or "dd" is used...but swapping
encodings is not very difficult :)...
The assembler is not where the actual problem lies...it's our _source code
editor_ that's going to be the "bitch" to code (though, X itself
comprehends UTF-8 and has UNICODE fonts...but some "intelligence" in how to
display the file on the screen might need to be added :)...
As for taking up more memory and disk space: UTF-8 does not take up a
single bit extra from ASCII for any ordinary ASCII characters...and the
other point, of course, is that you are _getting something_ for your
"money", so to speak...all those extra characters...plus, there is, of
course, NO compulsion on any program assembled by the assembler to use
anything but ASCII, if it doesn't want to do so...but it can process that
when it's there...
A good implementation shouldn't take appreciably more (on the assembler's
side of things)...with UTF-8 encoding, then it's NO DIFFERENT from ASCII,
while you're only using ASCII characters (so all your current ASCII source
code: Completely uneffected, exactly the same size as before...indeed, you
need do NOTHING to them at all...they will work "as is" :)...even with
"case insensitivity", the look-up table for that needn't list the entire
UNICODE range but just include "ranges" of "uppercase" and "lowercase"
characters (they are, in general, all put together, just like all the
uppercase and lowercase are next to each other in ASCII :)...stored as, for
example, "start: 'A', end: 'Z'" and that covers all the ASCII "uppercase"
in one go, so if you did that all the way along then it really wouldn't be
a big "look-up table" for the "case" either...all other characters can be
assumed to have "no case" (which is the vast majority of
characters..."case" only exists in some languages :)...if "case sensitive"
then you can ignore all that extra complication completely, as every
character is considered "unique" to all the others...
No; With a sensible implementation, then it can be made that it takes
appreciably little - if any extra at all (ASCII source code with UTF-8
remains completely unchanged) - to support, without making anyone's source
code any bigger...indeed, with UTF-8, it only gets "bigger" _IF_ you're
actually using non-ASCII characters...and if you're using them, then, well,
presumably you _WANT_ those characters in your program...the Japanese
character take more room: If that's a problem for you, then don't use
Japanese characters...if you _must_ use Japanese characters (because it's a
Japanese program :), then, you know, the extra is the "price you pay" for
that...even if using a non-UNICODE solution, things like "double byte
character sets" take up more room...simply: Japanese has more than 256
characters in it, so you're going to be using more than a byte, whatever
method you use...
> > *loss on source-portability and general readability
>
> Unicode support:
> MacOS - yes
> Linux and most modern UNIX - yes (UTF-8), even GNU Emacs supports
> UTF-8.
> Windows - yes
>
> Even notepad.exe in Windows XP supports unicode! (both UTF-8 and
> UTF-16)
Yeah, this is actually a "reverse" point on what's actually true...the
whole purpose of UNICODE is that it _gets rid_ of all the things like "code
page incompatibilities" once and for all...one standard character set for
_everyone_...
Perhaps the appropriate quote from Tolkien, who was a language scholar and
invented his own languages and scripts (indeed, wrote "the Lord of the
Rings" as a "background story" for where his languages came from because of
a belief that a language and its culture go hand-in-hand - one effects the
other - that you couldn't really invent a language without also inventing
the people who spoke it :):
"One Ring to rule them all,
One Ring to find them,
One Ring to bring them all...
...and, in the darkness, bind them"
> > *Net/NG-bandwidth
> > (my news-reader use 'Courier New, text-only', so I'd see only
> garbage)
>
> I'll give you that one.
Yes; The news software does not work particularly well, as I've just
discovered...UTF-8 doesn't always post properly (well, for some people,
your UTF-8 came through just fine, Chewy :)...and, if you choose "HTML",
then it's the most horrid implementation that duplicates the post content
twice, once in plain text, once in HTML...
Of course, if only everything moved to, say, UTF-8 (there is no need for
anything else because UNICODE guarantees "round-trip conversion" to all
other "code pages" out there...that is, if it exists in any other
"standard", then it's in UNICODE because the whole point, in fact, was to
"merge" all the standards into "one big one" :)...then we could all make
that _ONE_ change and never have to make a change ever again...
As for actually running the software in text-mode...well, it can simply
print "?" for what it cannot show...or, an interesting solution might be to
use a graphics mode, even for your "text-mode" console...divide the
_graphics_ screen into an 80 x 25 grid and use a "fixed size" font for that
grid...this isn't as silly as it sounds, as this is exactly what the BIOS
is doing when you use a DOS prompt or call the "write string" routines
while the screen is in a graphics mode like VGA mode 12h or mode 13h...
But this is the biggest problem, true...but it's a problem of "my software
is not up-to-date with UNICODE"...yes, but isn't the point Chewy's making
about: "let's all update things to use UNICODE from now on"?
The actual "transition" is always a problem...but one of the things about
UNICODE (why all the OSes listed above have just moved completely over to
it :), is that once you make the "transition", you shouldn't ever need to
make one again...
> > ..my new disassembler got separated decoder/interpreter routines,
> > so it is already prepared to output any syntax with any
> > character-set.
> > At the moment it just output ASCII, UTF could be added easy.
> >
> > But I think English is the proper language for technicians and
> > programmers. Without a global communication standard we had to
> > rely on Babel-fish, and its name include 'Babylonian confusion' ;)
>
> :D. I can definately see the point on the last one!
_THIS_ is the real problem, of course...
BUT is this a problem that's the tool's responsibility? For instance,
programming a protected mode OS is very confusing too...or using "direct
access" to hardware...should tools also start telling us we can't use "LGDT
/ LIDT" or "IN / OUT" instructions too? As they might cause "Babylonian
confusion"?
Or should the tools _support_ these things...but then it's up to the
programmers to come up with their own "policies" to avoid "Babylonian
confusion" themselves? That is, for an international "open source" project,
everyone agrees to keep all the code, labels, comments and such in
English...
But why should a tool be forcing a Russian programmer, writing a Russian
closed source program for Russian users, be _forced_ to speak English by a
tool? Isn't that a bit silly? You're forcing a "general case" onto a
"specific case", whether it fits or not...
I think it should be _separate_...the tool just does its job...the
programmers should sort out the "Babylonian confusion" problems
themselves...after all, we're all speaking English here in order to
understand each other, right? No-one is actually "forcing" us to do that
because, well, this is an unmoderated newgroup and you could post whatever
you like here...but we all naturally "agree" on this so we can understand
each other...yet, also, there is a specific German language CLAX, where
those who want to talk in German can do so too...you know, this allows for
all the possibilities...but if there was some "forced rule" that all
newsgroups must speak English? No, the point is not only to make the
correct "rules" but to apply them in the correct places...
The tool isn't the right place for this (after all, how can a tool tell if
you're typing in English or typing in German or typing in French? They all
use Latin characters...it's just NOT QUALIFIED to make these "judgements",
even if we'd like it to do so)...the programmers themselves should "agree
standards" for their projects...
After all, "the Tower of Babel" still exists in the world...we all speak
many languages in this newsgroup...but if we "agree" to stick with English,
then we can all understand each other...we don't need any "Big Brother"
software, trying to "guess" if we're speaking English or Spanish or Italian
and then trying to "ban" anyone from speaking anything but English...no, we
can do that kind of thing ourselves...we _DO_ do that kind of thing right
here and right now...and no-one twisted our arms to do it...we agree on it
because, simply, if we don't agree on that, then things become "Babylonian
confusion" here very, very quickly, nicht wahr? ;)...
Beth :)
Chew...@austarnet.com.au
> Chewy...@austarnet.com.au wrote:
<snip>
Point(s) taken.
> There are a number of proposed character sets still pending for
> approval for the next update. (One includes an interesting alphabet
> designed by some University of Toronto researchers to help retarded
> people.)
Any links? I know a few Star Trek fans tried to get Klingon and some of
the other scripts included, but IIRC UTC relegated those scripts to the
private use area...
> The standard does not say anything about whether or not any
> future encodings will include other redundant encodings that further
> complicates normalization.
I don't know if this has happened so far (which doesn't mean it won't),
but IIRC one the design goals was to ensure that this doesn't happen.
However I can understand your concern.
Darran (aka Chewy509).
Chew...@austarnet.com.au
> On 2005-04-20 BethS...@hotmail.NOSPICEDHAM.com wrote:
>
>
> > ...I've tried sending UTF-8 plain text and such before but it
> > didn't work (some stupid propogation software stripping the
> > 8th bit when no-one asked it to, no doubt ;)...
I'm posting from Google Groups (no 'net connection at home currently),
so I have no idea how Google's doing it... If beth has to use stinky
HTML (because of OE), then so be it.
> _____
> Duhhh! Usenet is a ((( `\
> 7-bit medium, Beth. _ _`\ )
> It worked exactly (^ ) )
> as it's supposed to. ~-( )
> _'((,,,)))
> And 173 trees were ,-' \_/ `\
> killed to produce the ( , |
> power necessary to `-.-'`-.-'/|_|
> propagate this spam-ish \ / | |
> message of yours. =()=: / ,' aa
> Shame!
>
> BTW, the whole idea of
> assemblers/compilers
> supporting 'Unicode'
> is totally nutty, prima
> facie...IMNSHO.
,
|'. ,
| '-._ / )
.' .._ ', /_'-,
' / _'.'_\ /._)')
: / '_' '_' / _.'
|E | |Q| |Q| / /
.' _\ '-' '-' /
.'--.(S ,__` ) /
'-. _.' /
__.--'----( /
_.-' : __\ /
( __.' :' :Y
'. '._, : :|
'. ) :.__:|
\ \______/
'._L/_H____]
/_ /
/ '-.__.-')
: / /
: / /
,/_____/----;
'._____)----'
/ / /
/ / /
.' / \
snd (______(-.____)
It's only sci-fi until we work out how to do it for real, then it just
becomes boring science.
--
Darran (aka Chewy509) brought to you by Google Groups.
"Walking into McDonalds and ordering off the new Healthy Options menu,
is like walking in to a Crazy Horse [adult shop] and reading the bible.
You just don't do it.". Dave Hughes, Melbourne Comedy Festival 2005.
Beth
> Hi Beth,
Hi wolfgang,
> | > What does and doesn't show up here?
> |
> | None of them...great...just great...
>
> You expected something else? :)
Well, not really...that is why I was "testing" because I knew at least some
of it wouldn't work...so, you know, "test" out some characters to see what
does and doesn't work exactly...turns out: None of them!! Excellent, just
excellent...
> I see every news-text with 'my' "Courier New 14pt" setting,
> regardless of the character-set you may have typed it.
Yeah, but that's your decision, isn't it? :)
Anyway, there are UNICODE "Courier New" fonts...that use "fixed size"
characters for everything (in fact, with the Chinese / Japanese / Korean
stuff, they usually get drawn in "squares", anyway...so it's always
"Courier" with them ;)...
In fact, XP Notepad supports UNICODE directly...but using the OpenType
"Courier New" font as the default, it's still correctly a "fixed size"
font...and that's what I used to create and write my little "Hiragana" text
file (has a "table" of the Japanese alphabet characters to make those
little "hello" and "goodbye" translations on the post I made to
Chewy...Japanese is interesting in that it _does_ also have an "alphabet"
system too, like this...and all the letters are "consonant, vowel": "ko",
"ni", "sa", "ro", etc...so it's not actually that complicated using this
little table: "sayonara" = "sa", "yo", "na", "ra" (look up each letter in
the "table" and copy the characters :)...this is also why Japanese has a
lot of vowels sounds when you listen to it because all the "letters" are
actually what we would consider two letters - a consonant and vowel
"pair" - except for "n", which can be said alone..."ko", "n", "ni", "chi",
"wa"...no, the real problems appear, of course, when it comes to the
"ideographs"...there's no simple way with this: Just learn each symbol
one-by-one...but it's a good system once you've learnt it, in that entire
words are just one character...for instance, UNICODE character 5315h means
"spoon", apparently (though, indeed, with your text-mode newsreader: "there
is no spoon" ;)...
So, this "Courier New" with Windows there must be a UNICODE version with
most of the charactes...but, being "Courier", of course, they should all be
"fixed size"...
> btw: You just filled my HD with Kenji/Russian-patterns?
> Don't care, I have the whole UTF-book already.
Ah, well...I was talking about UNICODE characters...so, you know, to talk
about specific characters and post a Japanese "hello, world!" program,
there isn't much choice but to use UNICODE for that...unfortunately, yes,
not everyone is able to see this directly...but, well, I can't do anything
about that from my end of things, if your software just doesn't understand
it...
Beth :)
Beth
> Rene wrote:
> > First, addressing both, at a time, "Randy, Rene" is insulting
> > for me, but, well... here we go.
>
> Ooh, please. Just because I mention 2 people on the NG that actively
> develop assemblers together your offended! In no way, do I imply that
> you are both of the same quality character or even ability in producing
> assemblers, just the fact that you both actively develop assemblers.
Ah, but that's your first mistake, Chewy: Randy is developing a "HLL
pre-parser sub-shit swindling trap", not an "assembler" ;)...
While hutch is developing his "dictionary arse"...Thomasz and John Found
are "assholes"...LuxAsm is "inherently corrupt" or something because we're
not listening to the "almight word of Rene, god of assembly" on everything
(even though we are based on "RosAsm for Linux" in design, that if we don't
qualify for simply adding "library support" and changing it around to fit
Linux better then no-one is able to qualify without a 100% identical
"clone" of RosAsm)...
Hence, you see, if addressing "assembler developers" then _ONLY_ Rene feels
he qualifies...so his name is so "holy" that it should NOT be mentioned in
conjunction with anyone else's name...it is a "blasphemy" to his "Holy
RosAsmness" to even contemplate "lesser mortals" in the same sentence as
him...
As for RosAsm supporting UNICODE, read this from B_U_ASM:
"Strings in Resources
Strings in Resources are rarely used because they are at least double size
(Unicode storage + Resources tree headers + dummy Strings, when required)
and because they are slower access. So, you should never consider Resources
as a good place where to store your Applications Strings. Usual Memory is
the preferred place for this."
Rene fails to comprehend the delibrate "internationalisation" support
Windows has built-in, for his "Windows specific assembler"...so don't
expect him to even comprehend what you're talking about (indeed, he does
effectively seem to say "what are you talking about?" a few times in his
reply, doesn't he? ;), let alone implementing it...
The point with string resources is, of course, that in a "modular"
environment (which RosAsm rejects) they'd be placed into a separate
"resource script" file...making it very convenient to send to "professional
translators" (go check their adverts, they often mention a qualification of
comprehending how Windows "resource scripts" work that they can deal with
this format :) to convert to other languages...
But the real winner is this:
-------------------------------------------
STRINGTABLE
LANGUAGE NEUTRAL, NEUTRAL
BEGIN
ID_STRING, "American English (default)"
END
STRINGTABLE
LANGUAGE LANG_ENGLISH, SUBLANG_ENGLISH_UK
BEGIN
ID_STRING, "British English spelling differences"
END
STRINGTABLE
LANGUAGE LANG_ENGLISH, SUBLANG_ENGLISH_AUS
BEGIN
ID_STRING, "Strewth! Aussie slang goes here!"
END
STRINGTABLE
LANGUAGE LANG_GERMAN, SUBLANG_NEUTRAL
BEGIN
ID_STRING, "Deutsch"
END
STRINGTABLE
LANGUAGE LANG_SPANISH, SUBLANG_NEUTRAL
BEGIN
ID_STRING, "Espanol"
END
STRINGTABLE
LANGUAGE LANG_FRENCH, SUBLANG_NEUTRAL
BEGIN
ID_STRING, "Francais"
END
-------------------------------------------
Note that all these strings have the same resource ID (ID_STRING :)...but
this is permitted because they all have different "language identifiers"...
And, simply, when you ask Windows to "LoadString", Windows automatically
pulls out the version of the string which corresponds to the user's
"locale" settings (or falls back on "neutral", if nothing matches)...
So, a single executable can be loaded up with resources appropriate for
many languages and it'll pull out the correct strings dependent on the user
"language / locale" settings...same .EXE but it pops up in French on a
French version of Windows, German on a German version, the British English
version spells it "colour picker" rather than "color picker" (and
"internationalisation" rather than "internationalization" ;), etc., etc....
Mind you, you might think "it's better to write separate versions of the
program for each language"...fine, you're probably right, in fact...but
Rene doesn't even seem to know about the "language ID" option because his
editor doesn't support it nor does he mention anything about it in
B_U_ASM...seems he skipped that chapter of the Windows SDK or
something...it is for reasons like _this_, you see, that Windows has the
"stringtable" resources...so, if you're going to have a "string editor"
then, yeah, you should be _supporting_ things like the "language ID" or,
yeah, the use of "string table" resources is a bit pointless (because the
other point about putting it in a separate file for sending it to
professional translators - who are experts in natural languages, not in
programming languages that handing them the RosAsm source code? Well, they
wouldn't know what should and shouldn't be translated! - is nullified by
RosAsm's "monofile programming" concept, so he's already ruled that
advantage out...without the "language ID", no wonder he thinks it's "rather
pointless" because he's accidentally "nullified" the main reasons for
wanting string resources, through his IDE "philosophy" and a crap editor
for it ;)...
Beth :)
Chew...@austarnet.com.au
ROFL :D
webs...@gmail.com
> websn...@gmail.com wrote:
> > Chewy...@austarnet.com.au wrote:
>
> <snip>
>
> Point(s) taken.
>
> > There are a number of proposed character sets still pending for
> > approval for the next update. (One includes an interesting
> > alphabet designed by some University of Toronto researchers to
> > help retarded people.)
>
> Any links?
Here's the links for all the stuff they are considering for the next
release oft he standard:
http://www.unicode.org/pending/pending.html
The character set I was referring to is "Blissymbolics". I may have
mischaracterized it, by claiming it was for retarded people -- its
apparently just for people who don't speak or something. I'm not sure
-- whatever -- its a recently invented alphabet with a serious research
based purpose. When I looked at them, I wondered whether or not
"traffic symbols" should be added. (Musical notation got added in the
last major release.)
> [...] I know a few Star Trek fans tried to get Klingon and some of
> the other scripts included, but IIRC UTC relegated those scripts to
the
> private use area...
Actually Klingon was categorically rejected:
http://www.unicode.org/alloc/rejected.html
But Tengwar (the Lord of the Rings elvish script) is apparently still
under investigation:
http://www.unicode.org/alloc/investigation.html
Don't ask me why one seems more legitimate than the other. But
assuming Tengwar is rejected, it doesn't seem clear to me whether or
not this bodes well for Blissymbolics.
> > The standard does not say anything about whether or not any
> > future encodings will include other redundant encodings that
> > further complicates normalization.
>
> I don't know if this has happened so far (which doesn't mean it
> won't), but IIRC one the design goals was to ensure that this
> doesn't happen. However I can understand your concern.
Right, I just don't want my software being held hostage by this
committee who has not exactly left me with the greatest impression,
judging from their history. The problem is, that there does not seem
to be a really good alternative.
See -- what bugs me is that they made no attempt to encode the
characters with certain bitpattern properties. Instead, the only real
property we have is that each alphabet tends to start on a 0x100
boundary, and are grouped together as much as possible. The Hangul
decomposition and composition at least follows a logical structure --
even though by itself it is an exception to the Normalization and
composition rules. But because accent combining characters, and other
similar things are just randomly strewn about the UnicodeData.txt file.
Really you have no choice but to implement a function which delivers
the mapping that is equivalent to the content in the UnicodeData.txt
file. And that's just to figure out if two unicode strings are the
same. Its fairly insane.
Beth
> Hi Beth,
Hi Chewy :)
> Thanks for the entertaining post.
Ah, I should start a career in "informmercials" or something, whatcha
think? ;)
> I had thought about many of the issues, including directionality of
> text, but as you said, an assembler/compiler should just see the
> logical order.
It's an "editor" issue (though, for RosAsm and LuxAsm - or those writing
IDEs like NaGoA or HIDE or RadAsm - then the "editor" is also something to
consider :)...but, you know, a file is still just a "stream of characters"
to the assembler reading it...and UNICODE delibrately presents it in
"logical order", regardless of what "direction" that might be when actually
writing it (hence, the need for the "directionality" marks, to add in this
information to help editors show it properly...as some languages can
actually be written either way around too...so, separating its "visual
order" from its "logical order" was delibrate to make that stuff easier
:)...
> Some of the other issues, such as non-english labels and
> numbers, can be a pain in the arse. Numbers not so much (AFIAK most of
> the world recognises the 0..9 numerical set),
Well, sort of...Tamil has characters for "ten", "one hundred" and "one
thousand", as well as _nine digits_ (no zero!! They've not been watching
that Stargate episode! ;)...but, yeah, numbers aren't so bad...
> but labels. The biggest
> issue for me, would be combinations or equivalence. How should those be
> handled?
There are "rules" for "normalising" strings in the UNICODE book...in fact,
some UTF-8 sequences are actually delibrately "invalid" so that only one
way to access a character is allowed and "overlong forms" (such as using an
"escape sequence" but accessing an ordinary ASCII character) are considered
"invalid"...
Follow these "rules" to put the string into a "normalised" form (that is,
there is only one valid way to address any particular character :) and then
there should be no problems with things like testing for equivalence
between two strings...
> Well, it appears that the answer (for LuxAsm) is don't. Just
> have labels/identifiers in the normal 7bit ASCII range as already
> defined by most other assemblers/HLLs.
Basically, yes...
Though, in principle, I don't have a problem with the idea of also applying
it to the labels / identifiers too...but, in practice, a lot of effort for
what exactly? As the mnemonics and everything are going to still be
ASCII-based, anyway...BUT, on the "principle" side of things, I wouldn't
have a problem with this as I don't think a tool should decide "policy", if
you know what I mean and any "Babylonian confusion" that might result is
for programmers to organise amongst themselves, not for a tool to
"dictate"...
For instance, wolfgang could decide to use German labels /
identifiers...you can validly do this without need for any accents, as an
"-e" after an "a" or "o" or "u" is considered the same as placing an
"umlaut" (two dots) above the vowel...that special B-shaped character is
just "ss" and then you can validly write German labels / identifiers in
strict ASCII...right, how do you propose the tool works out that wolfgang
is using "prohibited" German identifiers rather than "standard" English,
from looking at the characters he's typing alone? Including a "dictionary"
and an "AI" to determine the "Englishness" of identifiers? If you just drop
the accent marks, then Rene could use French identifiers...how's the tool
to know he's using a "prohibited" non-English language? There are even ways
for writing things like Vietnamese in ASCII characters...or what about
"Romanji" identifiers? Japanese words written with Latin ("Roman")
characters, like, in fact, writing "konnichiwa" or "sayonara" or "hiragana"
because, of course, these actually are written in Japanese with all those
"stroke" characters...
[ Indeed, if you know more than one language, this is a possible "trick" to
easily avoid conflicting with "reserved words" (say, in BASIC, where
there's a lot you could "conflict" with ;)...simply use identifiers from
that other language...anglicised and ASCII-ified, if necessary...usually,
this means you won't get any "conflicts" with reserved words - except for
the odd rare "coincidence" where a set of letters is a valid word in
English and in the other language - because, you know, they are all in
English, while all your identifiers are in "Romanji" Japanese or something
;) ]
In fact, with ASCII "dominating" a lot of things, then most languages
probably have their "ASCII-ified" versions...how does the tool work out the
"Englishness" of these identifiers, even written in strict ASCII?
It can't...it's not qualified or able to do so...this is NOT a job for a
tool to be doing...it's a job for "standards" to be agreed between
programmers...and it might not necessarily be English that's most
appropriate...a "joint project" between the Mexican and Spanish
governments, say? Spanish would make more sense...unless they're expecting
some "outside involvement" then forcing them to use English and verifying
the "Englishness" of their identifiers...well, this kind of logic is
leading to a very silly place, isn't it? ;)...
So, in principle, as I say, I have no problem with the idea of non-ASCII
identifiers...BUT, in practice, it's a lot of work to support that...and
we're NOT approaching this with a "clean slate"...programmers are already
used to having to switch to English / ASCII for this stuff...they might
want to stick with that, anyway...the Intel mnemonics won't change nor,
really, will the assembler directives (well, you could come up with
non-English directives but then you're "limiting your market" there, for
not much reason...on the other hand, you could use operators like "{ }" and
negate the natural language issue entirely *cough* like my LuxAsm "blocks"
do ;)...
Also, you know, as a native English speaker, I'm thinking "oh, other people
around the world might like this feature" (after all, for me, it's
_ALREADY_ in the most appropriate form ;)...but what are the French and
German and other people saying here? "No, not really interested...it's a
terrible idea because it'll be like the 'Tower of Babel'"! So, you know, if
your potential "audience" doesn't really want it, then, well, "the customer
is always right" (even when they are clearly wrong! ;)...you jump through
all the hoops to support it for them...none of them wants to actually use
it...well, that was a big fat wasted effort, eh? ;)...
LuxAsm will certainly be written to accept UTF-8 source files (indeed,
"accidentally", NASM appears already to do so with strings and comments in
my little "test program"...because of UTF-8's "ASCII compatibility" and
that ASCII characters "delimit" these things: quotation marks either side
of a string (ASCII) or ";" to start a comment, newline to end it (ASCII
again)...what's inside those ASCII characters is either "passed through" or
ignored (when you see Japanese characters, what NASM sees is a string of
characters with the 8th bit set - extended ASCII - and still processes that
quite correctly ;)...so, to a degree, this probably _already_ works with
some tools...try knocking up an example in (UNICODE-enabled) Notepad and
then run it through the tools as they stand already...NASM passed the
"test" and the others may "coincidentally" do so too...depends how they
treat bytes with the 8th bit set ;)...this is, indeed, a bit of a "no
brainer" to implement, as the fact that NASM "accidentally" already
supports the same kind of thing we were intending, anyway, shows just how
much of a "no brainer" (you can implement it "purely by accident", it's
that much of a "no brainer" ;)...with LuxAsm, though, it will be "aware"
that it's processing UTF-8...but, yeah, doesn't really do anything about
it, except to process it "sensibly" (it can, for example, provide a
"property" for string constants called "length" or something, which is in
terms of "characters", not bytes (a "sizeof" or whatever can deal with
"size in bytes"...characters, though, needs a tool that's "aware" of UTF-8
to provide an accurate and useful result ;)...ah, "properties" are like
"local defines"..."defines with scope", if you will...it's a "LuxAsm thing"
;)...
And one possibility is that - should it turn out that people actually
change their minds and think "yes, I want the UNICODE identifiers!!" - then
it can be "extended" later on...this would be perfectly "backwards
compatible", I think...or, if "bored" or something, then someone can add it
in later...everyone who's not interested can always completely ignore it,
after all...
But, as you've seen, there's not a great "popular uprising" with people
waving placards demanding this...so, LuxAsm will do "basic support" - it
"expects" UTF-8 and is "aware" of it for string constants and comments -
but, beyond that, there seems to be no "demand", so there's no priority to
"supply", if you know what I mean...it can be tacked on later,
maybe...maybe not...
In fact, my little "test" which demonstrates that NASM _already_ deals with
UTF-8 comments and strings, proves the point a different way...some tools
_already_ are "accidentally" supporting UTF-8 to that "basic level",
without even knowing it...and the fact that no-one has actually realised
this (I didn't either until I thought it would be interesting to see what
NASM would actually do ;), shows how great the "demand" is...if people were
regularly wanting UTF-8 source files passed through NASM, then your post
should have had Frank shouting "NASM already does it!!"...but, I bet, not
even the NASM developers have actually realised that it does already work
to this "basic level"...indeed, they could cheekily add it to the "features
list": "NASM has basic UTF-8 support!"...as if they actually "intended" it
or something...shhh! Don't tell anyone! ;)...
Beth :)
webs...@gmail.com
> > future encodings will include other redundant encodings that
> > further complicates normalization.
>
> I don't know if this has happened so far (which doesn't mean
> it won't), but IIRC one the design goals was to ensure that
> this doesn't happen.
I was just looking through the proposals and noticed the following --
look at Pahawh Hmong (the alphabet for a language in common use in
Laos, Thailand, and Vietnam). Character xx36 ("SIGN VOS NRUA") is a
"duplication character", that means "repeat" the previous character.
There aint no other characters in Unicode that I am aware of that has
such a property. Then there is the obvious xx3A ("SIGN VOS LUB") which
is a "shorthand" for a complex symbol sequence -- so normalization will
either have to compress the the complex sequence, or expand the xx3A.
This language is in actual use -- i.e., there's no good reason not to
include its alphabet. But in this case we can be sure that the
normalization algorithm will have to change to accomodate it. So my
concern is not theoretical or abstract.
Betov
@l41g2000cwc.googlegroups.com:
>> I must apologioze that i have no idea, at all,
>> about how this is usable or not usable for the real
>> "Unicode Users".
>
> Would you prefer:
> "Hello world"
> or
> 48h, 65h, 6ch, 6ch, 6fh, 020h, 77h, 6fh, 72h, 6ch, 64h
> within your source file.
>
> I would prefer the first form (as anyone would). Now take the view of
> the Hindi student in India who doesn't know english. Are you going to
> force the second option on him, so he can use your assembler?
>
> (I know this is a poor example, but think about other languages, eg
> hebrew, most asian, etc).
Of course. Evident. I would also like it to work
that way. What i am saying is that is simply
_DO NOT KNOW_, if it actually works that way or
not. I use an occidental KeyBoard, and i have no
idea, at all, of what is going on, when, say, an
oriental user inputs something directly in Unicode.
There is a Case, in the RosASm MainWindow Messages
holding for "&WM_IME_CHAR" actually, that is
effectively suposed to input Double-Bytes, into the
Source. But i do not know:
1) If it really works.
2) What goes on, in the life of the Source, with
the so many Edition and Compilation Functionalities,
and at what extend they could choke, on some Bytes
of the Double Bytes.
For example, RosAsm always parses the Strings
from Quotes to Quotes, on a _BYTE_ Basis scan.
Let us imagine, that, inside some Double Byte,
the same Byte as a Quote would be inserted from
a Unicode String... Can this happend? i simply
do not know (It would create an "Unpaired Quotes"
faultive error Message)...
>> As for writing, as you suggest, say, the labels
>> Names, in Unicode, directly in a RosAsm Source,
>> this will never be implemented. I do not see any
>> reason for doing so.
>
> So your source code will forever be ASCII?
The Assembly Source, Yes. Unicode will be (or are)
for Data Strings only, and possibly, maybe, for
Comments.
But, for naming the Labels, sure, not, and never.
That would be much no use complication, IMHO.
I Have read a long (as usaul...) Post, from Beth,
some long time ago, where she was, if i understood
well, promoting a kind of UTF something, that should
be the universal default. As i was not much interrested,
i did not read it carefully, but...
Beth... maybe.. a couple of words would help him...
:)
Betov.
Betov
@newsreader1.utanet.at:
> IIRC, one of the RosAsm guys tried to add Chinese pattern support
> onto the RosAsm-source. Rene may know more about it.
No, and this is my problem: I cannot really
answer anything usefull, because i do not
even know if it works, or not, i RosAsm.
I simply cannot test anything, and i entirely
rely on the "Unicode User", to do the job. Also,
i am not furiously interrested with Unicode:
I have never studied this problem, and i do not
know of the various formats. The ony thing i really
implemented was the:
[MyUstring: US 'This is a Unicode String', 0]
I do not even know if, say, it is possible to
really input a Chineese String that way. I just
wait for a Chineese user to tell me, and to help
me doing the job, if it causes problems. I cannot
write things that i cannot test, you know... ;)
Betov.
Betov
news:3Il9e.19368$443....@newsfe3-gui.ntli.net:
> [... Shames...]
Ah! I was wondering how long i would have to
wait before you could came at the rescue of
your good old friends, master Randall Hyde,
and Hutch--, our prefered Right-Wing bastards.
:)
Betov.
PS. Interresting to notice that, when it is
for denigrating RosAsm, you are able to read
B_U_Asm, and to Copy-Paste the parts where
i honestly talk about RosAsm real limitations,
whereas, when it is for some interresting
feature, you simply... do not know... :)
Yes, the RosAsm Resources Manager is limited.
The Lang IDs are not yet implemented, and the
Named IDs are only implemented for the Disassembler,
but not yet for the Assembler... Yes...
SHAME ON YOU, Beth!
wolfgang kern
Hi Beth,
[..]
| As for taking up more memory and disk space: UTF-8 does not take up a
| single bit extra from ASCII for any ordinary ASCII characters...
??
I may have missed the point of UTF-8, I'm not through the whole book yet,
how can I display 'mathematical operators' 2200..22FF (page 193) with UTF8
??
At the moment I use ASCII 20..7F as they are and 01..1F and 80..FF
as 'my personal character-set', while 00 is used to mark the end
followed by one ore more 'format/function/more../'-bytes.
ie:
40 41 42 43
00 0a xx yz nn nn nn mm ; print/edit a numeric variable
; (type,format,action,group,address)
will print: "@ABC-33.35 x10[sup3]-6[/sup]"
00 ... ; all var-types, input, locate, colours, ... just all needs ;)
00 E0 xx.. ; conditional call substring incl. format header (menus...)
00 FE ; end of a sub-string (menu/buttons/..)
00 FF ; end of the print job
[..support..]
| Yeah, this is actually a "reverse" point on what's actually true...the
| whole purpose of UNICODE is that it _gets rid_ of all the things like "code
| page incompatibilities" once and for all...one standard character set for
| _everyone_...
Even it will be a very huge pack of patterns for _everyony_ to carry,
it may make sense to set up an international agreement of tool-makers
to a limited standard use of UTF character-sets.
| But this is the biggest problem, true...but it's a problem of "my software
| is not up-to-date with UNICODE"...yes, but isn't the point Chewy's making
| about: "let's all update things to use UNICODE from now on"?
At least, it's worth to think about it.
['Babylonian confusion']
| _THIS_ is the real problem, of course...
| BUT is this a problem that's the tool's responsibility? For instance,
| programming a protected mode OS is very confusing too...or using "direct
| access" to hardware...should tools also start telling us we can't use "LGDT
| / LIDT" or "IN / OUT" instructions too? As they might cause "Babylonian
| confusion"?
My new disassembler will show PL0/IOPL/PM/64-only instructions
as "force EXC0D/06/00.." if the mode configuration wont match.
But this aren't language issues, except Intel-manuals are written
'that' confusing, it may even native English speakers drive crazy. :)
| Or should the tools _support_ these things ...
| But why should a tool be forcing a Russian programmer, ...
Wouldn't it help a lot if the whole globe talks only one language?
A long way to go, but when I look back 50 years, the world seems
to tend to everyone learn to understand written English at least,
local US/UK/AUS pronouncation excluded here of course :)
[..]
| ... if we don't agree on that, then things become "Babylonian
| confusion" here very, very quickly, nicht wahr? ;)...
That's very true indeed*, isn't it? *)indead for the French :)
__
wolfgang
wolfgang kern
Hi Beth,
| > You expected something else? :)
| Well, not really...that is why I was "testing" because I knew at least some
| of it wouldn't work...so, you know, "test" out some characters to see what
| does and doesn't work exactly...turns out: None of them!! Excellent, just
| excellent...
:)
| > I see every news-text with 'my' "Courier New 14pt" setting,
| > regardless of the character-set you may have typed it.
| Yeah, but that's your decision, isn't it? :)
It is, otherwise posted tables wont be readable and Annie become garbled.
| Anyway, there are UNICODE "Courier New" fonts...that use "fixed size"
[...]
Yes, sometimes I see a "Ç" (128) and the other time the Euro-sign.
| > btw: You just filled my HD with Kenji/Russian-patterns?
| > Don't care, I have the whole UTF-book already.
|
| Ah, well...I was talking about UNICODE characters...so, you know, to talk
| about specific characters and post a Japanese "hello, world!" program,
| there isn't much choice but to use UNICODE for that...unfortunately, yes,
| not everyone is able to see this directly...but, well, I can't do anything
| about that from my end of things, if your software just doesn't understand
| it...
I can fully see all foreign 'patterns' in your htm,
this means it loaded the fonts from somewhere.
And I don't see any text-double in your htm-post (OE6 here).
__
wolfgang
Betov
news:d45oal$jeq$4...@newsreader1.utanet.at:
>
> Hi Beth,
>
> [..]
>| As for taking up more memory and disk space: UTF-8 does not take up a
>| single bit extra from ASCII for any ordinary ASCII characters...
>
> ??
> I may have missed the point of UTF-8, I'm not through the whole book
> yet, how can I display 'mathematical operators' 2200..22FF (page 193)
> with UTF8 ??
There are other questions, that do not seem to be
considered, here, or that i fail to understand...
For example, when we implemented the National Strings
in RosAsm, there was a Chineese version... and a minor
problem with the viewings for this version. Then, to
fix the bug, i was in need to _SEE_ something on my
own screen. So, the Volunteer sent me... his FONT, that,
once saved in the required OS Folder, enabled me with
seeing the problem.
Now, with UTF something, would it be 8 or whatever...
Where are the FONTs, on the occidental Users PCs???
When implementing the Fonts Choice, for RosAsm, i had
to simply reduce the choice to the Fixed Pitch ones,
as you may guess,... QUITE FEW FONTS...
Then, there is one another problem with this UTF8 thing.
What with the occidental Chars above 127? There are
users who want them, would it only be for their Comments.
How would these ones come out in UTF8? What form?
Betov.
Evenbit
[Well, we all know what she said, and she kindly made use of the
world's biggest grid computer to strip the extranious bits from her
text]
Evenbit>
Wouldn't you be better off with a 'Keep It Simple' philosophy? If your
goal is the moon but you don't have NASA facilaties and a NASA crew to
get you there, shouldn't you settle for the summit of Mt. Everest
first?
This is just a suggestion, but I believe the first goal of a Linux
assembler should be to leverage the existing skillset. That means that
your first set of tools will be utilities to convert existing NASM and
FASM code to LuxAsm syntax. This will also drive your syntax design
because you'll want to make it as easy as possible to write those
conversion tools. The second goal should be to create a basic IDE to
get the Write/Compile/Link/Debug cycle started as soon as possible.
Since you've done a lot of investigation into how to use the xWindows
library, perhaps this could be your task? If you think it is too early
for that, then at least modify an existing Linux editor (like VI) to
give it a command to send the source to the assembler. Once you get
this Write/Compile/Link/Debug cycle running, you now have people
writing LuxAsm code -- thus you have a force driving the assembler
development crew to meet the needs of the LuxAsm coders. This is a
'form follows function' approach that should guarantee that the
assembler is practical and manages to develop a user base.
My two cents worth.
Nathan.
rand...@earthlink.net
Beth wrote:
>
> In fact, my little "test" which demonstrates that NASM _already_
deals with
> UTF-8 comments and strings, proves the point a different way...some
tools
> _already_ are "accidentally" supporting UTF-8 to that "basic level",
> without even knowing it...and the fact that no-one has actually
realised
> this (I didn't either until I thought it would be interesting to see
what
> NASM would actually do ;), shows how great the "demand" is...if
people were
> regularly wanting UTF-8 source files passed through NASM, then your
post
> should have had Frank shouting "NASM already does it!!"...but, I bet,
not
> even the NASM developers have actually realised that it does already
work
> to this "basic level"...indeed, they could cheekily add it to the
"features
> list": "NASM has basic UTF-8 support!"...as if they actually
"intended" it
> or something...shhh! Don't tell anyone! ;)...
Not knowing much about UTF-8 (my Unicode knowledge extends as far as
UTF-16 and that's about it), I would say that HLA v2.0 would handle
literal strings of this form as long as the character code for quote
can never appear in a MBCS (multibyte character sequence). HLA v1.x,
however, would not be happy with the character as Flex rejects all
character codes in the range $80..$FF out of hand.
There is, however, another issue that gets you into trouble with MBCS.
When you start adding sophisticated compile-time language facilities,
such as string functions, handling all the different character sets
becomes a nightmare. Then, in HLA's case, there is also the issue that
you need to provide standard library routine equivalents of string
functions for UTF-8 strings (you think zero-terminated strings are
painful to compute the length of? Try UTF-8!).
An assembler like NASM, that doesn't provide much in the way of
compile-time string handling, might actually get away with "accidental"
UTF-8 support. But when you've got a sophisticated macro system and
compile-time language, supporting MBCS turns out to be a *lot* of work.
Cheers,
Randy Hyde
webs...@gmail.com
> Chewy wrote:
> > Hi Beth,
> > Some of the other issues, such as non-english labels and
> > numbers, can be a pain in the arse. Numbers not so much (AFIAK most
> > of the world recognises the 0..9 numerical set),
>
> Well, sort of...Tamil has characters for "ten", "one hundred" and
> "one thousand", as well as _nine digits_ (no zero!! They've not been
> watching that Stargate episode! ;)...but, yeah, numbers aren't so
> bad...
There are many different numbers in different alphabets.
> > but labels. The biggest issue for me, would be combinations or
> > equivalence. How should those be handled?
>
> There are "rules" for "normalising" strings in the UNICODE book...in
> fact, some UTF-8 sequences are actually delibrately "invalid" so that
> only one way to access a character is allowed and "overlong forms"
> (such as using an "escape sequence" but accessing an ordinary ASCII
> character) are considered "invalid"...
These are just code point representation rules. Beyond that, there are
many illegal, and unassigned code points. Also there are nonsensical
code point expressions like just a singleton "grave" accent (no
language accepts that alone as a character in their alphabet).
> Follow these "rules" to put the string into a "normalised" form (that
> is, there is only one valid way to address any particular character
> :) and then there should be no problems with things like testing for
> equivalence between two strings...
Uhh ... normalized means something different. Unicode normalization is
not the only way a Unicode string should exist, its just a theoretical
form from which you can compare strings. There are two main endorsed
normalization forms -- compatible and canonical. They specify a
mechanism for rearranging and mapping of code point sequences to a more
unified version.
For example, there are letters which can take multiple accents. And
you can often specify them as base-char + accent1 + accent2. The
question is, if you flip the order of the accents, does that represent
a different character or not? The unicode normalization algorithms say
no for some cases, and yes in others.
> > Well, it appears that the answer (for LuxAsm) is don't. Just
> > have labels/identifiers in the normal 7bit ASCII range as already
> > defined by most other assemblers/HLLs.
>
> Basically, yes...
>
> Though, in principle, I don't have a problem with the idea of also
> applying it to the labels / identifiers too...but, in practice, a
> lot of effort for what exactly?
A lot of effort for cornering every market outside of english. If
programmers from non-english speaking countries could code in their own
native tongue, it would probably be very attractive to them.
> In fact, my little "test" which demonstrates that NASM _already_
> deals with UTF-8 comments and strings, proves the point a
> different way...some tools _already_ are "accidentally" supporting
> UTF-8 to that "basic level", without even knowing it...
Well that's kind of the point of UTF-8. For certain semantics, you get
it for free, simply by allowing characters in the range 0x80 - 0xF8 in
your tools. But of course, comments, and embedded strings in this case
are easy because, comments don't need any checking, and assembly
strings are essentially byte sequences which just correspond to raw
data (legality or not is not something that needs to be imposed or
checked).
> [...] and the fact that no-one has actually realised
> this (I didn't either until I thought it would be interesting to
> see what NASM would actually do ;), shows how great the "demand"
> is...if people were regularly wanting UTF-8 source files passed
> through NASM, then your post should have had Frank shouting
> "NASM already does it!!"...
Well hang on. If you can't make identifiers rendered in UTF-8 (or
possibly UTF-16) with proper normalization, then I wouldn't call that
real unicode support.
Chew...@austarnet.com.au
>
> For example, there are letters which can take multiple accents. And
> you can often specify them as base-char + accent1 + accent2. The
> question is, if you flip the order of the accents, does that
represent
> a different character or not? The unicode normalization algorithms
say
> no for some cases, and yes in others.
>
That appears to be the hardest bit of all. I've also found that in many
cases (now that I'm looking more closely at the actual encodings and
character maps), that some characters, particularly from the Latin set,
have direct equivalents, eg Latin Small Letter A With Tidle, can be
mapped to either U+00E3 or U+0061 + U+0303, (which is Latin Small
Letter A with Combining Tidle) which makes comparison even harder.
While it's stated that any editor conforming to Unicode 4.x must
produce shortest form (which negates the issue I raised), however
copying from other sources the exact encoding format MUST be preserved.
So that if I have an editor which only conforms to version 1.x, which
produces the long form (v1.x IIRC doesn't state which form to produce
to be conformant), and I copy it over to another editor which produces
short form, the second editor (to be conformant) cannot and should not
convert long form to short form, even though the short form is
considered correct.
So basically: If I copy over ã encoded as U+0061, U+0303 into a text
editor that is conformant to Unicode v4.x, it MUST remain in the long
format (eg 2 characters), even though that particular encoding is not
technically correct, (where U+00E3 is the technically correct
encoding).
That's if I'm reading Chap3 correctly. (If I'm wrong please let me
know).
Now my head hurts...
And I can see the resistance to allowing full unicode support for
labels/identifiers.
I would just like to thank everyone that has replied and voiced a
constructive opinion on this topic.
I will have to admit, supporting unicode is a bit more work than I
first thought! :(
wolfgang kern
Hi Rene,
Whenever I start with the KESYS-browser,
I'll come back to all this questions about UTF and stuff.
At the moment we just may say that we don't know much about it :)
Not that important from a tool-makers point of view yet, anyway.
Perhaps Beth can post some more details,
she's got just the same book as I have, but there may be more...
__
wolfgang
Beth
> Chewy wrote:
> >> I must apologioze that i have no idea, at all,
> >> about how this is usable or not usable for the real
> >> "Unicode Users".
> >
> > Would you prefer:
> > "Hello world"
> > or
> > 48h, 65h, 6ch, 6ch, 6fh, 020h, 77h, 6fh, 72h, 6ch, 64h
> > within your source file.
> >
> > I would prefer the first form (as anyone would). Now take the view of
> > the Hindi student in India who doesn't know english. Are you going to
> > force the second option on him, so he can use your assembler?
> >
> > (I know this is a poor example, but think about other languages, eg
> > hebrew, most asian, etc).
>
> Of course. Evident. I would also like it to work
> that way. What i am saying is that is simply
> _DO NOT KNOW_, if it actually works that way or
> not. I use an occidental KeyBoard, and i have no
> idea, at all, of what is going on, when, say, an
> oriental user inputs something directly in Unicode.
Okay...
First things first, when you register your RosAsm windows classes, you
should use the "RegisterClassW" API, not the "RegisterClassA" API...the
reason for this is that Windows takes this as a "cue" as to whether to send
the messages with ANSI / UNICODE parameters in ANSI or UNICODE form...
If you register with "RegisterClassA" - as you're probably doing - then
Windows takes this to mean that you want your application window to receive
ANSI messages (some messages have ANSI / UNICODE variants...for instance,
the character provided by "WM_CHAR" is an ANSI character when the window is
"registered" as ANSI...but it will be a UNICODE character returned by
"WM_CHAR" when the window is "registered" as UNICODE :)...so, yes, you
won't see any UNICODE characters appearing with "WM_CHAR", _IF_ you used
"RegisterClassA"...
So, the most important thing is to make your editor window a "UNICODE
window"...thus, register it with the UNICODE "RegisterClassW" to specify
that (this is a "backwards compatible" thing, of course...it assumes ANSI
unless you specifically call the UNICODE API, which tells Windows that the
application is "UNICODE aware" and then it operates and sends messages in
UNICODE format)...
The API "IsWindowUnicode" can be used on windows to find out if they are
registered as ANSI or UNICODE windows...and, note that if you send a
message from an ANSI window to your UNICODE registered window, then Windows
does automatically "translate" such messages while delivering them...
And, in fact, that is all you should need to do...when the user hits a key,
then the standard WM_CHAR should be delivered to the window...BUT, as this
window is "registered" as being a UNICODE window, then the "wParam"
parameter should now contain a full 16-bit UNICODE character, not only an
8-bit ANSI character...
So, when your Indian user presses a key for an Indian character, the
"wParam" parameter for the WM_CHAR message should now contain a 16-bit
UNICODE character code...
To deal with the UNICODE strings properly, obviously, then use the UNICODE
API...so, "TextOutW" with a string of 16-bit per character codes, not just
8-bits per character...
Be careful, of course, with some operations...for instance, to make an
ASCII character "uppercase" or "lowercase" is just a case of toggling a bit
in the character code...this won't work, in general, on UNICODE
characters...BUT, don't worry, you do not need to actually be an "expert"
in the character set...Windows provides a set of API functions which
correctly deal with UNICODE characters in the right way:
"CharLowerW" - converts UNICODE character to "lowercase" correctly
"CharUpperW" - converts UNICODE character to "uppercase" correctly
[ Note that, of course, not all languages and scripts actually have an
"uppercase" or "lowercase"...so, for some characters, calling "CharLower"
will have no effect...but, then again, this is the same as punctuation
characters like "<>,.@:;[]%&*" which also do not have any "case"...so this
idea is not unusual but just the case that some languages are also the same
with their alphabet characters, as with the numbers and punctuation...so,
you know, just something to remember, so you don't start thinking something
is "broken" when you can't get an "uppercase" Thai character...they don't
have any, so it's a not a mistake or anything :)... ]
There's also "lstrcmpW" - which "mimics" the C function of the same name -
which is also "UNICODE aware" when it makes string comparisons for you...
Well, I think you get the general point: Windows has "UNICODE awareness"
built-in, so if you use the Windows API functions on the UNICODE strings,
then it has all the "rules" built-in to do this correctly and sensibly...
Also, your UNICODE window procedure should call "DefWindowProcW" (because,
as noted, Windows now sends "UNICODE enabled" messages to your window, so
when passing it on to "default processing", you should call the UNICODE
default processing, which correctly understands the "UNICODE enabled"
messages)...
Generally speaking, if you change all your API calls that end with "A" to
those that end with "W" then this will put Windows into "UNICODE mode" when
dealing with your window...and the messages like WM_CHAR are automatically
"upgraded" to return UNICODE characters, rather than only ANSI
characters...this then allows your users to type in whatever characters
they like and the 16-bit "wide character" code returned has a unique "code"
for each character (basically, it's much the same as ASCII / ANSI but the
characters are 16-bits, so there are more characters...but, otherwise, the
way Windows implements UNICODE is only to make the characters
bigger...everything else should be more or less the same...but, as noted,
be careful about manipulating the character codes directly and, rather, use
the actual Windows API functions for things like "CharUpper" to take
advantage of the "built-in" UNICODE support in Windows...you know, Windows
itself has all the correct "look-up tables" to do it properly...just call
the API to make use of that then you yourself do not need to be any
"expert" :)...
And, of course, be aware that _ALL_ the messages sent to the window (when
registered as a "UNICODE window" :) will be sent in the UNICODE character
set...not only WM_CHAR but also WM_GETTEXT for the window title, WM_CREATE
(the CREATESTRUCT will have UNICODE strings inside it...but, then, you used
"CreateWindowExW" to create it, so you'd expect it to return the same
UNICODE strings you gave to create it...that makes sense, yes? ;) and so
forth...
You _ARE_ allowed, though, to call the ANSI API, if you really want (e.g.
"SetWindowTextA", supplying an ANSI string, not a UNICODE string) and
Windows will "translate" as necessary...in other words, it'll pad the ASCII
bytes with zero automatically...obviously, if you do that, though, then you
can only use ANSI characters in the string you supply)...but, well, once
you switch to UNICODE, there doesn't really seem to be too much need to do
that...but if you do need to, then you still can use the ANSI API...they
are just "limited" to the 8-bit ANSI strings only and get "translated"...
[ Mind you, Windows itself is inherently UNICODE internally...hence, in
fact, when you've been writing your ASCII / ANSI programs up to now, this
"translation" is actually happening all the time...you just don't see it
happening because it "translates" back and forth to always show you ANSI
strings...so, in fact, switching to UNICODE is not getting you any "extra
cost" but, actually, the opposite in that Windows actually _stops_
"translating" all the time - just "deals direct" with the program in its
_native_ UNICODE - except for when ANSI API are explicitly called :) ]
One problem with this, of course, is that UNICODE is not supported directly
on the 9x kernels...if this is a problem then send an Email to Jeremy
Gordon (of "GoAsm" fame :) because, as you know, he's got very good UNICODE
support...and there is some "UNICOWS.DLL" which helps "patch over"
this...but, well, Jeremy's the better one to ask about that because I've
never used it (indeed, I know of it through reading Jeremy's website...so,
all I know about it, is what he's told me...hence, better to ask the
"master" than the "student", yes? ;)...
Though, the obvious and simple way to deal with this is to "detect" what
Windows Rosasm is running on..."GetVersionEx" being the API to do that
(basically, look for an NT based kernel :)...and then use the ANSI /
UNICODE API, as appropriate...so, if UNICODE is available, then switch to
the UNICODE API or just carry on as normal with ANSI, if it cannot be
detected...this, though, might make the program bigger to deal with it this
way, as you might need "two versions" of many things (it's a simple
approach to it and, thus, might not be the best one to take...but it would
be sure to work ;)...so, go and take a look at the "UNICOWS" thing first,
as that might save time and effort...
> There is a Case, in the RosASm MainWindow Messages
> holding for "&WM_IME_CHAR" actually, that is
> effectively suposed to input Double-Bytes, into the
> Source. But i do not know:
>
> 1) If it really works.
>
> 2) What goes on, in the life of the Source, with
> the so many Edition and Compilation Functionalities,
> and at what extend they could choke, on some Bytes
> of the Double Bytes.
>
> For example, RosAsm always parses the Strings
> from Quotes to Quotes, on a _BYTE_ Basis scan.
> Let us imagine, that, inside some Double Byte,
> the same Byte as a Quote would be inserted from
> a Unicode String... Can this happend? i simply
> do not know (It would create an "Unpaired Quotes"
> faultive error Message)...
Note: There is a _DIFFERENCE_ between "multi-byte" and "wide character"...
"Multi-byte" - like you're talking about above - is how things like
oriental characters were dealt with _BEFORE_ UNICODE...
If you go for a UNICODE approach - using "RegisterClassW" and all the other
"W" API - then this should NOT be a problem...as this is the UNICODE "wide
character" approach...where - at least, in Windows' implementation of it -
every character is 16-bits wide...
So, if using "wide character" then your scan should be much the same,
except, obviously, that each character is 16-bits wide, so it's a _WORD_
basis scan, not a BYTE basis scan...all the ASCII codes are, as you
probably know, in the same places in UNICODE - just pad them with a zero
byte to make them 16-bits wide - so the actual logic shouldn't actually
need much, if any, changes but to, you know, make it a WORD-based rather
than BYTE-based...
The "multi-byte" stuff is NOT actually "UNICODE"...that is the "confusing"
stuff that existed before UNICODE to deal with oriental characters...
UNICODE is called "wide character" - which, by the way, is what the "W" in
the API names stands for - and "multi-byte" is the "old way" to deal with
oriental characters...
And, indeed, that's what's so useful about UNICODE: It makes a standard for
a "wide ASCII", so to speak...makes the characters bigger at 16-bits...and
then it can deal with all the characters in all the languages and scripts
in exactly the same way...it makes it all much simpler, as a kind of "big
ASCII"...instead of all the "multi-byte" / "code pages" confusion (which
_IS_ terribly confusing and awkward: That is, in fact, the underlying
reason why UNICODE was invented...to stop it being quite as "confusing", as
well as to allow all the different characters to be dealt with at the same
time (as with "code pages", you could only select one of them at a time, so
couldn't have oriental _AND_ English _AND_ Hebrew all in the same
document...which was another big "drawback" of the "code page"
system...very annoying, for instance, if your a language student and you
want to write an Email in English but, obviously, you want to be able to
include "Runic" characters in order to discuss the "Runic" language with
other people...as I said in that other post, without UNICODE, it would, in
fact, be _impossible_ to send that post any other way :)...and to try, as
far as possible, to make it work much like ASCII works but it's just a
"bigger character set" with "wider" characters :)...
[ Yes, since Windows included UNICODE support, the UNICODE standard now
includes _more_ than 16-bits of characters...note, though, that only
"historical scripts" - ancient egyptian hieroglyphics, for example (don't
think too many RosAsm users will want that, eh? ;) - and thousands of
oriental "ideographs" are actually in this "upper range" at the
moment...everything else is in the lower 16-bit range, so it's only a
problem for oriental ideographics, really...basically, Windows implements
an earlier standard...but one that's "good enough" for most purposes,
except these "extra characters" that have been added for oriental
"ideographs"...there are "special characters" which "escape" into the
"upper range" added to accomodate this...note that the problem is really
with Windows having "jumped on" early when it was only designed to be
16-bits in size...if implementing UNICODE _now_, then Microsoft would
probably use a "wider character" again to deal with it...16-bits, though,
covers 99% of the people (even the oriental ideographs: The ones in the
"upper range" are "additional"...plenty do exist in the first
16-bits...just that, well, they grossly underestimated just how many of
these ideographs actually exist...probably some 64K worth of ideographs
exist _alone_, without anything else)... ]
> >> As for writing, as you suggest, say, the labels
> >> Names, in Unicode, directly in a RosAsm Source,
> >> this will never be implemented. I do not see any
> >> reason for doing so.
> >
> > So your source code will forever be ASCII?
>
> The Assembly Source, Yes. Unicode will be (or are)
> for Data Strings only, and possibly, maybe, for
> Comments.
Yes; But, as noted, in order to get Windows working in "UNICODE mode" - so
it sends UNICODE characters with WM_CHAR - then you need to call a few
UNICODE "W" API to set that up...plus, to display the characters in the
editor, you'll need the "TextOutW" (and appropriate fonts...but, well, the
Indian user will surely have the right Indian fonts on their Windows, even
if you don't...and there is always that "Code2000" shareware font, I
mentioned before, delibrately designed to include as many characters as
possible, for "testing purposes" (and also just for "fun" in looking at all
the new UNICODE characters...I like the little "snowman" character in the
"dingbats" range...part of a set of "weather symbols", like a shining Sun
for "sunny" and an umbrella for "rain"...the snowman being "snowing", I
presume...there's a lot more characters than simply languages in UNICODE
too, you see...those DOS "box drawing characters"...the standard
mathematical symbols...circuit board diagram characters...arrow
shapes...chess pieces...signs of the zodiac...and many other things like
that ;)...Microsoft's own "Arial" and "Courier" fonts - included with
Windows - cover a large range too, on the UNICODE-based OSes like NT / XP
and so forth :)...
So, you will need to call UNICODE API here and there...but, yes, the basic
change is just that the data strings all have 16-bit wide characters...and
you _shouldn't_ need to do too many changes to support UNICODE...
At least, part of the idea behind UNICODE was to try to make supporting all
the various languages in the world just as simple and standard as with
ASCII...a "bigger ASCII", if you like...so, though there are some
"exceptions", the basic idea is that it _SHOULD_ more or less be as simple
as making your data strings use 16-bits per character rather than 8-bits
per character...but, otherwise, basically the same to program as ASCII
always was...so, generally, it _should_ just be (more or less) a case of
using the "W" API instead of the "A" API and that the strings are twice as
big (because each character is now 16-bits, not 8-bits :) BUT that there
shouldn't be any other real differences in your programming...a few
"exceptions" do exist - the "byte order mark", for example, isn't a real
character but an extra "control character" (there are a few extra "control
characters" added that you might want to process specially - just like the
ASCII "control characters" - that may add a few more "exceptions" to look
for in your processing :) - but the general "theory" of it all is that you
should really only need to call the UNICODE versions of the API (with
strings that are 16-bits per character instead) while everything else
_SHOULD_ broadly "stay the same"...
> But, for naming the Labels, sure, not, and never.
> That would be much no use complication, IMHO.
Yeah; It might be "interesting" - as some "mental exercise" - to do such a
thing...but, in practice, would anyone use it? Would it be useful at all?
In "principle", you could do it...but, in _practice_, the usefulness of it
is questionable...
Especially when the mnemonics and things are not going to change, so it
would probably prove incredibly awkward, anyway...you know, you type in
Latin for the mnemonics, then have to "switch keyboard layouts" to Russian,
type in a label, switch back to a Latin layout, type some more, switch
"layouts" again, etc., etc....you can set up "shortcut keys" to "switch
layouts" quickly in Windows and such...but, well, you also have to consider
the practicality of _typing_ this out...because the mnemonics are already
Latin, then the programmer will need to be able to type those
characters...and switching back and forth? Well, I think even if you
_could_ allow Russian or Japanese programmers to type in such labels, it
would prove so practically awkward for them to "switch" all the time, that
they'd willingly just stick to Latin to save the effort and bother...
I'm not against the idea itself, though...but, in practice, it'd be a lot
of effort to implement, when probably no-one really actually wants
it...that's why, for LuxAsm, I was only thinking about "basic support" for
the comments and the strings and that kind of thing...perhaps, if really,
really bored and there's nothing else that needs doing, it could be
"extended" to labels too...but it's hardly a "priority", even for those
programmers out there who like the idea of being able to type things in in
their native languages directly...
The point to remember, basically, is that a programming language is NOT
quite the same as a natural language...it is, to an extent, a "language of
its own"...after all, does assembly code really look like any natural
English text you'd read? No...show that source code to most ordinary
English speaking people (who are not programmers) and it might as well be
Chinese or some other language they know nothing about...
For some "international word-processor" then, yes, you need to have a high
level of "support"...but, for programming languages, the _real_ practical
benefit is just to be able to type those Russian "error message" strings in
Russian directly...it's more "convenient" and doesn't require too much
"messing around"...if you can type it with your keyboard then you can just
type it directly into the source code...this could be useful, if a
programmer is, for example, writing a program for a Russian audience, where
they want all the text and error messages and menus and things to appear in
Russian...it'll make it as easy for them to program such things as it is
for us Latin (especially English, where we don't even have "accent marks"
or anything, just the basic Latin characters ;) users to do with our source
code...you just type in the strings directly with your keyboard in the
"data section"...it just makes it as easy for them to do it as anyone
else...
Also, it should be stressed that UNICODE _ISN'T_ solely about "languages"
(that's the "main focus", obviously, but not everything :)...there are
actually other potentially useful characters in there, which can benefit
everyone, whatever their language...
You can pop some "box drawing characters" around your console applications
to get that DOS-style windowing working...or, if you want to write a chess
program, then there are actually chess characters in the "dingbats", so,
with UNICODE, you could potentially write that as a "text mode" program but
still have recognisable "king" / "rook" / "knight" pieces on the
screen...or, if you're doing a card game, then there are the "suits" of a
deck of cards...
Perhaps you want to do some "astronomy" program: There are the standard
symbols for all the planets (plus Sun and Moon :) included...if you want to
write some astrological "horoscopes" program then the signs of the zodiac
are there too...
Or, as noted, you might just simply use something like the historical
"Runic" characters as some "elf language" in a text-mode adventure game...
You can improve your "plain text editor" by using the "control pictures"
characters (these are _visible_ characters for the ASCII control
characters...so, you could have a "mode" - a bit like on Word - where you
can select to _see_ the spaces and tabs and newlines :)...
The IPA ("international phonetic alphabet") characters are included, so you
can use that to show "pronounciation" of words, in the usual "standard"
that dictionaries use to show how to pronounce words...that could be useful
for some program out there, perhaps...some "learn how to speak Spanish"
program (which can do it all and still actually be a "text-mode" console
program)?
There's even Braille pattern characters included...which might immediately
seem a bit silly but, of course, this could be used to send Braille files
to some "Braille printer", making it easier for blind and partially sighted
people to store text on their computers (note: These characters demonstrate
what I was saying about UNICODE being about "assigning numbers to
characters" and the actual "glyphs" for these is another matter...you know,
UNICODE supports the ability to _store_ Braille patterns in files, even if
"glyphs" for these characters would be a bit silly for blind
people...special programs can be written to interpret these characters in a
sensible way...UNICODE is just about the "storage" of the characters
themselves :)...
There are the six faces of a dice...some "dice game" could use those...and
there's just a bunch of "geometric shapes" - circles, half-circles,
diamonds, etc. - which could be used for lots of things...there's also
"terminal graphics" characters, which do that "teletext" type of simple
graphics...
Actual proper "roman numeral" characters (semantically distinct from just
using the Latin characters...so, for example, a program can know the
difference between "roman numerals" and letters...or, in drawing the
characters, it can automatically draw those horizontal lines above and
below the numerals, just like the Romans used to chisel their numbers into
stone :)...
Currency symbols for some "accounts" programs or whatever...musical symbols
for a music program (there's proper "treble clef" and "bass clef"
characters, for example...so, Rene, if you came up with some new jazz tune
with your guitar, then you might be able to use those to send it to other
guitarists, in a UNICODE text file, but where you've got the correct
musical notation :)...
Or how about an "advanced calculator"? There is a full complement of
mathematical characters to be used...so, you could even make a "calculator"
that could deal with mathematical predicates directly, perhaps?
There's also plenty of characters in the UNICODE standard that, in fact,
have nothing to do with "languages" at all...but are just very "handy"
characters to have, in general...
So, it isn't just about allowing those Russian programmers to input Russian
strings...you might be writing an English-based program but might find some
of these other characters "handy" for different purposes...there are enough
other "useful characters" in UNICODE to justify it all alone, even if you
didn't want to bother with "other languages"...I'm thinking, for instance,
that Paul's "adventure games" could be made very interesting indeed - "elf
language" (like Tolkien, just "borrow" a script no-one speaks anymore like
"Runic" or "Ogham" or "Gothic" ;), "box drawing" around things, an umbrella
appearing in the top-right corner to indicate that it's started "raining"
in the game world, you meet a dwarf who challenges you to a "dice game" in
the tavern, etc. - without even becoming a GUI (still just a "console"
program :)...
> I Have read a long (as usaul...) Post, from Beth,
> some long time ago, where she was, if i understood
> well, promoting a kind of UTF something, that should
> be the universal default. As i was not much interrested,
> i did not read it carefully, but...
Not a "universal default" - you weren't reading carefully, were you? - but
that UTF-8 is actually an "encoding" for UNICODE which is "ASCII
compatible" (for all the ASCII characters, the UTF-8 encoding is exactly
the same as it always was :)...and is "byte-based"...this UNICODE format
could be useful for programs that are already ASCII-based to easily
"upgrade" them...it's the "encoding" that Linux uses for UNICODE...
> Beth... maybe.. a couple of words would help him...
Maybe a couple of words may help you too about how to potentially implement
UNICODE in RosAsm, as you did (indirectly) ask how the WM_CHAR works for
Windows in "UNICODE mode" :)...
Beth :)
Beth
> Beth wrote:
> > [... Shames...]
>
> Ah! I was wondering how long i would have to
> wait before you could came at the rescue of
> your good old friends, master Randall Hyde,
A complete fabrication; At no point in my post did I make any reference to
HLA, AoA or Randy or anything remotely related to him or his software
tool...
Not a single thing...re-read it...don't "snip" all of it away, so that you
can pretend there's nothing I said which you need to address...as I made
very specific points regards RosAsm string support to which you _DO_ need
to answer...or, otherwise, other readers will rightly conclude that I am
_RIGHT_ about what I say in every word...and you are "snipping" my post
then insulting it, as a means to "evade" answering the specific charges
made against you and your software tool...
Re-read it and if you find a single reference to HLA, AoA, Randy, anything
Randy has said or otherwise then _POST IT HERE_...bring this to everyone's
attention to _DEMONSTRATE_ your "accusation" that I was in any way
"defending Randy"...not that I wouldn't do so or haven't done so...but, in
this case, you've made a specific charge about a specific post as doing so,
yet that post does NOT contain what you say it does whatsoever...
Indeed, you remove my post _ENTIRELY_ in quotation and then simultaneously
claim it says things it DOES NOT say...you are delibrately trying to
misrepresent what I'm saying while failing to provide even a single line of
the post to which you are objecting and creating "terrible illusions" about
what it _actually said_...
> and Hutch--, our prefered Right-Wing bastards.
I don't think I've ever uttered a word of support for hutch's repeated
childish postings...and would certainly not "defend" him whatsoever, as I
often find his postings to be repugnent in their content: All insult, no
actual useful point worth making (at least, in the majority of cases)...
What I specifically said was:
Beth wrote:
> Ah, but that's your first mistake, Chewy: Randy is developing a "HLL
> pre-parser sub-shit swindling trap", not an "assembler" ;)...
Does this in any way misrepresent your opinion of HLA?
Yes, I have fielded a number of your insults against Randy and HLA into
"one big insult"...but all of these are _YOUR_ charges against Randy and
HLA, are they not?
You consider me merely repeating _YOUR OWN ARGUMENTS_ back at you as being
"defending Randy and HLA" (while, in this post, NOT a single word is
mentioned of Randy, HLA, AoA or anything remotely related to them)?
So, you're telling us that your own argument "defends" Randy and "defends"
HLA?
Because in this post, all I did was begin by echoing _YOUR_ own arguments
back at you (and, yes, making a mockery of them...this would not be a
possibility, if they were not mock-worthy in themselves for being so vacant
in content, so transparent in motivation to attack others, so unfounded in
reality and such figments of your own imagination)...and then, after that,
I merely used _YOUR OWN WORDS_ from B_U_ASM to specifically demonstrate
your opinions and advice regards "string resources"...before explaining a
few of the uses - as suggested by Microsoft themselves (so how on Earth, in
creating a Win32 specific assembler, you did not yourself _READ_ these
pieces of suggestion and rationale from Microsoft yourself...well, this
suggests that you don't even really know the Win32 API set to a level of
sufficient qualification for the position of "Win32 specific assembler
author" to which you set yourself up as an "expert"...we've _ALREADY SEEN_
repeated and seriously misguided claims about Windows and Win32 from your
RosAsm "students", who appear NOT to be being taught the facts by you at
all...indeed, in your quest to "fight Microsoft", you've created an army of
"Windows defenders", who fight tooth and claw...not against Microsoft but
to _DEFEND_ them...hutch redistributes an actual Microsoft product and even
he has not been able to master creating such a devoted "army" of Microsoft
defenders as you are producing...other than MASM, the assemblers you
attack - FASM, HLA, NASM, etc. - are all capable of functioning on
non-Microsoft OSes...something your "weapon against Microsoft" cannot do...
No wonder you cannot address - or even stomach - looking and answering
_YOUR OWN ARGUMENT_ because it is so void of reality, it has gone and
achieved the _EXACT OPPOSITE_ aim to that which you claim you were a
"warrior" "fighting" against...you are such a disgrace to any "cause" of
fighting Microsoft that we seriously have to start considering whether you
are, in fact, _working for_ Microsoft to _promote_ Windows...and that your
claims otherwise are merely "reverse psychology" tactics (because though
you _claim_ to fight Microsoft, your every action _SUPPORTS_ them)...
What I specifically said was:
Beth wrote:
> While hutch is developing his "dictionary arse"...Thomasz and John Found
> are "assholes"...
Note: No mention of Randy or HLA or anything related to those things
whatsoever...
Did you or did you not charge hutch with being a "dictionary ass"?
Did you or did you not consider Thomasz and John Found to be "assholes"?
A simple "yes" or "no" answer will do...
I was only _REPRESENTING YOUR WORDS AND OPINIONS_...
Yes, I did so mockingly but I personally feel that such a vacant and
corrupt set of opinions deserves not to directly challenged and not
"tolerated" for politeness when they are so out-of-kilter with reality,
common decency, factual accuracy and so forth...
There are your own words...your own opinions...your own argument...if you
cannot bear to look at it...if you cannot in any way address it...if you
feel you must "evade" even your own opinions...well, then you already know
what's wrong...and you already know what must be done...
You claim to be a "warrior" and not a coward...so, well, if you would
rather it be you who dons the "white knight" armour and comes in "rescue"
and defence of the _TRUTH_, then I'd be more than happy to hand it over to
you, should you prove chivelrous enough to qualify for the position...I am
not fond of fighting or swords or such...it would make sense that a
"warrior for truth", as you seem to want to be, were to take this role of
defence rather than me...but as you're actually more concern with saving
your own ego (which you appear to "keep safe" up your own arse) than
usefully fighting for truth, then it would be grossly irresponsible to
leave things in your charge...
> < http://rosasm.org/ >
I urge others, who have the time and disk space, to go to this URL and
download RosAsm...give it a try...
I'm confident that most people will feel that Guga's gargantuan effort to
compile the "equates list" and other comprehensive Win32 support - which is
a true convenience (though, other than accessing it via a dialogue box, how
this differs from a set of full and comprehensive header files for other
assemblers like MASM, NASM and such, is anyone's guess) - has been _GROSSLY
LET DOWN_ by Rene's "barely passable" tool...
I second Randy's opinion that you should be pitied because the _potential_
of RosAsm was immense...but what you've created is a pale shadow of what
you've promised...indeed, the comprehensive "equates list" support _IS_ one
of the best things in RosAsm and it was _GUGA_ who worked on that...the new
"visual wizards" for creating windows that are being developed _by others_
looks excellent...while _your_ "icon editor" is a slap-dash affair that
only supports one size, 16 colours, provides no tool but pixel plotting and
has rather "cryptic" button controls...where, somehow, we're supposed to
guess whether "<" is "load" or is ">" the "load"? Left and right arrows
have no relation to "load" and "save" (except perhaps in your mind)...your
hash algorithm had problems that you wouldn't even acknowledge were
there...
There is a clear pattern: What you, Rene, code for RosAsm is typically
"slap-dash" with no care and "half done" while you hand over the _real
work_ to Guga and others, who's excellent efforts are _GROSSLY LET DOWN_ by
being attached to a tool that has "half done" written all over it...and,
from what you've said are your aims for RosAsm, then the actual
implementation of those aims are a severe let-down...you had such potential
but have, basically, squandered it all by diffusing your efforts and
spreading them so thinly - spending more time attacking others - that what
you personally have produced is nowhere near fulfilling the immense
potential RosAsm had...you've let your strange "philosophy" get in the way
of delivering the tool that _SHOULD HAVE BEEN_...
> PS. Interresting to notice that, when it is
> for denigrating RosAsm, you are able to read
> B_U_Asm, and to Copy-Paste the parts where
> i honestly talk about RosAsm real limitations,
> whereas, when it is for some interresting
> feature, you simply... do not know... :)
Untrue; There is no "editing" applied whatsoever...what is annoying you
really is that I did produce a RosAsm example (a "five minute" job...but it
was only to demonstrate a particular point being made about Windows, so
wasn't worth wasting more time than that on it)...so you know that there is
no "phantom book review" here whatsoever...I have used your tool to create
a working application (okay, not a terribly exciting or big
application...but it did the job it was designed to do, in making a point
clear)...and, as such, I _DO_ have at least a basic appreciation of what
RosAsm offers the programmer...and that I'm still not impressed in the
slightest...
It is possible to create slap-dash monolithic "demos" with RosAsm...but,
for anything more complicated, RosAsm would start to get in the way with
its backwards "philosophy" and inadequate "limitations" everywhere, while
NOT assisting in writing the code...
What is that phrase which teachers like to write on school report cards?
"Could do better"
Well, in your case, "could do an awful lot better"...as this is even what's
worse about RosAsm: It's not even really making it up to the standards of
what you talk about in your "philosophy" rants (which themselves are
unnecessarily limiting and restrictive to no useful purpose for the
programmer)...
> Yes, the RosAsm Resources Manager is limited.
And as there is no other choice with RosAsm "philosophy", don't you feel
that this limitation is an inherent limitation of RosAsm itself?
> The Lang IDs are not yet implemented, and the
> Named IDs are only implemented for the Disassembler,
> but not yet for the Assembler... Yes...
>
> SHAME ON YOU, Beth!
I see absolutely no shame whatsoever for bringing a legitimate and specific
concern about the limitations of RosAsm - that which you yourself above
completely confirm _ARE_ actual limitations on the programmer using RosAsm
(which I know because I've given full and fair attempt at using your tool
throughout its development - as I do for most tools - and tracked its
progress and have attempted to create code - such as the example posted -
using your tool...and, repeatedly, I always find the inherent limitations -
some of which, unbelievably, you _design_ to be present in the tool
(absurd! Creating a tool with the _intention_ of crippling it? Are you a
sadist or something?) - present in RosAsm to be so restrictive and that
continually getting in the way of progress, that, indeed, I have come to
the conclusions that I have and voice these _LEGITIMATE CONCERNS_ about
RosAsm publicly) - to yours and public attention...
On the contrary, Rene...shame on _YOU_...because you have stated on this
newsgroup repeatedly before that you felt RosAsm was effectively "complete"
as it stands...and now you openly admit to being perfectly aware of the
failing of "the RosAsm Resources Manager is limited" and a list of
unimplemented and half-done attributes in RosAsm?
That would be "false advertising", would it not? Delibrately and
consciously done that, yes, _SHAME ON YOU_ for willingly doing such a
thing...and as you seem to have no conpunction about outrightly _lying_
regards RosAsm, how exactly do you expect us to believe anything you say on
these points? You tell us that "monofile programming" is great and
excellent (though, those with sufficient programming experience _ALREADY
KNOW_ that's another falsehood to "sell" your assembler with "terrible
illusions" on "young heads")...but you've tried to suggest that RosAsm was
"complete"...yet when we ask you about specific facilities (and, as noted,
the "LANG ID" on string resources is pretty important because _THAT_ is the
typical advantageous use of having them in a program...the other advantages
of "string resources" you've already "nullified" by RosAsm's "monofile"
design itself), you tell us that "this is unimplemented, that is
unimplemented, something else is half-done"? You say "complete" when you
want to "sell" it but admit to "half done" and "unimplemented" when
actually forced to address _specific_ concerns about RosAsm...
If your word cannot be trusted on this point, then how can it be trusted on
other points like the supposed advantages of "monofile programming"?
Especially because "monofile programming" _IS_ completely supported by
"modular" toolsets...it's simply the "special case" of only using _ONE_
module...anyone who's used a modular tool such as NASM or HLA or FASM or
_ALL_ the others (as only your tool fails in this regard, by _delibrately
crippling itself_ to a single case, for no validly supplied reason you've
ever given) will know perfectly well that if they _WANT_ to put all their
source code into a single file, then they are completely free to do so
(provisor: Resources are placed in a separate source file for the resource
compiler...BUT, in RosAsm, resources are NOT available in source code form
_EITHER_ and are dealt with through the GUI interface...thus, you are no
more advantageous then any other tool on the "exception" of Windows
"resources"...indeed, only _FASM_ - as far as I'm aware - directly supports
source code-based resource definition and, thus, is a _MORE APPROPRIATE_
tool for "monofile programming" than RosAsm (other than any assembler with
"flat binary" mode - such as FASM or NASM - can also be used to, of course,
"hand-code" any file at all...including resources and such)...
Hence, you present a needless _CRIPPLING_ of your tool to fully
_UNNECESSARY_ limitations and restrictions - which provide NO ADVANTAGE to
the programmer at all (but perhaps _YOUR_ "convenience" in programming it,
when you've made it so intrinsically "simplistic") - as a "benefit" of
RosAsm...this is utter nonsense...
Indeed, all the other tools are just as capable of "monofile
programming"...which is proved _REPEATEDLY_ on this newsgroup all the
time...because most source code posted tends to only require a single
source file, anyway...they can do what RosAsm does, no problems...RosAsm
cannot do what they do because you've delibrately _crippled_ your
tool...and have NEVER satisfactorily provided an answer to why you've done
this..."because libraries bloat programs"...yes, if _ABUSED_, they very
much can do...but, by the same token, "cut and paste" methods will _EQUALLY
BLOAT_, if used incorrectly (for example, instead of creating a subroutine,
a programmer simply copies and pastes the same code all over the place
repeatedly...unnecessarily "bloating" their code for "convenience":
_EXACTLY THE SAME CHARGE_ that can be made at those who _ABUSE_
libraries...
You have made a fundamental philosophical mistake in thinking that a _tool_
in any way is responsible for the _ABUSES_ programmers make of it...this is
a fundamentally flawed assessment...and you've cornered yourself into
chasing an impossible "ideal"...
Libraries may be "abused"? Remove libraries so they can't be used!
Macros may be "abused" for HLL-like features? Cripple...nay, remove the
macros!
"Cut and paste" may be "abused" to get library-like functionality? Remove
"cut and paste" so it can be used!
Saving source files separately might be "abused" to get library-like
functionality? Remove saving source file separately!
The "DB" feature might be "absued"! Limit...no...remove the "DB" feature!!
The machine instructions might be "abused"! Remove the machine
instructions!!!
What is going to be left if you continually pursue this flawed thinking?
Can you see that the whole premise of what you're doing is based on "bad
thinking"...that it has falsely decided that it's a tool's responsibility
to limit and restrict the programmer's means of "abuse"...
Yes, the facilities you provide might very well be "abused" by the
programmer...they may produce "bloated" code from doing so...they may use
macros to do terrible "HLL" things...but removing those facilities does NOT
in any way limit or prevent the capability of the programmer to _ABUSE_
whatever features you provide in ways that you do not like...
But, thank you, you admirably prove the underlying philosophical point with
an actual, real example:
"Creating more laws only creates more criminals, not less crime"
America also learnt this lesson when it prohibited alcohol...this did not
make the abuse of alcohol any less...in fact, it made it dreadfully
worse...because, once illegal, there were no "checks" or "balances" - no
"regulation" - applied...driving the otherwise law-abiding citizens into
the "speakeasy", where it was NOT solely alcohol that was "illegal"
there...introducing people to _FURTHER ILLEGALITY_...forcing them into a
"life of crime"...being illegal, the price sky-rockets (well, got to pay
your dealer some "danger money" for their efforts)...those addicted to the
drug will not cease their addiction when it's unaffordable (it would not be
"addiction" otherwise)...they will just take to stealing and other crimes
to raise the money to get their "fix"...so, rather than "cleaning up the
streets", such attitudes contribute to making them far worse...
If you want to stop "bloat" and such, Rene, then _EDUCATE_ people as to
what causes "bloat", what they can do to avoid it, supply "tips and hints"
from your programming experience of how to achieve things in "non-bloated"
ways...and so forth...kill the problem at its _SOURCE_...which isn't the
tools they use but their _THINKING_...
It's amazing to hear you condemn "conservatives" in America as
"Nazis"...and, yet, the first thing you think of when there's a problem?
"Prohibition!", "Attack!! Kill!!"...claim whatever you like for yourself
but that _IS_ a "conservative" mindset...well, no, to be fair to the more
moderate conservatives out there, it is far further "right-extreme" than
that, heading towards "fascist", really...yes, the self-proclaimed
"left-wing anarchist" (an oxymoron in itself), who, in actual fact, behaves
more conservatively - so "right-extreme" as to begin to qualify as
"fascist" in sentiment - than nearly anyone else on this entire
newsgroup...
Liberty is not the problem, Rene...so "limiting" and "crippling" and
"prohibiting" won't do a blessed thing to "solve" anything...Liberty is the
_SOLUTION_: Use your "freedom of speech" to _EDUCATE_ people...use that
Liberty which allows you to shout your "cause" as loud as any "HLL
theoretician", in order to _EDUCATE_ people of the truth...of how to get
out of "bloatware" mindsets...of how to create _efficient_ software...of
the merits of assembly language..._USE_ your Liberty - do not abuse it - in
order to _DO GOOD_...
Beth :)
--
"We need to counter the shockwave of the evildoer...by thinking about tax
rebates."
[ George W. Bush, Jnr. ]
Betov
news:2CI9e.24732$443....@newsfe3-gui.ntli.net:
> A complete fabrication; At no point in my post did I make any
> reference to HLA, AoA or Randy or anything remotely related to him or
> his software tool...
You know, not only i am able to read your Post diagonaly,
but i am also able to read in between the lines you write.
... And what i see is really pathetic...
Yes, shame on you, Beth.
Betov.
Evenbit
webs...@gmail.com wrote:
> A lot of effort for cornering every market outside of english. If
> programmers from non-english speaking countries could code in their
own
> native tongue, it would probably be very attractive to them.
There is your "Assembly Rebirth" in a nutshell. If the currect
assemblers and their associated libraries, macro facilities, and
compile-time languages supported Unicode then programmers would have a
very compelling reason to learn assembly.
Just need Unicode support in our Usenet groups and forums. Perhaps
posting code examples using UUencode, Yenc, and such?
Nathan.
Betov
news:6CI9e.24733$443....@newsfe3-gui.ntli.net:
> First things first, when you register your RosAsm windows classes, you
> should use the "RegisterClassW" API, not the "RegisterClassA"
> API...the reason for this is that Windows takes this as a "cue" as to
> whether to send the messages with ANSI / UNICODE parameters in ANSI or
> UNICODE form...
>
> If you register with "RegisterClassA" - as you're probably doing -
> then Windows takes this to mean that you want your application window
> to receive ANSI messages (some messages have ANSI / UNICODE
> variants...for instance, the character provided by "WM_CHAR" is an
> ANSI character when the window is "registered" as ANSI...but it will
> be a UNICODE character returned by "WM_CHAR" when the window is
> "registered" as UNICODE :)...so, yes, you won't see any UNICODE
> characters appearing with "WM_CHAR", _IF_ you used "RegisterClassA"...
In fact, i do not want to upgrade RosAsm to Unicode,
at all. I just wish that the "Unicode Programmers",
writing for "Unicode Users" could write Applications
that would make use of Unicode Strings. Not RosAsm
itself. I see no reason, at all, for a Programming
Tool to be any unicode, internaly. The Programmation
is to be done in... English (unfortunately...) as
well for a Chineese as it is for me, as i have no
other choice for anything (Documentation, Menmonics
Names, exchanging with everybody, and so on...).
Personaly, i would much prefer that the international
Language could be Esperanto, but well, it is English,
and the game is over, at this point of view, also...
Also, switching to Unicode (whatever flavour of
Unicode...) can not be simple, in a monster App
like RosAsm, where nothing had ever been desing
for this. There are tons of features scanning the
Source, and all of them should, evidently, require
some modifications, even with UTF 8, in order to
not choke on any additional Char, that would conflict,
as i am afraid they could very easily do, when some
"critical Chars", that are ruling the scans...
The simper thing i want, is that, when declaring
Unicode Strings, like:
[MyUnicodeString: U§ '???????', 0]
The Programmer could input real Unicode Chars
into the String. But, even for something as
trivial as this, i am 100% sure, that there are
scan Functionalities that should be reviewed, in
order, for example, to not choke, on a _Byte_,
inside the Unicode String, that would have the
same walue as a Quote... This is an absurd problem,
when nothing has ever been designed since day one
to assume such cases >>> Much review work, for a
very small thing...
If this is too complicated, i will do it through
a "Tag" feature, runing a dedicated Unicode Dialog
for Unicode Strings pasting Back and Forth...
The RosAsm Editor Window is not an Edit Control.
This is just a flat "Drawing Surface". I do not
know at what extend, like you say, the fact of
having this window itself made Unicode Mode could
inpact on the Messages, but there had been an
oriental User, who implemented a "&WM_IME_CHAR"
case, in the MainWindowProc. So, i suppose that,
when he implemented this, he must have seen something
on his screen. Why would he have committed, if not?
For now, we have only one Oriental user (YeoHs),
who does not seem to be much concerned with these
problems, and if i understood well, in fact, i
think that he does not use really any Chineese
Chars Set, in his daily works... not sure...
Nevertheless i just asked him the question about
the effectiveness of this actual "&WM_IME_CHAR"
thingie... Wait and see.
Betov.
webs...@gmail.com
> websn...@gmail.com wrote:
> > For example, there are letters which can take multiple accents.
> > And you can often specify them as base-char + accent1 + accent2.
> > The question is, if you flip the order of the accents, does that
> > represent a different character or not? The unicode
> > normalization algorithms say no for some cases, and yes in others.
>
> many cases (now that I'm looking more closely at the actual
> encodings and character maps), that some characters, particularly
> from the Latin set, have direct equivalents, eg Latin Small Letter
> A With Tidle, can be mapped to either U+00E3 or U+0061 + U+0303,
> (which is Latin Small Letter A with Combining Tidle) which makes
> comparison even harder.
>
> While it's stated that any editor conforming to Unicode 4.x must
> produce shortest form (which negates the issue I raised), however
> copying from other sources the exact encoding format MUST be
> preserved. So that if I have an editor which only conforms to
> version 1.x, which produces the long form (v1.x IIRC doesn't state
> which form to produce to be conformant), and I copy it over to
> another editor which produces short form, the second editor (to be
> conformant) cannot and should not convert long form to short form,
> even though the short form is considered correct.
Huh? I don't think any sequence of Unicode code points is strictly
illegal. Its just a question of equivalence or what it is you could
possibly mean by certain encodings. For example if you just had a
string with a single accent combining code point, what kind of meaning
would you expect that to have for anyone? You can also do crazy things
like trying to put an accent on a letter than doesn't ordinarily take
one, like a q with a grave accent. It, of course, doesn't mean
anything in any language, but Unicode doesn't prevent you from trying
to express such a thing.
There is, of course, a funny grey area, where you can know just from
the encodings alone that something is wrong because you don't start
with a base (non-combining) character, or the same accent is being
repeated between base characters. Even though you know that's
gobbledygook in all languages, Unicode doesn't say you must detect and
reject such strings.
The point of normalization, as I keep saying, is for string comparison
purposes. This is how its even plausible for a glyph renderer to know
if certain characters are supposed to combine in a practical way.
First you normalize them, then you use whatever look ups or other
tables to tell you if the result is supposed to kern or combine or
whatever. Non-normalized Unicode strings are not illegal or even
deprecated -- they are just non-normalized. That's all.
> Now my head hurts...
Yeah, but that has a lot to do with the piss poor way the documentation
has been written. I had to read it multiple times before I realized
that the NormalizationTest.txt was the path to answering basically all
the mapping questions I needed answering.
> And I can see the resistance to allowing full unicode support for
> labels/identifiers.
>
> I would just like to thank everyone that has replied and voiced a
> constructive opinion on this topic.
>
> I will have to admit, supporting unicode is a bit more work than I
> first thought! :(
Its not that trivial if you want to code up the minimum expectations of
Unicode users. As was pointed out, NASM has lame "default"
(essentially, "ASCII-8") support already, but that's a little too weak.
It sounds like Randy Hyde can copped out with an partial I18N or
similar such solution (which allows him to just default to "MBCS" or
whatever.) Doing a full M17N (Unicode) implementation takes some
non-trivial, but not insurmountable effort.
Betov
>> < http://rosasm.org/ >
>
> I urge others, who have the time and disk space, to go to this URL and
> download RosAsm...give it a try...
So do it... :)) :)) :))
> I'm confident that most people will feel that Guga's gargantuan effort
> to compile the "equates list" and other comprehensive Win32 support -
> which is a true convenience (though, other than accessing it via a
> dialogue box, how this differs from a set of full and comprehensive
> header files for other assemblers like MASM, NASM and such, is
> anyone's guess) - has been _GROSSLY LET DOWN_ by Rene's "barely
> passable" tool...
???!!! Too bad that Guga is actually "Off"... he could have
answered better than me...
> I second Randy's opinion
Sure. When did you not? :)) :)) :))
> that you should be pitied because the
> _potential_ of RosAsm was immense...but what you've created is a pale
> shadow of what you've promised...indeed, the comprehensive "equates
> list" support _IS_ one of the best things in RosAsm and it was _GUGA_
> who worked on that...the new "visual wizards" for creating windows
> that are being developed _by others_ looks excellent...
Indead. The same guy who wrote the Form Wizard is also
the Maintainer of the Equal Pre-Parser. He just sent me
the next version today... Really great... :)
> while _your_
> "icon editor" is a slap-dash affair that only supports one size, 16
> colours, provides no tool but pixel plotting and has rather "cryptic"
> button controls...where, somehow, we're supposed to guess whether "<"
> is "load" or is ">" the "load"? Left and right arrows have no relation
> to "load" and "save" (except perhaps in your mind)...your hash
> algorithm had problems that you wouldn't even acknowledge were
> there...
True! The RosAsm Icon Editor is a little shit, i wrote
quick and durty, in the very first days of SpAsm. It
now deserves a complete rewrite. And then???!!!...
Mind you this is the kind of task i keep on the ToDo
List because it might be an interesting asignement for
a Volunteer who would ask about what to contribute to.
And then???!!!...
> There is a clear pattern: What you, Rene, code for RosAsm is typically
> "slap-dash" with no care and "half done"...
Sure. We will see if you are able to write a 100% done
job for LuxAsm in a couple of years... Good luck...
:)
>> [...]
>
> It is possible to create slap-dash monolithic "demos" with
> RosAsm...but, for anything more complicated, RosAsm would start to get
> in the way with its backwards "philosophy" and inadequate
> "limitations" everywhere, while NOT assisting in writing the code...
>
> What is that phrase which teachers like to write on school report
> cards? "Could do better"
Hi! Mistress Randall Hyde! How are you?
:)) :)) :))
> [...]
>
>
> ...the other
> advantages of "string resources" you've already "nullified" by
> RosAsm's "monofile" design itself), you tell us that "this is
> unimplemented, that is unimplemented, something else is half-done"?
> You say "complete" when you want to "sell" it but admit to "half done"
> and "unimplemented" when actually forced to address _specific_
> concerns about RosAsm...
Mind you, i have to focuse on what seems to me
to be the actual Priorities. For example, today,
i have fixed a couple of minor problems between
the IDE and the Equal Pre-Parser [Unfolding].
I have fixed a minor bug in the Editor. I finished
also the Implementation of the Messages Strings
Internationalization, that i will probably release
tonight.
... And it is like this, every days. So, is, or is not,
the Icon Editor a decent one, is, or is not, the Strings
Langs, an urgent task to do, and so on... these are problems
of choice of where to invest time, as i have only two hands,
one brain, and 10 hours a day, as my work power is slowly
degrading with age...
And then???!!!...
> [...]
>
> "resources"...indeed, only _FASM_ - as far as I'm aware - directly
> supports source code-based resource definition and, thus, is a _MORE
> APPROPRIATE_ tool for "monofile programming" than RosAsm (other than
> any assembler with "flat binary" mode - such as FASM or NASM - can
> also be used to, of course, "hand-code" any file at all...including
> resources and such)...
Mind you FASM, - that i _promote_ on a very regular Basis,
right here... - is so "well featured for Resources", that
... none of the FASM IDEs include any Resource management.
For example, Fresh, the most important one... Does not
do resources at all, and do all of the job with the help
of a Visual Designer for... windows...
:))
> [...]
>
> Indeed, all the other tools are just as capable of "monofile
> programming"...
Absolute LIE. None of the other Assemblers are able
of MonoFile Programming. That would be even more
difficult than with MultiFiles, mind you.
To assume real Mono-Files, you must implement the
required features enabling the Programmer with
dedicated features, like, typically, the RosAsm
Right-Click Feature, the TITLE Feature, the
Include Pre-Parser, and so on...
That is some REAL WORK!!! Mind you!!!...
> [...]
>
> Libraries may be "abused"? Remove libraries so they can't be used!
Hi Mistress Randall Hyde! How are you?
:))
> Macros may be "abused" for HLL-like features? Cripple...nay, remove
> the macros!
???!!!... Forgot your medication again???!!!...
> "Cut and paste" may be "abused" to get library-like functionality?
> Remove "cut and paste" so it can be used!
???!!!... Forgot your medication again???!!!...
> Saving source files separately might be "abused" to get library-like
> functionality?
>
> Remove saving source file separately!
Never heard of [Ctrl]/[S], and of its multiple options?
Too bad!!!... :))
> The "DB" feature might be "absued"! Limit...no...remove the "DB"
> feature!!
Never heard of DB in RosAsm???
Too bad!!!... :))
> The machine instructions might be "abused"! Remove the machine
> instructions!!!
??? Example ???
Did i miss something of the 32 Bits Set, Wolfgang? ;)
> What is going to be left if you continually pursue this flawed
> thinking?
Are you show this question should be asked to... _ME_ ???
Betov.
\o//annabee
>
> True! The RosAsm Icon Editor is a little shit, i wrote
> quick and durty, in the very first days of SpAsm. It
> now deserves a complete rewrite. And then???!!!...
I strongly disagree! The RosAsm icon editor is very nice. I created my
first real nice icon with it.
This is the "eagle icon" :))) And I think it became so nice, because I
spent so much time
with the editor.... The icon editor is holy to me !
One of the really nice things about RosAsm (programming atmosphere) is
EXACTLY that it does not do things exactly the way its been done
everywhere else. This is btw what *I* do many times, and this is why I
call my self wannabee. Someday I hope that I will have forgotten so much
of the past that I will be able to *create* something as well. As long as
the RosAsm tools remains usable, they should remain having their
unproffessional look. For instance, I instantly fell in love with the
original debugger, but it took a long time to appreciate the new debugger.
Atmosphere. Energy. Round things, as supposed to square things.
It was in fact RosAsm that reminded me of my first days with programming,
when I was actually able to do things in new ways, because I wasnt
burdened with redicolous socalled "knowledge". After being pressed through
the square asshole of 10 years of DELPHI programming, I allmost lost my
creativty, and RosAsm was the tool that reminded me of it. And my dream is
to be able to reenable it.
Do you think I am talking nonsense ? Well.. I think I am actually close to
some truth here. You just dont see how beautiful your solutions is. Dont
listen to the propaganda from theese infants !
>> There is a clear pattern: What you, Rene, code for RosAsm is typically
>> "slap-dash" with no care and "half done"...
>
> Sure. We will see if you are able to write a 100% done
> job for LuxAsm in a couple of years... Good luck...
:)))) This one probably has something stuck up its nose,
that should have been stuck up some other places instead.
:+)))) :+)))
> Betov.
>
> < http://rosasm.org/ >
>
Betov
news:opspk6hq...@news.broadpark.no:
> I strongly disagree! The RosAsm icon editor is very nice. I created my
> first real nice icon with it.
> This is the "eagle icon" :))) And I think it became so nice, because I
> spent so much time
> with the editor.... The icon editor is holy to me !
:))
Then, WE disagrea.
:))
The Icon Editor is evidently an uggly little thing, but
not only.
First, this is not at all what we need. We need, first,
a serious Paint _Control_, that could be called with
Parameters for defining the size of the object to
paint, so that we could use it, as well to draw an
Icon, a ToolBar Button image, or anything else.
Then, we need it to work in True Color, with downward
Computations for the other Formats, plus, assume other
Formats, in the Resources Main Icon than the poor
actual one.
Then, we need all of the usual Functionalities for
drawing other ways than by one by one Pixel Clicking.
:))
I just did this little Tool, because, when i started
RosAsm, i was in need of an Icon for... RosAsm itself
and, at that time (1998), i did not found out any GPL
Editor for doing this simple thing...
Betov.
\o//annabee
> "\\\\o//annabee" <http://www.TheWannabee.Org> écrivait
> news:opspk6hq...@news.broadpark.no:
>> The icon editor is holy to me !
>
> :))
>
> Then, WE disagrea.
>
> :))
>
> The Icon Editor is evidently an uggly little thing, but
> not only.
A bit uncommon, but it works very well.
> First, this is not at all what we need. We need, first,
> a serious Paint _Control_, that could be called with
> Parameters for defining the size of the object to
> paint, so that we could use it, as well to draw an
> Icon, a ToolBar Button image, or anything else.
ok.
> Then, we need it to work in True Color, with downward
> Computations for the other Formats, plus, assume other
> Formats, in the Resources Main Icon than the poor
> actual one.
What you mean True color ? All colors are available in the Icon editor !
> Then, we need all of the usual Functionalities for
> drawing other ways than by one by one Pixel Clicking.
>
> :))
Well, two pixels click, and you have a line.
> I just did this little Tool, because, when i started
> RosAsm, i was in need of an Icon for... RosAsm itself
> and, at that time (1998), i did not found out any GPL
> Editor for doing this simple thing...
Ok. If you will accept it, I can create you a bitmap editor, as an
external tool in a month.
Maybe two.
With sizeable, editable bitmaps. Actually, I allready have created some
framework for it. I could pass the bitmap on the clipboard back to Rosy ?
Reason I was not continuing with it, is because I had so many things todo
with the GUI before this could be needed, but now that my Open/Savedialogs
work, I have now code to load in a bitmap.
The only thing remaining is to be able to edit it. I allready have Paste
bitmap functionality.
I doubt it could take as much as a month if I let everything else go. I am
going to do it anyway.
>
> Betov.
>
> < http://rosasm.org/ >
>
>
>
>
Betov
news:opsplaea...@news.broadpark.no:
> What you mean True color ? All colors are available in the Icon editor
Unless "they" would have changed the Names... :)
... this is a Color Format, with full RGB, and
no Color Panel Header, in the File.
> [...]
>
>> I just did this little Tool, because, when i started
>> RosAsm, i was in need of an Icon for... RosAsm itself
>> and, at that time (1998), i did not found out any GPL
>> Editor for doing this simple thing...
>
> Ok. If you will accept it, I can create you a bitmap editor, as an
> external tool in a month.
In my opinion this is more than one month of work,
but well... as you like. ;)
[And this is not a "bitmap editor". It must look like
an Icon / Cursor / ... Editor, with Grid, like the
actual one... Plus Tools Box, and all...]
:)
Betov.
\o//annabee
> "\\\\o//annabee" <http://www.TheWannabee.Org> écrivait
> news:opsplaea...@news.broadpark.no:
>
>> What you mean True color ? All colors are available in the Icon editor
>
> Unless "they" would have changed the Names... :)
> ... this is a Color Format, with full RGB, and
> no Color Panel Header, in the File.
Ok.. You mean having a palette, where you dont ned to experiement a lot ?
I think the solution here was quite ok. A bit hard the first time maybe :))
>>
>> Ok. If you will accept it, I can create you a bitmap editor, as an
>> external tool in a month.
>
> In my opinion this is more than one month of work,
> but well... as you like. ;)
Great. I start today.
> [And this is not a "bitmap editor". It must look like
> an Icon / Cursor / ... Editor, with Grid, like the
> actual one... Plus Tools Box, and all...]
Ok. Save as ICON.
>
> :)
I promise you, unless someone shoot me, or I get lumonia, I have you a
icon/bitmapeditor within one month!
To not become the perfect idiot, I say two months. And not a day after !
But, I wount be here much answering post ! I will be vannished. Just say
hello.
> Betov.
>
> < http://rosasm.org/ >
>
>
>
Alan Anderson
> Chewy...@austarnet.com.au wrote:
> > [...] I know a few Star Trek fans tried to get Klingon and some of
> > the other scripts included, but IIRC UTC relegated those scripts to
the
> > private use area...
The proposal to encode Klingon wasn't done by "a few Star Trek fans".
It was done by a couple of *language* fans. (Klingon was already
somewhat standardized in the private use area before the proposal was
considered; its mapping predates the CSUR.)
> Actually Klingon was categorically rejected:
>
> http://www.unicode.org/alloc/rejected.html
>
> But Tengwar (the Lord of the Rings elvish script) is apparently still
> under investigation:
>
> http://www.unicode.org/alloc/investigation.html
>
> Don't ask me why one seems more legitimate than the other.
I don't need to ask, because I already know. The principal argument
against including an official Klingon mapping in Unicode was that the
people who actually use Klingon typically don't write with the "Klingon
alphabet", using instead an ASCII-compatible romanization.
Of course, at the time, the lack of a standard way to write in a
"Klingon alphabet" was a large reason nobody used it. Now that the
CSUR Klingon mapping is more widely supported (e.g. the Code2000 font),
lots of people do use it. I would hope that the relevant Unicode
committee would recognize that today if they were to reconsider the
matter.
Beth
> Hi Beth,
Hi wolfgang :)
> [..]
> | As for taking up more memory and disk space: UTF-8 does not take up a
> | single bit extra from ASCII for any ordinary ASCII characters...
>
> ??
> I may have missed the point of UTF-8, I'm not through the whole book yet,
> how can I display 'mathematical operators' 2200..22FF (page 193) with
UTF8
> ??
>
> At the moment I use ASCII 20..7F as they are and 01..1F and 80..FF
> as 'my personal character-set', while 00 is used to mark the end
> followed by one ore more 'format/function/more../'-bytes.
>
> ie:
> 40 41 42 43
> 00 0a xx yz nn nn nn mm ; print/edit a numeric variable
> ; (type,format,action,group,address)
> will print: "@ABC-33.35 x10[sup3]-6[/sup]"
>
> 00 ... ; all var-types, input, locate, colours, ... just all needs ;)
> 00 E0 xx.. ; conditional call substring incl. format header (menus...)
> 00 FE ; end of a sub-string (menu/buttons/..)
> 00 FF ; end of the print job
Right, UTF-8 is actually fairly easy to understand, so I might as well
describe it here...
I know you prefer concise tables, so let's see if I can make a simple table
for it:
U-00000000 to U-0000007F: 0xxxxxxx
U-00000080 to U-000007FF: 110xxxxx 10xxxxxx
U-00000800 to U-0000FFFF: 1110xxxx 10xxxxxx 10xxxxxx
U-00010000 to U-001FFFFF: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
*U-00200000 to U-03FFFFFF: 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
*U-04000000 to U-7FFFFFFF: 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
10xxxxxx
[ * Technically, it is intended that no UNICODE character will ever go
beyond FFFFFh (1024K characters)...so, though these encodings are defined
for UTF-8 (up to 2^31), you shouldn't ever actually see these in any UTF-8
file... ]
Examples:
So, for U-0041 (which is plain old ASCII "A" :), the byte is exactly the
same as it would be in ASCII:
01000001 (or 41h)
For U-03A0 (which is Greek letter "pi" :), the sequence would be:
11001110 10100000 (or CEh A0h)
For U-2200 (which is the mathematical "for all" character: That is, an
upside-down capital A :), the sequence would be:
11100010 10001000 10000000 (or E2h 88h 80h)
...so forth...I think you can work out the rest yourself...pick your
UNICODE character then check what "range" it's in with the table
above...place its binary digits into the spaces marked with the "x"...
Points to note:
1. 7-bit ASCII characters are encoded in exactly the same way in UTF-8
2. Byte-based (no "endianness" worries; Does not require any "BOM" ("byte
order mark") character at the start of the file or anything like that, as
Microsoft advise should be in their plain text file UTF-16
encoding...advice, in fact, that they were overstepping the mark to give
because UNICODE themselves do not specify this as anything but
"optional"...but, well, that's Microsoft for you, eh? ;)...
3. All non-ASCII characters use a multi-byte sequence
4. Each byte in that multi-byte sequence has the highest bit set (so, it's
clear what is ASCII and what is non-ASCII and they shouldn't become
confused)...
5. The first byte of the sequence has as many highest bits set as there are
bytes in the entire sequence (e.g. "110xxxxx 10xxxxxx" starts with two set
bits in the first byte, so the sequence is two bytes long :)...
6. After the first byte, all further bytes are of the form "10xxxxxx"
(providing another extra 6 bits of "address range" per byte :)...
7. The bytes 0xFE and 0xFF are never used in UTF-8 at all...
8. The first byte of a non-ASCII character is in the range C0h to FDh,
subsequent bytes in a multi-byte sequence are in the range 80h to BFh,
ASCII characters are in the range 00h to 7Fh...you can use this for easy
resynchronisation (if you start reading in the middle of a multi-byte
sequence, you can _know_ that this is the case by what range the byte is in
:)...
9. All UNICODE characters are available (ASCII bytes are still just one
byte long, all 16-bit "BMP" characters one to three bytes long, all defined
UNICODE characters four bytes...being "variable-length" then size of files
dependent on what characters encoded: If all ASCII, no different from
ordinary ASCII file...if all "upper range" Chinese ideographs, then UTF-8
encoding is 4 bytes long (though, note that UTF-16 - with 16-bit per
character, as Windows uses - is also 4 bytes long, so UTF-8, at worst, can
only be as big as UTF-16 but will typically probably be smaller))...
NOTE: You might notice that it is possible to create what are known as
"overlong forms" for characters...for example, you could unnecessarily take
two bytes to encode an ASCII character:
11000001 10000001 (or C1h 81h)
Instead of the simpler:
01000001 (or 41h :)
These "overlong forms" are _INVALID_ UTF-8...they should be rejected as
errors, if encountered...note that these "overlong forms", as the name
suggest, are _unnecessarily_ long, so rejecting these also ensures the
shortest possible encoding of the character, as well as "normalising"
everything so that "comparisons" are easy (i.e. there is only _one_ valid
way to encode any particular character :)...
In order to help detect "overlong forms" and reject them, here's another
simple table...if any of these bit patterns are detected, you have an
invalid "overlong form" (these are all invalid sequences in UTF-8):
1100000x (10xxxxxx)
11100000 100xxxxx (10xxxxxx)
11110000 1000xxxx (10xxxxxx 10xxxxxx)
11111000 10000xxx (10xxxxxx 10xxxxxx 10xxxxxx)
11111100 100000xx (10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx)
[ Note that you can work out if it's "overlong" from the first or second
bytes, irrespective of what the remaining bytes are (which I've placed in
brackets)... ]
As you can see, it's quite a nice "encoding" for UNICODE because it leaves
ASCII alone...and can access any defined UNICODE character in up to four
bytes (which is no worse in size than UTF-16 for the "worst case" (a file
of only "upper range" Chinese ideographs and nothing else...which is
probably not any file you or I would ever want to create, eh? ;) but will
generally be smaller in most cases...right down to, as noted, being exactly
the same size as ASCII, if the file is only 7-bit ASCII characters)...for
easy parsing, all non-ASCII multi-byte sequences have their highest bit set
for every byte in that sequence...and the first byte of that sequence is in
a different "range" to the subsequent bytes, with the number of "high bits"
set being equal to the number of bytes in the entire sequence...
> [..support..]
> | Yeah, this is actually a "reverse" point on what's actually true...the
> | whole purpose of UNICODE is that it _gets rid_ of all the things like
"code
> | page incompatibilities" once and for all...one standard character set
for
> | _everyone_...
>
> Even it will be a very huge pack of patterns for _everyony_ to carry,
> it may make sense to set up an international agreement of tool-makers
> to a limited standard use of UTF character-sets.
Well, yes, but this is confusing the purpose of UNICODE...it's a "standard"
for assigning _unique_ numbers to all of these characters...that's what
UNICODE is...just a "standard" for all the characters...
If you only want to support a limited set of characters then do so...if you
want to _STORE_ your strings in some "private format" and not one of the
UTF "encodings" then do so...
The point of UNICODE is just that when you need to send files and data
_internationally_, then the UNICODE standard ensures that there's _ONE_
standard for all characters...so that you never get the "code page"
problem, where a character you can see, can't be seen properly by someone
else because their "code page" is different to yours...if everyone sends
their files in UNICODE format then, you know, each character has a "unique"
character code which cannot be confused with any other character...
If you like, UNICODE is about _TRANSMITTING_ data internationally...a
"standard" that just makes completely sure that what people see on the
other end is _exactly_ what you sent to them, whatever "code page" someone
has set...
As for how many characters are in UNICODE? Well, it's Babylon that's to
blame for that, yes? They are just trying to put all known characters into
one standard...the fact that there are so many characters out there is to
do with the human race and it's "Babylon confusion"...
Indeed, it should be noted _HOW_ UNICODE came up with the characters:
The majority of UNICODE characters were introduced by _MERGING_ all the
other character set standards and "code pages" and such together...so, for
instance, many of the "dingbats" come from standard _PostScript_
fonts...the "box drawing characters", of course, come from IBM's character
set in the BIOS...and a large part of the UNICODE characters comes from
_MERGING_ previous standards together...
Hence, the amount of characters is a consequence of there being too many
"standards" out there...but note the reason _WHY_ UNICODE "merged" all
these standards: In order to provide what they call "round-trip
compatibility"...that UNICODE is a "superset" of all other standards...so
that if you converted from some other standard to UNICODE and then back to
that standard, there would be _ZERO_ data loss...no "translation errors"...
Indeed, much of the "problems" that exist in UNICODE relate to that...for
instance, there is a specific "capital A with umlaut" character..._AND_
it's possible to have "capital A" character with a "combining umlaut"
character after it (which puts the umlaut on top of the previous capital A
character)...why both forms? Because, for compatibility with other
standards (where the specific character "capital A with umlaut" existed),
they had to retain the specific character, as well as their new "combining"
characters (which are much more "versatile" because, you know, with a
separate "combining umlaut" character then you could place an umlaut over
_ANY_ character: A "snowman" character followed by a "combining umlaut"
becomes "snowman with umlaut" on the screen ;)...
It is, in fact, exactly because they were trying (too damn hard, if you ask
me) to be "backwards compatible" with all these other, older standards that
the "exceptions" here and there appear...
But, going back to the point about it being about "transmission", now you
might see why UNICODE went to the effort of all this "backwards
compatibility"...the _PURPOSE_ of UNICODE is a "standard" that "merges" all
these standards...so that if I have data in one "code page" and you have
data in a different "code page" and Rene has some "double byte character
set" oriental file he wants to send...well, if we all convert to UNICODE,
then _every single character in all the other standards_ is given a
_UNIQUE_ character code...so, UNICODE prevents any and all "conflicts"...it
_guarantees_ that the characters that appear when we all exchange files
with each other - no matter what "code page" or other "standard" we're
using - will be exactly the same characters seen that were sent...
That was the original and intended purpose of UNICODE...
But OSes and applications started deciding that they would just exclusively
use UNICODE everywhere...as, in fact, this is a far easier implementation
than trying to support each of the many "code pages" and "double byte
character set" stuff that's out there...
Indeed, if you think UNICODE is problematic, then try implementing "Double
byte character set" and a few thousand "code pages" for your OS: That'll be
far, far harder to implement and, for _MORE_ effort, you'd get less
functionality...because where UNICODE can show the oriental ideographs
_AND_ Hebrew _AND_ Cyrillic _AND_ Latin all at the same time, the "code
page" system does not allow that because only one "code page" can be active
at a time...
Also, there are some "special characters" that require "processing",
yes...but the vast majority of characters are just characters...you just
store them in strings...so, other than that the storage is bigger, what's
the difference if it's 95 characters, 950 characters or 9500 characters?
The only people who are going to have problems with this are the font
designers, drawing them all...but then, it is allowable to create fonts
that only covers certain "ranges"...and programmers can just use fonts from
the professional font designers...
Initially, granted, you might be "awe struck" with the amount of
characters...but, really, you don't have to become "personally acquainted"
with every Thai and Tamil character or anything like that...note: There's
many scripts and languages supported from all over the world...the person
who holds the world record for "most languages known" only knew 58
languages...NO-ONE comprehends it all...you're not expected to start
learning how to speak "Runic" (not least because it's a "historic language"
and NO-ONE out there speaks it anymore)...just store the characters in the
strings...take note of the "special characters" (and all the first chapters
of the UNICODE book explain all the "processing rules" for that)...but,
otherwise, just store the characters in the strings...if you're only
interested in reading ASCII then skip all the non-ASCII characters...
Note: I tested out that "Japanese Hello World" program as a UTF-8 file -
the strings and comments in the file were in Japanese...but, obviously,
following NASM's "rules" for identifiers and using the Intel mnemonics,
everything else was standard ASCII - and sent it through NASM...
NASM is _NOT_ designed to process UTF-8 files but it _WORKED_,
anyway...this is because NASM just looks for the ASCII characters in the
parsing...and the non-ASCII characters just "passed through" (for the
strings) or were "ignored" (for the comments)...
So, you see, to an extent, NASM _ALREADY_ supports UTF-8 files...and this
was completely _BY ACCIDENT_...hence, the "worry" that it's complicated to
implement? NASM implemented it without even trying!!! It can handle UTF-8
files and it wasn't even supposed to do that!!! So easy to implement, NASM
has done so _COMPLETELY BY ACCIDENT_...
> | But this is the biggest problem, true...but it's a problem of "my
software
> | is not up-to-date with UNICODE"...yes, but isn't the point Chewy's
making
> | about: "let's all update things to use UNICODE from now on"?
>
> At least, it's worth to think about it.
Well, the basic idea is simple: _IF_ everyone did move to UNICODE, then
this should "solve" all these problems, once and for all (mind you,
whenever people say that, it's normally "famous last words"...yeah, and the
Titanic was "the unsinkable ship" too...but, you know, this is the "theory"
behind it and even if there are problems, what's there already tends to be
better than trying to support a few thousand "code pages" and "double byte
character set" things and all the other stuff ;)...
Indeed, "one standard with too many characters" might be a problem...but
consider "too many standards with not enough characters" instead: This is a
worse fate ;)...
Part of the "problem" here is that, basically, what a lot of programmers
like us have been doing for many years is to write ASCII programs with
English (or French or German, which are also Latin-based so there's no big
difference, except for a few "accents" and that funny "double s" character,
which is officially being "phased out" of German, anyway, I notice with
those "spelling changes" introduced on July 1996)...and we haven't given
the slightest thought or done anything to do with "Internationalisation" at
all, really...BUT, for those that have been doing this, then it was a _LOT_
of work to support it all...and that's where UNICODE comes in...it was all
started by a bunch of people who'd been working on things like
"international word-processor" software and such, to try to, once and for
all, get rid of all the "Babylon confusions" of "code pages" and
such...create one "big" character set which includes all of it...there,
problem solved...and, okay, it's a little bit more complex than that (well,
you know, all those languages people speak around the world just don't all
work alike :) but this is much easier than the other "international"
options...so, you know, if UNICODE looks "bad" then just imagine what it
was like supporting everything else beforehand...
And, yeah, there's a lot there...and ASCII is, indeed, much simpler...but,
well, "you get what you pay for"...ASCII is simpler but, well, you "lock
out" people that way...ever noticed that we don't seem to have any Japanese
coders posting here very often, or Israeli coders and others who use
different alphabets...okay, there's Maxim and Alexei - Russian - on the
alt.os.development group but, of course, they've _TAUGHT ENGLISH_ as
"standard" in Russian schools for a long time (and are very good at it too
;)...but Maxim was telling me about how he actually has a Cyrillic keyboard
but has a "shortcut key" which switches back and forth from Cyrillic to
Latin (and he just "knows" where the Latin characters are, from
practice...mind you, I never look at the keyboard while typing either, so
you could mix up the keys and I'd still be able to type...indeed, on my
Linux machine, I popped out the keys because I thought I'd try that
"Dvorak" layout and pushed the keys back in the different layout...but
then, once I installed Linux, I wanted QWERTY back...so popped out all the
keys and pushed them back in the QWERTY way...except, oops, I popped "M"
and "N" back into the wrong places and they've swapped around...nevermind,
it makes no difference to me, really, as I never look at the keys while
typing...but, oh dear, it's confusing for everyone else! ;)...hence, yeah,
for _US_ ASCII is much simpler...but it's probably absolutely NO
COINCIDENCE whatsoever that those who generally show up on this newsgroup
and such, all have Latin-based alphabets or are taught compulsory English
in schools and so forth...
> ['Babylonian confusion']
> | _THIS_ is the real problem, of course...
>
> | BUT is this a problem that's the tool's responsibility? For instance,
> | programming a protected mode OS is very confusing too...or using
"direct
> | access" to hardware...should tools also start telling us we can't use
"LGDT
> | / LIDT" or "IN / OUT" instructions too? As they might cause "Babylonian
> | confusion"?
>
> My new disassembler will show PL0/IOPL/PM/64-only instructions
> as "force EXC0D/06/00.." if the mode configuration wont match.
>
> But this aren't language issues, except Intel-manuals are written
> 'that' confusing, it may even native English speakers drive crazy. :)
Oh, don't worry...when it comes to typical "technical English" and what's
written in many manuals with lots of jargon and bad descriptions, then,
yes, it might as well be a different language...and there's not much
advantage to being a native English speaker at all ;)...
But, yeah, back to the main point: This is what I was trying to say...a
tool _shouldn't_ really be "dictating" these kinds of things...it should
provide the "facilities" and then the programmer decides how to use those
facilities properly (or not to use them at all)...
You could say that non-ASCII characters are "confusing"...but many would
say protected mode is "confusing"...so, if a tool can say "we shall have no
more non-ASCII characters!", then is this also saying that the tool should
say "we shall have no more access to protected mode!"...
This is "nanny" mentality..."tool knows best"...if _I_ say it's too
"confusing" for you then you can't use it...this just isn't right...
I mean, let's say that I think "direct hardware access" is "too
confusing"...so, I come along and "for your own good", I remove all the "IN
/ OUT" instructions from your assembler...what would you feel about that?
That I'm "doing things for your own good"...or that I have "no right" to be
doing that and I'm a patronising cow, who should "piss off" and leave you
to do what you want to do?
This is the "principle" here...this just isn't what a tool is supposed to
do...it might not support something because, you know, it's supposed to be
"an applications assembler" (so, only 32-bit flat mode and no privileged
instructions is a consequence of it being an "applications assembler" -
fair enough - not because anyone is thinking "this is too confusing for
people, so let's remove to make things easy")...but to "ban" or "limit"
something because _you_ feel it's "too confusing"? To be "nanny" for
people, who, in fact, you've never met and don't know what they do or don't
want?
You know, the greatest wisdom is often to know when to _SHUT UP_...the
greatest power is to know when _NOT TO ACT_...often, the very best thing a
government can do is just to piss off and leave the people to do whatever
they want...
Well, perhaps one word makes the point clear: "Clippy"...
Yeah, "Clippy" was meant to make things "easier" by being your "personal
nanny" as you work...he was meant to be "helpful"...now, that's what
"Clippy" was supposed to be...but what was he _ACTUALLY_, in practice, not
just for "expert" but also "newbie" users? One of the most annoying
creatures ever created on the face of planet Earth...universally loathed by
literally _millions_ of people...
And, yes, wasn't "Microsoft Bob" a great idea?
Starting to see the "pattern" here yet? If you do, well done, because
Microsoft still ain't worked it out yet...
Sometimes, the most "helpful" thing a person can do for someone else is to,
in fact, do absolutely nothing...leave them be...you know, _wait_ until
you're asked to lend a hand, without rushing in, changing everything and,
well, not really knowing what you're doing...
For the Brits reading, it's the "Brittas Empire" syndrome...for everyone
else, that was a comedy show - starring Chris Barrie (Lara Croft's Butler
in the movie and the "smeghead" hologram in Red Dwarf :) - about this
manager called "Brittas"...and, basically, the joke of the show was that
this manager was utterly, utterly imcompetent...lethally so...everything
that he "managed" would crumble, collapse, explode, get killed or otherwise
fall apart completely...but - worryingly, very much like an awful lot of
"managers" out there - he was convinced that he was the smartest, cleverest
and best manager in the whole world...one of those people who thinks they
know best and interferes in _everything_...but the second they come along
and interfere, they don't help at all...they actually make things a million
times worse...
Well, sometimes when I think of Bill Gates and Microsoft, then this
"Brittas" character comes to mind...convinced they "know best"...convinced
everyone else is an "idiot", wo doesn't even know how to tie their shoe
laces...and then they come in to try to "help"...and make things 7 million
times worse...because, in fact, the only "idiots" are them but they are too
idiotic to even realise this...
> | Or should the tools _support_ these things ...
> | But why should a tool be forcing a Russian programmer, ...
>
> Wouldn't it help a lot if the whole globe talks only one language?
No, not in the slightest...I find that a terrible, horrible, awful and
repugnent idea...all that history, all that diversity, all those _DIFFERENT
WAYS TO LOOK AT THINGS_ lost, everyone in the world forced into a single
"NewSpeak" language? It would be a terrible oppression...spiritual
bankruptcy...we should NEVER give in to such "trans-culturalism" that tries
to "expunge" peoples, cultures, histories from existence, simply because it
makes "mass-production" a little easier and profits larger...no, no,
no...never...it's an awful idea...
And it would be an impossible idea too...remember that many languages are
"distortions" of the same language as it became "separated" by hills,
mountains, rivers, national borders and so forth..."dialects", "accents",
"slang" and such...you know, that English _IS_ Germanic but it "split
off"...and then along came French and Norse and this and that...and it
"evolved" from there...the "language" that just naturally came from people
speaking all those different languages trying to speak to each other (very
much part of the reason, for sure, that English is more "simplified",
having lost all its "word endings" and things like that...because it was
simply impossible to "resolve" German word endings with French word endings
and so on and so forth...so, simply, they just got "dropped" because no-one
could "agree" on which way it should be done...also, probably why English
is a pretty good "Lingua Franca" in that it's not as difficult as many
languages are to be learnt...because it literally did start as a "common
tongue" between peoples who all spoke different languages...a "common
ground"...you know, where Esparanto was "invented", English is its natural
counterpart...of course, it's where it is because of the British Empire and
America's "superpower" status and so forth...BUT, at the same time, the
language itself _IS_ actually "appropriate" in that it's "simpler" than
most languages...indeed, onr of the most difficult parts of English is the
pronouncation and spelling have no relation to each other...but, in fact,
that _ALSO_ comes from English being the product of a "clash of
languages"...and that, for a long time, it was a "spoken only"
language...and when written, there were no "standards" for a long
while...it's an infamous point that Shakespeare, in fact, spelt his own
name some 20 different ways in different places! :)...
What might be good is if the whole world had a shared _second language_,
perhaps...and through this language people could always communicate...but
the willing destruction of languages and culture and history? Absolutely
NOT...
When Tolkien created his languages, he wrote the stories as a "background"
to the people who spoke those languages...because he believed that
languages and the people who spoke it and their culture were instrinsically
linked...bound together...and he was exactly _RIGHT_...
For example, here in Wales, there's the Welsh language which is very
ancient and one of the oldest European languages...the English, earlier in
history, attempted to delibrately _exterminate_ it...punishing children
caught speaking it...burning any books with the language in it...an actual
100% _delibrate_ campaign of language extermination with the goal of its
utter _extinction_...and they came very, very, very close to
succeeding...but, in secret, parents would teach their children the
"forbidden" language...books would be hidden and concealed...people would
speak the language in secret...today, from near extinction, around 25% of
the people of Wales are able to speak Welsh...
Which is good because the only actual historical account of the real person
who was "King Arthur" is, of course, written in Welsh...the original Merlin
legend are written in Welsh too...there are the myths and legends of "the
dragons" who fought (and the red dragon is what sits on the Welsh flag from
this tale :)...the bardic triads...centuries of history and culture...
And if the English had succeeded in their quest to kill Welsh? It would all
be lost...no-one would be able to understand it or read it...an entire
culture and, in effect, people lost completely...the English were not the
only ones to realise that destroying a language destroys a people, as the
Roman Empire before them also came to Wales and headed straight for the
Druids - the spiritual leaders and keepers of knowledge in Celtic Britain -
and they slaughtered them...every last one...
What is known of the Druids and their ways? In fact, very little...next to
nothing...all that exists to document them are what the Romans
wrote...about their dress...where they lived...that they were spiritual
leaders and the "wise men" who retain all the diverse knowledge of the
their people (knowledge was passed orally through stories)...indeed, if you
look at the "jedi knights" or the shape of Merlin then they are clearly
partly "Druidic" in influence...certainly in the simple robes they
wear...part druid, part monk, part knight...
Who knows what stories they had to tell? They were known to have herbal
remedies and recipes for medicines...all lost...completely gone...really,
next to nothing remains but second-hand Roman accounts of them...everything
else the Romans _obliterated_ delibrately...they would, you see, have no
"rivals" to their power...no "split loyalties" from the British: They would
worship the Glory of Rome and the Emperor...so, the Druids - as local
leaders to whom the people may have "loyalty" to - were slaughtered...every
last one...every last bit of their knowledge...all of their history...
For example, who built Stonehenge (which is actually older than the
Pyramids at Giza, in fact)? What was it for? Who made the large chalk
figures on hills nearby? What was before the Romans?
Thanks to their quite delibrate massacre - which was exactly for this one
aim - no-one actually knows...almost everything about pre-Roman Britain is
from archeology alone...if the Druids kept any accounts of this history,
they must have been destroyed by the Romans...all the stories that would
have been "passed on" in myths and legends and tales of history: _DEAD_...
Some people in modern times pretend to be "druids" and that they are
enacting "druid rituals"...it's all made up in modern times...a bit like
the modern "wicca" movement, who claim to be "carrying on anicent pagan
traditions"...completely nonsense...it is based on no historical evidence
or accounts for these practices...exactly because there are _NO_ evidence
or accounts for either of these...or what they really might have
done...modern inventions with no basis in any historical reality...
And, note, there was no accident in this...the Romans were utterly
delibrate in this goal...what is the saying? "The victors write the history
books"...
To willingly surrender the past? The culture? The people? To hand over your
Forefathers...to what? Making "mass-production" easier with
"trans-culturalism" because it's easier to make a bigger profit when you
can just solely print English boxes and English manual "in bulk"? Sorry,
but do you really believe that the spread of English through commercialism
is entirely "accidental"?
"Those who do not know history are doomed to repeat it"
> A long way to go, but when I look back 50 years, the world seems
> to tend to everyone learn to understand written English at least,
> local US/UK/AUS pronouncation excluded here of course :)
Hail Ceasar! To the Glory of Rome!
Et tu, Brute?
> [..]
> | ... if we don't agree on that, then things become "Babylonian
> | confusion" here very, very quickly, nicht wahr? ;)...
>
> That's very true indeed*, isn't it? *)indead for the French :)
Oh, no...I salute the French that, every year, they enter a _French
language_ song in the Eurovision...they did, one year, break this and put
in an English language song...I was _VERY_ disappointed in that...don't do
that ever again, France! It is one of the most beautiful sounding languages
on the planet...everything spoken in it sounds like pure poetry...all that
history and culture of centuries can be heard in each word...
Let us not forget that Liberty is a French girl...let us NEVER forget
that...NEVER...
Mind you, the best Eurovison entry that never won was the bald surrealist
Guildo Horn...everyone says that the Germans have no sense of humour...and
then they enter a completely insane man, who runs around the stage and
climbs up the wall, ringing cow bells...it was hilarious...I voted for
him...but, no, everyone else couldn't understand that the Germans _do_ have
a sense of humour...instead, everyone said it was "terrible that the
Germans would insult Eurovision"...oh, come on...it's Eurovision...the
campness and crapness is the whole point...that's what so great about it...
Mind you, the British were also told off by other countries taking it all
far too seriously...because our commentator, Terry Wogan (who's actually
Irish but is a TV presenter in Britain that he does the commentary for us,
not Ireland...and he's actually done it every year for some 20 years or
something, so, to the Brits, is as much a part of "Eurovision" as anything
else :), has this "cheeky" way about him...he pokes fun at this and makes a
joke of that...very sarcastic in his comments...it's his style and it can
very funny...but, truth is, he really does like the Eurovision and thinks
it's great (as noted, he's been doing a _very long_ time, every single
year, and if he really didn't like it, he'd have given up the job long ago
:)...but, you know, he just doesn't see it all as "serious" at all...you
know, if the Croatian entry walks out wearing a really silly-looking
costume, he pokes some fun at it...but, a year or two ago, he was "caught"
by some other commentators from other countries he was sitting next to,
making his sarcastic comments and having some fun...and they took it all
very, very seriously and "reported" him to the Eurovision "authorities" or
something, asking that he be "banned" or something...
Although, thinking about it, I've just remembered what our entry is this
year...oh dear...oh dear, oh dear...it's page 3 model "Jordan", who seems
to want to show that she has "talent" at singing and isn't just a page 3
"bimbo" (of course, she isn't actually talented at all and _is_ just a page
3 "bimbo"...but, nevermind, as long as she's happy, let's not "burst the
bubble" for her, eh? ;)...and, well, they put the songs to a "phone vote"
to decide who will represent Britain...and she won (I suspect because a lot
of men phoned up in order to be sure to see her again on Eurovision ;)...so
that's going to be interesting on Eurovision night, when all the other
countries are confused as to why Britain seems to have entered a "pole
dancer"-like act into the Eurovision...
See? Think about it...if everyone spoke English then we wouldn't have those
hilarious things on Eurovision, where they cut over to some
over-enthusiastic Swedish presenters who make bad jokes that aren't
funny...before getting confused about who's supposed to be talking and then
talk over each other...and then they cut over to Moscow, where the
presenter is just sitting there, unaware that the camera has come on and
everyone can see them...and then the Swedish presenters are saying "hello?
Hello Moscow! Hello?"...and it all falls apart in a very "cheap and
cheerful" not-very-professional way that's just "Eurovision" through and
through...
Now, see..._THAT_ is "culture"...well, kind of...by the way, has anyone
worked out how Turkey and Israel are in the _EURO_vision? Since when was
Israel European? They really, really "bend the rules" for some of the
countries entering, eh? At least Turkey is technically half-on the European
plate and is trying to join the EU...but Israel? We might as well throw the
doors open and have Canada and America and Japan and everyone joining too!
And, think about it, if everyone spoke English all the time, then we
wouldn't have all that "dix points" and "nul points" stuff...that's, in a
funny way, part of the British vocabulary now...if you want to say
something's crap then it's "nul points" and it's "dix points", if it's
great (even though, since all the extra countries have been added to the
Eurovision, it's now gone up to 12 points, as the top marks)...
Oh, one last thing: Could countries actually vote for the _music_ rather
than the "block voting" for each other's neighbours in a "political" way?
Oh, look, all the Eastern European countries are voting for each other!
Although, that said, when Britain got "nul points" - absolutely no-one
voting for us at all (and the song wasn't that bad...and we normally get at
least one or two "political" votes, even if we're utter crap ;) - last
time, while Blair's following Bush into Iraq...yeah, don't worry...we "got"
that one and let's say we don't necessarily "disapprove" of that kind of
"political comment" through the voting...goodness knows what reaction
"Jordan" is going to have, if Bucks Fizz pulling off their skirts got such
a big reaction before...oh dear...oh dear, oh dear ;)...
I reckon, though, that Ireland really should enter "My Lovely Horse", as
their "Euro song"...ah, I Love the Eurovision...it's brilliant! ;)
Beth :)
--
"We're still being challenged in Iraq and the reason why is a free Iraq
will be a major defeat in the cause of freedom."
[ George Walker Bush Jnr., explaining his Iraq war policy ]
Beth
> Beth wrote:
> > In fact, my little "test" which demonstrates that NASM _already_
> > deals with UTF-8 comments and strings, proves the point a
> > different way...some tools _already_ are "accidentally"
> > supporting UTF-8 to that "basic level", without even
> > knowing it...and the fact that no-one has actually realised
> > this (I didn't either until I thought it would be interesting
> > to see what NASM would actually do ;), shows how great the
> > "demand" is...if people were regularly wanting UTF-8 source
> > files passed through NASM, then your post should have had Frank
> > shouting "NASM already does it!!"...but, I bet, not even the
> > NASM developers have actually realised that it does already
> > work to this "basic level"...indeed, they could cheekily add it
> > to the "features list": "NASM has basic UTF-8 support!"...as if
> > they actually "intended" it or something...shhh! Don't tell
> > anyone! ;)...
>
> Not knowing much about UTF-8 (my Unicode knowledge extends as far as
> UTF-16 and that's about it), I would say that HLA v2.0 would handle
> literal strings of this form as long as the character code for quote
> can never appear in a MBCS (multibyte character sequence).
It cannot appear in the multi-byte sequence...it's delibrately part of the
design that every single one of the multi-byte bytes has the high bit set,
so it can't be confused with any ASCII character (the "ASCII compatibility"
works both ways, so to speak ;)...
I put more of the exact details in my post to wolfgang just now...but,
basically, the design of UTF-8 is exactly designed for the "ASCII
compatibility", including that, so long as a program doesn't care about
bytes with the high bit set, then they may very well work without
modification...
It's very possible that many of these assemblers will work like NASM does
and even though they see some multi-byte sequence, this doesn't effect the
processing and it actually works (because "xterm", in this case, directly
understands the UTF-8 multi-bytes, so when NASM simply "passes it through"
to the xterm console then xterm interprets it as the UTF-8 :)...
Of course, what's actually happening is that...say I use Notepad to write
the source code, then XP Notepad directly understands UNICODE and you can
select the "save as UTF-8" option...then, when passed to NASM, what NASM
sees is _more than one character_ in the >127 range...but, as its a string,
NASM just passes it through to the output file "as is"...then when "xterm"
receives that string, it natively understands UTF-8 and displays the
correct character...
So, amusingly, even though NASM has no "special processing" for UTF-8 at
all, because it just passes the >127 bytes through "as is" for strings (and
just ignores any bytes between ";" and newline in comments :)...then I
could actually create that Japanese "hello, world!" program in Notepad,
being able to see only the Japanese characters...then, when I run it with
xterm, the Japanese strings appears (actually, the two ideographs for
"world" were missing and the "font missing" character appeared...obviously,
X lacks the fonts for that range of characters but it does have
"Hiragana" - the Japanese "alphabet" (well, one of them, they have more
than one: Talk about confusing, eh? ;) - but the very fact that _TWO_
"missing character" blocks appeared rather than more than two (because the
"multi-byte sequence" is more than two bytes, so though it doesn't have the
"glyph" to show the character in the font, it _IS_ interpreting it
correctly :) and that the "Hiragana" characters of the "konnichiwa" were
showing just fine...xterm is "interpreting" it correctly...I just need to
install the right fonts, it seems, if I want to do more Japanese programs
;)...
In simplest form - a basic "table" - UTF-8 works like this:
U-000000 to U-00007F: 0xxxxxxx
U-000080 to U-0007FF: 110xxxxx 10xxxxxx
U-000800 to U-00FFFF: 1110xxxx 10xxxxxx 10xxxxxx
U-010000 to U-1FFFFF: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
...where that's the binary of the bytes on the right, obviously...and the
"x"s are replaced with the binary of the character code you want...and,
other that "overlong forms" (trying to encode an ASCII character with one
of the longest multi-byte sequences) is to be rejected as "invalid" (in
order to ensure that the UTF-8 is "normalised" for comparisons and such :),
that's about it...
You'll notice that only ASCII bytes have the highest bit clear...and then
the first byte of the multi-byte sequence is different depending on the
number of bytes in the sequence (indeed, the number of high bits set in the
first byte is the amount of bytes in the sequence, you'll note :)...then
the rest of them start with "10"...values FEh and FFh are, in fact, not
possible and invalid...
Also, being byte-based throughout, there are no "endianness" or "byte order
mark" (BOM) stuff to worry about...plus, UTF-8 can address all UNICODE
characters (indeed, I've cut that down, you can validly extend the
multi-byte sequences to cover all 2^31 "UCS" values (up to a six byte
multi-byte sequence :)...but by the UNICODE design, there's no characters
that high up and they "intend" that there never should be...mind you, they
"intended" not to overflow 16-bits but did anyway...I suppose, though, if
we could _trust_ them not to "overflow" the 1024K characters they've
"reserved" then all the ones above it could be used for "private
use"...say, for "rich text formatting" or something like that...I don't
know...a possibility...but NOT a good idea, if we can't actually trust them
not to "overflow" 1024K or it'll then become "incompatible"...perhaps best,
unlike Microsoft jumping in early with the 16-bit encoding, not to do any
such thing until "mature" enough to be sure that any such characters are
really "free for use" ;)...
It's pretty simple and was delibrately designed, as you can tell, to be
"ASCII compatible" (including that no "multi-byte sequence" can be confused
with any ASCII character)...and the format of the "multi-byte sequence" is
also designed for easy "resynch" because the "first bytes" are all marked
as such (while ASCII characters are just ASCII characters, so those are
obvious too :)...so, even if you started half-way through a multi-byte
sequence, you could be "back in synch" by the next character (or could
"back up" to find this character's "first byte")...plus, the "nice" feature
that you can _TELL_ you're in the middle of "multi-byte sequence" too...
Ken Thompson obviously put on his thinking cap for this one because it's
quite a "nice" encoding for UNICODE...still relatively simple but has all
those "useful" properties: byte-based (no "endianness"), ASCII compatible,
addresses full range, easy "resynch", can immediately tell if you're
"mid-way" through a multi-byte character (by making the first bytes
different and actually can be easily tested as they are in a different
"range" to the other bytes :)...and, at least for all the "currently
defined and reserved" UNICODE characters, it's only four bytes long maximum
(which, in fact, compares directly to UTF-16 in the same situation, yes?
BUT UTF-8 is typically "smaller" for other cases :)...
The drawback, of course, is that it's "multi-byte", so that's not as simple
as UTF-16 to process (but, then again, for the >64K characters, UTF-16 can
go "multi-byte" too, due to that whole "the first standard was far too
inadequate thinking 16-bits was enough (when the Han ideographs are
probably 64K _by themselves_ without anything else)"...did anyone actually
bother to find out how many ideographs exist before that first draft?
Obviously not...the "variable length" not being quite as convenient to
process as fixed sized characters...
> HLA v1.x,
> however, would not be happy with the character as Flex rejects all
> character codes in the range $80..$FF out of hand.
Ah, well...yes, that won't be UTF-8 compatible then...
> There is, however, another issue that gets you into trouble with MBCS.
> When you start adding sophisticated compile-time language facilities,
> such as string functions, handling all the different character sets
> becomes a nightmare.
Yes, indeed...things like "ToUpper" suddenly become a little more
"non-trivial" than flipping a bit when the value is in a certain
"lowercase" range...
I've not exactly looked completely at what characters are "uppercase" and
"lowercase" and so forth (indeed, only a few scripts have "case": Latin,
Cyrillic and one other I can't remember off the top of my head ;)...but I
think it'll probably have to be a case of a "look-up table" for this kind
of thing...
UNICODE provide "data tables" for all the defined characters where various
"properties" are defined for each character, like its "case" and such
(comes on a CD, if you actually buy the UNICODE book itself :)...and then
there are "rules" explained in the first few chapters about how to process
them...so, you know, you shouldn't need to sit down and become some
"expert" in 50 languages or anything...
But, yeah, the "processing" becomes more complicated...mind you, Windows
already has things like the "CharUpperW" API, which uppercases a UNICODE
character correctly...indeed, all of Windows' string API in their UNICODE
versions are, of course, "UNICODE aware" and should correctly process
them...
BUT, of course, this partly negates one of the usual "advantages" of
assembly language coding that you can deal with the strings yourself in
"optimised" ways...calling an API to "uppercase" each letter? Well, not
"ideal", is it?
On Linux, there aren't any "string API" in the kernel but Linux is supposed
to use UTF-8 "natively"...in this case, then there are C standard libraries
with "wide character" string routines that are "UNICODE aware"...but, well,
again, that's "defeating the purpose" a little with using assembly
language, if you've now got to link in C libraries for this...
Mind you, processing strings in assembly language manually shouldn't
involve anything "OS specific", so perhaps some out there wants to write a
"UNICODE string library"?
> Then, in HLA's case, there is also the issue that
> you need to provide standard library routine equivalents of string
> functions for UTF-8 strings (you think zero-terminated strings are
> painful to compute the length of? Try UTF-8!).
Yeah, and being "multi-byte", you then have to distinguish between "number
of bytes" and "number of characters" because the relationship isn't simple
anymore either...this is the main "drawback" of UTF-8...
Plus, if HLA added UTF-8, then should it also add UTF-16 support and
UTF-32? Routines to "convert" between them?
Oh, yeah...I fully appreciate the problems...
With LuxAsm, though, we'll try to "think ahead" on this (so, even if not
implemented straight away, we should Hopefully not commit to anything that
screw up adding it in later on...so, for instance, remembering to
distinguish "number of bytes" and "number of characters" in any string
routines for the LuxAsm "standard library" or whatever ;)...and there's a
"nice" thing that Linux's "native" format is UTF-8 and UTF-16 and such
aren't really seen, except for "compatibility" with things sent over from
Windows and such...so, being "Linux specific" then it would not be
"unreasonable" for us to only bother with UTF-8 (which, as noted, is the
most "handy" with "ASCII compatibility" for the source files and such :)...
Ah, for sure, there's nothing particular "easy" in this...but, well,
supporting all those languages from around the world simultaneously? It's
never going to be "easy", is it? But, you know, "you reap what you sow",
isn't it? "No such thing as a free lunch" and so forth...it's more
complicated but that's because you're dealing with a bigger "kettle of
fish" here...and, truth is, if dealing with all the previous stuff to the
same kind of "support" level with thousands of "code pages" and "double
byte character set" and so forth...well, that would be _MORE WORK_ (kind of
by definition: This was the "problem" UNICODE was originally invented to
"solve" :)...
No, it's not going to require "more work" to support this...but, well, if
UNICODE picks up then, at some point, I'm going to have learn all about
it...so, dive in head first and look at it now...I did something similar
with Windows before...jumped into Windows (16-bit) coding when Windows was
only just picking up "serious" use generally (mind you, only C
coding...like others, I was "given the impression" by Microsoft and such
that ASM was "impossible"...also, I'm not any "anti-C" coder or
anything...so, I'm comfortable with that too and, to be honest, kind of
"didn't think" of assembly until later :)...and I suppose perhaps I'm sort
of also doing with Linux and X coding too, thinking about it...ah, it's the
way I am: I fear "standing still" far more than "fearing change" :)...
> An assembler like NASM, that doesn't provide much in the way of
> compile-time string handling, might actually get away with "accidental"
> UTF-8 support.
Ah, mind you...I've not tried any NASM "macros" on this...indeed, because
of the difference between "number of bytes" and "number of characters" in
UTF-8 then any kind of "macro loop" going through character-by-character?
Doomed to failure...well, it is "accidental" support, so to speak...hence,
"limits" to how far it extends are only to be expected...this, in a sense,
is really where LuxAsm will differ because UTF-8 in strings and comments is
about the "level" we've decided to aim for...but the support isn't
"accidental" and it realises its reading UTF-8...
As for the "string routines" in the LuxAsm "standard library"? Well, I've
got the CD with the UNICODE "data tables"...I suppose, when the time comes,
I'll just have to give it a go at making that "look-up table" routine...
Ooh, good point...LuxAsm team! Won't our "standard library" have to have
LGPL rather than GPL, if it's to be allowed that LuxAsm "standard library"
routines can also be used in non-GPL programs too...oh, boy...how many
different "licences" are we juggling already? MIT licence on X, GPL on
LuxAsm, LGPL on any "standard library"...
> But when you've got a sophisticated macro system and
> compile-time language, supporting MBCS turns out to be a *lot* of work.
Yeah...but we will be having a sophisticated macro system and compile-time
language, in our "plan"...so, well, looks like there's just got to be a
"lot of work" ahead of us...
Beth :)
--
"Let me put it to you bluntly: in a changing world, we want more people to
have control over your own life."
[ George Walker Bush Jnr., explaining how "globalisation" works ]
wolfgang kern
Many Thanks Beth,
[about UTF-8] this was the information I missed.
| I know you prefer concise tables, so let's see if I can make a simple table
| for it:
| U-00000000 to U-0000007F: 0xxxxxxx
** U-00000080 to U-000000BF: 10xxxxxx also just a single byte?
Ok No, as mentioned under PT.3 below
| U-00000080 to U-000007FF: 110xxxxx 10xxxxxx
| U-00000800 to U-0000FFFF: 1110xxxx 10xxxxxx 10xxxxxx
let me check on the latter:
xxxx'xxxxxx'xxxxxx range = 2^16 00800..107ff ??
I assume the 'overflown' aren't used then,
or may indicate an error condition as mentioned with overlong.
| U-00010000 to U-001FFFFF: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
| *U-00200000 to U-03FFFFFF: 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
| *U-04000000 to U-7FFFFFFF: 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
| 10xxxxxx
|
| [ * Technically, it is intended that no UNICODE character will ever go
| beyond FFFFFh (1024K characters)...so, though these encodings are defined
| for UTF-8 (up to 2^31), you shouldn't ever actually see these in any UTF-8
| file... ]
I see, the format is somehow logical organised and easy to understand,
[just count set MSB's to get involved bytes,
and valid start value of a group can be calculated by a few lines]
the (25% waste) synchron-bits in every byte may make sense for serial
transmission, .
Yes, got it. Implementation wont interfiere with my text-routines
as UTF-8 has no 00's in the extensions.
All I'd need to go for it is one additional conversion table
('my > 7f' -> UTF-8) and to insert a single instruction into my
text interpreter:
textout:
... ;getchar
OR al,al
jz NextFunction ;end of text or whatsoever
Js UNIcode ;*** and there he story continues :)
Call copychar ;display/print/file/buffer/..
inc ecx ;string cursur index
jmp txtout
[..the why..]
Yes, my long time schedule see a KESYS-browser since a while ;)
[..]
| Also, there are some "special characters" that require "processing",
| yes...but the vast majority of characters are just characters...you just
| store them in strings...so, other than that the storage is bigger, what's
| the difference if it's 95 characters, 950 characters or 9500 characters?
| The only people who are going to have problems with this are the font
| designers, drawing them all...but then, it is allowable to create fonts
| that only covers certain "ranges"...and programmers can just use fonts from
| the professional font designers...
You know, double size of all text is a very big increase,
especially for one like me who fight every wasted bit. ;)
| Indeed, "one standard with too many characters" might be a problem...but
| consider "too many standards with not enough characters" instead:
| This is a worse fate ;)...
Agreed, I can think of a limited to 'basic needs' standard font support,
like the few pages I printed out from my UNIcode4.0 file.
['Babylonian confusion'..]
| > Wouldn't it help a lot if the whole globe talks only one language?
| No, not in the slightest...I find that a terrible, horrible,...
| no...never...it's an awful idea...
:) Of course, local dialects and mother-tongue will remain alive anyway.
| What might be good is if the whole world had a shared _second language_,
| perhaps...and through this language people could always communicate...
| but the willing destruction of languages and culture and history?
| Absolutely NOT...
Everyone one this globe should know English. That's what I meant.
[side stories...] interesting.
[Druids...]
there are still more around than you may think ...
Understanding nature in a communicative way,
using roots and herbs instead of pharmaceutical products,
'control kids and pets' by charisma rather than by force,..
| > That's very true indeed*, isn't it? *)indead for the French :)
| Oh, no...I salute the French ...
This joke was just about how Rene usually type/spell 'indeed'.
| Let us not forget that Liberty is a French girl...
I've been there, a really b i g female :)
[Euro-'vision']
| Oh, one last thing: Could countries actually vote for the _music_ rather
| than the "block voting" for each other's neighbours in a "political" way?
| Oh, look, all the Eastern European countries are voting for each other!
What would you expect, these voters are humans.
| ... I Love the Eurovision...it's brilliant! ;)
It make me switch channels (if you've seen one you saw them all).
You may have seen the Austrian participant last year, we sent a Clown...
__
wolfgang