2009-09-09  Rendered into RfD form, added Forth200x words
1999-06-22  Original Text by John Rible

Problem
=======
A large number of words use "c-add u" to indicate the address of a
string (c-addr) and its length (u) on the stack.  With the
introduction of variable width characters, it is not clear if "u" is
referring to the number of characters or address units.

Solution
========
Introduce a new pseudo-type ("len") into the document of these words
to clarify the intent.  Replacing the "u" with a "len" should improve
the documentation of these words.  The words effected are:

    6.1.0040   #>
    6.1.0570   >NUMBER
    6.1.0980   COUNT
    6.1.1345   ENVIRONMENT?
    6.1.1360   EVALUATE
    6.1.1540   FILL
    6.1.2165   S"
    6.1.2216   SOURCE
    6.1.2310   TYPE
    6.2.2008   PARSE
    6.2.xxxx   PARSE-NAME
11.6.1.1010   CREATE-FILE
11.6.1.1190   DELETE-FILE
11.6.1.1718   INCLUDED
11.6.1.1970   OPEN-FILE
11.6.1.2080   READ-FILE
11.6.1.2090   READ-LINE
11.6.1.2165   S"
11.6.1.2480   WRITE-FILE
11.6.1.2485   WRITE-LINE
11.6.2.1524   FILE-STATUS
12.6.1.0558   >FLOAT
11.6.2.2130   RENAME-FILE
11.6.2.xxxx   REQUIRED
12.6.1.2143   REPRESENT
13.6.1.0086   (LOCAL)
16.6.1.2192   SEARCH-WORDLIST
17.6.1.0170   -TRAILING
17.6.1.0245   /STRING
17.6.1.0780   BLANK
17.6.1.0910   CMOVE
17.6.1.0920   CMOVE>
17.6.1.0935   COMPARE
17.6.1.2191   SEARCH
17.6.1.2212   SLITERAL

Proposal
========

1. Add the following to table 3.1 - Data Types

     len         character-string length               1 cell

2. Add the following to 3.1.1 Data-type relationships

     len => u => x

3. Replace "u" with "len" in 3.1.4.2 Character strings:

     A string is specified by a cell pair (c-addr len) representing
     its starting address and length in characters.

4. Add the following to table 3.5 - Environmental Query Strings:

     /CHARACTER-STRING   n  yes  maximum size of len in characters

5. Change "u" to "len" in the stack description, definition and
    rationale of the words listed under the Solution.

6. Replace "u" with "len" in section A.3.1.3.4 Counted Strings.

7. Change "u" to "len" in the rationale for A.6.2.0855 C".

At any rate, I think the definition at 3.1.4.2 Character strings makes
it clear that "c-addr u" as a unit means something special, so I don't
see any reason to replace the "u" with "len".

character:
    Depending on context, either 1) a storage unit capable of holding a
        character; or 2) a member of a character set.

I think that the presence of an address (i.e. the location of some
storage) makes it pretty clear that sense 1 is meant here, but if people
are confused by that, you might want to clarify.

----

Instead of adopting this (and that "pchar" rename proposal), I think it
would make much more sense to clarify things by leaving the existing
"char" and "character" alone, and instead adopting new terminology for
variable width characters.

As I see it, there's no reason to go changing terminology on people when
you could instead just adopt new terminology for the new concept.  Much
less potential for confusion that way.

The c-addr/len makes live easier as it definitely states that the length
is meant to be in characters (pchars), i.e. the storage unit as meant in
2.1. character 1).

IMO replacing "u" with "len" does improve readability, but does
not resolve the "which length" puzzle.

au-length, log-length, c-length ?

6.1.0898 CHARS
( n1 -- n2 )
n2 is the size in address units of n1 characters.

IMHO, the stack effect is at least misleading.  I find it difficult to
get a correct stack effect - we want -1 CHARS to be used to step through
strings backwards, so we want the sign.  I.e. "len" is not the right
left side of this stack effect (len is a subtype of u, no sign).  But we
basically use CHARS to convert +-len into a +-c-addr offset.  Works fine
on two's complement, might cause problems on one's complement ;-).

   len      length of a character-string in address units       1 cell

I guess whatever character you meant in this:

   3. Replace "u" with "len" in 3.1.4.2 Character strings:

       A string is specified by a cell pair (c-addr len) representing
       its starting address and length in characters.

It would be a substantial change if it were to be address units,
since 1 CHARS is not necessarily one address unit.

I'm unclear what you intend.  Is the meaning of "character
string" in the above being changed to allow for extended
characters?

   3.1.2 Character types
     Characters shall have the following properties:
       – at least one address unit wide;
       – contain at least eight bits;
       – be of fixed width;
       – have a size less than or equal to cell size;
       – be unsigned.

   3.1.2.3 Primitive Character
     A primitive character (pchar) is a character with no restrictions
     on its contents. Unless otherwise stated, a “character” refers to
     a primitive character.

Thus item 3 should be changed to refer to the "length in primitive
characters".  In this case I feel it probably is worth spelling out.

Me, too!

 >> It would be a substantial change if it were to be address units,
 >> since 1 CHARS is not necessarily one address unit.
 >
 > This is part of the problem, what does u mean in CMOVE?  According to
 > its definition "copy u consecutive characters", while most people
 > believe it refers to address units.

Not me!  :-)  MOVE is for that.

 >> I'm unclear what you intend.  Is the meaning of "character
 >> string" in the above being changed to allow for extended
 >> characters?
 >
 > No, but once extended characters are introduced there is the potential
 > for confusion, hence the introduction of a primitive character. Extended
 > characters will always be referenced as "extended character" or xchar,
 > while a "character" is a primitive characters or pchar.

Good!  I can probably wrap my head around that.

A change to let this parameter specify a number of address units would
break existing standard programs.  Granted, there are only few
standard programs that don't have an environmental dependency on
1 CHARS = 1, and all maintained systems support these programs, so
there would be little problem with such a change, but I see little
point in having such a change.  Better propose standardizing
1 CHARS = 1.

It definitely does not refer to address units in these words (only in
MOVE and ERASE, which don't deal with chars), although given that 1
chars = 1 au in all maintained systems, that distinction is of no
consequence.  Every word that refers to chars says so explicitly, and
every word that refers to aus says so explicitly, and if any word in
the xchars proposal refers to a number of xchars, it will say so
explicitly, too (but I don't think there is such a word).

Examples:

From 17.6.1.0910 CMOVE:
|[...] copy u consecutive characters [...]

From 6.1.1900 MOVE:
|[...]

Any system that cannot deal with strings of the length of the longest
data memory region that can be had from the system is broken.  And
that's not just IMO, but also in Forth-94.

So if the point of that query is to allow systems to not process some
of the strings that can be created, then existing standard programs
would become non-standard.  Such a restriction requires a two-step
process of first declaring the feature obsolescent, and eventually
removing it.  Moreover, I see no point in introducing such a
restriction.

If that's not the point of the query, then I see no point in it.  If
we can process all strings we can create, there is no point in
querying for the maximum size.

Otherwise the proposal looks fine.

Anyway, I'm in favour of using a single indication of the pair
whenever they cannot be logically separated.
This allows for a full explanation of "sc" at one place, instead of
limited explanations regarding address units/ character units at
several places. Maybe a distinction between "sc" and "xsc" is in
order.

Except a few experiments, all Forth systems have 1 CHARS = 1.  Most
programs rely on that, as well (i.e. they don't use CHARS where they
should, often, they also don't use CHAR+ but 1+ or so).

So, what is the point in dragging this "CHARS" stuff?

This brings another problem of Standard Forth: lack of internal consistency.
You either have overengineered parts, impractical parts, or lack of standard
tools to solve every day practical tasks (like reading non-textual streams).

Could you and Anton decide for yourself what you really want and stick to it?
Because as for now you easily jump from 1 CHARS being able to hold a byte,
i.e. real character be it 32-bit wide or octet-wide, to 1 CHARS being
addressable unit like it is in C.

Each variant has right to exist and has its own consequences.
If you decide 1 CHARS = 1, then how I access address units? Octets?
If you decide 1 CHARS to be byte width, how do I read non-textual file?

P.S. Most of UNIX text processing programs use "char" and don't care of
locales still, but there's some kind of general consensus that they
should be converted. So what's your argument about? I don't understand it.

Again, you overengineer standard in domain nobody has much experience with,
and skip fixing defects affecting practical everyday tasks.

A more interesting case is word-addressed machines: How should they
deal with BIN?  But I guess if the people implementing and programming
on such systems feel the need for standardization in this regard, they
will come forward and start discussing it.