Why use >r : r>

[santo ilardo]

Hi,
I am a newbie on Forth and I have a question.
From my reading of Programming Forth book I found the following code fragment:

: field \ offset1 n "name" -- offset2 ; addr1 -- addr2
over >r
: r> postpone literal postpone + postpone ;
+
;

What is the purpose of ">r ... r>"?
It seems useless to me, but without it I get an exception.
Thanks

[Alex]

The colon word : puts information on the stack. The >R saves the top of
stack on the return stack, : adds its information to the stack; then R>
gets it off again to use for generating the literal.

Some Forths might be ok without >R and R>, but most require it.

The stacks look like this

S:( n ) R:( )
>R S:( ) R:( n )
: S:( ccc ) R:( n )
R> S:( ccc n ) R:( )
postpone literal S:( ccc ) R:( ) \ generates a literal for n

Without it, this happens

S:( n ) R:( )
: S:( n ccc ) R:( )
postpone literal S:( n ??? ) R:( )

--
Alex

[santo ilardo]

Thank you Alex, crystal clear!
See you
Santo

[rickman]

So is >R and R> used because the stack effect of using : is not well defined
so SWAP would not be recommended?

--

Rick C

Viewed the eclipse at Wintercrest Farms,
on the centerline of totality since 1998

[Coos Haak]

Op Sat, 18 Nov 2017 16:48:05 -0500 schreef rickman:

That is not going to work. ccc is called colon-sys in the standard.
It might be some value on the stack, but traditionally (e.g. FIGFORTH)
stores the depth of the stack with !CSP in the word : and ; checks
if it is the same with ?CSP. In this case ccc might be nothing at all,
so SWAP messes up things.

groet Coos

[rickman]

So your answer is "yes, the stack effect of : is not well defined"... ?

From the standard, the use of : leaves "colon-sys" on the stack. "The
implementation-dependent data generated upon beginning to compile a
definition and consumed at its close is represented by the symbol colon-sys
throughout this standard."

[Coos Haak]

Op Sat, 18 Nov 2017 17:18:41 -0500 schreef rickman:

Yes. For example I tried this in Gforth:

Gforth 0.7.9_20170928, Copyright (C) 1995-2016 Free Software Foundation,
Inc.
Gforth comes with ABSOLUTELY NO WARRANTY; for details type `license'
Type `help' for basic help
: dot-s cr .s ; immediate ok
dot-s : nothing dot-s ; dot-s
<0>
<5> 7696573006048 0 7696573574232 7696573574232 0
<0> ok

So, colon-sys consists of 5 elements. Your SWAP would have
been 5 ROLL and its opposite (5 -ROLL).

Swiftforth leaves nothing on the stack, some other mechanism
must be at work.

Because the trick with >R and R> works in all implementations,
there is no need for other tricks ;-)

groet Coos

[hughag...@gmail.com]

That code is grossly inefficient. This is my version from the novice-package:
---------------------------------------------------------------------
: <field> ( offset -- ) \ run-time: struct-adr -- field-adr \ stream: name
?dup if >r
: state@, if, r@ lit, postpone lit+,
else, r@ lit+, then, ;,
immediate
rdrop
else
: ;, immediate \ the first field does nothing whatsoever
then ;

\ <FIELD> compiles an immediate word, which will compile inline code, so there is *no* CALL instruction executed.

: field ( offset size -- new-offset ) \ stream: name
aligned swap aligned tuck \ -- offset size offset
<field> \ -- offset size
+ ; \ -- new-offset

\ FIELD aligns the offset that it was given, so the new field will be aligned.
\ FIELD also aligns the size of the field so the new-offset is aligned. This is the record size if this is the last field.
\ The record size should be aligned so that in an array of records, each record is aligned (assuming the array is aligned).

\ As a convention, the field names should start with a dot (reminiscent of Pascal fields).
---------------------------------------------------------------------

Note that LIT, is defined like this:

: lit, ( val -- ) \ runtime: -- val
postpone literal ;

I have a lot of words like this provided. These are mostly for convenience, so you don't have a lot of POSTPONE cluttering your source-code.

In some cases however, these are necessary. For example, in <FIELD> above I have:
POSTPONE LIT+,
So, I'm postponing a POSTPONE of another word.

[Julian Fondren]

On Saturday, November 18, 2017 at 5:30:04 PM UTC-6, hughag...@gmail.com wrote:
> On Saturday, November 18, 2017 at 12:06:01 PM UTC-7, santo ilardo wrote:
> > Hi,
> > I am a newbie on Forth and I have a question.
> > From my reading of Programming Forth book I found the following code fragment:
> >
> > : field \ offset1 n "name" -- offset2 ; addr1 -- addr2
> > over >r
> > : r> postpone literal postpone + postpone ;
> > +
> > ;
> >
>

> That code is grossly inefficient.

If you're confused about this, what Hugh means is that some completely
different code than yours is inefficient because that code includes an
unnecessary memory-fetch of a constant at runtime. Your code is fine.

[Alex]

ccc could be from 0 to any number of cells; colon : doesn't specify how
many. >R : R> always works.

--
Alex

[Coos Haak]

Op Sat, 18 Nov 2017 16:16:50 -0800 (PST) schreef Julian Fondren:

As the Dutch proverb says:
He has heard the bell ringing,
(as usual, triggered by the his pet word field)
but did not know where the clapper hangs.
(the problem was not 'field', but why the use of >r and r>).

groet Coos

[dxf...@gmail.com]

Shouldn't that be STATE @ not STATE@, ?

[hughag...@gmail.com]

On Saturday, November 18, 2017 at 12:06:01 PM UTC-7, santo ilardo wrote:

This is what the ANS-Forth manual (section 3.2.3.2) says:
---------------------------------------
The control-flow stack may, but need not, physically exist in an implementation. If it does exist, it may be, but need not be, implemented using
the data stack. The format of the control-flow stack is implementation defined. Since the control-flow stack may be implemented using the data stack, items
placed on the data stack are unavailable to a program after items are placed on the control-flow stack and remain unavailable until the control-flow stack
items are removed.
---------------------------------------

ANS-Forth was a marketing gimmick of Forth Inc..
Elizabeth Rather wanted to get as many supporters as possible, so she told everybody that they were ANS-Forth compliant.
Some people had a separate control-flow stack (this is the intelligent design), so this is compliant.
Forth Inc. does not have a separate control-flow stack (because nobody at Forth Inc. understands meta-compiling), so this is compliant too.
The result is that the user has to assume the worst-case scenario (SwiftForth) and assume that the control-flow stack is not separate.

The control-flow data (produced by : and IF etc.) is on the data-stack, so it is on top of the user's data preventing the user from accessing his own data.
In the novice-package I use local variables to get around this bug in ANS-Forth.
It is also possible to use the return-stack.

[hughag...@gmail.com]

It is correct --- we are metacompiling a meta-compiling word.

: state@, ( -- ) \ runtime: -- state@
postpone state postpone @ ;

[Elizabeth D. Rather]

In the 80's some systems started putting a marker on the stack in : and
checking it in ; to detect a possible stack effect due to an incomplete
structure (such as an IF without a THEN, which would leave an address on
the stack). After extended discussion, ANS Forth decided to allow this
practice by defining this 'colon-sys' as something whose size and
content was entirely system-dependent: you don't know how many items, if
any, there are or what they are. In effect, this blocks the programmer
from passing data into or out of a colon definition. As Coos notes, the
>R ... R> trick avoids the problem.

Cheers,
Elizabeth

--
Elizabeth D. Rather
FORTH, Inc.
6080 Center Drive, Suite 600
Los Angeles, CA 90045
USA

[TG9541]

On Saturday, November 18, 2017 at 11:56:27 PM UTC+1, Coos Haak wrote:

> Because the trick with >R and R> works in all implementations,
> there is no need for other tricks ;-)

Coos, maybe the eForth I'm working on is obsolete, but like Swiftforth ":" leaves nothing on the stack. Passing an value into a colon definition "just works".

However, STM8 eForth is an STC Forth, and it doesn't tolerate using the return stack in the "outer interpreter", and >R and R> are flagged "compile only" for a good reason. The "inner interpreter" consists of the µC's code execution, and the "outer interpreter" also uses a CALL instruction for any word it executes.

For the sake of demonstration, I removed the "compile only" flag from the dictionary entries of >R and R>:

STM8eForth 2.2.20
10 >R ok
: test [ R> ] LITERAL ; ok
... crash ...
STM8eForth 2.2.20
10 >R ok
R> hex . 8E60 ok
... crash ...
STM8eForth 2.2.20

Calling R> fetches the return address of EVAL in the QUIT loop, which leads to a crash (fortunately the safety features of the STM8 core trigger a restart).

On the other hand, >R and R> are still useful for compiled code: it provides an extra stack which most often makes ROLL redundant (ROLL requires a memory move operation).

In Forth Dimensions Vol.III issue 1 p.20 Peter H. Helmers discussed the virtues of "userstacks", which might be a good (safe and portable) replacement return stack manipulations: https://www.complang.tuwien.ac.at/forth/forth-dimensions/FD-V3.pdf

Groetjes,
Thomas

[dxf...@gmail.com]

My mistake. IIUC it generates a unique compiler
word for each and every field defined. This seems
excessive. What about this:

: FIELD create over , + immediate
does> @ ?dup if
state @ if postpone literal postpone +
else + then
then ;

The effect is the same but uses less memory.

[Anton Ertl]

TG9541 <thomas....@gmail.com> writes:
>However, STM8 eForth is an STC Forth, and it doesn't tolerate using the ret=
>urn stack in the "outer interpreter", and >R and R> are flagged "compile on=

>ly" for a good reason.

The standard does not define the interpretation semantics of >R and
R>, that's all. In the definition of FIELD the interpretation
semantics of >R and R> are not used. I have no idea what you mean
with 'tolerate using the return stack in the "outer interpreter"'.
The definition is of FIELD is standard (but the word it defines is
called +FIELD in the standard). If it does not work as intended on
eForth, eForth is non-standard; if eForth is claimed to be standard,
it's a bug in eForth.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2017: http://euro.theforth.net/

[Anton Ertl]

...

>> > > : lit, ( val -- ) \ runtime: -- val
>> > > postpone literal ;

...

>> : state@, ( -- ) \ runtime: -- state@
>> postpone state postpone @ ;

Even with LIT, and STATE@, defined, <FIELD> is still incomplete:
"LIT+,", "IF,", "ELSE,", "THEN," and ";," are missing.

Anyway, about your claim of gross inefficiency, let's try it:

: field \ offset1 n "name" -- offset2 ; addr1 -- addr2
over >r
: r> postpone literal postpone + postpone ;
+
;

0
1 cells field x
1 cells field y
constant mystruct

create bla 1 , 2 ,

: x@ x @ ;
: y@ y @ ;
: foo dup x@ swap y@ + ;

see x@
see y@
see foo

On VFX Forth 4.72 this shows:

see x@
X@
( 0 bytes, 0 instructions )
see y@
Y@
( 080C0BA0 8B5B04 ) MOV EBX, [EBX+04]
( 080C0BA3 C3 ) NEXT,
( 4 bytes, 2 instructions )
see foo
FOO
( 080C0BC0 8B13 ) MOV EDX, 0 [EBX]
( 080C0BC2 035304 ) ADD EDX, [EBX+04]
( 080C0BC5 8BDA ) MOV EBX, EDX
( 080C0BC7 C3 ) NEXT,
( 8 bytes, 4 instructions )

On iforth I have to use IDIS instead of SEE to see the assembly code.
What I get is:

FORTH> ' x@ idis
$10226A80 : x@ 488BC04883ED088F4500 H.@H.m..E.
$10226A8A pop rbx 5B [
$10226A8B push [rbx] qword FF33 .3
$10226A8D ; 488B45004883C508FFE0 H.E.H.E..`
FORTH> ' y@ idis
$10226B00 : y@ 488BC04883ED088F4500 H.@H.m..E.
$10226B0A pop rbx 5B [
$10226B0B push [rbx 8 +] qwordFF7308 .s.
$10226B0E ; 488B45004883C508FFE0 H.E.H.E..` ok
FORTH> ' foo idis
$10226B80 : foo 488BC04883ED088F4500 H.@H.m..E.
$10226B8A pop rbx 5B [
$10226B8B mov rdi, [rbx] qword
488B3B H.;
$10226B8E add rdi, [rbx 8 +] qword
48037B08 H.{.
$10226B92 push rdi 57 W
$10226B93 ; 488B45004883C508FFE0 H.E.H.E..` ok

Does your version produce better results? I doubt it.

And of course, the STATE-smart fields your version produces will not
work in all cases.

> What about this:
>
>: FIELD create over , + immediate
> does> @ ?dup if
> state @ if postpone literal postpone +
> else + then
> then ;

At least it's complete and less complex, but again, it does not work
in all cases, e.g.:

1 cells 1 cells field foo drop
create bar 5 , 6 ,
defer flip ' foo is flip
: bla bar flip @ postpone literal ; immediate
: blubb bla ;
0 blubb cr . . \ prints 6 0 with a correct FIELD

[TG9541]

On Sunday, November 19, 2017 at 11:50:07 AM UTC+1, Anton Ertl wrote:

> TG9541 <thomas...> writes:
> >However, STM8 eForth is an STC Forth, and it doesn't tolerate using the ret=
> >urn stack in the "outer interpreter", and >R and R> are flagged "compile on=
> >ly" for a good reason.
>
> The standard does not define the interpretation semantics of >R and
> R>, that's all. In the definition of FIELD the interpretation
> semantics of >R and R> are not used. I have no idea what you mean
> with 'tolerate using the return stack in the "outer interpreter"'.
> The definition is of FIELD is standard (but the word it defines is
> called +FIELD in the standard). If it does not work as intended on
> eForth, eForth is non-standard; if eForth is claimed to be standard,
> it's a bug in eForth.

Dear Anton,

I was referring to a property of common STC Forth implementations, not to eForth (which doesn't rely on colon-sys). It may well be that ANS complient STC implementations are possible that also work with the >R .. R> method.

I think that Elisabeth explained quite well that the way the ANS Forth of old deals with "colon-sys" breaks passing data into a colon definition through the data stack, and >R .. R> is a work-around (not a very elegant one, as the initial question in this thread shows). Some claim that this work-around always works, but especially an STC ANS Forth would have be designed to work in spite of it. I hope that this makes my point clear.

I feel that this is an issue that deserves a robust solution, and, as I'm approaching Forth from the 80s perspective (when it was still simple enough to run on a very modest machine) I think that a "userstack" for the compiler points in the right direction. Please bear with me when I can't yet compare such a simple approach with the FIELD element you mention. I hope that I'll be getting there.

As far as I know, eForth predates the ANS standard, and I don't think that anybody would ever claim that it's ANS compliant. It works nicely as a lowly untethered Forth on a µC for less than 20ct.

[Anton Ertl]

TG9541 <thomas....@gmail.com> writes:
>I was referring to a property of common STC Forth implementations, not to e=
>Forth (which doesn't rely on colon-sys). It may well be that ANS complient =

>STC implementations are possible that also work with the >R .. R> method.

I don't see any reason why an STC implementation should have more
difficulties with this implementation of FIELD than other
implementation techniques.

On what which other implementations besides eForth do you base the
claim that this is a property of common STC implementations? I just
tried it on bigForth 2.3.1 (an STC Forth), and it works as expected.

>I feel that this is an issue that deserves a robust solution, and, as I'm a=
>pproaching Forth from the 80s perspective (when it was still simple enough =
>to run on a very modest machine) I think that a "userstack" for the compile=

>r points in the right direction.

Certainly having a separate stack for the control-flow stack would
make the >R ... R> complication unnecessary in this case, but cause
complications in the implementation (moving from the separate
control-flow stack to the data stack and back), i.e., something you
don't want on a very modest machine; and even on todays larger
machines, the number of Forth systems with a separate control-flow
stacks is small (I actually don't know any), so it seems that even on
not-so-modest machines that benefits of a separate control-flow stack
are not deemed to outweigh the drawbacks.

[TG9541]

On Sunday, November 19, 2017 at 3:14:11 PM UTC+1, Anton Ertl wrote:

> TG9541 <thomas...> writes:
> >I was referring to a property of common STC Forth implementations, not to e=
> >Forth (which doesn't rely on colon-sys). It may well be that ANS complient =
> >STC implementations are possible that also work with the >R .. R> method.
>
> I don't see any reason why an STC implementation should have more
> difficulties with this implementation of FIELD than other
> implementation techniques.

You're right, this was my mistake. There is never a return to the outer interpreter while this FIELD code is executed.

> >I feel that this is an issue that deserves a robust solution, and, as I'm a=
> >pproaching Forth from the 80s perspective (when it was still simple enough =
> >to run on a very modest machine) I think that a "userstack" for the compile=
> >r points in the right direction.
>
> Certainly having a separate stack for the control-flow stack would
> make the >R ... R> complication unnecessary in this case, but cause
> complications in the implementation (moving from the separate
> control-flow stack to the data stack and back), i.e., something you
> don't want on a very modest machine; and even on todays larger
> machines, the number of Forth systems with a separate control-flow
> stacks is small (I actually don't know any), so it seems that even on
> not-so-modest machines that benefits of a separate control-flow stack
> are not deemed to outweigh the drawbacks.

You're again right, using the return stack is the most efficient solution.

[Paul Rubin]

TG9541 <thomas....@gmail.com> writes:
> In Forth Dimensions Vol.III issue 1 p.20 Peter H. Helmers discussed
> the virtues of "userstacks", which might be a good (safe and portable)
> replacement return stack manipulations:
> https://www.complang.tuwien.ac.at/forth/forth-dimensions/FD-V3.pdf

That looks painful and I think it's preferable for the interpreter to
support locals for when they are needed. I haven't thought much about
how to implement them in the semi-tethered Forth, but I know they make
the text interpreter more complicated. Being able to support a fancy
text interpreter with locals was one of the motivations for separating
the host-side text interpreter from the small, low-level target-side
interpreter.

[Coos Haak]

Would the following (all on the same line) work in STM8eForth 2.2.20?

10 >R : test [ R> ] LITERAL ;

Here the return address of the interpret/eval loop
should not be affected.

groet Coos

[hughag...@gmail.com]

On Sunday, November 19, 2017 at 2:28:00 AM UTC-7, dxf...@gmail.com wrote:
> On Sunday, November 19, 2017 at 1:17:38 PM UTC+11, hughag...@gmail.com wrote:
> > On Saturday, November 18, 2017 at 7:09:01 PM UTC-7, dxf...@gmail.com wrote:
> > > Shouldn't that be STATE @ not STATE@, ?
> >
> > It is correct --- we are metacompiling a meta-compiling word.
>

> My mistake. IIUC it generates a unique compiler
> word for each and every field defined. This seems
> excessive. What about this:
>
> : FIELD create over , + immediate
> does> @ ?dup if
> state @ if postpone literal postpone +
> else + then
> then ;
>
> The effect is the same but uses less memory.

That is an okay solution. I wrote my version back in 2009 when I was just learning ANS-Forth.
This version is somewhat simpler, and it uses less memory (memory usage is not important though).
The run-time code generated is the same either way, and in both cases the state-smart problem is there (ticking the field word is beyond unlikely though).
The only problem that I see with your code is that you are not guaranteeing that the field offsets are aligned, which I do.

IIRC, that is Stephen Pelc's code --- I think it was in his book.

ANS-Forth is a terrible language standard! FIELD should be in the standard!
It is ridiculous to force novices to write code like this themselves.
It is impossible to write non-trivial programs without structs, but most novices aren't going to write FIELD themselves ---
most novices are going to give up on ANS-Forth as a toy language and go back to using C again.

Now Forth-200x has this --- but they gratuitously changed the name --- this is despite the fact that FIELD has been the common-practice name since the 1980s.
AFAIK, Forth-200x also fails to align the field offsets --- most likely they are using that code you showed above --- Stephen Pelc is the chair-person now.

It is also ridiculous that ANS-Forth lacks a control-flow stack and uses the data-stack instead.
This makes meta-compiling much more complicated than necessary. Our OP found this to be confusing --- it is confusing! --- ANS-Forth is a stupid design.

ANS-Forth and Forth-200x make Forth look stupid!
Forth is a good language --- Forth has a future --- Elizabeth Rather and Stephen Pelc will not be a part of that future though.

[dxf...@gmail.com]

Neither can DO LOOP be used in every imaginable
situation, yet used it still is.
State-smartness is a tool like any other. If
you get yourself into trouble through ignorance,
shall you condemn the tool and expend resources
avoiding it - or do you learn from your mistake and thereafter use the tool appropriately?

[Alex]

On 20-Nov-17 00:30, dxf...@gmail.com wrote:
> Neither can DO LOOP be used in every imaginable situation, yet used
> it still is. State-smartness is a tool like any other. If you get
> yourself into trouble through ignorance, shall you condemn the tool > and expend resources avoiding it - or do you learn from your mistake
> and thereafter use the tool appropriately?

It's possible to write Forth systems that not only avoid state smartness
and the problems, however slight, it brings, but that are also simpler
to maintain and that are easily and correctly extended.

http://amforth.sourceforge.net/Recognizers.html

I have written such a system based on those ideas, so it can't be that
hard to develop quality tools.

--
Alex

[Albert van der Horst]

We already have a global state that affects all human interpretation
and machine interpretation of words: the namespace stack (wordlists).
All languages have something similar and it is generally seen as
hard to avoid for non-trivial tasks.
Recognizers replace STATE (who is at least understandable) by another
global state the recognizer stack, that affects affects all human
interpretation and machine interpretation of words.

After all the talk I still feel to see why this could be
an improvement.

>--
>Alex
--
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

[Anton Ertl]

dxf...@gmail.com writes:
>> At least it's complete and less complex, but again, it does not work
>> in all cases, e.g.:
>>
>> 1 cells 1 cells field foo drop
>> create bar 5 , 6 ,
>> defer flip ' foo is flip
>> : bla bar flip @ postpone literal ; immediate
>> : blubb bla ;
>> 0 blubb cr . . \ prints 6 0 with a correct FIELD
>>
>
>Neither can DO LOOP be used in every imaginable
>situation, yet used it still is.
>State-smartness is a tool like any other. If
>you get yourself into trouble through ignorance,
>shall you condemn the tool and expend resources
>avoiding it - or do you learn from your mistake and thereafter use the tool appropriately?

If the problems of STATE-smartness just bit the programmer who uses
it, I would have no problem with it.

But the problem with STATE-smartness is that the trap tends to spring
very far, both in time and space, from where it was set.
E.g. consider the example above, and consider that the individual
pieces are parts of bigger libraries or programs.

Programmer A defines FIELD.
Programmer B defines FOO.
Programmer C defines BAR.
Programmer D defines BLA.
Programmer E writes "' foo is flip".
Programmer F defines BLUBB, and in testing, realizes that it does not
work as expected.

Who should fix the bug? A, B, C, D, E, or F? And how should it be
fixed?

Did they document the STATE-smartness of FIELD's children, of FOO, and
the potential STATE-smartness of FLIP, and of BLA? Now some Forthers
don't document their programs, and they hold the position that users
of a piece of code should read the source and discover by themselves
the traps (such as STATE-smartness) that this code's programmer put in
the code, and its effect on the applicability of this piece of code.

In that position F is at fault for not having read and completely
understood all the code that is used in the definition of BLUBB. The
bugfix? He cannot use FIELD, FOO, BAR, and BLA from the libraries in
pristine condition, so he either has to change some of these libraries
(so from then on he has to maintain them himself), or write BLA and
all that it uses by himself. Given that, in such a world, these
libraries probably contain other traps, too, the latter looks like the
better solution.

So if STATE-smartness is a tool like any other, we have an environment
where we better don't use libraries (is that what you mean with "learn
from your mistake and thereafter use the tool appropriately"?).

Not using libraries is actually Chuck Moore's position. Ironically,
Chuck Moore eliminated STATE from his Forths, certainly from cmForth,
but from what I read here, also from polyForth. So he apparently did
not consider STATE-smartness a legitimate tool, at least after a
while.

In any case there are those of us who think that it is useful to be
able to use libraries. Then we need an environment where the
libraries have simple and universal interfaces, not interfaces full of
traps and caveats. And in such an environment STATE-smartness is
inappropriate, because it leads to traps and caveats. And "I have not
had any problems yet" is not good enough if you program for general
consumption.

And in the case of FIELD there are much better alternatives:

A straighforward (and shorter) way to write FIELD is:

: field ( offset1 n "name" -- offset2 )
create over , +
does> ( addr1 -- addr2 )
@ + ;

Disadvantages of this implementation that the STATE-smart
implementations try to fix:

1) The @ cannot be optimized away, which costs cycles on some CPUs.

2) Simple Forth implementations won't inline the call to children of
FIELD.

The implementation at the start of the thread

: field \ offset1 n "name" -- offset2 ; addr1 -- addr2
over >r
: r> postpone literal postpone + postpone ;
+
;

solves problem 1, but not problem 2. Is problem 2 worth solving?
Maybe, but is it worth to exchange it for the problems caused by
STATE-smartness? Not in a library for general consumption.

What else can we do? In current development Gforth, you can write

: field ( offset1 n "name" -- offset2 )
\ you must not change the body of "name"
create over , +
[: @ + ;] set-does> ( addr1 -- addr2 )
[: >body @ ]] literal + [[ ;] set-optimizer ;

This solves both problem 1 and 2, and each child of FIELD consumes
less memory than in the definition from the start of the thread.

There is someone who prefers

OPT: <code>

to

[: <code> ;] SET-OPTIMIZER

With OPT: and DOES>, and without ]] ... [[, the definition would look
like this:

: field1 ( offset1 n "name" -- offset2 )
create over , +
does> ( addr1 -- addr2 )
@ + ;

: field ( offset1 n "name" -- offset2 )
\ you must not change the body of "name"
field1
OPT:
>body @ postpone literal postpone + ;

I leave it to you to decide which notation is better, but in any case,
both definitions are equivalent and have the same advantages when used.

Do we need STATE-smartness for implementing FIELD? No!

[Alex]

On 20-Nov-17 13:20, Albert van der Horst wrote:
> In article <ouud9u$icd$1...@dont-email.me>, Alex <al...@rivadpm.com>
> wrote:
>> On 20-Nov-17 00:30, dxf...@gmail.com wrote:
>>> Neither can DO LOOP be used in every imaginable situation, yet
>>> used it still is. State-smartness is a tool like any other. If
>>> you get yourself into trouble through ignorance, shall you
>>> condemn the tool > and expend resources avoiding it - or do you
>>> learn from your mistake and thereafter use the tool
>>> appropriately?
>>
>> It's possible to write Forth systems that not only avoid state
>> smartness and the problems, however slight, it brings, but that are
>> also simpler to maintain and that are easily and correctly
>> extended.
>>
>> http://amforth.sourceforge.net/Recognizers.html
>>
>> I have written such a system based on those ideas, so it can't be
>> that hard to develop quality tools.
>
> We already have a global state that affects all human interpretation
> and machine interpretation of words: the namespace stack
> (wordlists). All languages have something similar and it is generally
> seen as hard to avoid for non-trivial tasks. Recognizers replace
> STATE (who is at least understandable) by another global state the

Anton's example
https://groups.google.com/forum/#!msg/comp.lang.forth/_aaYZDoFvcM/1Z9ZIDtkAwAJ
shows how something "at least understandable" may have to span several
words.

> recognizer stack, that affects affects all human interpretation and
> machine interpretation of words.
>
> After all the talk I still feel to see why this could be an
> improvement.

As a gentle reminder, recognisers are not standardised. Therefore, you
are not required to use or even acknowledge the existence of the
recognizer stack. Just write standard Forth, which continues to work as
advertised on a system that provides them.

It may be that we find some way to coalesce wordlists and recognizers. I
have played with such an idea in my head, but have not yet implemented
it in code. It needs a different way of thinking about the parsing &
recognising processes in Forth, and how and what it means to identify a
token.

Wordlists are dynamically extensible (by adding : CREATE VALUE
definitions and so on that can have various outputs when executed or
compiled), but are based off a single fixed parse (i.e. they are all
whitespace delimited lookups).

On the other hand, recognisers as currently defined permit dynamic
parsing (anything goes) but generally produce a fixed output (for
instance, a cell sized number). That is true of all Forths btw; they all
have at least a number recogniser, although it may not be called that.

Then there are the attempts at defining some kind of object syntaxes;
object.method parses as used by Win32Forth, for instance.

These are all aspects of the same problem. I believe it worth continuing
the experiment to extend Forth programming to remove the requirement for
STATE and IMMEDIATE.

--
Alex

[Matthias Trute]

> Recognizers replace STATE

Not true.

They *may* make STATE less important.

Matthias

[Albert van der Horst]

In article <2017Nov2...@mips.complang.tuwien.ac.at>,
Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
<SNIP>

>
>Not using libraries is actually Chuck Moore's position. Ironically,
>Chuck Moore eliminated STATE from his Forths, certainly from cmForth,
>but from what I read here, also from polyForth. So he apparently did
>not consider STATE-smartness a legitimate tool, at least after a
>while.

Au contraire. Chuck replace the interpretation/compilation state with colors.
But the color is only valid for the following word.
There is not even an attempt to POSTPONE colored words, which would
or would not depend on their color and would make an awful mess.
The word is the word, and postponing them is just another color.

But who is introducing postpone state between [[ and ]]? Not me.

>In any case there are those of us who think that it is useful to be
>able to use libraries. Then we need an environment where the
>libraries have simple and universal interfaces, not interfaces full of
>traps and caveats. And in such an environment STATE-smartness is
>inappropriate, because it leads to traps and caveats. And "I have not
>had any problems yet" is not good enough if you program for general
>consumption.

In response I looked up where I use STATE in the ciforth library.
I came across this example which uses STATE where I'm interested to
hear your comment. It is used to allow LOOPs during interpretation


: LOOP POSTPONE LOOP POSTPONE T[ ; IMMEDIATE
: DO T] POSTPONE DO ; IMMEDIATE

\ Start compiling at temporary place : return START and STATE.
: T] STATE @ 0= IF SWAP-DP HERE THEN STATE @ ] ;
\ Execute code at START dropping STATE, restore dictionary.
: T[ 0= IF POSTPONE (;) SWAP-DP POSTPONE [ >R THEN ; IMMEDIATE

: SWAP-DP DP @ FAR-DP @ DP ! FAR-DP ! ;


Are their pitfalls of doing something like that?

Using control words in interpret mode is a great convenience
in scripting. I never realized it could spell problems
with libraries. (That also implies that I never came across
a problem.)

>
>And in the case of FIELD there are much better alternatives:
>
>A straighforward (and shorter) way to write FIELD is:
>
>: field ( offset1 n "name" -- offset2 )
> create over , +
>does> ( addr1 -- addr2 )
> @ + ;
>
>Disadvantages of this implementation that the STATE-smart
>implementations try to fix:
>
>1) The @ cannot be optimized away, which costs cycles on some CPUs.
>
>2) Simple Forth implementations won't inline the call to children of
> FIELD.
>

<SNIP>

>Do we need STATE-smartness for implementing FIELD? No!

As far as I can tell this STATE-smartness was only used to make
things faster. So it was never *needed* in the first place.

The argument doesn't strike home with me, because I wouldn't be
inclined to do things like that.

>
>- anton

[Alex]

On 20-Nov-17 21:35, Albert van der Horst wrote:
> In article <2017Nov2...@mips.complang.tuwien.ac.at>,
> Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
> <SNIP>
>>
>> Not using libraries is actually Chuck Moore's position. Ironically,
>> Chuck Moore eliminated STATE from his Forths, certainly from cmForth,
>> but from what I read here, also from polyForth. So he apparently did
>> not consider STATE-smartness a legitimate tool, at least after a
>> while.
>
> Au contraire. Chuck replace the interpretation/compilation state with colors.
> But the color is only valid for the following word.
> There is not even an attempt to POSTPONE colored words, which would
> or would not depend on their color and would make an awful mess.
> The word is the word, and postponing them is just another color.
>
> But who is introducing postpone state between [[ and ]]? Not me.

It's not a state per se; ]] A B C [[ is a shorthand for POSTPONE A
POSTPONE B POSTPONE C. Nothing can inspect STATE, since none of A B or C
can be executed.

--
Alex

[hughag...@gmail.com]

On Monday, November 20, 2017 at 6:44:53 AM UTC-7, Anton Ertl wrote:
> In any case there are those of us who think that it is useful to be
> able to use libraries. Then we need an environment where the
> libraries have simple and universal interfaces, not interfaces full of
> traps and caveats. And in such an environment STATE-smartness is
> inappropriate, because it leads to traps and caveats. And "I have not
> had any problems yet" is not good enough if you program for general
> consumption.

You are acting very scholarly, as if you are an expert on code-libraries --- but you haven't written any code-library comparable to my novice-package.

I agree that state-smartness is a major problem in code-libraries.
I rewrote MACRO: to use early-binding rather than late-binding, but the generated word is still state-smart.

MACRO: should be part of the language. In this case, the word would not be state-smart. It would just be a regular colon-word that can be ticked.
The dictionary header however, would contain a flag indicating that COMPILE, should inline the code rather than compile a function call.
Also, MACRO: words have looser rules --- a >R without a R> or vice-versa, should be okay.
Also, there should be a late-binding definer (I have LATE-MACRO: in the novice-package) also that allows IF without THEN etc., which is occasionally needed.

The lack of a MACRO: in ANS-Forth is another indication of the idiocy of the ANS-Forth language definition.
I was well aware of the need to inline some words in 1994 when I wrote MFX --- I had an early-binding MACRO: in MFX (MFX was written in UR/Forth).
None of the ANS-Forth committee apparently knew about inlining code --- they provided no support in ANS-Forth.
ANS-Forth is not comparable to my MFX, although both were written in 1994.

As for FIELD , that should be part of the language standard.
It is ridiculous that novices are required to write their own FIELD , which they aren't usually capable of.
Also, when everybody is required to write their own, they may be incompatible with each other (for example, Forth-200x changed the name gratuitously).
Field names should be like macros in that they can be ticked to get an xt, but COMPILE, inlines them (they have that header flag set to indicate this).

As a practical matter, the FIELD that I wrote or the FIELD that Stephen Pelc wrote (both in ANS-Forth), are okay.
The fact that field names are state-smart is not an issue because nobody ever needs to tick them.
My naming convention is for field names to have a dot prefix (reminiscent of Pascal field names), so this tells users that it is a field name.

Theoretically this leads to "traps and caveats" --- in practice, field names can be immediate without any confusion --- this is a non-problem.
I agree that willy-nilly use of state-smart words will cause problems.

None of these definitions are any good!
They don't guarantee that the fields will be on aligned addresses, or that the struct size will be a multiple of the cell size.
My EXCHANGE used in SORT requires the array-element size to be a multiple of the cell size, which my FIELD guarantees.
Also, making the field offsets aligned significantly boosts the speed for accessing them on any computer bigger than a 65c02.

You should know this!
Writing code that fails at this makes you look like a high-school student --- you Forth-200x committee members produce some awfully low-quality code!

> Do we need STATE-smartness for implementing FIELD? No!

I wrote a state-smart FIELD in 2009 when I was learning ANS-Forth, and I was using SwiftForth.
The problem with SwiftForth is that it does almost no optimization at all. The quality of SwiftForth is comparable to the shareware Forths of the 1980s.
Your code above assumes that there is an optimizing compiler.
I was primarily supporting SwiftForth when I manually did the optimization with my state-smart field words.
Elizabeth Rather did not thank me for going out of my way to support SwiftForth, working around its lack of optimization.
Instead, she continues to promote the simplistic FIELD that generates grossly inefficient code under SwiftForth, but does get optimized under VFX.
Not only is Elizabeth Rather dumber than a box of rocks, but she is too dumb to know that she is dumb! This is why she routinely makes a fool out of herself.

Anyway, this whole thread is stupid. I wrote FIELD in 2009 and we still have these grossly inefficient definitions of FIELD being promoted on C.L.F..
The year 2017 is really too late for Elizabeth Rather to learn about structs and learn how to write an efficient FIELD definer.
All of this has been discussed multiple times starting in 2009 and periodically every year or two, and nothing has been accomplished. Your code still sucks!

I really don't have the patience to continue repeating myself over and over, year after year.
I don't think the ANS-Forth or Forth-200x enthusiasts are ever going to learn to write Forth code beyond the most rudimentary high-school-student level.
Elizabeth Rather has never advanced beyond this level --- the code snippets she has posted on C.L.F. were copied directly out of the "Starting Forth" book.

ANS-Forth and Forth-200x are worthless --- actually, less than worthless because they make the whole Forth community look stupid.
Stop claiming to be the "Standard" (capital 'S') for Forth! Just admit that your ANS-Forth and Forth-200x are goofy amateurish attempts at Forth.
The FIND in GFORTH returns different xt values depending upon what STATE is at the time that FIND executes. That is amateurish! You call it "Standard" though.
Only after you admit that you don't really know anything about Forth, will you begin to learn Forth --- you have to be a student before you can be a teacher.

[Alex]

On 20-Nov-17 23:25, hughag...@gmail.com wrote:


> I really don't have the patience to continue repeating myself over
> and over, year after year.

A great blessing.

--
Alex

[dxf...@gmail.com]

I've written a few forth apps over the years.
All I needed was a few library routines to get
me started. Once I'd written those, it was plain
sailing. Can't say I've needed the quality tools
some hold in high esteem - particularly the ones
designed to protect the programmer from himself.

[Lars Brinkhoff]

Alex wrote:
>> But who is introducing postpone state between [[ and ]]? Not me.
> It's not a state per se; ]] A B C [[ is a shorthand for POSTPONE A
> POSTPONE B POSTPONE C.

I toyed with the idea to make it a proper state. [[ would be a new kind
of immediate word which exits the postpone state. Similarly, there
might be another state entered by [IF] or [ELSE], and exited by [THEN].

[dxf...@gmail.com]

> EuroForth 2017:the http://euro.theforth.net/

AFAIK +FIELD was standardized. If you don't want
child words to be state-smart that's where you
would specify it. Swiftforth has yet to implement
+FIELD and it remains to be seen how they do it.
Swiftforth doesn't inline colon defs and I'm
assume that's why Hugh chose state-smart for his
version. FI could decide CREATE ;CODE is fast
enough. On DTC forths it"s almost certainly
faster and more memory effective than laying down
LIT + .

For non-standard words or applications there's
nothing stopping a programmer defining state-
smart words. It comes with the same
responsibility as anything else. If you think
it should be documented, do so.

[a...@littlepinkcloud.invalid]

Albert van der Horst <alb...@cherry.spenarnc.xs4all.nl> wrote:
> In article <2017Nov2...@mips.complang.tuwien.ac.at>,
> Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
> <SNIP>
>>
>>Not using libraries is actually Chuck Moore's position. Ironically,
>>Chuck Moore eliminated STATE from his Forths, certainly from cmForth,
>>but from what I read here, also from polyForth. So he apparently did
>>not consider STATE-smartness a legitimate tool, at least after a
>>while.
>
> Au contraire. Chuck replace the interpretation/compilation state with colors.

That was a long time after the period that Anton is talking about.
Forth, Inc. was completely STATEless by the very early 1980s, and
STATE only came back, very sadly, because of ANS Forth.

> In response I looked up where I use STATE in the ciforth library.
> I came across this example which uses STATE where I'm interested to
> hear your comment. It is used to allow LOOPs during interpretation
>
>
> : LOOP POSTPONE LOOP POSTPONE T[ ; IMMEDIATE
> : DO T] POSTPONE DO ; IMMEDIATE
>
> \ Start compiling at temporary place : return START and STATE.
> : T] STATE @ 0= IF SWAP-DP HERE THEN STATE @ ] ;
> \ Execute code at START dropping STATE, restore dictionary.
> : T[ 0= IF POSTPONE (;) SWAP-DP POSTPONE [ >R THEN ; IMMEDIATE
>
> : SWAP-DP DP @ FAR-DP @ DP ! FAR-DP ! ;
>
> Are their pitfalls of doing something like that?

It's going to get quite entertaining if someone tries to POSTPONE your
new LOOP in a STATE-smart word. :-)

>>A straighforward (and shorter) way to write FIELD is:
>>
>>: field ( offset1 n "name" -- offset2 )
>> create over , +
>>does> ( addr1 -- addr2 )
>> @ + ;

And this would be the right way to do it.

> As far as I can tell this STATE-smartness was only used to make
> things faster. So it was never *needed* in the first place.
>
> The argument doesn't strike home with me, because I wouldn't be
> inclined to do things like that.

I agree with you.

Andrew.

[Anton Ertl]

Alex <al...@rivadpm.com> writes:
>It's possible to write Forth systems that not only avoid state smartness
>and the problems, however slight, it brings, but that are also simpler
>to maintain and that are easily and correctly extended.
>
>http://amforth.sourceforge.net/Recognizers.html

While recognizers help avoid a common use pattern of STATE-smart words
(code involving parsing words that is copy-paste-able between
interpreted and compiled code), the case being discussed here does not
fit that use pattern and recognizers don't help here.

What Hugh Aguilar is trying to achieve is to inline the code for the
field. A better approach is to use a Forth system that inlines such
small words automatically. And, e.g., VFX and iForth do that.

[Alex]

On 21-Nov-17 12:23, Anton Ertl wrote:
> Alex <al...@rivadpm.com> writes:
>> It's possible to write Forth systems that not only avoid state smartness
>> and the problems, however slight, it brings, but that are also simpler
>> to maintain and that are easily and correctly extended.
>>
>> http://amforth.sourceforge.net/Recognizers.html
>
> While recognizers help avoid a common use pattern of STATE-smart words
> (code involving parsing words that is copy-paste-able between
> interpreted and compiled code), the case being discussed here does not
> fit that use pattern and recognizers don't help here.

I know that; I was responding to

---

State-smartness is a tool like any other. If you get
yourself into trouble through ignorance, shall you condemn the tool
and expend resources avoiding it - or do you learn from your mistake
and thereafter use the tool appropriately?

---

As a big fan of recognisers (is it recognisers or recognizers? I have
been very inconsistent with my spelling) I think it's worth pointing out
that there is really no good reason to continue to tolerate the darker
corners of Forth when there's really no reason to do so.

>
> What Hugh Aguilar is trying to achieve is to inline the code for the
> field. A better approach is to use a Forth system that inlines such
> small words automatically. And, e.g., VFX and iForth do that.

Premature optimisation (or is it optimization?). Hugh would be better
served by better understanding algorithmic efficiencies; q.v. use of
splay trees for dictionaries.

--
Alex

[Albert van der Horst]

In article <ov1c81$je8$1...@dont-email.me>, Alex <al...@rivadpm.com> wrote:
>On 21-Nov-17 12:23, Anton Ertl wrote:
>> Alex <al...@rivadpm.com> writes:
>>> It's possible to write Forth systems that not only avoid state smartness
>>> and the problems, however slight, it brings, but that are also simpler
>>> to maintain and that are easily and correctly extended.
>>>
>>> http://amforth.sourceforge.net/Recognizers.html
>>
>> While recognizers help avoid a common use pattern of STATE-smart words
>> (code involving parsing words that is copy-paste-able between
>> interpreted and compiled code), the case being discussed here does not
>> fit that use pattern and recognizers don't help here.
>
>I know that; I was responding to
>
>---
>State-smartness is a tool like any other. If you get
>yourself into trouble through ignorance, shall you condemn the tool
>and expend resources avoiding it - or do you learn from your mistake
>and thereafter use the tool appropriately?

I think that Anton gave a pretty convincing argument that in the
end this attitude kills the chance of having the kind of plug in
and forget libraries we see in other languages.
Where most documentation exists of ( d n -- n ) there is no
culture of documentation caveat's to the point that A can build
on B who builds on C etc.

>---
>
>As a big fan of recognisers (is it recognisers or recognizers? I have
>been very inconsistent with my spelling) I think it's worth pointing out
>that there is really no good reason to continue to tolerate the darker
>corners of Forth when there's really no reason to do so.

I'm not a fan, because it comes with too many caveats, do's and dont's.
The reason is that they are in the base system, where a user cannot
avoid them.
This kind of extension are acceptable in an end user program.
My disassembler/assembler (ciasdis) uses ":" as a prefix to introduce
labels. ciasdis comes with an elaborate manual on itself, and
as long as a user follows it, he should be okay.
If an advanced user thinks he 1) can put `` [ postpone mythingy ] ''
in the middle of an assembler source, I couldn't tell what happens.
In that case he 1) can, because probably he has done the research.

>
>>
>> What Hugh Aguilar is trying to achieve is to inline the code for the
>> field. A better approach is to use a Forth system that inlines such
>> small words automatically. And, e.g., VFX and iForth do that.
>
>Premature optimisation (or is it optimization?). Hugh would be better
>served by better understanding algorithmic efficiencies; q.v. use of
>splay trees for dictionaries.
>
>--
>Alex

1) Genderless pronoun, as in "los hermanos".

[Anton Ertl]

alb...@cherry.spenarnc.xs4all.nl (Albert van der Horst) writes:
>In article <2017Nov2...@mips.complang.tuwien.ac.at>,
>Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
><SNIP>
>>
>>Not using libraries is actually Chuck Moore's position. Ironically,
>>Chuck Moore eliminated STATE from his Forths, certainly from cmForth,
>>but from what I read here, also from polyForth. So he apparently did
>>not consider STATE-smartness a legitimate tool, at least after a
>>while.
>
>Au contraire. Chuck replace the interpretation/compilation state

The thing to say when you agree is "Indeed!", not "Au contraire".

>with colors.

He also replaced it earlier, in polyForth and cmForth (no colours
there). He never returned to STATE.

>But who is introducing postpone state between [[ and ]]? Not me.

You can implement ]] ... [[ as a separate loop (as PolyForth does for
"]"), or inside the existing outer interpreter loop, in a way similar
to what standard Forth ] and [ do. Both approaches have been used.

You cannot use STATE for determining whether text interpretation is
inside ]] ... [[, because STATE only can represent compilation state
(every non-zero value) and interpretation state (0). Also, because
there is no double-immediacy (apart from, in a way, [[), there would
be no way to write POSTPONE-state-smart words anyway. So there is no
need to be able to determine whether one is inside ]] ... [[.

>In response I looked up where I use STATE in the ciforth library.
>I came across this example which uses STATE where I'm interested to
>hear your comment. It is used to allow LOOPs during interpretation
>
>
>: LOOP POSTPONE LOOP POSTPONE T[ ; IMMEDIATE
>: DO T] POSTPONE DO ; IMMEDIATE
>
>\ Start compiling at temporary place : return START and STATE.
>: T] STATE @ 0= IF SWAP-DP HERE THEN STATE @ ] ;
>\ Execute code at START dropping STATE, restore dictionary.
>: T[ 0= IF POSTPONE (;) SWAP-DP POSTPONE [ >R THEN ; IMMEDIATE
>
> : SWAP-DP DP @ FAR-DP @ DP ! FAR-DP ! ;
>
>
>Are their pitfalls of doing something like that?

If you have something like

: generate-something ( n -- )
... POSTPONE do ... POSTPONE loop ... ;

that is later used as

: bla
... [ 5 generate-something ] ... ;

I think this will not do on your system what is intended. You can
find such a usage (but without DO LOOP in the word MARK in gc.fs in
<https://www.complang.tuwien.ac.at/forth/garbage-collection.zip>).

[hughag...@gmail.com]

On Tuesday, November 21, 2017 at 11:09:31 AM UTC-7, Anton Ertl wrote:
> You cannot use STATE for determining whether text interpretation is
> inside ]] ... [[, because STATE only can represent compilation state
> (every non-zero value) and interpretation state (0). Also, because
> there is no double-immediacy (apart from, in a way, [[), there would
> be no way to write POSTPONE-state-smart words anyway. So there is no
> need to be able to determine whether one is inside ]] ... [[.

Another failing of ANS-Forth was that STATE only has two values.
It should have three:
0: compiling inside of a colon word
1: interpreting inside of a colon word (inside of the [ ] brackets)
-1: interpreting outside of a colon word

As for your ]] ... [[ I haven't examined that. I have my early-binding MACRO: that I use.
My early-binding MACRO: depends upon the disambiguifiers to work. I doubt that your ]] ... [[ is going to work without disambiguifiers too.
I have no interest in ]] ... [[ because you have got the brackets backwards. That is so confusing! Much better would be: [[ ... ]] instead.

[Matthias Trute]

> is it recognisers or recognizers?

Depends on the spell checker I used. the British
English one insists in the S, the US American
favors the Z. IIRC Oscar Wilde made wise statement
long ago...

Honestly: I'm not a native English speaker, I leave it
to them to decide.

Matthias

[Alex]

On 21-Nov-17 18:19, hughag...@gmail.com wrote:
> On Tuesday, November 21, 2017 at 11:09:31 AM UTC-7, Anton Ertl
> wrote:
>> You cannot use STATE for determining whether text interpretation
>> is inside ]] ... [[, because STATE only can represent compilation
>> state (every non-zero value) and interpretation state (0). Also,
>> because there is no double-immediacy (apart from, in a way, [[),
>> there would be no way to write POSTPONE-state-smart words anyway.
>> So there is no need to be able to determine whether one is inside
>> ]] ... [[.
>
> Another failing of ANS-Forth was that STATE only has two values. It
> should have three: 0: compiling inside of a colon word 1:
> interpreting inside of a colon word (inside of the [ ] brackets) -1:
> interpreting outside of a colon word

Why?

(The example below uses a STATE of -2 to indicate a POSTPONE "state",
but it's important to note that words cannot detect that state when they
are being postponed. It's there to allow the recognizer a simple index
into a 3 element table; 0 for interpret action, -1 for compile, and -2
for postpone. It could just as easily be a special internal flag.)

>
> As for your ]] ... [[ I haven't examined that. I have my
> early-binding MACRO: that I use. My early-binding MACRO: depends upon
> the disambiguifiers to work. I doubt that your ]] ... [[ is going to
> work without disambiguifiers too. I have no interest in ]] ... [[

It doesn't require "disambiguifiers" to work. I can demonstrate that it
does with running code, not speculation.

\ ------------------ ]] macros [[ ---------------------------------

:noname -600 ( throw_macroerr ) throw ;
' noop
' noop
dt-token value dt:macro ( -- )

: rec:macro ( addr u -- dt:macro | dt:null )
s" [[" str= if \ switch into compile state
get-recognizers 1- set-recognizers drop
]
dt:macro
else dt:null then ;

: ]] ( -- ) \ switch into postpone state
['] rec:macro get-recognizers 1+ set-recognizers
-2 state ! ; immediate

That's all that's required to support it in my Forth when using recognizers.

> because you have got the brackets backwards. That is so confusing!
> Much better would be: [[ ... ]] instead.

You're thinking like a C programmer.

] moves from interpret to compile. ]] moves from compile to postpone.
[[ moves from postpone to compile. [ moves from compile to interpret.

[[ ]] would not have the same symmetry.

--
Alex

[a...@littlepinkcloud.invalid]

Matthias Trute <matthia...@gmail.com> wrote:
>> is it recognisers or recognizers?
>
> Depends on the spell checker I used. the British English one insists
> in the S, the US American favors the Z.

Your so-called "British spell checker" is wrong. "-ize" is the
correct spelling and "is favoured on etymological grounds, in that
-ize corresponds more closely to the Greek root, -izo, of most -ize
verbs." [1] "-ize" has the added advantage of being the same as
American English, so can be use without fear of error.

Andrew.


[1] https://en.wikipedia.org/wiki/Oxford_spelling

[Alex]

I work for an American company and use a US keyboard with a US spell
checker for work. I'm now very accustomed to the different key placement
of the @ # and | from a UK keyboard, which I struggle to use to be
frank. The only downside is having to use the ALT key to get £
(alt-0163) and € (alt-0128).

But as to spelling; I prefer the S versions (recognise) and our liberal
use of U as in colour.

--
Alex

[Alex]

Zee my other pozt. Perzonally, I prefer the S.

--
Alex

[Rod Pemberton]

I'm a native American English speaker. I use "recognize".

Maybe, this helps?
http://grammarist.com/spelling/recognize-vs-recognise/


Rod Pemberton

--
Does the increase in pedestrian deaths correspond with the increase in
Millennials?

[Albert van der Horst]

In article <2017Nov2...@mips.complang.tuwien.ac.at>,
Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
<SNIP>
>

>If you have something like
>
>: generate-something ( n -- )
> ... POSTPONE do ... POSTPONE loop ... ;
>
>that is later used as
>
>: bla
> ... [ 5 generate-something ] ... ;
>
>I think this will not do on your system what is intended. You can
>find such a usage (but without DO LOOP in the word MARK in gc.fs in
><https://www.complang.tuwien.ac.at/forth/garbage-collection.zip>).

"Your system" may suggest a misconception about how I go about a tools
like -scripting-. This is how I would phrase it.
"
I think this will not do what is intended with "-scripting- wanted".
"
In trying to run gc.zip I would not even consider loading this tool.
In fact in such a case I would try to find the minimum of extra words
to load which allows me to compile on ciforth. I'm very aware of
the potential risk of unneeded extensions to a standard system.
This is different from having a fat system that is intended to run
a great number of programs without further attention by maybe
naive users.

I will study gc.zip in this context, but gc.fs doesn't contain an
example, it is a small 4 line loader for "gc4image.fs".

>

>- anton
>--
>M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
>comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
> New standard: http://www.forth200x.org/forth200x.html
> EuroForth 2017: http://euro.theforth.net/

[Gerry Jackson]

On 20/11/2017 14:19, Alex wrote:
> It may be that we find some way to coalesce wordlists and recognizers. I
> have played with such an idea in my head, but have not yet implemented
> it in code. It needs a different way of thinking about the parsing &
> recognising processes in Forth, and how and what it means to identify a
> token.
>
> Wordlists are dynamically extensible (by adding : CREATE VALUE
> definitions and so on that can have various outputs when executed or
> compiled), but are based off a single fixed parse (i.e. they are all
> whitespace delimited lookups).
>
> On the other hand, recognisers as currently defined permit dynamic
> parsing (anything goes) but generally produce a fixed output (for
> instance, a cell sized number). That is true of all Forths btw; they all
> have at least a number recogniser, although it may not be called that.

Surely the recogniser (I automatically spell it and all -ise words with
an 's') proposal permits the mixing up of wordlists and recognisers. For
example a dictionary search recogniser can be written that emulates the
dictionary search of standard Forth and can be placed anywhere in the
recogniser stack. The built in dictionary search can then be disabled by
executing
0 SET-ORDER
permitting things to be recognised before any wordlist is searched.

Given that, ISTM that it is better that an agreed way of coalescing
wordlists and recognisers is developed before recognisers are
standardised to avoid a future problem where different systems take
different approaches and standardisation of that becomes impossible.

The concept of recognisers is more general, the standard dictionary
search mechanism is a particular sort of recogniser.

--
Gerry

[Helmar Wodtke]

Why not do different direction? That is assign *one* recognizer to one wordlist/dictionary. Then there are two words to maintain the things:

SET-RECOGNIZER \ xt wid -- \\ xt: p c -- flag
GET-RECOGNIZER \ wid -- xt

The recognizer takes a string and returns a flag if it did the work or not.
It's actived if a searched word was not found in a active wordlist/vocabulary by the "outer interpreter".
You can have multiple recognizers by manipulation of the search order.
A recognizer in this simple schematic would always be STATE-smart. But as part of the system it's not a problem.

I personally would prefer not to have another "stack" to handle...

Regards,
-Helmar


>
> --
> Gerry

[Alex]

On 22-Nov-17 08:27, Gerry Jackson wrote:
> On 20/11/2017 14:19, Alex wrote:
>> It may be that we find some way to coalesce wordlists and recognizers.
>> I have played with such an idea in my head, but have not yet
>> implemented it in code. It needs a different way of thinking about the
>> parsing & recognising processes in Forth, and how and what it means to
>> identify a token.
>>
>> Wordlists are dynamically extensible (by adding : CREATE VALUE
>> definitions and so on that can have various outputs when executed or
>> compiled), but are based off a single fixed parse (i.e. they are all
>> whitespace delimited lookups).
>>
>> On the other hand, recognisers as currently defined permit dynamic
>> parsing (anything goes) but generally produce a fixed output (for
>> instance, a cell sized number). That is true of all Forths btw; they
>> all have at least a number recogniser, although it may not be called
>> that.
>
> Surely the recogniser (I automatically spell it and all -ise words with
> an 's') proposal permits the mixing up of wordlists and recognisers. For
> example a dictionary search recogniser can be written that emulates the
> dictionary search of standard Forth and can be placed anywhere in the
> recogniser stack. The built in dictionary search can then be disabled by
> executing
>     0 SET-ORDER
> permitting things to be recognised before any wordlist is searched.

They return different things. One is an XT; it can be EXECUTEd or
COMPILE,d. The other is (eventually) a cell or cells that represent,
normally, a literal.

The action taken from the uniform token found for these might differ;
for instance, a recognizer token for a name (for example) uses the table
(interpret, compile, postpone):

: plit postpone literal ;

:noname name>interpret execute ;
:noname name>compile execute ;
:noname name>compile swap plit compile, ;
dt-token value dt:name ( nt -- )

whereas a number recognizer token is

' noop
:noname plit ;
:noname plit postpone plit ;
dt-token value dt:single ( n -- )

Recognizers return a "something-on-the-stack dt-token" and not an XT (or
NT if you prefer the output from FIND-NAME). There's only one thing you
can do with them; STATE @ CELLS- @ to get an "dt-token-XT" you can
execute. But that needs to be executed now, because there are a variable
number of cells on the stack that only this "dt-token-XT" knows about.

>
> Given that, ISTM that it is better that an agreed way of coalescing
> wordlists and recognisers is developed before recognisers are
> standardised to avoid a future problem where different systems take
> different approaches and standardisation of that becomes impossible.

Yes. I'm not in favour of standardization as yet, although recognizers
(which I have implemented) are an incredibly simple and flexible facility.

>
> The concept of recognisers is more general, the standard dictionary
> search mechanism is a particular sort of recogniser.
>

This goes back to a discussion that (if I remember correctly) Jenny
Brien and Anton Ertl had about numbers being represented as first class
citizens. That is, the number 1234 is the equivalent of

1234 constant 1234

and it can be ticked, EXECUTEd and COMPILE,d etc.

--
Alex

[minf...@arcor.de]

I can imagine that the scheme can be handy for compiler writers.
But for the rest?

And it "feels" convoluted, against Forth's keep it simple spirit.

[Alex]

On 22-Nov-17 16:42, minf...@arcor.de wrote:
> I can imagine that the scheme can be handy for compiler writers.
> But for the rest?
>
> And it "feels" convoluted, against Forth's keep it simple spirit.

Which part feels convoluted?

Certainly, the compiler part of implementing a recognizer takes around
30 lines of code, or about 70 to 80 tokens or so.

Writing a recognizer is pretty simple for the user[*]. For example,
here's a recognizer that can be used by any system that provides them,
for single character numbers of the form 'X'.

: q? ( addr -- f ) \ points to ' ?
c@ $27 = ;
: quoted? ( str k -- flag ) \ check for '.' strings
3 = if dup q? swap 2 + q? and else drop 0 then ;

: rec:quot ( addr u -- n dt:num | dt:null ) \ 'x' type numbers
2dup quoted? if
drop 1+ c@ dt:single
else
2drop dt:null
then ;

' rec:quot get-recognizers 1+ set-recognizers

That's it. Simple. (As an aside, this recognizer uses the number prefix
recognizer for hex strings like $x.)

About the only downside is the inefficiency of repeatedly scanning
tokens when the parse fails (as indicated by DT:NULL). It could be
mitigated, but it's only an overhead while scanning source code.

One solution is a full blown dynamically extensible character token
parsing scheme. Now that *would* be complicated.


[*] You and others here keep talking about "users" of Forth or in this
instance "the rest" as though they are in some way not in the same Venn
intersect as "compiler writers"; they seem to you to be a quite
different species. What is the difference?

--
Alex

[minf...@arcor.de]

Please, no offense or disrespect intended.

With that said, what is so difficult to understand that there are compiler
writers who know all the innards - and on the other side of the road there
are users who just want their job done and unfortunately have to deal with
that crazy looking compiler?

In my world the latter would be service engineers of controller boards.
They have their DSL written in Forth. Never would they tinker around with
recognizers or somesuch intratestinal stuff - it is just too dangerous.

[Alex]

On 22-Nov-17 20:29, minf...@arcor.de wrote:
> Am Mittwoch, 22. November 2017 18:42:06 UTC+1 schrieb Alex:

>>
>>
>> [*] You and others here keep talking about "users" of Forth or in this
>> instance "the rest" as though they are in some way not in the same Venn
>> intersect as "compiler writers"; they seem to you to be a quite
>> different species. What is the difference?
>>
>
> Please, no offense or disrespect intended.

I didn't take any, don't worry.

>
> With that said, what is so difficult to understand that there are compiler
> writers who know all the innards - and on the other side of the road there
> are users who just want their job done and unfortunately have to deal with
> that crazy looking compiler?
>
> In my world the latter would be service engineers of controller boards.
> They have their DSL written in Forth. Never would they tinker around with
> recognizers or somesuch intratestinal stuff - it is just too dangerous.
>
>

I call those consumers.

--
Alex

[hughag...@gmail.com]

On Sunday, November 19, 2017 at 1:41:14 AM UTC-7, Elizabeth D. Rather wrote:
> On 11/18/17 12:56 PM, Coos Haak wrote:
> > Because the trick with >R and R> works in all implementations,
> > there is no need for other tricks ;-)
>
> In the 80's some systems started putting a marker on the stack in : and
> checking it in ; to detect a possible stack effect due to an incomplete
> structure (such as an IF without a THEN, which would leave an address on
> the stack). After extended discussion, ANS Forth decided to allow this
> practice by defining this 'colon-sys' as something whose size and
> content was entirely system-dependent: you don't know how many items, if
> any, there are or what they are. In effect, this blocks the programmer
> from passing data into or out of a colon definition. As Coos notes, the
> >R ... R> trick avoids the problem.

The obvious solution is to have a control-flow stack where colon IF BEGIN etc. put their data.
This leaves the data-stack available for use in the meta-compiling word. Anybody who writes meta-compiling words knows this.
This solution was well-known in 1994. The fact that this solution is legal in ANS-Forth proves that this solution was known to the ANS-Forth committee:
-------------------------------------------------------------------------
3.2.3.2 Control-flow stack
The control-flow stack is a last-in, first out list whose elements define the permissible matchings of control-flow words and the restrictions imposed on
data-stack usage during the compilation of control structures.
The elements of the control-flow stack are system-compilation data types.
The control-flow stack may, but need not, physically exist in an implementation. If it does exist, it may be, but need not be, implemented using
the data stack. The format of the control-flow stack is implementation defined. Since the control-flow stack may be implemented using the data stack, items
placed on the data stack are unavailable to a program after items are placed on the control-flow stack and remain unavailable until the control-flow stack
items are removed.
-------------------------------------------------------------------------

You created this problem by failing to standardize the control-flow stack.
You allowed the data-stack to be used for control-flow data, preventing the data-stack from being used to hold data passed into the meta-compiling word.

Now you say: "the >R ... R> trick avoids the problem."

Realistically, nobody wants to use ANS-Forth because doing so requires weird tricks to solve pointless problems introduced by an idiot (you).
This "trick" is not documented anywhere in the ANS-Forth document. Most novices, including our OP, are not going to figure this out on their own.

ANS-Forth is not "crystal clear" --- programmers don't want to: " tinker around with ... intratestinal stuff - it is just too dangerous."
If the word "intratestinal" refers to what is inside of the large intestine, then it does accurately describe what is inside of ANS-Forth

I think that your primary goal with ANS-Forth was to make sure that Forth Inc. compilers would be standard.
Forth Inc. compilers use the data-stack for control-flow data from colon IF BEGIN etc., so section 3.2.3.2. makes this legal ANS-Forth.
Your secondary goal was to get as many supporters for ANS-Forth as possible.
Intelligently written Forth systems have a separate control-flow stack, so section 3.2.3.2. also makes this legal ANS-Forth.
This is ambiguous though! Apparently ambiguity doesn't bother you at all though.

Ambiguity actually makes a language standard worthless though.
We have programs that work under some ANS-Forth compilers but don't work under other ANS-Forth compilers.
You famously said: "Just declare a dependency and forge ahead!"
This is called "vendor lock-in" --- the purpose of a standard is to avoid dependencies --- the purpose of a standard is to allow for portable programs.

> Cheers,
> Elizabeth

You're an alcoholic, aren't you?

[Elizabeth D. Rather]

On 11/22/17 7:42 AM, Alex wrote:
> On 22-Nov-17 16:42, minf...@arcor.de wrote:
>> I can imagine that the scheme can be handy for compiler writers.
>> But for the rest?
>>
>> And it "feels" convoluted, against Forth's keep it simple spirit.
>
> Which part feels convoluted?
>
> Certainly, the compiler part of implementing a recognizer takes around
> 30 lines of code, or about 70 to 80 tokens or so.
>
> Writing a recognizer is pretty simple for the user[*]. For example,
> here's a recognizer that can be used by any system that provides them,
> for single character numbers of the form 'X'.

...

> [*] You and others here keep talking about "users" of Forth or in this
> instance "the rest" as though they are in some way not in the same Venn
> intersect as "compiler writers"; they seem to you to be a quite
> different species. What is the difference?

I would define a "user" as someone who uses a pre-existing Forth
implementation to write applications, as opposed to wrestling with the
complier/system issues directly. And, yes, they are pretty different.
They're interested not in new compiler models or adding whizzy features
that some other popular languages have, but in looking for and utilizing
a system with the tools and functions needed to solve the specific
problem at hand, and then using them effectively.

Cheers,
Elizabeth

--
Elizabeth D. Rather
FORTH, Inc.
6080 Center Drive, Suite 600
Los Angeles, CA 90045
USA

[m...@iae.nl]

On Thursday, November 23, 2017 at 8:57:04 AM UTC+1, Elizabeth D. Rather wrote:
> On 11/22/17 7:42 AM, Alex wrote:
> > On 22-Nov-17 16:42, minf...@arcor.de wrote:
> >> I can imagine that the scheme can be handy for compiler writers.
> >> But for the rest?

[..]

> I would define a "user" as someone who uses a pre-existing Forth
> implementation to write applications, as opposed to wrestling with the
> complier/system issues directly. And, yes, they are pretty different.
> They're interested not in new compiler models or adding whizzy features
> that some other popular languages have, but in looking for and utilizing
> a system with the tools and functions needed to solve the specific
> problem at hand, and then using them effectively.

In my view, a compiler writer is developing tools and
functions to solve observed problems with his product
more efficiently and effectively, whilst on the lookout
for the problems his "user" (defined above) does not yet
have.

I think this makes the Venn diagram more interesting.

-marcel

[Alex]

Which is where we started; having a compiler that acts predictably.
Anton's example is about a chain of causality that end up in error
because the language specification is broken. As he says;

<quote>

In any case there are those of us who think that it is useful to be
able to use libraries. Then we need an environment where the
libraries have simple and universal interfaces, not interfaces full of
traps and caveats. And in such an environment STATE-smartness is
inappropriate, because it leads to traps and caveats. And "I have not
had any problems yet" is not good enough if you program for general
consumption.

</quote>

Whizzy features or increasing language popularity has little to do with it.

--
Alex

[minf...@arcor.de]

Yes it has. Not being able to rely on field-tested libraries is no popularity
booster. Only chronic wheel-reinventors love it....

F.ex. even the venerable FSL Needs lots of compiler adaptations to be usable.

I always thought it might help to have a minimal but complete Forth
reference implementation as Standard reference basis, It is not required to be
fast, but it should be full-featured with warnings instead of crashing.

When libraries work with the reference, they should work off the shelf
with any standard-compliant Forth.

But I am only dreaming here....
compliant

[a...@littlepinkcloud.invalid]

Alex <al...@rivadpm.com> wrote:
>
> Which is where we started; having a compiler that acts predictably.
> Anton's example is about a chain of causality that end up in error
> because the language specification is broken. As he says;
>
> <quote>
>
> In any case there are those of us who think that it is useful to be
> able to use libraries. Then we need an environment where the
> libraries have simple and universal interfaces, not interfaces full
> of traps and caveats. And in such an environment STATE-smartness is
> inappropriate, because it leads to traps and caveats. And "I have
> not had any problems yet" is not good enough if you program for
> general consumption.
>
> </quote>
>
> Whizzy features or increasing language popularity has little to do
> with it.

It has everything to do with it. The best solution to STATE-smartness
is not to write STATE-smart words. The second-best solution is, if
you really must, to write STATE-smart words in a very simple way and
to make their factors available so that no-one is ever tempted to try
to POSTPONE a STATE-smart word. This is not a language issue, it's an
education issue.

Andrew.

[m...@iae.nl]

On Thursday, November 23, 2017 at 11:15:50 AM UTC+1, minf...@arcor.de wrote:
[..]

> When libraries work with the reference, they should work off the shelf
> with any standard-compliant Forth.
>
> But I am only dreaming here....

Well, the originators of the FSL where not putting
"standard-compliant" first (or high) on their list. They were trying
to disarm preconceptions that high-quality and efficient numeric
libraries could not possibly be written in Forth.

The FSL's "preamble files" for different compilers were invented to
paper over syntactical and functional differences. (As I was
a reviewer, I know for certain that non-compatibility after applying
the fsl_util pre-load meant "no-approval.") If you found any problem
there, please report them to the compiler vendors.

I know that e.g. Forth Inc. always complained that the FSL was of
'low quality', which might well be because they never bothered to
report these problems and never submitted a SF-specific fsl_util
preamble file.

-marcel

[minf...@arcor.de]

Yes, understandable. IMO the FSL is not a library but a source code pool
using a sometimes very special syntax.

And it was never intended to compete against BLAS, GSL or such.

IMO signal and image processing could well be done with Forth, but then
many other factors come into play (code size, CPU/GPU assembler support etc)
that determine whether Forth remains a good choice at the end of the day.

[Stephen Pelc]

On Sun, 19 Nov 2017 16:30:42 -0800 (PST), dxf...@gmail.com wrote:

>Neither can DO LOOP be used in every imaginable
>situation, yet used it still is.

>State-smartness is a tool like any other. If
>you get yourself into trouble through ignorance,
>shall you condemn the tool and expend resources
>avoiding it - or do you learn from your mistake
>and thereafter use the tool appropriately?

In terms of Forth standards from 1994 onwards, current
implementations do not permit state-smart words to be
correctly implemented. It is possible. See my paper
"Special Words in Forth", which shows how to do it
properly.
http://www.complang.tuwien.ac.at/anton/euroforth/ef17/genproceedings/papers/pelc.pdf

Stephen

--
Stephen Pelc, steph...@mpeforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441
web: http://www.mpeforth.com - free VFX Forth downloads

[Alex]

On 23-Nov-17 11:52, a...@littlepinkcloud.invalid wrote:
> Alex <al...@rivadpm.com> wrote:
>>
>> Which is where we started; having a compiler that acts predictably.
>> Anton's example is about a chain of causality that end up in error
>> because the language specification is broken. As he says;
>>
>> <quote>
>>
>> In any case there are those of us who think that it is useful to be
>> able to use libraries. Then we need an environment where the
>> libraries have simple and universal interfaces, not interfaces full
>> of traps and caveats. And in such an environment STATE-smartness is
>> inappropriate, because it leads to traps and caveats. And "I have
>> not had any problems yet" is not good enough if you program for
>> general consumption.
>>
>> </quote>
>>
>> Whizzy features or increasing language popularity has little to do
>> with it.
>
> It has everything to do with it.

Why does fixing STATE smartness... oh wait

> The best solution to STATE-smartness
> is not to write STATE-smart words. The second-best solution is, if
> you really must, to write STATE-smart words in a very simple way and
> to make their factors available so that no-one is ever tempted to try
> to POSTPONE a STATE-smart word. This is not a language issue, it's an
> education issue.

Of course. Cowpox would not have been a problem if everyone had been
taught not to milk cows.

This is an argument that flies in the face of much of what we see from
even relatively experienced programmers here. Hugh's eponymous novice
package is littered with STATE. They infest the Forth standard; see IS
or the equally dreadful ACTION-OF for examples.

And we have very experienced compiler writers (to differentiate from
users) that have implemented such things as the non-standard
interpretative S". With STATE. Oh dear.

STATE is not entirely to blame here; it's the desire to parse & provide
different interpret/compilation actions that drives much of its use. So
we have, for better or for worse, many words that parse in the standard,
many that are implemented with STATE, and consequently a list of
"ambiguous when POSTPONEd" words.

It's been a language issue from the introduction of STATE. Which was a
long time ago.

--
Alex

[a...@littlepinkcloud.invalid]

LOL! Cows are useful. :-)

> This is an argument that flies in the face of much of what we see
> from even relatively experienced programmers here. Hugh's eponymous
> novice package is littered with STATE. They infest the Forth
> standard; see IS or the equally dreadful ACTION-OF for examples.

Well, yes, those are excellent examples.

> And we have very experienced compiler writers (to differentiate from
> users) that have implemented such things as the non-standard
> interpretative S". With STATE. Oh dear.

Indeed, but in that case it's understandable: S" and TO are in the
standard and require handling somehow. There's a choice between
kludging around the problem with STATE or special-casing them in the
interpreter. The latter is arguably correct. There is also the
possibility of even more elaborate compiler structures, as discussed
here at some length by Spephen pelc and others, but IMO it's an
unnecessary compilication of the classic Forth compiler.

> STATE is not entirely to blame here; it's the desire to parse & provide
> different interpret/compilation actions that drives much of its use.

Exactly so. You have put your finger on the exact issue. And, as you
can see above, if you really cannot overcome the desire to write a
STATE-smart word, provide its factors and there is no need ever to
POSTPONE it, so the problem will never arise.

It might be nice to have a way to mark a STATE-smart word so that
POSTPONE would immediately abort.

> So we have, for better or for worse, many words that parse in the
> standard, many that are implemented with STATE, and consequently a
> list of "ambiguous when POSTPONEd" words.

Yes. It's "several" rather than "many", but yes.

> It's been a language issue from the introduction of STATE. Which was a
> long time ago.

That's true. Every programming language has dusty corners, and has
ways to work around their problems. Forth is no different, of course,
and STATE-smartness has been a plague ever since ANS '94, and (in some
systems) long before that.

Andrew.

[Albert van der Horst]

In article <7182d7d6-02ba-439a...@googlegroups.com>,

I see a lot of rough edges in the FSL.

E.g. the bignum library. One goes to great lengths not to use
S" MAX-N"
The next thing is assuming that MAX-N is defined in the ENVIRONMENT?
One can define a minimal ENVIRONMENT? like
: ENVIRONMENT? 2DROP FALSE ;
and it works. No small feat I must admit, but
ENVIRONMENT? really is almost no use if one has no strings anyway.
On a 64 bit Forth with no MAX-N in the environment
the components of big nums are 1/4 of what they could be.
So it will run 4, 4^2 or 4^3 times slower.
In practice I would start editing the source immediately.
With ISO 2012 (obligatory 2-complement) one could do
-1 1 RSHIFT biggest
Less code, and useable on more systems without adaptation,
less portable.

Gforth needs a large preambule, even if gforth itself is large.
The preambule contains `cell- , but that word is already present
in gforth itself.
Having preambules kind of defeats the purpose of having a fat
forth because all the stuff is redefined anyway.
The preambule is all or nothing. So you get a lot of words
that you may not need.

With adding more programs, a preambule that works for all
of them becomes harder to make and more bulky.

I see a possible improvement by formalising the lines that
document which utilities are needed

Take elip12.seq

\ 2. Uses words 'Private:', 'Public:' and 'Reset_Search_Order'
\ to control the visibility of internal code. ( see fsl_util file )

could be replaced by

\ 2. To control the visibility of internal code. ( see fsl_util file ) :
WANT Private: Public: Reset_Search_Order

Now an automated tool can add those definitions to the source,
without loading unneeded, possibly conflicting stuff.
Even without the tool the names stand out better.
It is sufficient to add
: WANT POSTPONE \ ; IMMEDIATE
to preambules that have no such tool and one can start slowly
improving.
We get the hard and fast, and easily testable rule, that
after WANT there are standard words or words with a FSL
controled definition.

[I have experience with such a mechanism on ciforth and I
can tell you, it works nicely. It solves a similar problem
that FSL faces to get the code for 16/32/64 , windows32,
MSDOS, linux 64 etc. at an even level for the actual
application code.]

Groetjes Albert

[hughag...@gmail.com]

On Thursday, November 23, 2017 at 7:01:38 AM UTC-7, Stephen Pelc wrote:
> On Sun, 19 Nov 2017 16:30:42 -0800 (PST), dxf...@gmail.com wrote:
>
> >Neither can DO LOOP be used in every imaginable
> >situation, yet used it still is.
> >State-smartness is a tool like any other. If
> >you get yourself into trouble through ignorance,
> >shall you condemn the tool and expend resources
> >avoiding it - or do you learn from your mistake
> >and thereafter use the tool appropriately?
>
> In terms of Forth standards from 1994 onwards, current
> implementations do not permit state-smart words to be
> correctly implemented. It is possible. See my paper
> "Special Words in Forth", which shows how to do it
> properly.
> http://www.complang.tuwien.ac.at/anton/euroforth/ef17/genproceedings/papers/pelc.pdf
>
> Stephen

I read through that article. A lot of it is borrowed from me. For example:
"IMMEDIATE? Xt -- flag ; return true if the word is immediate "
I have repeatedly said that this was necessary!

I don't think there is anything of value in Stephen Pelc's article.
He wants to put all of the complexity inside of the COMPILE, word.
The COMPILE, word can only be modified by the compiler-writer however, not by the Forth programmer.
He is trying to maintain a monopoly on extensibility of Forth, which will result in vendor lock-in for programs.
He is trying to prevent the Forth programmer from extending a standard Forth system in a standard way.

I haven't forgotten that Stephen Pelc is a liar. I don't think anybody else should forget either:

On Tuesday, August 1, 2017 at 2:27:41 AM UTC-7, Stephen Pelc wrote:
> On Tue, 18 Jul 2017 21:28:18 -0700 (PDT), hughag...@gmail.com
> wrote:
>
> >My complaint against Stephen Pelc is that he uses dishonest business practi=
> >ces. He supports ANS-Forth and Forth-200x for the purpose of making all For=
> >th programmers look stupid. He doesn't actually use ANS-Forth himself. He r=
> >outinely provides vendor-specific code in VFX, even when it is easy to writ=
> >e the code in ANS-Forth, for the purpose of trapping his customers in vendo=
> >r lock-in. A good example is SYNONYM --- I can write this in ANS-Forth usin=
> >g my disambiguifiers --- he insists that this is impossible to write in ANS=
> >-Forth --- he refuses to admit that the disambiguifiers exist because he wa=
> >nts ANS-Forth's FIND to behave differently in every ANS-Forth compiler.
>
> Hugh's wonderful disambiguifiers do NOT do what the great Hugh thinks
> they do. What the great Hugh has done is to redefine a large number of
> words so that they behave in a very restricted way to support the
> great Hugh's version of Forth. Hugh's SYNONYM is not portable ANS
> Forth unless you use Hugh's Forth. Bah, humbug. Another emperor
> with no clothes.
>
> Stephen

Stephen Pelc is lying. My disambiguifers do make all of the words such as IF etc. behave consistently between ANS-Forth implementations.
They are called "disambiguifiers" because the ambiguity (different behavior on different ANS-Forth implementations) is fixed.
They do not behave in a "very restrictive way" and there is no "Hugh's version of Forth" (the disambiguifiers work on VFX, SwiftForth, GFORTH, etc.).

Stephen Pelc is lying because he wants FIND to be ambiguous (different behavior on different ANS-Forth implementations) so that it will be impossible
to write an outer-interpreter in ANS-Forth that is portable between different ANS-Forth implementations. Stephen Pelc wants vendor lock-in.
I have said repeatedly that LITERAL needs to be vectored for a cross-compiler to support literal numbers in TARG mode.
ANS-Forth failed to make LITERAL (and DLITERAL FLITERAL etc.) vectored. Because of this failing, the cross-compiler writer needs his own outer-interpreter.
ANS-Forth failed to support a FIND that is unambiguous however, so for a long time I didn't think a cross-compiler could be written in ANS-Forth.
Anton Ertl invented the disambiguifiers however, and I realized that this was the key to working around the ambiguous FIND in ANS-Forth.
Stephen Pelc is lying about the disambiguifiers because he wants to prevent ANS-Forth programmers from writing cross-compilers in competition with MPE.
He want cross-compilers to be vendor specific.

Anton Ertl lost his job as Forth-200x chair-person. Stephen Pelc is now the Forth-200x chair-person.
Most likely, Anton Ertl lost his job because he invented the disambiguifiers and he made this public on comp.lang.forth where I learned about it.
Bernd Paysan uses disambiguifiers in his MiniOOF program. He is another one who has zero chance of being promoted to the chair-person position.
Stephen Pelc is the worst possible candidate however, because his goal is to cripple the standard so that VFX will look good by comparison.
Stephen Pelc is motivated entirely by his goal of making MPE profitable --- he considers all other Forth programmers to be his enemy in the Forth arena.

It is very ironic that Stephen Pelc accuses me of being an: "emperor with no clothes."
As chair-person for the Forth-200x committee, Stephen Pelc has declared himself to be the emperor of Forth --- and he is a liar, so he has "no clothes."
I just write ANS-Forth code that works and is portable --- this is what the ANS-Forth cult hates and fears the most --- I'm not being an "emperor" though.

Forth-200x is deeply corrupt. Peter Knaggs is Stephen Pelc's employee, yet he is on the Forth-200x committee.
This is a blatant violation of the rule that no company has more than one vote on the committee.
I recommend that everybody ignore Forth-200x --- this is "astroturf" (a corporate project faked up to look like a grassroots project).

[dxf...@gmail.com]

On Wednesday, November 22, 2017 at 4:56:40 AM UTC+11, Albert van der Horst wrote:
> In article <ov1c81$je8$1...@dont-email.me>, Alex <al...@rivadpm.com> wrote:
> >On 21-Nov-17 12:23, Anton Ertl wrote:
> >> Alex <al...@rivadpm.com> writes:
> >>> It's possible to write Forth systems that not only avoid state smartness
> >>> and the problems, however slight, it brings, but that are also simpler
> >>> to maintain and that are easily and correctly extended.
> >>>
> >>> http://amforth.sourceforge.net/Recognizers.html
> >>
> >> While recognizers help avoid a common use pattern of STATE-smart words
> >> (code involving parsing words that is copy-paste-able between
> >> interpreted and compiled code), the case being discussed here does not
> >> fit that use pattern and recognizers don't help here.
> >
> >I know that; I was responding to
> >
> >---

> >State-smartness is a tool like any other. If you get
> >yourself into trouble through ignorance, shall you condemn the tool
> >and expend resources avoiding it - or do you learn from your mistake
> >and thereafter use the tool appropriately?
>

> I think that Anton gave a pretty convincing argument that in the
> end this attitude kills the chance of having the kind of plug in
> and forget libraries we see in other languages.
> Where most documentation exists of ( d n -- n ) there is no
> culture of documentation caveat's to the point that A can build
> on B who builds on C etc.

Those who bemoan Forth doesn't have what other
languages do have had five decades to turn it
around. Nobody has been prevented from writing standard programs or libraries. Forth's danger
lies not in what a programmer may do but in
becoming a designer who has lost all interest
in programming and now designs for some mythical
user who doesn't even exist.

[minf...@arcor.de]

You seem to nurture your own myth about Forthers. ;-)

I can assure you I know VERY excellent smart engineers
who did VERY practical jobs.

[Gerry Jackson]

What's stopping a user defined dictionary search recogniser such as I
postulated from actually executing or compiling the found word and
returning some dummy "something-on-the-stack dt-token" that causes a
noop. That would be a bodge with the current recogniser proposal and not
nice - but is that impossible?.

Conceptually at a higher level both recognisers and wordlist searches
parse some text and do something with it and it should be possible to
devise a scheme that encompasses both.

>
>>
>> Given that, ISTM that it is better that an agreed way of coalescing
>> wordlists and recognisers is developed before recognisers are
>> standardised to avoid a future problem where different systems take
>> different approaches and standardisation of that becomes impossible.
>
> Yes. I'm not in favour of standardization as yet, although recognizers
> (which I have implemented) are an incredibly simple and flexible facility.
>
>>
>> The concept of recognisers is more general, the standard dictionary
>> search mechanism is a particular sort of recogniser.
>>
>
> This goes back to a discussion that (if I remember correctly) Jenny
> Brien and Anton Ertl had about numbers being represented as first class
> citizens. That is, the number 1234 is the equivalent of
>
> 1234 constant 1234
>
> and it can be ticked, EXECUTEd and COMPILE,d etc.
>

I don't remember that discussion but don't see that's particularly
relevant.
--
Gerry

[hughag...@gmail.com]

On Friday, November 24, 2017 at 7:20:15 PM UTC-7, hughag...@gmail.com wrote:
> On Thursday, November 23, 2017 at 7:01:38 AM UTC-7, Stephen Pelc wrote:
> > In terms of Forth standards from 1994 onwards, current
> > implementations do not permit state-smart words to be
> > correctly implemented. It is possible. See my paper
> > "Special Words in Forth", which shows how to do it
> > properly.
> > http://www.complang.tuwien.ac.at/anton/euroforth/ef17/genproceedings/papers/pelc.pdf
> >
> > Stephen
>
> I read through that article. A lot of it is borrowed from me. For example:
> "IMMEDIATE? Xt -- flag ; return true if the word is immediate "
> I have repeatedly said that this was necessary!

Also useful is:
VOC? xt -- voc# ; return the voc# for the word

In my experience, when a cross-compiler goes haywire, it is almost always because an xt is being used in the wrong place.
This is very difficult to debug because the behavior of the system becomes extremely bizarre and you don't know where it went off the rails.
I wrote some assertion code to check every xt before it was used to make sure that it was the expected voc# and the expected immediate or not.
After I did this, debugging became very easy because I immediately found out when there was a bug, before the system went off the rails and became bizarre.
At this time, I quickly got rid of all the bugs and began making rapid progress. :-)

It is really not difficult to fix the standard. Forth-83 was screwed up, but I fixed it pretty easily.
I had all of the UR/Forth-specific code (mostly assembly-language) in one small file. This small file contained all the needed upgrades from Forth-83.
ANS-Forth is more screwed up than Forth-83 was. By going back to Forth-83 and then adding a few upgrades it is possible to obtain a viable Forth standard.
I would call this Traditional Forth because it is pretty close to traditional Forth with just a few upgrades to make it usable for cross-compilation.

I also have my Straight Forth design that is more ambitious, with some non-traditional aspects.
For example, I have fixed-point numbers that are 64-bit and assume unity to be 2^32.
I think Straight Forth is a good design. It is not just for cross-compilation, but is also for numerical programming and text processing.
In the meantime, Traditional Forth is a very easy upgrade from Forth-83 that would be useful for cross-compilation.

I think that my disambiguifiers can totally kill ANS-Forth.
If Emperor Pelc accepted the disambiguifiers, this would hurt the credibility of ANS-Forth because this would shine a spotlight on FIND being ambiguous.
ANS-Forth would survive though. It is just a bug-fix, which is not that big of a problem --- somewhat late in the year 2017 --- but just a bug-fix.
If Emperor Pelc becomes a blatant liar and says that the disambiguifiers don't work, this utterly destroys the credibility of ANS-Forth and Forth-200x.
Nobody will ever believe anything that he says on any subject.

----------------------------------------------------------------------------

The control-flow stack may, but need not, physically exist in an implementation. If it does exist, it may be, but need not be, implemented
using the data stack. The format of the control-flow stack is implementation defined. Since the control-flow stack may be implemented using the data stack,
items placed on the data stack are unavailable to a program after items are placed on the control-flow stack and remain unavailable until the control-flow
stack items are removed.

----------------------------------------------------------------------------
Most likely, Elizabeth Rather was drunk when she wrote this.
This looks like the pseudo-intellectual blather that drunkards spew when they are trying to fake up expertise on a subject they know nothing about.
Continuing to defend ANS-Forth in the year 2017 is absurd. Stephen Pelc is an emperor with no clothes! Continuing to support ANS-Forth will ruin him.

[Gerry Jackson]

I thought I'd replied to this a few days ago but it seems to have got
lost - so I'm trying again.

On 22/11/2017 15:11, Alex wrote:

What's stopping a user defined dictionary search recogniser such as I
postulated from actually executing or compiling the found word and

returning a dummy "something-on-the-stack dt-token" that causes a noop.

That would be a bodge with the current recogniser proposal and not nice
- but is that impossible?.

Conceptually at a higher level both recognisers and wordlist searches
parse some text and do something with it and it should be possible to
devise a scheme that encompasses both.

>
>>

>> Given that, ISTM that it is better that an agreed way of coalescing
wordlists and recognisers is developed before recognisers are
standardised to avoid a future problem where different systems take
different approaches and standardisation of that becomes impossible.
>
> Yes. I'm not in favour of standardization as yet, although
recognizers (which I have implemented) are an incredibly simple and
flexible facility.
>
>>
>> The concept of recognisers is more general, the standard dictionary
search mechanism is a particular sort of recogniser.
>>
>
> This goes back to a discussion that (if I remember correctly) Jenny
Brien and Anton Ertl had about numbers being represented as first class
citizens. That is, the number 1234 is the equivalent of
>
> 1234 constant 1234
>
> and it can be ticked, EXECUTEd and COMPILE,d etc.
>

[Alex]

On 26-Nov-17 10:45, Gerry Jackson wrote:

Excuse the formatting, but your message seems to have disrupted the indents.

Yes, but it's not clear what shape this scheme might take without
completely redesigning what is a quite simple (if flawed around the
edges) white space parse execute/compile cycle.

>
>>
>>>
>>> Given that, ISTM that it is better that an agreed way of
>>> coalescing
> wordlists and recognisers is developed before recognisers are
> standardised to avoid a future problem where different systems take
> different approaches and standardisation of that becomes impossible.
>>
>> Yes. I'm not in favour of standardization as yet, although
> recognizers (which I have implemented) are an incredibly simple and
> flexible facility.
>>
>>>
>>> The concept of recognisers is more general, the standard
>>> dictionary
> search mechanism is a particular sort of recogniser.
>>>
>>
>> This goes back to a discussion that (if I remember correctly) Jenny
>>
> Brien and Anton Ertl had about numbers being represented as first
> class citizens. That is, the number 1234 is the equivalent of
>>
>> 1234 constant 1234
>>
>> and it can be ticked, EXECUTEd and COMPILE,d etc.
>>
>
> I don't remember that discussion but don't see that's particularly
> relevant.

It's about the nature of the parse; there are effectively two things
that Forth recognizes, an XT type thing and a number type thing. Perhaps
the can be unified. I've not given it a huge amount of thought though.

--
Alex

[Albert van der Horst]

In article <ovenn4$58q$1...@dont-email.me>, Alex <al...@rivadpm.com> wrote:
>On 26-Nov-17 10:45, Gerry Jackson wrote:
>
>Yes, but it's not clear what shape this scheme might take without
>completely redesigning what is a quite simple (if flawed around the
>edges) white space parse execute/compile cycle.

Plus. If definitions can be marked prefix and found in the dictionary
if they match the first part of the space delimited name, some of the
rough edges disappear. Things like 'a' and "aap" can be handled by
adding just two lines to a forth system.

>--
>Alex

[Gerry Jackson]

On 26/11/2017 15:47, Alex wrote:
>> What's stopping a user defined dictionary search recogniser such as I
>>  postulated from actually executing or compiling the found word and
>> returning a dummy "something-on-the-stack dt-token" that causes a
>> noop. That would be a bodge with the current recogniser proposal and
>> not nice - but is that impossible?.
>>
>> Conceptually at a higher level both recognisers and wordlist searches
>>  parse some text and do something with it and it should be possible
>> to devise a scheme that encompasses both.
>
> Yes, but it's not clear what shape this scheme might take without
> completely redesigning what is a quite simple (if flawed around the
> edges) white space parse execute/compile cycle.
>

Perhaps the view to take is that recognisers are the general concept and
dictionary search is a type of recogniser that can be achieved as I
outlined. I don't see the white space parse as a problem as a recogniser
has access to the line of source code via SOURCE and >IN and can do what
it wants by re-parsing etc.

You seem to have more experience than most with recognisers, what do you
see as the 'flaws round the edges'? I've yet to complete my
implementation of recognisers as progress has been overtaken by other
priorities.

--
Gerry

[dxf...@gmail.com]

Good. Forth could do with some.