WL-TOOLS and ?IMMEDIATE

[Krishna Myneni]

In an earlier discussion of using TRAVERSE-WORDLIST to write tools for
introspection, e.g. WL-INFO which displays detailed information about
entries in a wordlist, I had a need for a standard way of obtaining the
immediacy (precedence) of a word from its nt. Anton Ertl had suggested
the following definition:

: IMMEDIATE? ( nt -- flag )
NAME>COMPILE ['] COMPILE, <> NIP ;

The above definition assumes that NAME>COMPILE will return

-- xt-word xt-compile,

for a non-immediate word. This approach may not work on some systems,
such as mine, in which specialized compilation behavior is needed for
certain words. In my system, NAME>COMPILE will return

-- nt-word xt-compile-name

for non-immediate words, and return

-- xt-word xt-execute

for immediate words. The latter, I believe, should be applicable to all
Forth systems which implement NAME>COMPILE . Therefore, I propose a
standard way of defining IMMEDIATE? as

: IMMEDIATE? ( nt -- flag )
NAME>COMPILE ['] EXECUTE = NIP ;

Will the above not work on standard systems for some reason which I may
have overlooked?

Krishna Myneni

[Alex McDonald]

Or you can take my approach, and just abandon the concept of immediacy.
I wrote as much on clf some time ago and annoyed at least one Forth
purist. But I do believe that immediacy is dead, and that non-default
compilation semantics (NDCS to use Stephen Pelc's ugly if accurate
shorthand) are a much more sensible and consistent way of looking at it.
IMMEDIATE then becomes "a word that makes the NDCS the interpretation
semantics". [COMPILE] is already obsolescent, and I haven't heard a
great deal of squealing over its timely death.

--
Alex

[Mark William Humphries]

Non-default compilation semantics, special semantics, dual behaviors, or whatever you want to call them, are very rare self-imposed exceptions rather than the rule. The traditional Forth approach of immediate and non-immediate words is still the simplest way to handle the common default cases.

[A. K.]

.. including state smartness

[Anton Ertl]

Krishna Myneni <krishna...@ccreweb.org> writes:
>In an earlier discussion of using TRAVERSE-WORDLIST to write tools for
>introspection, e.g. WL-INFO which displays detailed information about
>entries in a wordlist, I had a need for a standard way of obtaining the
>immediacy (precedence) of a word from its nt. Anton Ertl had suggested
>the following definition:
>
>: IMMEDIATE? ( nt -- flag )
> NAME>COMPILE ['] COMPILE, <> NIP ;
>
>The above definition assumes that NAME>COMPILE will return
>
> -- xt-word xt-compile,
>
>for a non-immediate word. This approach may not work on some systems,
>such as mine, in which specialized compilation behavior is needed for
>certain words. In my system, NAME>COMPILE will return
>
> -- nt-word xt-compile-name
>
>for non-immediate words, and return
>
> -- xt-word xt-execute
>
>for immediate words. The latter, I believe, should be applicable to all
>Forth systems which implement NAME>COMPILE .

IMO a system will encounter the least problems if

NAME>COMPILE ( nt -- xt1 xt2 )

only ever produces the xt of EXECUTE or of COMPILE, for xt2.

>Therefore, I propose a
>standard way of defining IMMEDIATE? as
>
>: IMMEDIATE? ( nt -- flag )
> NAME>COMPILE ['] EXECUTE = NIP ;

If a system satisfies the condition above, this will work just as
well. If it deviates, one of the two may be better, or one may have
to write IMMEDIATE? in a completely system-specific way.
Interestingly, Bernd Paysan changed the implementation of IMMEDIATE?
from the latter to the former in
<http://git.savannah.gnu.org/cgit/gforth.git/commit/?id=abe4e30d8c3e70900cf245572508a3cb01f52c34>,
but I see no particular reason for it.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2019: http://euro.theforth.net/

[Krishna Myneni]

On Tue, 17 Sep 2019 13:50:52 +0100, Alex McDonald wrote:

> On 17-Sep-19 13:35, Krishna Myneni wrote:

>> ... In my system, NAME>COMPILE will return

>>
>> -- nt-word xt-compile-name
>>
>> for non-immediate words, and return
>>
>> -- xt-word xt-execute
>>
>> for immediate words. The latter, I believe, should be applicable to all
>> Forth systems which implement NAME>COMPILE . Therefore, I propose a
>> standard way of defining IMMEDIATE? as
>>
>> : IMMEDIATE? ( nt -- flag )
>> NAME>COMPILE ['] EXECUTE = NIP ;
>>
>> Will the above not work on standard systems for some reason which I may
>> have overlooked?
>>

> Or you can take my approach, and just abandon the concept of immediacy.
> I wrote as much on clf some time ago and annoyed at least one Forth

> purist. But I do believe that immediacy is dead, ...

I'll consider it at a future date but it is low on my list of development
priorities. Last I checked, IMMEDIATE is still a standard word and the
concept of immediacy is not declared obsolescent in the Forth-2012
standard (unless I missed such a declaration).

So, will your system accept the above definition of IMMEDIATE? or not?

Krishna

[Krishna Myneni]

On Tue, 17 Sep 2019 15:24:39 +0000, Anton Ertl wrote:

>
> IMO a system will encounter the least problems if
>
> NAME>COMPILE ( nt -- xt1 xt2 )
>
> only ever produces the xt of EXECUTE or of COMPILE, for xt2.
>

What types of problems do you anticipate other than possibly not being
able to check for immediacy? For xt2 I think a form of COMPILE based on
the name token is more generally useful.

>>Therefore, I propose a standard way of defining IMMEDIATE? as
>>
>>: IMMEDIATE? ( nt -- flag )
>> NAME>COMPILE ['] EXECUTE = NIP ;
>
> If a system satisfies the condition above, this will work just as well.
> If it deviates, one of the two may be better, or one may have to write
> IMMEDIATE? in a completely system-specific way.
> Interestingly, Bernd Paysan changed the implementation of IMMEDIATE?
> from the latter to the former in
> <http://git.savannah.gnu.org/cgit/gforth.git/commit/?
id=abe4e30d8c3e70900cf245572508a3cb01f52c34>,
> but I see no particular reason for it.
>

It will be interesting to know if he changed it because of an anticipated
problem.

Krishna

[Krishna Myneni]

On Tue, 17 Sep 2019 13:50:52 +0100, Alex McDonald wrote:

> ... But I do believe that immediacy is dead, ....

> [COMPILE] is already obsolescent, and I haven't heard a
> great deal of squealing over its timely death.

Both COMPILE and [COMPILE] are dead because their functionality has been
replaced by POSTPONE (Forth 2012, A.6.1.2033) which checks the immediacy
of a word to decide which of the two functions to perform. How will
removing immediacy affect a word like POSTPONE ?

Krishna

[dxforth]

On Tuesday, 17 September 2019 22:50:54 UTC+10, Alex McDonald wrote:
> ...

> Or you can take my approach, and just abandon the concept of immediacy.
> I wrote as much on clf some time ago and annoyed at least one Forth
> purist. But I do believe that immediacy is dead, and that non-default
> compilation semantics (NDCS to use Stephen Pelc's ugly if accurate
> shorthand) are a much more sensible and consistent way of looking at it.
> IMMEDIATE then becomes "a word that makes the NDCS the interpretation
> semantics". [COMPILE] is already obsolescent, and I haven't heard a
> great deal of squealing over its timely death.

POSTPONE didn't get rid of IMMEDIATE or the concept of immediacy - just
made it easier to use. When it came to explaining POSTPONE, ANS did
so in terms of immediacy, knowing 'compilation semantics' used in the
formal definition would cut no ice with the average forther.

"POSTPONE replaces most of the functionality of COMPILE and [COMPILE].
[…] COMPILE was designed to be applied to non-immediate words and
[COMPILE] to immediate words."

[Alex McDonald]

While the concept of immediacy is useful, trying to identify IMMEDIATE
words post hoc is less so. About the only reason I can think of is to
textually reconstitute a word in support of things like SEE.


--
Alex

[Anton Ertl]

Nevertheless, the current implementation of Gforth has no
immediate/precedence bit, and NAME>COMPILE (and compilation semantics)
as basic concept. Like in the standard, "immediate word" is just
defined as a property of compilation semantics.

>So, will your system accept the above definition of IMMEDIATE? or not?

In Gforth it works.

[Anton Ertl]

Krishna Myneni <krishna...@ccreweb.org> writes:
>On Tue, 17 Sep 2019 13:50:52 +0100, Alex McDonald wrote:
>Both COMPILE and [COMPILE] are dead because their functionality has been
>replaced by POSTPONE (Forth 2012, A.6.1.2033) which checks the immediacy
>of a word to decide which of the two functions to perform.

No. POSTPONE /name/ appends the compilation semantics of /name/ to
the current definition.

>How will
>removing immediacy affect a word like POSTPONE ?

Not at all. You can still define it as, e.g.,:

: ?parse-name ( "name" -- c-addr u )
parse-name dup 0= -16 and throw ;

: ?find-name ( c-addr u -- nt )
find-name dup 0= -13 and throw ;

: postpone ( "name" -- )
?parse-name ?find-name name>compile
( xt1 xt2 ) swap postpone literal compile, ;

[Anton Ertl]

Alex McDonald <al...@rivadpm.com> writes:
>While the concept of immediacy is useful, trying to identify IMMEDIATE
>words post hoc is less so. About the only reason I can think of is to
>textually reconstitute a word in support of things like SEE.

And indeed this is the only occurence of IMMEDIATE? in Gforth.

The other reason is if you want to define FIND. However, Gforth does
not use IMMEDIATE? there, but equivalent code.

[Mark William Humphries]

Le mercredi 18 septembre 2019 21:58:39 UTC+8, Anton Ertl a écrit :
> Alex McDonald <al...@rivadpm.com> writes:
> >While the concept of immediacy is useful, trying to identify IMMEDIATE
> >words post hoc is less so. About the only reason I can think of is to
> >textually reconstitute a word in support of things like SEE.
>
> And indeed this is the only occurence of IMMEDIATE? in Gforth.
>
> The other reason is if you want to define FIND. However, Gforth does
> not use IMMEDIATE? there, but equivalent code.

I use it in my COMPILER

: compiler ( -- ? )
compilation lookup 0= if number lit, then;
dup immediate? if >code execute then;
>code compile, ; compiled

... and in POSTPONE (which I name "&")

: & { <name> -- }( -- ? )
compilation defined
dup immediate? if >code compile, then;
>code lit, [^] compile, call, ; directive

[none albert]

In article <qlt8hk$5j8$1...@dont-email.me>,
Alex McDonald <al...@rivadpm.com> wrote:
<SNIP>

>
>While the concept of immediacy is useful, trying to identify IMMEDIATE
>words post hoc is less so. About the only reason I can think of is to
>textually reconstitute a word in support of things like SEE.

Whether it is called immediacy or non-default compilation, immediacy is
not just useful, it is at the core of Forth, an extensible compiler.

>--
>Alex

Groetjes Albert
--
This is the first day of the end of your life.
It may not kill you, but it does make your weaker.
If you can't beat them, too bad.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

[Alex McDonald]

Yes, it does. I don't have such a definition since I have no use for it;
FIND uses an alternative called NAME>XTIMM ( nt -- 0 | xt imm-flag ).


--
Alex

[dxforth]

Less useful for ANS because it was faced with having to support dual-xt
systems. To the users of classical systems you were compiling words
(easy to fathom) rather than compilation semantics, and words were
plainly immediate or not. There's no problem with immediacy - other
than perhaps reminding us of the days when forth was simpler.

[Alex McDonald]

That immediate simplicity delivered state smartness.

--
Alex

[a...@littlepinkcloud.invalid]

Not at all: stateless Forth, pre-ANS, had IMMEDIATE.

Andrew.

[Alex McDonald]

My point.

--
Alex

[a...@littlepinkcloud.invalid]

> My point.

I read it twice, and you still seem to me to have suggested that
immediate's simplicity in some way caused state smartness. Was that
not your point? But surely that cannot be so, because immediate
doesn't even require STATE to exist.

Andrew.

[Alex McDonald]

I know immediacy doesn't require STATE to exist. But state smartness
requires immediacy, and made the complexity of state smartness possible.
Hence: immediate simplicity delivered state smartness.

It's not an important point.

--
Alex

[Ruvim]

It depends what does this flag mean: the standard notion of immediacy,
or any non default compilation semantics.


" ['] EXECUTE = " produces TRUE for immediate word (in the standard
notion), and FALSE otherwise (i.e. for other non default compilation
semantics and for default compilation semantics).

" ['] COMPILE, <> " produces TRUE for any non default compilation
semantics (including immediate in the standard notion), and FALSE for
the default compilation semantics only.


OTOH, a system may return other variant for some immediate words.

E.g. if "TO" or "IF" is implemented as immediate STATE-smart word (NB:
the standard allows such approach), NAME>COMPILE may return
( xt 'EXECUTE-COMPILING ) to accurately represent the compilation semantics.

See also: https://forth-standard.org/standard/core/FIND#reply-334


--
Ruvim

[Krishna Myneni]

On Wed, 18 Sep 2019 18:08:06 +0100, Alex McDonald wrote:

> On 17-Sep-19 23:51, Krishna Myneni wrote:

>>>> : IMMEDIATE? ( nt -- flag )
>>>> NAME>COMPILE ['] EXECUTE = NIP ;
>>>>

...

>>
>> So, will your system accept the above definition of IMMEDIATE? or not?
>
> Yes, it does. I don't have such a definition since I have no use for it;
> FIND uses an alternative called NAME>XTIMM ( nt -- 0 | xt imm-flag ).

Is it not useful to know when writing a word definition whether or not
you are using an immediate word inside the definition, particularly for
some word from a library?

IMMEDIATE? together with TRAVERSE-WORDLIST and GET-ORDER provides a way
of listing all of the immediate words in the search order. Whether or not
it is helpful for writing and debugging code remains to be seen, because
we have not had the capability before TRAVERSE-WORDLIST of listing only
the immediate words.

Krishna

[Krishna Myneni]

On Wed, 18 Sep 2019 12:54:06 +0000, Anton Ertl wrote:

> Krishna Myneni <krishna...@ccreweb.org> writes:
>>On Tue, 17 Sep 2019 13:50:52 +0100, Alex McDonald wrote:
>>Both COMPILE and [COMPILE] are dead because their functionality has been
>>replaced by POSTPONE (Forth 2012, A.6.1.2033) which checks the immediacy
>>of a word to decide which of the two functions to perform.
>
> No. POSTPONE /name/ appends the compilation semantics of /name/ to the
> current definition.
>

To be more specific, in systems which use a precedence/immediacy bit and
don't provide dual tokens for execution and compilation, POSTPONE needs
to query the immediacy bit for a word to determine the compilation
semantics. This is still well within the standard, since the standard
does not demand a dual token system. The presence of the immediacy bit
usually means that compilation semantics are computed instead of simply
retrieved.

Krishna

[Krishna Myneni]

On Wed, 18 Sep 2019 23:28:18 +0000, Krishna Myneni wrote:

...

> IMMEDIATE? together with TRAVERSE-WORDLIST and GET-ORDER provides a way
> of listing all of the immediate words in the search order. Whether or
> not it is helpful for writing and debugging code remains to be seen,
> because we have not had the capability before TRAVERSE-WORDLIST of
> listing only the immediate words.
>

With the introduction of NAME>COMPILE in kForth-32 (as of commit
6ceee33), I have added a word in the program wl-tools.4th to show all of
the immediate words in a wordlist. The example below shows all of the
immediate words in the kForth-32 Forth wordlist.

----
Ready!
include wl-tools
ok
forth-wordlist wl-immediate

[CHAR]
."
S"
C"
.(
(
#!
\
ABORT"
RECURSE
ENDOF
OF
ENDCASE
CASE
THEN
ELSE
IF
AGAIN
UNTIL
REPEAT
WHILE
BEGIN
LEAVE
?DO
DO
FLITERAL
SLITERAL
2LITERAL
LITERAL
;
]
[
POSTPONE
[UNDEFINED]
[DEFINED]
[']
ok
----

The program wl-tools.4th may be viewed at the link below. Some notes:

1) The word WL-INFO is now properly implemented and will show the proper
xts, body addresses, and immediacies for multiple definitions with the
same name.

2) The word WL-CREATED is not portable to other standard systems since
>BODY may not work as assumed here.

3) Other words should be portable to Forth-2012 compatible systems.

4) The wl-tools program is still in its infancy.

https://github.com/mynenik/kForth-32/blob/master/forth-src/wl-tools.4th

KM

[dxforth]

On Thursday, 19 September 2019 04:42:33 UTC+10, Alex McDonald wrote:
> ...

> That immediate simplicity delivered state smartness.

With which Forth Inc and others appear to have no problem. Most
implementation examples one sees e.g. quotations are written in
terms of state smartness because it's common and well-understood.

[a...@littlepinkcloud.invalid]

I know this isn't your main point, but I'm not sure what about
quotations might use state-smartness. It all looks pretty
straightforward to me.

Andrew.

[none albert]

In article <MsqdnZxsda-P2x7A...@supernews.com>,

For me it requires the state smartness that numbers ("literals") have. 1]
I use { and } instead of [: and ;] and use it freely inside or
outside definitions, the way numbers work.
>
>Andrew.
1]
The state smartness of numbers is to compile a literal using LITERAL
instead of leaving it on the stack, if compilation state is on.

[Mark William Humphries]

Le jeudi 19 septembre 2019 18:37:15 UTC+8, none albert a écrit :
> In article <MsqdnZxsda-P2x7A...@supernews.com>,
> <a...@littlepinkcloud.invalid> wrote:
> >dxforth <dxf...@gmail.com> wrote:
> >> On Thursday, 19 September 2019 04:42:33 UTC+10, Alex McDonald wrote:
> >>> ...
> >>> That immediate simplicity delivered state smartness.
> >>
> >> With which Forth Inc and others appear to have no problem. Most
> >> implementation examples one sees e.g. quotations are written in
> >> terms of state smartness because it's common and well-understood.
> >
> >I know this isn't your main point, but I'm not sure what about
> >quotations might use state-smartness. It all looks pretty
> >straightforward to me.
>
> For me it requires the state smartness that numbers ("literals") have. 1]
> I use { and } instead of [: and ;] and use it freely inside or
> outside definitions, the way numbers work.
> >
> >Andrew.
> 1]
> The state smartness of numbers is to compile a literal using LITERAL
> instead of leaving it on the stack, if compilation state is on.

I find that perspective confusing. I don't think of it as numbers being state-smart, I think of it as the compiler being number-smart. :)

[Krishna Myneni]

On Wed, 18 Sep 2019 12:54:06 +0000, Anton Ertl wrote:

...

>
> : postpone ( "name" -- )
> ?parse-name ?find-name name>compile ( xt1 xt2 ) swap postpone literal
> compile, ;

> ...

The definition of POSTPONE refers to an earlier definition of POSTPONE .
You did not give the earlier definition.

Krishna

[none albert]

In article <181a79de-401d-4616...@googlegroups.com>,
Mark William Humphries <mark.willia...@gmail.com> wrote:

<SNIP>

>I find that perspective confusing. I don't think of it as numbers
> being state-smart, I think of it as the compiler being number-smart.

The whole point of Forth is to avoid smart compilers.
Think about it, and you will be enlightened.

My compiler is already pretty dumb but I want to publish an even
dumber one, with the result that STATE is only known and used
by INTERPRET .

Groetjes Albert

[Mark William Humphries]

Le jeudi 19 septembre 2019 20:51:14 UTC+8, none albert a écrit :
> In article <181a79de-401d-4616...@googlegroups.com>,
> Mark William Humphries <mark.willia...@gmail.com> wrote:
>
> <SNIP>
> >I find that perspective confusing. I don't think of it as numbers
> > being state-smart, I think of it as the compiler being number-smart.
>
> The whole point of Forth is to avoid smart compilers.
> Think about it, and you will be enlightened.

I thought about having a truly number-dumb compiler and just using prefix words for all numbers, but in the end I settled on using prefix words for non-decimal bases, e.g. $ ffff and % 10101, while the compiler still deals with decimals on its own. I found that to be the most convenient combination for me.

> My compiler is already pretty dumb but I want to publish an even
> dumber one, with the result that STATE is only known and used
> by INTERPRET .

I haven't had STATE in 20+ years, don't need it, haven't missed it.

[Anton Ertl]

Mark William Humphries <mark.willia...@gmail.com> writes:
>Le jeudi 19 septembre 2019 18:37:15 UTC+8, none albert a =C3=A9crit=C2=A0:

>> 1]
>> The state smartness of numbers is to compile a literal using LITERAL
>> instead of leaving it on the stack, if compilation state is on.
>
>I find that perspective confusing.

That's not surprising. "None) (albert" lives in his own world, with
his own terminology, like Humpty-Dumpty (the literary character, not
the c.l.f poster).

>I don't think of it as numbers being sta=
>te-smart,

They are not, as shown in the following code:

: foo postpone 1 ;
: bar [ foo ] ; .s
bar .s

If "1" was STATE-smart, FOO on the second line would push 1 on the
stack right away. In SwiftForth, iForth and Gforth 0.7, POSTPONE does
not find a word "1", so it cannot be STATE-smart. In VFX and
development Gforth, BAR pushes 1 on the stack, so for POSTPONE "1"
behaves like a normal (not STATE-smart) word defined as follows:

1 constant 1

Neither VFX nor development Gforth can tick "1", so it obviously does
not behave like a STATE-smart word when ticked.

[Anton Ertl]

Krishna Myneni <krishna...@ccreweb.org> writes:
>On Wed, 18 Sep 2019 12:54:06 +0000, Anton Ertl wrote:
>
>...
>>
>> : postpone ( "name" -- )
>> ?parse-name ?find-name name>compile ( xt1 xt2 ) swap postpone literal
>> compile, ;

Append IMMEDIATE

>> ...
>
>The definition of POSTPONE refers to an earlier definition of POSTPONE .
>You did not give the earlier definition.

It's the same one (or an equivalent one in the cross-compiler). You
can expand it if you want:

: postpone ( "name" -- )
?parse-name ?find-name name>compile ( xt1 xt2 ) swap

[ s" literal" ?find-name name>compile swap ] literal [ compile, ]
compile, ; immediate

The second line is the expansion of "postpone literal".

[none albert]

In article <2019Sep1...@mips.complang.tuwien.ac.at>,

Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
>Mark William Humphries <mark.willia...@gmail.com> writes:
>>Le jeudi 19 septembre 2019 18:37:15 UTC+8, none albert a =C3=A9crit=C2=A0:
>>> 1]
>>> The state smartness of numbers is to compile a literal using LITERAL
>>> instead of leaving it on the stack, if compilation state is on.
>>
>>I find that perspective confusing.
>
>That's not surprising. "None) (albert" lives in his own world, with
>his own terminology, like Humpty-Dumpty (the literary character, not
>the c.l.f poster).
>
>>I don't think of it as numbers being sta=
>>te-smart,
>
>They are not, as shown in the following code:
>
>: foo postpone 1 ;
>: bar [ foo ] ; .s
>bar .s

This 1 is of course POSTPONE-smart:
it can be postponed while it is not even a word in a dictionary.
I think that is worse than STATE-smart.

>
>If "1" was STATE-smart, FOO on the second line would push 1 on the
>stack right away. In SwiftForth, iForth and Gforth 0.7, POSTPONE does
>not find a word "1", so it cannot be STATE-smart. In VFX and
>development Gforth, BAR pushes 1 on the stack, so for POSTPONE "1"
>behaves like a normal (not STATE-smart) word defined as follows:
>
>1 constant 1
>
>Neither VFX nor development Gforth can tick "1", so it obviously does
>not behave like a STATE-smart word when ticked.
>
>- anton
>--

[Alex McDonald]

On 19-Sep-19 16:05, albert wrote:
> In article <2019Sep1...@mips.complang.tuwien.ac.at>,
> Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
>> Mark William Humphries <mark.willia...@gmail.com> writes:
>>
>>> I don't think of it as numbers being sta=
>>> te-smart,
>>
>> They are not, as shown in the following code:
>>
>> : foo postpone 1 ;
>> : bar [ foo ] ; .s
>> bar .s
>
> This 1 is of course POSTPONE-smart:
> it can be postponed while it is not even a word in a dictionary.

That's a function of recognizers, not POSTPONE, so you might say it's
recognizer smart. Recognizers pose the issue of managing and not finding
words that aren't in wordlists, but we've always had that issue with
numbers anyway.


--
Alex

[Anton Ertl]

albert@cherry.(none) (albert) writes:
>This 1 is of course POSTPONE-smart:
>it can be postponed while it is not even a word in a dictionary.
>I think that is worse than STATE-smart.

Why?

[Anton Ertl]

Ruvim <ruvim...@gmail.com> writes:
>It depends what does this flag mean: the standard notion of immediacy,
>or any non default compilation semantics.
>
>
>" ['] EXECUTE = " produces TRUE for immediate word (in the standard
>notion), and FALSE otherwise (i.e. for other non default compilation
>semantics and for default compilation semantics).

There are two notions of immediacy in the standard: What IMMEDIATE
does (change compilation semantics), and what FIND returns.

In cmForth words such as + (which have default compilation semantics,
i.e., are not immediate as far as compilation semantics are concerned)
have a header in the compilation wordlist that appears to be an
immediate word (executing that word performs the compilation
semantics).

In Gforth, EXIT (which has default compilation semantics in the
standard document, i.e., not immediate) is reported by FIND as
immediate.

I expect that we can find similar examples in other systems.

In any case, whether ['] EXECUTE = produces true for immediate words
as seen by FIND depends on the implementation.

>" ['] COMPILE, <> " produces TRUE for any non default compilation
>semantics (including immediate in the standard notion), and FALSE for
>the default compilation semantics only.

Likewise, whether that produces any different result from ['] EXECUTE =
depens on the implementation.

In Gforth both produce the same result. I checked the change from the
former to the latter. It was caused by a change where the two
variants would produce different results. However, when I pointed out
the problems this causes, Bernd changed that, so again the two
variants produce the same result.

>OTOH, a system may return other variant for some immediate words.
>
>E.g. if "TO" or "IF" is implemented as immediate STATE-smart word (NB:
>the standard allows such approach)

Yes for TO, no for IF. If POSTPONE special-cases IF to work around
the STATE-smartness, IF is no longer STATE-smart, but something else.

> NAME>COMPILE may return
>( xt 'EXECUTE-COMPILING ) to accurately represent the compilation semantics.

The question is, how you implement FIND.

[Anton Ertl]

Krishna Myneni <krishna...@ccreweb.org> writes:
>Is it not useful to know when writing a word definition whether or not
>you are using an immediate word inside the definition, particularly for
>some word from a library?

It is useful to know what the word does, but immediacy alone is not
helpful in my experience.

[Anton Ertl]

Krishna Myneni <krishna...@ccreweb.org> writes:
>On Wed, 18 Sep 2019 12:54:06 +0000, Anton Ertl wrote:
>
>> Krishna Myneni <krishna...@ccreweb.org> writes:
>>>On Tue, 17 Sep 2019 13:50:52 +0100, Alex McDonald wrote:
>>>Both COMPILE and [COMPILE] are dead because their functionality has been
>>>replaced by POSTPONE (Forth 2012, A.6.1.2033) which checks the immediacy
>>>of a word to decide which of the two functions to perform.
>>
>> No. POSTPONE /name/ appends the compilation semantics of /name/ to the
>> current definition.
>>
>
>To be more specific, in systems which use a precedence/immediacy bit and
>don't provide dual tokens for execution and compilation, POSTPONE needs
>to query the immediacy bit for a word to determine the compilation
>semantics.

You can still define POSTPONE as I have done. Yes, NAME>COMPILE will
have to look at the immediate bit in such a system at some point, but
it will have to do that anyway.

[Krishna Myneni]

On Thu, 19 Sep 2019 17:18:35 +0000, Anton Ertl wrote:

> Krishna Myneni <krishna...@ccreweb.org> writes:
>>Is it not useful to know when writing a word definition whether or not
>>you are using an immediate word inside the definition, particularly for
>>some word from a library?
>
> It is useful to know what the word does, but immediacy alone is not
> helpful in my experience.
>

Looking at the list generated by WL-IMMEDIATE for the Forth wordlist in
my system, it is immediately (unintended pun) clear that most of the
words have to do with compilation. No surprise. So, in using a library
that one didn't write, it may be useful to know which words perform
compilation.

Krishna

[Ruvim]

On 2019-09-19 06:15, Krishna Myneni wrote:
> On Wed, 18 Sep 2019 23:28:18 +0000, Krishna Myneni wrote:

> With the introduction of NAME>COMPILE in kForth-32 (as of commit
> 6ceee33), I have added a word in the program wl-tools.4th to show all of
> the immediate words in a wordlist. The example below shows all of the
> immediate words in the kForth-32 Forth wordlist.
>
> ----
> Ready!
> include wl-tools
> ok
> forth-wordlist wl-immediate

[...]
> ]
> [

By the specification (6.1.2540), Right-bracket ']' should have the
default compilation semantics.

The following is a standard code:

: state-on ( -- ) ] ; \ set compilation state

: state-off ( -- ) postpone [ ; \ set interpretation state


It seems "state-on" definition will not work in your system.

[Krishna Myneni]

On Fri, 20 Sep 2019 10:56:06 +0300, Ruvim wrote:

> On 2019-09-19 06:15, Krishna Myneni wrote:

...

>> forth-wordlist wl-immediate
> [...]
>> ]
>> [
>
> By the specification (6.1.2540), Right-bracket ']' should have the
> default compilation semantics.
>
> The following is a standard code:
>
> : state-on ( -- ) ] ; \ set compilation state
>
> : state-off ( -- ) postpone [ ; \ set interpretation state
>
>
> It seems "state-on" definition will not work in your system.

Hi Ruvim,

I noticed the difference in immediacy of ']' between kForth and Gforth,
but I did not believe it to be a standards issue. Does the specification
(6.1.2540) actualy require this?

----
6.1.2540 “right-bracket” ]
( –– )
Enter compilation state.

See: 3.4 The Forth text interpreter, 3.4.5 Compilation, 6.1.2500 [, A.
6.1.2540 ].
----

If non-immediacy of ']' is standard, or even common practice, it should
not be a problem to change it in kForth; however, I don't recall this
causing portability issues in the past.

Krishna

[Krishna Myneni]

On Fri, 20 Sep 2019 09:42:58 +0000, Krishna Myneni wrote:

> On Fri, 20 Sep 2019 10:56:06 +0300, Ruvim wrote:
>
...
>> By the specification (6.1.2540), Right-bracket ']' should have the
>> default compilation semantics.

> ... Does the specification (6.1.2540) actualy require this?

>
> ----
> 6.1.2540 “right-bracket” ]
> ( –– )
> Enter compilation state.
>
> See: 3.4 The Forth text interpreter, 3.4.5 Compilation, 6.1.2500 [, A.
> 6.1.2540 ].
> ----

Ok, I see why it should be a non-immediate word per the spec.:

----
3.4.3.3
Compilation semantics

Unless otherwise specified in a “ Compilation: ” section of the glossary
entry, the compilation semantics of a Forth definition shall be to append
its execution semantics to the execution semantics of the current
definition.
----

Thanks. I'll fix the issue.

KM

[none albert]

In article <qm0vas$mld$1...@dont-email.me>,

I don't dig that. The first fundamental flaw in it is that it doesn't
make a distinction between words to be used (by the application of the
library) and words to be supplied (by the system you use, if indeed
the library is supplied in source form).
Concerning the words to be used, its use must be documented, and that
is the end of it.
Concerning the words to be supplied, they are either standard --
and that is the end of it -- or they place a demand on the system
on which they are used. If a word say XXX is used, and you have XXX
on your system, and it is *not* a standard word, there is not a shred
of proof that it is doing the same.

So the library has to document that kind of demands. If in weird cases
it is required that a particular word is IMMEDIATE that should be
documented among all other things needed to be able to use the
library. Like Anton Ertl I have a hard time dreaming up an
example of that, but anyway.

>
>Krishna

Groetjes Albert

[Krishna Myneni]

On Fri, 20 Sep 2019 11:27:22 +0000, none) (albert wrote:

> In article <qm0vas$mld$1...@dont-email.me>,
> Krishna Myneni <krishna...@ccreweb.org> wrote:
>>On Thu, 19 Sep 2019 17:18:35 +0000, Anton Ertl wrote:
>>
>>> Krishna Myneni <krishna...@ccreweb.org> writes:
>>>>Is it not useful to know when writing a word definition whether or not
>>>>you are using an immediate word inside the definition, particularly
>>>>for some word from a library?
>>>
>>> It is useful to know what the word does, but immediacy alone is not
>>> helpful in my experience.
>>>
>>>
>>Looking at the list generated by WL-IMMEDIATE for the Forth wordlist in
>>my system, it is immediately (unintended pun) clear that most of the
>>words have to do with compilation. No surprise. So, in using a library
>>that one didn't write, it may be useful to know which words perform
>>compilation.
>
> I don't dig that. The first fundamental flaw in it is that it doesn't
> make a distinction between words to be used (by the application of the
> library) and words to be supplied (by the system you use, if indeed the
> library is supplied in source form).

Library words could and should be loaded into their own wordlists. This
is an explicit feature of the modules system which I use. Then, it is
quite simple to distinguish between words in the Forth wordlist and
another wordlist. WL-IMMEDIATE takes a wordlist as an argument, if you
didn't notice.

> Concerning the words to be used, its use must be documented, and that is
> the end of it.
> Concerning the words to be supplied, they are either standard --
> and that is the end of it -- or they place a demand on the system on
> which they are used. If a word say XXX is used, and you have XXX on your
> system, and it is *not* a standard word, there is not a shred of proof
> that it is doing the same.
>

The introspection tools are not intended to be a substitute for
documentation. They are tools to display attributes of words loaded in
the system, which may be helpful during debugging or developing code.

> So the library has to document that kind of demands. If in weird cases
> it is required that a particular word is IMMEDIATE that should be
> documented among all other things needed to be able to use the library.

That wasn't a case I was envisioning.

> Like Anton Ertl I have a hard time dreaming up an example of that, but
> anyway.
>

Whether or not the tools will be useful, I can't say. We haven't had them
for very long (maybe other systems did, but I didn't), so I expect to
find out how useful they may be. I think dependency graphs for words will
be extremely useful to see the overall architecture of a program. But,
we're not there yet.

Krishna

[Anton Ertl]

Krishna Myneni <krishna...@ccreweb.org> writes:
>The introspection tools are not intended to be a substitute for
>documentation. They are tools to display attributes of words loaded in
>the system, which may be helpful during debugging or developing code.

I am doubtful about the "debugging or developing code" part, but they
may be helpful in understanding how a system is implemented. In that
sense:

: filter ( xt nt -- xt f )
2dup swap execute if
dup cr name>string type then
drop true ;

: filtered-words ( ... xt -- ... )
\ show all words in the search order for which xt ( nt -- f ) returns true
>r get-order r> swap 0 ?do ( widn ... widm xt )
['] filter rot traverse-wordlist loop
drop ;

: dual? ( nt -- f )
dup name>interpret swap name>compile drop <> ;

: funny? ( nt -- f )
name>compile nip dup ['] execute = swap ['] compile, = or 0= ;

: nt<>xt? ( nt -- f)
dup name>interpret <> ;

previous \ get rid of the double occurence of FORTH
\ Here are some calls and their outputs on development Gforth:
' dual? filtered-words
\ sizeof
\ ;code
\ .\"
\ s\"
\ ctrl
\ ~~
\ what's
\ action-of
\ ."
\ s"
\ [:
\ does>
\ addr
\ +TO
\ IS
\ TO ok
' funny? filtered-words \ ok
' nt<>xt? filtered-words
\ Because of the long output, I have reformatted the output of this one:
\ clear-libs add-lib c-function sizeof ;code .\" s\" ctrl ekey? k12 k11
\ k10 k9 k8 k7 k6 k5 k4 k3 k2 k1 voctable offset .elapsed timer-reset
\ dbg-shell ~~ xc-width prim>name >head xt>name name>compile
\ name>interpret lle wle xdle xdbe xle xbe lbe wbe xd@ x@ l@ w@ what's
\ action-of needs [AGAIN] $[]off nocomp {: d>s .name .id %size
\ %alignment nalign vlist defined ." s" [: ENDIF $off recursive does>
\ addr +TO IS TO defer! defer@-opt: comp: set-compiler compile-only
\ name>view recurse lastxt const A, cfalign name parse-word
\ code-address! >code-address ((name>)) A! cfaligned r@ order
\ set-order forth-wordlist Forth words

The large number of matches for NT<>XT? is due to aliases for the most
part.

[Krishna Myneni]

On Sat, 21 Sep 2019 06:33:42 +0000, Anton Ertl wrote:

> Krishna Myneni <krishna...@ccreweb.org> writes:
>>The introspection tools are not intended to be a substitute for
>>documentation. They are tools to display attributes of words loaded in
>>the system, which may be helpful during debugging or developing code.
>
> I am doubtful about the "debugging or developing code" part, but they

> may be helpful in understanding how a system is implemented. ...

I certainly agree about being helpful in understanding a system
implementation -- we already (inadvertently) demonstrated that in this
thread with my demo of WL-IMMEDIATE for kForth. Ruvim (and I) noticed
that ']' was immediate in kForth, and Ruvim noted that this was non-
standard. This information is helpful to me to better align kForth with
the standard.

To the extent that the tools are standard, we can use them to compare
differences between systems also. For example, your filter NT<>XT? is
Gforth-specific. In kForth nt is never equal to xt. Similarly, my WL-
CREATED is specific to kForth, although it could possibly be made
standard by replacing the ambiguous condition in the standard spec for
>BODY with a requirement to return NULL for non-CREATEd words. However,
this may prove too challenging for system implementers, and may not be
worth the cost to do so for some.

> In that sense:
>
> : filter ( xt nt -- xt f )
> 2dup swap execute if
> dup cr name>string type then
> drop true ;
>
> : filtered-words ( ... xt -- ... )
> \ show all words in the search order for which xt ( nt -- f ) returns
> true
> >r get-order r> swap 0 ?do ( widn ... widm xt )
> ['] filter rot traverse-wordlist loop
> drop ;
>
> : dual? ( nt -- f )
> dup name>interpret swap name>compile drop <> ;
>
> : funny? ( nt -- f )
> name>compile nip dup ['] execute = swap ['] compile, = or 0= ;
>
> : nt<>xt? ( nt -- f)
> dup name>interpret <> ;

> ...

Nice! Thanks for the examples.

The challenge is to extend these tools from simply inspecting a system's
implementation to inspecting application code loaded into the system. You
may be correct that, within the constraint of standard words which exist
now, there's not much we can infer about application code to assist us in
debugging and writing code. We need to keep an eye open for such
possibilities. I'm interested in the possibility of generating dependency
trees for application words -- whether or not it is practical to do so
remains to be seen, although Stephen Pelc voiced his opinion years ago
that it would not be practical for his large applications. There may be
other possibilities which aren't as demanding. What (standard) words
might be useful as factors for such capabilities? Alex McDonald was
considering words to implement user-defined fields in a dictionary
header, and I think that's an intriguing possibility.

Krishna

[Ruvim]

On 2019-09-19 19:55, Anton Ertl wrote:
> Ruvim <ruvim...@gmail.com> writes:
>> It depends what does this flag mean: the standard notion of immediacy,
>> or any non default compilation semantics.
>>
>>
>> " ['] EXECUTE = " produces TRUE for immediate word (in the standard
>> notion), and FALSE otherwise (i.e. for other non default compilation
>> semantics and for default compilation semantics).

I was wrong here. "['] EXECUTE = " can produce TRUE not only for the
standard immediate words, since xt1 (and the corresponding behavior) can
be different in the results of NAME>INTERPRET and NAME>COMPILE

A word that produce true for only immediate words (in the standard
notion) is:

: IMMEDIATE-STANDARD? ( nt -- flag )
DUP NAME>COMPILE ['] EXECUTE <> IF 2DROP FALSE EXIT THEN
SWAP NAME>INTERPRET =
;

>
> There are two notions of immediacy in the standard: What IMMEDIATE
> does (change compilation semantics), and what FIND returns.

We already discussed this issue somewhat [1].

The two notions of immediacy (that are not both well defined and not
expressed unambiguously) — is an inconsistency.

If the normative text is consistent regarding immediacy, we should
accept the only one notion of immediacy (from the term definitions section).

If there are two notions of immediacy, we should admit that the
corresponding normative text is inconsistent.

At the moment I'm inclined to interpret the normative text as consistent
in regards to immediacy.

> In cmForth words such as + (which have default compilation semantics,
> i.e., are not immediate as far as compilation semantics are concerned)
> have a header in the compilation wordlist that appears to be an
> immediate word (executing that word performs the compilation
> semantics).

Could you point to the corresponding place in the cmForth source code?

In any case, an additional header in the COMPILER wordlist means the
different definition.

And it seems that all the words from COMPILER wordlist are immediate.
I.e. immediacy attribute is maintained via belonging to the special
wordlist.

A factor for standard FIND in such Forth system can look like:

: SFIND ( c-addr u -- c-addr u 0 | xt code )
STATE @ IF
COMPILER-WL LOOKUP-WORDLIST IF 1 EXIT THEN
THEN
FORTH-WL LOOKUP-WORDLIST ( c-addr u 0 | xt -1 )
;

So FIND returns honest immediacy flag.

Definition of IMMEDIATE can look like:

: IMMEDIATE ( -- )
FORTH-WL @ COMPILER-WL ( nt wid ) APPEND-WORDLIST-NT
;

NB: The optional Search-Order word set is not provided in such Forth system.

> In Gforth, EXIT (which has default compilation semantics in the
> standard document, i.e., not immediate) is reported by FIND as
> immediate.

If we interpret this case as an ambiguous condition (due to undefined
interpretation semantics for EXIT), a standard system may return
whatever it wants in this case. An implementation just usually
guarantees that the user defined text interpreter is able to either
produce a correct result or report an error explicitly.

If we interpret this case (i.e. C" EXIT" FIND ) as a standard code, it
should return either ( xt -1 ) in accordance with the single notion of
immediacy, or (c-addr 0). The proper compilation for EXIT (and similar
words) can be implemented inside "COMPILE,".

Actually, I think that the specifications for such words should be
rewritten. Execution semantics should be removed, and compilation
semantics should be specified. After that ( xt 1 ) will be a possible
result on find "EXIT" (the same as it is a possible result on find "IF"
at the moment).

>
> I expect that we can find similar examples in other systems.
>
> In any case, whether ['] EXECUTE = produces true for immediate words
> as seen by FIND depends on the implementation.

Yes.

>> " ['] COMPILE, <> " produces TRUE for any non default compilation
>> semantics (including immediate in the standard notion), and FALSE for
>> the default compilation semantics only.
>
> Likewise, whether that produces any different result from ['] EXECUTE =
> depens on the implementation.

Yes.

> In Gforth both produce the same result. I checked the change from the
> former to the latter. It was caused by a change where the two
> variants would produce different results. However, when I pointed out
> the problems this causes, Bernd changed that, so again the two
> variants produce the same result.

Since in Gforth no either EXECUTE or COMPILE, is returned, and no other
variants.

Since in Gforth no other variants are returned except "EXECUTE" and
"COMPILE,".

>> OTOH, a system may return other variant for some immediate words.
>>
>> E.g. if "TO" or "IF" is implemented as immediate STATE-smart word (NB:
>> the standard allows such approach)
>
> Yes for TO, no for IF. If POSTPONE special-cases IF to work around
> the STATE-smartness, IF is no longer STATE-smart, but something else.

Well, let's define "STATE-smart" as following:

A definition is *STATE-smart* if its execution semantics
depends on STATE.

From this point of view STATE-smartness nothing to do with POSTPONE.

>> NAME>COMPILE may return
>> ( xt 'EXECUTE-COMPILING ) to accurately represent the compilation semantics.
>
> The question is, how you implement FIND.

This case is not a problem when FIND is used in a user-defined text
interpreter (in the classic notion).

FIND just returns ( xt 1 ) for "IF" or "TO". And it is OK while this
xt can be only immediately executed or compiled. No other options.

But it is a problem if FIND is used in other purpose. OTOH trying to
obtain xt for these words is an ambiguous condition.

Perhaps I would make FIND return ( c-addr 0 ) in such cases.


In any case, there is no standard way to implement a full-fledged
user-defined text interpreter in Forth-2012. You need to use
system-specific APIs to handle the literals. Why don't to use
system-specific APIs to also handle the words (if you need to be able to
handle any word)?
If you want to be limited by a standard code, and you write your own
handlers for the literals, you can also write them for the non ordinary
words.


I agree that we need a standard way (a high level API) to implement a
user-defined text interpreter. But we don't need to use the
old-fashioned FIND for that. FIND should be obsoleted, early or late.

Re NAME>* API — it is not enough general (it is too specific on dual-xt
implementation). Particularly, NAME>INTERPRET is not fine with a
STATE-smart implementation of "dual-semantics" words. [2]

I think that Recognizer/Resolver mechanism can be used to handle (and
perhaps define [3]) all the non-ordinary words and be a high-level API
to implement a full-fledged user-defined text interpreter.




[1] https://forth-standard.org/standard/core/FIND#reply-294
[2] https://forth-standard.org/standard/core/FIND#reply-334
[3] https://gist.github.com/ruv/dbe99ace903b86063610dbae81f3c69f

--
Ruvim

[Ruvim]

On 2019-09-21 17:00, Ruvim wrote:
> On 2019-09-19 19:55, Anton Ertl wrote:
>> Ruvim <ruvim...@gmail.com> writes:

[...]

>>> NAME>COMPILE may return
>>> ( xt 'EXECUTE-COMPILING ) to accurately represent the compilation
>>> semantics.
>>
>> The question is, how you implement FIND.
>
> This case is not a problem when FIND is used in a user-defined text
> interpreter (in the classic notion).
>
>   FIND just returns ( xt 1 ) for "IF" or "TO". And it is OK while this
>   xt can be only immediately executed or compiled. No other options.

Correction: xt can be only immediately executed. Since compiling
(performing compilation semantics) for them means to execute it. And it
actually cannot be COMPILE,d in the text interpreter due to 1 flag.

[Anton Ertl]

Ruvim <ruvim...@gmail.com> writes:
>On 2019-09-19 19:55, Anton Ertl wrote:
>> There are two notions of immediacy in the standard: What IMMEDIATE
>> does (change compilation semantics), and what FIND returns.

...

>The two notions of immediacy (that are not both well defined and not
>expressed unambiguously) — is an inconsistency.

It is. The "Clarify FIND" proposal
<http://forth-standard.org/standard/core/FIND#reply-165> is intended
to resolve this.

>> In cmForth words such as + (which have default compilation semantics,
>> i.e., are not immediate as far as compilation semantics are concerned)
>> have a header in the compilation wordlist that appears to be an
>> immediate word (executing that word performs the compilation
>> semantics).
>
>Could you point to the corresponding place in the cmForth source code?

For the source code at
<https://github.com/ForthHub/cmFORTH/blob/combined/cmforth.fth>

Definition of COMPILER +: line 110
Definition of FORTH +: line 391
Definition of ] (the compiler): line 411
Definition of INTERPRET: line 357

Note that 2 is the wid of COMPILER, and 1 the wid of FORTH.

>In any case, an additional header in the COMPILER wordlist means the
>different definition.
>
>And it seems that all the words from COMPILER wordlist are immediate.
>I.e. immediacy attribute is maintained via belonging to the special
>wordlist.

Right. This is nice to see for (:

Definition of COMPILER (: line 486
Definition of FORTH (: line 488

cmForth does not have an immediate flag and not the word IMMEDIATE, so
you have to define ( in both vocabularies. cmForth makes it
relatively hard to define immediate words.

>> In Gforth, EXIT (which has default compilation semantics in the
>> standard document, i.e., not immediate) is reported by FIND as
>> immediate.
>
>If we interpret this case as an ambiguous condition (due to undefined
>interpretation semantics for EXIT), a standard system may return
>whatever it wants in this case. An implementation just usually
>guarantees that the user defined text interpreter is able to either
>produce a correct result or report an error explicitly.
>
>If we interpret this case (i.e. C" EXIT" FIND ) as a standard code, it
>should return either ( xt -1 ) in accordance with the single notion of
>immediacy, or (c-addr 0). The proper compilation for EXIT (and similar
>words) can be implemented inside "COMPILE,".

If it was so easy... We wasted a lot of time on trying to turn EXIT
into a tickable word with default compilation semantics; i.e., if you
EXECUTE its xt, it works as if DROP EXIT had been in place of the
EXECUTE. This attempt resulted in a cascade of other changes, some of
them complications, and at some point in the paper
<http://www.complang.tuwien.ac.at/anton/euroforth/ef15/papers/ertl.pdf>.
But all the complications there were still not sufficient, and it
would have required further complications. In the end we reverted the
complications and EXIT is an immediate word again.

>Actually, I think that the specifications for such words should be
>rewritten. Execution semantics should be removed, and compilation
>semantics should be specified.

Yes. Who will propose it?

>> Yes for TO, no for IF. If POSTPONE special-cases IF to work around
>> the STATE-smartness, IF is no longer STATE-smart, but something else.
>
>Well, let's define "STATE-smart" as following:
>
> A definition is *STATE-smart* if its execution semantics
> depends on STATE.
>
> From this point of view STATE-smartness nothing to do with POSTPONE.

This definition is incomplete. STATE-smart words are immediate, and
therefore the compilation semantics of STATE-smart words vary with
STATE. So if POSTPONE eliminates this STATE-dependence, the word is
not STATE-smart.

>>> NAME>COMPILE may return
>>> ( xt 'EXECUTE-COMPILING ) to accurately represent the compilation semantics.
>>
>> The question is, how you implement FIND.
>
>This case is not a problem when FIND is used in a user-defined text
>interpreter (in the classic notion).
>
> FIND just returns ( xt 1 ) for "IF" or "TO". And it is OK while this
> xt can be only immediately executed or compiled. No other options.
>
>But it is a problem if FIND is used in other purpose. OTOH trying to
>obtain xt for these words is an ambiguous condition.
>
>Perhaps I would make FIND return ( c-addr 0 ) in such cases.

FIND does not know whether it is used in a text interpreter or for
something else. If you return c-addr 0, a text interpreter will not
be able to text-interpret TO and IF.

>In any case, there is no standard way to implement a full-fledged
>user-defined text interpreter in Forth-2012.

Yes, FIND is defined too loosely for that. The "Clarify FIND"
proposal is intended to fix that.

>You need to use
>system-specific APIs to handle the literals.

No. You can use >NUMBER and LITERAL to deal with literals.

>I agree that we need a standard way (a high level API) to implement a
>user-defined text interpreter. But we don't need to use the
>old-fashioned FIND for that.

That was the intention of Forth-94.

>FIND should be obsoleted, early or late.

Even if we do, FIND will be in the standard for another release, and
the code using it will not go away fast, so it should be specified to
satisfy the intention.

>Re NAME>* API — it is not enough general (it is too specific on dual-xt
>implementation). Particularly, NAME>INTERPRET is not fine with a
>STATE-smart implementation of "dual-semantics" words. [2]

Good point. Systems that want to implement dual-semantics words with
some STATE-smart internals should special-case these words in
NAME>INTERPRET and NAME>COMPILE, e.g.:

: <is> ' defer! ;
: [is] ' postpone literal postpone defer! ; immediate
: is
state @ if postpone [is] else <is> then ; immediate

: name>interpret ( nt -- xt )
case
nt-of-is of ['] <is> endof
nt-of-to of ['] <to> endof
...
default-name>interpret exit \ default case
endcase ;

: ' ?parse-name ?find-name name>interpret 0= -14 and throw ;

: name>compile ( nt -- xt1 xt2 )
case
nt-of-is of ['] [is] ['] execute endof
...
default-name>compile exit \ default case
endcase ;

: postpone ?parse-name ?find-name name>compile swap postpone literal compile, ;

: find ( c-addr -- xt n | c-addr 0 )
dup count find-name ?dup if
nip dup name>interpret swap name>compile >r ( xti xt1 R: xt2 )
state @ if swap then drop ( xt R: xt2 )
r> ['] execute = ( xt 0|-1 ) -2 * 1- ( xt -1|1 )
then ;

This is actually a way to get rid of STATE-smartness in small steps:
Once you have converted everything to use NAME>INTERPRET and
NAME>COMPILE, you can just change the definition of IS to:

: IS ABORT ;

and you should notice any uses that do not go through NAME>INTERPRET
and NAME>COMPILE pretty quickly.

>I think that Recognizer/Resolver mechanism can be used to handle (and
>perhaps define [3]) all the non-ordinary words

Doubtful. You still have to deal correctly with the TRAVERSE-WORDLIST
(which has to report these words) and redefining these words. Even if
you can get it to work correctly, the approach above (special-casing
in NAME>INTERPRET and NAME>COMPILE) is simpler to implement.

>and be a high-level API
>to implement a full-fledged user-defined text interpreter.

Recognizers are a refactoring of the text interpreter, and you can
define a user-defined text interpreter that relies on them.

[Ruvim]

On 2019-09-21 17:41, Anton Ertl wrote:
> Ruvim <ruvim...@gmail.com> writes:
>> On 2019-09-19 19:55, Anton Ertl wrote:

[...]

>>> Yes for TO, no for IF. If POSTPONE special-cases IF to work around
>>> the STATE-smartness, IF is no longer STATE-smart, but something else.
>>
>> Well, let's define "STATE-smart" as following:
>>
>> A definition is *STATE-smart* if its execution semantics
>> depends on STATE.
>>
>> From this point of view STATE-smartness nothing to do with POSTPONE.
>
> This definition is incomplete. STATE-smart words are immediate,

If STATE-smartness already implicates immediacy, why do you mention
"immediate" separately? I always thought that you use them as
independent properties because of that.

Could you define your "STATE-smart" term?

> STATE-smart words are immediate,
> and therefore the compilation semantics of STATE-smart words vary with
> STATE.

It sounds very odd for me that the compilation semantics of some word
can vary with STATE. But I see that it is a consequence of your
interpretation for some standard terms.

Actually, it is our long dispute that was not finished. You didn't
explicitly accept the premise regarding the term definitions [1]. Well,
have a look at other arguments.


1. The specification and the reference implementation for SYNONYM don't
support your interpretation.

Let's consider the following example:

SYNONYM 0EQ 0=

In a system with the reference implementation of SYNONYM, '0EQ' is a
STATE-dependent immediate word.

Also SYNONYM guarantees that the compilation semantics are equal for
'0EQ' and '0='. The compilation semantics for '0=' are not depend on
STATE, and therefore the compilation semantics for '0EQ' are not depend
on STATE too. But by your interpretation they depend on STATE.
Therefore, you interpretation of the "compilation semantics" term isn't
the same as its meaning in the standard. Q.E.D.

In the same time the execution semantics of '0EQ' are unequal to the
execution semantics of '0='.

: [E] EXECUTE ; IMMEDIATE
0 ' 0= ] [E] [ . \ prints -1
0 ' 0EQ ] [E] [ . \ prints 0 (or even throw exception)
\ NB: with the reference implementation of SYNONYM

Therefore, the execution semantics for "0EQ" are unequal to the
interpretation semantics for "0EQ" (that are equal to the int. sem. of
'0=').


2. A similar conclusion can be shown for 'TO' (or 'S"') word.
'TO' is allowed to be implemented as a STATE-dependent immediate word.
In the same time the compilation semantics for 'TO' are not equal to the
interpretation semantics for 'TO'. But they are equal (in such
implementation) by your interpretation of the corresponding terms.

> So if POSTPONE eliminates this STATE-dependence, the word is
> not STATE-smart.

Why does a word property depend on how POSTPONE is implemented?

If STATE-smartness is a property of a word (e.g. 'TO' word, or 'IF'
word), why does it depend on how POSTPONE is implemented?

What depends on POSTPONE is the semantics, that are appended. POSTPONE
does not eliminate STATE-dependency of a word, and don't change the word
at all.



--

[1] news://qi6con$1bdk$1...@gioia.aioe.org
https://groups.google.com/forum/message/raw?msg=comp.lang.forth/vrq2f2XXy1Q/tiuq7hJTFgAJ


--
Ruvim

[Anton Ertl]

Ruvim <ruvim...@gmail.com> writes:
>On 2019-09-21 17:41, Anton Ertl wrote:
>> Ruvim <ruvim...@gmail.com> writes:
>>> On 2019-09-19 19:55, Anton Ertl wrote:
>
>[...]
>>>> Yes for TO, no for IF. If POSTPONE special-cases IF to work around
>>>> the STATE-smartness, IF is no longer STATE-smart, but something else.
>>>
>>> Well, let's define "STATE-smart" as following:
>>>
>>> A definition is *STATE-smart* if its execution semantics
>>> depends on STATE.
>>>
>>> From this point of view STATE-smartness nothing to do with POSTPONE.
>>
>> This definition is incomplete. STATE-smart words are immediate,
>
>If STATE-smartness already implicates immediacy, why do you mention
>"immediate" separately?

Some people have used "STATE-smart" for STATE-dependent words that are
not immediate. So I usually mention "immediate" to make the immediacy
clear.

>Could you define your "STATE-smart" term?

Here's a description: A STATE-smart word is an immediate word with
STATE-dependent execution semantics used to approximate
dual-semantics. The approximation works as intended when the word is
text-interpreted, but breaks down when it is ticked and the EXECUTEd
in compile state, or POSTPONEd, and the containing word is then
performed in interpret state.

>1. The specification and the reference implementation for SYNONYM don't
>support your interpretation.

The implementation section of SYNONYM says that the implementation of
SYNONYM requires knowledge of the host implementation. The provided
definition is broken (for the reasons outlined above), and in some
sense, this is not a problem, because the reference implementation is
not normative. Of course, in the real world it is a problem, because
people are misled by broken reference implementations.

>In the same time the execution semantics of '0EQ' are unequal to the
>execution semantics of '0='.
>
> : [E] EXECUTE ; IMMEDIATE
> 0 ' 0= ] [E] [ . \ prints -1
> 0 ' 0EQ ] [E] [ . \ prints 0 (or even throw exception)
> \ NB: with the reference implementation of SYNONYM
>
>Therefore, the execution semantics for "0EQ" are unequal to the
>interpretation semantics for "0EQ" (that are equal to the int. sem. of
>'0=').

So you provide a demonstration of the breakage of the reference
implementation of SYNONYM.

>2. A similar conclusion can be shown for 'TO' (or 'S"') word.
>'TO' is allowed to be implemented as a STATE-dependent immediate word.
>In the same time the compilation semantics for 'TO' are not equal to the
>interpretation semantics for 'TO'. But they are equal (in such
>implementation) by your interpretation of the corresponding terms.

TO has an ambiguous condition that prevents a standard program from
determining the difference between a proper dual-semantics
implementation and a STATE-smart implementation. This ambiguous
condition allows to implement TO as a STATE-smart word. That does not
mean that STATE-smart words are proper implementations of
dual-semantics words in general (and for S", they are not).

>> So if POSTPONE eliminates this STATE-dependence, the word is
>> not STATE-smart.
>
>Why does a word property depend on how POSTPONE is implemented?

I am sure you are adept enough at this kind of stuff to finish the
reductio ad absurdum yourself, but for your pleasure, let me formulate
it more clearly.

STATE-smart words have STATE-dependent compilation semantics. We can
detect it by putting the compilation semantics in a word X with
POSTPONE, and then performing X in interpret STATE. If we write a
STATE-smart word, and it does not behave as a STATE-smart word when
POSTPONEd, the implementation of POSTPONE is incorrect.

>If STATE-smartness is a property of a word (e.g. 'TO' word, or 'IF'
>word), why does it depend on how POSTPONE is implemented?

TO and IF are not defined as STATE-smart words. TO is allowed to be
implemented as STATE-smart word, IF isn't.

[Ruvim]

On 2019-10-01 20:05, Anton Ertl wrote:
> Ruvim <ruvim...@gmail.com> writes:
>> On 2019-09-21 17:41, Anton Ertl wrote:
>>> Ruvim <ruvim...@gmail.com> writes:
>>>> On 2019-09-19 19:55, Anton Ertl wrote:
>>
>> [...]
>>>>> Yes for TO, no for IF. If POSTPONE special-cases IF to work around
>>>>> the STATE-smartness, IF is no longer STATE-smart, but something else.
>>>>
>>>> Well, let's define "STATE-smart" as following:
>>>>
>>>> A definition is *STATE-smart* if its execution semantics
>>>> depends on STATE.
>>>>
>>>> From this point of view STATE-smartness nothing to do with POSTPONE.
>>>
>>> This definition is incomplete. STATE-smart words are immediate,
>>
>> If STATE-smartness already implicates immediacy, why do you mention
>> "immediate" separately?
>
> Some people have used "STATE-smart" for STATE-dependent words that are
> not immediate. So I usually mention "immediate" to make the immediacy
> clear.

I see. And you make them independent as a funny side effect.

>> Could you define your "STATE-smart" term?
>
> Here's a description: A STATE-smart word is an immediate word with
> STATE-dependent execution semantics used to approximate
> dual-semantics.

Can a STATE-dependent immediate word be used for another purpose?
(i.e. not to "approximate dual-semantics")

What the words are "dual-semantics"? Are they those five well known
standard words only or something else?

> The approximation works as intended when the word is
> text-interpreted, but breaks down when it is ticked and the EXECUTEd
> in compile state, or POSTPONEd, and the containing word is then
> performed in interpret state.

Is this breaking is essential part of a STATE-smart word?
What if such behavior is not breaking but is intended one?

Actually this description of "STATE-smart word" looks too arbitrary.
It resembles me "No true Scotsman" fallacy [1].

The /true/ STATE-smart word works not as intended when it is POSTPONEd,
but if it works as intended, then POSTPONE works incorrectly, or the
word is not /true/ STATE-smart word. And you can also arbitrary choose
what is intended.

In any case, it does not matter. See (2) below.

>> 1. The specification and the reference implementation for SYNONYM don't
>> support your interpretation.
>
> The implementation section of SYNONYM says that the implementation of
> SYNONYM requires knowledge of the host implementation. The provided
> definition is broken (for the reasons outlined above), and in some
> sense, this is not a problem, because the reference implementation is
> not normative. Of course, in the real world it is a problem, because
> people are misled by broken reference implementations.

In important premise of a reader is that the reference implementations
are correct and consistent even if they are specific for some system only.

The broken definitions have no place in the standard. If a definition in
a reference implementation becomes known as broken, it should be excluded.

But in the case of SYNONYM the reference implementation does not violate
the specification and it is not broken.

>> In the same time the execution semantics of '0EQ' are unequal to the
>> execution semantics of '0='.
>>
>> : [E] EXECUTE ; IMMEDIATE
>> 0 ' 0= ] [E] [ . \ prints -1
>> 0 ' 0EQ ] [E] [ . \ prints 0 (or even throw exception)
>> \ NB: with the reference implementation of SYNONYM
>>
>> Therefore, the execution semantics for "0EQ" are unequal to the
>> interpretation semantics for "0EQ" (that are equal to the int. sem. of
>> '0=').
>
> So you provide a demonstration of the breakage of the reference
> implementation of SYNONYM.

It is not breakage of the reference implementation. Since such
implementation is allowed by the specification.

The specification does not require to have equal execution semantics.
Perhaps it is a hole in the specification or just a lack of a
declaration for the corresponding ambiguous condition, but it is not a
problem in the reference implementation.


Perhaps you can show a breakage using your interpretation of
"compilation semantics" and "interpretation semantics" terms, but there
is no any breakage if we use the notion from the standard term definitions.

>> 2. A similar conclusion can be shown for 'TO' (or 'S"') word.
>> 'TO' is allowed to be implemented as a STATE-dependent immediate word.
>> In the same time the compilation semantics for 'TO' are not equal to the
>> interpretation semantics for 'TO'. But they are equal (in such
>> implementation) by your interpretation of the corresponding terms.
>
> TO has an ambiguous condition that prevents a standard program from
> determining the difference between a proper dual-semantics
> implementation and a STATE-smart implementation. This ambiguous
> condition allows to implement TO as a STATE-smart word. That does not
> mean that STATE-smart words are proper implementations of
> dual-semantics words in general (and for S", they are not).

Well, perhaps your are right using your definition for "STATE-smart
word" (that is too vague for me).

So, take a note, in the citation above I talk about STATE-dependency and
immediacy (in the standard notion) only — they seem clear enough.

Yes, the ambiguous condition prevents the standard program from
determining that, but why don't to use a non standard program to check
'TO'? We know that in the case of POSTPONE the corresponding ambiguous
condition is just a concession to some Forth system implementations.
OTOH, at the moment a standard program can still check 'S"' word.

(2) I can just claim that the standard words with undefined execution
semantics (including 'TO', 'S"', 'IF') may and can be implemented as
STATE-dependent immediate word: i.e. FIND returns ( xt 1 ) for them and
this result of FIND does not depend on STATE. And value 1 means
immediacy in the standard notion. And POSTPONE and [COMPILE] can be be
implemented in such a way to work correctly for them.


Moreover, a special choice of wording is used in some specifications in
the standard to allow such implementation. It is the next argument.

3.
Let's take for shortness:
ES — execution semantics
IS — interpretation semantics
CS — compilation semantics

In the glossary entries for all those five "dual-semantics" words the
standard specifies IS and CS, but ES remain unspecified.

By your interpretation the ES are equal to the IS for these words. To
explicitly express this idea, ES could be specified instead of IS.

E.g. for 'S"' word:

*Execution*:
( "ccc<quote>" -- c-addr u )
Parse ccc delimited by " (double quote). Store the resulting string
in a transient buffer c-addr u.

Compilation:
( "ccc<quote>" -- )
Parse ccc delimited by " (double quote). Append the run-time semantics
given below to the current definition.

Run-time:
( -- c-addr u )
Return c-addr and u that describe a string consisting of the
characters ccc.

And the IS become just the default case (by 3.4.3.2).

NB: this approach is used for the words like EXIT — when ES are given
and CS are defined as the default case (by 3.4.3.3).

But this approach in the specifications wording excludes implementation
(2) for these words. Since in such case ES can't be STATE-dependent.
(but, as it is obvious for me, TC X3J14 felt to not exclude such
implementations).

So, the chosen wording (i.e. specifying IS and CS but not ES) allows
implementation (2) for these words, it is a correct implementation
choice, and the IS are unequal to the ES for them in such case (namely,
the IS are STATE-independent but the ES are STATE-dependent).

But this contradicts with your interpretation that the IS are equal to
the ES for these words.

>>> So if POSTPONE eliminates this STATE-dependence, the word is
>>> not STATE-smart.
>>
>> Why does a word property depend on how POSTPONE is implemented?
>
> I am sure you are adept enough at this kind of stuff to finish the
> reductio ad absurdum yourself, but for your pleasure, let me formulate
> it more clearly.
>
> STATE-smart words have STATE-dependent compilation semantics. We can
> detect it by putting the compilation semantics in a word X with
> POSTPONE, and then performing X in interpret STATE. If we write a
> STATE-smart word, and it does not behave as a STATE-smart word when
> POSTPONEd, the implementation of POSTPONE is incorrect.

Actually we cannot detect it via POSTPONE when the correct
implementation of POSTPONE is a controversial point.




[1] https://en.wikipedia.org/wiki/No_true_Scotsman


--
Ruvim

[Ruvim]

On 2019-09-21 17:41, Anton Ertl wrote:

> Ruvim <ruvim...@gmail.com> writes:
>> On 2019-09-19 19:55, Anton Ertl wrote:
>>> There are two notions of immediacy in the standard: What IMMEDIATE
>>> does (change compilation semantics), and what FIND returns.

> ....

>> The two notions of immediacy (that are not both well defined and not
>> expressed unambiguously) — is an inconsistency.
>
> It is.

And, by your interpretation, what immediacy notion does FIND return?
I.e., in the clause, "If the definition is immediate, also return 1",
what does "definition is immediate" mean?

> The "Clarify FIND" proposal
> <http://forth-standard.org/standard/core/FIND#reply-165> is intended

> to resolve this.[1]

[...]

>>>> NAME>COMPILE may return
>>>> ( xt 'EXECUTE-COMPILING ) to accurately represent the compilation semantics.
>>>
>>> The question is, how you implement FIND.
>>
>> This case is not a problem when FIND is used in a user-defined text
>> interpreter (in the classic notion).
>>
>> FIND just returns ( xt 1 ) for "IF" or "TO". And it is OK while this
>> xt can be only immediately executed or compiled. No other options.
>>
>> But it is a problem if FIND is used in other purpose. OTOH trying to
>> obtain xt for these words is an ambiguous condition.
>>
>> Perhaps I would make FIND return ( c-addr 0 ) in such cases.
>
> FIND does not know whether it is used in a text interpreter or for
> something else. If you return c-addr 0, a text interpreter will not
> be able to text-interpret TO and IF.

I'm now inclined to think that FIND was *intended* to be used in the
Forth text interpreter *only*. In such case the STATE before FIND and
before processing its results is always the same (1). Perhaps FIND may
be used for other purposes too but the condition (1) shall be kept when n=1.

[...]

>> In any case, there is no standard way to implement a full-fledged
>> user-defined text interpreter in Forth-2012.
>
> Yes, FIND is defined too loosely for that.

Not only that.

Another important thing is that there is no way to restore (or reset)
the interpreter state after an error. For example, locals for an
unfinished colon definition could be left in the scope.

A system's text interpreter is provided as QUIT word. But a user can't
implement an alternative functionality without re-implementing a much
part of the Forth system.

>> I agree that we need a standard way (a high level API) to implement a
>> user-defined text interpreter. But we don't need to use the
>> old-fashioned FIND for that.
>
> That was the intention of Forth-94.
>
>> FIND should be obsoleted, early or late.
>
> Even if we do, FIND will be in the standard for another release, and
> the code using it will not go away fast, so it should be specified to
> satisfy the intention.

[...]

> : find ( c-addr -- xt n | c-addr 0 )
> dup count find-name ?dup if

Don't you need 'dup' instead of '?dup' here?

> nip dup name>interpret swap name>compile >r ( xti xt1 R: xt2 )
> state @ if swap then drop ( xt R: xt2 )
> r> ['] execute = ( xt 0|-1 ) -2 * 1- ( xt -1|1 )
> then ;

It seems the variant when n=1 in interpretation state doesn't have much
sense in your proposal [1].

Why not the following?

: find ( c-addr -- xt n | c-addr 0 )

dup count find-name dup 0= if exit then ( c-addr nt )
state @ if nip name>compile ['] execute = -2 * 1- exit then
name>interpret ( c-addr xt|0 ) dup if nip -1 exit then
;

Also, in the general case, when 'name>compile' can return xt2 other than
'compile,' and 'execute', the code becomes slightly more complex: it
should return (c-addr 0) if xt2 is neither of them.


--
Ruvim

[Coos Haak]

Op Mon, 7 Oct 2019 18:45:23 +0300 schreef Ruvim:

In the source the definition is followed by the word IMMEDIATE.

;-)

groet Coos

[Coos Haak]

Op Mon, 7 Oct 2019 18:45:23 +0300 schreef Ruvim:

> And, by your interpretation, what immediacy notion does FIND return?
> I.e., in the clause, "If the definition is immediate, also return 1",
> what does "definition is immediate" mean?

[Ruvim]

It would mean the single notion of immediacy — the notion from the term
definition. And I'm inclined to this interpretation too.

But if somebody claims that in the specification for FIND the different
immediacy notion is used, I would be interested what this notion is.

[Anton Ertl]

Ruvim <ruvim...@gmail.com> writes:
>On 2019-09-21 17:41, Anton Ertl wrote:
>> Ruvim <ruvim...@gmail.com> writes:
>>> On 2019-09-19 19:55, Anton Ertl wrote:
>>>> There are two notions of immediacy in the standard: What IMMEDIATE
>>>> does (change compilation semantics), and what FIND returns.
>> ....
>>> The two notions of immediacy (that are not both well defined and not
>>> expressed unambiguously) — is an inconsistency.
>>
>> It is.
>
>And, by your interpretation, what immediacy notion does FIND return?
>I.e., in the clause, "If the definition is immediate, also return 1",
>what does "definition is immediate" mean?

If FIND returns xt 1 in compile state, the compilation semantics of
the found word is to EXECUTE the xt.

>> The "Clarify FIND" proposal
>> <http://forth-standard.org/standard/core/FIND#reply-165> is intended
>> to resolve this.[1]
...

>> FIND does not know whether it is used in a text interpreter or for
>> something else. If you return c-addr 0, a text interpreter will not
>> be able to text-interpret TO and IF.
>
>I'm now inclined to think that FIND was *intended* to be used in the
>Forth text interpreter *only*.

I don't think that the Forth-94 committee had that intention. I think
the intention was to expose a classically available word that is used
in the text interpreter, in ' and POSTPONE to the user, in particular
for user-defined text interpreters. And I think that you can, e.g.,
implement POSTPONE and ' based on FIND once FIND is sufficiently
defined (as in my proposal).

[user-defined text interpreter]

>In such case the STATE before FIND and
>before processing its results is always the same (1).

Yes.

>>> In any case, there is no standard way to implement a full-fledged
>>> user-defined text interpreter in Forth-2012.
>>
>> Yes, FIND is defined too loosely for that.
>
>Not only that.
>
>Another important thing is that there is no way to restore (or reset)
>the interpreter state after an error. For example, locals for an
>unfinished colon definition could be left in the scope.

POSTPONE ;

might do the trick. And if that produces an error, you have to think
about how to stay within the user-defined text interpreter.

>A system's text interpreter is provided as QUIT word. But a user can't
>implement an alternative functionality without re-implementing a much
>part of the Forth system.

True. The recognizer proposal proposes to replace the core of the
text interpreter with one that works through the user-reconfigurable
recognizer queue/stack, allowing to reuse the rest of the text
interpreter infrastructure (error catching, INCLUDE, EVALUATE, LOAD,
etc.).

>> : find ( c-addr -- xt n | c-addr 0 )
>> dup count find-name ?dup if
>Don't you need 'dup' instead of '?dup' here?

Yes.

>> nip dup name>interpret swap name>compile >r ( xti xt1 R: xt2 )
>> state @ if swap then drop ( xt R: xt2 )
>> r> ['] execute = ( xt 0|-1 ) -2 * 1- ( xt -1|1 )
>> then ;
>
>It seems the variant when n=1 in interpretation state doesn't have much
>sense in your proposal [1].

Yes, it makes no difference for a typical user-defined text
interpreter. However, if you have a classic
single-xt-plus-immediate-flag system, FIND returns the same values in
either STATE, and the FIND implementation above will behave the same
way. So if you have a usage of FIND that relies on this property, the
FIND above will work.

>Why not the following?
>
>: find ( c-addr -- xt n | c-addr 0 )
> dup count find-name dup 0= if exit then ( c-addr nt )
> state @ if nip name>compile ['] execute = -2 * 1- exit then
> name>interpret ( c-addr xt|0 ) dup if nip -1 exit then
>;

This version would always return "xt -1" in interpret state, and some
(system-specific) users of FIND might prefer my FIND implementation
(after bug-fixing).

>Also, in the general case, when 'name>compile' can return xt2 other than
>'compile,' and 'execute', the code becomes slightly more complex: it
>should return (c-addr 0) if xt2 is neither of them.

In this case FIND should return something equivalent to:

:noname xt1 xt2 execute ;

as xt and 1 as n.

[Ruvim]

On 2019-10-08 19:35, Anton Ertl wrote:
> Ruvim <ruvim...@gmail.com> writes:
>> On 2019-09-21 17:41, Anton Ertl wrote:
>>> Ruvim <ruvim...@gmail.com> writes:
>>>> On 2019-09-19 19:55, Anton Ertl wrote:
>>>>> There are two notions of immediacy in the standard: What IMMEDIATE
>>>>> does (change compilation semantics), and what FIND returns.
>>> ....
>>>> The two notions of immediacy (that are not both well defined and not
>>>> expressed unambiguously) — is an inconsistency.
>>>
>>> It is.
>>
>> And, by your interpretation, what immediacy notion does FIND return?
>> I.e., in the clause, "If the definition is immediate, also return 1",
>> what does "definition is immediate" mean?
>
> If FIND returns xt 1 in compile state, the compilation semantics of
> the found word is to EXECUTE the xt.

In some Forth systems (namely in the classic
single-xt-plus-immediate-flag systems) it always means that the
corresponding word is immediate in the standard notion.

Wrt other Forth systems.

A crucial point is that FIND is allowed to resolve a name into the
different definitions depending on STATE.

Let's consider for example a system that doesn't provide the
Search-Order word set and has the following *internal* definitions:

wordlist constant comp

: find state @ if comp >order find previous exit then find ;

comp exch-current
: foo postpone + ; immediate
set-current

: foo + ;

\ test foo via find:

c" foo" find swap . . \ prints some xt1 and -1
c" foo" 'find ]execute[ swap . . \ prints another xt2 and 1

So, from the system's point of view, n=1 in the second line means

immediacy in the standard notion.

From a program's point of view, foo is not immediate in the standard
notion, despite n=1, since:

\ the execution semantics in compilation state:
2 3 'foo ]execute[ . \ prints 5

\ the compilation semantics:
2 3 ] foo [ . \ prints 3

\ — they produce the different effects.
\ For immediate words the effects should be the same.


The standard allows to implement some standard words in such a way, but
a standard program cannot detect that since obtaining the execution
token for them is not allowed.

Therefore, when a program may check immediacy, n=1 means that the
corresponding word is immediate in the standard notion.


How can we disprove this reasoning?



The only issue is Sq ('S"') word in some Forth systems. If ticking is
not ambiguous for Sq, then Sq must be implemented as single-xt immediate
word. Otherwise the result of FIND will be detectable as incorrect for
Sq. And it is yet another argument, why ticking Sq is implicitly ambiguous.



OTOH, I agree that the specification for FIND should be clarified and
have made a proposal too.


--
Ruvim

[Ruvim]

On 2019-10-08 19:35, Anton Ertl wrote:
> Ruvim <ruvim...@gmail.com> writes:
>> On 2019-09-21 17:41, Anton Ertl wrote:

[...]

>>> The "Clarify FIND" proposal
>>> <http://forth-standard.org/standard/core/FIND#reply-165> is intended
>>> to resolve this.[1]

> ....

>>> FIND does not know whether it is used in a text interpreter or for
>>> something else. If you return c-addr 0, a text interpreter will not
>>> be able to text-interpret TO and IF.
>>
>> I'm now inclined to think that FIND was *intended* to be used in the
>> Forth text interpreter *only*.
>
> I don't think that the Forth-94 committee had that intention.

Perhaps I have expressed my thought wrongly. I wanted to say that FIND,
at the first, should respond to the requirements of the text
interpreter. And other possible use cases have far less weight.

> I think
> the intention was to expose a classically available word that is used
> in the text interpreter, in ' and POSTPONE to the user, in particular
> for user-defined text interpreters. And I think that you can, e.g.,
> implement POSTPONE and ' based on FIND once FIND is sufficiently
> defined (as in my proposal).

Yes, Tick and POSTPONE should be implementable over FIND.

>>>> In any case, there is no standard way to implement a full-fledged
>>>> user-defined text interpreter in Forth-2012.
>>>
>>> Yes, FIND is defined too loosely for that.
>>
>> Not only that.
>>
>> Another important thing is that there is no way to restore (or reset)
>> the interpreter state after an error. For example, locals for an
>> unfinished colon definition could be left in the scope.
>
> POSTPONE ;
>
> might do the trick. And if that produces an error, you have to think
> about how to stay within the user-defined text interpreter.

It can produce an error due to control structure mismatch.

Therefore, I even didn't try, and was forced to restart my "quit" again
after any error.

>> A system's text interpreter is provided as QUIT word. But a user can't
>> implement an alternative functionality without re-implementing a much
>> part of the Forth system.
>
> True. The recognizer proposal proposes to replace the core of the
> text interpreter with one that works through the user-reconfigurable
> recognizer queue/stack, allowing to reuse the rest of the text
> interpreter infrastructure (error catching, INCLUDE, EVALUATE, LOAD,
> etc.).

Yes, recognizers help.
But they cannot help to implement QUIT alternative in regards to reset
the states after an error. There need another API.

>>> : find ( c-addr -- xt n | c-addr 0 )
>>> dup count find-name ?dup if
>> Don't you need 'dup' instead of '?dup' here?
>
> Yes.
>
>>> nip dup name>interpret swap name>compile >r ( xti xt1 R: xt2 )
>>> state @ if swap then drop ( xt R: xt2 )
>>> r> ['] execute = ( xt 0|-1 ) -2 * 1- ( xt -1|1 )
>>> then ;
>>
>> It seems the variant when n=1 in interpretation state doesn't have much
>> sense in your proposal [1].
>
> Yes, it makes no difference for a typical user-defined text
> interpreter. However, if you have a classic
> single-xt-plus-immediate-flag system, FIND returns the same values in
> either STATE, and the FIND implementation above will behave the same
> way. So if you have a usage of FIND that relies on this property, the
> FIND above will work.

It seems for me that your proposal is not fine enough for
single-xt-plus-immediate-flag systems, see bellow.

>> Why not the following?
>>
>> : find ( c-addr -- xt n | c-addr 0 )
>> dup count find-name dup 0= if exit then ( c-addr nt )
>> state @ if nip name>compile ['] execute = -2 * 1- exit then
>> name>interpret ( c-addr xt|0 ) dup if nip -1 exit then
>> ;
>
> This version would always return "xt -1" in interpret state, and some
> (system-specific) users of FIND might prefer my FIND implementation
> (after bug-fixing).

It is reasonable, but I think a system-specific using of FIND already
doesn't matter, since FIND is provided for compatibility only (i.e. for
programs).

So, the question is — what about programs?

: foo ; immediate
: bar c" foo" find nip 1 = ; bar .

A standard program may assume that bar returns -1.

But according to your proposal, bar *may* return 0 in this case. Since
your specification for FIND does not dictate what to return in
interpretation state (1 or -1) and in what case. And what it means.

I think it is a problem.

Another problem is that your proposal doesn't allow to implement the
words with undefined execution semantics as STATE-dependent immediate
words. But the original specification allows that.

>> Also, in the general case, when 'name>compile' can return xt2 other than
>> 'compile,' and 'execute', the code becomes slightly more complex: it
>> should return (c-addr 0) if xt2 is neither of them.
>
> In this case FIND should return something equivalent to:
>
> :noname xt1 xt2 execute ;
>
> as xt and 1 as n.

Yes, it is technically possible, but perhaps it isn't worth effort in
all the cases.


--
Ruvim

[Ruvim]

On 2019-09-21 17:41, Anton Ertl wrote:

> Ruvim <ruvim...@gmail.com> writes:
>> On 2019-09-19 19:55, Anton Ertl wrote:

[...]

>>> In cmForth words such as + (which have default compilation semantics,
>>> i.e., are not immediate as far as compilation semantics are concerned)
>>> have a header in the compilation wordlist that appears to be an
>>> immediate word (executing that word performs the compilation
>>> semantics).
>>
>> Could you point to the corresponding place in the cmForth source code?
>
> For the source code at
> <https://github.com/ForthHub/cmFORTH/blob/combined/cmforth.fth>
>
> Definition of COMPILER +: line 110
> Definition of FORTH +: line 391
> Definition of ] (the compiler): line 411
> Definition of INTERPRET: line 357
>
> Note that 2 is the wid of COMPILER, and 1 the wid of FORTH.
>

Thanks, I see.

A one important question arises.

Is a standard Forth system allowed to implement an ordinary word (i.e. a
word with default interpretation semantics) in this way, returning the
different xt in the different STATE?

The issue is that FIND should return n = 1 for such word in compilation
state, but that violates the letters of the specification for FIND,
since a standard program can check that the word is not immediate.

So, my conclusion is that the standard implicitly denies such
implementation for the ordinary words. Despite of the intention
mentioned in the rationale section for FIND.

Therefore, for the ordinary words FIND should return the same values
independent on STATE.

>> In any case, an additional header in the COMPILER wordlist means the
>> different definition.
>>
>> And it seems that all the words from COMPILER wordlist are immediate.
>> I.e. immediacy attribute is maintained via belonging to the special
>> wordlist.
>
> Right. This is nice to see for (:
>
> Definition of COMPILER (: line 486
> Definition of FORTH (: line 488
>
> cmForth does not have an immediate flag and not the word IMMEDIATE, so
> you have to define ( in both vocabularies. cmForth makes it
> relatively hard to define immediate words.

cmForth is not a standard system. But we talk about its approach
(implementation variant). cmForth approach can be used in the internals
of a standard system that has IMMEDIATE word.

>>> In Gforth, EXIT (which has default compilation semantics in the
>>> standard document, i.e., not immediate) is reported by FIND as
>>> immediate.
>>
>> If we interpret this case as an ambiguous condition (due to undefined
>> interpretation semantics for EXIT), a standard system may return
>> whatever it wants in this case. An implementation just usually
>> guarantees that the user defined text interpreter is able to either
>> produce a correct result or report an error explicitly.
>>
>> If we interpret this case (i.e. C" EXIT" FIND ) as a standard code, it
>> should return either ( xt -1 ) in accordance with the single notion of
>> immediacy, or (c-addr 0). The proper compilation for EXIT (and similar
>> words) can be implemented inside "COMPILE,".
>
> If it was so easy... We wasted a lot of time on trying to turn EXIT
> into a tickable word with default compilation semantics; i.e., if you
> EXECUTE its xt, it works as if DROP EXIT had been in place of the
> EXECUTE. This attempt resulted in a cascade of other changes, some of
> them complications, and at some point in the paper
> <http://www.complang.tuwien.ac.at/anton/euroforth/ef15/papers/ertl.pdf>.
> But all the complications there were still not sufficient, and it
> would have required further complications. In the end we reverted the
> complications and EXIT is an immediate word again.

Yes, I see. There are a lot of pitfalls.
But I didn't mean making EXIT a tickable word. A standard code can't
tick EXIT due to undefined interpretation semantics for it.

EXIT can be correctly implemented as following (just an idea):

: RET, POSTPONE EXIT ;

: EXIT -32 THROW ; \ "invalid name argument"

: COMPILE, ( xt -- )
['] EXIT =? IF RET, [ RET, ] THEN
COMPILE,
;

So FIND will return n = -1 for "EXIT", and both POSTPONE and COMPILE,
will work correctly for it.

This approach conforms to the letters of the specification, without much
practical sense though. Updating the specification for FIND to reflect
the current Forth system implementations will make admissible their
approach for EXIT.

>> Actually, I think that the specifications for such words should be
>> rewritten. Execution semantics should be removed, and compilation
>> semantics should be specified.
>
> Yes. Who will propose it?

Perhaps I will try, after fixing more critical parts.


[...]

>> Re NAME>* API — it is not enough general (it is too specific on dual-xt
>> implementation). Particularly, NAME>INTERPRET is not fine with a
>> STATE-smart implementation of "dual-semantics" words. [2]
>
> Good point. Systems that want to implement dual-semantics words with
> some STATE-smart internals should special-case these words in
> NAME>INTERPRET and NAME>COMPILE,

Such systems have also another option: don't provide NAME>INTERPRET and
NAME>COMPILE at all.

Example of a correct implementation: special-words-as-immediate.fth
https://gist.github.com/ruv/21e7b4d0693dce587adeecb824e9c5aa
(the code can be just loaded into a standard Forth system for test)

There some "dual-semantics" words and words with undefined
interpretation semantics are implemented as STATE-dependent immediate
words. POSTPONE and [COMPILE] correctly work for them. Tick returns
execution semantics for them (as it may). Performing these execution
semantics in interpretation state performs the corresponding
interpretation semantics (as it should). In compilation state they
perform the corresponding compilation semantics (as it may due to
undefined execution semantics).


As I mentioned before (in a sibling sub-thread), the only reason of
undefined execution semantics for many words in the standard is to allow
the different implementation for them, including implementation via
immediate words.

It is not fair. You just suggest to remove STATE-dependent internals in
a system that wants to implement "dual-semantics" words with some
STATE-dependent internals. But the system might have various reasons for
that. For example, to have multiple levels of postponing, or a macro
mode for macros, or interpretive control flow structures that are not
affected by string literals and comments.

> and you should notice any uses that do not go through NAME>INTERPRET
> and NAME>COMPILE pretty quickly.
>
>> I think that Recognizer/Resolver mechanism can be used to handle (and
>> perhaps define [3]) all the non-ordinary words
>
> Doubtful. You still have to deal correctly with the TRAVERSE-WORDLIST
> (which has to report these words) and redefining these words.

I think if a name is not resolvable via SEARCH-WORDLIST, it might be not
resolvable via TRAVERSE-WORDLIST too. Therefore, TRAVERSE-WORDLIST
doesn't lay more requirements on a system than SEARCH-WORDLIST.

OTOH, "markup" wordlist (from my example [1]) can be easily connected
with FORTH-WORDLIST inside TRAVERSE-WORDLIST word.

Redefining (shadowing) by an ordinary word just works, no need for any
special handling. Also, they can be shadowed by other user-defined
resolvers (via Resolver API). No need for special handling too.

POSTPONE works for them. Tick is not applicable to them, and can be
implemented in any way: to fail on them, or to return an xt for them.


[1] dual-semantics-via-recognizer
https://gist.github.com/ruv/dbe99ace903b86063610dbae81f3c69f

> Even if
> you can get it to work correctly, the approach above (special-casing
> in NAME>INTERPRET and NAME>COMPILE) is simpler to implement.

If a system already has Recognizers/Resolvers mechanism, it seems
simpler to use this mechanism to support "dual-semantics" words than to
implement the mentioned special-casing or a separate "dual-xt" mechanism
for them.

Also, Recongizers/Resolvers mechanism provides a common API to define
new "dual-semantics" or other non ordinary words. This API will be in
any case. So there is no a big reason to standardize another "dual-xt"
based API to define the "dual-semantics" words.



--
Ruvim