Demonstration of Dual Semantics Forth: Tracking Word Dependencies

[Krishna Myneni]

I've written Forth code (below) to demonstrate keeping track of word
dependencies using a Forth system capable of defining dual-semantics
words. Here I use Gforth for the example. I believe the following may
also be doable on VFX, using its NDCS: word. The other requirements of
the system are that the name token (nt) of the last word definition is
accessible, and that there is a mechanism to define a closure.

A word definition may be immediately followed by the word TRACK-DEPS
which will set its compilation semantics to not only compile its
interpretation semantics but also to add an entry into a dependency tree
to keep track of the words into which it is compiled. For example,

: FOO ." foo" ; TRACK-DEPS

: T1 foo ;

The word SHOW-DEPS prints the names of all words in which FOO is compiled.

s" foo" SHOW-DEPS \ prints T1

To continue,

: BAR ." bar" ; TRACK-DEPS

: T2 foo bar ;

s" foo" SHOW-DEPS \ prints the list: T2 T1

s" bar" SHOW-DEPS \ prints the list: T2

With the other definitions in the example code, it should be possible to
follow the entire dependency tree branch which terminates on a word for
which TRACK-DEPS has been applied (this assumes all of the nodes leading
to the word are also being tracked).

For simplicity, the dependency tree is implemented as a fixed size
table. This is not memory efficient and flexible enough for real use,
but is suitable for a demonstration. A tree structure should be used for
actual applications. The main point is to show that being able to set
the compilation semantics for a word independently allows for
capabilities which are likely not possible (or considerably more
difficult) in Forth source on a single-xt + immediate flag system only.

--
Krishna Myneni


dependencies.4th
-----------------
\ dependencies.4th
\
\ Demonstrate tracking first-level word dependencies
\ in a dual-xt system.
\
\ K. Myneni, 2022-10-10
\
\ This program is written for Gforth. It uses the following
\ non-standard words
\
\ LATESTNT ( -- nt )
\ SET-COMPSEM ( xt -- )
\ [N:D ( x -- )

require set-compsem.fs
synonym a@ @

100 constant MAX-TRACKS
64 constant MAX-DEPS
MAX-TRACKS MAX-DEPS * cells constant MAX-DTABLE-SIZE

create DEPENDENCY-TABLE MAX-DTABLE-SIZE allot
DEPENDENCY-TABLE MAX-DTABLE-SIZE erase

: th-track ( u -- a )
DEPENDENCY-TABLE swap MAX-DEPS * cells + ;

0 value curr-track

: get-track ( nt -- n ) \ return -1 if not found
curr-track 0 ?DO
I th-track a@ \ nt nt-track
over = IF drop I UNLOOP EXIT THEN
LOOP
drop -1 ;

: track-dependency ( nt -- )
curr-track th-track !
1 curr-track + to curr-track ;

: add-dependency ( nt-track nt-newdep -- )
swap get-track dup 0< IF
." Not tracking." cr abort
ELSE
th-track \ -- nt-dep a
BEGIN cell+ dup a@ 0= UNTIL !
THEN ;

\ Track first-level dependencies for last defined word
: track-deps ( -- )
latestnt dup track-dependency
[n:d dup name>interpret compile, latestnt add-dependency ;]
set-compsem ;

0 value ndeps

: what-requires? ( nt-track -- nt1 nt2 ... u | 0 )
0 to ndeps
get-track dup 0< IF
drop 0
ELSE
th-track cell+ \ -- a
BEGIN
dup a@ dup \ -- a nt|0 nt|0
WHILE
swap
ndeps 1+ to ndeps
cell+
REPEAT
2drop ndeps
THEN ;

: show-deps ( caddr u -- )
find-name ?dup IF
what-requires?
0 ?DO cr name>string type LOOP cr
then ;


0 [IF] \ Demonstration

\ Define two words for which first-level deps will be tracked
: foo ." foo" ; track-deps
: bar ." bar" ; track-deps

foo cr
bar cr
s" foo" show-deps \ list of words dependent on FOO is empty
s" bar" show-deps \ list of words dependent on BAR is empty
: t1 foo ;
: t2 foo bar ;
s" foo" show-deps \ shows T1 and T2 are dependent on FOO
s" bar" show-deps \ shows T2 is dependent on BAR

[THEN]

[minf...@arcor.de]

FWIW I use a dependency table for recursive elimination of
dead words (ie those not needed for an application).

Dual semantic words, f.ex. here defined in 'my private Idaho syntax':
: MYWORD ." interpreting myword" ;
: [MYWORD] ." compiling myword" ; ' [MYWORD] COMPILES MYWORD
\ only for clarification, [MYWORD] is usually a :NONAME

Compilation of these 2 words sets dependency flags 'on the fly'
in dependency table columns for
MYWORD <- ." [MYWORD]
[MYWORD] <- ." MYWORD
\ ." brings its own dependencies of course

This is ultra-easy for 'static' code. However parsing words or words
involving dictionary search are unpredictable.

Keeping track would mean to maintain the dependency table
as integral Forth compiler database. And it would grow over time.
To keep life easy, dead word elimination stops then and the table is
eliminated from memory.

But back to your proposed dependency tracker:
a) a.m. situation for dual-semantic parsing words applies as well?
b) what is your benefit or use case for dependency introspection?

[Krishna Myneni]

On 10/10/22 12:26, minf...@arcor.de wrote:
> Krishna Myneni schrieb am Montag, 10. Oktober 2022 um 17:44:24 UTC+2:
>> I've written Forth code (below) to demonstrate keeping track of word
>> dependencies using a Forth system capable of defining dual-semantics
>> words.

..

> FWIW I use a dependency table for recursive elimination of
> dead words (ie those not needed for an application).
>

..

> But back to your proposed dependency tracker:
> a) a.m. situation for dual-semantic parsing words applies as well?

I don't understand the term "a.m. situation" -- please clarify.
Regarding parsing words, the following parsing word appears to work ok.

include dependencies.4th
: foo ' ; track-deps
: bar ." bar" ; track-deps
foo bar \ returns the xt-interp for bar
drop
: test foo ;
test foo \ returns the xt-interp for foo
s" foo" show-deps \ prints test

> b) what is your benefit or use case for dependency introspection?

For now, this is just a demonstration of being able to store, retrieve,
and output dependency information. It shows how to write an
introspection tool in Forth source on a dual-xt system. Actual benefits
are that if we wanted to know which words referenced a given word by
name, we can do so from within the Forth system. Note that it does not
involve searching the source code but uses directly the internal
dictionary information.

--
Krishna

[Krishna Myneni]

On 10/10/22 12:26, minf...@arcor.de wrote:

> Krishna Myneni schrieb am Montag, 10. Oktober 2022 um 17:44:24 UTC+2:
>> I've written Forth code (below) to demonstrate keeping track of word
>> dependencies using a Forth system capable of defining dual-semantics
>> words. Here I use Gforth for the example.

..

>> \ Track first-level dependencies for last defined word
>> : track-deps ( -- )
>> latestnt dup track-dependency
>> [n:d dup name>interpret compile, latestnt add-dependency ;]
>> set-compsem ;
>>

.
> But back to your proposed dependency tracker:
> a) a.m. situation for dual-semantic parsing words applies as well?
> b) what is your benefit or use case for dependency introspection?

I'm guessing you mean immediate words. Indeed, in the version of
TRACK-DEPS shown above, the code does not work for immediate words. I
found I had to revise TRACK-DEPS as follows to make it work in Gforth
for both precedence cases:

\ Track first-level dependencies for last defined word
: track-deps ( -- )

latestnt dup track-dependency ( -- nt-track)
dup name>compile nip
['] compile, = if

[n:d dup name>interpret compile, latestnt add-dependency ;]

else
[n:d dup name>interpret execute latestnt add-dependency ;]
then
set-compsem ;

Demonstrated as follows:

include dependencies.4th ok
: foo ." foo" ; track-deps ok
: t1 foo ; ok
s" foo" show-deps
t1
ok
: bar ." bar" ; immediate track-deps ok
: t2 foo bar ; bar ok
s" bar" show-deps
t2
ok
s" foo" show-deps
t2
t1
ok

In the above FOO is an "ordinary word", and BAR is an immediate word.
The above shows that TRACK-DEPS and SHOW-DEPS work for both types of words.

I don't understand why the following simpler definition of TRACK-DEPS
does not work under Gforth (0.7.9_20220120):

: track-deps ( -- )
latestnt dup track-dependency ( -- nt-track)
[n:d dup name>compile execute latestnt add-dependency ;]
set-compsem ;

The above definition, which (I thought) should be equivalent to the
working definition shown above leads to some sort of infinite recursion
resulting in a return stack overflow in Gforth v. 0.7.9_20220120.

: foo ." foo" ; track-deps ok
: t1 foo ;
*the terminal*:3:6: error: Return stack overflow
: t1 >>>foo<<< ;

--
Krishna

[Anton Ertl]

Krishna Myneni <krishna...@ccreweb.org> writes:
>Indeed, in the version of
>TRACK-DEPS shown above, the code does not work for immediate words. I
>found I had to revise TRACK-DEPS as follows to make it work in Gforth
>for both precedence cases:
>
>\ Track first-level dependencies for last defined word
>: track-deps ( -- )
> latestnt dup track-dependency ( -- nt-track)
> dup name>compile nip
> ['] compile, = if
> [n:d dup name>interpret compile, latestnt add-dependency ;]
> else
> [n:d dup name>interpret execute latestnt add-dependency ;]
> then
> set-compsem ;

The NAME>INTERPRETs here should be left away: You pass an xt to the
closure, not an nt. Fortunately (or unfortunately), in Gforth an xt
can be treated as an nt, so this mistake does not result in unintended
behaviour, at least for most words.

>I don't understand why the following simpler definition of TRACK-DEPS
>does not work under Gforth (0.7.9_20220120):
>
>: track-deps ( -- )
> latestnt dup track-dependency ( -- nt-track)
> [n:d dup name>compile execute latestnt add-dependency ;]
> set-compsem ;
>
>The above definition, which (I thought) should be equivalent to the
>working definition shown above leads to some sort of infinite recursion
>resulting in a return stack overflow in Gforth v. 0.7.9_20220120.
>
>: foo ." foo" ; track-deps ok
>: t1 foo ;
>*the terminal*:3:6: error: Return stack overflow
>: t1 >>>foo<<< ;

SET-COMPSEM changes what NAME>COMPILE does for the word, and the end
result is that the NAME>COMPILE EXECUTE inside the closure calls the
closure again, which calls NAME>COMPILE EXECUTE again until the return
stack overflows.

One way to avoid the problem is to do the NAME>COMPILE outside
(untested):

: track-deps ( -- )
latestnt dup track-dependency name>compile ( xt1 xt2 )
[d:d over >r execute r> latestnt add-dependency ;] set-compsem ;

Another alternative is to change NAME>COMPILE directly (with
SET->COMP) instead of using SET-COMPSEM, and call the old NAME>COMPILE
from there (untested):

: track-deps ( -- )
latestnt dup track-dependency ( nt-track )
dup >namehm @ >hm>comp @ ( nt-track old-name>comp )
[d:d swap >r execute r> latestnt add-dependency ;] set->comp ;

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: https://forth-standard.org/
EuroForth 2022: https://euro.theforth.net

[Krishna Myneni]

On 10/12/22 11:55, Anton Ertl wrote:
> Krishna Myneni <krishna...@ccreweb.org> writes:
>> Indeed, in the version of
>> TRACK-DEPS shown above, the code does not work for immediate words. I
>> found I had to revise TRACK-DEPS as follows to make it work in Gforth
>> for both precedence cases:
>>
>> \ Track first-level dependencies for last defined word
>> : track-deps ( -- )
>> latestnt dup track-dependency ( -- nt-track)
>> dup name>compile nip
>> ['] compile, = if
>> [n:d dup name>interpret compile, latestnt add-dependency ;]
>> else
>> [n:d dup name>interpret execute latestnt add-dependency ;]
>> then
>> set-compsem ;
>
> The NAME>INTERPRETs here should be left away: You pass an xt to the
> closure, not an nt. Fortunately (or unfortunately), in Gforth an xt
> can be treated as an nt, so this mistake does not result in unintended
> behaviour, at least for most words.
>

An nt is being passed into the closure, so the NAME>INTERPRET is needed
inside the closure. Note the DUP before NAME>COMPILE and the NIP afterwards.

>> I don't understand why the following simpler definition of TRACK-DEPS
>> does not work under Gforth (0.7.9_20220120):
>>
>> : track-deps ( -- )
>> latestnt dup track-dependency ( -- nt-track)
>> [n:d dup name>compile execute latestnt add-dependency ;]
>> set-compsem ;
>>
>> The above definition, which (I thought) should be equivalent to the
>> working definition shown above leads to some sort of infinite recursion
>> resulting in a return stack overflow in Gforth v. 0.7.9_20220120.
>>
>> : foo ." foo" ; track-deps ok
>> : t1 foo ;
>> *the terminal*:3:6: error: Return stack overflow
>> : t1 >>>foo<<< ;
>
> SET-COMPSEM changes what NAME>COMPILE does for the word, and the end
> result is that the NAME>COMPILE EXECUTE inside the closure calls the
> closure again, which calls NAME>COMPILE EXECUTE again until the return
> stack overflows.
>

It's too bad that SET-COMPSEM doesn't simply create a new code sequence
for the compilation semantics and store the corresponding xt. This is
similar to the problem with SET-OPTIMIZER. These words should not change
the behavior of "COMPILE," or of "NAME>COMPILE". Doing so makes these
standard words non-transparent to the programmer, resulting in
unexpected problems like this.

--
Krishna

[Anton Ertl]

Krishna Myneni <krishna...@ccreweb.org> writes:
>On 10/12/22 11:55, Anton Ertl wrote:
>> Krishna Myneni <krishna...@ccreweb.org> writes:
>>> Indeed, in the version of
>>> TRACK-DEPS shown above, the code does not work for immediate words. I
>>> found I had to revise TRACK-DEPS as follows to make it work in Gforth
>>> for both precedence cases:
>>>
>>> \ Track first-level dependencies for last defined word
>>> : track-deps ( -- )
>>> latestnt dup track-dependency ( -- nt-track)
>>> dup name>compile nip
>>> ['] compile, = if
>>> [n:d dup name>interpret compile, latestnt add-dependency ;]
>>> else
>>> [n:d dup name>interpret execute latestnt add-dependency ;]
>>> then
>>> set-compsem ;
>>
>> The NAME>INTERPRETs here should be left away: You pass an xt to the
>> closure, not an nt. Fortunately (or unfortunately), in Gforth an xt
>> can be treated as an nt, so this mistake does not result in unintended
>> behaviour, at least for most words.
>>
>
>An nt is being passed into the closure, so the NAME>INTERPRET is needed
>inside the closure. Note the DUP before NAME>COMPILE and the NIP afterwards.

NAME>COMPILE produces xt1 xt2; you consume xt2 in "['] COMPILE, =",
leaving xt1, which you then pass into a closure. You may expect an nt
and treat it like an nt (and it happens to work to some extent), but
conceptually it's an xt; if we changed Gforth so that an xt would not
be a valid xt, this code would break very obviously.

As it is, you may see some lossage in connection with, e.g., synonyms.
E.g., try

: a ." a" ;
synonym b a track-deps
: c b ;
s" b" show-deps

Does it show that c depends on b?

>>> I don't understand why the following simpler definition of TRACK-DEPS
>>> does not work under Gforth (0.7.9_20220120):
>>>
>>> : track-deps ( -- )
>>> latestnt dup track-dependency ( -- nt-track)
>>> [n:d dup name>compile execute latestnt add-dependency ;]
>>> set-compsem ;
>>>
>>> The above definition, which (I thought) should be equivalent to the
>>> working definition shown above leads to some sort of infinite recursion
>>> resulting in a return stack overflow in Gforth v. 0.7.9_20220120.
>>>
>>> : foo ." foo" ; track-deps ok
>>> : t1 foo ;
>>> *the terminal*:3:6: error: Return stack overflow
>>> : t1 >>>foo<<< ;
>>
>> SET-COMPSEM changes what NAME>COMPILE does for the word, and the end
>> result is that the NAME>COMPILE EXECUTE inside the closure calls the
>> closure again, which calls NAME>COMPILE EXECUTE again until the return
>> stack overflows.
>>
>
>It's too bad that SET-COMPSEM doesn't simply create a new code sequence
>for the compilation semantics and store the corresponding xt. This is
>similar to the problem with SET-OPTIMIZER. These words should not change
>the behavior of "COMPILE," or of "NAME>COMPILE". Doing so makes these
>standard words non-transparent to the programmer, resulting in
>unexpected problems like this.

If SET-OPTIMIZER would not change COMPILE,, it would not work (it
would fail to set the optimizer). For Gforth's current
implementation, if SET-COMPSEM did not change NAME>COMPILE, it would
not work (it would fail to change the compilation semantics).

But these problems are just like any other case where you
inadvertently introduce endless recursion; e.g., such endless
recursions happen easily if you try to reduce the number of primitives
by replacing them with colon definitions in a cross-compiler that
allows forward references. You just have to break the cycle, in the
present case by explicitly performing the old action of NAME>COMPILE
instead of just calling NAME>COMPILE.

Concerning SET-COMPSEM, one could imagine an extended header structure
that also contains a COMPSEM field. SET-COMPSEM then would set that
field and would change NAME>COMPILE to perform:

: compsem-name>comp ( nt -- xt1 xt2 )
>namehm @ >hmcompsem @ ['] execute ;

But that still would not help with the code above, because
NAME>COMPILE still would give you the new compilation semantics, which
would again result in endless recursion.

Another idea is to have a NAME>COMPILE-ORIGINAL and a
NAME>COMPILE-USER. SET-COMPSEM would change NAME>COMPILE-USER, but
not NAME>COMPILE-ORIGINAL. You would call NAME>COMPILE-ORIGINAL in
the code above, while NAME>COMPILE-USER would be called by system
words like the text interpreter and POSTPONE.

This would solve your immediate problem, but it would mean that your
extension would not compose with other extensions that use
SET-COMPSEM: E.g., if I have an extension that counts the number of
times the compilation semantics of this word is invoked, and then
apply your TRACK-DEPS extension to the word, the counting extension is
disabled. If the counting extension also uses NAME>COMPILE-ORIGINAL
instead of the previous NAME>COMPILE-USER action, then applying it
after TRACK-DEPS would disable TRACK-DEPS.

Therefore I think that just invoking the old NAME>COMPILE result
explicitly is the way to go. It's a little more complicated, but
composable.

[Krishna Myneni]

In the case of standard "COMPILE," the problem is that on a dual-xt
system, "COMPILE," operating on an xt is a fundamentally incorrect way
of compiling a named definition (word). It may still be used for
compiling the xt's obtained from :NONAME definitions, quotations, or
closures; however, compilation of all word definitions should involve
the name token (nt) and NAME>COMPILE followed by EXECUTE. I understand
your "hack" in Gforth is for the purpose of maintaining backwards
compatibility.

> But these problems are just like any other case where you
> inadvertently introduce endless recursion; e.g., such endless
> recursions happen easily if you try to reduce the number of primitives
> by replacing them with colon definitions in a cross-compiler that
> allows forward references. You just have to break the cycle, in the
> present case by explicitly performing the old action of NAME>COMPILE
> instead of just calling NAME>COMPILE.
>

Does SET-COMPSEM erase all memory of the original compilation semantics,
or is the old xt-comp still available? If not, we should be able to fix
this.

> Concerning SET-COMPSEM, one could imagine an extended header structure
> that also contains a COMPSEM field. SET-COMPSEM then would set that
> field and would change NAME>COMPILE to perform:
>
> : compsem-name>comp ( nt -- xt1 xt2 )
> >namehm @ >hmcompsem @ ['] execute ;
>
> But that still would not help with the code above, because
> NAME>COMPILE still would give you the new compilation semantics, which
> would again result in endless recursion.
>

..

>
> This would solve your immediate problem, but it would mean that your
> extension would not compose with other extensions that use
> SET-COMPSEM: E.g., if I have an extension that counts the number of
> times the compilation semantics of this word is invoked, and then
> apply your TRACK-DEPS extension to the word, the counting extension is
> disabled. If the counting extension also uses NAME>COMPILE-ORIGINAL
> instead of the previous NAME>COMPILE-USER action, then applying it
> after TRACK-DEPS would disable TRACK-DEPS.
>
> Therefore I think that just invoking the old NAME>COMPILE result
> explicitly is the way to go. It's a little more complicated, but
> composable.
>

Ok, I see that there is a problem with using NAME>COMPILE within the
the closure's xt, passed to SET-COMPSEM.

NAME>COMPILE should be used outside the closure to obtain the
compilation semantics at the time that TRACK-DEPS was invoked. Thus a
permanent data structure, 3 cells long, may be passed into the closure.
The data structure contains the nt of the word, and the original
compilation semantics, which consists of two cells of data: x xt. Then
the closure can invoke both the original compilation semantics and
append the extra actions needed to add information to the dependency table.

Here's my new version of TRACK-DEPS. It invokes NAME>COMPILE outside of
the closure to obtain and store the current compilation semantics for
the word before SET-COMPSEM is invoked. It should satisfy all cases,
including preserving any extensions to the compilation semantics prior
to invoking TRACK-DEPS. I've tested it on the single-xt+immediate flag
cases. So it appears that the previous xt-comp is still valid.

: track-deps ( -- )
latestnt dup track-dependency ( -- nt-track)

dup name>compile ( -- nt-track x xt )
3 cells allocate drop \ reserve memory for old compsem
dup >r 2! r@ 2 cells + ! r>
[n:d dup 2@ execute 2 cells + a@ latestnt add-dependency ;]
set-compsem ;


--
Krishna

[Krishna Myneni]

On 10/13/22 14:40, Krishna Myneni wrote:
> On 10/13/22 10:30, Anton Ertl wrote:

>> As it is, you may see some lossage in connection with, e.g., synonyms.
>> E.g., try
>>
>> : a ." a" ;
>> synonym b a track-deps
>> : c b ;
>> s" b" show-deps
>>
>> Does it show that c depends on b?

Yes, the revised version of TRACK-DEPS gives the following, showing that
c depends on b.

: a ." a" ; ok
synonym b a track-deps
*terminal*:59:11: warning: redefined b
locate1.fs:207:3: warning: original location ok
: c b ; ok
s" b" show-deps
c
ok

>
> : track-deps ( -- )
>     latestnt dup track-dependency  ( -- nt-track)
>     dup name>compile               ( -- nt-track x xt )
>     3 cells allocate drop          \ reserve memory for old compsem
>     dup >r 2! r@ 2 cells + ! r>
>     [n:d dup 2@ execute 2 cells + a@ latestnt add-dependency ;]
>     set-compsem ;
>
>

Using the revised version of TRACK-DEPS I decided to try it out on a
real application, Ian Osgood's chess game, fcp.f (ver 1.3). I marked
most word definitions, with the exception of a few output words, using
TRACK-DEPS.

Below are some interesting results:

curr-track . 204 ok \ constant MAX-TRACKS set to 250

The number of words being tracked is 204.

s" rank" show-deps
evalDR
evalLR
evalDP
evalLP
taxicab
taxicab
evalSetupSq
evalSetupSq
sqCastleMask
sqCastleMask
updatePawnFile
updatePawnFile
genDP
genDP
genLP
genLP
genCapsDP
genCapsLP
genDPCaptures
genLPCaptures
epdSq
cRank
ok

The word RANK is referenced in the above list of words. At first I was
confused by multiple appearances of the same word, but it is simply
because the dependent word references RANK multiple times. Hence, we can
count the number of references as well for each dependent word.

Words defined by the macro :INLINE are also able to be tracked.

s" rotate" show-deps
evalDB
evalDN
evalDP
?rotate
ok

Above, we see that there are four words which reference the word ROTATE.

s" tolower" show-deps
epd
char>piece
str>sq
keyHitQ?
piece
ok

Some of the words which reference the character case lowering word
TOLOWER. This is probably not a complete list since I didn't mark every
output word with TRACK-DEPS.

s" forEveryRow" show-deps
forEverySq
forEverySq
forEverySq
forEverySq
forEverySq
forEverySq
forEverySq
forEverySq

The word forEverySq appears to be the only word to reference
forEveryRow, and it does so eight times.

s" switchPieces" show-deps
epd
undoNullMove
makeNullMove
makeMove
takeBack
ok

s" mvFrom" show-deps
reps
makeMove
makeMove
takeBack
takeBack
takeBack
updatePawnFiles
genPushCapture
mvHistory
move
epCapSq
epSq
ok

s" .move" show-deps
ex
go
bkNode
curMove
pv
makeMove
moveList
ok

s" makeMove" show-deps
redo
ex
(mv)
go
line:
noLegalMoves?
_search
quiesce
ok

s" quiesce" show-deps
_search
_search
ok

s" _search" show-deps
thinker
failLowSearch
failHighSearch
ok

etc.

Obviously we may gain considerable insight into the structure of a
program by viewing the word dependencies in this manner. It is also
possible to recurse through the dependencies to follow the branches. For
example, a section of a branch may be gleaned from the results above:

mvFrom -> makeMove -> quiesce -> _search -> thinker

mvFrom is a primitive word in the program:

:inline mvFrom $FF AND ; ( mv -- sqFrom )

and THINKER is a high-level word in the program.

This shows the utility of the dependency tracking words, which
themselves are written in Forth source on a dual-xt system.

Below, I give the full listing of the current version of dependencies.4th.

--
Krishna

dependencies.4th

----
\ dependencies.4th
\
\ Demonstrate tracking first-level word dependencies
\ in a dual-xt system.
\
\ K. Myneni, 2022-10-10

\ Revised: 2022-10-13

\
\ This program is written for Gforth. It uses the following
\ non-standard words
\
\ LATESTNT ( -- nt )
\ SET-COMPSEM ( xt -- )
\ [N:D ( x -- )

require set-compsem.fs
synonym a@ @

250 constant MAX-TRACKS

64 constant MAX-DEPS
MAX-TRACKS MAX-DEPS * cells constant MAX-DTABLE-SIZE

create DEPENDENCY-TABLE MAX-DTABLE-SIZE allot
DEPENDENCY-TABLE MAX-DTABLE-SIZE erase

: th-track ( u -- a )
DEPENDENCY-TABLE swap MAX-DEPS * cells + ;

0 value curr-track

: get-track ( nt -- n ) \ return -1 if not found
curr-track 0 ?DO
I th-track a@ \ nt nt-track
over = IF drop I UNLOOP EXIT THEN
LOOP
drop -1 ;

: track-dependency ( nt -- )
curr-track th-track !
1 curr-track + to curr-track ;

: add-dependency ( nt-track nt-newdep -- )
swap get-track dup 0< IF
." Not tracking." cr abort
ELSE
th-track \ -- nt-dep a
BEGIN cell+ dup a@ 0= UNTIL !
THEN ;

\ Track first-level dependencies for last defined word
: track-deps ( -- )
latestnt dup track-dependency ( -- nt-track)

dup name>compile ( -- nt-track x xt )
3 cells allocate drop \ reserve memory for old compsem
dup >r 2! r@ 2 cells + ! r>
[n:d dup 2@ execute 2 cells + a@ latestnt add-dependency ;]
set-compsem ;

[Anton Ertl]

Krishna Myneni <krishna...@ccreweb.org> writes:
>On 10/13/22 10:30, Anton Ertl wrote:
>> If SET-OPTIMIZER would not change COMPILE,, it would not work (it
>> would fail to set the optimizer). For Gforth's current
>> implementation, if SET-COMPSEM did not change NAME>COMPILE, it would
>> not work (it would fail to change the compilation semantics).
>>
>
>In the case of standard "COMPILE," the problem is that on a dual-xt
>system, "COMPILE," operating on an xt is a fundamentally incorrect way
>of compiling a named definition (word).

If you mean that, e.g.,

: [compile,] compile, ; immediate
: foo [ ' s" ] [compile,] bla" ;

is not (and must not be) equivalent to

: foo s" bla" ;

i.e., performing the compilation semantics of S", you are correct.

It's also incorrect for performing the compilation semantics of

: bar ." bar" ; immediate

But

: foo1 [ ' bar ] [compile,] ;

appends the execution=interpretation=compilation semantics to foo1, so
it is equivalent to

: foo1 postpone bar ;

Ticking named words is unambiguous for most named words, and
COMPILE,ing the resulting xt is just as standard as EXECUTEing it.

>I understand
>your "hack" in Gforth is for the purpose of maintaining backwards
>compatibility.

I don't know what you mean with '"hack"'. Forth has COMPILE, for
appending semantics to the current definition, and NAME>COMPILE for
getting the compilation semantics of a named word. These are two
different jobs, and Gforth provides SET-OPTIMIZER and SET->COMP,
respectively, to provide implementations for these words for the
current definition. And that's the right way to do it.

Because one often follows the other, there have been attempts to use
one mechanism for both purposes:

* It is incorrect to use SET-OPTIMIZER for changing the compilation
semantics.

* If you use SET->COMP for changing the code generation, you don't
cover the cases that COMPILE, xts directly instead of going through
NAME>COMPILE, and you have to provide a fallback mechanism for
COMPILE,ing that; and if you implement [COMPILE], you also have to
tell it which uses of SET->COMP change the default compilation
semantics and which don't. You may think that you are saving
complexity by eliminating SET-OPTIMIZER, but the complexity pops up
elsewhere.

I think that providing both mechanisms is a good solution, because it
also provides a nice separation of concerns: If you want to change the
generated code without changing the semantics (i.e., if you want to
optimize), use SET-OPTIMIZER; if you want to change the compilation
semantics, use SET->COMP (or, by extension, SET-COMPSEM).

It also gives different results for your dependency-tracking (which is
outside semantics, so you can use either mechanism). Consider:

: bar ." bar" ; track-deps
: foo postpone bar postpone bar ; immediate
: bla foo foo ;
: blub [ ' bar compile, ] ;

If you track dependencies throug NAME>COMPILE as mechanism, you will
see that BAR is used twice in FOO. If you track dependencies using
"COMPILE,", you will see that BAR is used four times in BLA and one
time in BLUB.

>> You just have to break the cycle, in the
>> present case by explicitly performing the old action of NAME>COMPILE
>> instead of just calling NAME>COMPILE.
>>
>
>Does SET-COMPSEM erase all memory of the original compilation semantics,
>or is the old xt-comp still available? If not, we should be able to fix
>this.

The compilation semantics is specified by the >HM>COMP field (which
contains the xt performing NAME>COMPILE for the word), but may access
other parts of the word. SET-COMPSEM changes the >HM>COMP field, but
not the other parts of the word. Since you do not change the other
parts of the word, you could just remember the old contents of
>HM>COMP and execute that; that's what the untested version I posted
earlier does:

: track-deps ( -- )

latestnt dup track-dependency ( nt-track )
dup >namehm @ >hm>comp @ ( nt-track old-name>comp )
[d:d swap >r execute r> latestnt add-dependency ;] set->comp ;

Here I directly called SET->COMP instead of going through SET-COMPSEM.
Benefits: No additional closure from SET-COMPSEM (saving memory), and
it works correctly with [COMPILE].

>Ok, I see that there is a problem with using NAME>COMPILE within the
>the closure's xt, passed to SET-COMPSEM.
>
>NAME>COMPILE should be used outside the closure to obtain the
>compilation semantics at the time that TRACK-DEPS was invoked. Thus a
>permanent data structure, 3 cells long, may be passed into the closure.
>The data structure contains the nt of the word, and the original
>compilation semantics, which consists of two cells of data: x xt. Then
>the closure can invoke both the original compilation semantics and
>append the extra actions needed to add information to the dependency table.
>
>Here's my new version of TRACK-DEPS. It invokes NAME>COMPILE outside of
>the closure to obtain and store the current compilation semantics for
>the word before SET-COMPSEM is invoked. It should satisfy all cases,
>including preserving any extensions to the compilation semantics prior
>to invoking TRACK-DEPS. I've tested it on the single-xt+immediate flag
>cases. So it appears that the previous xt-comp is still valid.
>
>: track-deps ( -- )
> latestnt dup track-dependency ( -- nt-track)
> dup name>compile ( -- nt-track x xt )
> 3 cells allocate drop \ reserve memory for old compsem
> dup >r 2! r@ 2 cells + ! r>
> [n:d dup 2@ execute 2 cells + a@ latestnt add-dependency ;]
> set-compsem ;

Another way to do that is to simply pass the three values to the
closure. Gforth does not provide pure-stack closure construction
words for that, but unless you are a locals-hater, you may prefer the
following (untested):

: track-deps ( -- )
latestnt dup track-dependency ( nt-track)
dup name>compile ( nt-track x xt )
[{: nt-track x xt :}d x xt execute nt-track latestnt add-dependency ;]
set-compsem ;

or alternatively, turning xt into a defer-flavoured local:

: track-deps ( -- )
latestnt dup track-dependency ( nt-track)
dup name>compile ( nt-track x xt )
[{: nt-track x xt: xt :}d x xt nt-track latestnt add-dependency ;]
set-compsem ;

[Anton Ertl]

an...@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>It also gives different results for your dependency-tracking (which is
>outside semantics, so you can use either mechanism). Consider:
>
>: bar ." bar" ; track-deps
>: foo postpone bar postpone bar ; immediate
>: bla foo foo ;
>: blub [ ' bar compile, ] ;
>
>If you track dependencies throug NAME>COMPILE as mechanism, you will
>see that BAR is used twice in FOO. If you track dependencies using
>"COMPILE,", you will see that BAR is used four times in BLA and one
>time in BLUB.

Correction: How often you see it with the NAME>COMPILE approach
depends on the implementation of POSTPONE, in particular when it
performs NAME>COMPILE. The current implementation of POSTPONE in
Gforth performs NAME>COMPILE four times in this example, when
compiling BLA.

[Krishna Myneni]

The above is only true when you are not talking about dual-semantics
words, where the compilation semantics is not simply that of "ordinary
words" or of immediate words. Dual-semantics systems permit extended
compilation semantics. I think your term for such words is "combined
words". "COMPILE," is inadequate for combined words.

>> I understand
>> your "hack" in Gforth is for the purpose of maintaining backwards
>> compatibility.
>
> I don't know what you mean with '"hack"'. Forth has COMPILE, for
> appending semantics to the current definition, and NAME>COMPILE for
> getting the compilation semantics of a named word. These are two
> different jobs, and Gforth provides SET-OPTIMIZER and SET->COMP,
> respectively, to provide implementations for these words for the
> current definition. And that's the right way to do it.
>

I haven't implemented a dual-semantics system, so I won't strongly opine
on this other than to say, at the present time it does not seem to me
the right way to do it. "COMPILE," when used with NAME>INTERPRET, or
equivalently with '(tick) is fine when it is desired to append the
interpretation semantics, but for appending the compilation semantics of
a "combined word", one should use NAME>COMPILE and EXECUTE. Both of
these procedures must start with the name token (nt). After I implement
dual-semantics, I may be in a better position to understand your rationale.

> Because one often follows the other, there have been attempts to use
> one mechanism for both purposes:
>
> * It is incorrect to use SET-OPTIMIZER for changing the compilation
> semantics.
>
> * If you use SET->COMP for changing the code generation, you don't
> cover the cases that COMPILE, xts directly instead of going through
> NAME>COMPILE, and you have to provide a fallback mechanism for
> COMPILE,ing that; and if you implement [COMPILE], you also have to
> tell it which uses of SET->COMP change the default compilation
> semantics and which don't. You may think that you are saving
> complexity by eliminating SET-OPTIMIZER, but the complexity pops up
> elsewhere.
>

I think what I'm advocating is that, in a dual-semantics system, the
programmer should be aware of when to use "COMPILE," and when not to use
it -- namely use it only for those definitions for which a name token is
meaningless, and for words (named definitions) always use a compilation
chain which starts with the name token. When I use "COMPILE,"
deliberately I don't want the Forth system to alter the xt I give to
"COMPILE," behind the scenes due to the presence of arbitrary
compilation semantics.

You're entirely right that the complexity pops up elsewhere if you
eliminate SET-OPTIMIZER -- it rests on the programmer to use the
appropriate compilation sequence.

> I think that providing both mechanisms is a good solution, because it
> also provides a nice separation of concerns: If you want to change the
> generated code without changing the semantics (i.e., if you want to
> optimize), use SET-OPTIMIZER; if you want to change the compilation
> semantics, use SET->COMP (or, by extension, SET-COMPSEM).
>

Optimization is simply one rationale for changing the compilation
semantics. I don't feel strongly about SET-OPTIMIZER as a substitute for
SET-COMPSEM. What concerns me is that it modifies the behavior of
"COMPILE,". That seems to me to be unnecessary.

> It also gives different results for your dependency-tracking (which is
> outside semantics, so you can use either mechanism). Consider:
>
> : bar ." bar" ; track-deps
> : foo postpone bar postpone bar ; immediate
> : bla foo foo ;
> : blub [ ' bar compile, ] ;
>
> If you track dependencies throug NAME>COMPILE as mechanism, you will
> see that BAR is used twice in FOO. If you track dependencies using
> "COMPILE,", you will see that BAR is used four times in BLA and one
> time in BLUB.
>

Trying this gives the following (with the current version of
dependencies.4th):

include dependencies.4th ok
: bar ." bar" ; track-deps ok
: foo postpone bar postpone bar ; immediate track-deps ok
: bla foo foo ; ok
: blub [ ' bar compile, ] ; ok
s" foo" show-deps
bla
bla
ok
s" bar" show-deps
bla
bla
bla
bla
ok

This shows me that FOO is used twice in BLA and that BAR is used four
times in BLA. POSTPONE works nicely with dependency tracking.

Ticking BAR and compiling it with "COMPILE," in BLUB does not allow for
tracking the dependency, but this is ok because the proper way to
compile BAR in a dual semantics system is not with "COMPILE,". If
instead we use

: blub [ s" bar" find-name name>compile execute ] ;

Then, we get the desired list of words dependent on BAR.

s" bar" show-deps
blub
bla
bla
bla
bla
ok

A word such as COMPILE-NAME as a shortcut for NAME>COMPILE EXECUTE helps
shorten this. Also, we can be specific about which wordlist the name is
compiled from using FIND-NAME-IN. In short, '(tick) and "COMPILE," have
their purpose in a dual-semantics system but it is not a sequence which
should be applied to compiling a word.

>>> You just have to break the cycle, in the
>>> present case by explicitly performing the old action of NAME>COMPILE
>>> instead of just calling NAME>COMPILE.
>>>
>>
>> Does SET-COMPSEM erase all memory of the original compilation semantics,
>> or is the old xt-comp still available? If not, we should be able to fix
>> this.
>

..

..

I don't have anything against locals. Will test this and if it works I
can update TRACK-DEPS to use the easier to read version above, since we
are dealing with Gforth-specific code. Incidentally, I'd like to
acknowledge your help in the comments of dependencies.4th unless you
object. It's obvious in the newsgroup thread that I couldn't have made
this work in Gforth without your input.

--
Krishna

[Krishna Myneni]

On 10/14/22 12:16, Krishna Myneni wrote:
> On 10/14/22 02:21, Anton Ertl wrote:

>>
>> I don't know what you mean with '"hack"'.  Forth has COMPILE, for
>> appending semantics to the current definition, and NAME>COMPILE for
>> getting the compilation semantics of a named word.  These are two
>> different jobs, and Gforth provides SET-OPTIMIZER and SET->COMP,
>> respectively, to provide implementations for these words for the
>> current definition.  And that's the right way to do it.
>>

> ... "COMPILE," when used with NAME>INTERPRET, or

> equivalently with '(tick) is fine when it is desired to append the
> interpretation semantics, but for appending the compilation semantics of

> a "combined word", one should use NAME>COMPILE and EXECUTE. ...

My statement above is nonsense. What I should have said is that
compilation of a word should not be done with '(tick) and "COMPILE," but
start with the name token, e.g. S" <name>" FIND-NAME and then use
NAME>COMPILE and EXECUTE to append the appropriate semantics.

--
Krishna

[Krishna Myneni]

On 10/14/22 12:56, Krishna Myneni wrote:

>
> ... What I should have said is that

> compilation of a word should not be done with '(tick) and "COMPILE," but
> start with the name token, e.g. S" <name>" FIND-NAME and then use
> NAME>COMPILE and EXECUTE to append the appropriate semantics.
>

A concrete illustration in Gforth is useful for understanding the above
statement.

require set-compsem.fs

I. Compilation semantics for an "ordinary word".

: foo ." foo" ;

s" foo" find-name name>compile .s
\ prints: <2> 140188477601736 140188477079032 ok

' compile, . \ prints: 140188477079032 ok
' foo . \ prints: 140188477601736 ok

Thus, we see that from FOO's nt, NAME>COMPILE returns on the stack

nt -- xt-foo xt-compile,

These two xt's describe the compilation semantics for FOO (or what has
been called "default compilation semantics"). The interpretation
semantics of FOO will be compiled when EXECUTE is performed with the two
xt's on the stack.

II. Compilation semantics for an IMMEDIATE word

: foo ." foo" ; IMMEDIATE

s" foo" find-name name>compile .s
\ prints: <2> 140188477601840 140188477026696 ok

' execute . \ prints: 140188477026696 ok
' foo . \ prints: 140188477601840 ok

From FOO's nt, NAME>COMPILE returns on the stack

nt -- xt-foo xt-execute

The compilation semantics for this version of FOO are the same as its
interpretation semantics. The interpretation semantics of FOO will be
executed when EXECUTE is performed with these two xt's on the stack.

III. Compilation semantics for a dual-semantics word

: foo ." foo" ; \ define word with default compilation semantics
compsem: 'f' emit 'o' emit 'o' emit ;

This example is neither case I nor case II.

s" foo" find-name name>compile .s
\ prints: <2> 140188477602024 140188477026696

' execute . \ prints: 140188477026696 ok
' foo . \ prints: 140188477601944 ok

From FOO's nt, NAME>COMPILE returns on the stack

nt -- xt-? xt-execute

xt-? represents the compilation semantics of FOO and does not involve
xt-foo. xt-? will be performed when EXECUTE is performed with these two
xt's on the stack. The output will look the same as for case II, because
of the way we defined COMPSEM: ... ; to give the same output behavior,
but could also have done

: foo ." foo" ;
compsem: ." bar" ;

Then,

foo foo ok
: test foo ; bar \ bar is printed when FOO is compiled into TEST


The above three cases demonstrate that the sequence

s" <name>" FIND-NAME NAME>COMPILE EXECUTE

will perform the specified compilation semantics for all cases. Cases I
and II apply to single-xt + immediate flag Forth systems, while Cases I
-- III apply to dual-xt systems.


--
Krishna

[Krishna Myneni]

On 10/14/22 17:00, Krishna Myneni wrote:
..
> A concrete illustration in Gforth ...

>
> require set-compsem.fs
>
> I. Compilation semantics for an "ordinary word".
>
> : foo ." foo" ;
>
> s" foo" find-name name>compile .s

..

> Thus, we see that from FOO's nt, NAME>COMPILE returns on the stack
>
> nt -- xt-foo xt-compile,
>

..

>
> II. Compilation semantics for an IMMEDIATE word
>
> : foo ." foo" ; IMMEDIATE
>
> s" foo" find-name name>compile .s

..

> From FOO's nt, NAME>COMPILE returns on the stack
>
> nt -- xt-foo xt-execute
>

..

> III. Compilation semantics for a dual-semantics word
>
> : foo ." foo" ;  \ define word with default compilation semantics
> compsem: 'f' emit 'o' emit 'o' emit ;
>
> This example is neither case I nor case II.
>
> s" foo" find-name name>compile .s

..

> From FOO's nt, NAME>COMPILE returns on the stack
>
> nt -- xt-? xt-execute
>
> xt-? represents the compilation semantics of FOO and does not involve

> xt-foo. ...

>
> s" <name>" FIND-NAME NAME>COMPILE EXECUTE
>
> will perform the specified compilation semantics for all cases. Cases I
> and II apply to single-xt + immediate flag Forth systems, while Cases I
> -- III apply to dual-xt systems.
>

It is further worth noting that on an *ideal* dual-xt system, as opposed
to a system which allows for dual-semantics, NAME>COMPILE would simply
have the following stack diagram for all three cases:

nt -- xt-comp xt-execute \ xt-comp represents the compilation semantics

Then the xt-execute may be dispensed with and NAME>INTERPRET and some
differently named NAME>COMPILE both would simply return a single xt. In
the *ideal* dual-xt system, we should be able to define

: NAME>COMPSEM ( nt -- xt-comp ) NAME>COMPILE DROP ;

EXECUTE(ing) xt-comp will perform the compilation semantics. This is not
how Gforth works at present (see case I). But, in my opinion, it would
be a cleaner and more consistent design, and easier to understand for
the programmer.

For NAME>INTERPRET, xt is simply xt-interp which is the same value as
returned by '(tick) in all three cases, and this is the behavior in Gforth:

case I:

: foo ." foo" ; ok
s" foo" find-name name>interpret ' foo = . -1 ok
ok

case II:
: foo ." foo" ; IMMEDIATE
s" foo" find-name name>interpret ' foo = . -1 ok
ok

case III:
: foo ." foo" ;
compsem: 'f' emit 'o' emit 'o' emit ; ok
s" foo" find-name name>interpret ' foo = . -1 ok


The issue of Gforth's non-standard word SET-OPTIMIZER altering the
behavior of standard "COMPILE," is a separate issue also, but one on
which our we have already had sufficient discussion.

--
Krishna

[Ruvim]

On 2022-10-10 15:44, Krishna Myneni wrote:
>
> A word definition may be immediately followed by the word TRACK-DEPS
> which will set its compilation semantics to not only compile its
> interpretation semantics

By this wording you just skew the language of the standard.

By default, not the interpretation semantics, but the execution
semantics of a word are appended (compiled):

| 3.4.3.3 Compilation semantics
| Unless otherwise specified in a "Compilation:" section
| of the glossary entry, the compilation semantics of a Forth
| definition shall be to append its *execution semantics*
| to the execution semantics of the current definition.

(emphasized by me)


There is a number of cases when interpretation semantics for a word are
undefined by the standard, but execution semantics are defined.

But when interpretation semantics are defined via execution semantics,
it's still incorrect to say that they are equivalent in the general case.

Otherwise you introduce such a beast as interpretation semantics in
compilation state — i.e., a behavior that takes place when the Forth
text interpreter encounters a word name in interpretation state, if the
Forth text interpreter in compilation state.

It's absurd. It's like to ask what is your behavior when it rains, if
there isn't raining?



[...]

--
Ruvim

[Krishna Myneni]

On 10/19/22 04:16, Ruvim wrote:
> On 2022-10-10 15:44, Krishna Myneni wrote:
>>
>> A word definition may be immediately followed by the word TRACK-DEPS
>> which will set its compilation semantics to not only compile its
>> interpretation semantics
>
> By this wording you just skew the language of the standard.
>

The original wording of the statement is flawed. A more accurate
statement follows:

A word definition may be immediately followed by the word TRACK-DEPS

which will set its new compilation semantics to append the original
compilation semantics and additional semantics to keep track of the

words into which it is compiled.

The above statement explicitly indicates that IMMEDIATE may be used
between the end of the definition ";" and TRACK-DEPS

> By default, not the interpretation semantics, but the execution
> semantics of a word are appended (compiled):
>
> | 3.4.3.3 Compilation semantics
> | Unless otherwise specified in a "Compilation:" section
> | of the glossary entry, the compilation semantics of a Forth
> | definition shall be to append its *execution semantics*
> | to the execution semantics of the current definition.
>
> (emphasized by me)
>
>

The wording of the standard reflects a strict adherence to the single-xt
model, while we are discussing here a dual-xt model, in which the term
"execution semantics" can be omitted. In a single-xt system the
compilation semantics and the interpretation semantics are strongly
connected, while in a dual-xt system they are free to be specified
separately.

In the ideal implementation of a dual-xt system, every word provides
execution tokens for both interpretation semantics and compilation
semantics. When the text interpreter parses the word name it will
determine whether to perform the interpretation semantics or the
compilation semantics based on STATE. There is no need to talk of
"execution semantics". For :NONAME definitions, quotations, and other
compositions which result in an execution token, we may simply speak of
the semantics resulting from applying EXECUTE to xt.

With the existence of dual-semantics Forth systems which are able to
properly run code written for single-xt systems, I would argue that the
standard's definition of "compilation semantics", given in 3.4.3.3 is
becoming obsolete.

> There is a number of cases when interpretation semantics for a word are
> undefined by the standard, but execution semantics are defined.
>

In a dual-semantics system, let's say you want to declare a word which
is not allowed to be used in interpretation state. We may do so, as
follows (in Gforth):

: NAME -14 throw ;
\ -14 is throw code for interpreting a compile-only word
COMPSEM: ( i*x -- j*x ) ... ;

Obviously this word has both interpretation semantics (throw an error)
and compilation semantics.

> But when interpretation semantics are defined via execution semantics,
> it's still incorrect to say that they are equivalent in the general case.
>

The "execution semantics" of a word is STATE-dependent when parsed and
executed by the text interpreter. In a dual-xt system we have only
"interpretation semantics" *and* "compilation semantics". Then, the
correct way to compile a word is through the name token, from which
xt-comp would be obtained via NAME>COMPILE and then executed.

> Otherwise you introduce such a beast as interpretation semantics in
> compilation state — i.e., a behavior that takes place when the Forth
> text interpreter encounters a word name in interpretation state, if the
> Forth text interpreter in compilation state.
>

"Interpretation semantics" and "compilation semantics" only refer to
what the text interpreter does with the name of a word.

compilation semantics: The behavior of a Forth definition when its name
is encountered by the *text interpreter* in compilation state.

interpretation semantics: The behavior of a Forth definition when its
name is encountered by the *text interpreter* in interpretation state.

The text interpreter's behavior, as dictated by the standard, is very
clear when it encounters a name in a given state -- perform the
interpretation semantics *or* the compilation semnatics. As far as the
text interpreter is concerned, there is no such beast as you are claiming.

If you want to perform the compilation semantics in interpretation
state, you can do that even on a (Forth 200x) standard single-xt system:

s" <name>" FIND-NAME NAME>COMPILE EXECUTE

It will likely result in inconsistent behavior on different Forth
systems. This is not the point of a dual-xt system though. The point is
to have less constraint on the compilation semantics of a word. We have
illustrated the usefulness for tasks like optimizing the implementation
of the compilation semantics, and for appending additional semantics to
the original compilation semantics (e.g. for instrumentation or for
dependency tracking).

--
Krishna

[Ruvim]

On 2022-10-19 22:33, Krishna Myneni wrote:
> On 10/19/22 04:16, Ruvim wrote:

[...]

>> By default, not the interpretation semantics, but the execution
>> semantics of a word are appended (compiled):
>>
>> | 3.4.3.3 Compilation semantics
>> | Unless otherwise specified in a "Compilation:" section
>> | of the glossary entry, the compilation semantics of a Forth
>> | definition shall be to append its *execution semantics*
>> | to the execution semantics of the current definition.
>>
>> (emphasized by me)
>>
>>
>
> The wording of the standard reflects a strict adherence to the single-xt
> model,

Probably it does, in few places (like "find"), but not in this place.

> while we are discussing here a dual-xt model, in which the term
> "execution semantics" can be omitted. In a single-xt system the
> compilation semantics and the interpretation semantics are strongly
> connected,
> while in a dual-xt system they are free to be specified separately.

You are wrong: in a single-xt system they can be specified separately too.

: compilation ( -- flag ) state @ 0<> ;

: dual` ( xt.int xt.comp "name" -- )
2>r : 2r> 2lit,
[: ( i*x xt.int xt.comp -- j*x )
compilation if nip else drop then execute
;] compile, postpone ; immediate
;

:noname ( -- ) ." foo" ;
:noname ( -- ) ." bar" ;
dual` foo

Where xt.int and xt.comp are execution tokens for definitions that are
*used* to perform the interpretation semantics and compilation semantics
for /name/.

They don't identify the corresponding behaviors in the general case.
(it's true for dual-xt systems too).

> In the ideal implementation of a dual-xt system, every word provides
> execution tokens for both interpretation semantics and compilation
> semantics. When the text interpreter parses the word name it will
> determine whether to perform the interpretation semantics or the
> compilation semantics based on STATE. There is no need to talk of
> "execution semantics".

I totally disagree.

> For :NONAME definitions, quotations, and other
> compositions which result in an execution token, we may simply speak of
> the semantics resulting from applying EXECUTE to xt.

These semantics are included into the notion of "execution semantics".
There is no any profit to use the phrase:
"the semantics resulting from applying EXECUTE to xt of a definition"
instead of the phrase:
"the execution semantics of a definition"

> With the existence of dual-semantics Forth systems which are able to
> properly run code written for single-xt systems, I would argue that the
> standard's definition of "compilation semantics", given in 3.4.3.3 is
> becoming obsolete.

It makes standard systems non standard.

>> There is a number of cases when interpretation semantics for a word
>> are undefined by the standard, but execution semantics are defined.
>>
>
> In a dual-semantics system, let's say you want to declare a word which
> is not allowed to be used in interpretation state. We may do so, as
> follows (in Gforth):
>
> : NAME  -14 throw ;
> \ -14 is throw code for interpreting a compile-only word
> COMPSEM: ( i*x -- j*x ) ... ;
>
> Obviously this word has both interpretation semantics (throw an error)
> and compilation semantics.

Your example is not a case when interpretation semantics for a word are
undefined by the standard, and execution semantics are defined.


[...]


--
Ruvim

[none albert]

In article <tiptve$4kuu$2...@dont-email.me>,
Krishna Myneni <krishna...@ccreweb.org> wrote:
<SNIP>

>In a dual-semantics system, let's say you want to declare a word which
>is not allowed to be used in interpretation state. We may do so, as
>follows (in Gforth):
>
>: NAME -14 throw ;
>\ -14 is throw code for interpreting a compile-only word
>COMPSEM: ( i*x -- j*x ) ... ;
>
>Obviously this word has both interpretation semantics (throw an error)
>and compilation semantics.

You contradict yourself. You stated that the word is not allowed to be
used in interpretation state.
So it cannot be even used to throw an error. 1)
(Language precision, important in this context, "this word cannot
be used in a valid ISO94 program." is correct.)
<SNIP>

1) Anton Ertl would say that "pigs can fly out your nose", useful
excuse to compile something insensible to spite you.

>--
>Krishna

Groetjes Albert
--
"in our communism country Viet Nam, people are forced to be
alive and in the western country like US, people are free to
die from Covid 19 lol" duc ha
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

[Krishna Myneni]

On 10/20/22 03:18, Ruvim wrote:
> On 2022-10-19 22:33, Krishna Myneni wrote:
>> On 10/19/22 04:16, Ruvim wrote:
>
> [...]

>
>

>> while we are discussing here a dual-xt model, in which the term
>> "execution semantics" can be omitted. In a single-xt system the
>> compilation semantics and the interpretation semantics are strongly
>> connected, while in a dual-xt system they are free to be specified
>> separately.
>
> You are wrong: in a single-xt system they can be specified separately too.
>
>   : compilation ( -- flag ) state @ 0<> ;
>
>   : dual` ( xt.int xt.comp "name" -- )
>     2>r :   2r> 2lit,
>     [: ( i*x xt.int xt.comp -- j*x )
>       compilation if nip else drop then execute
>     ;] compile, postpone ; immediate
>   ;
>
>   :noname ( -- ) ." foo" ;
>   :noname ( -- ) ." bar" ;
>   dual` foo
>
> Where xt.int and xt.comp are execution tokens for definitions that are
> *used* to perform the interpretation semantics and compilation semantics
> for /name/.
>

It's clever, but its state dependency means that FOO is no longer a
"first-class" word in Forth. For example, if we want to execute xt.comp
from the interpreter (the action of which is not ambiguous),

s" foo" find-name name>compile execute \ prints "foo"

which gives an incorrect result with your definition (it prints "foo").
FIND-NAME and NAME>COMPILE are standard words. You have created a
state-smart word which no longer behaves as expected for standard
operations. In a true dual-semantics system, the above sequence should
print "bar", e.g. in Gforth,

: foo ." foo" ; ok
compsem: ." bar" ; ok
s" foo" find-name name>compile execute \ prints "bar"

> They don't identify the corresponding behaviors in the general case.
> (it's true for dual-xt systems too).
>

I don't know what you mean by the above statement.

>
>
>> In the ideal implementation of a dual-xt system, every word provides
>> execution tokens for both interpretation semantics and compilation
>> semantics. When the text interpreter parses the word name it will
>> determine whether to perform the interpretation semantics or the
>> compilation semantics based on STATE. There is no need to talk of
>> "execution semantics".
>
> I totally disagree.
>

I would ask you to elaborate, but it's clear from your prior posts that
your disagreement rests on the basis of the term, "execution semantics,"
being used in the current standard (which appears to be circular logic
to me).

>> For :NONAME definitions, quotations, and other compositions which
>> result in an execution token, we may simply speak of the semantics
>> resulting from applying EXECUTE to xt.
>
> These semantics are included into the notion of "execution semantics".
> There is no any profit to use the phrase:
>   "the semantics resulting from applying EXECUTE to xt of a definition"
> instead of the phrase:
>   "the execution semantics of a definition"
>

A word definition has two types of semantics in a dual-semantics system,
by definition.

>
>
>
>> With the existence of dual-semantics Forth systems which are able to
>> properly run code written for single-xt systems, I would argue that the
>> standard's definition of "compilation semantics", given in 3.4.3.3 is
>> becoming obsolete.
>
> It makes standard systems non standard.
>

A standard fixed in stone is the sign-post for a dead language.
Standards for languages which are still used evolve to take into account
common practice and new functionality desired by programmers and users.
One type of new functionality is the ability to define "first-class"
words (see above) with flexible specification of compilation behavior.
This has led to the development of new Forth systems which adopt
dual-semantics, either with full dual-xt implementation or with mixed
single-xt and dual-xt implementation.

>
>>> There is a number of cases when interpretation semantics for a word
>>> are undefined by the standard, but execution semantics are defined.
>>>
>>
>> In a dual-semantics system, let's say you want to declare a word which
>> is not allowed to be used in interpretation state. We may do so, as
>> follows (in Gforth):
>>
>> : NAME  -14 throw ;
>> \ -14 is throw code for interpreting a compile-only word
>> COMPSEM: ( i*x -- j*x ) ... ;
>>
>> Obviously this word has both interpretation semantics (throw an error)
>> and compilation semantics.
>
> Your example is not a case when interpretation semantics for a word are
> undefined by the standard, and execution semantics are defined.
>

A system still has to do something when you try to invoke interpretation
semantics in the case they are undefined. The semicolon ";" operator has
undefined interpretation semantics. Try executing it in the interpreter
in your system. The above example is no different, except it gives the
programmer full control of the system response when a word designed to
be used only in compilation state is executed in interpretation state.
In contrast, single-xt systems give the programmer no control unless a
state-smart word is used, resulting in second-class words.


--
Krishna

[Krishna Myneni]

On 10/20/22 06:25, albert wrote:
> In article <tiptve$4kuu$2...@dont-email.me>,
> Krishna Myneni <krishna...@ccreweb.org> wrote:
> <SNIP>
>> In a dual-semantics system, let's say you want to declare a word which
>> is not allowed to be used in interpretation state. We may do so, as
>> follows (in Gforth):
>>
>> : NAME -14 throw ;
>> \ -14 is throw code for interpreting a compile-only word
>> COMPSEM: ( i*x -- j*x ) ... ;
>>
>> Obviously this word has both interpretation semantics (throw an error)
>> and compilation semantics.
>
> You contradict yourself. You stated that the word is not allowed to be
> used in interpretation state.
> So it cannot be even used to throw an error. 1)
> (Language precision, important in this context, "this word cannot
> be used in a valid ISO94 program." is correct.)
> <SNIP>

My intended meaning by "not allowed to be used in interpretation state"
is that the word takes no action other than indicating such usage is an
error. If the word doesn't notify the user, the system may or may not.
See my response to ruvim.

--
Krishna

[Ruvim]

On 2022-10-20 11:41, Krishna Myneni wrote:
> On 10/20/22 03:18, Ruvim wrote:
>> On 2022-10-19 22:33, Krishna Myneni wrote:

[...]

>>> while in a dual-xt system they are free to be specified
>>> separately.
>>
>> You are wrong: in a single-xt system they can be specified separately
>> too.
>>
>>    : compilation ( -- flag ) state @ 0<> ;
>>
>>    : dual` ( xt.int xt.comp "name" -- )
>>      2>r :   2r> 2lit,
>>      [: ( i*x xt.int xt.comp -- j*x )
>>        compilation if nip else drop then execute
>>      ;] compile, postpone ; immediate
>>    ;
>>
>>    :noname ( -- ) ." foo" ;
>>    :noname ( -- ) ." bar" ;
>>    dual` foo
>>
>> Where xt.int and xt.comp are execution tokens for definitions that are
>> *used* to perform the interpretation semantics and compilation
>> semantics for /name/.
>>
>
> It's clever, but its state dependency means that FOO is no longer a
> "first-class" word in Forth.

What is a first-class word?

> For example, if we want to execute xt.comp
> from the interpreter (the action of which is not ambiguous),
>
> s" foo" find-name name>compile execute   \ prints "foo"
>
> which gives an incorrect result with your definition (it prints "foo").

How do you know?

If a single-xt system provides a correct "name>compile", as:

: name>compile ( nt -- x xt )
name> ['] execute-compilatively
;

your test prints "bar".

> FIND-NAME and NAME>COMPILE are standard words. You have created a
> state-smart word which no longer behaves as expected for standard
> operations. In a true dual-semantics system, the above sequence should
> print "bar", e.g. in Gforth,
>
> : foo ." foo" ;  ok
> compsem: ." bar" ;  ok
> s" foo" find-name name>compile execute  \ prints "bar"

So, it is the same behavior that the system demonstrates when it
encounters the word name "foo" in compilation state.

And it's an expected behavior for your test, regardless how a word "foo"
is defined. Isn't it?


[...]


--
Ruvim

[Krishna Myneni]

On 10/20/22 15:25, Ruvim wrote:
> On 2022-10-20 11:41, Krishna Myneni wrote:
>> On 10/20/22 03:18, Ruvim wrote:
>>> On 2022-10-19 22:33, Krishna Myneni wrote:
>
> [...]
>>>> while in a dual-xt system they are free to be specified separately.
>>>
>>> You are wrong: in a single-xt system they can be specified separately
>>> too.
>>>
>>>    : compilation ( -- flag ) state @ 0<> ;
>>>
>>>    : dual` ( xt.int xt.comp "name" -- )
>>>      2>r :   2r> 2lit,
>>>      [: ( i*x xt.int xt.comp -- j*x )
>>>        compilation if nip else drop then execute
>>>      ;] compile, postpone ; immediate
>>>    ;
>>>
>>>    :noname ( -- ) ." foo" ;
>>>    :noname ( -- ) ." bar" ;
>>>    dual` foo
>>>
>>> Where xt.int and xt.comp are execution tokens for definitions that
>>> are *used* to perform the interpretation semantics and compilation
>>> semantics for /name/.
>>>
>>
>> It's clever, but its state dependency means that FOO is no longer a
>> "first-class" word in Forth.
>
> What is a first-class word?
>

For the purpose of this discussion, let us agree that a "first class
word" is one which behaves as expected when the following standard words
are used in connection with the word:

-- '(tick)
-- [']
-- postpone
-- name>interpret
-- name>compile
-- compile,
-- [compile] (if your system provides it)

>
>
>> For example, if we want to execute xt.comp from the interpreter (the
>> action of which is not ambiguous),
>>
>> s" foo" find-name name>compile execute   \ prints "foo"
>>
>> which gives an incorrect result with your definition (it prints "foo").
>
> How do you know?
>

Output from Gforth:

: compilation state @ 0<> ; ok
: dual` 2>r : 2r> postpone 2literal [: compilation if nip else drop then
execute ;] compile, postpone ; immediate ; ok
:noname ." foo" ; ok 1
:noname ." bar" ; ok 2
dual` foo ok
s" foo" find-name name>compile execute foo ok

Output from kForth (single-xt system):

: compilation state @ 0<> ;
ok
: dual` 2>r : 2r> postpone 2literal [: compilation if nip else drop then
execute ;] compile, postpone ; immediate ;
ok
:noname ." foo" ;
ok
:noname ." bar" ;
ok
dual` foo

ok
s" foo" find-name name>compile execute

foo ok

> If a single-xt system provides a correct "name>compile", as:
>
>   : name>compile ( nt -- x xt )
>     name> ['] execute-compilatively
>   ;
>
> your test prints "bar".
>
>

I don't know what your NAME> and EXECUTE-COMPILATIVELY are supposed to
do -- they are not standard words. NAME>COMPILE in both Gforth and
kForth follow the behavior prescribed by the Forth 200x standard:

15.6.2.1909.10 NAME>COMPILE
“name-to-compile”
TOOLS EXT
( nt – – x xt ) x xt represents the compilation semantics of the word
nt. The returned xt has the stack effect ( i * x x – – j * x ) .
Executing xt consumes x and performs the compilation semantics of the
word represented by nt.

NAME>COMPILE returns the compilation semantics, which when executed
should print "bar". Your implementation of dual-semantics words using
DUAL` does not work with NAME>COMPILE under Gforth or kForth, and I
suspect it will not work on most other Forth systems.

>
>
>> FIND-NAME and NAME>COMPILE are standard words. You have created a
>> state-smart word which no longer behaves as expected for standard
>> operations. In a true dual-semantics system, the above sequence should
>> print "bar", e.g. in Gforth,
>>
>> : foo ." foo" ;  ok
>> compsem: ." bar" ;  ok
>> s" foo" find-name name>compile execute  \ prints "bar"
>
>
> So, it is the same behavior that the system demonstrates when it
> encounters the word name "foo" in compilation state.
>

Printing "bar" is the correct behavior for my test -- Gforth provides
the correct behavior when the compilation semantics are specified by its
COMPSEM: word. I've demonstrated that your DUAL` does not work with
NAME>COMPILE on a standard system.

--
Krishna

Krishna

[Ruvim]

On 2022-10-20 22:01, Krishna Myneni wrote:
> On 10/20/22 15:25, Ruvim wrote:
[...]

> I've demonstrated that your DUAL` does not work with
> NAME>COMPILE on a standard system.

I cannot agree. There should be admitted either an ambiguous condition,
or an incompletely standard system.

But, before I try to convince you, could you please answer to my
question that you missed:

>> And it's an expected behavior for your test, regardless how a word "foo"
>> is defined. Isn't it?

Regardless how the word "foo" is defined, should your test demonstrate
the same behavior that the system demonstrates when the Forth text
interpreter encounters the word name "foo" in compilation state?


--
Ruvim

[Krishna Myneni]

Perhaps you can pose your question in the form of a test, using only
standard words, so that there is no possibility of misunderstanding.

Also, you may want to apply my test of your word DUAL` in other standard
systems to see what they do.

--
Krishna

[Krishna Myneni]

On 10/21/22 10:11, Krishna Myneni wrote:
..

> Also, you may want to apply my test of your word DUAL` in other standard
> systems to see what they do.
>

..

Also, here is the test under VfxLin64 (Vfx Forth for Linux;
noncommercial version). It appears to be missing the word FIND-NAME but
the nt of a word may be obtained through its xt via >NAME. Then, the
sequence " ' foo >NAME" is equivalent to s" foo" FIND-NAME.

: compilation state @ 0<> ; ok
: dual` 2>r : 2r> postpone 2literal
[: compilation if nip else drop then execute ;]
compile, postpone ; immediate ; ok

:noname ." foo" ; ok-1
:noname ." bar" ; ok-2
dual` foo ok

\ The test prints FOO
' foo >name name>compile execute foo ok

A dual-semantics word may be defined in Vfx Forth using NDCS: as follows:

: foo ." foo" ; ok
ndcs: ." bar" ; ok

Then,

' foo >name name>compile execute bar ok

gives the correct output, "bar".

Thus, Gforth, Vfx Forth, and kForth all give the same test output for
the test. kForth, being a single-xt system, does not presently allow for
dual-semantics; however, Gforth and Vfx Forth do provide a way to define
arbitrary compilation semantics for a word. NAME>COMPILE returns the
proper xt for the compilation semantics of a word, when the word's
compilation semantics is set (using COMPSEM: in Gforth, and NDCS: in Vfx
Forth).

--
Krishna

[Ruvim]

On 2022-10-21 15:11, Krishna Myneni wrote:
> On 10/21/22 06:23, Ruvim wrote:
>> On 2022-10-20 22:01, Krishna Myneni wrote:
>>> On 10/20/22 15:25, Ruvim wrote:
>> [...]

>>>> And it's an expected behavior for your test, regardless how a word
>>>> "foo" is defined. Isn't it?
>>
>> Regardless how the word "foo" is defined, should your test demonstrate
>> the same behavior that the system demonstrates when the Forth text
>> interpreter encounters the word name "foo" in compilation state?
>>
>
> Perhaps you can pose your question in the form of a test, using only
> standard words, so that there is no possibility of misunderstanding.

Actually this my question is close to my another one[1], where I ask for
a counter example to my statement. Maybe that question is more clear for
understanding.

[1] Re: POSTPONE and semantics of immediate words, 2022-10-20 19:26Z
news://tis7e6$bmv6$1...@dont-email.me
https://groups.google.com/g/comp.lang.forth/c/rjbRkJPLMpY/m/UwKFSswDAAAJ



My point is that for *some* definitions of "foo", and your test in
interpretation state:

s" foo" find-name name>compile execute

this test on your Forth system (and many other) demonstrates a behavior
that is substantively distinct from a behavior that the Forth system
demonstrates when the text interpreter encounters the word "foo" in
compilation state. (even with reservation concerning the different input
source, that can be easy made identical).


Whether such a difference is acceptable or not (according to the current
wording of the "name>compile" glossary entry) depends on understanding
of the "compilation semantics" notion (see the mentioned thread
"POSTPONE and semantics of immediate words").





-----


If you want a test, it's following.

Let us have the helpers:

: leave-compilation ( -- ) postpone [ ;
: enter-compilation ( -- ) ] ;

The behavior that the Forth system demonstrates when the text
interpreter encounters the word name "foo" in compilation state can be
identified by the execution semantics of the following word:

: behave-via-evaluate ( i*x -- j*x )
enter-compilation \ start in compilation state
s" foo" evaluate
leave-compilation \ revert interpretation state at the end
;

The behavior of your test can be identified by the word:

: behave-via-name-compile ( i*x -- j*x )
leave-compilation \ ensure interpretation state
s" foo" find-name name>compile execute-foo
leave-compilation \ ensure interpretation state at the end
;

We ensure interpretation state at the beginning of this word since your
test is started in interpretation state:

s" foo" find-name name>compile execute

For the sake of simplicity, the input source was not made identical in
these two ways of performing the behavior, but it can easy corrected.


According the the term definitions for the term "compilation semantics",
the words "behave-via-evaluate" and "behave-via-name-compile" should
always produce the same effects (with mentioned reservation about
different input sources).



--
Ruvim

[Ruvim]

On 2022-10-24 19:16, Ruvim wrote:
[...]

... regardless how "foo" is defined, if it can be a part of a standard
program, and there is no ambiguous conditions.


--
Ruvim

[Anton Ertl]

I give concrete examples examples involving S" (a word with
non-default non-immediate compilation semantics) and the immediate
word BAR. What is not true in your opinion?

>Dual-semantics systems permit extended
>compilation semantics. I think your term for such words is "combined
>words". "COMPILE," is inadequate for combined words.

I fail to see the inadequacy.

>>> I understand
>>> your "hack" in Gforth is for the purpose of maintaining backwards
>>> compatibility.
>>
>> I don't know what you mean with '"hack"'. Forth has COMPILE, for
>> appending semantics to the current definition, and NAME>COMPILE for
>> getting the compilation semantics of a named word. These are two
>> different jobs, and Gforth provides SET-OPTIMIZER and SET->COMP,
>> respectively, to provide implementations for these words for the
>> current definition. And that's the right way to do it.
>>
>
>I haven't implemented a dual-semantics system, so I won't strongly opine
>on this other than to say, at the present time it does not seem to me
>the right way to do it. "COMPILE," when used with NAME>INTERPRET, or
>equivalently with '(tick) is fine when it is desired to append the
>interpretation semantics, but for appending the compilation semantics of
>a "combined word", one should use NAME>COMPILE and EXECUTE.

Sure, but that does not make COMPILE, "inadequate". In particular, in
Gforth NAME>COMPILE for words with default compilation semantics is

: default-name>comp ( nt -- w xt )
name>interpret ['] compile, ;

[Actually, this is not quite true. With the current Gforth header
layout NAME>INTERPRET is a noop for the words with DEFAULT-NAME>COMP
as NAME>COMPILE implementation, so the code above has been simplified
to

' compile, AConstant default-name>comp ( nt -- w xt )
]

Essentially, NAME>COMPILE and COMPILE, are two words for two different
purposes that work at two different times. When you text-interpret,
e.g., DUP in compilation state, the text interpreter first gets the NT
of DUP, then performs NAME>COMPILE on that, and then EXECUTEs the
result of that; this EXECUTE performs "COMPILE," and that then appends
the execution semantics of DUP to the current definition (in an
optimized way, thanks to SET-OPTIMIZER).

For another example, if you text-interpret IF in compilation state,
NAME>COMPILE produces the xt1 xt2, where xt2 is the xt of EXECUTE and
xt1 is the xt of

: IF ( compilation -- orig ; run-time f -- ) \ core
POSTPONE ?branch >mark? ; immediate restrict

No COMPILE, involved, right? Wrong! When xt1 is performed, it
performs the compilation semantics of ?BRANCH, which leads to
COMPILE,ing the xt of ?BRANCH.

>> Because one often follows the other, there have been attempts to use
>> one mechanism for both purposes:
>>
>> * It is incorrect to use SET-OPTIMIZER for changing the compilation
>> semantics.
>>
>> * If you use SET->COMP for changing the code generation, you don't
>> cover the cases that COMPILE, xts directly instead of going through
>> NAME>COMPILE, and you have to provide a fallback mechanism for
>> COMPILE,ing that; and if you implement [COMPILE], you also have to
>> tell it which uses of SET->COMP change the default compilation
>> semantics and which don't. You may think that you are saving
>> complexity by eliminating SET-OPTIMIZER, but the complexity pops up
>> elsewhere.
>>
>
>I think what I'm advocating is that, in a dual-semantics system, the
>programmer should be aware of when to use "COMPILE," and when not to use
>it -- namely use it only for those definitions for which a name token is
>meaningless,

There is no reason for this restriction.

>and for words (named definitions) always use a compilation
>chain which starts with the name token.

Apart from the rare (and non-standard) uses of LASTXT, you can always
follow a chain back to the name token: The only way to get xts of such
words are NAME>INTERPRET, NAME>COMPILE, and '/[']; and the latter can
be seen as using NAME>INTERPRET as its factor.

>>When I use "COMPILE,"
>deliberately I don't want the Forth system to alter the xt I give to
>"COMPILE," behind the scenes due to the presence of arbitrary
>compilation semantics.

I have no idea why you would be worried about that.

>> I think that providing both mechanisms is a good solution, because it
>> also provides a nice separation of concerns: If you want to change the
>> generated code without changing the semantics (i.e., if you want to
>> optimize), use SET-OPTIMIZER; if you want to change the compilation
>> semantics, use SET->COMP (or, by extension, SET-COMPSEM).
>>
>
>Optimization is simply one rationale for changing the compilation
>semantics.

It should not be.

>I don't feel strongly about SET-OPTIMIZER as a substitute for
>SET-COMPSEM.

It's not a substitute for SET-COMPSEM. Using SET-OPTIMIZER where
SET-COMPSEM is appropriate (i.e., for changing compilation semantics)
is incorrect.

>What concerns me is that it modifies the behavior of
>"COMPILE,". That seems to me to be unnecessary.

If "," is fine for your system as implementation of "COMPILE,", you
don't need SET-OPTIMIZER (and you can implement it as

: set-optimizer drop ;

). Otherwise, that's what it is there for.

[Anton Ertl]

Krishna Myneni <krishna...@ccreweb.org> writes:
>What I should have said is that
>compilation of a word should not be done with '(tick) and "COMPILE," but
>start with the name token, e.g. S" <name>" FIND-NAME and then use
>NAME>COMPILE and EXECUTE to append the appropriate semantics.

These are two different operations in general:

: foo [ ' <word> compile, ] ;

makes FOO perform the interpretation semantics of <word>.

: bar [ s" <word>" find-name name>compile execute ] ;

is equivalent to

: bar <word> ;

If you want to perform the former operation, you will in general not
get it by doing the latter operation. So you cannot replace the
former usage with the latter usage (or vice versa) in the general
case.

[Anton Ertl]

Krishna Myneni <krishna...@ccreweb.org> writes:
>It is further worth noting that on an *ideal* dual-xt system, as opposed
>to a system which allows for dual-semantics, NAME>COMPILE would simply
>have the following stack diagram for all three cases:
>
>nt -- xt-comp xt-execute \ xt-comp represents the compilation semantics
>
>Then the xt-execute may be dispensed with and NAME>INTERPRET and some
>differently named NAME>COMPILE both would simply return a single xt. In
>the *ideal* dual-xt system, we should be able to define
>
>: NAME>COMPSEM ( nt -- xt-comp ) NAME>COMPILE DROP ;
>
>EXECUTE(ing) xt-comp will perform the compilation semantics. This is not
>how Gforth works at present (see case I). But, in my opinion, it would
>be a cleaner and more consistent design, and easier to understand for
>the programmer.

I thought so at some time (maybe 20 years ago). I no longer do so.
The reasons are:

* Implementation cost: For every word X with default compilation
semantics, you would need something like:

:noname ['] X compile, ;

to have an xt-comp that NAME>COMPSEM would return. Ok, one can work
out ways to make this more space-efficient, but that eats up the
little understandability advantage that NAME>COMPSEM gives over
NAME>COMPILE.

* This approach emphasizes words with non-default non-immediate
compilation semantics. I think that such words should be avoided,
and with recognizers we have found ways to get rid of most of them;
unfortunately, they rear their ugly heads in people wanting
interpretive quotations and closures, so we may not be able to get
rid of them (and of course we still need them for backwards
compatibility with Forth-2012).

* And while we are at backwards compatibility, there is still
[COMPILE]. In Gforth it can recognize that a word has default
compilation semantics by looking at xt2 returned by
NAME>COMPILE ( nt -- xt1 xt2 )
EXIT is the exception that proves the rule.

[Krishna Myneni]

On 10/29/22 08:41, Anton Ertl wrote:
> Krishna Myneni <krishna...@ccreweb.org> writes:
>> What I should have said is that
>> compilation of a word should not be done with '(tick) and "COMPILE," but
>> start with the name token, e.g. S" <name>" FIND-NAME and then use
>> NAME>COMPILE and EXECUTE to append the appropriate semantics.
>
> These are two different operations in general:
>
> : foo [ ' <word> compile, ] ;
>
> makes FOO perform the interpretation semantics of <word>.
>

It is not true that FOO will perform the interpretation semantics of
<word> in Gforth if SET-OPTIMIZER has been used to change the
implementation of its compilation semantics. This is exactly my
objection to SET-OPTIMIZER -- it changes what COMPILE, does.

--
Krishna

[Ruvim]

On 2022-10-29 13:41, Anton Ertl wrote:
[...]

>
> : bar [ s" <word>" find-name name>compile execute ] ;
>
> is equivalent to
>
> : bar <word> ;

Yes, they shall be equivalent according to the specification for
"name>compile", but in Gforth v0.7.9_20221027 these expressions are not
equivalent (in some cases, e.g. for immediate STATE-dependent words).

For example:

: foo state @ 0= lit, ; immediate

: bar [ s" foo" find-name name>compile execute ] ; \ (1)
bar . \ prints "-1" in Gforth

: bar foo ; \ (2)
bar . \ prints "0" in Gforth


So, equivalence of the given expressions does not hold in Gforth.


A practical question: do you have an example when you need the
expression (1) to produce a different result than the expression (2)?


--
Ruvim

[Ruvim]

It's a weak argument, since it can be easily eliminated by a proper
specification for "set-optimizer" (or a similar word).

For example:

set-xt-compiler ( xt.compiler xt -- )

Associate xt.compiler with xt. After that xt.compiler is performed to
append the execution semantics identified by xt to the current definition.
An ambiguous condition exists if the execution semantics identified by
xt.compiler are not equivalent to appending the execution semantics
identified by xt to the current definition.


In this variant we don't change compilation semantics of any word, and
it's impossible without falling into an ambiguous condition.




My objection is to the Anton's (and your also) wording only — the use of
"interpretation semantics" in the sense of "execution semantics".

If we use the term "interpretation semantics" in the meaning that is
distinct from its normative notion (according to the Definitions of
terms), we have to introduce other name for this original normative
notion. It only produces more confusions and forces us to use more
verbose wordings to clarify what namely is meant.


--
Ruvim

[Krishna Myneni]

On 10/29/22 16:41, Ruvim wrote:
> On 2022-10-29 16:10, Krishna Myneni wrote:
>> On 10/29/22 08:41, Anton Ertl wrote:
>>> Krishna Myneni <krishna...@ccreweb.org> writes:
>>>> What I should have said is that
>>>> compilation of a word should not be done with '(tick) and "COMPILE,"
>>>> but
>>>> start with the name token, e.g. S" <name>" FIND-NAME and then use
>>>> NAME>COMPILE and EXECUTE to append the appropriate semantics.
>>>
>>> These are two different operations in general:
>>>
>>> : foo [ ' <word> compile, ] ;
>>>
>>> makes FOO perform the interpretation semantics of <word>.
>>>
>>
>> It is not true that FOO will perform the interpretation semantics of
>> <word> in Gforth if SET-OPTIMIZER has been used to change the
>> implementation of its compilation semantics. This is exactly my
>> objection to SET-OPTIMIZER -- it changes what COMPILE, does.
>
> It's a weak argument, since it can be easily eliminated by a proper
> specification for "set-optimizer" (or a similar word).

> ...

You can't unbreak "COMPILE," with a specification for SET-OPTIMIZER. I
expect "COMPILE," per the standard, to append exactly the semantics
given by xt, in this case the xt corresponding to the interpretation
semantics of the word. Although you are fixed on the term "execution
semantics", for a dual-semantics system, '(tick) returns the
interpretation semantics for the word.

The user (or programmer) must rely on "COMPILE," appending exactly the
semantics of the xt that is specified, and not substitute some other xt.

6.2.0945
COMPILE, “compile-comma” CORE EXT
Interpretation: Interpretation semantics for this word are undefined.
Execution: ( xt – – )
Append the execution semantics of the definition represented by xt to
the execution semantics of the current definition.

Not doing so causes problems such as the incompatibility I ran into
between using COMPSEM: in Gforth vs NDCS: in Vfx Forth. Gforth's
COMPSEM: does not change the action of "COMPILE," but Vfx Forth's NDCS:
does, making changing the compilation semantics non-portable between the
two systems if "COMPILE," is used to define the new semantics.

--
Krishna

[Ruvim]

On 2022-10-29 23:14, Krishna Myneni wrote:
> On 10/29/22 16:41, Ruvim wrote:
>> On 2022-10-29 16:10, Krishna Myneni wrote:
>>> On 10/29/22 08:41, Anton Ertl wrote:
>>>> Krishna Myneni <krishna...@ccreweb.org> writes:
>>>>> What I should have said is that
>>>>> compilation of a word should not be done with '(tick) and
>>>>> "COMPILE," but
>>>>> start with the name token, e.g. S" <name>" FIND-NAME and then use
>>>>> NAME>COMPILE and EXECUTE to append the appropriate semantics.
>>>>
>>>> These are two different operations in general:
>>>>
>>>> : foo [ ' <word> compile, ] ;
>>>>
>>>> makes FOO perform the interpretation semantics of <word>.
>>>>
>>>
>>> It is not true that FOO will perform the interpretation semantics of
>>> <word> in Gforth if SET-OPTIMIZER has been used to change the
>>> implementation of its compilation semantics. This is exactly my
>>> objection to SET-OPTIMIZER -- it changes what COMPILE, does.
>>
>> It's a weak argument, since it can be easily eliminated by a proper
>> specification for "set-optimizer" (or a similar word).
>> ...
>
> You can't unbreak "COMPILE," with a specification for SET-OPTIMIZER.
> I expect "COMPILE," per the standard, to append exactly the semantics
> given by xt,

In what case does it append the semantics given by xt *inexactly*? Could
you provide an example?

Have a look:

: tuck ( x2 x1 -- x1 x2 x1 ) swap over ;

: compile-tuck-v1 ['] tuck compile, ;

: compile-tuck-v2 ['] swap compile, ['] over compile, ;

Are you agree that "compile-tuck-v1" and "compile-tuck-v2" both append
the same semantics?

> in this case the xt corresponding to the interpretation
> semantics of the word.

It makes impression that (according to your understanding) the
interpretation semantics of this word and the execution semantics of
this word are *not* the same in this case. It is right?

If they are the same, could you provide an example when they are not the
same?

> Although you are fixed on the term "execution
> semantics", for a dual-semantics system, '(tick) returns the
> interpretation semantics for the word.

I just use the language of the standard, and the terminology of the
standard.

The language of the standard is agnostic on whether a system employs the
dual-xt or single-xt approach. There is no one reason to expose the
internal details of an dual-xt system when we talk about behavior of a
system that is observable by a standard program.

> The user (or programmer) must rely on "COMPILE," appending exactly the
> semantics of the xt that is specified, and not substitute some other xt.
>
> 6.2.0945
> COMPILE, “compile-comma” CORE EXT
> Interpretation: Interpretation semantics for this word are undefined.
> Execution: ( xt – – )
> Append the execution semantics of the definition represented by xt to
> the execution semantics of the current definition.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

BTW, take a look, during compilation of a definition, namely the
execution semantics of this definition are created. And Tick returns an
identifier namely of these semantics (this behavior).

Please, don't tell me that in a dual-xt system it creates the
interpretation semantics — it does not matter how your system
intertwists things under the hood — it's its internal business.

From a standard program point of view, this behavior is execution
semantics of the word, which in some cases can be equivalent to the
interpretation semantics of the word, of course. But there is no one
case when it's incorrect to call this behavior "execution semantics".



--
Ruvim

[Krishna Myneni]

See

https://groups.google.com/g/comp.lang.forth/c/dh347IHLDtw/m/YQVN1g-kBAAJ


..

>
>> in this case the xt corresponding to the interpretation semantics of
>> the word.
>
> It makes impression that (according to your understanding) the
> interpretation semantics of this word and the execution semantics of
> this word are *not* the same in this case. It is right?
>
> If they are the same, could you provide an example when they are not the
> same?
>

Even in a single-xt system with only one xt per word, it makes sense to
talk of interpretation semantics and compilation semantics. Depending on
the immediate flag, the compilation semantics are derived from either
executing or compiling the xt. Both involve using the single xt as an
argument to the xt of EXECUTE or COMPILE, . Thus tick can always simply
return the single xt.

For a dual-xt system, one must choose which xt, xt-interp or xt-comp is
returned by '(tick). It makes sense for backwards compatibility to
return xt-interp for application of '(tick) (and for [']). Here it makes
no sense to say that '(tick) returns the "execution semantics" of the
word. '(tick) specifically returns xt-interp. The language of the
standard is insufficient (imprecise) to describe the action of '(tick)
for a dual-xt system.

6.1.0070
6. CORE Word Set
“tick”
’
CORE
( “<spaces>name” – – xt )
Skip leading space delimiters. Parse name delimited by a space. Find
name and return xt, the execution token for name. An ambiguous condition
exists if name is not found. When interpreting, ’ xyz EXECUTE is
equivalent to xyz.

>
>
>
>> Although you are fixed on the term "execution semantics", for a
>> dual-semantics system, '(tick) returns the interpretation semantics
>> for the word.
>
> I just use the language of the standard, and the terminology of the
> standard.
>
> The language of the standard is agnostic on whether a system employs the
> dual-xt or single-xt approach. There is no one reason to expose the
> internal details of an dual-xt system when we talk about behavior of a
> system that is observable by a standard program.
>

It may be agnostic, but it is not sufficiently precise for a dual-xt
implementation.

>
>
>> The user (or programmer) must rely on "COMPILE," appending exactly the
>> semantics of the xt that is specified, and not substitute some other xt.
>>
>> 6.2.0945
>> COMPILE, “compile-comma” CORE EXT
>> Interpretation: Interpretation semantics for this word are undefined.
>> Execution: ( xt – – )
>> Append the execution semantics of the definition represented by xt to
>> the execution semantics of the current definition.
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> BTW, take a look, during compilation of a definition, namely the
> execution semantics of this definition are created. And Tick returns an
> identifier namely of these semantics (this behavior).
>
> Please, don't tell me that in a dual-xt system it creates the
> interpretation semantics — it does not matter how your system
> intertwists things under the hood — it's its internal business.
>
> From a standard program point of view, this behavior is execution
> semantics of the word, which in some cases can be equivalent to the
> interpretation semantics of the word, of course.  But there is no one
> case when it's incorrect to call this behavior "execution semantics".
>

I've pretty much said all I have to say on both the matters of how I
think "COMPILE," should behave, and why "execution semantics" is no
longer an adequate term to use in the specification of some words, if
the standard is to be applicable to both single-xt and dual-xt system.
My views are expressed within my last three topical posts on c.l.f.

There appears to be no immediate resolution for a consensus on these
topics, which is fine. However, I won't engage in an endless discussion
of these topics.

--
Krishna

[Ruvim]

| If SET-OPTIMIZER is used to alter
| the compilation semantics of a word

If "set-optimizer" is proper specified, it's impossible to use it to
alter the compilation semantics for a word in a standard program.
Concerning nonstandard programs — they are not in the scope of the
standard at all.

> ..
>>
>>> in this case the xt corresponding to the interpretation semantics of
>>> the word.
>>
>> It makes impression that (according to your understanding) the
>> interpretation semantics of this word and the execution semantics of
>> this word are *not* the same in this case. It is right?
>>
>> If they are the same, could you provide an example when they are not
>> the same?
>>
>
> Even in a single-xt system with only one xt per word, it makes sense to
> talk of interpretation semantics and compilation semantics. Depending on
> the immediate flag, the compilation semantics are derived from either
> executing or compiling the xt. Both involve using the single xt as an
> argument to the xt of EXECUTE or COMPILE, . Thus tick can always simply
> return the single xt.
>


> For a dual-xt system, one must choose which xt, xt-interp or xt-comp is
> returned by '(tick).

In my dual-xt system one xt identifies the execution semantics of a
word, and another one is used to perform the compilation semantics for
the word. This way better fits the standard model.

> It makes sense for backwards compatibility to
> return xt-interp for application of '(tick) (and for [']). Here it makes
> no sense to say that '(tick) returns the "execution semantics" of the
> word. '(tick) specifically returns xt-interp.

> The language of the
> standard is insufficient (imprecise) to describe the action of '(tick)
> for a dual-xt system.

The standard should not describe a particular dual-xt system. But the
system, to be a standard system, should fit what the standard describes.
If the system does not fit that, then the system just is not standard.



[...]

>
> I've pretty much said all I have to say on both the matters of how I
> think "COMPILE," should behave, and why "execution semantics" is no
> longer an adequate term to use in the specification of some words, if
> the standard is to be applicable to both single-xt and dual-xt system.
> My views are expressed within my last three topical posts on c.l.f.

If it is no longer adequate, it's unclear since what a moment (and what
was changed in that moment).

I still don't understand your grounds. And some simple closed questions
(yes/no) have remained unanswered.

From my point of view, "compile," and "execution semantics" are pretty
adequate in the model that the standard describes.

> There appears to be no immediate resolution for a consensus on these
> topics, which is fine. However, I won't engage in an endless discussion
> of these topics.

Thank you for the discussion, it was useful anyway :)


--
Ruvim

[Krishna Myneni]

On 11/1/22 03:18, Ruvim wrote:
> On 2022-10-30 20:02, Krishna Myneni wrote:

>   | If SET-OPTIMIZER is used to alter
>   | the compilation semantics of a word
>
> If "set-optimizer" is proper specified, it's impossible to use it to
> alter the compilation semantics for a word in a standard program.
> Concerning nonstandard programs — they are not in the scope of the
> standard at all.

> ...

Anton corrected me earlier when I claimed SET-OPTIMIZER (in Gforth) is
used to alter the compilation semantics of a word. Although it *can*
alter the compilation semantics of a word, it is intended to be used to
change the implementation of the compilation semantics, i.e. append the
same semantics but which executes more efficiently. In Gforth,
SET-COMPSEM is used to change the compilation semantics of a word in a
dual-semantics system. Doing so may well be outside of the scope of the
current standard, but, as we have seen it can be a useful tool to extend
the capabilities of a Forth system.

> From my point of view,  "compile," and "execution semantics" are pretty
> adequate in the model that the standard describes.
>
>
>> There appears to be no immediate resolution for a consensus on these
>> topics, which is fine. However, I won't engage in an endless
>> discussion of these topics.
>
>
> Thank you for the discussion, it was useful anyway :)
>
>

Finding a commonly agreed-upon way of talking about topics like
dual-semantics systems seems to be challenging. I have no doubt the
discussion will come up again, but it's worthwhile to take a break from
it too.


--
Krishna