STATE IMMEDIATE and POSTPONE

[Mark Wills]

I thought this article might stimulate some debate!

------------------------------------------------------------------------

All you need to know about STATE, IMMEDIATE and POSTPONE
Jack Brien
Forthwrite FIG-UK Newsletter, Issue 106, April 2000

The issues which Jack reviews here by special invitation have been the
source of confusion and many postings to comp.lang.forth. Jack
provides us with the definitive reference to this thorny topic.

Using and Controlling STATE
---------------------------
STATE is the flag which tells programs whether the text interpreter is
interpreting or compiling (i.e. making a new definition).

STATE @ IF compiling ELSE interpreting THEN

While compiling, STATE can be temporarily turned off with [ to begin
interpreting, and restored with ] to return to compiling. This may be
done, for example, to calculate a literal value, e.g.

VARIABLE FOO
...
: BAR ... [ foo @ ] LITERAL ;
: BAR1 ... foo @ ;

"Foo @" is interpreted at compile time in the definition of BAR, and
the result compiled by LITERAL. In the definition of BAR1, "foo @"
is compiled to be executed later, at run time. BAR always returns the
same initial value of FOO, whereas BAR1 returns whatever FOO holds
when it is executed.

' name returns name's execution token (xt) which represents its
behaviour
' name EXECUTE performs name's behaviour

EXECUTE does not either know or care about the current STATE, so,
given the same xt, it will perform the same behaviour whether
interpreting or compiling. That's how the text interpreter treats
IMMEDIATE words, and normal words when interpreting. If it encounters
a normal word while compiling, the text interpreter adds its behaviour
to the current definition. The Standard word to do this, given an xt,
is COMPILE, .

Immediate Words
---------------
When an immediate word is met during compilation, it is executed
instead of compiled. They are easy to create - all you have to do is
use the word IMMEDIATE directly after the definition. (IMMEDIATE
changes the most recent definition). You can have IMMEDIATE colon
definitions, IMMEDIATE variables, IMMEDIATE constants, IMMEDIATE
anything that has a name. FIND can distinguish between immediate and
normal words. It returns the xt and a flag of -1 for normal words; the
xt and a flag of 1 for immediate words. That is the only difference
between normal and immediate words. Neither EXECUTE nor COMPILE, know
or care if the xt they receive comes from a normal or an immediate
word.

Here's a simple text interpreter (so simple, it gives up at the first
unknown word):

BEGIN
BL WORD FIND DUP WHILE
STATE @ IF \ compiling?
-1 = IF \ normal word?
COMPILE, ELSE \ add its behaviour to the definition
EXECUTE THEN \ immediate - perform its behaviour
ELSE \ interpreting
EXECUTE THEN \ perform its behaviour
REPEAT
2DROP

That's all you need to create a simple compiler on most Forths. In
fact, you would also need just one immediate word - [ - to switch from
compiling to interpreting. Normal words can switch in the other
direction. But it would be more convenient to have a number of
immediate words:

*Structure Words: BEGIN , DO , IF , THEN , etc.
*Literal Words: ['] , [CHAR] , LITERAL , S" etc.
*Words to stop compiling:[ , ;
*Comment Words: ( , \ , [IF] etc.

Compile-Only Words
------------------
However, you will notice that most of these words have behaviours that
make no sense while interpreting. Only the comment words can actually
be executed regardless of STATE. The rest are "compile-only" words
which should never be invoked (by executing them either directly, or
in a definition in which they've been compiled) when interpreting.
It's possible to implement compile-only words as normal or immediate
definitions (and most Forth systems do) but you have to be careful
never to invoke them in the state for which they were not intended.

*****************************************************
Rule 1:
Never try to get or use the xt of a compile-only word
*****************************************************

It's not good practice - and on some systems it may not even be
possible - to re-define compile-only words. Fortunately, these
restrictions only matter if you want to write your own compiling text
interpreter - and that's outside the scope of the present article.

Using POSTPONE
--------------
There is a Standard way to get a compile-only word's compilation
action:

POSTPONE name

POSTPONE itself is also compile-only. If name is compile-only, its
compilation action is added to the current definition. If name is
immediate, code equivalent to ' name COMPILE, will be executed at
compile-time.

If name is a normal word, will be code equivalent to ' name COMPILE,
will be executed at run-time. In any case, code compiled by POSTPONE
should itself never be invoked while interpreting. The only possible
exception is where name is an immediate word:

: MY( POSTPONE ( ; IMMEDIATE

will work on every system I know, because it could be equally well
defined as:

: MY( ['] ( EXECUTE ; IMMEDIATE

which is better practice.

So, does the use of POSTPONE make a definition compile-only? Not
quite. A simple example comes from the Standard document:

: NOP : POSTPONE ; IMMEDIATE ; NOP ALIGN NOP ALIGNED

NOP is a defining word that creates do-nothing colon definitions and
makes them immediate, so they do nothing regardless of STATE (they are
never compiled into other words). Here STATE gets switched when NOP
executes - : turns it on, and the POSTPONEd ; turns it off . So
although NOP performs the compilation action of ; it does so when
STATE is ON.

**************************************************************
Rule 2: Words defined with POSTPONE are usually compile-only.
**************************************************************

The same principle is used in so-called STATE-smart words.

State-Smart Words
-----------------
STATE-smart words are a high-level convenience. They are good when you
don't want the user to have to remember two separate words - one for
interpreting and one for compiling. So F83 had a word ASCII which
could be defined:

: ASCII STATE @ IF POSTPONE [CHAR] ELSE CHAR THEN ; IMMEDIATE

The compilation action of ASCII is guarded by the test of STATE. When
compiling, it's equivalent to [CHAR]. When interpreting, ASCII invokes
CHAR instead. It's also safe to POSTPONE ASCII and other STATE-smart
words, so long as you observe the restriction on never invoking the
resulting code while interpreting. Then what you have POSTPONEd will
always use the compilation action, since STATE will always be on when
it's invoked.

Otherwise, here's an example of what can go wrong:

Say you want a word CONTROL that returns a control code instead of an
ASCII value.

Interpret version:

: CONTROL POSTPONE ASCII \ OK. ASCII is an immediate word
>CONTROL ; \ Convert the ASCII value returned

Test it and it works. CONTROL invokes CHAR while interpreting, and it
also compiles into other definitions which also work while
interpreting. However because ASCII is state- smart, the phrase
POSTPONE ASCII can lead to problems later on. For example, we can
build a compile-only version [CONTROL] - by analogy with

: [CHAR] CHAR POSTPONE LITERAL ; IMMEDIATE

we get:


: [CONTROL] CONTROLPOSTPONE LITERAL ; IMMEDIATE

Doesn't work as you might expect. CONTROL only invokes CHAR if STATE
is OFF. When it is executed by [CONTROL] it will invoke the
compilation action of [CHAR], which is certainly not what is required.

It wouldn't have helped to write


: CONTROL ['] ASCII EXECUTE ;

['] , ' or FIND cannot unaided distinguish a STATE-smart word from
any other immediate word, and so extract the STATE-dumb interpretation
action. There's no easy way to invoke the interpretation action of a
STATE-smart while compiling. The safest option is to treat STATE-smart
words like compile-only words and not mess with their
xt's.

*************************************************************************
Rule 3: State-smart words defined with POSTPONE are always compile-
only.
*************************************************************************

COMPINT below is an un-easy option, inspired1 by Anton Ertl's combined
words at http://www.complang.tuwien.ac.at/forth/combined.zip It's
messy, but it's a good illustration of the power of POSTPONE. COMPINT
is an enhanced POSTPONE which will avoid these problems:

: [[ POSTPONE [ ; \ Turn STATE off at run time, not compile time

: COMPINT \ Compile code equivalent to the interpret
\ action of a STATE-smart word
' \ Get the xt
POSTPONE STATE POSTPONE @ POSTPONE IF
POSTPONE [[ \ Code to turn STATE off if it is ON
DUP COMPILE, \ Compile the xt
POSTPONE ] \ and restore STATE
POSTPONE ELSE \ Interpreting already; don't mess with STATE

1 See also Anton Ertl's comments http://www.complang.tuwien.ac.at/forth/dpans-
html/comment-semantics.html. and Mitch Bradley's proposal on
'syntactic elements' ftp://ftp.minerva.com/pub/x3j14/proposal/99-032-proposal-impl-q5-7-8-9.txt
for a different perspective.

COMPILE, \ Just do it
POSTPONE THEN ; IMMEDIATE

Used as : CONTROL COMPINT ASCII >CONTROL ; this produces the code:


: CONTROL STATE @ IF [[ ASCII ] ELSE ASCII THEN >CONTROL ;

***********************************************
Rule 4: COMPINT is a safe version of POSTPONE.
***********************************************

Better still, avoid state-smartness altogether and just factor the
interpretation action into a definition of its own and compile that
instead!

This brings us to the last and simplest rule:

**********************************
Rule 5: Avoid state-smart words.
**********************************