evolution of forth to OK, aha, ColorForth and reasons/results etc
peekay
i guess this will help many new colorForthers to understand
and trace the historical lineage .. tho it may be more convenient
to NOT get into it in the initial learning stages
so the messsage is re-posted with a fresh subject line
please bear with me for the repetition
(the other subject line is better closed/deleted due to the
concept which was misunderstood by me)
> MarkWills wrote:
> > Any idea quite what Chuck meant when he said 'sourceless'? Did his
> > 'programs' just consist of a list of addresses entered into memory?
> > Effectively op-codes? It seems a very painful way to program.
His programs were native code, factored as Forth but without source
code or compiler. He originally called it 3/Forth as in 3/4 of
Forth.
Forth minus source code and compiler and with a key table interpreter
instead of a text string interpreter. Code was entered via the
hex editor tool in the OK tool suite. The MISC design made the
process easy. Programming a Pentium in hex IS only for bare metal
types not programmers used to more abstractions.
> The program editor of OK included a disassembler, so you could disassemble
> some object code, edit it, and reassemble it. To be honest, it was so
> painful that even Chuck Moore declared it a failure and moved on to
> ColorForth, which again has source code.
> --
> Bernd Paysan
> "If you want it done right, you have to do it yourself"http://www.jwdt.com/~paysan/
Not quite. There was no assembler.
There was a name dictionary and a decompiler
that could show dissasembled native opcodes
and named Forth word calls. But the
output was never captured or fed
back to an assembler or compiler. The programmer
was the compiler and it was all mental.
There was a hex editor and a decompiler
but one had to enter word names into
the name dictionary by hand too to use them
with the decompiler tool.
There was a seven key and timeout input
routine with eight vectored addresses
the eighth one being a time-out.
There was a hex editor used to enter inlined
opcodes for primitives like DUP DROP OVER
etc. and subroutine calls or jumps
for high level words just like the
machineForth compiler below colorforth.
There was font editor to create the fonts used.
There was a decompiler and
application but that was it. There was no
assembler, no compiler, no source.
Dr. Ting did write a 386 OK manual available from Offete.
The way Chuck put it, he wanted to explore
the notion of reduced size of source
code to programs and when carried to its
conclusion the minimum source code
needed is zero. In fact the experiment
worked pretty well as an existence
proof of how little code was needed
to generate what were in effect Forth
systems without compilers or source
code and only a seven key input table
as the interpreter.
When I wrote the first p21 simulator,
s21, I used what was a sourceless
approach quite independently to what
Chuck was doing and just by cooincedence.
I wanted an educational tool that
was obvious and intuitive that even a
child could use. With so few opcodes
in MISC this was pretty easy.
The user interface it used was thumbwheels.
Since there were so few opcodes I used a thumbwheel
on each of the five-bit opcode fields so you could
create a packed opcode like "dup !r+ !r+ drop"
and see the hex and symbolic representation and
single step opcodes and see all the contents of all
the registers change. (admittedly five bits at a
time was not the best way
to enter address fields for branch opcodes.)
Unlike OK it could capture disassembled
code to a file that could be fed
back to an assember. But one just
created a forth program using the mouse
on some thumbwheels in s21. But in OK
one actually entered code through
the hex editor. It is what is known as
bare metal programming. Being
the compiler was not hard for Chuck, and
being a MISC compiler is not a complicated task.
It isn't true that it was abandoned because
it was painful or difficult.
It was in fact easy and reasonably productive as a way to enter code.
At least for Chuck who has bootstrapped
many systems and is comfortable
working at or below the level of a hex editor (rememeber that you
have
to start at an even lower level than that) and that Chuck
demonstrated
the bootstrap binary editor and binary video dump programs he made
for
p21 before he wrote the hex tools.
The shortcoming of the sourceless
programming approach was that without
source it was not possible to specify what we think of as compile
time
vs runtime stuff. What was lost was the ability to specify code
that is interpreted at compile time. Chuck decided that this was
something that Forth had that he really did need. Code like:
[ sectors @ tracks @ * ] literal
Now we know that says to compile a literal after first evaluating
an expression involving the contents of names system variables.
The source code tells the programmer what that literal represents
rather than just displaying the numeric value of the calculation
that was done at compile time. A comment in source could also
tell the programming what abstract expression it is.
So OK lacked the ability to communicate to the programmer what these
sorts of fields meant. They were just magic numbers and magic
numbers are not good. That was the problem. Something was lost.
Chuck's sourceless programming experiment led to some interesting
discoveries and was considered a win. It explored the notion of
mininal program source and showed what it can and cannot do. It
was not abandoned because it was painful to create code, it was
just a matter of it not having the full symbolic power of source.
Like colorforth it also explored the notion that ascii bytes
is not the only possible format for source code.
My aha compiler experiment revived the notion that ascii is not
the only possible representation of source code. Aha combined
things learned from the sourceless programming experiment with
the intrinsic properties of the packed misc opcode architecture.
Different representations of source were used in different
places to give both the advantages of symbolic source and
of the sourceless programming. The main point it showed was
that at times it is perfectly valid to use a binary opcode as
the symbol for a binary opcode. At other times that doesn't work
as shown by source specification for the evaluation of symbolic
terms at compile time.
Some of the code required references to high level words as
calls. These could be represetnted to the compiler in a way
that required the compiler to basically just relocate them
sort of like a ddl. Some of the code could be represented
by tokens like in colorforth, but tokens using the object
code as source code and which could be compiled even faster
than the relocated code, it was simply copied. Comments
are counted and simply skipped over by the compiler by
advancing the pointer by the comments length.
This resulted in a dramatically smaller and very dramatically
faster compiler than a conventional Forth compiler. It
comblined ideas from machineforth, colorforth, and sourceless
programming. And like machineForth, colorforth, and
the sourceless OK stuff from twenty years ago when Chuck
was using his chorded keyboards it is a close match to the
misc architecture.
This stuff is so old now that many people may have
forgotten about it. In one interview Chuck said that part
of the motivation for his sourceless programming experiment
was that after having every suggestion that he made for the
proposed ANSI Forth standard rejected out of hand he
conclude that other people had decided that ANSI Forth was
not for him. His abondonment of the language that he had
invented was also somewhat of a protest of the way lines
were drawn in the ANSI standard to carefully exclude him.
He concluded that the sourceless programming experiment
let to better understanding of the process and brought him
back to understanding that he really needed to use more
than 3/4 of Forth to be most productive. This is what
led to machineForth and later colorforth for Chuck and
to my aha experiment.
The colorforth and aha approaches do simplify the
Forth compiler and make it considerablly smaller, faster,
and simpler. But they do this by shifting the burden
of compilexity to the editor which has to identify the
type of compilation needed for each piece of source at
edit time and tag the source tokens. This is how for
instance the forty year old and rather slow process
of seaching the name dictionary and failing then
converting from source to binary in the current number
base when encountering a number in source can be
abandoned and replaced by a mechanism where the
compiler just loads the binary and avoids all that
unnecessary compile time mucking about with dictionary
searches that are just going to fail and character
by character conversion of strings to numbers after
dictionary searches fail.
Since Chuck worked on some word addressing machines in
the old days, and with machines with things other than
eight-bit bytes I think he was able to easily abandon
the notion that source has to be ascii bytes in a
file and C file system like it has to be for so many
other people. It is part of what Chuck means when
he says that C is byte oriented in contrast to Forth
being word oriented.
Reviewing the history of OK and how it led to OKAD
and later colorforth and later OKAD II from time to
time may be useful to frame how we got to where we
are today. I have always said that it must be
hard for people to try to jump from a knowledge
of forty year old threaded interpreted Forth to
a clear understanding of the reasons for each of
the changes to the virtual machine and compilers
that Chuck did from the first optimizing native
code compilers for Novix to the simplified MISC
compilers, simplified and updated virtual machine,
and today the colorforth compiler internals and
user interfaces.
Best Wishes,
Jeff Fox