Writing a Forth compiler in C or Go

[Cecil Bayona]

I been doing some experiments in writing a Forth in C or in Go for
Windows and I want to make it more efficient. Right now Primitives are
written in C or Go and the compiler generates tokens referring to those
primitives, the "assembler" is the C tokens but I would like to change
that to make it more efficient.

I have looked at two approaches, one works well if one is building a
meta-compiler but not as well for an interactive Forth.

First there is a DLL that will quickly compile and optimize C code
passed to it as a string and places it in an area pointed too. So the
goal will be to have the DLL be a sort of JIT compiler. The generated
code has to be placed in a place that Windows will allow writing and
execution of the code in order for it be capable of being interactive.


Scheme #1 Useful for a meta compiler.
Create a large array where the target code will be placed, the
dictionary goes somewhere else. One would create the PE header of a
Windows program at the start of the array, the generated code is added
to the array, when finished the array is written out as a .exe file
which can execute the code that was compiled. This will work for
creating a separate file that executes code.

Scheme #2 a little more complicated but allows Forth to compile and be
interactive.
Have Windows allocate a chunk of of memory with Read/Write/Execute
permission this will be the Code Area. Create a large array to hold the
dictionary, code compiled is placed in the Code Area which has execute
rights, the dictionary points to the Code Area of RAM so basically one
ends up with the assembler being C code that gets compiled on the fly
and can execute immediately.

Are there other schemes that will work? Any improvements to these schemes?

Unless otherwise convinced I tend to favor scheme # 2 as it's more
flexible and will allow meta-compilation and interactive coding.

Thanks
--
Cecil - k5nwa

[Kip Ingram]

I wrote a Forth system in gcc, and by taking advantage of the
"pointers to labels" extension I was able to do a true indirect
threaded system, in the traditional way. It's quite complete
and has very good performance.

I'd be happy to discuss with you if you like; just let me know
and we can discuss.

Cheers,
Kip

[Cecil Bayona]

On 5/21/2018 7:26 PM, Kip Ingram wrote:
> I wrote a Forth system in gcc, and by taking advantage of the
> "pointers to labels" extension I was able to do a true indirect
> threaded system, in the traditional way. It's quite complete
> and has very good performance.
>
> I'd be happy to discuss with you if you like; just let me know
> and we can discuss.
>
> Cheers,
> Kip
>
>

I'm trying to do a subroutine threaded Forth, at worst I'll settle for
direct threaded. Still the question is where did you put the code so
Windows will allow execution in interactive mode?

At best I come up with asking Windows to allocate a block of RAM with
Read/Write/Execute permissions so code placed there at run time can execute.


--
Cecil - k5nwa

[Pablo Hugo Reda]

my mv is in C, one function and the switch of OP is a enum, then when compile is a table jump, not make any parameter push or pop when ejecute, very fast

[Mark Humphries]

The simplest way to write a c-based host platform Forth is to write a token-threaded Virtual Machine executable, e.g. byte-sized tokens (bytecodes) and a big switch statement. To the rest of your Forth running atop the VM, the bytecodes are machine code albeit for a virtual rather than a real machine, and you can use subroutine-threading, inlining, and native code generation techniques.

Personally I don't find much use any more for indirect and direct threading, I tend to use either one extreme, i.e. token threading, or the other, i.e. subroutine threading with inlining and native code generation.

[Cecil Bayona]

On 5/21/2018 9:24 PM, Mark Humphries wrote:
> The simplest way to write a c-based host platform Forth is to write a token-threaded Virtual Machine executable, e.g. byte-sized tokens (bytecodes) and a big switch statement. To the rest of your Forth running atop the VM, the bytecodes are machine code albeit for a virtual rather than a real machine, and you can use subroutine-threading, inlining, and native code generation techniques.
>
> Personally I don't find much use any more for indirect and direct threading, I tend to use either one extreme, i.e. token threading, or the other, i.e. subroutine threading with inlining and native code generation.
>

The version I been working on is token threaded using a byte opcode
using a jump table. It seems fast but of course it could be faster,
tokens are the low level assembler to create new words so they are added
to the data area which is fine since the tokens don't actually run but
used to refer to native code in the code area, but effectively once
compiled no new tokens can be added unless they are added to the C code
and re-compiled.

Now on subroutine threading where do you add the new entered words
native code generated while entering words at the console?

The only way I can come up with is to have Windows allocate a chunk of
RAM with execute privileges, and compile C code on the fly and store it
there, C then becomes the low level assembler.

--
Cecil - k5nwa

[Mark Humphries]

If you take the simplest approach the VM is completely sandboxed, similar to Pascal's p-code machine, the only c code is the source of the VM itself. When you compile new definitions you are not laying down actual native code for your physical machine, you are laying down VM opcodes and data in your VM's memory. The VM's memory can be a static chunk who's size is predetermined when the VM executable is compiled, or determined at VM startup through a command line parameter, or if you prefer it could be a pre-allocated stub that you can grow at runtime if you include VM opcodes for that purpose in your VM instruction set.

[Paul Rubin]

Mark Humphries <m...@intranetsys.com> writes:
> Personally I don't find much use any more for indirect and direct
> threading, I tend to use either one extreme, i.e. token threading, or
> the other, i.e. subroutine threading with inlining and native code
> generation.

That's a nice summary of an interesting philosophy, at least for desktop
systems. Traditional threading might still have benefits for small
embedded resident interpreters.

[Cecil Bayona]

Similar to your description my present way of doing the Forth System is
similar. In the current system the only C or Go used is in writing the
primitives and the VM, all generated code are tokens representing the C
code primitives so that works OK, now I want to try something new that
is more efficient by having the new defined words generate machine code
via a C JIT compiler. It's a learning exercise, token threading is
working fine, so it's time to move to something more efficient.

My big question is where to place the generated code so it can run
without Windows objections, the only way I can think of is to have the
generated code stored in a Window allocated chunk of RAM that allows
code execution, I was hoping that there are other methods that I have
not though of, I'm not an expert on Windows, all prior Windows
programming was with Delphi which handles most of the Window issues for you.

--
Cecil - k5nwa

[Cecil Bayona]

I have it working with tokens, so now I want to try it using subroutine
threading and in-lining while generating native code using a C JIT code
generator. It's a learning experience, actually playing with the token
version and modern CPUs it's more than fast enough but I want to try it
a different way.

So when you are generating native code where do you store the code so
Windows does not complain? Windows can allocate a chunk of RAM where
code can be written and executed on the fly but I wonder if there are
other ways that I'm not aware of. It is not difficult to do but I wonder
if that is the only way?

--
Cecil - k5nwa

[Mark Humphries]

Can't help you there, my last forays in MS programming were with Windows 3.1 :-)

[Ron Aaron]

On 05/22/18 10:28, Cecil Bayona wrote:

> So when you are generating native code where do you store the code so
> Windows does not complain?

You need to use VirtualAlloc() and specify the memory is executable (and
writeable). You might be able to get away with executing off the HW
stack (e.g. RSP etc), but probably not.

[peter....@gmail.com]

Hi Cecil,

I am almost at the same stage with my 64bit tokentreaded forth.
The difference is that the VM is written in assembler. The OS interface
is written in C. From the tokenized code I generate native X64 assembler.
This is feed into an assembler, FCML, that I compiled as a dll.

To your question of allocating memory. I do this via the PE header where I
allocate more address space then needed by the image. This space is set as
r/w/x. This will always load as the same address which means that the image
can be saved and reloaded without problems.

I have written the PE header and image loader in asm. for this reason I have
full control of it. Depending on what linker your C or Go compiler uses
there are different ways to change the parameters of the allocated space.
The last way is to directly edit the PE header. There are tools for this
but also a Hex editor and the specification makes it possible.

Peter

[Lars Brinkhoff]

peter....@gmail.com writes:
> I have written the PE header and image loader in asm. for this reason
> I have full control of it.

Do you have any good documentation for the PE format? In particular,
what's acceptable or illegal in different versions of Windows?

[peter....@gmail.com]

No unfortunately not. I have downloaded the specification and worked with
that. Then of course disassembled and analysed small files to understand
it better. There are several PE editors that greatly helps.

I have done the same to get a working ELF header for linux. That was more
difficult.

Peter

[Alex McDonald]

I have a Forth version of the PE. It can build both 32 and 64 bit
versions of the PE.

See here; https://github.com/alextangent/wf32/blob/master/src/imageman.fs
specifically
https://github.com/alextangent/wf32/blob/master/src/imageman.fs


IMAGEMAN builds Windows EXE images.

For documentation on the PECOFF format, see
http://www.microsoft.com/hwdev/hardware/PECOFF.asp. Note: not included
because of copyright restrictions, but freely downloadable. Also see
"Peering Inside the PE: A Tour of the Win32 Portable Executable File
Format" by Matt Pietrek, and "An In-Depth Look into the Win32 Portable
Executable File Format", same author, both findable on http://msdn.
microsoft.com.

There is very little documentation on the loader. An exception is "What
Goes On Inside Windows 2000: Solving the Mysteries of the Loader" by
Russ Osterlund, also from MSDN.


--
Alex

[Kip Ingram]

First off, I've made this system work in Linux and on MacOS, but
I haven't tried it in Windows. I don't know for sure there's not
some sort of show-stopper there. But here are some more details.

For the initial construction of the thing the "primitives" were written
in C - most of them were very short "one liners." The key Forth
registers were housed in C variables which were directly manipulated
by the primitives. These primitives of course resided whereever
gcc put them in the program code.

The primary system lived in a memory block I allocated with malloc. The
structure was pretty much standard FIG: name string, link field, code
field, parameter region. The code field held the address of a primitive
or some other bit of code that was written in C. It was all very
standard.

In that situation I could not create new *code* words, but I could make
new colon definitions in a completely straightforward way - I just
added the new header and definition onto the end of the growing image.

Since then I've made progress toward getting EVERYTHING into the image
buffer. First step there is to use a library call at the outset to
make that image have executable permissions. I have confirmed that I
can add new code words to the system by manually poking the appropriate
bytes into the image - it works.

Then I created (in C) a "primitive wrapper" that moves the C variables
of the Forth VM into registers, jumps to an in-image address that is
expected to house machine code, and then when that code passes control
back it moves the VM regs back into the C variables. This of course
is not efficient and no good for a final working point, but it does
allow me to develop Forth source code for primitives that would
function eventually in a fully final way.

That's where I am right now - this is just a hobby project and I don't
have a lot of time to work on it. But the next step is to get all of
the primitives (all things that require C code be called) implemented
in-image, and fully "cut the cord" to C.

After that, I plan to learn enough about executable file formats to
be able to write my image out to a file that can be run stand-alone.

So I have a game plan for eventually removing the C "scaffolding"
that's been required to bring the thing into existence - when I'm done
it would be a fully native Forth. Finally, I want to work toward
Forth source for the whole system, so that it can "recompile itself."

Did that answer your question?

Cheers,
Kip

[Alex McDonald]

That MS link is bust. It's
http://www.microsoft.com/whdc/system/platform/firmware/PECOFF.mspx.

--
Alex

[Stephen Pelc]

On Tue, 22 May 2018 02:28:36 -0500, Cecil Bayona <cba...@cbayona.com>
wrote:

>So when you are generating native code where do you store the code so
>Windows does not complain? Windows can allocate a chunk of RAM where
>code can be written and executed on the fly but I wonder if there are
>other ways that I'm not aware of. It is not difficult to do but I wonder
>if that is the only way?

The sections of a PE file have r/w/x permissions. An NCC system
needs r/w/x. The section is also defined by its size on the disc
and the size in memory. Just make the memory size as large as you
need.

Stephen

--
Stephen Pelc, steph...@mpeforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441
web: http://www.mpeforth.com - free VFX Forth downloads

[Cecil Bayona]

You can, eventually I will get around to Linux, I keep debating
switching to Linux on most of my PCs but there is a couple of expensive
software packages that keep me with Windows but Microsoft keeps pissing
me off by breaking the machine with major updates. Now I have one PC
with Mint Linux only, more to come in the future.

--
Cecil - k5nwa

[Cecil Bayona]

Thanks, that was my first thought on adding executable code on the fly,
I was thinking about editing the Forth Compiler's PE either on the fly
or statically. I assume you are taking about adding additional RAM where
the current code lies for the Forth compiler, a little simpler than
allocating a new area but I have not thought out if it will cause any
additional problems, I can't think of any at present.

I looked and printed some documents on the PE of an executable file and
how to make changes by extending the area, there is no lack of
documentation on the PE and it's a reasonable approach also. I'll have
to do a few experiments with the C JIT module and see how it handles
allocating Variables.

The C JIT I been planning on using is TCC it has the assembler/linker
built in so it generates ready to run code on the fly, it has limited
code optimization but it's small and extremely fast at compiling. For C
there are a couple of other JIT packages available including GCC, I have
not checked into Go but I believe it has one available from Google, it
needs to be really fast to make it practical.

--
Cecil - k5nwa

[Cecil Bayona]

Thanks, that is the method I thought of using, but I wanted to know if
there are alternatives, with a meta-compiler there are simple methods
available but they don't work with interactive Forth.

Writing code to the stack even if it works promises to be messy compared
to VirtualAlloc().

I swore off C a long time back but now that I'm getting back into it
it's not so bad, of course I have not run yet into a nasty bug where C
help you easily get there, that is why I been looking a Go. Still my
favorite Algol like language is Embarcadero Delphi but it's only for
rich people now.

--
Cecil - k5nwa

[Cecil Bayona]

On 5/22/2018 4:31 AM, peter....@gmail.com wrote:

By the way and I forgot to mention it, I like your ntf2017 Forth
compiler, it generates decent code, I often refer to it to see what code
it generates.

I noticed that PARSE and WORD do not seem to get their data as easily
corrupted as some of the other compilers which is handy when entering
words one at a time and see what is going on.

I assume it modifies the PE like you mentioned.

--
Cecil - k5nwa

[Cecil Bayona]

There is a lot of documentation from Microsoft and others on the
structure of the PE header, search for ".exe PE header". Microsoft has a
lot of information and charts that can be printed well.

A nice free tool for looking and changing the PE header is stud_pe.exe

--
Cecil - k5nwa

[minf...@arcor.de]

Am Dienstag, 22. Mai 2018 17:10:12 UTC+2 schrieb Cecil - k5nwa:
> On 5/22/2018 3:17 AM, Ron Aaron wrote:
> >
> >
> > On 05/22/18 10:28, Cecil Bayona wrote:
> >
> >> So when you are generating native code where do you store the code so
> >> Windows does not complain?
> >
> > You need to use VirtualAlloc() and specify the memory is executable (and
> > writeable). You might be able to get away with executing off the HW
> > stack (e.g. RSP etc), but probably not.
> >
>
> Thanks, that is the method I thought of using, but I wanted to know if
> there are alternatives, with a meta-compiler there are simple methods
> available but they don't work with interactive Forth.

Why?? Just put the primitive address into a C function pointer and execute it.

> Writing code to the stack even if it works promises to be messy compared
> to VirtualAlloc().
>
> I swore off C a long time back but now that I'm getting back into it
> it's not so bad, of course I have not run yet into a nasty bug where C
> help you easily get there, that is why I been looking a Go. Still my
> favorite Algol like language is Embarcadero Delphi but it's only for
> rich people now.

FreePascal is not just for the poor.. ;-)

[Cecil Bayona]

On 5/22/2018 1:19 PM, minf...@arcor.de wrote:
> Am Dienstag, 22. Mai 2018 17:10:12 UTC+2 schrieb Cecil - k5nwa:
>> On 5/22/2018 3:17 AM, Ron Aaron wrote:
>>>
>>>
>>> On 05/22/18 10:28, Cecil Bayona wrote:
>>>
>>>> So when you are generating native code where do you store the
>>>> code so Windows does not complain?
>>>
>>> You need to use VirtualAlloc() and specify the memory is
>>> executable (and writeable). You might be able to get away with
>>> executing off the HW stack (e.g. RSP etc), but probably not.
>>>
>>
>> Thanks, that is the method I thought of using, but I wanted to know
>> if there are alternatives, with a meta-compiler there are simple
>> methods available but they don't work with interactive Forth.
>
> Why?? Just put the primitive address into a C function pointer and
> execute it.
>

That is what I'm doing in the Token Threaded version but I want to move
to the next step which is generating native code on the fly.

>> Writing code to the stack even if it works promises to be messy
>> compared to VirtualAlloc().
>>
>> I swore off C a long time back but now that I'm getting back into
>> it it's not so bad, of course I have not run yet into a nasty bug
>> where C help you easily get there, that is why I been looking a Go.
>> Still my favorite Algol like language is Embarcadero Delphi but
>> it's only for rich people now.
>
> FreePascal is not just for the poor.. ;-)
>

I have not tried it out, I have it installed on my two main PCs but have
not had time to try it out with a serious application

--
Cecil - k5nwa
on but I want to move on to generate

[Paul Rubin]

Cecil Bayona <cba...@cbayona.com> writes:
> You can, eventually I will get around to Linux..., I have one PC with

> Mint Linux only, more to come in the future.

Well you could get some raspberry pis or something like that.

Don't know about Windows but if you're writing a C-based Forth under
Linux, you could possibly implement CODE words using dlsym. You compile
the C code into the a .so (Linux's counterpart to a .dll), and dlsym
takes the name of a function, loads the .so if it isn't already loaded,
and gives you the function's address. Then you can call the function
using normal C calling conventions.

It would be nice to define CODE words that generate inline code but that
would require more platform specific hackery. The dlsym method should
be portable across Linuxes.

[minf...@arcor.de]

Am Dienstag, 22. Mai 2018 21:46:46 UTC+2 schrieb Cecil - k5nwa:
> > Why?? Just put the primitive address into a C function pointer and
> > execute it.
> >
> That is what I'm doing in the Token Threaded version but I want to move
> to the next step which is generating native code on the fly.

In the olden days one would do macros like this:

: MYWORD
state @
IF <comma some hex code as native code macro>
ELSE <do the runtime boogie>
ENDIF ;

Today such doings are too dangerous. Postpone might get a hickup. :-)

[Rod Pemberton]

On Tue, 22 May 2018 11:19:07 -0700 (PDT)
minf...@arcor.de wrote:

> Am Dienstag, 22. Mai 2018 17:10:12 UTC+2 schrieb Cecil - k5nwa:
> > On 5/22/2018 3:17 AM, Ron Aaron wrote:
> > > On 05/22/18 10:28, Cecil Bayona wrote:

> > >> So when you are generating native code where do you store the
> > >> code so Windows does not complain?
> > >
> > > You need to use VirtualAlloc() and specify the memory is
> > > executable (and writeable). You might be able to get away with
> > > executing off the HW stack (e.g. RSP etc), but probably not.
> >
> > Thanks, that is the method I thought of using, but I wanted to know
> > if there are alternatives, with a meta-compiler there are simple
> > methods available but they don't work with interactive Forth.
>
> Why?? Just put the primitive address into a C function pointer and
> execute it.

That's what I did for my ITC Forth interpreter. At this point, the
primitives are in C, and the high-level words are in Forth. It uses
actual address/inner interpreter, instead of a switch() statement
like most Forths in C.


Rod Pemberton
--
I believe in the right to life. That's why I oppose gun control.

[Rod Pemberton]

On Tue, 22 May 2018 14:46:44 -0500
Cecil Bayona <cba...@cbayona.com> wrote:

> On 5/22/2018 1:19 PM, minf...@arcor.de wrote:

> > Why?? Just put the primitive address into a C function pointer and
> > execute it.
> >
> That is what I'm doing in the Token Threaded version but I want to
> move to the next step which is generating native code on the fly.
>

You're not going to generate C code on the fly? Why not?

Advanced C compilers can do all kinds of optimizations that your
compiler probably won't have.

[Rod Pemberton]

On Tue, 22 May 2018 12:48:11 GMT
Kip Ingram <kipi...@kips-air.attlocal.net> wrote:

> First off, I've made this system work in Linux and on MacOS, but
> I haven't tried it in Windows. I don't know for sure there's not
> some sort of show-stopper there. But here are some more details.

My Forth is only working for DOS, with two C compilers, OpenWatcom and
DJGPP. I have some hardcoded offsets for the dictionary that need
adjusting for 64-bit Linux.

> For the initial construction of the thing the "primitives" were
> written in C - most of them were very short "one liners." The key
> Forth registers were housed in C variables which were directly
> manipulated by the primitives.

I worked from C first, then migrated, transformed, rewrote, to the
point that the primitives are in C and the high-level words are in
Forth.

Unlike virtually every other Forth in C, I don't use a switch()
statement as an "interpreter", but use an actual ITC interpreter coded
in C. The address/inner interpreter is coded in C using W, IP etc. The
outer/text interpreter is high-level Forth word, pre-"compiled" in C to
match the same layout as a high-level colon-def.

> The primary system lived in a memory block I allocated with malloc.

Ditto.

My allocation amount is completely arbitrary at this point, since I
don't know how large the dictionary will grow. I took a guess at the
average Forth word size and quantity of words in the dictionary.

> The structure was pretty much standard FIG: name string, link field,
> code field, parameter region.

Ditto.

Except, I moved the LFA to the start of the dictionary header. This
was originally so I could move through the dictionary quickly in C
using a linked list. The C code no longer accesses the dictionary
directly, as all of the high-level words are now in Forth.

> The code field held the address of a primitive or some other bit of
> code that was written in C. It was all very standard.

Ditto.

I have the standard CFA routines, except for DOUSER.

> In that situation I could not create new *code* words, but I could
> make new colon definitions in a completely straightforward way - I
> just added the new header and definition onto the end of the growing
> image.

Ditto.

Yes, I hadn't thought about that, but the primitives (or "code" words)
must be compiled in for mine as well, as the low-level is in C and
there is no assembler. An assembler in Forth would have register
incompatibilities with the C code.

> Since then I've made progress toward getting EVERYTHING into the image
> buffer. First step there is to use a library call at the outset to
> make that image have executable permissions. I have confirmed that I
> can add new code words to the system by manually poking the
> appropriate bytes into the image - it works.

And, earlier,

> These primitives of course resided
> whereever gcc put them in the program code.

You've gone further than I have here. My primitives and their
dictionary headers are wherever the C compiler places them in the
executable image. The dictionary is in malloc'd space. I.e., the
primitives and high-level colon-def words not together. I'm not sure
that copying the primitives or relocating them to the malloc'd
dictionary space would have any advantage. I would need some C code to
patch up their field addresses for the relocation.

> Then I created (in C) a "primitive wrapper" that moves the C
> variables of the Forth VM into registers, jumps to an in-image
> address that is expected to house machine code, and then when that
> code passes control back it moves the VM regs back into the C
> variables. This of course is not efficient and no good for a final
> working point, but it does allow me to develop Forth source code for
> primitives that would function eventually in a fully final way.

Well, you've progressed more to that of a standalone OS-like Forth, as
you've transferred execution directly into the image. You did
mention working with embedded environments. It's also interesting that
you're saving and restoring the registers similar to DOS DPMI calls.

I could do these things, but I don't see the point for a generic
Forth, not embedded, which starts from an existing OS. Why does your
Forth need to be entirely within a self-contained image which can be
saved and restarted, apparently? Is this just for embedded? Or, do you
intend to use it as an OS too, like early Forths?

> That's where I am right now - this is just a hobby project and I don't
> have a lot of time to work on it.

I haven't worked on mine in a few years either, also hobby project. I
started it about a decade ago or so. I have a small set of high-level
Forth words to complete to fully pass Hayes core. It was a challenge
to me as I'm not a Forth programmer, but a C coder.

> But the next step is to get all of
> the primitives (all things that require C code be called) implemented
> in-image,

Ok.

> and fully "cut the cord" to C.

How is that possible? You still need C to be compiled to initially
produce the binary code for the primitives or other low-level code
words, yes? Do you plan to re-implement this code from Forth assembly?

> After that, I plan to learn enough about executable file formats to
> be able to write my image out to a file that can be run stand-alone.

This reminds me of how I created my OS. I started with C code for
DOS. DOS has virtually no OS or hardware protections. Then, I started
my OS image from DOS, and worked backwards to booting the image from a
boot loader.

> So I have a game plan for eventually removing the C "scaffolding"
> that's been required to bring the thing into existence - when I'm done
> it would be a fully native Forth.

Ok.

> Finally, I want to work toward
> Forth source for the whole system, so that it can "recompile itself."

I started doing that with my Forth, but backed away from that path.
I.e., I started re-implementing the low-level primitives and outer
interpreter in high-level Forth, but realized that the code would still
call some low-level code, e.g., primitives or inner interpreter, which
was coded in C. I would still prefer to bootstrap the system from a
lower level than it is presently, but that appeared to be much more
complicated than having a few extra words created using C.

[Lars Brinkhoff]

Rod Pemberton writes:
>> Just put the primitive address into a C function pointer and execute
>> it.
> That's what I did for my ITC Forth interpreter. At this point, the
> primitives are in C, and the high-level words are in Forth. It uses
> actual address/inner interpreter,

My first Forth did that too. I think the technique is quite obvious.

> instead of a switch() statement like most Forths in C.

Do you have actual numbers on how many Forth in C use switch() versus
how many use address interpreters?

[Ron Aaron]

8th is written mostly in C, but it uses native-code subroutine
threading, not tokens or address threading etc.

[Ron Aaron]

On 05/22/18 18:10, Cecil Bayona wrote:

> I swore off C a long time back but now that I'm getting back into it
> it's not so bad, of course I have not run yet into a nasty bug where C
> help you easily get there, that is why I been looking a Go. Still my
> favorite Algol like language is Embarcadero Delphi but it's only for
> rich people now.

I was a fan of Turbo Pascal back in ancient times. But it never really
got popular enough in commercial-software development, so I almost never
worked with it for pay.

There are a variety of options for making executable code, but IMHO it's
easiest and most straight-forward to compile into properly allocated
memory. On non-Windows systems you can use "mmap()", with pretty much
the same semantics as VirtualAlloc(). And the OS will only actually
commit the memory you touch (so allocating 1M only gets allocated in
page-sized chunks as you need it).

[minf...@arcor.de]

That's also how it works here:

: EXECUTE \ ( xt -- .. ) execute given execution token
C mfw=(mfXT*)mfpop();
C (*mfw)(); ;

An execution token is a pointer cell pointing to machine code:

typedef void (*mfXT)(void); // xts are function pointers

Hilevel words are function pointer lists that are executed sequentially
without any VM giant switch.

[Rod Pemberton]

On Wed, 23 May 2018 05:35:36 +0000
Lars Brinkhoff <lars...@nocrew.org> wrote:

> Rod Pemberton writes:

> > instead of a switch() statement like most Forths in C.
>
> Do you have actual numbers on how many Forth in C use switch() versus
> how many use address interpreters?

No.

But, I've only seen code one other address interpreter in C, and read
of another in this thread. So, I know of 3 Forths in C that don't use
a switch() statement.

[Rod Pemberton]

On Wed, 23 May 2018 04:33:04 -0400
Rod Pemberton <inv...@lkntrgzxc.com> wrote:

> On Wed, 23 May 2018 05:35:36 +0000
> Lars Brinkhoff <lars...@nocrew.org> wrote:
> > Rod Pemberton writes:

> > > instead of a switch() statement like most Forths in C.
> >
> > Do you have actual numbers on how many Forth in C use switch()
> > versus how many use address interpreters?
>
> No.
>
> But, I've only seen code one other address interpreter in C, and read
> of another in this thread. So, I know of 3 Forths in C that don't use
> a switch() statement.
>

Sorry, that would be four, including yours.

[Rod Pemberton]

On Wed, 23 May 2018 04:37:05 -0400

Rod Pemberton <inv...@lkntrgzxc.com> wrote:

> On Wed, 23 May 2018 04:33:04 -0400
> Rod Pemberton <inv...@lkntrgzxc.com> wrote:
> > On Wed, 23 May 2018 05:35:36 +0000
> > Lars Brinkhoff <lars...@nocrew.org> wrote:
> > > Rod Pemberton writes:

> > > > instead of a switch() statement like most Forths in C.
> > >
> > > Do you have actual numbers on how many Forth in C use switch()
> > > versus how many use address interpreters?
> >
> > No.
> >
> > But, I've only seen code one other address interpreter in C, and
> > read of another in this thread. So, I know of 3 Forths in C that
> > don't use a switch() statement.
> >
>
> Sorry, that would be four, including yours.
>

FYI, the fourth one is Bradford Rodriguez's recent CamelForth in C:

See "CamelForth in C!"
http://www.camelforth.com/news.php

download
http://www.camelforth.com/download.php?view.35

[Albert van der Horst]

In article <pe09gj$u5q$1...@dont-email.me>,

Cecil Bayona <cba...@cbayona.com> wrote:
>On 5/21/2018 9:24 PM, Mark Humphries wrote:
>> The simplest way to write a c-based host platform Forth is to write a token-threaded Virtual Machine executable, e.g. byte-sized tokens (bytecodes) and a big switch statement. To the rest of your Forth running atop the VM, the bytecodes are machine code albeit for a virtual rather than a real machine, and you can use subroutine-threading, inlining, and native code generation techniques.
>>

>> Personally I don't find much use any more for indirect and direct threading, I tend to use either one extreme, i.e. token threading, or the other, i.e. subroutine threading with inlining and native code generation.
>>
>

>The version I been working on is token threaded using a byte opcode
>using a jump table. It seems fast but of course it could be faster,
>tokens are the low level assembler to create new words so they are added
>to the data area which is fine since the tokens don't actually run but
>used to refer to native code in the code area, but effectively once
>compiled no new tokens can be added unless they are added to the C code
>and re-compiled.
>
>Now on subroutine threading where do you add the new entered words
>native code generated while entering words at the console?
>
>The only way I can come up with is to have Windows allocate a chunk of
>RAM with execute privileges, and compile C code on the fly and store it
>there, C then becomes the low level assembler.

My stance is that indirect threaded is probably fast enough.
If it isn't, indirect threading allows so much introspection
that a separate optimiser blows everything out of the water,
regarding speed and simplicity.
(One of the things on my to do list: write an optimiser for ciforth
that beats ciforth in speed.
First I need a program that is not fast enough without optimiser. My
OCR may just be such a program.)

>
>--
>Cecil - k5nwa

Groetjes Albert
--
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

[Albert van der Horst]

In article <pe0gdc$9p$1...@dont-email.me>,

Cecil Bayona <cba...@cbayona.com> wrote:
>
>My big question is where to place the generated code so it can run
>without Windows objections, the only way I can think of is to have the
>generated code stored in a Window allocated chunk of RAM that allows
>code execution, I was hoping that there are other methods that I have
>not though of, I'm not an expert on Windows, all prior Windows
>programming was with Delphi which handles most of the Window issues for you.

This is a protection problem. In my experience protection problems
are less on Windows than on Linux. If you manage to arrive
at a single segment, write read execute permissions, your problems are over.
Then you can always define you own machine code and execute it.
Going through C is difficult, it is like brain surgery on the moon
via a remote controled robot. FASM rules!

>
>--
>Cecil - k5nwa

[Albert van der Horst]

In article <pe3051$4qi$1...@dont-email.me>,

Ron Aaron <ramb...@gmail.com> wrote:
>
>
>On 05/22/18 18:10, Cecil Bayona wrote:
>
>> I swore off C a long time back but now that I'm getting back into it
>> it's not so bad, of course I have not run yet into a nasty bug where C
>> help you easily get there, that is why I been looking a Go. Still my
>> favorite Algol like language is Embarcadero Delphi but it's only for
>> rich people now.
>
>I was a fan of Turbo Pascal back in ancient times. But it never really
>got popular enough in commercial-software development, so I almost never
>worked with it for pay.

Isn't that because it evolved into Delphi and Kylix?
I've seen very sophisticated production planning done in Delphi.

Groetjes Albert

[m...@iae.nl]

On Wednesday, May 23, 2018 at 11:49:16 AM UTC+2, Albert van der Horst wrote:

[..]

> My stance is that indirect threaded is probably fast enough.
> If it isn't, indirect threading allows so much introspection
> that a separate optimiser blows everything out of the water,
> regarding speed and simplicity.
> (One of the things on my to do list: write an optimiser for ciforth
> that beats ciforth in speed.

I predict that that can be done with a one line program :-)

Didn't you add one character too many in that line?

> First I need a program that is not fast enough without optimiser. My
> OCR may just be such a program.)

The problem here is to find an algorithm that can't
be written by a normally gifted programmer in a form
that can be trivially optimized (i.e. needs no sophisticated
techniques). If it can be written in a trivially
optimizable way, the competing compiler will beat your
optimizer, as it has probably seen decades of fine-tuning
for trivial code.

There been threads in CLF where people showed surprising
ingenuity in simplifying complicated looking stuff, once
it was pointed out what and where the problem was. I will
be very interested in your benchmark.

-marcel

[Ron Aaron]

On 23/05/2018 13:03, Albert van der Horst wrote:
> In article <pe3051$4qi$1...@dont-email.me>,
> Ron Aaron <ramb...@gmail.com> wrote:

>> I was a fan of Turbo Pascal back in ancient times. But it never really
>> got popular enough in commercial-software development, so I almost never
>> worked with it for pay.
>
> Isn't that because it evolved into Delphi and Kylix?
> I've seen very sophisticated production planning done in Delphi.

Yes, because Borland took an amazingly excellent product (Turbo Pascal)
and didn't know how to develop and market it effectively.

Delphi was OK, but you could see the lack of overall vision with each
new version. I was a beta-tester for years, I saw it all...

[Kip Ingram]

It sounds like you're on a VERY similar path to my work. My
primitives are also still in C, wherever the compiler placed
them. The extent to which I've run "in image" machien code
is still very much "proof of principle." I've seen enough
things work to know that I can "get there," but I haven't
"gotten there" yet. I only manage to find time to work on
this occasionally. But I've had thoughts similar to what
you express - that this is a "real" ITC Forth as opposed to
a "Forth-like behavioral model."

I'm a hardware guy by training, and my eventual goal (sort of
a "bucket list" thing) is to build a serious computer fully
from scratch, using an FPGA implemented processor core that I
have some rather large number of in the system. This Forth
will become the operating system for that machine. So I want
to play with things like clean Forth support for multiple
cores, code parallelization mechanisms, and so on.

It's a hobby, but it's a pretty darn serous one.

Good luck with your continued work!

-- Kip

[Kip Ingram]

Of course, regarding turning this thing into the OS of a future
custom machine, it goes without saying that everything that's a
"primitive" in this system would have to be re-implemented in
the machine code (or hardware) of that future machine. But my
hope is to gradually build up a fairly sophisticated collection
of defined words that would port to the new environment
unchanged.

[Cecil Bayona]

Thanks, so far either make it a token compiler which I tried, or use
Windows or Linux functions to extend the code area using two different
calls which are similar.

After using C for years I got tired of the errors which were very
difficult to find sometimes. Delphi and it's predecessor Turbo Pascal
had optional error checking that helped one catch many mistakes that
were difficult to find in C, and it could pretty much could do anything
C could do with some extra features.

--
Cecil - k5nwa

[Cecil Bayona]

On 5/23/2018 3:33 AM, Rod Pemberton wrote:
> On Wed, 23 May 2018 05:35:36 +0000
> Lars Brinkhoff <lars...@nocrew.org> wrote:
>
>> Rod Pemberton writes:
>
>>> instead of a switch() statement like most Forths in C.
>>
>> Do you have actual numbers on how many Forth in C use switch() versus
>> how many use address interpreters?
>
> No.
>
> But, I've only seen code one other address interpreter in C, and read
> of another in this thread. So, I know of 3 Forths in C that don't use
> a switch() statement.
>
>
> Rod Pemberton
>

I don't use a switch statement either, a big array with the addresses of
the tokens, use the token as an index into the array, fetch the address
and execute it.

--
Cecil - k5nwa

[minf...@arcor.de]

The assumed disadvantage of VMs with giant switches are often overrated.
Any decent C compiler will translated the switch to an indexed jump table
anyhow.

More important are branch prediction issues and CPU cache misses.

[Ilya Tarasov]

понедельник, 14 мая 2018 г., 3:07:02 UTC+3 пользователь Cecil - k5nwa написал:
> I been doing some experiments in writing a Forth in C or in Go for
> Windows and I want to make it more efficient. Right now Primitives are
> written in C or Go and the compiler generates tokens referring to those
> primitives, the "assembler" is the C tokens but I would like to change
> that to make it more efficient.
>
> I have looked at two approaches, one works well if one is building a
> meta-compiler but not as well for an interactive Forth.
>
> First there is a DLL that will quickly compile and optimize C code
> passed to it as a string and places it in an area pointed too. So the
> goal will be to have the DLL be a sort of JIT compiler. The generated
> code has to be placed in a place that Windows will allow writing and
> execution of the code in order for it be capable of being interactive.
>
>
> Scheme #1 Useful for a meta compiler.
> Create a large array where the target code will be placed, the
> dictionary goes somewhere else. One would create the PE header of a
> Windows program at the start of the array, the generated code is added
> to the array, when finished the array is written out as a .exe file
> which can execute the code that was compiled. This will work for
> creating a separate file that executes code.
>
> Scheme #2 a little more complicated but allows Forth to compile and be
> interactive.
> Have Windows allocate a chunk of of memory with Read/Write/Execute
> permission this will be the Code Area. Create a large array to hold the
> dictionary, code compiled is placed in the Code Area which has execute
> rights, the dictionary points to the Code Area of RAM so basically one
> ends up with the assembler being C code that gets compiled on the fly
> and can execute immediately.
>
> Are there other schemes that will work? Any improvements to these schemes?
>
> Unless otherwise convinced I tend to favor scheme # 2 as it's more
> flexible and will allow meta-compilation and interactive coding.
>
> Thanks
> --
> Cecil - k5nwa

I use C-written Forth VM now for both Windows and ARM bare metal platforms. It is 32/64 datawidth (regulated by CELL typedef, so all data has CELL type defined at the top) and code is fully high-level, without any inlines. My goal was to achieve compilation for both platforms with simple #include directive. Well, it works fine.

All Forth words are described as void() functions, with special kind of execution code, acceoting "pointer to void() function as a parameter". This avoids possible problem of huge switch code.

With this scheme I can accept lower performance comparing with machine code described in assembler, because I rely on hardware acceleration. Finally, I want to have the same sources in Forth for x86/ARM/Forth CPU.

[Paul Rubin]

Rod Pemberton <inv...@lkntrgzxc.com> writes:
> But, I've only seen code one other address interpreter in C, and read
> of another in this thread. So, I know of 3 Forths in C that don't use
> a switch() statement.

I have a partly written one where you write C functions with special
names like CODE_swap, and the build script figures out the code
addresses for those functions by scanning the symbol table generated by
the compiler.

[Rod Pemberton]

On Wed, 23 May 2018 10:59:24 GMT
Kip Ingram <kipi...@kips-air.attlocal.net> wrote:

<re-organized>

> I'm a hardware guy by training, and my eventual goal (sort of
> a "bucket list" thing) is to build a serious computer fully
> from scratch, using an FPGA implemented processor core that I
> have some rather large number of in the system.

Wow. Ok. Why?

The last time I had an inkling to build a processor was due to a Byte
magazine article in the 1980s.

> This Forth will become the operating system for that machine.

Would Linux be easier? faster?

> It sounds like you're on a VERY similar path to my work.

I'm definitely not going to implement a Forth which is embeddable.
I've decided that I'm basically done, except finishing Hayes core
compliance, and fixing my current set of problems. While I could
technically implement an embeddable version, that's yet another project
I don't have time for, and I have no desire to do either.

> It sounds like you're on a VERY similar path to my work.

I've written about my project a few times over many years here, from
summaries to in-depth, and haven't heard of anything similar until this
thread. It's amazing that there are a bunch of guys here and now at
this time who've done similar things! I thought and felt I was alone
for over a decade.

> But I've had thoughts similar to what you express - that this is a
> "real" ITC Forth as opposed to a "Forth-like behavioral model."

My original goal was to implement a Forth using C as identically as
possible to a Forth in assembly. I've done that, or very close
to it, although I'm not finished. At first, it was all in C.

I started with fig-Forth words that survived into modern Forth, adopted
the eForth primitive and high-level Forth model, added things from
other standards that made sense to me (e.g., CELLS CELL+ etc). You
wouldn't believe how many versions I've gone through to get it to where
it is today. The backups I still have are well over 200.

You wouldn't believe how difficult it is to convert C into Forth, or
share data between C and Forth, unless you've BTDT. Forth is so much
more primitive than ANSI C. You wouldn't believe how difficult it was
for a non-Forther to find out how certain aspects of Forth are
implemented either. Forth has it's own language to describe standard
computer science concepts. Forth has plenty of undocumented "features"
or boundary situations or things not easy to implement in ANSI C, e.g.,
DOES> or KEY (non-echo getc AKA getch).

So, I spent plenty of time testing various Forths to determine the
common behavior even for words defined in multiple Forth standards.
Some things have a common behavior, while other things don't, e.g.,
multiple common behaviors, no common behavior, etc. I wrote my own
tests for Forth words for Josh Grams' small tester to test other
Forths and then my own, and modified and extended a stack optimizer by
Peter Sovietov to find better definitions.

> My primitives are also still in C, wherever the compiler placed
> them.

Pure ANSI C unfortunately has some limitations on the placement of data
as compared to assembly which can't be overridden with casts. So, my
Forth primitive words are currently split into two parts, e.g., header
(LFA, NFA, etc) and address list (CFA, PFA), which C may store
separately or insert padding between, since they're not a single C
object. That causes problems with primitives and words which access
the body of a word. High-level words are compiled into the dictionary
at run-time and don't have such issues.

I can pack everything into a single C object except for the name field,
which C converts into an address and stores the data for elsewhere.
Doing so eliminates the split data, except for the name field, and makes
the implementation of a few high-level Forth words more correct. But,
there isn't much advantage otherwise. The disadvantage of an incorrect
name field is that it makes a few other high-level Forth words less
correct. I.e., either way it may be a wash. However, locality of
reference of a single C object may improve processor cache
performance. So, it's just a matter of do I want to do all the work to
rewrite it to find out if it's better one way or the other? ...

I was also avoiding the use of 'unions'. Although, a union might help
to solve the issue of accurate data implementation for the Forth
dictionary entries, i.e., the name field. Unfortunately, early C
compilers had issues with "advanced" C things like structs, unions,
enums, etc and I was hoping that maybe I could bootstrap my Forth with a
primitive C compiler in the future, but that is not a serious goal.

[minf...@arcor.de]

If I may ask:
could you publish your C code of this Forth 1-liner?

: MAIN cr ." Hello world " 2018 . ;

[Rod Pemberton]

On Thu, 24 May 2018 08:38:54 -0700 (PDT)
minf...@arcor.de wrote:

> If I may ask:
> could you publish your C code of this Forth 1-liner?
>
> : MAIN cr ." Hello world " 2018 . ;
>

A) Technically, it's a 2-liner as you must type MAIN on a new line.
The Forth word MAIN is code.

B) I don't have C code for that offhand, but will post something for
you below.

C) Which C code? You didn't say if you want the ANSI C code
equivalent as I expect, or if you wanted the pre-compiled ITC Forth
expressed in C, that can be executed with my Forth's inner/address
interpreter, once compiled. I'm not providing the latter as I've not
released my Forth interpreter.

D) Why are you trolling me? ... Whatever.

E) It's a 2-liner in pure ANSI C as well. Unlike Forth, the program
name used to execute the compiled C code is not code. Removing C's
return value for main(), as I did, matches your code more closely
syntactically.

#include <stdio.h>
int main(void){puts("Hello world 2018");}


If you want the 2018 to be an integer, not a string, since I converted
it to a string, it's still a 2-liner:

#include <stdio.h>
int main(void){printf("Hello world %d\n",2018);}


BTW, C doesn't required formatted or indented code, which is why the
main program is on one line. C only requires spaces in a few locations
to parse correctly, e.g., between "long long" and "int main" etc.

Technically, a return(0); or return 0; is not required, nor the more
formal exit(EXIT_SUCCESS) either, but is usually coded that way by C
programmers. So, I'll add it in the C examples below.


With the return value, it would look like one of these:

#include <stdio.h>
int main(void){puts("Hello world 2018");return(0);}

#include <stdio.h>
int main(void){puts("Hello world 2018");return 0;}


Using exit() requires an extra include, #include <stdlib.h>, turning it
into a 3-liner.

#include <stdio.h>
#include <stdlib.h>
int main(void){puts("Hello world 2018");exit(EXIT_SUCCESS);}


With formatting, the 3-liner becomes a C program coded the way
you'd normally see it:

#include <stdio.h>
#include <stdlib.h>
int main (void)
{
puts ("Hello world 2018");
exit (EXIT_SUCCESS);

[minf...@arcor.de]

I am sorry if my question led you to the impression that I was trolling you.
I am too old for such silly games.

My question was triggered by my impression that you had developed a Forth
compiler too, that uses C as language for Forth primitives. It would have
been nice to share ideas.

Tit for tat here is what MinForth generates from the 1-liner
(here for a stand-alone executable, embeddable C-code can be controlled by flags):

Forth source:

\ Forth to C Test
#REQUIRE lib/kernel.mfw
#REDUCE
: MAIN cr ." Hello Forth " 2018 . ;

Generated C code:

/* --------- MinForth V3 C Source ---------
- Transpiled by MF2C ----- do not edit -
- Forth source : hello.mf
- Built Fri May 25 19:36:49 2018
*/

#include "mf3h.h"

void mf64D9F6E0(void); // CR
void mf2B0C98F1(void); // .
void mf96272888(void); // MAIN

mfHdr mfdict[4] = { // MinForth diction8ary
{NULL,-1,NULL,NULL},
{mfdict,-1,"\002CR",(mfXT)&mf64D9F6E0},
{mfdict+1,-1,"\001.",(mfXT)&mf2B0C98F1},
{mfdict+2,-1,"\004MAIN",(mfXT)&mf96272888}};

#define mflast (mfdict+3)

void mf64D9F6E0(void) { // : CR
mfemit('\n');
}

void mf2B0C98F1(void) { // : .
printf("%d ",mfpop());
}

void mf96272888(void) { // : MAIN
mf64D9F6E0(); // CR
mfprint("Hello Forth "); // ."
mfpush(2018); // $7E2
mf2B0C98F1(); // .
}
int __cdecl main(void) {
mf96272888(); // MAIN
}

Here . is coded using the C printf function. For embedded targets the executable
might become too large.

For such cases a classic hilevel Forth definition of . is used.
The generated intermediate C source would become by ~ 30 small functions
larger, but the overall executable size is much smaller.

[Kip Ingram]

As far as "why" goes, really no objective justification. It's just
become something I want to do. I don't really know how to explain
it. :-)

So I guess I mis-phrased. Our whole paths aren't similar, but
we've done some similar work. My implementation is utterly
reliant on the pointer to label gcc extension - it's not an
ANSI C compliant implementation in any way.

[Rod Pemberton]

On Fri, 25 May 2018 11:02:10 -0700 (PDT)
minf...@arcor.de wrote:

Sorry about my misunderstandings.

> My question was triggered by my impression that you had developed a
> Forth compiler too, that uses C as language for Forth primitives.

No, I have no Forth compiler. I did create a rudimentary Forth-to-C
converter program, which I've not done much work on. It's not part of
my Forth program, but is a separate parser. Originally, it produced
two output files: C code, interpretable Forth code in C. Currently, it
only produces C code.

My rudimentary converter program isn't capable of converting the
1-liner to C correctly, e.g., doesn't recognize strings etc. The
current version only outputs C code, not the interpretable Forth code
as C. This is what it does currently:

: MAIN cr ." Hello Forth " 2018 . ;

#include "f_cnv.h"

int main(void)
{
cr();
dot_quote();
hello();
forth();
2();
2(2018);
ERROR
dot__();

return(0);
}

So, here is my test file and two outputs I had for it with the earlier
version. The one output gives clues about how my Forth interpreter
works, which I haven't revealed, but now that there are 5 guys who've
done something similar ... The format has changed somewhat since then.


The garbage test input:

: PRINTX > < ? ; : FUNC1 = IF < PRINTX ELSE PRINTX THEN ; : FUNC2 DO
PRINTX WHILE ; : MAIN FUNC1 FUNC2 ;


First output file (interpretable Forth in C):

#include "f_cnv.h"

void* _printx_[]=
{
cfa enter__,
_greater_,
_less_,
_O_,
_exit_
};

struct dict_ _printx=
{
(struct dict_ *)NULL,
"PRINTX",
(void *) _printx_
};

void* _func1_[]=
{
cfa enter__,
_equal_,
_if_,
_less_,
_printx_,
_else_,
_printx_,
_then_,
_exit_
};

struct dict_ _func1=
{
(struct dict_ *)&_printx,
"FUNC1",
(void *) _func1_
};

void* _func2_[]=
{
cfa enter__,
_do_,
_printx_,
_while_,
_exit_
};

struct dict_ _func2=
{
(struct dict_ *)&_func1,
"FUNC2",
(void *) _func2_
};

void* _main_[]=
{
cfa enter__,
_func1_,
_func2_,
_exit_
};

struct dict_ _main=
{
(struct dict_ *)&_func2,
"MAIN",
(void *) _main_
};


Second output file (C code):

#include "f_cnv.h"

void _printx_(void)
{
_greater_();
_less_();
_O_();
};

void _func1_(void)
{
_equal_();
_if_();
_less_();
_printx_();
_else_();
_printx_();
_then_();
};

void _func2_(void)
{
_do_();
_printx_();
_while_();
};

int _main_(void)
{
_func1_();
_func2_();

return(0);

[luser droog]

I've been working along similar lines. I have a virtual machine
which has C functions for each opcode. And I'm trying to write
forth primitives that work on top of it. The instruction set of
the VM is a subset of the 8086 IA.

And just to be completely idiosyncratic, the assembly is all done
with C pre-processor macros.

https://groups.google.com/d/topic/comp.lang.asm.x86/vcxBUCvc4MQ/discussion

Since it's all virtualized, my code perhaps does not bear upon
OP's question. It doesn't try to execute native code.

[Kip Ingram]

> Thanks

Cecil, thought I'd let you know what my most immediate work on my system
is. As I mentioned earlier, I initially constructed this thing using a
farily standard FIG dictionary structure, with NFAs, LFAs, CFAs, and PFAs
all mingled together. I'm currently in the process of migrating these
into separate regions, so that I have one contiguous region of
definition cells into which the headers point. This will let me have
multiple entry points and is somewhat like what Chuck Moore did in
ColorForth. I intend eventually to implement a good bit of his latter
era stuff that promoted high compilation speed and so on.

I currently have the old headers and so on still in place, but all of
the headers created by the C code itself are also placed in their "new
home." Next step is to modify CREATE so that it creates that second
copy as well, then transition to USING the new headers, and then finally
to remove the creation of the old ones.

Eventually the header region will become a disk-resident array. Names
will appear there if they appear ANYWHERE in my source code - whether
that name has been compiled or not (if it's uncompiled the CFA pointer
will be zero). Moore didn't really discuss the implementation of his
latest stuff, but Jeff Fox wrote some stuff that gave a fair bit of
insight and I'm trying to "reverse engineer" from there.

I toyed with the idea of swallowing the whole ColorForth pill, but I
think I'll probably stop short. I want the fast compilation, the
source compression, and the support for multiple entry points, but
I wrote a "toy front end" that let me create colored source and found
it to be somewhat cumbersome. I'm glad it works well for CM, but
I don't think it will turn out to be my cup of tea. I am thinking
of adding some degree of intelligence to the compiler (primarily
in the area of handling literals beyond just integers) and I've
also given some thought to how I might support strong typing, which
would let me use the system for the sort of things I currently use
Python and Octave for.

I don't want that to be part of the base system, though; I don't
want to give up the clean simplicity except in situations where I
*need* the added power. Also, I want it to be a compile time
thing - once the code is compiled I want it to run as fast as
ever. I'll probably add a "type stack" that keeps up with what
the type of the items on the stack are - then I can use that
info as part of my search criteria and thus overload words.
For example, I could have + add two integers, two floats, two
matrices, etc. etc. etc. Whatever it's asked to add. There
would have to be a version of + in the dictionary for each
pattern I wanted to support.

Since I don't intend to make this the ground-level system, a
better way to describe what I'd be doing is, say, "writing a
Forth-like application similar to Octave," or whatever.

[minf...@arcor.de]

Thanks, although it is really rudimentary yet.
I guess your function FUNC2 needs some polishing too:
: FUNC2 DO PRINTX WHILE ;

;-)

[Rod Pemberton]

It appears that your conversion to C code is emitted from the Forth
environment. I noticed that some words use names, but others use
addresses. Do these correlate to the Forth words and primitives? It
appears to me that this isn't a 1:1 translation of compiled Forth
address list.

E.g., I'm assuming that you've replaced . (dot) with C code for the
primitive, and you've done the same for CR which probably isn't a
primitive. However, the one which sticks out the most to me is
mfprint(). I'd expect it to be a few operations if converted 1:1 from
Forth, e.g., one function for ." (dot-quote), another for allocating
the string, and another to emit the string. In other words, it seems
like some high-level parsing of the Forth code, or an optimization is
occurring, instead of a 1:1 translation from the address list or
threaded code.

> [snip]

>
> Thanks, although it is really rudimentary yet.

Yes, it doesn't have my Forth interpreter built-in, nor is it run from
my Forth interpreter. I.e., it's just C code based parsing. So, it
doesn't understand Forth. It just converts text-to-text.

It was written very early in the development of my Forth, before my
Forth was developed much. Obviously, the converter wasn't something I
continued developing, but it could be useful in the future, with some
work, perhaps. I do see a few things that could be quickly fixed which
would improve it, implement strings and integers, implement Forth-to-C
control-flow for common sequences, fix main's naming or set up a proper
main.

However, today, it may be better to emit C code from Forth, as I believe
you've done. My understanding is that is how Forth's metacompilation
works. I've not coded the Forth word SEE yet, but I would suspect that
it's a good starting point to selectively emit high-level Forth words as
C, similarly to my converter. For conversion, I would probably leave
the primitives blank, e.g., "void dup(void){/* FIX ME */};", as I could
copy the C code from my Forth interpreter to implement them.

Of course, except for the primitives and few other words, the majority
of my Forth is now written in high-level Forth words (colon-defs) which
are compiled at run-time. The high-level words can mostly be converted
1:1 to "void proc(void)" procedure calls for C and colon-defs to
procedures, like my converter did. So, the entire dictionary could
probably be converted to C upon Forth interpreter start up.

If I was to convert the dictionary to C upon start up of my Forth
interpreter, I'd probably do this from the C side, e.g., a modified
version of my Forth interpreter. OTH, doing that seems like too much
work. Every time I updated the Forth interpreter, I'd also have to
update the Forth interpreter program that was modified to emit C. So,
it might be easier to just fix the converter, since it's independent of
the Forth interpreter. Today, I have almost an entire high-level
dictionary to throw at the converter to detect it's flaws.

> I guess your function FUNC2 needs some polishing too:
> : FUNC2 DO PRINTX WHILE ;
>
> ;-)

:-)

Yes, I probably wasn't aware of Forth's BEGIN AGAIN or BEGIN UNTIL at
that point in time. So, C sufficed.

Early on, I didn't have Forth's control-flow words, and used a word I
called STAY which allowed me to then execute the current word again.
So, STAY would be followed by the word it was within at the end of the
definition. E.g., in high-level Forth, looping was implemented as:

: QUIT ... STAY QUIT ;

STAY discarded the top of the return stack, e.g., similar to R> DROP
but without placing data onto the data stack. This was actually used
in the address list for pre-compiled Forth words. This was very
effective, as I didn't need any of Forth's control-flow words for a
long time period during development. It reminds me of BRANCH
and ?BRANCH being so effective in implementing Forth's higher level
control-flow words.


Rod Pemberton
--
Opposition claims. "That's illegal." "That's unsustainable."

[minf...@arcor.de]

Am Montag, 28. Mai 2018 10:37:34 UTC+2 schrieb Rod Pemberton:

>
> It appears that your conversion to C code is emitted from the Forth
> environment. I noticed that some words use names, but others use
> addresses. Do these correlate to the Forth words and primitives? It
> appears to me that this isn't a 1:1 translation of compiled Forth
> address list.

The converter transpiles text to text and is a standalone C program.
Its main usage is to create C apps including subroutine-threaded
Forth functions. For instance
#REQUIRE lib/kernel.mfw includes the CORE and CORE-EXT wordset plus some few
MinForth specific words in source form
#REDUCE eliminates all words that are not needed in the MAIN Forth functions
by backtracking through the transpiler's symbol table
Words like mfdup, mfdrop etc are just syntactic sugar implemented a C
preprocessor macros

One "app" is a stand-alone MinForth interpreter in text mode (console mode in
Windows) with all #REQUIRED compiled to machine code. Hilevel Forth words
can be interpreted and compiled as address lists (C function pointer
equivalents). As a user you wouldn't see a difference between lolevel and
hilevel words as the precompiled dictionary is an extensible linked list.

>
> E.g., I'm assuming that you've replaced . (dot) with C code for the
> primitive, and you've done the same for CR which probably isn't a
> primitive. However, the one which sticks out the most to me is
> mfprint(). I'd expect it to be a few operations if converted 1:1 from
> Forth, e.g., one function for ." (dot-quote), another for allocating
> the string, and another to emit the string. In other words, it seems
> like some high-level parsing of the Forth code, or an optimization is
> occurring, instead of a 1:1 translation from the address list or
> threaded code.

Many simple primitives are declared as static inline functions. But decent
C compilers will inline anyhow where appropriate and minimize stack juggling
depending on the -O compiler flag option.

mfprint() is just syntactic sugar. For a desktop app it is:
#define mfprint(s) fputs((char*)(s),stdout)
(for umbilical controller apps there are other definitions)

For S" strings the transpiler generates C code to define an equivalent C string
and push its address and length onto the Forth stack:
mfpushs("i-am-a-string");

The hilevel interpreter does it the classic Forth way: it compiles the xt of
the SLIT primitive followed by a copy of the string and jumps over it.

[hughag...@gmail.com]

On Wednesday, May 23, 2018 at 3:59:25 AM UTC-7, Kip Ingram wrote:
> I'm a hardware guy by training, and my eventual goal (sort of
> a "bucket list" thing) is to build a serious computer fully
> from scratch, using an FPGA implemented processor core that I
> have some rather large number of in the system. This Forth
> will become the operating system for that machine. So I want
> to play with things like clean Forth support for multiple
> cores, code parallelization mechanisms, and so on.
>
> It's a hobby, but it's a pretty darn serous one.

I have a Forth processor design that I'm working on.
I can email it to you if you are interested. Contact me privately: hughag...@gmail.com

It doesn't support interrupts, so you are not going to have an OS --- you are going to have a paced-loop --- it is mostly for motion-control.
It has a fast multiply for the PID algorithm.

It does have three instructions executing concurrently in a single clock-cycle --- should be pretty fast --- I have an FPGA implementation in the works.
The design is loosely based on the MiniForth (now known as RACE) from Testra.

[hughag...@gmail.com]

On Tuesday, May 22, 2018 at 8:08:17 AM UTC-7, Cecil - k5nwa wrote:
> The C JIT I been planning on using is TCC it has the assembler/linker
> built in so it generates ready to run code on the fly, it has limited
> code optimization but it's small and extremely fast at compiling. For C
> there are a couple of other JIT packages available including GCC...

I'm not aware of any C compiler that does incremental compilation --- what I've seen is that the C compiler only compiles entire .C files into .OBJ files.

TCC means "Borland Turbo C" to me --- that was MS-DOS though --- what is this TCC that you are mentioning?

[Cecil Bayona]

It's not Borland C , it's Tiny C Compiler available in versions that
generate 32 bit or 64 bit code for Windows or Linux.

It has a DLL file that with calls to it will compile a piece of C code
passed to it in a string and generate executable code really fast. GCC
also has a similar offering but it's more complicated to use. TCC is
also blinding fast so it makes it more practical to use, no linker
necessary either although you have the option of doing so. It allows one
to generate machine code for i386 or ARM CPUs on the fly from a text string.

< https://bellard.org/tcc/ >

This is just an experiment for me to try out new things and see how they
work. Right now I been working on a Forth Compiler that uses C to create
the primitives and VM for a forth, there is no assembler as the machine
is Token Threaded, the tokens are used to index into an array that
contains a list of the addresses of the primitive C tokens. A very
portable way to move the Forth to any CPU that has a decent C compiler.
You can not add a new primitive token on the fly to do so one must edit
the C source code and recompile, not the end of the world but a pain.

I want to make C code the assembler which will generate inline machine
code on the fly instead of tokens it won't be so portable a JIT C
compilers are a rare beast to find and don't exist for most CPUs.

As mentioned before I been ill lately but I'm getting better so soon I
will be getting back to this project.
--
Cecil - k5nwa

[Cecil Bayona]

Looks like the topic is coming to a slow end so I would like to thank
all those that gave comments and ideas. I looks like most of the
comments are in line with what I was thinking of doing a modification of
the original idea is to expand the size of the current code segment
instead of creating a new segment, either way it would work but I will
try out expanding the existing segment as it is somewhat simpler to use.

Any ideas is welcomed, Thanks.

--
Cecil - k5nwa