how to create standalone EXE from inside forth with actual working environment (in forth or c)

[make....@gmail.com]

Hi all,
I'm a "old" hobby programmer (windows) and i try to write a tcl-pkg (in c with poor knowlege) to use forth in tcl/tk. The goal is to generate a minimal tclkit (bundle all relevants programms, tools etc. in one execute file) --> Less is more

I found several forth implementations in c wich i can use to create the package and that works fine. Most of the have system function to save an image of the memory to file to use later the last state of forth working environment.

But inside forth i want to save the working environment with a new standalone execute file like as:

on command prompt:
xyz_forth.exe -f core.fth (minimal forth words in forth language)
inside xyz_forth:
: test 1 1 + . ;
(and now) fsave new_forth.exe
bye

now on command prompt:
new_forth.exe
inside new_forth:
test (--> output 2)
bye


I believe in earlier time i saw that on fpc (tom zimmer).

What is the trick, how is the way to define a word inside forth or a system call in c to do that?

thanks a lot and greetings - kmatze

[Julian Fondren]

On Wednesday, December 7, 2016 at 4:55:20 AM UTC-6, make....@gmail.com wrote:
> Hi all,

>
> But inside forth i want to save the working environment with a new standalone execute file like as:
>
> on command prompt:
> xyz_forth.exe -f core.fth (minimal forth words in forth language)
> inside xyz_forth:
> : test 1 1 + . ;
> (and now) fsave new_forth.exe
> bye
>
> now on command prompt:
> new_forth.exe
> inside new_forth:
> test (--> output 2)
> bye
>

Hello,

What you're describing is a normal feature of any Forth that does
offer any kind of 'turnkey' feature. I don't even know how you
could've saved the state without being able to do as you describe.
So if you tell us what Forth you're working with, someone can
probably tell you exactly what to do.

-- Julian

[make....@gmail.com]

>
> Hello,
>
> What you're describing is a normal feature of any Forth that does
> offer any kind of 'turnkey' feature. I don't even know how you
> could've saved the state without being able to do as you describe.
> So if you tell us what Forth you're working with, someone can
> probably tell you exactly what to do.
>
> -- Julian

Hi Julian,
thx for your fast answer. Even I play a little bit with zforth (a tiny forth implementation in c by zevv - https://github.com/zevv/zForth). I don't now it's a really forth but it's enough for me.
Inside this forth there is a c function "save" which saves / dumps a memory image as "zforth.save". In zforth i can define ": save 131 sys ;" to use this function.

if you now start "zforth.exe" with the argument "-l zforth.save" from command line you have a forth with the saved working environment.

I thought now to 'bundle' this two files to a single execute file and I don't know how. That's is the background to my question.

I hope you anderstand my bad old school english ;-)

Greetings - kmatze

[Julian Fondren]

On Wednesday, December 7, 2016 at 7:31:07 AM UTC-6, make....@gmail.com wrote:
>
> Hi Julian,
> thx for your fast answer. Even I play a little bit with zforth (a tiny forth implementation in c by zevv - https://github.com/zevv/zForth). I don't now it's a really forth but it's enough for me.
> Inside this forth there is a c function "save" which saves / dumps a memory image as "zforth.save". In zforth i can define ": save 131 sys ;" to use this function.
>
> if you now start "zforth.exe" with the argument "-l zforth.save" from command line you have a forth with the saved working environment.
>
> I thought now to 'bundle' this two files to a single execute file and I don't know how. That's is the background to my question.
>
> I hope you anderstand my bad old school english ;-)
>
> Greetings - kmatze

OK. I would say, unless zforth's own docs are hiding the
functionality, that there's no way to do this. zforth can dump an
image but can't create an executable. Under windows you can still
create a shortcut (right-click zforth.exe, create shortcut, then edit
the shortcut) that includes your own arguments--so you can have
something that behaves like an executable, that actually calls
zforth.exe and loads your code.

win32forth, ciforth (wina), SwiftForth, and VFX Forth can all probably
create executables as you'd like, however. The first two are free.


-- Julian

[Alex]

It's not easy to do; and the only Windows Forth I know that can build a
native EXE file from a running copy as you indicate is Win32Forth
version 6. (Does MPE's VFX do this? It may, since IIRC it can build DLLs.)

I wrote the code for Win32Forth native EXEs. You might want to look at
it; the source modules are imagehds.f (which descrbes the PE headers)
and imageman.f (which builds the EXE file). It is over 1000 lines of
Forth and comment, and has several notable restrictions that I hadn't
got round to addressing:

1. It can't build relocatable files; they can only execute at fixed
addresses (the default for EXE files is 0x0040000)
2. It doesn't build DLLs (although it could; issue 1 would need fixed
first).
3. It doesn't handle resources, so a seperate resource editor has to be
used.

If you require these features, the solutions will be messy, though not
impossible. There is some non-Forth literature on the subject; search on
object serialization for example.

There are other ways of writing FSAVE though, and I would suggest the
method that a number of other Forth's use. Write an FSAVE that creates
an image file of your design; and write an EXE that can load & execute
it. IIRC that's the mechanism gforth uses. This also has the advantage
of possibly making it easier to support both a 32bit and 64bit Forth
with a single image (or very similar images).

Regardless of which method you choose, these are still considerations
that make the task complex. Initialisation needs to be done to make the
newly restored environment look like the FSAVEd environment.
. File handles do not persist
. Dynamically allocated memory (heaps for instance) does not persist,
and repeating allocations doesn't guarantee the same memory addresses
. The stack and rstack may be difficult to copy & restore
. Error recovery may not work correctly on the saved EXE.

--
Alex

[Albert van der Horst]

In article <332b84e4-3e3d-426c...@googlegroups.com>,

<make....@gmail.com> wrote:
>Hi all,
>I'm a "old" hobby programmer (windows) and i try to write a tcl-pkg (in c with poor knowlege) to use forth in tcl/tk. The goal is to generate a minimal tclkit (bundle all relevants programms, tools etc. in one execute file) --> Less is more
>
>I found several forth implementations in c wich i can use to create the package and that works fine. Most of the have system function to save an image of the memory to file to use later the last state of forth working environment.
>
>But inside forth i want to save the working environment with a new standalone execute file like as:
>
>on command prompt:
> xyz_forth.exe -f core.fth (minimal forth words in forth language)
> inside xyz_forth:
> : test 1 1 + . ;
>(and now) fsave new_forth.exe
> bye

In ciforth this is (for windows)
"
wina.exe (Now you have a minimum system)
1 LOAD (get library mechanism)
WANT SAVE-SYSTEM ( get a tool from the library)
: test 1 1 + . ; (still inside wina.exe)
"newforth.exe" SAVE-SYSTEM
BYE
"
You can also abuse the compiler mechanism, defeating error handling for
turnkey systems by just starting the interpreter (QUIT) as the turnkey
command.

-------------- newforth.frt ------

: test 1 1 + . ;

: main 'ERROR RESTORED 'OK RESTORED QUIT ;
-------------------------------------------

then
wina -c newforth.frt ( generates newforth.exe)

This has the advantage that the -c option (through carnal
trickery) avoids that the SAVE-SYSTEM stuff remains in the
executable.

>
>now on command prompt:
> new_forth.exe
> inside new_forth:
> test (--> output 2)
> bye
>

You can send the newforth.exe as a single file through e-mail
to your mother and she can do the above.

This "just works" (tm).

All tools in ciforth are made to work properly after SAVE-SYSTEM, 1]
e.g after WANT ALLOCATE and using ALLOCATE all allocated stuff
is accessable and you can go on allocating and freeing.

AMDX86 ciforth beta $RCSfile: ci86.gnr,v $ $Revision: 5.140 $
DSP@ .
8394850304 OK
1 LOAD WANT ALLOCATE
OK
100 ALLOCATE CONSTANT AAP
OK
WANT SAVE-SYSTEM
OK
"q" SAVE-SYSTEM
OK
BYE

As you see 8 gbyte is available within this Forth,
and a quarter is taken by the heap as soon as you load the
ALLOCATE mechanism.

~/PROJECT/ciforth$ ls -l q
-rwxr-xr-x 1 albert albert 2097712480 Dec 7 15:48 q
~/PROJECT/ciforth$

So the executable q is huge too.

<SNIP>

>
>thanks a lot and greetings - kmatze
>

Groetjes Albert

1] With the exception of multitasking
--
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

[Stephen Pelc]

On Wed, 7 Dec 2016 13:50:53 +0000, Alex <al...@rivadpm.com> wrote:

>It's not easy to do; and the only Windows Forth I know that can build a
>native EXE file from a running copy as you indicate is Win32Forth
>version 6. (Does MPE's VFX do this? It may, since IIRC it can build DLLs.)

Of course VFX can do this. VFX is for writing real applications.
Our customers release real applications all the time.

Stephen

--
Stephen Pelc, steph...@mpeforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

[Alex]

On 12/7/2016 20:24, Stephen Pelc wrote:
> On Wed, 7 Dec 2016 13:50:53 +0000, Alex <al...@rivadpm.com> wrote:
>
>> It's not easy to do; and the only Windows Forth I know that can build a
>> native EXE file from a running copy as you indicate is Win32Forth
>> version 6. (Does MPE's VFX do this? It may, since IIRC it can build DLLs.)
>
> Of course VFX can do this. VFX is for writing real applications.
> Our customers release real applications all the time.
>
> Stephen
>

I stand (just about, since that was quite a poke in the eye ;)
corrected. There appear to be several.

Does VFX build a PE reloc section? I found it quite a challenge; the
easiest (but untried) method seemed to be to build the image B at a
different addresses to the running version A. Compare the two, and any
contents that differ A-B require a relocation entry.

--
Alex

[Elizabeth D. Rather]

On 12/7/16 10:24 AM, Stephen Pelc wrote:
> On Wed, 7 Dec 2016 13:50:53 +0000, Alex <al...@rivadpm.com> wrote:
>
>> It's not easy to do; and the only Windows Forth I know that can build a
>> native EXE file from a running copy as you indicate is Win32Forth
>> version 6. (Does MPE's VFX do this? It may, since IIRC it can build DLLs.)
>
> Of course VFX can do this. VFX is for writing real applications.
> Our customers release real applications all the time.
>
> Stephen

Likewise SwiftForth.

Cheers,
Elizabeth

--
==================================================
Elizabeth D. Rather (US & Canada) 800-55-FORTH
FORTH Inc. +1 310.999.6784
5959 West Century Blvd. Suite 700
Los Angeles, CA 90045
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

[Ron Aaron]

On 08/12/2016 2:59, Elizabeth D. Rather wrote:
> On 12/7/16 10:24 AM, Stephen Pelc wrote:

>> Of course VFX can do this. VFX is for writing real applications.
>> Our customers release real applications all the time.
>>
>> Stephen
>
> Likewise SwiftForth.
>

Likewise 8th (in a different manner)

[Stephen Pelc]

On Wed, 7 Dec 2016 22:05:16 +0000, Alex <al...@rivadpm.com> wrote:

>Does VFX build a PE reloc section? I found it quite a challenge; the
>easiest (but untried) method seemed to be to build the image B at a
>different addresses to the running version A. Compare the two, and any
>contents that differ A-B require a relocation entry.

We do not build a PE relocation section because we use the same
relocation code for all systems. This code uses the "compile twice at
different addresses" approach. We also used this technique for
binary overlays back in the dim and distant.

[make....@gmail.com]

Wow,so many answers. Thx for all.
I'm confuse a little bit but i will studdy your messages and will inform you about my forth steps - progressions ;-)

[Anton Ertl]

Alex <al...@rivadpm.com> writes:
>There are other ways of writing FSAVE though, and I would suggest the
>method that a number of other Forth's use. Write an FSAVE that creates
>an image file of your design; and write an EXE that can load & execute
>it. IIRC that's the mechanism gforth uses. This also has the advantage
>of possibly making it easier to support both a 32bit and 64bit Forth
>with a single image (or very similar images).

Gforth has different images for 32-bit and 64-bit, as well as
different byte orders. You can use the same image on different
architectures, as long as cell size and byte order agree; if the image
includes an assember or disassembler, you can still run it on a
different architecture, but of course the assembler and disassembler
won't be very useful.

A common image format for different cell sizes and byte orders is
possible, but not easy. The cell size is particularly troublesome:
Consider a structure with a mixture of cell-sized and char-sized
elements, with alignment padding. How do you represent field
offsets in a cell-size-parametric way?

>Regardless of which method you choose, these are still considerations
>that make the task complex. Initialisation needs to be done to make the
>newly restored environment look like the FSAVEd environment.
>. File handles do not persist
>. Dynamically allocated memory (heaps for instance) does not persist,
>and repeating allocations doesn't guarantee the same memory addresses
>. The stack and rstack may be difficult to copy & restore

These are all things that could be restored, if we wanted to.
However, Forth systems have traditionally not done so, and I have
heard few complaints about that.

>. Error recovery may not work correctly on the saved EXE.

What do you mean with that?

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2016: http://www.euroforth.org/ef16/

[Alex]

On 12/8/2016 15:58, Anton Ertl wrote:

> Alex <al...@rivadpm.com> writes:

>
>> . Error recovery may not work correctly on the saved EXE.
>
> What do you mean with that?
>

Exception handling may change the environment of the system between an
initialisation and the point where it's saved; but when the saved EXE is
run, everything the system manages returns to the initialisation state
(qv file handles). Under Windows, automated guard page exceptions to
change reservations to allocations on the stack are one where the
application programmer has no control.

--
Alex

[make....@gmail.com]

Hi all,
over the time i have found a solution. Maby not so elegant (i think) but it works fine for me:

I work with windows vista and gcc 3.4.5.

The Idea:
- I created a minimal execute file (mForth.exe) only with bootstrap
- inside i can load forth src-files and then
- i can create a new execute file with my new command fsave (system call in c)
- fsave
- uses the core of mForth.exe as and copy this to a new exe file
- appends to the end of the new exe file the actual used memory (dynamicle size) as image
- if the new exe file started:
- it looks first for a first argument and load an image with this name directly
- if not it scans himself to look if image at the end of file and start it
- if not it loads standard bootstrap

The clou: I compress the mForth.exe (29.708 bytes) first with strip command and the result (14.336 bytes) with upx command and get a very tiny

exe file (7.186 bytes).

With this now I load a minimal word set:
words name next ." s" loop loop+ do j i fi else unless if times until again begin
var allot here .. cr != not =0 >= <= > < dec inc +! over postpone ] [ # , @ ! fsave
version save include sin bye tell . emit _postpone compiling latest h & ## lits
key ,, pick sys = r> >r ( ' jmp0 jmp rot swap !! @@ immediate pickr dup dr op
% / * - + ; : <0 lit exit

and with fsave i get a new exe file with 8.285 bytes that runs!

READY ;-))

Greetings - kmatze -

[ste...@mpeforth.com]

> It's not easy to do; and the only Windows Forth I know that can build a
> native EXE file from a running copy as you indicate is Win32Forth
> version 6. (Does MPE's VFX do this? It may, since IIRC it can build DLLs.)

VFX Forth can save both EXE files and DLLs.

Most serious Windows Forth can do this.

Stephen

[ste...@mpeforth.com]

> It's not easy to do; and the only Windows Forth I know that can build a
> native EXE file from a running copy as you indicate is Win32Forth
> version 6. (Does MPE's VFX do this? It may, since IIRC it can build DLLs.)

[Albert van der Horst]

In article <17a81d8a-e5a8-4b51...@googlegroups.com>,

Even a less serious Forth like wina.exe can make EXE files.
DLL is a little harder, and a separate effort.
So I don't think being able to do DLL's doesn't warrant any
conclusions about EXE's.

>
>Stephen

Groetjes Albert

[Elizabeth D. Rather]

Yes, including SwiftForth.

Cheers,
Elizabeth

--
Elizabeth R. Conklin
FORTH, Inc.
6080 Center Drive, Suite 600
Los Angeles, CA 90045
USA

[Albert van der Horst]

The proper exception handling for an application program is
to have all foreseen exceptions caught and consider all unforeseen
exceptions fatal. Fatal exceptions should abort the application,
not fall into the QUIT interpreter loop. 1]
This is what is built in into the -c (compile) option of ciforth).
Assuming the user of an application can do some useful error
recovery within Forth is in general invalid.

This is not to say that special situations may not arise.
The application may be geared towards very knowledgeable users, or
a turnkey control system may be designed to do a warm start
with as little danger as possible.

>--
>Alex

Groetjes Albert

1] This might even be a security leak.

[JUERGEN]

MPE did a special exe for the VXXTESTAPP - the 35 Word Forth Example - see https://wiki.forth-ev.de/doku.php/en:projects:a-start-with-forth:start there is a link to download it and it includes SAVE to save your work.

[Alex]

On 4/29/2017 12:09, Albert van der Horst wrote:
> In article <o2coqg$o7l$1...@dont-email.me>, Alex <al...@rivadpm.com> wrote:
>> On 12/8/2016 15:58, Anton Ertl wrote:
>>> Alex <al...@rivadpm.com> writes:
>>
>>>
>>>> . Error recovery may not work correctly on the saved EXE.
>>>
>>> What do you mean with that?
>>>
>>
>> Exception handling may change the environment of the system between an
>> initialisation and the point where it's saved; but when the saved EXE is
>> run, everything the system manages returns to the initialisation state
>> (qv file handles). Under Windows, automated guard page exceptions to
>> change reservations to allocations on the stack are one where the
>> application programmer has no control.
>
> The proper exception handling for an application program is
> to have all foreseen exceptions caught and consider all unforeseen
> exceptions fatal. Fatal exceptions should abort the application,
> not fall into the QUIT interpreter loop. 1]

Well, of course, since by definition fatal errors are fatal, but not all
unforeseen errors can be expected to be fatal and abort; since that
would make a development environment very difficult indeed.

Re-entering QUIT allows the application to be debugged.

> This is what is built in into the -c (compile) option of ciforth).
> Assuming the user of an application can do some useful error
> recovery within Forth is in general invalid.

I respectfully disagree; for example, division by zero.

>
> This is not to say that special situations may not arise.
> The application may be geared towards very knowledgeable users, or
> a turnkey control system may be designed to do a warm start
> with as little danger as possible.

Yes. WF32 does have a mechanism for this; there's a sequence that does
initialization which can then call the end user's application. The
default application is to have QUIT executed.

>
>> --
>> Alex
>
> Groetjes Albert
>
> 1] This might even be a security leak.

I doubt it; one could consider the entire Forth system as a security
leak in that case.

>

--
Alex

[Albert van der Horst]

In article <oe2jbs$6bp$1...@dont-email.me>, Alex <al...@rivadpm.com> wrote:
>On 4/29/2017 12:09, Albert van der Horst wrote:
>> In article <o2coqg$o7l$1...@dont-email.me>, Alex <al...@rivadpm.com> wrote:
>>> On 12/8/2016 15:58, Anton Ertl wrote:
>>>> Alex <al...@rivadpm.com> writes:
>>>
>>>>
>>>>> . Error recovery may not work correctly on the saved EXE.
>>>>
>>>> What do you mean with that?
>>>>
>>>
>>> Exception handling may change the environment of the system between an
>>> initialisation and the point where it's saved; but when the saved EXE is
>>> run, everything the system manages returns to the initialisation state
>>> (qv file handles). Under Windows, automated guard page exceptions to
>>> change reservations to allocations on the stack are one where the
>>> application programmer has no control.
>>
>> The proper exception handling for an application program is
>> to have all foreseen exceptions caught and consider all unforeseen
>> exceptions fatal. Fatal exceptions should abort the application,
>> not fall into the QUIT interpreter loop. 1]
>
>Well, of course, since by definition fatal errors are fatal, but not all
>unforeseen errors can be expected to be fatal and abort; since that
>would make a development environment very difficult indeed.
>
>Re-entering QUIT allows the application to be debugged.

No it doesn't. A real application cannot be expected to be debugged
by a user.
On division by zero, MS-Windows sees that the program has crashed
and suggest to the user to send a message to the maker of the
program to the effect that there is a severe defect.

>
>> This is what is built in into the -c (compile) option of ciforth).
>> Assuming the user of an application can do some useful error
>> recovery within Forth is in general invalid.
>
>I respectfully disagree; for example, division by zero.

What kind of applications do you have in mind?
I think of a demonstration of midi voices where the user can select
the midi voice to play "mary had a little lamb" on.

>>
>> This is not to say that special situations may not arise.
>> The application may be geared towards very knowledgeable users, or
>> a turnkey control system may be designed to do a warm start
>> with as little danger as possible.
>
>Yes. WF32 does have a mechanism for this; there's a sequence that does
>initialization which can then call the end user's application. The
>default application is to have QUIT executed.

So you don't deal in applications where the user need not be
aware that it is written in Forth?

>>> --
>>> Alex
>>
>> Groetjes Albert
>>
>> 1] This might even be a security leak.
>
>I doubt it; one could consider the entire Forth system as a security
>leak in that case.

We don't disagree. Having Forth at disposal is a security leak in itself.
So in a secure application you can't have an escape into Forth,
not even for errors that "cannot/should not occur".

>--
>Alex

Groetjes Albert

[Elizabeth D. Rather]

An application that is to be released to "real world" users should be
debugged thoroughly before being released. It certainly shouldn't have
bugs that allow dividing by zero or other fatal errors. During
development, errors need to be handled in such a way as to facilitate
debugging.

>>> This is what is built in into the -c (compile) option of ciforth).
>>> Assuming the user of an application can do some useful error
>>> recovery within Forth is in general invalid.
>>
>> I respectfully disagree; for example, division by zero.
>
> What kind of applications do you have in mind?
> I think of a demonstration of midi voices where the user can select
> the midi voice to play "mary had a little lamb" on.
>
>>>
>>> This is not to say that special situations may not arise.
>>> The application may be geared towards very knowledgeable users, or
>>> a turnkey control system may be designed to do a warm start
>>> with as little danger as possible.
>>
>> Yes. WF32 does have a mechanism for this; there's a sequence that does
>> initialization which can then call the end user's application. The
>> default application is to have QUIT executed.
>
> So you don't deal in applications where the user need not be
> aware that it is written in Forth?

We certainly do. We test them pretty intensively before they're
released. Most do not have an end-user interface that works at the QUIT
level, though. That's mainly for debugging.

>>>> --
>>>> Alex
>>>
>>> Groetjes Albert
>>>
>>> 1] This might even be a security leak.
>>
>> I doubt it; one could consider the entire Forth system as a security
>> leak in that case.
>
> We don't disagree. Having Forth at disposal is a security leak in itself.
> So in a secure application you can't have an escape into Forth,
> not even for errors that "cannot/should not occur".

In most cases, that's true. So, the summary on this issue is that error
detecting that goes to QUIT is essential in the debugging process, but
not relevant in a released, end-user application.

Cheers,
Elizabeth

--
Elizabeth D. Rather

[Alex]

On 4/30/2017 20:35, Elizabeth D. Rather wrote:
>
> So, the summary on this issue is that error
> detecting that goes to QUIT is essential in the debugging process, but
> not relevant in a released, end-user application.

Except that every application (even those of modest size & ambition)
will have bugs; and I would want my application to give me the
opportunity to get better diagnostics than "hey, I'm bust". Or even, as
I have seen with some, an application that simply disappears in a puff
of transparent smoke.

--
Alex

[Elizabeth D. Rather]

Sure. You want it to communicate somehow what went wrong. The most
common failures in production programs are some kind of hardware glitch
and wrong user input. User input should be vetted thoroughly at the
point of entry, so you don't have bogus numbers turning up down below.
If you have hardware, you can pretty well guarantee it *will* fail, so
you put in diagnostics that will tell you something useful, if possible.
Unfortunately, most of the programs we work with don't have real
terminals, which makes it a little harder.