Essential properties of current standard Forth

[Anton Ertl]

At EuroForth Andrew Read gave an impromptu talk where he suggested a
next-standard effort that would not be shackled by
backwards-compatibility with the current (Forth-83/94/2012) line of
standards.

While there was not that much desire for such an effort in the
audience, the talk got me thinking about what is essential in the
current line of Forth standards and which features of a Forth system
would require such a next-standard effort (if there are people that
want to have a standard for that).

What is not essential:

Of course, there are people (including committee members) who disagree
with some choices made in the standard, but that's a matter of opinion
and not the kind of "essential" I have in mind; In particular, if
people could not find a consensus on a different choice earlier, will
it help to start from scratch? In at least one case, there actually
has been consensus on taking a different choice than an earlier
standard (standardizing "{:" and making "LOCALS|" obsolescent), so if
that happens, the current standard process can change choices, too,
slowly. We don't need a from-scratch effort for that.

Another issue is removing features, to achieve some form of
minimalism. We have a mechanism in the standards process for removing
features (it takes it's time, though), so if the community decided we
really want minimality, and if we actually could find consensus on
what we want in a minimal system, we could go there; in practice, this
will not happen, but that has more to do with there being no consensus
that we want a minimal system and no consensus on what it would look
like.

Another issue is that of names. While machine Forth calls the
exclusive-or operation OR, I don't consider that an essential feature
of machine Forth. It would be pretty much the same if Chuck Moore had
chosen to call it XOR. Conversely, if we wanted to remove inclusive
OR, later add an OR that's a synonym for XOR, and then remove XOR,
this could be done, very slowly, over several standards releases.

Possibly essential:

The syntax difference between machine Forth and Color Forth (color
that indicates how to text-interpret the string) cannot be
accommodated in the current line of standards; while it's just syntax,
it's a change that does not allow a transition period, so it would
make no sense to try to do it in the current line of standards.

Pretty essential:

Statically determined stack depth: Requiring that, as sketched for
PAF, would disallow some standard programs (and a lot of buggy
programs). I guess we could get there in the current line of
standards, but I don't expect it.

Static type checking: StrongForth did that, but also implemented
overloading (e.g., DUP worked as 2DUP when there was a double on the
TOS). Static type checking would first require a type system; it
would probably disallow more standard programs, especially if you want
to have a somewhat safe type system; the actual repercussions would
depend on the type system. Overloading would be even more disruptive,
by a lot.

Data stack items: In the current standards, the data stack item is a
machine word (cell), used as integer, address, or a group of bits and
its interpretation depends on the operation; there are a number of
systems that deviate from this:

Floats: There is one suggestion (and a Russion system that works this
way) to use floats as stack items, with words like + working on
floats, and IIRC addresses represented by NaNs (or maybe by
interpreting the bit pattern as address). That's pretty far from the
current standard.

Dynamically typed stack items (e.g., Factor, 8th, Oforth, with Factor
also doing static type checking to some extent), with dynamic type
checking in the operations and possible overloading, and possible
automatic memory reclamation. That's even further from the current
line of standards. Looking at the number of systems that go that way,
there seems to be quite a bit of interest in that direction, so maybe
that's something where such a standard that's not rooted in the
current line of standards might make sense. But I guess that the
implementors and users of these systems have to work it out if they
need a standard for that, or if they are happy with the different
systems that they have now.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2016: http://www.euroforth.org/ef16/

[hughag...@gmail.com]

On Thursday, September 15, 2016 at 10:24:49 AM UTC-7, Anton Ertl wrote:
> At EuroForth Andrew Read gave an impromptu talk where he suggested a
> next-standard effort that would not be shackled by
> backwards-compatibility with the current (Forth-83/94/2012) line of
> standards.
>
> While there was not that much desire for such an effort in the
> audience, the talk got me thinking about what is essential in the
> current line of Forth standards and which features of a Forth system
> would require such a next-standard effort (if there are people that
> want to have a standard for that).

In this thread,
https://groups.google.com/forum/#!msg/comp.lang.forth/D0PmblPgthM/o6Ivh2RbAgAJ;context-place=forum/comp.lang.forth

I said this:

On Wednesday, May 4, 2016 at 5:03:16 PM UTC-7, hughag...@gmail.com wrote:
> I said earlier that I would be the "chief" and I would appoint the committee members (similar to how Elizabeth Rather appoints the committee members for Forth-200x). I'm not going to be like Leon Wagner though and tell everybody exactly what to believe, give them assignments, and write the specs for them (basically, treat them like code-monkeys). I just want to appoint the members so I can weed out people like Alex McDonald who claimed to "do standards for a living" (whatever the hell that means) and who knew very little about Forth. People can do whatever they want, so long as they adhere to the basic principles:
>
> 1.) Minimalism. All that cruft in ANS-Forth will be discarded (for example: DO loops will go). I would expect the language to be less than half the size of ANS-Forth. As for the extension code-libraries, they will make the language bigger, but they are optional. They are containers (my list.4th and association.4th) or they are associated with a particular data-type (my string-stack.4th). The emphasis here should be on usefulness --- I was impressed by how Factor used "sequences" as a standard data-type, and this was my inspiration for list.4th --- also, almost all programs for desktop-computers are going to use strings, so having a standard library such as string-stack.4th is going to make the programs easy to read, rather than everybody doing something different in every program.
>
> 2.) Behavior. The focus should be on standardizing behavior rather than implementation. One of the problems with ANS-Forth was that it standardized implementation. For example, ANS-Forth assumes that the >R data and the DO loops use the same stack, so they were standardizing on the PDP-11 implementation that had to do this because of a shortage of registers. Quite a lot of ANS-Forth assumes that the target processor is a PDP-11, apparently because that is the only processor that Elizabeth Rather is familiar with.
>
> 3.) Quotations. My system with the fat-xt will be used to allow quotations to access the parent function's local variables. This is trivial to implement! This is the foundation to writing general-purpose data-structures --- I really see this as bring Forth into the 21st century --- make it comparable to the other desktop-computer language such as Lisp --- unlike Lisp though, we will not have garbage-collection or tagged data (we will not have Dynamic-OOP). The emphasis will still be on speed, and Dynamic-OOP is a dog, so leave that to the script kiddies.
>
> 4.) Numerical. The 64-bit cell with the Fixed or Float data-type is fundamental to the language. I think this makes the language much simpler than ANS-Forth where integers and floats are distinct (so, this fits in with the minimalism idea). The "killer app" is CAM, but most machinists have experience primarily with using pocket calculators for figuring out gcode, and pocket calculators don't consider integers and floats to be distinct data-types. Also, it is important to define what the precision of the numbers is in the standard so the programmer knows what the results of numerical calculations are going to be and can expect the same results on every compliant compiler (so long as he sticks with either the Fixed or Float version, as they give different results from each other). I originally was only going to have the Fixed version, but Ilya Vasilyev convinced me to have a Float version also and to make them compatible with each other (except for a slight difference in the numeric results in some cases). This was here:
> https://groups.google.com/forum/#!searchin/comp.lang.forth/ilya%7Csort:date/comp.lang.forth/R7K3FWXnyKE/0_LJ13ZfBQAJ
>
> 5.) Cross-Compilation. The compiler should have all the necessary features for writing a cross-compiler. This is actually pretty easy --- it is amazing that ANS-Forth failed at such a simple task (I'm highly suspicious that ANS-Forth was purposely sabotaged to make cross-compilation difficult in order to prevent common users from writing a cross-compiler in competition with Forth Inc. and MPE who make most of their money selling vendor-specific cross-compilers) --- I know how to write a cross-compiler, so I will make sure that this is supported.
>
> These are the 5 principles of Straight-Forth. Who disagrees with any of these principles? Which ones? Why?
>
> Note that LFTforth is for micro-controllers and would have different principles that it is based on --- that is really Raimond's thing, not mine.

[anothe...@gmail.com]

(I tried to post this last night, unfortunately via google groups... the text box grayed out after clicking "post" and I assumed it was going to post it... came back this morning, it's still grayed out and not posted. Wow. The text isn't selectable, so I can't even copy and paste it here, I have to manually re-type it.)

I (relative noob here), in my silliness, think that if I were to think up a separate standard from ANS Forth, I would emphasize optional wordsets. One of the problems I've seen in the Lisp world (or at least, I think I've seen) is that often each implementation will provide useful capabilities outside of the standard, but with different names and interfaces and slightly different behavior. Consequently portability libraries had to be developed that abstract away the differences.

I guess what I'm trying to say is that it would be great to have standardized libraries. And by "libraries" I don't mean common abstractions that can be built using the core wordset like it seems a lot of other languages have, I mean a minimalist set of capabilities that couldn't be used portably otherwise - not even with a C interface, because that would only work on systems with that shared library. Optionally-included bare-minimum forth-style batteries, basically. And the core wordset could include a simple way to test whether optional wordsets are supported, which might vary not only based on implementation but also based on hardware ("how the heck am I supposed to use graphics without a display, or networking without any physical way to connect, or serial without any serial port..." etc). Perhaps this is a symptom of coming from languages where interaction with assembler is extremely uncommon (where being part of the language is often the entire difference between "works smoothly" and "almost impossible"), but having these capabilities be accessible in a standard way (or inaccessible in a standard way, for that matter) sounds really useful.

In short, what I'm trying to say is that it seems (I could be wrong, like I said, I'm a noob) like it's hard to handle non-portability portably. Different Forth implementations often have different goals, and the least we can do is have the Forths with similar capabilities present similar interfaces. To this end, I'd like more optional wordsets that may be more specialized and a way in the core wordset to query which wordsets the current implementation supports for simple compatibility matching. Sockets, graphics (even if it's just a common interface to openGL or Vulkan), I/O beyond just text (mouse? sound?), possible a C interface - these are the sort of things that are necessary in order to make generally useful programs (I love playing with algorithms and extending the language as much as anyone else, but at some point stuff needs to get done), and there should be a common way of using that functionality where it is available.

Am I off-base here? Is this contrary to the goals of the Forth community? Is this an irrelevant comment for the current discussion? Is the wall of text uncomfortably large? I don't know. I hope it's relevant. Thanks for taking the time to read!

[Cecil Bayona]

I'm by no means an expert but it seems that there are conflicting
requirements for Forth but they are not unsolvable.

On the one hand some want Forth to be lean and mean, leaner that the
Standard calls for this would be handy for embedded applications where
one does not want any fluff that is above bare minimum, some also prefer
a lean Forth for desktop applications simply because that is what they like.

The other option is Rolls Royce Forth for use with desktops where lots
of resources are available and one could use more tools to make the
writing of software easier.

There is something to be said by both cases and I would think with
careful thought both classes of programming style can be accommodated.
The present Standard allows for a Core and for additions to the core,
that feature can be used to help both camps to be better pleased with Forth.

To make Forth leaner the core functions can be reduced to bare minimum,
with additional optional word sets to build Forth to the level one
desires, it could be lean, or it could be fat and juicy, or it could be
something in-between, it would be up to the individual.

There are some items that would pose some issues such as type checking
of variables but with careful though they might be possible to implement
them while still retaining the ability to not have such a feature after
all Forth allows words to be re-defined at ones pleasure.
--
Cecil - k5nwa

[minf...@arcor.de]

Not to forget those who think Forth is a compiler, while otheres like it to be a "system", and then the fraction of DSL enthusiasts.

But the most important issue is: why standardize a half-dead horse?

[Albert van der Horst]

In article <2016Sep1...@mips.complang.tuwien.ac.at>,

Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
>At EuroForth Andrew Read gave an impromptu talk where he suggested a
>next-standard effort that would not be shackled by
>backwards-compatibility with the current (Forth-83/94/2012) line of
>standards.

I'm sympathetic to that idea. With 64 bit systems in mind we could
get rid of some cruft.
This is a setup for the 202x standard

- unsigned numbers can go, or at least must be banned to an
optional wordset
- double numbers can go, or at least must be banned equally.
As part of this, only one non-obsolescent notation for doubles
remains: a point before *and* after a number, like so
.340282366920938463463374607431768211455.
This notation must not be mixed with any other prefix, or recognizer
shit, just the current BASE, that's it.
- related to this, number formatting must be cleaned up to
use only signed single precision numbers. All # stuff banned to the
double wordset. %-words can be used instead. We must have something
like that for floats.
- only one floating point wordset with an "implementation defined" precision,
which soon will be 256 bits.
- >IN must be replaced by a handle that abstracts the position
in the input stream. UNFILL replaces the idiom `` 0 >IN ! ''
- The normal casing of Forth words becomes lower case.
ISO Forth words must be recognized if in all upper case.
- Embracing bytes (address units) as the basic storage units.
All c-cruft must go, that refers to characters where we want bytes.
The following words become byte-based
B@ (or we could keep C@)
B! (or we could keep C!)
ERASE FILL MOVE (maybe BLANKs)
new : CORA (compare two byte areas)
EXCHANGE (exchange areas over a byte length)
Dictionary search becomes byte based, hence language
dependancy is restricted to screen views
So if I talk to a chinese forth I can build a string using
c"\67~_\A0\90" and it will find it. (that is the chinese
character for DROP, by the way). 1]
(c" is cooked string, " is literal).

I hope this doesn't look like I'm promoting yourforth. It is just
that I've been experimenting there.

>- anton

1] Just kidding.
--
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

[foxaudio...@gmail.com]

An idea I have been toying with is that a minimalist Forth system should have a standard way to be extended. We have never specified in detail the order of "includes" that one should expect to work correctly. (as far as I know)

So starting from some "standard" minimal system, what file can I next load that will give me the next "level" of Forth and so on.

I guess I am asking for some system level standards as well as language internals standards.

In other words, can we specify how to move to a higher levels of functionality in a standard way so that after including a set number of "standard" files, "myForth" has the same words as "yourforth" so to speak. Dependancies could still be stated (word size, memory allocations etc) but at least a programmer would know the words that she has at her disposal and that stack arguments are consistent for those words.


BF

[Andrew Haley]

minf...@arcor.de wrote:
>
> Not to forget those who think Forth is a compiler, while otheres
> like it to be a "system", and then the fraction of DSL enthusiasts.

Is anyone who knows Forth crazy enough to describe it as merely "a
compiler?" Of course, it is often a whole system and sometimes runs
atop some OS. And right from the get go, Forth has always been about
creating a domain-specific language: Chuck Moore originally called
Forth programming as creating a "Problem-Oriented-Language".

> But the most important issue is: why standardize a half-dead horse?

Eh? Is this flamebait, or is there a real question?

Andrew.

[JUERGEN]

Cecil, on a completely other front ( like regarding Novix ): as you know I am CDP1802.
There will be a free implementation for FPGA on github this year in VHDL.
As such, my BMP802 will be more interesting for people.
Any chance you run the BMP802 through your system so I can update the book with ASCII as you suggested for Ting's Footsteps ? At 12 MB it is a bit large and could look a lot better in ASCII. See http://www.exemark.com/Microcontrollers/BMP802.pdf

[Anton Ertl]

alb...@cherry.spenarnc.xs4all.nl (Albert van der Horst) writes:
>- Embracing bytes (address units) as the basic storage units.
> All c-cruft must go, that refers to characters where we want bytes.
> The following words become byte-based
> B@ (or we could keep C@)
> B! (or we could keep C!)
> ERASE FILL MOVE (maybe BLANKs)
> new : CORA (compare two byte areas)
> EXCHANGE (exchange areas over a byte length)
> Dictionary search becomes byte based, hence language
> dependancy is restricted to screen views
> So if I talk to a chinese forth I can build a string using
> c"\67~_\A0\90" and it will find it. (that is the chinese
> character for DROP, by the way). 1]
> (c" is cooked string, " is literal).

That's pretty much already there. 1 chars = 1, for comparing two
blocks of address units/chars, you can use COMPARE; dictionary search
is now address-unit based, and you can build a string using
s\" \x67~_\xA0\x90"; or you can just write the Chinese character you
are interested in directly in S"; that will be no less readable to
those who don't read Chinese, and more readable to those who can.

[Cecil Bayona]

Thanks for the manual.

I am very interested in the Novix 4016 and the RTX2000 to that end I was
willing to create a computer readable text version of the Footsteps
book, I have a book copy now, and hopefully soon I will have a up to
date electronic version to hand back to Dr Ting. It turns out to be a
horrendous amount of work so I took a lot more than what I can chew, but
we will try it anyway.

I do have some interest in the CDP1802 CPU also but not as high as the
Novix 4016 and the RTX 2000, so I will be interested in a VHDL version
when available but at this point I'm not ready to commit to re-type
another book as it's a lot harder than anticipated. Maybe later I might
get interested but right now I have too many projects going on.
--
Cecil - k5nwa

[minf...@arcor.de]

Am Montag, 19. September 2016 16:49:19 UTC+2 schrieb Andrew Haley:
> minf...@arcor.de wrote

> > But the most important issue is: why standardize a half-dead horse?
>
> Eh? Is this flamebait, or is there a real question?
>
> Andrew.

No and yes. ;-) Forth

a) lacks mass and momentum i.e. the user base is so small that standardization seems a microscopic endeavour

b) by its nature Forth embraces flexibility - however standardization can be countereffective to flexibility

Please understand me correctly: I do not want to belittle the work and love others are putting into new standards. But I'd rather prefer to see more new features instead of decades old repetitions.

[Andrew Haley]

minf...@arcor.de wrote:
> Am Montag, 19. September 2016 16:49:19 UTC+2 schrieb Andrew Haley:
>> minf...@arcor.de wrote
>> > But the most important issue is: why standardize a half-dead horse?
>>
>> Eh? Is this flamebait, or is there a real question?
>

> No and yes. ;-) Forth
>
> a) lacks mass and momentum i.e. the user base is so small that
> standardization seems a microscopic endeavour

There are real users, and it is an interesting language. I think it's
worth the effort. Not every worthwhile language has ten million
developers.

> b) by its nature Forth embraces flexibility - however
> standardization can be countereffective to flexibility

I don't believe that at all. In fact, I'm sure that is nonsense.
There is no way that having a standard small language core can
possibly adversely affect flexibility. This is obviously true, given
that any system developer can take or leave any standard language
features.

> Please understand me correctly: I do not want to belittle the work
> and love others are putting into new standards. But I'd rather
> prefer to see more new features instead of decades old repetitions.

Standardization is not language design. They are separate (but
related) activities.

Andrew.

[lawren...@gmail.com]

On Wednesday, September 21, 2016 at 4:23:26 AM UTC+12, Andrew Haley wrote:
> Not every worthwhile language has ten million
> developers.

What is the ratio of people doing things with the language, to those coming up with yet another implementation of the language?

[hughag...@gmail.com]

On Tuesday, September 20, 2016 at 9:23:26 AM UTC-7, Andrew Haley wrote:
> minf...@arcor.de wrote:
> > Am Montag, 19. September 2016 16:49:19 UTC+2 schrieb Andrew Haley:
> >> minf...@arcor.de wrote
> >> > But the most important issue is: why standardize a half-dead horse?
> >>
> >> Eh? Is this flamebait, or is there a real question?
> >
> > No and yes. ;-) Forth
> >
> > a) lacks mass and momentum i.e. the user base is so small that
> > standardization seems a microscopic endeavour
>
> There are real users, and it is an interesting language. I think it's
> worth the effort. Not every worthwhile language has ten million
> developers.

Andrew Haley is an employee of Forth Inc. and so Elizabeth Rather appointed him to the Forth-200x committee --- he has no purpose except to vote as his boss at Forth Inc. tells him to vote --- he doesn't think for himself.

This 10,000,000 figure is indicative of Andrew Haley's delusions of grandeur regarding his own importance --- he fantasizes about having millions of followers depending upon the Forth-200x committee to provide them with a standard --- he doesn't want to face the truth that ANS-Forth (and now Forth-200x) are just a marketing gimmick from Forth Inc. and that nobody has ever written a commercial ANS-Forth program (for example, Stephen Pelc routinely abandons ANS-Forth and makes all of his software VFX-specific).

I would estimate that there are 200 to 400 Forth programmers in the world.

There are about 20 ANS-Forth programmers in the world, and all of them are on the Forth-200x mailing list braying about how they set the standard for Forth that 10 million Forth programmers depend upon --- none of them are actually writing Forth programs --- most of them know little or nothing about computer programming.

> > b) by its nature Forth embraces flexibility - however
> > standardization can be countereffective to flexibility
>
> I don't believe that at all. In fact, I'm sure that is nonsense.
> There is no way that having a standard small language core can
> possibly adversely affect flexibility. This is obviously true, given
> that any system developer can take or leave any standard language
> features.

The problem with ANS-Forth is not that it has too small of a core. I say that ANS-Forth is at least twice as big as necessary. ANS-Forth has a lot of redundancy, such as having both VALUE and VARIABLE that do the same thing, and this causes confusion among novices because they don't know which they should use. In Straight Forth I get rid of TO and VALUE (and also get rid of TO for local variables) and use @ and ! throughout for consistency. I also get rid of DO loops and use BEGIN WHILE REPEAT throughout for consistency. I also get rid of UNTIL which is yet more cruft. A lot of this cruft was introduced in the 1970s when optimizing compilers weren't available. Now a compiler should be smart enought to notice that WHILE and REPEAT don't have any code between them, and compile the code efficiently --- in the 1970s compilers weren't this smart and it was necessary to provide UNTIL (in those days, every source-code word compiled into threaded-code word one-to-one). Similarly, I get rid of AGAIN --- I allow any number of WHILE to be between the BEGIN and REPEAT, so if you have no WHILE in there then you effectively have an infinite loop (typically exited with EXIT).

The problem with ANS-Forth is that it was written by a sales-person who has never written a Forth program in her life and whose only programming experience was in COBOL --- if you want to have "mass and momentum" you have to have a standard written by actual Forth programmers --- this is my goal with Straight Forth, and is why I solicit input on the design from Forth programmers (so far the only input I've gotten is for the ANS-Forth cult members to say that my Forth code is "crap" and "sucks").

> > Please understand me correctly: I do not want to belittle the work
> > and love others are putting into new standards. But I'd rather
> > prefer to see more new features instead of decades old repetitions.
>
> Standardization is not language design. They are separate (but
> related) activities.

That isn't true. ANS-Forth was full of new "features" (arguably "bugs" would be a better term) that had never been seen before in any previous Forth.

For example, ANS-Forth has a very bizarre construct in which THEN can be used after a BEGIN WHILE REPEAT loop and will match up with the WHILE inside. Perhaps later a doctor with a flashlight can show us where this idea came from.

ANS-Forth has LOCAL| for defining local variables. The names of the locals are listed backwards though! Obviously, nobody ever tested this before they put it into the ANS-Forth standard document and shipped the document off to ANSI to be rubber-stamped --- even one day of testing would have been adequate to reveal that the names were backwards --- the reason why nobody tested ANS-Forth in 1994 is that no ANS-Forth compiler existed until 1997 (that was SwiftForth which was so bug-ridden as to be unuable; it wasn't until many years later that a usable ANS-Forth compiler was written).

[Albert van der Horst]

In article <b0d05878-6616-4349...@googlegroups.com>,

Let's hope that those coming up with implementations, also do things
with the language ...
Anyway, at a certain point the board of the Dutch Forth consists of
5 persons. I was the last among them to build my own Forth.
The percentage of Forthers that build their own compiler is probably
larger than for most languages, except maybe Lisp.

Groetjes Albert

[humptydumpty]

Hi!

Maybe lack of (domestic use of) continuations?
Is something very basic, that could be useful,
as stack of continuations provides subroutines,
and queue of continuations provides coroutines.

Have a nice day,
humptydumpty

[Stephen Pelc]

On Sun, 18 Sep 2016 09:01:46 -0700 (PDT), anothe...@gmail.com
wrote:

>I (relative noob here), in my silliness, think that if I were to think up a=

> separate standard from ANS Forth, I would emphasize optional wordsets.

All the wordsets in the Forth-2012 are optional. If you want to
provide them, you can even supply them as source code. This has been
true for at least 20 years now.

>I guess what I'm trying to say is that it would be great to have standardiz=
>ed libraries. And by "libraries" I don't mean common abstractions that can =
>be built using the core wordset like it seems a lot of other languages have=
>, I mean a minimalist set of capabilities that couldn't be used portably ot=
>herwise - not even with a C interface, because that would only work on syst=
>ems with that shared library.

Source libraries and experimental proposals were discussed at the
standards meeting in Reichenau on September 9-11, 2016. They will
appear. See
http://theForth.net
http://forth-standard.org
http://www.forth200x.org/

Stephen

--
Stephen Pelc, steph...@mpeforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

[Anton Ertl]

humptydumpty <oua...@gmail.com> writes:
>Hi!
>
>Maybe lack of (domestic use of) continuations?
>Is something very basic, that could be useful,
>as stack of continuations provides subroutines,
>and queue of continuations provides coroutines.

Do you mean return-address manipulations (I have noticed that you use
"continuation" to mean one return address, whereas in general usage
the term means much more)?

The main problem with return-address manipulations is that they don't
combine well with locals. So if you want to standardize them, you
either have to de-standardize locals, or you create a bunch of
ambiguous conditions when return-address manipulations are combined
with locals. Both are not very likely to find consensus.

Is a "next standard" without locals and with return-address
manipulation likely to find widespread acceptance? I doubt it, but I
may be wrong.

What else could be done? Instead of return-address manipulations,
something closer to the general concept of continuations could be
created; the general concept includes the whole program state, and is
hard to implement in Forth. But something more in the Forth tradition
is tasks (which include only the stacks and the user variable).

By transferring control (and data) between two tasks, you get
coroutines, and more general ones than available by just changing one
return address (e.g., the two coroutines can each traverse a tree
recursively, with different tree structures).

There is a subcommittee working on multi-tasking, but from what I
overheard from them, the present idea is to have traditional
statically allocated and named Forth tasks, not dynamically created
tasks. But that's still subject to change.

[hughag...@gmail.com]

On Wednesday, September 21, 2016 at 7:39:26 AM UTC-7, Anton Ertl wrote:
> humptydumpty <oua...@gmail.com> writes:
> >Hi!
> >
> >Maybe lack of (domestic use of) continuations?
> >Is something very basic, that could be useful,
> >as stack of continuations provides subroutines,
> >and queue of continuations provides coroutines.
>
> Do you mean return-address manipulations (I have noticed that you use
> "continuation" to mean one return address, whereas in general usage
> the term means much more)?
>
> The main problem with return-address manipulations is that they don't
> combine well with locals. So if you want to standardize them, you
> either have to de-standardize locals, or you create a bunch of
> ambiguous conditions when return-address manipulations are combined
> with locals. Both are not very likely to find consensus.
>
> Is a "next standard" without locals and with return-address
> manipulation likely to find widespread acceptance? I doubt it, but I
> may be wrong.

My rquotations (a slightly upgraded version of HumptyDumpty's rquotations) work fine with locals. The rule is that the hof has to have local variables (HumptyDumpty's rule was that the hof was not allowed to have local variables) --- my rule is better because you usually want the hof to have local variables (because you need to hold the xt in a local out of the way while doing the iteration which is typically a BEGIN WHILE REPEAT loop).

Most of the complaints about rquotations come from people who don't understand how they work --- just a lot of hand-wringing and: "OMG! This can't possibly work! This should be banned because I don't understand it!" --- I didn't understand how HumptyDumpty's rquotations worked either at first, and it took me a couple of hours of thinking to figure them out, and now I consider them to be pretty awesome.

Thinking is the key to understanding stuff!

> What else could be done? Instead of return-address manipulations,
> something closer to the general concept of continuations could be
> created; the general concept includes the whole program state, and is
> hard to implement in Forth. But something more in the Forth tradition
> is tasks (which include only the stacks and the user variable).
>
> By transferring control (and data) between two tasks, you get
> coroutines, and more general ones than available by just changing one
> return address (e.g., the two coroutines can each traverse a tree
> recursively, with different tree structures).
>
> There is a subcommittee working on multi-tasking, but from what I
> overheard from them, the present idea is to have traditional
> statically allocated and named Forth tasks, not dynamically created
> tasks. But that's still subject to change.

P.S. for HumptyDumpty --- do you know Icon?

I know Icon somewhat, but I haven't played with that in 20 years --- I intend to get back into that again, for the purpose of possibly implementing something similar in Straight Forth.

Since you seem to be big on continuations, I would definitely be interested in your input on the subject. The reason I talk about Straight Forth but don't release it, is because I want input on the design --- I don't want to be like Elizabeth Rather and say: "Here is a language design that has never been tested, but it is now written in stone and can never be changed, and everybody must follow it or be non-standard." My name is not Moses --- I'm not bring Straight Forth down from the mountain --- I want Straight Forth to be a grassroots effort.

I have a document started that describes Straight Forth. It is not complete, and is very much in flux. If anybody wants to read what I have so far, I can email it.

[humptydumpty]

On Wednesday, September 21, 2016 at 5:39:26 PM UTC+3, Anton Ertl wrote:

> humptydumpty writes:
> >Hi!
> >
> >Maybe lack of (domestic use of) continuations?
> >Is something very basic, that could be useful,
> >as stack of continuations provides subroutines,
> >and queue of continuations provides coroutines.
>
> Do you mean return-address manipulations (I have noticed that you use
> "continuation" to mean one return address, whereas in general usage
> the term means much more)?
>

Hi!

I think that Andrew Read title "An axiomatic approach to Forth" give
an insight that concepts that compose "forth axioms" should not be
formulated in Forth terms. To be safe, I'll put here on not
computer related terms, but somehow in a intuitive way.

Let here be a imaginary dictionary book and suppose I search definition
of "Word", whose entry looks like:

Word : word0 word1 .. wordN ;

Now suppose that I don't know what "word0" means.
I'll put bookmark (near as possible to hinge of dictionary,
with condition to not shadow earlier placed bookmarks)
on *continuation* of "word0" inside definition of "Word"
( sequence "word1 .. wordN" till ";" ) and go search for "word0".

When done understanding "word0" ( hit "word0" end-of-definition ";" )
I'll return to continuation on last-bookmark (far away from hinge),
clean-up and put away that bookmark.

At a given time, this ordered set of bookmarks (by distance from
book hinge) is a stack of continuations.

Digression:
Dictionary book compress knowledge in limited number of files,
as factored subroutines compress "actions" in limited memory.

Now, back to Forth, an return address is an unique identifier for
a continuation.

> The main problem with return-address manipulations is that they don't
> combine well with locals. So if you want to standardize them, you
> either have to de-standardize locals, or you create a bunch of
> ambiguous conditions when return-address manipulations are combined
> with locals. Both are not very likely to find consensus.
>

Now, only I could think of a stack of locals that somehow synchronizes
with return stack, for automatic de-allocation (question:
is automatic de-allocation worth to complexify things? ).

> Is a "next standard" without locals and with return-address
> manipulation likely to find widespread acceptance? I doubt it, but I
> may be wrong.
>
> What else could be done? Instead of return-address manipulations,
> something closer to the general concept of continuations could be
> created; the general concept includes the whole program state, and is
> hard to implement in Forth. But something more in the Forth tradition
> is tasks (which include only the stacks and the user variable).
>


> By transferring control (and data) between two tasks, you get
> coroutines, and more general ones than available by just changing one
> return address (e.g., the two coroutines can each traverse a tree
> recursively, with different tree structures).
>

I looked over "tasker.fs"; for me is a bit overhead to think coroutines
in terms of tasks. Intuitively, for me is straightforward to say
"put this continuation to wait its turn and continue with whatever
is first on queue".
Provided '>co' which put into a circular queue and 'co>' which retrive it,
coroutines could be provided with:
: co-start co> >R ;
: 0CO ( -- ) CQ Head ! CQ Tail ! ; \ init circular queue
: CO: ( -- ) R> >co ; \ register into coroutines queue
: CO ( -- ) R> >co co> >R ; \ coroute
: GO ( -- ) BEGIN Head @ Tail @ <> WHILE co-start REPEAT ; \ squeeze out

I'm interested of such example where two coroutines can each traverse a tree
recursively, with different tree structures.

> There is a subcommittee working on multi-tasking, but from what I
> overheard from them, the present idea is to have traditional
> statically allocated and named Forth tasks, not dynamically created
> tasks. But that's still subject to change.
>
> - anton
> --
> M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
> comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
> New standard: http://www.forth200x.org/forth200x.html
> EuroForth 2016: http://www.euroforth.org/ef16/

[humptydumpty]

On Thursday, September 22, 2016 at 9:41:05 PM UTC+3, hughag...@gmail.com wrote:
> On Wednesday, September 21, 2016 at 7:39:26 AM UTC-7, Anton Ertl wrote:

Hi!

I don't know Icon. I was interested on Icon, had some lectures from
an Icon book. Very interesting, but no more than Forth :)

Somehow I'm interested to know of Forth, from much basic concepts.

[Paul Rubin]

an...@mips.complang.tuwien.ac.at (Anton Ertl) writes:
> At EuroForth Andrew Read gave an impromptu talk where he suggested a
> next-standard effort that would not be shackled by

Are there any slides or video about this?

> the talk got me thinking about what is essential in the current line
> of Forth standards

I'd be interested in your thoughts about what is essential in the
current line of Forth standards, and which parts if any of that are NOT
essential to a (subjective) higher-level concept of what Forth is.

> Possibly essential: The syntax difference between machine Forth and
> Color Forth (color that indicates how to text-interpret the string)
> cannot be accommodated in the current line of standards; while it's

> just syntax ...

Is it really just syntax? I don't really understand how Colorforth
works but I gather that it separates "edit time" from "compile time"
much more than traditional or ANS Forth does. Also its treatment of
macros (black words) seems different, though I don't know exactly how.
I've been wondering about this.

> Pretty essential: Statically determined stack depth: Requiring that,
> as sketched for PAF,

At that point maybe you want to depart from Forth tradition and have
named stack slots (ouch!). I do think dynamic stack effects (while
misused sometimes) have valid uses, like variable sized arg lists, e.g.

: foo ( a1 a2 .. an n -- ) ... \ n is # of args

Most languages have some mechanism for that, such as varargs in C etc.

Also, do you want for conforming implementations to actually enforce
this rule? Overall it sounds like a difficult sell even though PAF and
other compilers would benefit from it.

> Static type checking: StrongForth did that

This also seems like a hard sell, since StrongForth didn't get much
traction. It also sounds harder to implement than traditional Forth.
Do you really think of it as essential?

> Floats: There is one suggestion (and a Russion system that works this
> way) to use floats as stack items, with words like + working on
> floats, and IIRC addresses represented by NaNs (or maybe by
> interpreting the bit pattern as address).

No please don't do that ;-). It's one of the fail points of Javascript
and older versions of Lua and maybe others as well. And users of low
level languages like Forth will often expect to be able to twiddle the
bits in cells.

> Dynamically typed stack items (e.g., Factor, 8th, Oforth,...)

I wonder how many users of those systems actually started out as Forth
users. I haven't tried them since I lump them with other dynamic
languages such as Python, so their attractiveness mostly comes from
features like GUI toolkit integration, which I haven't needed so far.
If Forth users are actually willing to try this approach, it might be a
good way to move the community forward.

Another idea is to have a standard Forth with pseudo-static typing:
i.e., have a testing/debugging mode where every cell is accompanied by a
tag field that's checked at runtime, signalling errors that might
otherwise be missed. Then once debugged, you can turn off the checks.
No idea how useful this is.

Meanwhile some other imho essentials missing from your list: 1) design
changes to accomodate processors with flash program storage, 2) changes
to accomodate cross-compilation, 3, 4...) (forgot).

[Anton Ertl]

Paul Rubin <no.e...@nospam.invalid> writes:
>an...@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>> At EuroForth Andrew Read gave an impromptu talk where he suggested a
>> next-standard effort that would not be shackled by
>
>Are there any slides or video about this?

IIRC he used the flip chart, but in any case I have no electronic
slides. There may be a video, the videos have not been published yet.

>> the talk got me thinking about what is essential in the current line
>> of Forth standards
>
>I'd be interested in your thoughts about what is essential in the
>current line of Forth standards, and which parts if any of that are NOT
>essential to a (subjective) higher-level concept of what Forth is.

My original posting was an exploration of that. My conclusion is that
the lines are not as clearly drawn as I thought before:

For the standard, the issue is not so much what is essential, but
what finds consensus, and how long it takes and how much work it takes
to change it. In theory, we could move almost everywhere. In
practice, we hardly move at all.

But does that mean there should be a "next-standard" effort that goes
where it is too hard to reach with the current standards effort? I
have my doubts that there will be consensus on such a next standard.
How much commonality is there among systems that do not follow the
standard? And would the implementors of these systems really be
interested in finding consensus on a next standard?

I think that a Turbo-Pascal like effect is more likely: While the
Pascal standard was pretty limited, and it took until 1990 to publish
the "Extended Pascal" standard, Turbo Pascal added all the features
its designers considered useful, without letting themselves be
shackled by the Pascal standard, and became the dominant Pascal
implementation in terms of percentage of written code and installed
systems, making the Pascal standards irrelevant. So, if a Forth
system manages to grab a large percentage of the Forth market, that
would be the "next standard".

Which parts of the standard are not essential? Lots; machine Forth is
a Forth, and it has little in common with the standard. I did not
consider systems with any form of type checking to be "Forth" before
starting this thread, but now this does not seem so clear-cut to me.

>> Possibly essential: The syntax difference between machine Forth and
>> Color Forth (color that indicates how to text-interpret the string)
>> cannot be accommodated in the current line of standards; while it's
>> just syntax ...
>
>Is it really just syntax? I don't really understand how Colorforth
>works but I gather that it separates "edit time" from "compile time"
>much more than traditional or ANS Forth does.

Yes, I dimly remember something along these lines; I am not sure that
that really makes a fundamental difference.

By contrast, I found Holon/HolonForth's approach does take away some
features that I consider fundamental in Forth, while also providing
cool features (Wolf Wejgaard gave a nice presentation at EuroForth).
In Holon the dictionary is constructed at edit time, and the source
code and executable code hangs from the dictionary, which is
browsable.

An important (IMO) Forth feature Holon takes away is the ability to
create words at run-time.

>> Pretty essential: Statically determined stack depth: Requiring that,
>> as sketched for PAF,
>
>At that point maybe you want to depart from Forth tradition and have
>named stack slots (ouch!).

Actually Glass [glass83] proposed this 30 years ago, and it has a
certain cuteness, but I don't see any benefits over removing them from
the data stack. For PAF, for which both stack items and locals are
just pseudo-registers, there is little difference, and for the
programmer, it means that he has a hard time accessing stack items
lower on the stack than the locals.

@InCollection{glass83,
author = "Harvey Glass",
title = "Towards a More Writable {Forth} Syntax",
crossref = "ouverson86",
chapter = "21",
pages = "169--181",
note = "Reprinted from {\em Proceedings of the 1983 Rochester
Forth Applications Conference}"
}
@Book{ouverson86,
booktitle = {Dr.\ Dobb's Toolbook of Forth},
publisher = {M\&T Books},
year = {1986},
editor = {Marlin Ouverson},
address = {Redwood City, CA 94063}

}

>I do think dynamic stack effects (while
>misused sometimes) have valid uses, like variable sized arg lists, e.g.
>
> : foo ( a1 a2 .. an n -- ) ... \ n is # of args
>
>Most languages have some mechanism for that, such as varargs in C etc.

And they cause complications and external costs in the calling
convention.

There are uses for that kind of stuff, but they are rare and can be
avoided without too much effort. If avoiding it did cost too much, I
would add the common patterns to PAF; basically a sequence like
"a1...an n" would be treated as one stack item, allowing the rest of
the stack to be handled statically.

>Also, do you want for conforming implementations to actually enforce
>this rule? Overall it sounds like a difficult sell even though PAF and
>other compilers would benefit from it.

A PAF-only compiler should produce an error message on such code,
because it cannot produce the machine code that does what the
programmer expects. If a compiler is extended from PAF to Forth, it
would generate code that does what the programmer expects, and such a
compiler would either produce a warning (in PAF mode) or be quiet (in
Forth mode) when encountering such code.

It may be a hard sell to Forth programmers (not very hard, though,
IMO), but the idea of PAF is not as a replacement for Forth, but as an
intermediate language for compilers; and the compiler writers will not
find it at all hard to comply with these restrictions; the hard sell
to them will be to sell them a Forth-like language.

>> Static type checking: StrongForth did that
>
>This also seems like a hard sell, since StrongForth didn't get much
>traction. It also sounds harder to implement than traditional Forth.
>Do you really think of it as essential?

Obviously not, or I would have worked on it more, and maybe used
StrongForth.

The question was if the "weak"ness of standard Forth is essential.
Before to original posting, I thought so, but, depending on the type
system, this is just a step further in the Forth->PAF progression.
The problem is actually defining a type system that catches most/all
of the things that Forth programmers consider a type error, but does
not or only rarely annoys programmers with false positives. It may be
an issue of the programmer and the type system getting used to each
other (the Stockholm syndrome seems to be very common in these kinds
of situations).

>> Floats: There is one suggestion (and a Russion system that works this
>> way) to use floats as stack items, with words like + working on
>> floats, and IIRC addresses represented by NaNs (or maybe by
>> interpreting the bit pattern as address).
>
>No please don't do that ;-). It's one of the fail points of Javascript
>and older versions of Lua and maybe others as well.

Lua went away from that? Interesting. What's the story behind that?

>> Dynamically typed stack items (e.g., Factor, 8th, Oforth,...)
>
>I wonder how many users of those systems actually started out as Forth
>users.

Does it matter? If these systems attract new users to the family of
Forth-like languages, is that bad?

>Another idea is to have a standard Forth with pseudo-static typing:
>i.e., have a testing/debugging mode where every cell is accompanied by a
>tag field that's checked at runtime, signalling errors that might
>otherwise be missed. Then once debugged, you can turn off the checks.
>No idea how useful this is.

Again, you need a type system, but this one might be easier than a
statically checkable one.

>Meanwhile some other imho essentials missing from your list: 1) design
>changes to accomodate processors with flash program storage,

Does not appear too hard to me and reachable within the standard
framework, at least in theory. In practice people who used to
complain about the problems CREATE DOES> causes in this regard have
apparently given up or are content to have found workarounds for the
problems.

> 2) changes
>to accomodate cross-compilation

I think that cross-compilation and the hard compilation/run-time
boundary it requires destroys an essential property of Forth, but
that's a minority position. Anyway, many systems have some form of
cross compilation, but there does not seem to be consensus on a common
approach. And apparently there is still enough movement in this area
that the existing cross-compilation proposal is now deemed outdated by
its proponents, and dropped from the standardization effort.

[Matthias Trute]

>
> For the standard, the issue is not so much what is essential, but
> what finds consensus, and how long it takes and how much work it takes
> to change it. In theory, we could move almost everywhere. In
> practice, we hardly move at all.

I think that's the consequence of the design principle of the Forth
standard process: codify existing practice and consolidate it. Other
standard bodies follow a different approach: define something and declare
it standard, or standard extension. E.g. the WLAN or Ethernet standards
follow this model. First you get the specs, the implementation follows.

Forth could use both. There is a core feature set like the stacks, some words (in CORE) and the interpreter/compile model, and there are a number of extension wordsets. These extension wordsets could be managed more freely than the core.

No, I'm not complaining about the quotations or recognizers beeing discussed so badly. Or the object models. But they are a good example that the community will hardly reach "consensus before standardization" and progress is no longer possible. In such situations many forks happen. Strongforth, oforth or factor or or. All of them are nice and find their followers. They follow the TurboPascal model, hopefully with better luck. But converging them back to a common Forth base seems unlikely.

[tabcom...@gmail.com]

On Thursday, September 15, 2016 at 8:58:15 PM UTC-4, hughag...@gmail.com wrote:

> > 3.) Quotations. My system with the fat-xt will be used to allow quotations to access the parent function's local variables. This is trivial to implement! This is the foundation to writing general-purpose data-structures --- I really see this as bring Forth into the 21st century

Could you explain this more? Maybe an example?

[hughag...@gmail.com]

The quotation gets compiled elsewhere (not in the middle of the parent function). What gets compiled into the parent function is QXT that is like LIT except that it pushes the quotation's xt onto the data-stack. It knows the cfa at compile-time and it packs this into an xt. An xt is a 64-bit struct (Straight Forth has a 64-bit cell, but has 32-bit pointers). The xt contains the cfa (32-bit) and the pointer to the parent function's local-frame (32-bit) packed together. QXT has to obtain the pointer to the parent function's local-frame at run-time and pack it together with the quotations cfa which is a literal known at compile-time. An xt of a colon word is similar to an xt of a quotation except that the pointer to the local-frame is zero. EXECUTE can test the pointer to the local-frame to determine if the xt is of a quotation or a colon-word, and it behaves slightly differently. This is called the "fat-xt" system because the xt is a struct containing two pointers --- it is not just the cfa like in most Forth systems, but it also contains the parent-function's local-frame pointer (for a quotation).

What I have described above is pretty trivial to implement. I have a slightly more complicated version that allows EXECUTE to determine if a quotation is valid before executing it --- it can test that the parent function's local-frame is still valid, meaning that the quotation is okay to execute --- if the parent function's local-frame is no longer valid, that means that the parent function has executed, which means that EXECUTE will abort with an error message because this is illegal.

As for an example of using quotations, my hottie-counting program is pretty good:
https://groups.google.com/forum/#!topic/comp.lang.forth/ohE8mx7tWQU%5B101-125%5D
(see page-5)
This example uses rquotations rather than fat-xt quotations, but it does show quotation use. The fat-xt method is more robust --- we don't have the limitations of rquotations --- but those limitations aren't a problem most of the time anyway, so I don't actually have any examples of when the fat-xt method is needed and the rquotation method won't work.

I have a document started describing Straight Forth. I have two data-structures built-in (the array and the "chain"). The array is mostly for strings. I recommend that the chain be used for pretty much everything. I was impressed by how Factor has the "sequence" for everything, and this makes programs readable because they all look alike, and my chain (I just made up that term; it is something I invented) is based on this idea. Another advantage is that the compiler-writer can put some effort into making the hofs supporting the chain to be efficient (mine are pretty much all assembly-language, and they use registers internally). If the programmers use chains for all of their data, their programs should be pretty efficient, even if the compiler is doing minimal optimization for general code. Uniformity not only helps to make programs readable (uniform) but it also helps to make programs efficient --- the downside is that the chain may not always be the best choice for a particular application --- I think that it is a reasonable choice most of the time though (I would expect that 99% of all Straight Forth programs can use the chain as their only data-structure, and it will work fine).

The chain is an ordered container using a string as a key. Previously, Bernd Paysan called me stupid because I use a binary tree rather than a hash-table, because a hash-table provides faster random access (given enough memory that it doesn't overflow into a mess of linked-lists). My chains can be used as associations (which are all about random access), but this isn't my primary purpose for them, so I'm not all that concerned about efficiency of random-access. I am primarily assuming that the key will be a string representation of a number --- examples would be line-numbers in a cross-compiler, or a sparse array in a numeric program. The two "killer apps" for Straight Forth are the Forth cross-compiler and numeric programs --- the numeric programs would include CAM (Computer-Aided Manufacturing) that involves generating gcode for CNC machines, and statistics programs such as R is used for (R is slow as molasses though).

This is an excerpt from my document:
-----------------------------------------------------------------------------
A novice-level Straight Forth programmer should only use the built-in hofs; he should not write his own hofs and he should not use BEGIN WHILE REPEAT (iteration is a common source of bugs, which is why novices are discouraged from using BEGIN WHILE REPEAT). An intermediate-level Straight Forth programmer may write his own hofs and use BEGIN WHILE REPEAT inside of them, but should not use BEGIN WHILE REPEAT anywhere except inside of his hofs, and he should not write very many hofs. It is idiomatic for all Straight Forth programmers from novice to advanced to program in a novice-level manner (using the built-in hofs) as much as possible because this makes the programs easy to read by others who may be novice-level. The built-in data-structures are the array and the chain (an ordered container with a string as a key). These should be adequate for 99% of all Straight Forth programs. In rare circumstances it will be necessary to use some other data-structure, in which case hofs will have to be written to support that data-structure (a hof is used to traverse a data-structure). Straight Forth programmers should not introduce new data-structures unless necessary, because doing so makes their programs hard to read, but they should stick with novice-level techniques (using the two built-in data-structures) as much as possible --- the programmers should not strive to prove that they are intermediate-level or (woo hoo!) advanced-level --- the goal with Straight Forth is to write programs that do something useful and which are easy to read by everybody including novices.

This document describes how novice-level Straight Forth programmers write their programs (they use the built-in hofs and the two built-in data-structures). This document also describes how intermediate-level Straight Forth programmers write their programs (they implement new data-structures and write hofs containing BEGIN WHILE REPEAT to support those data-structures, but only when necessary). This document does not describe how advanced-level Straight Forth programmers write their programs --- this is because there are no advanced-level Straight Forth programmers (not even me the inventor) so we can't speculate on how they will write their programs --- after the language has been in use for a few years, advanced-level Straight Forth programmers might evolve (maybe not though; Python has been in use for decades, but no sober Python programmer claims to be advanced-level, and it is not expected to ever happen).

Straight Forth is completely different in spirit from ANS-Forth. Straight Forth is intended to be useful, and programs are intended to be simple and straight-forward (hence the name: Straight Forth). By comparison, ANS-Forth has been described as "sudoku for programmers," meaning that it is a toy language, but it can be fun like solving a puzzle (actually, like creating a puzzle). ANS-Forth programmers delight in writing complicated and hard-to-read programs to prove that they are more clever than the poor devil who is unable to figure out what they are doing. The ANS-Forth programmers motto seems to be: "If you can't dazzle them with brilliance, then baffle them with bull!" Forth-200x is the same pile of manure grown to a gigantic size and getting bigger every year. The Forth-200x programmers delight in writing "recognizers" that change the syntax of Forth-200x and hence totally baffle the reader. By comparison, Straight Forth is a traditional Forth that doesn't have any syntax (well, almost no syntax; we do have a few minor syntactical features that help to improve readability) --- Straight Forth programmers are definitely not allowed to create their own funky syntax, effectively creating their own custom language --- Straight Forth programs are supposed to be easily readable by all Straight Forth programmers (the maintainence programmer's life is already pretty miserable, but there is no good purpose in adding to his misery gratuitously).

The point of Straight Forth is that you don't have to be advanced-level, or even intermediate-level, for the language to be useful to you --- it is designed to be useful for novice-level programmers --- it is acceptable for a Straight Forth programmer to stay at the novice-level indefinitely (the point is to write useful programs, not to show off how clever you are; stick with sudoku puzzles for showing off how clever you are).
-----------------------------------------------------------------------------

[Andrew Haley]

Matthias Trute <matthia...@gmail.com> wrote:
>>
>> For the standard, the issue is not so much what is essential, but
>> what finds consensus, and how long it takes and how much work it takes
>> to change it. In theory, we could move almost everywhere. In
>> practice, we hardly move at all.
>
> I think that's the consequence of the design principle of the Forth
> standard process: codify existing practice and consolidate it. Other
> standard bodies follow a different approach: define something and declare
> it standard, or standard extension. E.g. the WLAN or Ethernet standards
> follow this model. First you get the specs, the implementation follows.
>
> Forth could use both. There is a core feature set like the stacks,
> some words (in CORE) and the interpreter/compile model, and there
> are a number of extension wordsets. These extension wordsets could
> be managed more freely than the core.
>
> No, I'm not complaining about the quotations or recognizers beeing
> discussed so badly. Or the object models. But they are a good
> example that the community will hardly reach "consensus before
> standardization" and progress is no longer possible.

Huh? I don't understand what you believe is bad about the dicussions
around quotations and recognizers.

> In such situations many forks happen. Strongforth, oforth or factor
> or or. All of them are nice and find their followers. They follow
> the TurboPascal model, hopefully with better luck. But converging
> them back to a common Forth base seems unlikely.

The early history of Forth is instructive here. Forth, Inc. wrote
applications for customers. Sometimes, particular factors turned out
to be useful, and entered their toolbox. Some of these were generally
useful, and eventually entered the standard system and the language.
Some didn't make the grade and were kicked out. But everything was
tested in the fire of real-world usage. That's a distinctive feature
of Forth: it's about getting stuff done, not "cute" features.

Quotations are neat, but they don't add a great deal of power to the
language. On the other hand, their cost and footprint is small. I
think that they'll turn out to be useful enough to be standardized,
but once a word is in the standard it's hard to get rid of it.

Andrew.

[Matthias Trute]

> The early history of Forth is instructive here. Forth, Inc. wrote
> applications for customers.

That was like the Turbo Pascal era for Pascal. Borland did add
a great amount of library code (anyone remember the BGI or the
Turbo Vision libraries?) that matched the demands of many users.

Expectations moved since then. Today a language by itself is less important.
Erlang or ruby did not run into real world application because of their
theoretical features but because of their libraries. Has anyone written
a real forth application server? Ruby got it's success with
rails (IMHO only because of it). Erlang came from the telcos and now a
few real hot software packages use it (CouchDB, RabbitMQ). Python
replaced perl because perl de facto stopped development for 20 years.
(perl6 only recently came into public).

> tested in the fire of real-world usage. That's a distinctive feature
> of Forth: it's about getting stuff done, not "cute" features.

Forth became an unknown hidden language, that's a tragedy. Erlang is a
hidden language too, but a well known one, considered for new projects.

> Quotations are neat, but they don't add a great deal of power to the
> language. On the other hand, their cost and footprint is small. I
> think that they'll turn out to be useful enough to be standardized,
> but once a word is in the standard it's hard to get rid of it.

Maybe this is the gist of the matter. I think that there should be a
language core feature set, more or less what CORE is now. Discussions
here are worth any effort. The same effort for the optional word sets
is overkill. The optional wordsets are more or less a standard library
that shall be changeable more easily (that is extended, keeping
backward compatibility is not a bug).

So, quotations are fine, just have them as candidate wordset in the
standard (a new section in the text). published as standard 2017, to
be reviewed in standard 2019. Next one, please. (the years are examples).
I agree that recognizers are not that fine tuned yet. That's why I didn't
start a CfV.

Matthias

[tabcom...@gmail.com]

On Sunday, September 25, 2016 at 12:47:10 AM UTC-4, hughag...@gmail.com wrote:
> On Saturday, September 24, 2016 at 6:48:13 PM UTC-7, tabcom...@gmail.com wrote:
> > On Thursday, September 15, 2016 at 8:58:15 PM UTC-4, hughag...@gmail.com wrote:
> >
> > > > 3.) Quotations. My system with the fat-xt will be used to allow quotations to access the parent function's local variables. This is trivial to implement! This is the foundation to writing general-purpose data-structures --- I really see this as bring Forth into the 21st century
> >
> > Could you explain this more? Maybe an example?
>
> The quotation gets compiled elsewhere (not in the middle of the parent function). What gets compiled into the parent function is QXT that is like LIT except that it pushes the quotation's xt onto the data-stack. It knows the cfa at compile-time and it packs this into an xt. An xt is a 64-bit struct (Straight Forth has a 64-bit cell, but has 32-bit pointers). The xt contains the cfa (32-bit) and the pointer to the parent function's local-frame (32-bit) packed together. QXT has to obtain the pointer to the parent function's local-frame at run-time and pack it together with the quotations cfa which is a literal known at compile-time. An xt of a colon word is similar to an xt of a quotation except that the pointer to the local-frame is zero. EXECUTE can test the pointer to the local-frame to determine if the xt is of a quotation or a colon-word, and it behaves slightly differently. This is called the "fat-xt" system because the xt is a struct containing two pointers --- it is not just the cfa like in most Forth systems, but it also contains the parent-function's local-frame pointer (for a quotation).
>
> What I have described above is pretty trivial to implement. I have a slightly more complicated version that allows EXECUTE to determine if a quotation is valid before executing it --- it can test that the parent function's local-frame is still valid, meaning that the quotation is okay to execute --- if the parent function's local-frame is no longer valid, that means that the parent function has executed, which means that EXECUTE will abort with an error message because this is illegal.

Thanks. Very helpful. I am not adept in all the implementation details of Forth so this a touch over my head, but I understood enough. What notation are you using in Straight Forth -- `[: ;]` ?

> The chain is an ordered container using a string as a key. Previously, Bernd Paysan called me stupid because I use a binary tree rather than a hash-table,

I don't. BST is probably better for many reasons.

> The Forth-200x programmers delight in writing "recognizers" that change the syntax of Forth-200x and hence totally baffle the reader. By comparison, Straight Forth is a traditional Forth that doesn't have any syntax (well, almost no syntax; we do have a few minor syntactical features that help to improve readability) --- Straight Forth programmers are definitely not allowed to create their own funky syntax, effectively creating their own custom language --- Straight Forth programs are supposed to be easily readable by all Straight Forth programmers (the maintainence programmer's life is already pretty miserable, but there is no good purpose in adding to his misery gratuitously).

I basically agree with you about recognizers. They are new to me. But it didn't take me long to think maybe that is just going too far. Maybe it makes sense as an optional add on (however that might work) but for general use it would make programs harder to read. I think things need to be moving in the opposite direction.

OTOH, they are interesting b/c it gives the programmer the power to create DSLs of almost any form. Of course using an extensible lexer could probably achieve much the same thing (which I think Factor does?).

[Rod Pemberton]

On Sat, 24 Sep 2016 21:47:09 -0700 (PDT)
hughag...@gmail.com wrote:

> I have a document started describing Straight Forth. I have two
> data-structures built-in (the array and the "chain"). The array is
> mostly for strings. I recommend that the chain be used for pretty
> much everything.

Forth is criticized for all the stack manipulation. C should be
criticized for professors over-emphasizing linked-lists and binary
trees. What I've found over the years is that nearly every problem
that can be solved by using them, can be also be solved by using arrays
and LIFO stacks. This takes far less coding and has less code overhead.
This produces faster code too since stacks are push/pop and arrays are
simply indexed. There is no searching nor traversing of them.
Reworking the problem for this is much like finding the optimal stack
juggling solution in Forth.

> Previously, Bernd Paysan called me stupid because I use a binary tree
> rather than a hash-table, because a hash-table provides faster random
> access (given enough memory that it doesn't overflow into a mess of
> linked-lists).

Que?


Rod Pemberton

[lawren...@gmail.com]

On Sunday, September 25, 2016 at 10:21:38 PM UTC+13, Matthias Trute wrote:
> Today a language by itself is less important.

Good libraries need a good core language to build on. This is why Python has become popular. For example, Python offers an elegantly simple way to define custom overloads of operators: compare the cumbersome way C++ does it, and Java doesn’t allow it at all--and both are much more complex languages than Python.

For instance, I am able to define vector arithmetic in my own Python module, so that a geometrical calculation like <https://github.com/ldo/qahirah_examples/blob/master/fillet>

t1 = p1 + Vector.unit((p0 - p1).angle()) * inset
# fillet tangent point on line (p0, p1)

can determine both x- and y-coordinates at once, instead of having to write out the same formulas twice.

[Paul Rubin]

Matthias Trute <matthia...@gmail.com> writes:
> Forth became an unknown hidden language, that's a tragedy. Erlang is a
> hidden language too, but a well known one, considered for new projects.

Forth historically solved a problem (interactive, reasonable performance
development environment on low-resource machines) that's less important
now than it used to be, and most programmers these days don't want to
deal with the tradeoffs Forth made to solve that problem, unless they
are dealing with the same problem.

The Arduino really does seem like a natural Forth target, though. It
would be great if Amforth had more uptake. Do you think of encouraging
vendors of Atmel boards to ship with Amforth already installed? Or
maybe just flash a pile of Atmega328 parts and sell them to people?

> I agree that recognizers are not that fine tuned yet. That's why I
> didn't start a CfV.

Recognizers seem like a good idea, even if there are issues still to
work out.

[lawren...@gmail.com]

On Monday, September 26, 2016 at 11:04:43 AM UTC+13, Paul Rubin wrote:
> The Arduino really does seem like a natural Forth target, though.

I don’t think the Arduino can self-host any kind of development environment; you usually cross-compile from some other machine. Something as modest as a Raspberry π will do.

[Paul Rubin]

lawren...@gmail.com writes:
> On Monday, September 26, 2016 at 11:04:43 AM UTC+13, Paul Rubin wrote:
>> The Arduino really does seem like a natural Forth target, though.
> I don’t think the Arduino can self-host any kind of development
> environment

Try amforth.sourceforge.net, written by Mathias Trute who is present in
this thread. See also ulisp.com though that's much less practical.

[hughag...@gmail.com]

On Sunday, September 25, 2016 at 8:11:15 AM UTC-7, tabcom...@gmail.com wrote:
> On Sunday, September 25, 2016 at 12:47:10 AM UTC-4, hughag...@gmail.com wrote:
> What notation are you using in Straight Forth -- `[: ;]` ?

No, I use { ... } for quotations in Straight Forth.
I use LOCAL{ ...} for defining local variables.

I encourage programmers to write explicit iteration using BEGIN WHILE REPEAT but to use the hofs for the two built-in data-structures (arrays and chains) --- and if they do use BEGIN WHILE REPEAT to bury this inside of a hof --- consequently, quotations are going to be used quite a lot and so they get { } which is easy to type and easy to read.

I have the first chapter of my document about Straight Forth written, if you want to read it --- I can email it to you --- it includes the hottie-counting program upgraded for Straight Forth.

> I basically agree with you about recognizers. They are new to me. But it didn't take me long to think maybe that is just going too far. Maybe it makes sense as an optional add on (however that might work) but for general use it would make programs harder to read. I think things need to be moving in the opposite direction.

The problem with recognizers is that they don't allow the reader of the source-code to look up words that he doesn't know. Usually, if you see a word that you don't know, then you can look it up with LOCATE or in a reference manual. You can't look up a recognizer-specific word however --- for example, if you don't know what $BEAD means, how are you going to look it up?

Also, there are likely going to be a lot of recognizers, and they will clash with each other --- a program that runs on one system won't run on another system unless the same recognizers are used and the recognizers are loaded in the same order --- programs aren't going to be portable between systems, which is supposed to be the point of having a Standard.

> OTOH, they are interesting b/c it gives the programmer the power to create DSLs of almost any form. Of course using an extensible lexer could probably achieve much the same thing (which I think Factor does?).

VFX has a BNF lexer --- I haven't looked into it --- I might provide something like this for Straight Forth (I glanced over VFX's BNF lexer and it appeared to be pretty easy to implement).

[Mikael Nordman]

On Monday, September 26, 2016 at 1:04:43 AM UTC+3, Paul Rubin wrote:
>
> The Arduino really does seem like a natural Forth target, though. It
> would be great if Amforth had more uptake. Do you think of encouraging
> vendors of Atmel boards to ship with Amforth already installed? Or
> maybe just flash a pile of Atmega328 parts and sell them to people?
>

I am selling Atmega328 parts flashed with the latest version of FlashForth for the Arduino.

It is usually purchased by people who don't want to bother with flashing arrangements or compilation setup.

[Paul Rubin]

Mikael Nordman <oh2...@gmail.com> writes:
> I am selling Atmega328 parts flashed with the latest version of
> FlashForth for the Arduino.

I had forgotten that FlashForth runs on the Arduino! That's also good
to know.

[Andrew Haley]

Paul Rubin <no.e...@nospam.invalid> wrote:
>
> Recognizers seem like a good idea, even if there are issues still to
> work out.

Well, yes, that's the problem. Lots of things seem like a good idea,
but have unexpected consequences. Recognizers are probably the most
significant change to the Forth language that has (ever?) been
proposed, so a very high bar should be required for their adoption as
standard.

Andrew.

[Albert van der Horst]

In article <e8f43052-8ead-4025...@googlegroups.com>,
<hughag...@gmail.com> wrote:
<SNIP>
>The problem with recognizers is that they don't allow the reader of the sou=
>rce-code to look up words that he doesn't know. Usually, if you see a word =
>that you don't know, then you can look it up with LOCATE or in a reference =
>manual. You can't look up a recognizer-specific word however --- for exampl=

>e, if you don't know what $BEAD means, how are you going to look it up?

That is a good point.
And remember if it is hard for a computer to parse it probably is
also for a human, and vice versa.
In my book only prefixes are allowed. That
means you look up $ , then $B, then $BEA etc. If it is known that
the largest prefix is one character, it is swiftly found as $ would be
the only candidate.
(or e.g. you only restrict prefixes to the minimum search order aka ONLY
wordlist).
ciforth interprets/compiles all char's, strings, numbers in weird BASE's
etc. using only one char prefixes.
The only 2-char prefix is "0x" for hex (which goes better if you want
reserve $ for Unix environment queries). It is not even loadable,
you must define it yourself with
WANT $
'$ ALIAS 0x
(To be honest. If it starts with a 7 it is a number. What kind of
number, single double float hex, is left to the prefix 7 to sort out.)

I originally invented this mechanism from my experience with tForth
and Marcel Hendrix. There were too many prefixes, and I absolutely
didn't want the $-prefix and the #-prefix etc. in my kernel and indeed
they are made loadable in the ciforth (lina) kernel.
That is the same feeling you have about recognizers now.

>
>Also, there are likely going to be a lot of recognizers, and they will clas=
>h with each other --- a program that runs on one system won't run on anothe=
>r system unless the same recognizers are used and the recognizers are loade=
>d in the same order --- programs aren't going to be portable between system=

>s, which is supposed to be the point of having a Standard.

I can see that you fear that from the proposal. It probably isn't that
bad, but maybe it is.

>
>> OTOH, they are interesting b/c it gives the programmer the power to creat=
>e DSLs of almost any form. Of course using an extensible lexer could probab=

>ly achieve much the same thing (which I think Factor does?).
>

>VFX has a BNF lexer --- I haven't looked into it --- I might provide someth=
>ing like this for Straight Forth (I glanced over VFX's BNF lexer and it app=

>eared to be pretty easy to implement).

I doubt that BNF is an advantage, not for the kernel system.

[Alex]

On 26/09/16 10:12, Albert van der Horst wrote:
> In article <e8f43052-8ead-4025...@googlegroups.com>,
> <hughag...@gmail.com> wrote:
> <SNIP>
>> The problem with recognizers is that they don't allow the reader of the sou=
>> rce-code to look up words that he doesn't know. Usually, if you see a word =
>> that you don't know, then you can look it up with LOCATE or in a reference =
>> manual. You can't look up a recognizer-specific word however --- for exampl=
>> e, if you don't know what $BEAD means, how are you going to look it up?
>
> That is a good point.
> And remember if it is hard for a computer to parse it probably is
> also for a human, and vice versa.
> In my book only prefixes are allowed. That
> means you look up $ , then $B, then $BEA etc. If it is known that
> the largest prefix is one character, it is swiftly found as $ would be
> the only candidate.
>

We already have several classes of words like 1.0e-1 that cannot be
LOCATEd.

Use a 4 STATE recognizer; EXECUTE COMPILE, POSTPONE and EXPLAIN. It can
then provide an explanation if it recognises the string (and a LOCATEion
if that's appropriate).


--
Alex

[Albert van der Horst]

In article <nsb02g$ple$1...@dont-email.me>, Alex <al...@rivadpm.com> wrote:
>On 26/09/16 10:12, Albert van der Horst wrote:

<SNIP>

>
>We already have several classes of words like 1.0e-1 that cannot be
>LOCATEd.

LOCATE means look the source up. Nobody expects denotations to be
programmed in the source. So that's okay.

>
>Use a 4 STATE recognizer; EXECUTE COMPILE, POSTPONE and EXPLAIN. It can
>then provide an explanation if it recognises the string (and a LOCATEion
>if that's appropriate).

I would say featuritis. Having EXPLAIN could be nice, it could try a
LOCATE and failing that it could try to tell you what denotation you
have. There is no STATE for SEE and I see no state for EXPLAIN.

But really. Somebody who needs EXPLAIN to understand what 12 means,
is certainly not able to read tutorials where EXPLAIN is explained.

>--
>Alex

[Coos Haak]

Op Mon, 26 Sep 2016 14:30:26 +0200 (CEST) schreef Albert van der Horst:

> In article <nsb02g$ple$1...@dont-email.me>, Alex <al...@rivadpm.com> wrote:
>>On 26/09/16 10:12, Albert van der Horst wrote:
> <SNIP>
>>
>>We already have several classes of words like 1.0e-1 that cannot be
>>LOCATEd.
>
> LOCATE means look the source up. Nobody expects denotations to be
> programmed in the source. So that's okay.
>
>>
>>Use a 4 STATE recognizer; EXECUTE COMPILE, POSTPONE and EXPLAIN. It can
>>then provide an explanation if it recognises the string (and a LOCATEion
>>if that's appropriate).
>
> I would say featuritis. Having EXPLAIN could be nice, it could try a
> LOCATE and failing that it could try to tell you what denotation you
> have. There is no STATE for SEE and I see no state for EXPLAIN.
>
> But really. Somebody who needs EXPLAIN to understand what 12 means,
> is certainly not able to read tutorials where EXPLAIN is explained.
>
>>--
>>Alex
>
> Groetjes Albert

I am that one. I can type 'help forth' or 'help if' and the glossary
of these words are displayed.
But I can also type 'help %1100' and read that this is the number twelve.
help "abacadabra" says that this is a string of ten characters.
help &A says that this is the letter A with value 65.
Just in case anyone wants it.

groet Coos

[Albert van der Horst]

In article <1j1l33ju3gvb4$.9drt1iosj0f2$.d...@40tude.net>,

And would you've found that if it was called EXPLAIN ?
>
>groet Coos

[Anton Ertl]

Alex <al...@rivadpm.com> writes:
>On 26/09/16 10:12, Albert van der Horst wrote:
>> In article <e8f43052-8ead-4025...@googlegroups.com>,
>> <hughag...@gmail.com> wrote:
>> <SNIP>
>>> The problem with recognizers is that they don't allow the reader of the sou=
>>> rce-code to look up words that he doesn't know. Usually, if you see a word =
>>> that you don't know, then you can look it up with LOCATE or in a reference =
>>> manual. You can't look up a recognizer-specific word however --- for exampl=
>>> e, if you don't know what $BEAD means, how are you going to look it up?

Well, $BEAD is already a "word" that is recognized by the standard,
how do you look that up? Or the "word" 1.

I am more of the school of thought of avoiding conflicting recognizers
(at least not having them intentionally) and having tight recognizers
(that do recognizes only the minimal set of "words"), but Bernd Paysan
embraces loose recognizers, conflicting recognizers, and letting thr
priority order resolve conflicts. In order to allow the user to sort
out what's going on in puzzling cases, Bernd has added a trace
facility to Gforth's recognizers; you can turn it on with

' .name is trace-recognizer

and it shows which recognizer is used for every recognized string.

So when I write "1 .", it says

1 . rec:num rec:word 1 ok

So "1" is recognized by rec:num, and "." is recognized by rec:word;
the rest is the usual output of interpreting "1 .".

So, let's answer your question:

$BEAD rec:num ok

>Use a 4 STATE recognizer; EXECUTE COMPILE, POSTPONE and EXPLAIN. It can
>then provide an explanation if it recognises the string (and a LOCATEion
>if that's appropriate).

TRACE-RECOGNIZER is just a hook in the recognizer handling, no extra
state necessary. I am not sure if it is the right user interface in
most cases, however. Maybe SEE should also be reocgnizer-aware, so
that the user can do

see 1

and gets information that 1 is recognized by REC:NUM, and maybe a
decompilation of REC:NUM; and similar for LOCATE and EDIT on those
systems that support them.

[Albert van der Horst]

In article <2016Sep2...@mips.complang.tuwien.ac.at>,

Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
>Alex <al...@rivadpm.com> writes:
>>On 26/09/16 10:12, Albert van der Horst wrote:
>>> In article <e8f43052-8ead-4025...@googlegroups.com>,
>>> <hughag...@gmail.com> wrote:
>>> <SNIP>
>>>> The problem with recognizers is that they don't allow the reader of the sou=
>>>> rce-code to look up words that he doesn't know. Usually, if you see a word =
>>>> that you don't know, then you can look it up with LOCATE or in a reference =
>>>> manual. You can't look up a recognizer-specific word however --- for exampl=
>>>> e, if you don't know what $BEAD means, how are you going to look it up?
>
>Well, $BEAD is already a "word" that is recognized by the standard,
>how do you look that up? Or the "word" 1.

I hope it is not in the CORE set. This is what you get in lina64

OK
$BEAD
$BEAD ? ciforth ERROR # 12 : NOT RECOGNIZED

OK
WANT $-PREFIX
# : ISN'T UNIQUE
$-PREFIX : (WARNING) NOT PRESENT, THOUGH WANTED
OK
$BEAD
OK
.
48813 OK

>
>I am more of the school of thought of avoiding conflicting recognizers
>(at least not having them intentionally) and having tight recognizers

I agree with that. 7 is a recognizer in my book. There is a kind of
conflict here that 7 may be the prefix of a single or a double number.
That conflict is an internal affair in the word 7 itself.

>(that do recognizes only the minimal set of "words"), but Bernd Paysan
>embraces loose recognizers, conflicting recognizers, and letting thr
>priority order resolve conflicts. In order to allow the user to sort
>out what's going on in puzzling cases, Bernd has added a trace
>facility to Gforth's recognizers; you can turn it on with
>
>' .name is trace-recognizer
>
>and it shows which recognizer is used for every recognized string.
>
>So when I write "1 .", it says
>
>1 . rec:num rec:word 1 ok
>
>So "1" is recognized by rec:num, and "." is recognized by rec:word;
>the rest is the usual output of interpreting "1 .".
>
>So, let's answer your question:
>
>$BEAD rec:num ok
>
>>Use a 4 STATE recognizer; EXECUTE COMPILE, POSTPONE and EXPLAIN. It can
>>then provide an explanation if it recognises the string (and a LOCATEion
>>if that's appropriate).
>
>TRACE-RECOGNIZER is just a hook in the recognizer handling, no extra
>state necessary. I am not sure if it is the right user interface in
>most cases, however. Maybe SEE should also be reocgnizer-aware, so
>that the user can do
>
>see 1
>
>and gets information that 1 is recognized by REC:NUM, and maybe a
>decompilation of REC:NUM; and similar for LOCATE and EDIT on those
>systems that support them.

I've still 1 as a constant in the kernel.
The recognizer is only visible after hiding that.

OK
SEE 1

0000,0000,0000,0001 CONSTANT 1
OK
'1 HIDDEN
OK
SEE 1

: 1
FFFF,FFFF,FFFF,FFFF PP +! (NUMBER) POSTPONE SDLITERAL
; IMMEDIATE PREFIX
OK

SEE 12
SEE 12 ? ciforth ERROR # 11 : WORD IS NOT FOUND

So in my book 12 is not a recognizer, it is a recognizee.
recognizees are not Forth words, because they are not in
the dictionary. SEE works on Forth words.

(Strange word recognizee, but the term denotation didn't catch on).

>
>- anton

[Anton Ertl]

On the contrary, recognizers are not a change to the Forth language at
all. They have been in Forth since day 1. The very first Forth has
recognized 1 as number.

Ok, you might say, but that does not allow the user to add
recognizers. However, users have been able to add recognizers of all
kinds to Forth systems for many decades. E.g., if I look at the
floating point packages of SwiftForth, VFX, and Gforth, I see:

SwiftForth:
: FNUMBER? ( addr len 0 | ... xt -- addr len 0 | ... xt )
?DUP ?EXIT BASE @ 10 = IF
R-BUF 2DUP R@ PLACE BL R@ COUNT + C! R> (REAL) IF
2DROP ['] FLITERAL EXIT
ELSE DROP THEN THEN 0 ;

' FNUMBER? NUMBER-CONVERSION <CHAIN

VFX:
defer old-number?
action-of number? to-do old-number?

: fnumber? \ c-addr -- 0 | n 1 | d 2 | -2 ; F: -- r
\ *G Behaves like the integer version of *\fo{NUMBER?} except
\ ** that if integer conversion fails, and *\fo{BASE} is decimal,
\ ** a floating point conversion is attempted. If conversion is
\ ** successful, the floating point number is left on the float
\ ** stack and the result code is -2.
dup >r old-number? dup 0= if \ test for integer first
base @ #10 = if \ not integer, floats must be in DECIMAL
drop r@ count (>float) \ >FLOAT returns well formed flag
-2 and \ good float returns -2,
then
then
r> drop
;

assign fnumber? to-do number?
assign #fliteral to-do c#fliteral

Gforth 0.7.0:
:noname ( c-addr u -- ... xt )
2dup sfnumber
IF
2drop [comp'] FLiteral
ELSE
defers compiler-notfound1
ENDIF ;
IS compiler-notfound1

:noname ( c-addr u -- ... xt )
2dup sfnumber
IF
2drop ['] noop
ELSE
defers interpreter-notfound1
ENDIF ;
IS interpreter-notfound1


And just like the floating point recognizer can be added to these
Forth systems, every other recognizer can be added, too.

What the recognizer proposal does is to add a standardized interface
for adding recognizers and dealing with them, such that portable
libraries can provide recognizers. But the basic capability has been
there in a non-portable way for decades. There have been regular
discussions about providing a standardized interface for these
capabilities, demonstrating that there is demand for making such
things portable. And at one point one of these discussions came up
with an embryonic version of the current recognizer proposal.

If you think that user-defined recognizers are a significant change to
Forth, you should have voiced your concerns when Forth, Inc, added the
NUMBER-CONVERSION chain, MPE made NUMBER? a deferred word (or whatever
it is), and when Gforth added NOTFOUND (the pre-predecessor of
COMPILER-NOTFOUND1 and INTERPRETER-NOTFOUND1).

[Anton Ertl]

Andrew Haley <andr...@littlepinkcloud.invalid> writes:
>Quotations are neat, but they don't add a great deal of power to the
>language. On the other hand, their cost and footprint is small. I
>think that they'll turn out to be useful enough to be standardized,
>but once a word is in the standard it's hard to get rid of it.

Currently it is much easier to get rid of words than to add words;
admittedly, for the getting-rid part to actually take effect takes a
lot of time, but the amount of work required is smaller by several
orders of magnitude:

For getting rid of a word, you don't even need to make an RfD; someone
proposes to make the word obsolescent at a meeting, without providing
any numbers on usage or anything, the committee takes a vote, and the
word is obsolescent, and will be gone in the next-but-one standard
unless there is a concerted effort to keep it in, as happened for
FORGET (and even then the committee decided to keep it obsolescent, so
the fans of FORGET may have to make the same effort the next time).
This has happened to LOCALS| and [COMPILE], and almost to (LOCAL).

For adding a word, you can jump through all the hoops, write a
proposal, get it voted on, have implementations for several Forth
systems, demonstrate that the programmers feel that there is a need
for it by resorting to return-address manipulation to get the feature,
show a count of how many times the proposed form of the feature is
used, answer all the questions that the committee has about the
proposal (with an explicit "no" to the question whether there are
further questions), and on committee vote the proposal fails to get
consensus. This has happened to quotations.

[Andrew Haley]

Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
> Andrew Haley <andr...@littlepinkcloud.invalid> writes:
>>Paul Rubin <no.e...@nospam.invalid> wrote:
>>>
>>> Recognizers seem like a good idea, even if there are issues still to
>>> work out.
>>
>>Well, yes, that's the problem. Lots of things seem like a good idea,
>>but have unexpected consequences. Recognizers are probably the most
>>significant change to the Forth language that has (ever?) been
>>proposed, so a very high bar should be required for their adoption as
>>standard.
>
> On the contrary, recognizers are not a change to the Forth language
> at all. They have been in Forth since day 1. The very first Forth
> has recognized 1 as number.

I'm fairly sure it was never described in such a way. I don't think
it's fair to use the term "recognizers" for both the recognizer
proposal and what Forths have always done.

> What the recognizer proposal does is to add a standardized interface
> for adding recognizers and dealing with them, such that portable
> libraries can provide recognizers.

I agree. However, its scope is not limited to literals, and some
non-literal recognizers have been discussed here. (e.g. the ">X"
syntax to mean "TO X"). I don't doubt that this has been possible
with some vendor-specific Forth code. But there is a very significant
difference between something vendors do (and can be revised or removed
completely) and something which is standard, and therefore much harder
to remove if it turns out not to have been a good idea.

> But the basic capability has been there in a non-portable way for
> decades.

Great! So all you have to do is persuade the authors of Forth
systems, particularly the professional vendors, to support the
proposal you favour on their systems. And do that based on the actual
merits of what you propose.

Andrew.

[Andrew Haley]

Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
> Andrew Haley <andr...@littlepinkcloud.invalid> writes:
>>Quotations are neat, but they don't add a great deal of power to the
>>language. On the other hand, their cost and footprint is small. I
>>think that they'll turn out to be useful enough to be standardized,
>>but once a word is in the standard it's hard to get rid of it.
>
> Currently it is much easier to get rid of words than to add words;
> admittedly, for the getting-rid part to actually take effect takes a
> lot of time, but the amount of work required is smaller by several
> orders of magnitude:

By "it's hard" I don't necessarily mean that it's less work: I mean
that it is difficult (perhaps impossible) to get rid of a feature once
it is in use in applications because -- quite rightly -- people tend
to treat not breaking code as very important.

Andrew.

[Anton Ertl]

Matthias Trute <matthia...@gmail.com> writes:
>I think that's the consequence of the design principle of the Forth
>standard process: codify existing practice and consolidate it. Other
>standard bodies follow a different approach: define something and declare
>it standard, or standard extension. E.g. the WLAN or Ethernet standards
>follow this model. First you get the specs, the implementation follows.
>

>Forth could use both. There is a core feature set like the stacks, some wor=
>ds (in CORE) and the interpreter/compile model, and there are a number of e=
>xtension wordsets. These extension wordsets could be managed more freely th=
>an the core.

Well, we have that already. We have the CORE wordset, and IIRC
nothing has been added to it or removed from it between Forth-94 and
Forth-2012 except a feature or two already declared obsolescent in
Forth-94 (space at the end of the WORD buffer IIRC). The only thing
that was really added to the core in Forth-2012 is number prefixes,
and that's because making them optional would have turned the document
into an ifdef maze.

All the extension wordsets are optional. But even adding things to
extension wordsets is very tough going, to put it mildly.

So in the latest meeting the committee has decided to encourage
"experimental proposals". It's not quite clear to me how that differs
from an ordinary RfD, but my impression is the idea is to put up
another hoop for proponents so that the ossified standard can be
protected from dangerous ideas.

My take on this is to revert back to the original idea of the RfD/CfV
process: Have an RfD phase where you hash out the proposal, and when
you think you have a good and as widely acceptable as possible
proposal, freeze it and make a CfV; and because it is frozen, now more
systems can implement it (or one can have implementations for systems
where the maintainers are unwilling), and users can use it, and one
can indicate the level of support of the proposal in the votes (and
maybe one day also in the number of dependences on theforth.net). If
the proposal does not get accepted into the standard, but is widely
supported and used, the only consequence is that you have to look it
up in the CfV (and that the standard makes itself less relevant).

>They fol=

>low the TurboPascal model, hopefully with better luck.

I think we could all wish that any Forth system had as much luck as
Turbo Pascal.

[Stephen Pelc]

On Mon, 26 Sep 2016 16:01:16 GMT, an...@mips.complang.tuwien.ac.at
(Anton Ertl) wrote:

>So in the latest meeting the committee has decided to encourage
>"experimental proposals". It's not quite clear to me how that differs
>from an ordinary RfD, but my impression is the idea is to put up
>another hoop for proponents so that the ossified standard can be
>protected from dangerous ideas.

The remit of the standard is to codify common practice. This means
that the practice to be standardised is widespread. The idea of
experimental proposals is to allow proposals for new practice to be
taken seriously. and perhaps to become common practice.

Regardless of the 1984-style redefinition of the mean of
"recognition", recognisers are not widespread at present. They
are a perfect example of experimental practice, and by having a
formal proposal, systems are encouraged to experiment with them.

Stephen


--
Stephen Pelc, steph...@mpeforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

[Anton Ertl]

What you describe is they way it should work. But it actually does
not work that way. It actually works the way I described in the part
that you conveniently deleted, so here it is again:

|For getting rid of a word, you don't even need to make an RfD; someone
|proposes to make the word obsolescent at a meeting, without providing
|any numbers on usage or anything, the committee takes a vote, and the
|word is obsolescent, and will be gone in the next-but-one standard
|unless there is a concerted effort to keep it in, as happened for
|FORGET (and even then the committee decided to keep it obsolescent, so
|the fans of FORGET may have to make the same effort the next time).
|This has happened to LOCALS| and [COMPILE], and almost to (LOCAL).

So while parts of the committee are extremely reluctant when it comes
to adding new words (because it is supposedly so difficult to get rid
of them), the committee is quite keen on making words obsolescent
(which leads to deletion from the standard in a later version), and
does not do any research about the usage of the word in applications
before taking that decision.

A nice example is LOCALS|. It was added in Forth-94, and was not a
good solution, so it was declared obsolescent in Forth-2012 and will
be removed from the standard in the next version. This was not at all
difficult.

What actually is difficult is getting a de-standardized word really
out of use, as we have seen in occasional discussions of NOT. But
such problems can just as well occur if the word has never been
standardized; example: "{".

[trans]

On Saturday, September 24, 2016 at 8:14:58 AM UTC-4, Anton Ertl wrote:

> >Another idea is to have a standard Forth with pseudo-static typing:
> >i.e., have a testing/debugging mode where every cell is accompanied by a
> >tag field that's checked at runtime, signalling errors that might
> >otherwise be missed. Then once debugged, you can turn off the checks.
> >No idea how useful this is.
>
> Again, you need a type system, but this one might be easier than a
> statically checkable one.

I think one of trending ideas now in language research is hybrid type systems.[1] Basically the language uses static typing where it can and falls back to dynamic typing when it can't. It's been dubbed "soft typing" from Mike Fagan's dissertation on the subject.[2]

1. https://www.ics.uci.edu/~lopes/teaching/inf212W12/readings/rdl04meijer.pdf
2. https://scholarship.rice.edu/handle/1911/16439

Static tying works fine until you have to load a YAML document, or want to write a method than can return 12 different types of objects.

On the other hand dynamic type systems can eat memory. A 32 bit type header for every 8 bit integer seems pretty wasteful. And of course they are slower. Although V8 has proven they don't necessarily have to be so slow.

Forth is an interesting case, b/c it is (typically) statically typed, but you wouldn't know it b/c there are only a few types: integer, fixed and/or float and the cell. Everything else is just built on top of those. In a way it is bit of trick. We just use different words for for different types. That's basically how it works for any actual type that gets implemented in Forth. It transfers responsibility from a type system to word names and the programmer has to use the appropriate ones. Which isn't type safe, obviously, but can be quite flexible in the hands of a good coder.

[Andrew Haley]

Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
> Andrew Haley <andr...@littlepinkcloud.invalid> writes:
>>Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
>>> Andrew Haley <andr...@littlepinkcloud.invalid> writes:
>>>>but once a word is in the standard it's hard to get rid of it.
>>>
>>> Currently it is much easier to get rid of words than to add words;
>>> admittedly, for the getting-rid part to actually take effect takes a
>>> lot of time, but the amount of work required is smaller by several
>>> orders of magnitude:
>>
>>By "it's hard" I don't necessarily mean that it's less work: I mean
>>that it is difficult (perhaps impossible) to get rid of a feature once
>>it is in use in applications because -- quite rightly -- people tend
>>to treat not breaking code as very important.
>
> What you describe is they way it should work. But it actually does
> not work that way. It actually works the way I described in the part
> that you conveniently deleted, so here it is again:

I will again cite FORGET as the counter-example.

> |For getting rid of a word, you don't even need to make an RfD; someone
> |proposes to make the word obsolescent at a meeting, without providing
> |any numbers on usage or anything, the committee takes a vote, and the
> |word is obsolescent, and will be gone in the next-but-one standard
> |unless there is a concerted effort to keep it in, as happened for
> |FORGET (and even then the committee decided to keep it obsolescent, so
> |the fans of FORGET may have to make the same effort the next time).
> |This has happened to LOCALS| and [COMPILE], and almost to (LOCAL).
>
> So while parts of the committee are extremely reluctant when it comes
> to adding new words (because it is supposedly so difficult to get rid
> of them), the committee is quite keen on making words obsolescent
> (which leads to deletion from the standard in a later version),

Good. Forth is a small language. It should be easier to remove
standard features than to add them.

> and does not do any research about the usage of the word in
> applications before taking that decision.

I think that's a bit of a stretch.

> What actually is difficult is getting a de-standardized word really
> out of use, as we have seen in occasional discussions of NOT. But
> such problems can just as well occur if the word has never been
> standardized; example: "{".

My point stands. If a word is in the standard, it's much harder to
de-standardize it than if it is not in the standard.

Andrew.

[Alex]

The "not breaking code" argument doesn't necessarily win by saying it's
very important. It has to demonstrate that stability is vital.

Currently there's a pretty heated argument over Docker about release
compatibility & standards. Docker Inc. seem to believe that breaking
code is not only OK, but that it's desirable -- nay, vital.

http://www.infoworld.com/article/3105857/open-source-tools/save-the-whale-docker-rightfully-shuns-standardization.html

--
Alex

[Anton Ertl]

trans <tran...@gmail.com> writes:
>Forth is an interesting case, b/c it is (typically) statically typed, but y=
>ou wouldn't know it b/c there are only a few types: integer, fixed and/or f=
>loat and the cell. Everything else is just built on top of those. In a way =
>it is bit of trick. We just use different words for for different types. Th=
>at's basically how it works for any actual type that gets implemented in Fo=
>rth. It transfers responsibility from a type system to word names and the p=
>rogrammer has to use the appropriate ones. Which isn't type safe, obviously=

>, but can be quite flexible in the hands of a good coder.

Every useful program has certain static properties; that follows from
programmers having to reason how the program works. In Forth that's
all there is: the system does not check any of the properties, so it's
all in the programmer's head (and in the comments in the program).

Some languages try to check static properties with the compiler; they
either force the programmer to state properties (e.g., how many
parameters are passed, and what types they have), or they try to infer
it from the program and still check it (type inference).

Applied to Forth, a pretty good static property is stack depth.

If we then try to extend this to types, there are things that have to
be wrong (at least in portable code), such as

5 'a' !

but others are less clear:

5 'a' *

That may be an index into an array with 5-byte elements; or maybe some
part of a hash computation; or ... So we should probably accept it.

However, if we have

4 5 'a' * X + !

with X being an array of cells, there's probably a type error here
(and, on some architectures, an alignment error); the array is not
accessed using a multiple-of-cell offset, and there my be a CELLS
missing, or something else is funny.

Designing Forth type-checking rules beyond stack depth checking that
catch many errors without producing lots of false positives is an
unsolved problem. And even stack-depth checking is not trivial; e.g.,
if you write a defining word for a method, the defined methods can
have very different stack effects, but each method typically has only
one stack effect.

[Anton Ertl]

ste...@mpeforth.com (Stephen Pelc) writes:
>On Mon, 26 Sep 2016 16:01:16 GMT, an...@mips.complang.tuwien.ac.at
>(Anton Ertl) wrote:
>
>>So in the latest meeting the committee has decided to encourage
>>"experimental proposals". It's not quite clear to me how that differs
>>from an ordinary RfD, but my impression is the idea is to put up
>>another hoop for proponents so that the ossified standard can be
>>protected from dangerous ideas.
>
>The remit of the standard is to codify common practice. This means
>that the practice to be standardised is widespread. The idea of
>experimental proposals is to allow proposals for new practice to be
>taken seriously. and perhaps to become common practice.

You mean, until now you don't take RfDs for new practice seriously,
but once the "experimental proposal" sticker is put on it, you will?

>Regardless of the 1984-style redefinition of the mean of
>"recognition", recognisers are not widespread at present. They
>are a perfect example of experimental practice, and by having a
>formal proposal, systems are encouraged to experiment with them.

Sure. That proposal is at the RfD stage, so accepting it into the
standard is not even on the table. However, what I am missing is the
willingness for experimentation, and using the face-to-face time that
you think is so valuable for hashing out technical (or documentation)
issues. Given that, your reaction will be the same next year, in five
years, in twenty years, etc.

As for recognizers not being widespread, they are present in at least
two systems, more than the Text Substitution proposal when it was put
up for committee vote, and maybe as many as the Text Substitution
proposal is implemented in now. So, when the proposal reaches
stability (CfV stage), that level of spreading should be enough for
you to accept it.

[Andrew Haley]

Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
> ste...@mpeforth.com (Stephen Pelc) writes:
>>On Mon, 26 Sep 2016 16:01:16 GMT, an...@mips.complang.tuwien.ac.at
>>(Anton Ertl) wrote:
>>
>>>So in the latest meeting the committee has decided to encourage
>>>"experimental proposals". It's not quite clear to me how that differs
>>>from an ordinary RfD, but my impression is the idea is to put up
>>>another hoop for proponents so that the ossified standard can be
>>>protected from dangerous ideas.
>>
>>The remit of the standard is to codify common practice. This means
>>that the practice to be standardised is widespread. The idea of
>>experimental proposals is to allow proposals for new practice to be
>>taken seriously. and perhaps to become common practice.
>
> You mean, until now you don't take RfDs for new practice seriously,
> but once the "experimental proposal" sticker is put on it, you will?

That sounds reasonable. All that "RfD" means is that someone
considers something worth discussing. An Experimental Proposal is
something that the committee as a body thinks is a real candidate for
standardization.

Andrew.

[Anton Ertl]

Andrew Haley <andr...@littlepinkcloud.invalid> writes:
>Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
>> Andrew Haley <andr...@littlepinkcloud.invalid> writes:
>>>Paul Rubin <no.e...@nospam.invalid> wrote:
>>>>
>>>> Recognizers seem like a good idea, even if there are issues still to
>>>> work out.
>>>
>>>Well, yes, that's the problem. Lots of things seem like a good idea,
>>>but have unexpected consequences. Recognizers are probably the most
>>>significant change to the Forth language that has (ever?) been
>>>proposed, so a very high bar should be required for their adoption as
>>>standard.
>>
>> On the contrary, recognizers are not a change to the Forth language
>> at all. They have been in Forth since day 1. The very first Forth
>> has recognized 1 as number.
>
>I'm fairly sure it was never described in such a way.

Maybe not. Concepts evolve, and terms are introduced to go with that.
E.g., Luigi Galvani did not describe his discovery as
"bioelectromagnetics" ("electromagnetism" had not been discovered at
the time), yet <https://en.wikipedia.org/wiki/Electricity#History>
says: "In 1791, Luigi Galvani published his discovery of
bioelectromagnetics [...]".

>I don't think
>it's fair to use the term "recognizers" for both the recognizer
>proposal and what Forths have always done.

I think it's entirely fair. When the difference plays a role, you can
add adjectives like "standardized", "system-pecific", "user-defined",
"system-defined", "extensible", and "non-extensible".

So what Forth systems have always done is to have a bunch of
standardized system-defined recognizers and system-specific hooks for
adding user-defined or more system-defined recognizers. What is being
proposed came out of discussions for standardizing these hooks.

>I agree. However, its scope is not limited to literals, and some
>non-literal recognizers have been discussed here. (e.g. the ">X"
>syntax to mean "TO X").

Actually "->X". Yes, that is possible with user-defined recognizers,
and it is also possible with the system-specific interpreter hooks
that Forth systems have always had. And these system-specific hooks
have actually been used for non-literals, e.g., the dot-parser of VFX.
So that's nothing new that the recognizer proposal introduces.

>But there is a very significant
>difference between something vendors do (and can be revised or removed
>completely) and something which is standard, and therefore much harder
>to remove if it turns out not to have been a good idea.

As we have seen with "{", vendors do not revise or remove non-standard
words. And as we have seen with LOCALS|, it is easy to de-standardize
words that turned out not to have been a good idea.

You can think only about removing any proposed feature, and when
that's your only goal, then yes, de-standardizing a feature that has
been standardized is harder than de-standardizing it if it has not
been.

Or you can think about what benefits the proposed feature would
provide, what costs it would impose on system implementors and program
readers, if the proposed interface is good or if you can suggest
improvements.

I don't think an attitude that only takes de-standardization into
consideration is cunducive to the aim of having programs and libraries
that are portable between systems. The latter is certainly an aim I
am pursuing. What aims are you pursuing?

[Andrew Haley]

Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
> Andrew Haley <andr...@littlepinkcloud.invalid> writes:
>>Anton Ertl <an...@mips.complang.tuwien.ac.at> wrote:
>>> Andrew Haley <andr...@littlepinkcloud.invalid> writes:
>>>>Paul Rubin <no.e...@nospam.invalid> wrote:
>>>>>
>>>>> Recognizers seem like a good idea, even if there are issues still to
>>>>> work out.
>>>>
>>>>Well, yes, that's the problem. Lots of things seem like a good idea,
>>>>but have unexpected consequences. Recognizers are probably the most
>>>>significant change to the Forth language that has (ever?) been
>>>>proposed, so a very high bar should be required for their adoption as
>>>>standard.
>>>
>>> On the contrary, recognizers are not a change to the Forth language
>>> at all. They have been in Forth since day 1. The very first Forth
>>> has recognized 1 as number.
>>
>>I'm fairly sure it was never described in such a way.
>
> Maybe not. Concepts evolve, and terms are introduced to go with that.

I'm sure, but I need perhaps to explain that when I talk about
"recognizers" I'm talking bout the recognizer proposal on the table.

>>I don't think it's fair to use the term "recognizers" for both the
>>recognizer proposal and what Forths have always done.
>
> I think it's entirely fair.

There we will have to agree to differ.

> When the difference plays a role, you can add adjectives like
> "standardized", "system-pecific", "user-defined", "system-defined",
> "extensible", and "non-extensible".
>
> So what Forth systems have always done is to have a bunch of
> standardized system-defined recognizers and system-specific hooks for
> adding user-defined or more system-defined recognizers. What is being
> proposed came out of discussions for standardizing these hooks.

Indeed.

>>I agree. However, its scope is not limited to literals, and some
>>non-literal recognizers have been discussed here. (e.g. the ">X"
>>syntax to mean "TO X").
>
> Actually "->X". Yes, that is possible with user-defined recognizers,
> and it is also possible with the system-specific interpreter hooks
> that Forth systems have always had. And these system-specific hooks
> have actually been used for non-literals, e.g., the dot-parser of VFX.
> So that's nothing new that the recognizer proposal introduces.

No, that is not true. It adds to the standard language a feature
which allows libraries to add arbitrary strings to the source
language, and for each library to define its own set of strings which
will be recognized. And not only to change the meaning of words but
also to change the way that source strings are scanned. I do not deny
that this was possible already in some Forth systems, but we are
talking about a proposal for standardization. I suspect that it
wasn't standardized, even though some systems have system-specific
hooks, because it is inherently highly problematic.

>>But there is a very significant difference between something vendors
>>do (and can be revised or removed completely) and something which is
>>standard, and therefore much harder to remove if it turns out not to
>>have been a good idea.
>
> As we have seen with "{", vendors do not revise or remove non-standard
> words. And as we have seen with LOCALS|, it is easy to de-standardize
> words that turned out not to have been a good idea.
>
> You can think only about removing any proposed feature, and when
> that's your only goal, then yes,

Destandardizing a feature that has been standardized is harder than
destandardizing it if it has not been. This is true no matter what
you're thinking about.

> Or you can think about what benefits the proposed feature would
> provide, what costs it would impose on system implementors and program
> readers, if the proposed interface is good or if you can suggest
> improvements.

That sounds very sensible.

> I don't think an attitude that only takes de-standardization into
> consideration is cunducive to the aim of having programs and libraries
> that are portable between systems.

Neither do I. Nobody does, as far as I'm aware.

> The latter is certainly an aim I am pursuing. What aims are you
> pursuing?

The issue here isn't only writing portable libraries. There is also
the coherence, readability, and maintainability of programs to
consider. This proposal is very general and might not help people to
write portable libraries at all. In fact, recognizers in libraries
have the potential of changing a language which is, at base, words and
literals into an incomprehensible alphabet soup with insufficient
tools to understand the resulting confusion. And this will work
*against* the adoption of portable libraries.

I repeat my earlier comment: you are trying to jump the starting gun.
If you want a proposal to be standardized, you should persuade
implementers to use it in their systems and programmers to try it.
And you must do so on its own merits. If a proposal has significant
benefits for them and their users then they will want to do that
anyway.

Andrew.

[Stephen Pelc]

On Tue, 27 Sep 2016 14:49:12 GMT, an...@mips.complang.tuwien.ac.at
(Anton Ertl) wrote:

>ste...@mpeforth.com (Stephen Pelc) writes:
>>The remit of the standard is to codify common practice. This means
>>that the practice to be standardised is widespread. The idea of
>>experimental proposals is to allow proposals for new practice to be
>>taken seriously. and perhaps to become common practice.
>
>You mean, until now you don't take RfDs for new practice seriously,
>but once the "experimental proposal" sticker is put on it, you will?

There are so many ideas to be tried that, as always, we encourage
natural selection by asking a "champion" to put a proposal together.

>
>>Regardless of the 1984-style redefinition of the mean of
>>"recognition", recognisers are not widespread at present. They
>>are a perfect example of experimental practice, and by having a
>>formal proposal, systems are encouraged to experiment with them.
>
>Sure. That proposal is at the RfD stage, so accepting it into the
>standard is not even on the table. However, what I am missing is the
>willingness for experimentation, and using the face-to-face time that
>you think is so valuable for hashing out technical (or documentation)
>issues. Given that, your reaction will be the same next year, in five
>years, in twenty years, etc.

The file Examples/recogniser.fth is in all VFX distributions.
Personally I will look at recognisers in my ample spare time.
However, recognisers implicitly suggest a major restructuring
of the Forth text interpreter. That veers towards mandating
implementation, which worries me. There are other committee
members who have expressed worry in public.

>As for recognizers not being widespread, they are present in at least
>two systems, more than the Text Substitution proposal when it was put
>up for committee vote, and maybe as many as the Text Substitution
>proposal is implemented in now. So, when the proposal reaches
>stability (CfV stage), that level of spreading should be enough for
>you to accept it.

You didn't have to vote for it. SUBSTITUTE has the benefit of nearly
20 years usage in MPE's ProForth and VFX lines. Text macro
substitution exists in all Unix shells and other languages. It has
been in the internationalisation proposal for 18 years.

That you see no benefit to it suggests that you do not write
internationalised applications. It is already clear that Anton and
Stephen have very different views on source file naming and
management.

[jim.bra...@ieee.org]

On Thursday, September 15, 2016 at 12:24:49 PM UTC-5, Anton Ertl wrote:
> At EuroForth Andrew Read gave an impromptu talk where he suggested a
> next-standard effort that would not be shackled by
> backwards-compatibility with the current (Forth-83/94/2012) line of
> standards.
>
> While there was not that much desire for such an effort in the
> audience, the talk got me thinking about what is essential in the
> current line of Forth standards and which features of a Forth system
> would require such a next-standard effort (if there are people that
> want to have a standard for that).
>
> What is not essential:
>
> Of course, there are people (including committee members) who disagree
> with some choices made in the standard, but that's a matter of opinion
> and not the kind of "essential" I have in mind; In particular, if
> people could not find a consensus on a different choice earlier, will
> it help to start from scratch? In at least one case, there actually
> has been consensus on taking a different choice than an earlier
> standard (standardizing "{:" and making "LOCALS|" obsolescent), so if
> that happens, the current standard process can change choices, too,
> slowly. We don't need a from-scratch effort for that.
>
> Another issue is removing features, to achieve some form of
> minimalism. We have a mechanism in the standards process for removing
> features (it takes it's time, though), so if the community decided we
> really want minimality, and if we actually could find consensus on
> what we want in a minimal system, we could go there; in practice, this
> will not happen, but that has more to do with there being no consensus
> that we want a minimal system and no consensus on what it would look
> like.
>
> Another issue is that of names. While machine Forth calls the
> exclusive-or operation OR, I don't consider that an essential feature
> of machine Forth. It would be pretty much the same if Chuck Moore had
> chosen to call it XOR. Conversely, if we wanted to remove inclusive
> OR, later add an OR that's a synonym for XOR, and then remove XOR,
> this could be done, very slowly, over several standards releases.
>
> Possibly essential:
>
> The syntax difference between machine Forth and Color Forth (color
> that indicates how to text-interpret the string) cannot be
> accommodated in the current line of standards; while it's just syntax,
> it's a change that does not allow a transition period, so it would
> make no sense to try to do it in the current line of standards.
>
> Pretty essential:
>
> Statically determined stack depth: Requiring that, as sketched for
> PAF, would disallow some standard programs (and a lot of buggy
> programs). I guess we could get there in the current line of
> standards, but I don't expect it.
>
> Static type checking: StrongForth did that, but also implemented
> overloading (e.g., DUP worked as 2DUP when there was a double on the
> TOS). Static type checking would first require a type system; it
> would probably disallow more standard programs, especially if you want
> to have a somewhat safe type system; the actual repercussions would
> depend on the type system. Overloading would be even more disruptive,
> by a lot.
>
> Data stack items: In the current standards, the data stack item is a
> machine word (cell), used as integer, address, or a group of bits and
> its interpretation depends on the operation; there are a number of
> systems that deviate from this:
>
> Floats: There is one suggestion (and a Russion system that works this
> way) to use floats as stack items, with words like + working on
> floats, and IIRC addresses represented by NaNs (or maybe by
> interpreting the bit pattern as address). That's pretty far from the
> current standard.
>
> Dynamically typed stack items (e.g., Factor, 8th, Oforth, with Factor
> also doing static type checking to some extent), with dynamic type
> checking in the operations and possible overloading, and possible
> automatic memory reclamation. That's even further from the current
> line of standards. Looking at the number of systems that go that way,
> there seems to be quite a bit of interest in that direction, so maybe
> that's something where such a standard that's not rooted in the
> current line of standards might make sense. But I guess that the
> implementors and users of these systems have to work it out if they
> need a standard for that, or if they are happy with the different
> systems that they have now.

>
> - anton
> --
> M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
> comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
> New standard: http://www.forth200x.org/forth200x.html
> EuroForth 2016: http://www.euroforth.org/ef16/

]> what is essential in the current line of Forth standards and which
]> features of a Forth system would require such a next-standard effort

Not a Forth expert, think I know enough to pose the following:

Been watching the "recognizer" discussion and have realized that my problem
is not there but with the interpret/compile distinction. Immediate words that apply only to the following word are OK. Basically, want to eliminate the compile state!(1) This implies that quotations are a form of string/list constant and are used in word definitions: ":" word "{:" quotation ";}" ";" or better yet: "'" word "{:" quotation ";}" ";" ( where "{:" and ";}" are adjusted to suit your tastes. Also implies a formal stack effect string/list "constant" or an inferred stack effect (added by the text editor?).

(1) 8th allows stack operations within a quotation by use of a start/stop single character word. This seems OK.

The basic problem is in reading Forth code and knowing where the changes between interpret and compile take place. Now in line with the Forth spirit it is good to let the programmer have access and control over this state.(2) However, for the uninitiated, limiting such state changes to three or four single character bracket words is much better.

(2) another way to distinguish is to show compile text in italics or underline?

[Bernd Paysan]

Am Wed, 28 Sep 2016 12:07:00 +0000 schrieb Stephen Pelc:

> However, recognisers implicitly suggest a major restructuring of the
> Forth text interpreter.

It replaces a hard-coded structure by one that is extensible by the user.
We all have long done that, but in an incompatible way that is sometimes
not even stable across versions (Klaus Schleisiek got stuck with Gforth
0.6.2, because we changed one of the lesser documented hooks in the text
interpreter, and his code that extends the text interpreter doesn't work
with newer versions).

So the problem of extending the text interpreter has always been there,
and had various incompatible solutions, which have been added in an ad hoc
fashion to the diverging implementations.

Conceptually, the Forth outer interpreter always was a sequence of
matches, first words in the dictionary, then (if that failed) numbers,
and then, if loaded, floating point numbers. The recognizers generalize
that concept, makes it extensible, but does not fundamentally change it.

> That veers towards mandating implementation, which worries me.

I'm trying hard to make the recognizers abstract enough that it's not
mandating too much of a particular implementation. E.g. amForth's
recognizer return tables are actual tables, Gforth's are not. The way a
word is dealt with can be different from system to system, my
implementation for VFX e.g. puts uses the immediate flag to select the
appropriate return table; other options are to return a name token
instead of an xt.

Same with numbers, to satisfy the existing hook in VFX, there's only one
R:NUM as a recognizer return token for literals, while Gforth has R:NUM,
R:DNUM, R:FLOAT for the different data types.

> There are other committee members who have expressed
> worry in public.

Yes, Andrew Haley was worried about "too much empowerment" for the
programmer. Hey, we are not doing Java here, we do Forth.

The recognizer proposal came partly as a response of exactly you two
asking for standardizing OOP or thinking about some "common grounds" like
Common Lisp's generic OOP system, so one obviously open question was how
to add something like a dot parser, which you advocated, to a system
without making that a special-purpose hook.

Your outer interpreter has a CLASSHOOK before, and a CUNDEFINED after the
traditional interpreter; however, the actual components of the text
interpreter like DONUMBER?, C#FLITERAL and C#LITERAL are also all hooks.
The only thing that isn't a hook in the outer part is FIND, even COMPILE,
and CEXECUTE are hooks.

This entire thing is hooks calling other hooks. Do you see the design
problem with this? Once you have the recognizer embraced, you can stop
that: There's just one list of hooks which gets processed in a generic
loop: All are the same structure. If you need to replace parts of the
outer interpreter, e.g. for a cross compiler, just replace them on the
recognizer stack. No need for special-purpose-hooks everywhere.

--
Bernd Paysan
"If you want it done right, you have to do it yourself"
net2o ID: kQusJzA;7*?t=uy@X}1GWr!+0qqp_Cn176t4(dQ*
http://bernd-paysan.de/

[hughag...@gmail.com]

On Wednesday, September 28, 2016 at 7:07:45 PM UTC-7, Bernd Paysan wrote:
> Am Wed, 28 Sep 2016 12:07:00 +0000 schrieb Stephen Pelc:
> > That veers towards mandating implementation, which worries me.
>
> I'm trying hard to make the recognizers abstract enough that it's not
> mandating too much of a particular implementation.

> ...

> > There are other committee members who have expressed
> > worry in public.
>
> Yes, Andrew Haley was worried about "too much empowerment" for the
> programmer. Hey, we are not doing Java here, we do Forth.

Well, Bernd Paysan assumes that he alone knows how "we do Forth" --- very arrogant!

Recognizers are the complete opposite of how we "do" Straight Forth --- the name "Straight Forth" implies that it is a traditional Forth without all of the weird ambiguity of ANS-Forth, and without all of the weird everything of Forth-200x.

I see recognizers as being the wedge that splits the Forth community into two camps --- Forth-200x for those who want recognizers --- Straight Forth for those who just want a straight Forth.

> The recognizer proposal came partly as a response of exactly you two
> asking for standardizing OOP or thinking about some "common grounds" like
> Common Lisp's generic OOP system, so one obviously open question was how
> to add something like a dot parser, which you advocated, to a system
> without making that a special-purpose hook.
>
> Your outer interpreter has a CLASSHOOK before, and a CUNDEFINED after the
> traditional interpreter; however, the actual components of the text
> interpreter like DONUMBER?, C#FLITERAL and C#LITERAL are also all hooks.
> The only thing that isn't a hook in the outer part is FIND, even COMPILE,
> and CEXECUTE are hooks.

Straight Forth won't support polymorphism.

Most likely I will just have a super-simple OOP system such as described here:
https://groups.google.com/forum/#!searchin/comp.lang.forth/is-a%7Csort:relevance/comp.lang.forth/cH-pGXhBCfE/b13O8gNFAwAJ

I might go with a VMT like in Mini-OOF but I doubt it as the method calls are slower.

[Andrew Haley]

Bernd Paysan <be...@net2o.de> wrote:
> Yes, Andrew Haley was worried about "too much empowerment" for the
> programmer. Hey, we are not doing Java here, we do Forth.

Heh. If anyone wants to know what I actually said, they can read it.
:-)

Andrew.

[Stephen Pelc]

On Thu, 29 Sep 2016 02:07:32 -0000 (UTC), Bernd Paysan
<be...@net2o.de> wrote:

>> However, recognisers implicitly suggest a major restructuring of the
>> Forth text interpreter.
>
>It replaces a hard-coded structure by one that is extensible by the user.
>We all have long done that, but in an incompatible way that is sometimes
>not even stable across versions

A very nice post - I've even printed it. Thanks.

[Matthias Trute]

> >You mean, until now you don't take RfDs for new practice seriously,
> >but once the "experimental proposal" sticker is put on it, you will?
>
> There are so many ideas to be tried that, as always, we encourage
> natural selection by asking a "champion" to put a proposal together.

It seems that system providers are not the best people to talk
with when it comes to changes, may they be as little as 1char=1 or
as (only potentially fundamental) as recognizers. The users are the
ones to ask the question "is feature xy useful". System providers
will always follow their customers.

[Ilya Tarasov]

> I see recognizers as being the wedge that splits the Forth community into two camps --- Forth-200x for those who want recognizers --- Straight Forth for those who just want a straight Forth.

It seems like theory of compilation becomes more popular and gets over the edge where has been found by forthers. This is absolutely useless thing to discuss recognizers as a part of standard. It is just a kind of application which can be written on Forth.