String processing, for example

[Andy Valencia]

So if you wanted a "descendent of Forth" which fit the hand nicely in the
current compute environment, how about strings? I thought of the toy problem
of counting the number of words in the diction with an "a" in them:

>>> f = open("/usr/share/dict/words", "r")
>>> na = 0
>>> for l in f:
... if "a" in l:
... na += 1
...
>>> print na
52849
>>>

(Python, obviously, and it took .13 CPU seconds on an arm64
device, the RockPRO64. The file is 102,401 lines.)

I typed it in just on the fly, no IDE or any dev references needed, and got
it right on the first go--not too surprising.

It could go shorter:

>>> print len([l for l in f if ("a" in l)])
52849
>>>

And--I was surprised--using the same amount of CPU.

Forth as hand written assembly code for a stack machine will never touch
this. I've tried many times to find the language which brings higher level
of expressiveness. Failed every time so far.

Andy Valencia
Home page: https://www.vsta.org/andy/
To contact me: https://www.vsta.org/contact/andy.html

[Paul Rubin]

Andy Valencia <van...@vsta.org> writes:
> Forth as hand written assembly code for a stack machine will never touch
> this. I've tried many times to find the language which brings higher level
> of expressiveness. Failed every time so far.

SQL: SELECT COUNT(word) FROM words WHERE word LIKE '%a%';

I used to disdain SQL since lots of times for large datasets, I could
write a much more practical and performant program by using old
fashioned tools like the Un*x sorting utility. Lately I've become a
convert.

Not long ago I wrote a Python script to match up a bunch of urls (that
contained numeric identifiers) with retrieved filenams (that also had
those identifiers). So I had a dictionary mapping urls to id's, and
another dictionary mapping id's to filenames, and IIRC there was another
dictionary that was a reverse mapping. All very standard stuff in
Python. It was a page or so of code, complex enough that I had to fix a
few bugs during testing, but nothing awful. There were a few hundred
filenames involved, not enough for performance or memory consumption to
matter on my laptop.

Then I realized that what I had actually written was a relational JOIN,
so I ripped out my stuff with dictionaries, imported the sqlite3 module,
made an in-memory SQL table, then populated and extracted it with SQL
queries.

That cut the program size in half, made its functionality much more
obvious, and made it simpler to modify (just add ORDER BY to change the
output sorting, etc). But it did that by pulling in a full-blown SQL
implementation. I think that amount of bloat and dependency is counter
to the Forth spirit. I still have to say that it was convenient.

[meff]

On 2021-12-24, Andy Valencia <van...@vsta.org> wrote:
> Forth as hand written assembly code for a stack machine will never touch
> this. I've tried many times to find the language which brings higher level
> of expressiveness. Failed every time so far.

Is the question about whether Forth can be expressive enough to implement string
processing? Is the question about whether a fluent Forth programmer can easily
implement something like the one-line Python example? I'm not clear what this
thread is about.

- meff

[Paul Rubin]

meff <em...@example.com> writes:
> Is the question about whether Forth can be expressive enough to
> implement string processing? Is the question about whether a fluent
> Forth programmer can easily implement something like the one-line
> Python example?

I think it postulates a hypothetical Forth-descended language that lets
users implement the Python example with comparable ease, and asks what
that Forth-descended language would look like. Forth as we know it
today is much more cumbersome.

We have Oforth, 8th, Factor, and maybe some others as Forth-descended
languages that might fit the bill. These all have their aficionados.
I'm personally not that enthused about any of them, since the Forth
features they preserve (RPN and visible stacks) are imho not that
helpful once the language implementation is that complicated. But
that's just me.

For an article on a distantly comparable situation with Lisp, see:

http://www.cs.kent.ac.uk/people/staff/dat/miranda/wadler87.pdf

[Andy Valencia]

Paul Rubin <no.e...@nospam.invalid> writes:
> (solve w. SQL)

> That cut the program size in half, made its functionality much more
> obvious, and made it simpler to modify (just add ORDER BY to change the
> output sorting, etc). But it did that by pulling in a full-blown SQL
> implementation. I think that amount of bloat and dependency is counter
> to the Forth spirit. I still have to say that it was convenient.

And I have to say, I've used sqlite3 many, many times. It's not svelte, but
in bang/buck for database functionality, it's the best bargain going.

[Nickolay Kolchin]

On Saturday, December 25, 2021 at 12:01:50 AM UTC+3, Andy Valencia wrote:
> So if you wanted a "descendent of Forth" which fit the hand nicely in the
> current compute environment, how about strings? I thought of the toy problem
> of counting the number of words in the diction with an "a" in them:
>
> >>> f = open("/usr/share/dict/words", "r")
> >>> na = 0
> >>> for l in f:
> ... if "a" in l:
> ... na += 1
> ...
> >>> print na
> 52849
> >>>
>
> (Python, obviously, and it took .13 CPU seconds on an arm64
> device, the RockPRO64. The file is 102,401 lines.)
>
> I typed it in just on the fly, no IDE or any dev references needed, and got
> it right on the first go--not too surprising.
>
> It could go shorter:
>
> >>> print len([l for l in f if ("a" in l)])
> 52849
> >>>
>
> And--I was surprised--using the same amount of CPU.
>
> Forth as hand written assembly code for a stack machine will never touch
> this. I've tried many times to find the language which brings higher level
> of expressiveness. Failed every time so far.
>

Factor variant:

0 "/usr/share/dict/words" utf8 file-lines [ CHAR: a swap index [ 1 + ] [ ] if ] each

Probably, this can be written shorter, but I'm not familiar with Factor enough.

[minf...@arcor.de]

That's what had me always kept away from Factor: you have to read it forward
and backwards. Your eyes always have to 'go zig-zag' to read Factor code.
I find its syntax annoying and tiring.

[Nickolay Kolchin]

Yes, I've also noticed that. Maybe it is possible to get used to it after some time,
just like with Forth RPN.

But the very concept of combinators is noteworthy and should be borrowed.
Sometimes they allow you to avoid complicated manipulations with the stack.

[Anton Ertl]

Andy Valencia <van...@vsta.org> writes:
>So if you wanted a "descendent of Forth" which fit the hand nicely in the
>current compute environment, how about strings? I thought of the toy problem
>of counting the number of words in the diction with an "a" in them:
>
>>>> f = open("/usr/share/dict/words", "r")
>>>> na = 0
>>>> for l in f:
>... if "a" in l:
>... na += 1
>...
>>>> print na
>52849
>>>>
>
>(Python, obviously, and it took .13 CPU seconds on an arm64
>device, the RockPRO64. The file is 102,401 lines.)

We don't have /usr/share/dict/words on our Rockpro 64 (nor on the two
other systems I checked). Looking around for it, I did not find it,
but found a pointer to
<http://downloads.sourceforge.net/project/wordlist/SCOWL/2020.12.07/scowl-2020.12.07.tar.gz>

I don't see anything with 102401 lines there, so instead I used
final/english-words.80, which has 139209 lines.

Let's see how this works out (I have disabled the slow cores to get
more reliable timings):

[rockpro64:/tmp/scowl-2020.12.07:76145] time python
Python 2.7.13 (default, Sep 26 2018, 18:42:22)
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open("final/english-words.80","r")

>>> print len([l for l in f if ("a" in l)])

78886
>>>

real 0m23.269s
user 0m0.136s
sys 0m0.024s

Ok, but Python 2 is no longer supported, so let's try Python 3:

[rockpro64:/tmp/scowl-2020.12.07:76146] time python3
Python 3.5.3 (default, Sep 27 2018, 17:25:39)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open("final/english-words.80","r")

>>> print len([l for l in f if ("a" in l)])

File "<stdin>", line 1

print len([l for l in f if ("a" in l)])

^
SyntaxError: invalid syntax

Ok, so fall back to your first variant:

>>> na = 0
>>> for l in f:
... if "a" in l:
... na += 1
...

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2797: invalid continuation byte

Great going, Python!

What would I use for this task?

[rockpro64:/tmp/scowl-2020.12.07:76147] time sh -c "grep a final/english-words.80|wc -l"
78683

real 0m0.092s
user 0m0.048s
sys 0m0.032s

Quite a bit shorter and faster than your short Python 2 version, but
with a different result. Investigation reveals that GNU grep 2.27
apparently silently does not output lines that contain Latin-1
characters when LANG=C.UTF-8, which is even worse than Python 3's
behaviour. This can be fixed with

[rockpro64:/tmp/scowl-2020.12.07:76148] time env LANG=C sh -c "grep a final/english-words.80|wc -l"
78886

real 0m0.077s
user 0m0.064s
sys 0m0.008s

but I guess that the same fix may help Python 3.

Anyway, let's see about Forth:

[rockpro64:/tmp/scowl-2020.12.07:76149] time ~/nfstmp/gforth-arm64/gforth -e "s\" final/english-words.80\" slurp-file :noname 0 >r begin 'a' scan dup while r> 1+ >r #lf scan dup 0= until then 2drop r> ; execute . bye"
78886
real 0m0.105s
user 0m0.044s
sys 0m0.032s

On my first try I forgot the second DUP, but the bug and its fix was
obvious once I saw the stack underflow error message and it worked on
the second try. For this code, gforth is faster than gforth-fast,
because gforth-fast has more startup overhead (especially on Aarch64),
and much of the time is spent in SCAN (which does not benefit much
from the differences between gforth and gforth-fast).

The layout of the program is not so nice in this command-line form, so
here it's with nicer layout:

s" final/english-words.80" slurp-file
:noname
0 >r begin
'a' scan dup while
r> 1+ >r
#lf scan dup 0= until then
2drop r> ;
execute .

Longer than the two python variants, but not by much. Looking at it
again, I would now code it as:

: counts
0 >r begin
'a' scan dup while
r> 1+ >r
#lf scan dup 0= until then
2drop r> ;
s" final/english-words.80" slurp-file counts .

>Forth as hand written assembly code for a stack machine will never touch
>this.

It's not there, but it's close. And concerning compatibility: 'a' did
not work on Gforth 0.6, but 'a did and still does, so by changing the
'a' into 'a, you are compatible back from 0.6 (2003) to the current
development version. SLURP-FILE did not work on Gforth 0.5 (2000), so
to be compatible with that more work would be needed.

>I've tried many times to find the language which brings higher level
>of expressiveness. Failed every time so far.

The Unix shell combined with the shell tools are the language of
choice for this kind of task, as demonstrated here again.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

[minf...@arcor.de]

Once you have quotations in your Forth, eg
https://www.complang.tuwien.ac.at/forth/ansforth-cvs/quotations.txt
combinators shouldn't be too difficult to implement.

[Nickolay Kolchin]

What a great example.

So, i've tried to run it on various forths:

: counts
0 >r begin
'a' scan dup while
r> 1+ >r
#lf scan dup 0=
until then
2drop r>
;

: main-ertl s" /usr/share/dict/words" slurp-file counts . ;

$ lxf include ertl.fth

evaluate: include ertl.fth col: 16
ertl.fth line: 7 col: 8
undefined word #lf
no block file

No luck with LXF. Probably it is not standard conformant enough.

$ sf ertl.fth
/some/path/ertl.fth
8: #lf scan dup 0= >>> #lf ?
Exit? (Yes/No/Edit)

No luck with SwiftForth.

$ vfxlin32 include ertl.fth
VFX Forth 5.20 [build 0749] 2021-05-27 for Linux x86
© MicroProcessor Engineering Ltd, 1998-2021

Including ertl.fth
Err# -13 ERR: Undefined word.
Source: "ertl.fth" on line 8
-> #lf scan dup 0=
^
ERROR on command line

Vfx also refuses to compile this code.

$ i4 include /some/path/ertl.fth
...
Thrown out of `/some/path/ertl.fth'
Error -13 in line #8 at nesting level 1
#lf scan dup 0=
-------^
#lf ?

iForth also refuses to build it.

Well, we also have Forth2012 compatible IKForth

$ ikforth -f ertl.fth
...
Undefined word #lf
at '(TRACKED-INCLUDED)' (H# 2000D89C)
LINE# H# 00000008 #lf scan dup 0=

No, luck.

But this should be smth simple. Probably #lf is just 10. Let's define it and
check.

[undefined] #lf [if] 10 CONSTANT #lf [then]

But it still doesn't compile on any of them...

It is fun to see how core "Standards Team Member" uses THREE non-standard words
in a simple example.

[Nickolay Kolchin]

This is a bit more complex. In Forth quotations are just :NONAME inside colon
definition. I.e. they are compiled and there is no way except MARKER to remove
them. In Factor (AFAIK) they are garbage collected.

This prevents us from using RTCG in Forth. Eventually we will run out of
memory. Consider this example.

: GEN [: POSTPONE LIT ;] ; \ simple demo
: TTT 0 0 1000000 DO I GEN EXECUTE + LOOP ;
TTT

[minf...@arcor.de]

I think I get your meaning, although IMHO your demo just does what it had been told
namely compile an excessive number of literals.

I don't know Factor well enough, but contrary to Forth all data (including numbers)
reside in the heap. The stack just contains pointers to data items, which then can be
all kinds of data including code lists = quotations. At least that's how I understand
it (which might be wrong).

Consequently DROP in addition to decrement the stack point has to free assigned data.
ABORT/THROW et cetera have to walk the stack to free all assigned data when needed.
So it's a stack-based garbage collection. MARKER would have to free other data of course,
like VALUES and VARIABLEs (don't know if they exist in Factor).

BTW I use a similar mechanism in my own Linear Algebra package in Forth.

[Jan Coombs]

On Sat, 25 Dec 2021 09:50:35 GMT
an...@mips.complang.tuwien.ac.at (Anton Ertl) wrote:

> [rockpro64:/tmp/scowl-2020.12.07:76146] time python3
> Python 3.5.3 (default, Sep 27 2018, 17:25:39)
> [GCC 6.3.0 20170516] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> f = open("final/english-words.80","r")
> >>> print len([l for l in f if ("a" in l)])
> File "<stdin>", line 1
> print len([l for l in f if ("a" in l)])
> ^
> SyntaxError: invalid syntax

Andy's original and short versions run and produce same result using python 2 or 3 if print statement parens are added:

x86_64:$ time python
Python 2.7.16 (default, Oct 10 2019, 22:02:15)
[GCC 8.3.0] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> f = open("/usr/share/dict/words", "r")
>>> print( len([l for l in f if ("a" in l)]) )
52849

real 0m6.197s
user 0m0.039s
sys 0m0.000s


x86_64:$ time python3
Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> f = open("/usr/share/dict/words", "r")
>>> print( len([l for l in f if ("a" in l)]) )
52849

real 0m4.048s
user 0m0.040s
sys 0m0.011s


For a quick result I would use the more explicit version, which I could also write and modify faster.
Timings were similar:

user 0m0.029s Python 2
user 0m0.050s Python 3

Jan Coombs

[Marcel Hendrix]

On Saturday, December 25, 2021 at 12:51:37 PM UTC+1, Nickolay Kolchin wrote:
[..]
> What a great example.

> iForth also refuses to build it.

Really?

FORTH> : counts ( c-addr u -- n )
<3>[FORTH>] 0 >r
<3>[FORTH>] begin
[3]<3>[FORTH>] 'a' scan dup
[3]<3>[FORTH>] while
[6]<3>[FORTH>] r> 1+ >r
[6]<3>[FORTH>] #10 scan dup 0=
[6]<3>[FORTH>] until then
<3>[FORTH>] 2drop r> ; ok
FORTH> S" Hallo, aarde." counts . 1 ok
FORTH> S\" Hallo, aarde.\nMaan hier." counts . 2 ok
FORTH> S\" Hallo, aarde.\nMa\na\nn hier." counts . 3 ok

-marcel

[Nickolay Kolchin]

That's unfair. You haven't compiled 'main-ertl'.

P.S. But I'm on iForth4, so who knows.

[luser droog]

I thought PostScript might be good at this, and it's broadly construed
as a Forth variant (though its actually history suggests a common ancestor
in the Burrough's Large System architecture). It has a string type, at least.

I had to add a few helper functions to my toolbag.

each-line {src proc}{
src type /stringtype eq { /src src (r) file store } if
{
src 512 string readline {proc}{pop exit} ifelse
} loop
} @func
contains { search { pop pop pop true }{ pop false } ifelse }

And there's a bit of boilerplate to use it from the command line.

$ gsnd -q -dNOSAFER struct2.ps -c "[(struct2.ps){}each-line]{(a)contains{1}{0}ifelse}map{add}reduce = quit"
60
$ gsnd -q -dNOSAFER struct2.ps -c "[(struct2.ps){(a)contains{1}{0}ifelse}each-line]{add}reduce = quit"
60

It can probably be shorter using Anton's idea to suppress the zeros and
take the length of the array. (I also don't have /usr/share/dict/word but
I doubt ghostscript is going to win any speed contests.)

[Hugh Aguilar]

Factor's combinators were the inspiration for my rquotations.
It is necessary for the rquotation to have access to the parent function's local variables
despite the HOF (higher-order-function called by the parent function that executes
the rquotation) having local variables of its own.

The idea that the Paysan-faked quotations, that lack this feature, are a step toward
combinators is total baloney! This is just Forth-200x fantasy, not based on reality at all.

My rquotations only work one level deep, so they are not like Factor combinators that can
stay in scope at any level deep from the parent function and can even stay in scope after the
parent function has exited. Factor combinators are like closures in Lisp, and Common Lisp
is in fact Slava's background. This isn't necessary for my purposes. I am just supporting
general-purpose data-structures. The idea is that the HOF traverses a data-structure and
executes the rquotation that it is given for every node in the data-structure. The programmer
who writes the HOF has to know how the data-structure is implemented internally, but does
not need to know what they payload is. The application programmer who writes the parent
function and the rquotation has to know what the payload is (this is specific to the application)
but does not have to know how the data-structure or HOF are implemented internally.
This is called information hiding!

In regard to your string pattern-matching and replacement program, that would be trivial
given the novice-package that has STRING-STACK.4TH, rquotations, ASSOCIATION.4TH etc..
The novice--package really makes programming easy!

[Nickolay Kolchin]

On Saturday, December 25, 2021 at 11:54:37 PM UTC+3, Hugh Aguilar wrote:
> Factor's combinators were the inspiration for my rquotations.

Can you share a link to your factor vs forth comparison?

[Jali Heinonen]

lauantai 25. joulukuuta 2021 klo 0.49.18 UTC+2 Paul Rubin kirjoitti:

> meff <em...@example.com> writes:
> We have Oforth, 8th, Factor, and maybe some others as Forth-descended
> languages that might fit the bill. These all have their aficionados.
> I'm personally not that enthused about any of them, since the Forth
> features they preserve (RPN and visible stacks) are imho not that
> helpful once the language implementation is that complicated. But
> that's just me.

8th fits the bill nicely as there are high level words for string processing. I would probably write something like:

0 "/usr/share/dict/words" f:slurp
( "a" search null? nip nip if n:1+ then ) s:eachline . cr

or a little slower:

0 "/usr/share/dict/words" f:open-ro
( "a" search null? nip nip if n:1+ then ) f:eachline drop . cr

My dictionary is a lot bigger: 654299 lines. First code runs in: 0m0.341s on my Raspberry PI 4B.

[Ron AARON]

I also took a look, my version:

0 >r "/usr/share/dict/words" f:open-ro
( "a" s:search number? if 1 r+ then 2drop ) f:eachline drop
r> . cr bye

[Hugh Aguilar]

I haven't written any Factor vs Forth comparison.
In regard to how combinators inspired rquotations, that is obvious.
Here are a couple of HOFs that use rquotations similar to Factor:
------------------------------------------------------------------------------------------------------------------
: |iterate { cnt rq -- }
begin cnt while
rq rex
-1 +to cnt repeat ;

: |until { cnt rq | original -- completed }
cnt to original
begin cnt while
rq rex
-1 +to cnt
if original cnt - exit then
repeat
original ;
------------------------------------------------------------------------------------------------------------------

Here are some functions that use |ITERATE internally.
------------------------------------------------------------------------------------------------------------------

: old-up. \ limit n -- \ assumes LIMIT is above N
?do I . loop ;

: up. ( limit n -- ) \ assumes LIMIT is above N
tuck - r[ dup . 1+ ]r |iterate drop ;

: old-down. \ limit n -- \ assumes LIMIT is below N
swap 1+ swap ?do I . -1 +loop ;

: down. ( limit n -- ) \ assumes LIMIT is below N
tuck - negate r[ dup . 1- ]r |iterate drop ;

: design ( val -- |val| sign )
dup 0< if negate -1 exit then
1 ;

: through. ( limit n -- )
tuck - \ -- n count
design -rot \ -- step n |count|
r[ dup . over + ]r |iterate 2drop ;

: through. { limit n | step -- }
limit n - \ -- count
design to step \ -- |count|
r[ n . step +to n ]r |iterate ;
------------------------------------------------------------------------------------------------------------------

For the most part though, I don't think it is a good idea to use rquotations
for iteration. The rquotations are primarily for supporting general-purpose
data-structures.

[Nickolay Kolchin]

On Monday, December 27, 2021 at 6:12:27 AM UTC+3, Hugh Aguilar wrote:
> On Sunday, December 26, 2021 at 2:43:41 AM UTC-7, Nickolay Kolchin wrote:
> > On Saturday, December 25, 2021 at 11:54:37 PM UTC+3, Hugh Aguilar wrote:
> > > Factor's combinators were the inspiration for my rquotations.
> > Can you share a link to your factor vs forth comparison?
> I haven't written any Factor vs Forth comparison.

I thought that was your text: http://www.rosycrew.org/FactorVsForth.pdf

- https://twitter.com/shishini/status/1060677855946915841

[Paul Rubin]

Nickolay Kolchin <nbko...@gmail.com> writes:
> I thought that was your text: http://www.rosycrew.org/FactorVsForth.pdf
> - https://twitter.com/shishini/status/1060677855946915841

The rosycrew url won't connect and the weibo one linked in the tweet
goes to weibo's main page in Chinese. Factorcode.org is also down.

I haven't looked closely at Factor but I think of it (maybe incorrectly)
as Lisp-like but with postfix syntax.

[Nickolay Kolchin]

On Monday, December 27, 2021 at 11:09:26 AM UTC+3, Paul Rubin wrote:
> Nickolay Kolchin <nbko...@gmail.com> writes:
> > I thought that was your text: http://www.rosycrew.org/FactorVsForth.pdf
> > - https://twitter.com/shishini/status/1060677855946915841
> The rosycrew url won't connect and the weibo one linked in the tweet
> goes to weibo's main page in Chinese. Factorcode.org is also down.

Yes, that's why I asked Hugh for pdf. It was written ages ago when Factor
was different, but still can be interesting as a "die-hard" forther view on
a different stack-based language.

>
> I haven't looked closely at Factor but I think of it (maybe incorrectly)
> as Lisp-like but with postfix syntax.

And I thought about APL with RPN notation. :)

[minf...@arcor.de]

Really? See a tiny APL in Forth:
https://github.com/chmykh/apl-life

File apl-life-anno.fs explains it nicely.

[Nickolay Kolchin]

That's just brilliant!

[Hans Bezemer]

On Friday, December 24, 2021 at 10:01:50 PM UTC+1, Andy Valencia wrote:

Oh, c'mon - gimme a break and stop whining:

s" /usr/share/dict/words" input open dup use
0 begin
refill
while
0 parse bounds ?do i c@ char a = if 1+ leave then loop
repeat close

$ time pp4th -x words.4th
52849
real 0m0,019s
user 0m0,019s
sys 0m0,000s
$ time python3 words.py
52849

real 0m0,026s
user 0m0,026s
sys 0m0,000s
$

It's faster - and I don't have to deal with an ever complaining Python interpreter. Barf!

Hans Bezemer

[Ulrich Hoffmann]

Anton Ertl schrieb am Samstag, 25. Dezember 2021 um 12:06:38 UTC+1:

> [rockpro64:/tmp/scowl-2020.12.07:76147] time sh -c "grep a final/english-words.80|wc -l"
> 78683

> The Unix shell combined with the shell tools are the language of
> choice for this kind of task, as demonstrated here again.

Right, and those tools get improved even in 202x. A faster version of grep ist ripgrep [1]

/usr/local/opt/ripgrep a /usr/share/dict/words | wc -l

is a log faster than the standard grep on my system.

Andy Valencia writes:
> >I've tried many times to find the language which brings higher level
> >of expressiveness. Failed every time so far.

Just for the records, the awk version looks like this:

awk '/a/ {n+=1} END {print n}' /usr/share/dict/words

Andy, what kind of improved expressiveness you are looking for? (Serious question).

Greetings,
Ulrich

[1] https://github.com/BurntSushi/ripgrep

[Hans Bezemer]

On Monday, December 27, 2021 at 12:57:55 PM UTC+1, Hans Bezemer wrote:
> On Friday, December 24, 2021 at 10:01:50 PM UTC+1, Andy Valencia wrote:

Good chance I'd not even write custom code for it: I'd probably let a template take care of it:

aka refill read-file
: usage abort" aword <file.txt> <count.txt>" ;
: preprocess 0 ;
: postprocess . cr ;
: process 0 parse bounds ?do i c@ char a = if 1+ leave then loop ;
include lib/convert.4th

$ pp4th -x aword.4th
aword <file.txt> <count.txt>
$ pp4th -x aword.4th /usr/share/dict/words count.txt
$ cat count.txt
52849
$

Hans Bezemer

[Howerd Oakford]

Hi minf,

Thanks for posting this :-)
I just love it!!!!

Cheers, a belated merry Christmas and Happy New Year,
Howerd

[Hugh Aguilar]

I apparently wrote that a decade ago and forgot about it.
The rosycrew.org website has been defunct for at least 10 years.
I haven't looked at Factor in that much time.

I don't recommend using rquotations for iteration as shown above with my
|ITERATE and |UNTIL --- this doesn't really give you any functionality that you
don't have with ordinary iteration (although |ITERATE is much easier to use
than DO loops especially when you have a negative step value).
The reason I don't recommend rquotations for iteration is that you can't
EXIT out of the rquotation and exit out of the HOF. I suppose that I could
upgrade the rquotations to allow this, but I don't think it is worth the complication.

The purpose of the rquotations is to support general-purpose data-structures.
The idea is to have a HOF that traverses the data-structure executing the
rquotation for every node. The rquotation communicates information back to
the parent function that called the HOF by modifying the parent function's
local variables.

In the novice-package I had pseudo-quotations, (called "touchers" in my documentation
because I didn't know the word "pseudo-quotation" at the time. These were just colon words.
I expected the HOF to not have any data on the data-stack at the time that the
pseudo-quotation was executed. This is the pseudo-quotation could communicate information
back to the parent function that called the HOF by modifying the parent function's
data on the data-stack (also, the pseudo-quotation could communicate information
back to the parent function in global variables). This is a very kludgy work-around for
ANS-Forth's lack of rquotations that allow information to be communicated back to the
parent function via the parent function's local variables. When I wrote the rquotations
(an upgrade on HumptyDumpty's prototype that lacked the crucial feature) this really
made pseudo-quotations obsolete. Rquotations are a major break-through for Forth.

The Forth-200x committee has standardized the Paysan-faked quotations that are just
syntactic sugar for pseudo-quotations (they use :NONAME rather than colon to provide
a cute but fake quotation-like syntax, but they lack the functionality of rquotations).
The Forth-200x committee deserves to be completely discredited and disbanded
for standardizing the Paysan-faked quotations --- this is like spray-painting
a brick with gold paint, and then claiming that it is a solid-gold brick.
This is disgraceful fakery! The Forth-200x committee don't deserve to be forgiven.

[minf...@arcor.de]

Thanks Howerd

there is also a rather strange (at least to me) mixed APL+Forth system:
http://www.cosy.com/language/
with Bob Armstrong as its most avid promoter.
My impression is that it is more of a notebook than of a calculation tool.

But you probably know this already.

Happy New Year!
Andreas

[Doug Hoffman]

On 12/24/21 3:49 PM, Andy Valencia wrote:

[snip]

> Forth as hand written assembly code for a stack machine will never touch

> this. I've tried many times to find the language which brings higher level

> of expressiveness. Failed every time so far.

With Forth it often seems to come down to library code. Granted, each
Forther tends to write their own.

Using my string and file library code(ANS compatible):
https://github.com/DouglasBHoffman/FMS2

include FMS2VT.f
include string.f
include file.f

s" english-words.80" file f
: main
0 [: 'a' swap :chsearch if 1+ then ;] f :apply . ;

\ => 78886 0.290 seconds

-Doug

[Andy Valencia]

Ulrich Hoffmann <ulrich.e...@gmail.com> writes:
> Andy, what kind of improved expressiveness you are looking for? (Serious
> question).

Running with our /usr/share/dict/words a little further:

wcounts = {}
for l in open("/usr/share/dict/words", "r"):
l = l.strip().lower()
wcounts[l] = wcounts.get(l, 0) + 1

So take each word, strip whitespace and normalize to lower case,
and run a count of the number of times it's in the file. For my
copy of "words", it looks like there are 1,620 words which are in there
with more than one case--100,781 entries out of 102,401 lines.

I could time it, but it's well under a second, and well above 99%
of the time an interpreted language goes fast enough that performance
is not of any interest at all. (I understand that, say, a data scientist
would not say this.)

I could write this in Forth, but I don't want to. It would be tedious
and take a LOT more time than these four lines of Python. What would
a Forth-inspired language look like where I would _rather_ write
it in this new language?

[minf...@arcor.de]

You know that standard Forth defines only 8 words within the String wordset.
It may perhaps be sufficient for number crunching or control applications, but
it is way too little for string or text processing.

IOW comparing Python vs Forth is like comparing apples "batteries included"
with pears. So before making comparisons, first write your required
string processing functions (or download some Forth 'library').

Of course you know all this, but my point is that you don't need a
Forth-inspired new language, just more powerful wordsets. IMHO it would
also require an extended Memory wordset for managing heap objects
like dynamic strings of variable length.

[Ron AARON]

Your example doesn't print out the results, which this 8th code does:

{} "/usr/share/dict/words" f:slurp
( s:trim s:lc true m:[]! ) s:eachline
( nip array? nip ) m:filter
m:len . cr bye

As well as yours, 1620 duplicates (determined in 'real' of 0.087 seconds
on my machine).

[Paul Rubin]

"minf...@arcor.de" <minf...@arcor.de> writes:
> Of course you know all this, but my point is that you don't need a
> Forth-inspired new language, just more powerful wordsets. IMHO it would
> also require an extended Memory wordset for managing heap objects
> like dynamic strings of variable length.

You also want: 1) memory safety and lots of error checking which (for
Python-style languages) includes runtime type tags. That catches a lot
of bugs early, and when there is an error, gives a stack dump that
identifies the offending source code line. You can break to a debugger
there, though Python's builtins for this aren't so great compared to
Lisp systems of yore. So that speeds up your debugging a lot.

2) garbage collection or other automatic storage management. This stops
a lot of bugs and headaches from happening in the first place.

In the end, you end up with something like 8th or Factor. That's very
nice, but the main thing you've kept from Forth is RPN, and it's not
clear to me that this gains you anything significant by then.

[Doug Hoffman]

My Forth library code is heavily oop-centric. You are right about the
error checking and runtime type tags, which I use extensively, being
useful.

I manage to get by without automatic garbage collection. I am aware of
the benefits but one can use manual gc and it works out fine as long as
one is careful to not do something like grabbing an address of a
substring (and expecting that address to be valid after the string is
freed) instead of copying the substring.

In the end I still have Forth. ;-)

Oforth is a nice alternative to Factor, though Factor has a larger and
more complete library. Oforth feels a lot more like Forth, but
unfortunately it seems Franck Bensusan is no longer supporting it. Not
sure. For example I can not run Oforth on the 64-bit MacOS that I use.
Oforth incorporated automatic gc which I have to admit is nice. Franck
has supplied the complete Oforth source, in C, so anyone with the
motivation could theoretically pick up where Franck left off.

RPN in and of itself gives one the benefits of parenthesis-free
calculation and implicit parameter passing within definitions and
between definitions. That is a lot of benefit IMO.

-Doug

[Andy Valencia]

"minf...@arcor.de" <minf...@arcor.de> writes:
> ... but my point is that you don't need a

> Forth-inspired new language, just more powerful wordsets. IMHO it would
> also require an extended Memory wordset for managing heap objects
> like dynamic strings of variable length.

I've thought this at various points. My challenge to you is to posit those
words, then write this code using such words.

I've tried my hand at this, and never thought my results were even as good as
the Python version.

[dxforth]

On 29/12/2021 01:44, Andy Valencia wrote:
> ...

> I could write this in Forth, but I don't want to. It would be tedious
> and take a LOT more time than these four lines of Python.

"We choose to go to the Moon [...] and do the other things, not because
they are easy, but because they are hard; because that goal will serve
to organize and measure the best of our energies and skills"

The de-skilling of people that has occurred over the last 40 years because
subsequent govts chose the easy way, has made us all dumb and vulnerable.

[Nickolay Kolchin]

Nah, that's not proper sentence. This is "Our grandfathers lived badly, our fathers lived badly, we live badly -- why should our children live well?"

[dxforth]

'By the year AD 802,701, humanity had evolved into two separate species.
The Eloi live a banal life of ease on the surface of the Earth while the
Morlocks live underground, tending machinery and providing food, clothing,
and inventory for the Eloi.' - Wikipedia

[Paul Rubin]

dxforth <dxf...@gmail.com> writes:
> "We choose to go to the Moon ... The de-skilling of people that has

> occurred over the last 40 years because subsequent govts chose the
> easy way, has made us all dumb and vulnerable.

Forthers went to the moon in the 1960s and they are still going to the
moon. Question is what languages interplanetary and interstellar
missions will use.

[dxforth]

Question is who will write those languages - users who find everything
too hard?

[Nickolay Kolchin]

Yeah, the keepers of sacred knowledge who make simple things in the complex
way. Finding words with letter 'a' -- is not rocket science.

There is an anecdote on the topic, btw:

During first human flights to space a very complex problem raised. Standard
ink pens don't work in space, due to zero gravity. So NASA spend several
dozen of millions of dollars and developed ink pen that can work in space
conditions. Russians just used a pencil.

[Anton Ertl]

Paul Rubin <no.e...@nospam.invalid> writes:
>Forthers went to the moon in the 1960s

No. "AGC software was written in AGC assembly language"
<https://en.wikipedia.org/wiki/Apollo_Guidance_Computer#Software>.

>Question is what languages interplanetary and interstellar
>missions will use.

The ACE and Philae use (used) the RTX 2010, so I expect that the are
programmed in Forth.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

[Marcel Hendrix]

On Wednesday, December 29, 2021 at 4:39:05 AM UTC+1, dxforth wrote:
[..]

> 'By the year AD 802,701, humanity had evolved into two separate species.
> The Eloi live a banal life of ease on the surface of the Earth while the
> Morlocks live underground, tending machinery and providing food, clothing,
> and inventory for the Eloi.' - Wikipedia

My, how time flies.

-marcel

[Marcel Hendrix]

That is a hardware engineering question. Are any of the current
crop of popular languages fit for running on rad-hard processors?
And if they can in principle, how would they rate performance- and
productivity-wise compared to a professional Forth?

-marcel

[dxforth]

So it happened sooner than H.G. Wells predicted.

[Nickolay Kolchin]

1. There are RAD protected Cortex-M/A CPUs. So, all popular languages
run just fine on them.
2. Space-X (AFAIK) doesn't use RAD protected CPU at all. They simply put
intel and pray on triple redundancy.

[S Jack]

On Tuesday, December 28, 2021 at 9:39:05 PM UTC-6, dxforth wrote:
> 'By the year AD 802,701, humanity had evolved into two separate species.
> The Eloi live a banal life of ease on the surface of the Earth while the
> Morlocks live underground, tending machinery and providing food, clothing,
> and inventory for the Eloi.' - Wikipedia

It already had by Well's lifetime; a reflection of the early 20th century.

[Hugh Aguilar]

On Tuesday, December 28, 2021 at 8:07:01 PM UTC-7, dxforth wrote:
> On 29/12/2021 01:44, Andy Valencia wrote:
> > ...
> > I could write this in Forth, but I don't want to. It would be tedious
> > and take a LOT more time than these four lines of Python.

Andy Valencia doesn't want to do this program in Forth because he doesn't know
Forth very well and hence even simple programs seem tedious and difficult to him.
A program such as this is trivial given my novice-package which is ANS-Forth.
ASSOCIATION.4TH can be used to collect all of the distinct words. Each node's key
would be the string and it would have one datum tacked on which would be the count.
STRING-STACK.4TH can be used to parse out the words from the text.
STRING-STACK.4TH has pretty good pattern-matching and sub-string extraction
capability, so it could use a much more sophisticated definition of what a word is
than just a blank-delimited string --- no word-count program has been so crude as to
use blank-delimited strings since the early 1970s --- that is kindergarten-level programming.

> "We choose to go to the Moon [...] and do the other things, not because
> they are easy, but because they are hard; because that goal will serve
> to organize and measure the best of our energies and skills"
>
> The de-skilling of people that has occurred over the last 40 years because
> subsequent govts chose the easy way, has made us all dumb and vulnerable.

DXforth says this as if he had skill --- but he doesn't --- he is all talk and no programming
beyond kindergarten-level programming such as his END macro.

[dxforth]

END - easy to remember, easy to use and no shortage of ELSE to eliminate.
It doesn't get better than that. It'll do me as an epitaph.

[Anton Ertl]

Nickolay Kolchin <nbko...@gmail.com> writes:
>So, i've tried to run it on various forths:
>
>: counts
> 0 >r begin
> 'a' scan dup while
> r> 1+ >r
> #lf scan dup 0=3D
> until then
> 2drop r>
>;
>
>: main-ertl s" /usr/share/dict/words" slurp-file counts . ;
>
>$ lxf include ertl.fth
>
>evaluate: include ertl.fth col: 16
>ertl.fth line: 7 col: 8
>undefined word #lf
> no block file
>
>No luck with LXF. Probably it is not standard conformant enough.

What makes you think so?

[...]
>No luck with SwiftForth.
[...]
>Vfx also refuses to compile this code.
[...]
>iForth also refuses to build it.
[And kForth doesn't, either]

>It is fun to see how core "Standards Team Member" uses THREE non-standard w=
>ords
>in a simple example.

I did not have the goal to limit myself to standard words in this
example, so that should be no surprise.

About the three words:

SCAN is pretty common, but scans only for a char, not an xchar, and we
have SEARCH as the more general replacement. It would be a bit more
cumbersome to use here, though.

#LF can indeed be replaced with #10, but #LF tells you more about the
intent. Should it be standardized? If I used SEARCH for standards
conformance, I would also use S\" \l" or S\" \n" instead of #lf.

SLURP-FILE takes more work to replace. I posted an implementation in
1998 <72v4vm$g6p$1...@reader3.wxs.nl> (interestingly, it was not included
in Gforth 0.5 (2000), but only in 0.6 (2003)), and have mentioned this
word a number of times since then (34 occurences in my Articles file).
Apparently non-Gforth users do not find it as useful as I do,
otherwise I would have expected it to spread to other systems. And of
course, with only one system having it, the chances of standardization
are slim (and nobody has tried).

[Marcel Hendrix]

On Saturday, January 1, 2022 at 7:24:34 PM UTC+1, Anton Ertl wrote:
> Nickolay Kolchin <nbko...@gmail.com> writes:

[..]

> SLURP-FILE takes more work to replace. I posted an implementation in
> 1998 <72v4vm$g6p$1...@reader3.wxs.nl> (interestingly, it was not included
> in Gforth 0.5 (2000), but only in 0.6 (2003)), and have mentioned this
> word a number of times since then (34 occurences in my Articles file).
> Apparently non-Gforth users do not find it as useful as I do,
> otherwise I would have expected it to spread to other systems. And of
> course, with only one system having it, the chances of standardization
> are slim (and nobody has tried).

I have been using it since 2011.

D:\dfwforth\examples\DOMI\domiapi.frt(117): S" json.txt" SLURP-FILE TO posted ; PRIVATE
D:\dfwforth\examples\DOMI\domiapi.frt(450): S" json.txt" SLURP-FILE TO posted ; PRIVATE
D:\dfwforth\examples\DOMI\domiapi.frt(511): S" json.txt" SLURP-FILE TO posted ;
D:\dfwforth\examples\SPICE\ispice\ispice.frt(1832): NETFILE-NAME FILE-EXISTS? IF NETFILE-NAME SLURP-FILE
D:\dfwforth\examples\SPICE\ispice\ispice.frt(1834): NETFILE-NAME FILE-EXISTS? IF NETFILE-NAME SLURP-FILE
D:\dfwforth\examples\SPICE\ispice\ispice.frt(5018): MEASFILE-NAME SLURP-FILE ( -- c-addr u ) DLOCAL measures
D:\dfwforth\examples\SPICE\ispice\old_code\ispice.frt(1759): NETFILE-NAME SLURP-FILE ; PRIVATE
D:\dfwforth\examples\SPICE\ispice\old_code\ispice.frt(4808): MEASFILE-NAME SLURP-FILE ( -- c-addr u ) DLOCAL
D:\dfwforth\examples\SPICE\Mag_Tool\dispxml.frt(212): SLURP-FILE ( c-addr u -- )
D:\dfwforth\examples\SPICE\Mag_Tool\dispxml.frt(301): SLURP-FILE ( c-addr u -- )
D:\dfwforth\examples\SPICE\Mag_Tool\mlibread.frt(41): fname COUNT SLURP-FILE
D:\dfwforth\examples\SPICE\Mag_Tool\mlibread.frt(48): fname COUNT SLURP-FILE
D:\dfwforth\examples\SPICE\Mag_Tool\readmag.frt(839): SLURP-FILE ( c-addr u -- )
Found 13 occurrence(s) in 6 file(s), 7752 ms

Searching for: slurp-file
D:\dfwforth\include\miscutil.frt(25): * LAST-CHANGE : Saturday, April 19, 2014, 15:24, mhx; bug in SLURP-FILE
D:\dfwforth\include\miscutil.frt(33): * LAST CHANGE : Friday, April 08, 2011, 22:55, mhx added SLURP-FILE sxint
D:\dfwforth\include\miscutil.frt(2174): : (SLURP-FILE) ( c-addr1 u1 boffset bsize -- addr2 u2 )
D:\dfwforth\include\miscutil.frt(2178): sz bsize U> DUP IF CR c-addr1 u1 TYPE ENDIF ABORT" (SLURP-FILE)
D:\dfwforth\include\miscutil.frt(2184): : SLURP-FILE ( c-addr1 u1 -- addr2 u2 )
D:\dfwforth\include\miscutil.frt(2185): : SLURP-FILE(1) ( c-addr1 u1 -- addr2 u2 ) safeUU 4 / safeUU 4 /
D:\dfwforth\include\miscutil.frt(2186): : SLURP-FILE(2) ( c-addr1 u1 -- addr2 u2 ) safeUU 2/ safeUU 4 /
D:\dfwforth\include\miscutil.frt(2187): : SLURP-FILE(3) ( c-addr1 u1 -- addr2 u2 ) safeUU 3 4 */ safeUU 4 /
D:\dfwforth\include\miscutil.frt(2196): S" _proctxt_" SLURP-FILE S" USERNAME" SEARCH NIP NIP ;
D:\dfwforth\include\miscutil.frt(2547): CR ." SLURP-FILE \ ( c-addr u -- addr2 u2 ) The location addr2 is transient"
D:\dfwforth\include\miscutil.frt(2548): CR ." SLURP-FILE(1) \ ( c-addr u -- addr2 u2 ) Skips space at HERE (2 slurps),
D:\dfwforth\include\spifsim.frt(1848): SLURP-FILE(3) ( c-addr u -- )
D:\dfwforth\include\spifsim.frt(1886): SLURP-FILE(1)
D:\dfwforth\include\spifsim.frt(2027): SLURP-FILE(2) $add-stream ; PRIVATE
D:\dfwforth\include\spifsim.frt(2072): SLURP-FILE(3) $add-stream
D:\dfwforth\include\spifsim.frt(2580): S" spice.errors" SLURP-FILE ( c-addr u) DLOCAL str
D:\dfwforth\include\spifsim.frt(2584): S" SPF.log" SLURP-FILE ( c-addr u) DLOCAL str
D:\dfwforth\include\spifsim.frt(3125): : ECHO-CIRCUIT ( -- ) S" SPF_net.frt" SLURP-FILE cirfile WRITE-FILE ?FILE ;
D:\dfwforth\include\spifsim.frt(3154): S" SPF.net" SLURP-FILE(3) ( c-addr u) -TRAILING-WS
D:\dfwforth\include\syssim.frt(619): prog NETFILE-NAME SLURP-FILE TO $handle
Found 25 occurrence(s) in 3 file(s), 1999 ms

-marcel

[Anton Ertl]

Nickolay Kolchin <nbko...@gmail.com> writes:
>On Saturday, December 25, 2021 at 2:25:22 PM UTC+3, minf...@arcor.de wrote:
>> Nickolay Kolchin schrieb am Samstag, 25. Dezember 2021 um 11:45:19 UTC+1:
>> > But the very concept of combinators is noteworthy and should be borrowed.
>> > Sometimes they allow you to avoid complicated manipulations with the stack.
>> Once you have quotations in your Forth, eg
>> https://www.complang.tuwien.ac.at/forth/ansforth-cvs/quotations.txt
>> combinators shouldn't be too difficult to implement.
>
>This is a bit more complex. In Forth quotations are just :NONAME inside colon
>definition. I.e. they are compiled and there is no way except MARKER to remove
>them.

There is also no need to remove them.

>This prevents us from using RTCG in Forth.

[RTCG=run-time code generation] It doesn't. But RTCG has little to do
with quotations.

>Eventually we will run out of
>memory.

Eventually we are all dead. In practice, it depends on the use case.

The parser generator Gray generates the parser as Forth code at its
run-time. You don't want to garbage-collect it, because it is alive
while there may still be input to parse (and even if there is no
longer any input, I have my doubts that a garbage collector could
determine that; and in any case, the garbage collector would usually
be larger than the generated parser). Gray also leaves some unused
data in the dictionary that could in theory be garbage collected. In
practice, all the data it leaves in the dictionary (including live
data) is only a few KB in size, so again a garbage-collector would not
pay for itself. Eliminating most of the parser generator once the
parser has been generated would be more fruitful.

Anyway, back to combinators or higher-order words. Quotations have no
associated data (that's why no memory management is necessary), so you
have to keep any data on the stack that you want to deal with. This
is possible, but it becomes pretty hard to understand when you have to
deal with multiple levels of higher-order words.

So one idea is to make locals of an outer word visible in a quotation.
Humpty-dumpty's rquotations and Hugh Aguilar's rquotations (which are
somewhat different from humpty-dumpty's) provide a limited way to do
that.

Together with Bernd Paysan, I explored a more general path to doing
that, but eventually we ended up with explicit flat closures: You
provide the data you want associated with the closure explicitly
instead of having nested scoping of locals. You can read all about
it, including using quotations and passing stuff on the stack, in our
EuroForth 2018 paper:

@InProceedings{ertl&paysan18,
author = {M. Anton Ertl and Bernd Paysan},
title = {Closures --- the {Forth} way},
crossref = {euroforth18},
pages = {17--30},
url = {http://www.complang.tuwien.ac.at/papers/ertl%26paysan.pdf},
url2 = {http://www.euroforth.org/ef18/papers/ertl.pdf},
slides-url = {http://www.euroforth.org/ef18/papers/ertl-slides.pdf},
video = {https://wiki.forth-ev.de/doku.php/events:ef2018:closures},
OPTnote = {refereed},
abstract = {In Forth 200x, a quotation cannot access a local
defined outside it, and therefore cannot be
parameterized in the definition that produces its
execution token. We present Forth closures; they
lift this restriction with minimal implementation
complexity. They are based on passing parameters on
the stack when producing the execution token. The
programmer has to explicitly manage the memory of
the closure. We show a number of usage examples.
We also present the current implementation, which
takes 109~source lines of code (including some extra
features). The programmer can mechanically convert
lexical scoping (accessing a local defined outside)
into code using our closures, by applying assignment
conversion and flat-closure conversion. The result
can do everything one expects from closures,
including passing Knuth's man-or-boy test and living
beyond the end of their enclosing definitions.}
}

@Proceedings{euroforth18,
title = {34th EuroForth Conference},
booktitle = {34th EuroForth Conference},
year = {2018},
key = {EuroForth'18},
url = {http://www.euroforth.org/ef18/papers/proceedings.pdf}

[Anton Ertl]

luser droog <luser...@gmail.com> writes:
>I thought PostScript might be good at this, and it's broadly construed
>as a Forth variant (though its actually history suggests a common ancestor
>in the Burrough's Large System architecture). It has a string type, at least.
>
>I had to add a few helper functions to my toolbag.
>
> each-line {src proc}{
> src type /stringtype eq { /src src (r) file store } if
> {
> src 512 string readline {proc}{pop exit} ifelse
> } loop
> } @func
> contains { search { pop pop pop true }{ pop false } ifelse }
>
>And there's a bit of boilerplate to use it from the command line.
>
>$ gsnd -q -dNOSAFER struct2.ps -c "[(struct2.ps){}each-line]{(a)contains{1}{0}ifelse}map{add}reduce = quit"

Putting the stuff above in a file struct2.ps, and then performing this
command line, I get:

Error: /undefined in each-line

There seems to be quite a bit missing.

[dxforth]

On 2/01/2022 05:46, Marcel Hendrix wrote:
> On Saturday, January 1, 2022 at 7:24:34 PM UTC+1, Anton Ertl wrote:
>> Nickolay Kolchin <nbko...@gmail.com> writes:
> [..]
>> SLURP-FILE takes more work to replace. I posted an implementation in
>> 1998 <72v4vm$g6p$1...@reader3.wxs.nl> (interestingly, it was not included
>> in Gforth 0.5 (2000), but only in 0.6 (2003)), and have mentioned this
>> word a number of times since then (34 occurences in my Articles file).
>> Apparently non-Gforth users do not find it as useful as I do,
>> otherwise I would have expected it to spread to other systems. And of
>> course, with only one system having it, the chances of standardization
>> are slim (and nobody has tried).
>
> I have been using it since 2011.
>

MPE has something similar 'FileHacks.fth' which includes the notice:

Copyright (c) 2003
Wil Baden

Though 'SLURP' is referenced on Baden's former website it doesn't appear to
have been included. Wil had a propensity for choosing quirky names which
probably didn't help. While SLURP et al suffices for a quick hack, it's
still quite bare-bones e.g. no printing of filename on error.

[Nickolay Kolchin]

On Tuesday, December 28, 2021 at 6:39:51 PM UTC+3, minf...@arcor.de wrote:
>
> IOW comparing Python vs Forth is like comparing apples "batteries included"
> with pears. So before making comparisons, first write your required
> string processing functions (or download some Forth 'library').
>

This applies to comparing Forth with any language nowadays. You can replace
Python with Go, Rust, Ruby, Elm, even Idris. They all come with basic standard
library.

For Forth, lack of standard library is the obvious flaw.

[Nickolay Kolchin]

On Saturday, January 1, 2022 at 9:24:34 PM UTC+3, Anton Ertl wrote:
> Nickolay Kolchin <nbko...@gmail.com> writes:
>

> >It is fun to see how core "Standards Team Member" uses THREE non-standard

> >words in a simple example.

>
> I did not have the goal to limit myself to standard words in this
> example, so that should be no surprise.

Cause the standard is so bad that it is hard to write even such simple
example in it. Right?

>
> SCAN is pretty common, but scans only for a char, not an xchar, and we
> have SEARCH as the more general replacement. It would be a bit more
> cumbersome to use here, though.

And is not part of standard, despite common use for at least 50 years...

[Nickolay Kolchin]

You should probably update iForth trial to some more recent version.

iForth version 4.0.631, generated 00:53:30, December 26, 2010.

FORTH> s" test" slurp-file
Error -13
s" test" slurp-file
------------------^
slurp-file ?

[dxforth]

A standard library is a lot of work just to be able to say: 'Look, Forth
doesn't stack up'. Why else compare it with other languages if not to find
reason not to use it? I certainly didn't compare languages and then choose
Forth. All I knew was Forth was unlike anything I'd seen and something I
could get my hands into. And that was good enough for me.

[Nickolay Kolchin]

So, the Forth will die with you and you are ok with it?

Cause currently Forth is even out of Tiobe top 100 languages.

[Marcel Hendrix]

On Sunday, January 2, 2022 at 6:37:37 AM UTC+1, Nickolay Kolchin wrote:
> On Saturday, January 1, 2022 at 9:46:45 PM UTC+3, Marcel Hendrix wrote:
> > On Saturday, January 1, 2022 at 7:24:34 PM UTC+1, Anton Ertl wrote:

[..]

> > I have been using it since 2011.

[..]

> You should probably update iForth trial to some more recent version.
>
> iForth version 4.0.631, generated 00:53:30, December 26, 2010.
>
> FORTH> s" test" slurp-file
> Error -13
> s" test" slurp-file
> ------------------^
> slurp-file ?

-marcel

[Anton Ertl]

Paul Rubin <no.e...@nospam.invalid> writes:
>I haven't looked closely at Factor but I think of it (maybe incorrectly)
>as Lisp-like but with postfix syntax.

With that description I think of Postscript. Factor has static type
checking (plus, I think, dynamic typechecking), is closer to Forth in
syntax, and higher-order words are more idiomatic in Factor than in
Forth or Postscript.

[Anton Ertl]

Andy Valencia <van...@vsta.org> writes:
>So take each word, strip whitespace and normalize to lower case,
>and run a count of the number of times it's in the file.

Sounds like Ben Hoyt's example
<https://benhoyt.com/writings/count-words/>, which has been discussed
here at length (but with a focus on performance). You can find my
first take on <2021Mar1...@mips.complang.tuwien.ac.at>.

>I could write this in Forth, but I don't want to. It would be tedious
>and take a LOT more time than these four lines of Python.

Given that you use Python rather than the shell tools, you apparently
have no problem spending a LOT more time than necessary:-)

sed 's/ //g' /usr/share/dict/words|tr '[:upper:]' '[:lower:]'|sort|uniq -c

Untested, because you still have not told us where to get
/usr/share/dict/words.

[Stephen Pelc]

On 1 Jan 2022 at 19:02:42 CET, "Anton Ertl" <Anton Ertl> wrote:

> What makes you think so?
>
> [...]
>> No luck with SwiftForth.
> [...]
>> Vfx also refuses to compile this code.
> [...]
>> iForth also refuses to build it.
> [And kForth doesn't, either]
>
>> It is fun to see how core "Standards Team Member" uses THREE non-standard w=
>> ords
>> in a simple example.
>
> I did not have the goal to limit myself to standard words in this
> example, so that should be no surprise.

The file word SLURP-FILE comes from Wil Baden's Toolbox. An equivalent
set with deliberately facetious word names can be fond in the VFX file:
Lib/FileHacks.fth
This file has been around since 2011 or so.

If these words were standardised, I would probably add them to VFX, but
the current set use global data, are not thread safe and so on.

Stephen

--
Stephen Pelc, ste...@vfxforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, +44 (0)78 0390 3612, +34 649 662 974
http://www.mpeforth.com - free VFX Forth downloads

[S Jack]

On Sunday, January 2, 2022 at 12:58:00 AM UTC-6, Nickolay Kolchin wrote:
> So, the Forth will die with you and you are ok with it?

i. Simple text editor, Runoff when needed, Not word processor
i. Macro assembler, ultimate programming experience
i. Command line with Midnight commander Not GUI menus
i. HTML,XML but No javascript

Won't convince anyone of the above. So be it; just do
what works for me while I can. Eccles 2:16 3:22

--
me

[Andy Valencia]

an...@mips.complang.tuwien.ac.at (Anton Ertl) writes:
> >So take each word, strip whitespace and normalize to lower case,
> >and run a count of the number of times it's in the file.
> Sounds like Ben Hoyt's example
> <https://benhoyt.com/writings/count-words/>, which has been discussed
> here at length (but with a focus on performance). You can find my
> first take on <2021Mar1...@mips.complang.tuwien.ac.at>.

Would you say the Forth in that web page is representative of how
you'd solve the problem? The after or before one?

> Given that you use Python rather than the shell tools, you apparently
> have no problem spending a LOT more time than necessary:-)

I feel like it was a motivating example within the domain of programming
languages, rather than a challenge to find the shortest/sweetest way
to solve the problem. I think we've reached reductio ad absurdum when
I post it on Usenet and an "anton" algorithm solves it. :)

> Untested, because you still have not told us where to get
> /usr/share/dict/words.

Rather silly that it doesn't just jump out in a search. Your example appears
to work fine, but if it's of any interest:

https://vsta.org/andy/pickup/words.txt

Thanks for the pointer to that article.

Andy Valencia
Home page: https://www.vsta.org/andy/
To contact me: https://www.vsta.org/contact/andy.html

[luser droog]

On Saturday, January 1, 2022 at 12:57:30 PM UTC-6, Anton Ertl wrote:
> luser droog <luser...@gmail.com> writes:
> >I thought PostScript might be good at this, and it's broadly construed
> >as a Forth variant (though its actually history suggests a common ancestor
> >in the Burrough's Large System architecture). It has a string type, at least.
> >
> >I had to add a few helper functions to my toolbag.
> >
> > each-line {src proc}{
> > src type /stringtype eq { /src src (r) file store } if
> > {
> > src 512 string readline {proc}{pop exit} ifelse
> > } loop
> > } @func
> > contains { search { pop pop pop true }{ pop false } ifelse }
> >
> >And there's a bit of boilerplate to use it from the command line.
> >
> >$ gsnd -q -dNOSAFER struct2.ps -c "[(struct2.ps){}each-line]{(a)contains{1}{0}ifelse}map{add}reduce = quit"
>
> Putting the stuff above in a file struct2.ps, and then performing this
> command line, I get:
>
> Error: /undefined in each-line
>
> There seems to be quite a bit missing.
> - anton

Apologies. I was using my own syntax extension, which would've been polite to mention.
The extension is described here:
https://codereview.stackexchange.com/questions/193520/an-enhanced-syntax-for-defining-functions-in-postscript

Alternatively, here's a complete example of what I was bragging about in standard PostScript (Level 2).

$ cat toolbag.ps
%!
<<
/each-line { 1 dict begin {proc src}{exch def}forall

src type /stringtype eq { /src src (r) file store } if

{ src 1024 string readline {proc}{pop exit} ifelse } loop
end }
/contains { search { pop pop pop true }{ pop false } ifelse }
/map { 1 index xcheck 3 1 roll [ 3 1 roll forall ] exch {cvx} if }
/reduce { exch dup first exch rest 3 -1 roll forall }
/first { 0 get }
/rest { 1 1 index length 1 sub getinterval }
>> begin

$ gsnd -q -dNOSAFER toolbag.ps -c '[(toolbag.ps){(a)contains{1}{0}ifelse}each-line]{add}reduce = quit'
6
$ gsnd -q -dNOSAFER toolbag.ps -c '[(toolbag.ps){(a)contains{1}if}each-line]length = quit'
6

[Hugh Aguilar]

On Sunday, January 2, 2022 at 8:26:38 AM UTC-7, Stephen Pelc wrote:
> On 1 Jan 2022 at 19:02:42 CET, "Anton Ertl" <Anton Ertl> wrote:
>
> > What makes you think so?
> >
> > [...]
> >> No luck with SwiftForth.
> > [...]
> >> Vfx also refuses to compile this code.
> > [...]
> >> iForth also refuses to build it.
> > [And kForth doesn't, either]
> >
> >> It is fun to see how core "Standards Team Member" uses THREE non-standard w=
> >> ords
> >> in a simple example.
> >
> > I did not have the goal to limit myself to standard words in this
> > example, so that should be no surprise.
> The file word SLURP-FILE comes from Wil Baden's Toolbox. An equivalent
> set with deliberately facetious word names can be fond in the VFX file:
> Lib/FileHacks.fth
> This file has been around since 2011 or so.
>
> If these words were standardised, I would probably add them to VFX, but
> the current set use global data, are not thread safe and so on.
>
> Stephen

You don't know how to implement general-purpose data-structures.
Nobody on the Forth-200x committee has ever succeeded in implementing
a linked list.

Here the READ-FILE word reads a file into a linked list.
I assume that SLURP-FILE reads a file into memory somehow, although I haven't
been paying attention to this thread. It is pathetic when the Forth-200x committee
fail at even the most basic programming tasks.
------------------------------------------------------------------------------------------------------
\ ******
\ ****** These are lists of strings.
\ ******

list
w field .line
constant seq

: init-seq ( str node -- node )
init-list >r
hstr r@ .line !
r> ;

: new-seq ( str -- node )
seq alloc
init-seq ;

: <kill-seq> ( node -- )
dup .line @ dealloc
dealloc ;

: kill-seq ( head -- )
each[ <kill-seq> ]each ;

: <clone-seq> ( node -- new-node )
clone-node
dup .line @ hstr over .line ! ;

: clone-seq ( head -- new-head )
nil
swap each[ <clone-seq> link ]each ;

: concrete-str ( str -- new-str )
here swap count ,str ;

: <concrete-seq> ( node -- new-node )
concrete-node
dup .line @ concrete-str
over .line ! ;

: concrete-seq ( head -- new-head )
nil
swap each[ <concrete-seq> link ]each ;

: <read-seq> ( adr cnt -- head )
r/o <open-file> >r \ r: -- file-id
<cstr nil begin \ -- head
cstr 1+ cstr-size 1- r@ read-line abort" *** READ-SEQ failed to read line ***"
while \ -- head cnt
cstr c! \ -- head
cstr new-seq \ -- head node
link repeat drop \ -- head
r> <close-file>
cstr> drop ;

: read-seq ( name -- head )
count <read-seq> ;

: <write-seq> ( head adr cnt -- )
w/o <create-file> swap \ -- file-id head
each[ .line @ count rover write-line abort" *** <WRITE-SEQ> failed to write ***" ]each
<close-file> ;

: write-seq ( head name -- )
count <write-seq> ;

: <sort-seq> ( new-node node -- new-node flag )
.line @ count rover .line @ count compare A>B = ;

: sort-seq ( head -- new-head )
['] <sort-seq> sort-list ;

: <show-seq> ( node -- )
.line @ count type cr ;

: show-seq ( head -- )
cr
['] <show-seq> each ;

\ SHOW-SEQ uses EACH rather than EACH[...]EACH because <SHOW-SEQ> is useful in its own right.

: trim-seq ( head -- new-head ) \ trim nodes of trailing whitespace, and kill any that are entirely whitespace
nil swap each[ \ -- head node \ HEAD is of the new seq and NODE is of the seq we iterate through (called HEAD in the stack picture)
init-list \ prepare node to be linked into new seq
dup .line @ count -trailing \ -- head node adr cnt
nip \ -- head node cnt
dup 0= if
drop <kill-seq> \ -- head
else
over .line @ c! \ -- head node \ store CNT in string as count byte (this doesn't change how much memory is allocated in the heap)
link then \ -- head
]each ;

\ In TRIM-SEQ we don't bother to RESIZE the string to use less memory in the heap. That would not be worth the extra complication, and it would be slow.
\ When the seq is given to KILL-SEQ later on, all of that memory will get recovered at that time.

: <split> { adr cnt delimiter left right | lit-str? -- head }
nil <cstr
adr cnt + adr ?do \ -- head
lit-str? if \ if inside of a literal string
I c@ right = if false to lit-str? \ trip flag when we pass the right bracket
else I c@ char+cstr then \ pass all chars through except the right bracket
else
I c@ delimiter = if \ if we hit the delimiter outside of a literal string
cstr> new-seq link \ -- new-head \ conclude substring and link it
<cstr \ start a new substring
else
I c@ left = if true to lit-str? \ trip flag when we pass the left bracket
else I c@ char+cstr then \ pass all chars through except the left bracket
then
then
loop
cstr> dup c@ if new-seq link \ if last substring has any chars in it, link it in
else drop then ;


\ <SPLIT> splits the string on the delimiter into substrings dumped into a SEQ. The delimiter(s) are removed.
\ Delimiters inside of left/right brackets are passed through. The brackets are not passed through.
\ Use 0 for LEFT and RIGHT if you don't want to have literal strings.

vector split ( adr cnt -- head )

: comma-split ( adr cnt -- head )
[char] , [char] " [char] " <split> ;

' comma-split is-split

\ The most common format that you get from database dumps is comma delimited with quote-mark brackets.
\ If you have something else though, you can use IS-SPLIT to set what you need.

: combine ( head delimiter -- str ) \ combines seq strings into a single delimited string
swap <cstr
each[ .line @ +cstr dup char+cstr ]each drop
1 -cstr \ get rid of the delimiter appended on the last substring
cstr> ;

\ Be careful in regard to the fact that <SPLIT> removes the brackets from literal strings.
\ If you use COMBINE to put the string back together, those brackets will no longer be there.
\ If you <SPLIT> the string again, it will split on any delimiters that were inside of literal strings.
\ As a rule, I use a different delimiter with COMBINE than I had in <SPLIT>. For example, | rather than the comma.
\ The comma is a bad choice considering how common the comma is in data, but the | is rare in data.

: remove-padding ( head -- ) \ removes leading and trailing blank characters from strings in seq
each[ >r
r@ .line @ \ -- str
dup count -leading -trailing \ -- str adr cnt
<hstr> r@ .line ! \ -- str
dealloc \ --
rdrop ]each ;
------------------------------------------------------------------------------------------------------

[Nickolay Kolchin]

On Saturday, January 1, 2022 at 9:53:13 PM UTC+3, Anton Ertl wrote:
> Nickolay Kolchin <nbko...@gmail.com> writes:
> >On Saturday, December 25, 2021 at 2:25:22 PM UTC+3, minf...@arcor.de wrote:
> >> Nickolay Kolchin schrieb am Samstag, 25. Dezember 2021 um 11:45:19 UTC+1:
> >> > But the very concept of combinators is noteworthy and should be borrowed.
> >> > Sometimes they allow you to avoid complicated manipulations with the stack.
> >> Once you have quotations in your Forth, eg
> >> https://www.complang.tuwien.ac.at/forth/ansforth-cvs/quotations.txt
> >> combinators shouldn't be too difficult to implement.
> >
> >This is a bit more complex. In Forth quotations are just :NONAME inside colon
> >definition. I.e. they are compiled and there is no way except MARKER to remove
> >them.
>
> There is also no need to remove them.
>
> >This prevents us from using RTCG in Forth.
>
> [RTCG=run-time code generation] It doesn't. But RTCG has little to do
> with quotations.
>

You misunderstood me here. Quotations are an important
building block for RTCG. Imagine that we have a word

: compile-expr ( c-addr u -- xt )

Which takes string expression as input and return address of
quotation which can be used to evaluate expression. I.e.

"cos(x)*sin(y)-1" compile-expr .5e .6e EXECUTE

That all works fine, but there is no way to remove "compiled
expressions" except MARKER.

[Marcel Hendrix]

FORTH> NEEDS -xopg
FORTH> 0-VALUE GVAL a 0-VALUE GVAL b 0-VALUE GVAL c 0-VALUE GVAL x 0-VALUE GVAL w ok
FORTH> : test1 LET a = b*c-3.17e-5/TANH(w)+ABS(x): CR LET. a: ; ok
FORTH> S" LET w = 1.e-3: LET x = -2.5: CR CR test1" EVALUATE


2.4682999894333340377777106878373968247191e+0000 ok
FORTH> FORGET test1 ok
FORTH> test1
Error -13
test1
----^
test1 ?

-marcel

[Nickolay Kolchin]

"This word is obsolescent and is included as a concession to
existing implementations."

[Nickolay Kolchin]

Also I was talking about quotations removal. You can't "forget" them.

[Anton Ertl]

Marcel Hendrix <m...@iae.nl> writes:
>On Saturday, January 1, 2022 at 7:24:34 PM UTC+1, Anton Ertl wrote:
>> Nickolay Kolchin <nbko...@gmail.com> writes:
>[..]
>> SLURP-FILE takes more work to replace. I posted an implementation in
>> 1998 <72v4vm$g6p$1...@reader3.wxs.nl> (interestingly, it was not included
>> in Gforth 0.5 (2000), but only in 0.6 (2003)), and have mentioned this
>> word a number of times since then (34 occurences in my Articles file).
>> Apparently non-Gforth users do not find it as useful as I do,
>> otherwise I would have expected it to spread to other systems. And of
>> course, with only one system having it, the chances of standardization
>> are slim (and nobody has tried).
>
>I have been using it since 2011.

...

>Found 13 occurrence(s) in 6 file(s), 7752 ms

...

>Found 25 occurrence(s) in 3 file(s), 1999 ms

Great! Now we have two systems, standardization is only a proposal
and some stamina away:-)

[Anton Ertl]

Stephen Pelc <ste...@vfxforth.com> writes:
>The file word SLURP-FILE comes from Wil Baden's Toolbox.

Citation needed.

I believe that I came up with it.

I find in my postings <72v4vm$g6p$1...@reader3.wxs.nl> from 1998, where I
give a full implementation of SLURP-FILE, and an answer to Wil Baden's
2001 posting <B7DD15E5.1FC41%neil...@earthlink.net>, where he
mentions his SLURP (and in my answer
<B7DD15E5.1FC41%neil...@earthlink.net> I mentioned that the similar
SLURP-FID was in the Gforth sources at the time).

>An equivalent
>set with deliberately facetious word names can be fond in the VFX file:
> Lib/FileHacks.fth
>This file has been around since 2011 or so.

The file header in the version that came with vfxlin 4.72 says

|Copyright (c) 2003
|Wil Baden
|

|Copyright (c) 2004, 2005
|MicroProcessor Engineering

I don't find SLURP-FILE in this file. I do find:

slurp \ file-id -- addr length
which has a stack effect like Gforth's SLURP-FID

Inhale \ fname flen -- caddr len
which has a stack effect like Gforth's SLURP-FILE

Concerning the name SLURP, it's from (MIT) hacker usage, see
<http://hackersdictionary.com/html/entry/slurp.html>
<https://www.dourish.com/goodies/jargon.html>. I checked a few other
words from FileHacks.fth, but did not find them there.

>If these words were standardised, I would probably add them to VFX, but
>the current set use global data, are not thread safe and so on.

If you look at SLURP-FILE from 1998 <72v4vm$g6p$1...@reader3.wxs.nl>,
there is no global data there, and it looks thread-safe to me.

[Anton Ertl]

Andy Valencia <van...@vsta.org> writes:
>an...@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>> >So take each word, strip whitespace and normalize to lower case,
>> >and run a count of the number of times it's in the file.
>> Sounds like Ben Hoyt's example
>> <https://benhoyt.com/writings/count-words/>, which has been discussed
>> here at length (but with a focus on performance). You can find my
>> first take on <2021Mar1...@mips.complang.tuwien.ac.at>.
>
>Would you say the Forth in that web page is representative of how
>you'd solve the problem?

No, you can find in <2021Mar1...@mips.complang.tuwien.ac.at> how
I solved the problem. There was also a long discussion where many
people (including me) had a go at the problem and discussed various
solutions, including Ben Hoyt's variants, and presented additional
variants.

> https://vsta.org/andy/pickup/words.txt

Thanks.

[Anton Ertl]

luser droog <luser...@gmail.com> writes:
>$ cat toolbag.ps
>%!
><<
> /each-line { 1 dict begin {proc src}{exch def}forall
> src type /stringtype eq { /src src (r) file store } if
> { src 1024 string readline {proc}{pop exit} ifelse } loop
> end }
> /contains { search { pop pop pop true }{ pop false } ifelse }
> /map { 1 index xcheck 3 1 roll [ 3 1 roll forall ] exch {cvx} if }
> /reduce { exch dup first exch rest 3 -1 roll forall }
> /first { 0 get }
> /rest { 1 1 index length 1 sub getinterval }
>>> begin
>
>$ gsnd -q -dNOSAFER toolbag.ps -c '[(toolbag.ps){(a)contains{1}{0}ifelse}each-line]{add}reduce = quit'
>6
>$ gsnd -q -dNOSAFER toolbag.ps -c '[(toolbag.ps){(a)contains{1}if}each-line]length = quit'
>6

Thanks. For those interested in the performance comparison:

[rockpro64:/tmp/scowl-2020.12.07:76226] taskset -c 4,5 perf stat -r20 -e cycles gsnd -q -dNOSAFER ~/forth/toolbag.ps -c '[(final/english-words.80){(a)contains{1}{0}ifelse}each-line]{add}reduce = quit'
78886
...
78886

Performance counter stats for 'gsnd -q -dNOSAFER /nfs/a5/anton/forth/toolbag.ps -c [(final/english-words.80){(a)contains{1}{0}ifelse}each-line]{add}reduce = quit' (20 runs):

1487887895 cycles

0.816152385 seconds time elapsed ( +- 0.30% )

[rockpro64:/tmp/scowl-2020.12.07:76227] taskset -c 4,5 perf stat -r20 -e cycles ~/nfstmp/gforth-arm64/gforth -e "s\" final/english-words.80\" slurp-file :noname 0 >r begin 'a' scan dup while r> 1+ >r #lf scan dup 0= until then 2drop r> ; execute . bye"
78886 78886 78886 78886 78886 78886 78886 78886 78886 78886 78886 78886 78886 78886 78886 78886 78886 78886 78886 78886
Performance counter stats for '/nfs/a5/anton/nfstmp/gforth-arm64/gforth -e s" final/english-words.80" slurp-file :noname 0 >r begin 'a' scan dup while r> 1+ >r #lf scan dup 0= until then 2drop r> ; execute . bye' (20 runs):

48881662 cycles

0.064227552 seconds time elapsed ( +- 5.20% )

[S Jack]

On Monday, January 3, 2022 at 5:52:15 AM UTC-6, Nickolay Kolchin wrote:
> Also I was talking about quotations removal. You can't "forget" them.

I can but I'm non-standard and nobody else cares about building words
that contain a data area.
--
me

[Anton Ertl]

an...@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>Stephen Pelc <ste...@vfxforth.com> writes:
>>The file word SLURP-FILE comes from Wil Baden's Toolbox.
>
>Citation needed.
>
>I believe that I came up with it.
>
>I find in my postings <72v4vm$g6p$1...@reader3.wxs.nl> from 1998, where I

It's <7346e1$c4h$2...@news.tuwien.ac.at> (an answer to the posting
mentioned above).

>give a full implementation of SLURP-FILE, and an answer to Wil Baden's
>2001 posting <B7DD15E5.1FC41%neil...@earthlink.net>, where he
>mentions his SLURP (and in my answer
><B7DD15E5.1FC41%neil...@earthlink.net> I mentioned that the similar

Sorry, I have no message-Id for my answer, it's a followup to that
posting.

>SLURP-FID was in the Gforth sources at the time).
>
>>An equivalent
>>set with deliberately facetious word names can be fond in the VFX file:
>> Lib/FileHacks.fth
>>This file has been around since 2011 or so.
>
>The file header in the version that came with vfxlin 4.72 says
>
>|Copyright (c) 2003
>|Wil Baden
>|
>|Copyright (c) 2004, 2005
>|MicroProcessor Engineering
>
>I don't find SLURP-FILE in this file. I do find:
>
>slurp \ file-id -- addr length
> which has a stack effect like Gforth's SLURP-FID
>
>Inhale \ fname flen -- caddr len
> which has a stack effect like Gforth's SLURP-FILE
>
>Concerning the name SLURP, it's from (MIT) hacker usage, see
><http://hackersdictionary.com/html/entry/slurp.html>
><https://www.dourish.com/goodies/jargon.html>. I checked a few other
>words from FileHacks.fth, but did not find them there.
>
>>If these words were standardised, I would probably add them to VFX, but
>>the current set use global data, are not thread safe and so on.
>
>If you look at SLURP-FILE from 1998 <72v4vm$g6p$1...@reader3.wxs.nl>,

Again, <7346e1$c4h$2...@news.tuwien.ac.at>

[lehs]

fredag 24 december 2021 kl. 22:01:50 UTC+1 skrev Andy Valencia:
> So if you wanted a "descendent of Forth" which fit the hand nicely in the
> current compute environment, how about strings? I thought of the toy problem
> of counting the number of words in the diction with an "a" in them:
>
> >>> f = open("/usr/share/dict/words", "r")
> >>> na = 0
> >>> for l in f:
> ... if "a" in l:
> ... na += 1
> ...
> >>> print na
> 52849
> >>>
>
> (Python, obviously, and it took .13 CPU seconds on an arm64
> device, the RockPRO64. The file is 102,401 lines.)
>
> I typed it in just on the fly, no IDE or any dev references needed, and got
> it right on the first go--not too surprising.
>
> It could go shorter:
>
> >>> print len([l for l in f if ("a" in l)])
> 52849
> >>>
>
> And--I was surprised--using the same amount of CPU.
>
> Forth as hand written assembly code for a stack machine will never touch
> this. I've tried many times to find the language which brings higher level
> of expressiveness. Failed every time so far.

>
> Andy Valencia
> Home page: https://www.vsta.org/andy/
> To contact me: https://www.vsta.org/contact/andy.html

Forth is a processor independent macro assembler and shouldn't be compared to enourmos compilators with professional libraries.

\ text processing

: @bl \ ad n -- ad n flag
over c@ bl = ;

: >lowcase 96 or ;

: intervall \ ad n -- ad+n ad
over + swap ;

: >bl \ a n -- a' n' to next bl
begin @bl 0=
while 1 /string
repeat ;

: bl> \ a n -- a' n' to next non bl
begin @bl
while 1 /string
repeat ;

: nextword \ a n -- a' n' ad nr ?
-trailing bl>
2dup locals| nr ad | >bl
nr over - to nr bl> ad nr ;

: cwc \ a n xt -- m conditional word count
locals| xt |
0 >r
begin nextword xt execute
if r> 1+ >r then
dup 0>
while bl>
repeat 2drop r> ;

create vowels s" aeiouy" s,

: isvowel \ ch -- flag
>lowcase false locals| flag |
vowels count intervall
do dup i c@ >lowcase =
if true to flag leave then
loop drop flag ;

: #vowels \ ad n -- nr
intervall 0 -rot
do i c@ isvowel - loop ;

: test \ ad n -- flag
#vowels 1 = ;

s" Okay, Houston we've had a problem here. This is Houston. Say again, please. Uh, Houston, we've had a problem. We've had a Main B Bus Undervolt." ' test cwc . 10 ok

[Andy Valencia]

an...@mips.complang.tuwien.ac.at (Anton Ertl) writes:
> >Would you say the Forth in that web page is representative of how
> >you'd solve the problem?

> No, you can find in <2021Mar1...@mips.complang.tuwien.ac.at> how
> I solved the problem.

Thanks. I'm looking at this one:
http://www.complang.tuwien.ac.at/anton/count-unique2.fs

> There was also a long discussion where many
> people (including me) had a go at the problem and discussed various
> solutions, including Ben Hoyt's variants, and presented additional
> variants.

I'm not a Google user, but I found an archive at:
https://news.novabbs.com/programming/article-flat.php?id=12344&group=comp.lang.forth

Thanks again.

[dxforth]

There are more excuses in heaven and earth, Horatio,
Than are dreamt of in your philosophy.

[Anton Ertl]

Nickolay Kolchin <nbko...@gmail.com> writes:
>On Saturday, January 1, 2022 at 9:53:13 PM UTC+3, Anton Ertl wrote:
>> Nickolay Kolchin <nbko...@gmail.com> writes:
>> >On Saturday, December 25, 2021 at 2:25:22 PM UTC+3, minf...@arcor.de wrote:
>> >> Nickolay Kolchin schrieb am Samstag, 25. Dezember 2021 um 11:45:19 UTC+1:
>> >> > But the very concept of combinators is noteworthy and should be borrowed.
>> >> > Sometimes they allow you to avoid complicated manipulations with the stack.
>> >> Once you have quotations in your Forth, eg
>> >> https://www.complang.tuwien.ac.at/forth/ansforth-cvs/quotations.txt
>> >> combinators shouldn't be too difficult to implement.
>> >
>> >This is a bit more complex. In Forth quotations are just :NONAME inside colon
>> >definition. I.e. they are compiled and there is no way except MARKER to remove
>> >them.
>>
>> There is also no need to remove them.
>>
>> >This prevents us from using RTCG in Forth.
>>
>> [RTCG=run-time code generation] It doesn't. But RTCG has little to do
>> with quotations.
>>
>
>You misunderstood me here. Quotations are an important
>building block for RTCG. Imagine that we have a word
>
>: compile-expr ( c-addr u -- xt )
>
>Which takes string expression as input and return address of
>quotation which can be used to evaluate expression. I.e.
>
>"cos(x)*sin(y)-1" compile-expr .5e .6e EXECUTE

You don't need quotations for that. :NONAME (Forth-94) is sufficient.

>That all works fine, but there is no way to remove "compiled
>expressions" except MARKER.

Yes. The question is if RTCG-with-removal would be used frequently
enough to justify the complexity of such a removal mechanism. Looking
beyond Forth, there were a number of research papers on RTCG in the
1990s, but it seems to have had little practical impact. So it looks
unlikely that the answer to the question above is "yes".

[Nickolay Kolchin]

":NONAME" can't be used inside colon definition, afaik.

> >That all works fine, but there is no way to remove "compiled
> >expressions" except MARKER.
> Yes. The question is if RTCG-with-removal would be used frequently
> enough to justify the complexity of such a removal mechanism. Looking
> beyond Forth, there were a number of research papers on RTCG in the
> 1990s, but it seems to have had little practical impact. So it looks
> unlikely that the answer to the question above is "yes".

Yes. This is the attitude we expect from Forth200X member. :)

[Marcel Hendrix]

On Wednesday, January 5, 2022 at 8:49:58 PM UTC+1, Nickolay Kolchin wrote:
> On Wednesday, January 5, 2022 at 9:23:38 PM UTC+3, Anton Ertl wrote:
> > Nickolay Kolchin <nbko...@gmail.com> writes:
> > >On Saturday, January 1, 2022 at 9:53:13 PM UTC+3, Anton Ertl wrote:

[..]

> > >You misunderstood me here. Quotations are an important
> > >building block for RTCG. Imagine that we have a word
> > >
> > >: compile-expr ( c-addr u -- xt )
> > >
> > >Which takes string expression as input and return address of
> > >quotation which can be used to evaluate expression. I.e.
> > >
> > >"cos(x)*sin(y)-1" compile-expr .5e .6e EXECUTE
> > You don't need quotations for that. :NONAME (Forth-94) is sufficient.
> ":NONAME" can't be used inside colon definition, afaik.
> > >That all works fine, but there is no way to remove "compiled
> > >expressions" except MARKER.
> > Yes. The question is if RTCG-with-removal would be used frequently
> > enough to justify the complexity of such a removal mechanism. Looking
> > beyond Forth, there were a number of research papers on RTCG in the
> > 1990s, but it seems to have had little practical impact. So it looks
> > unlikely that the answer to the question above is "yes".
> Yes. This is the attitude we expect from Forth200X member. :)

I use RTCG as the main mechanism in my SPICE simulator, and yes,
I want to remove the generated code without having to restart iForth.
This means FORGET, and this FORGET must work nicely with interwoven
vocabularies and wordlists.

I have trouble coming up with reasons to remove generated code in *a
running program*. What would immediately useful examples of that
feature be?

-marcel

[Ruvim]

On 2022-01-03 16:59, Anton Ertl wrote:
> Marcel Hendrix <m...@iae.nl> writes:
>> On Saturday, January 1, 2022 at 7:24:34 PM UTC+1, Anton Ertl wrote:
>>> Nickolay Kolchin <nbko...@gmail.com> writes:
>> [..]
>>> SLURP-FILE takes more work to replace. I posted an implementation in
>>> 1998 <72v4vm$g6p$1...@reader3.wxs.nl> (interestingly, it was not included
>>> in Gforth 0.5 (2000), but only in 0.6 (2003)), and have mentioned this
>>> word a number of times since then (34 occurences in my Articles file).
>>> Apparently non-Gforth users do not find it as useful as I do,
>>> otherwise I would have expected it to spread to other systems. And of
>>> course, with only one system having it, the chances of standardization
>>> are slim (and nobody has tried).
>>
>> I have been using it since 2011.
> ...
>> Found 13 occurrence(s) in 6 file(s), 7752 ms
> ...
>> Found 25 occurrence(s) in 3 file(s), 1999 ms
>
> Great! Now we have two systems, standardization is only a proposal
> and some stamina away:-)

I never faced "SLURP-FILE" before, but I implemented "FILENAME-CONTENT"
with the same interface: https://git.io/JSSah

FILE-CONTENT ( fileid -- sd.content ior )
FILENAME-CONTENT ( sd.filename -- sd.content )
FOR-FILENAME-CONTENT ( i*x sd.filename xt -- j*x )
\ xt ( i*x sd.content -- j*x )

They are only used about ten times in my projects.

--
Ruvim

[Paul Rubin]

Marcel Hendrix <m...@iae.nl> writes:
> I have trouble coming up with reasons to remove generated code in *a
> running program*. What would immediately useful examples of that
> feature be?

It's just like freeing up any other memory that is no longer in use.
Maybe you have a text editor that compiles a regular expression into
code, or maybe a database that compiles a query. Once you have run the
regexp search or db quary, you no longer need the code. In e.g. a Lisp
system, it gets garbage collected. How should Forth deal with it?

[Nickolay Kolchin]

On Saturday, January 1, 2022 at 9:24:34 PM UTC+3, Anton Ertl wrote:
> SLURP-FILE takes more work to replace. I posted an implementation in
> 1998 <72v4vm$g6p$1...@reader3.wxs.nl> (interestingly, it was not included
> in Gforth 0.5 (2000), but only in 0.6 (2003)), and have mentioned this
> word a number of times since then (34 occurences in my Articles file).
> Apparently non-Gforth users do not find it as useful as I do,
> otherwise I would have expected it to spread to other systems. And of
> course, with only one system having it, the chances of standardization
> are slim (and nobody has tried).

I've investigated several "SLURP-FILE" implementations and I'm really
depressed.

This is GForth (from stuff.fs), my comments start with \ !!

: slurp-file ( c-addr1 u1 -- c-addr2 u2 ) \ gforth
\G @var{c-addr1 u1} is the filename, @var{c-addr2 u2} is the file's contents
r/o bin open-file throw >r
\ !! Resource leak here. File not closed. (twice)
r@ file-size throw abort" file too large"
\ !! Resource leak here. File not closed.
dup allocate throw swap
\ !! Resource leak here. File not closed. Memory not freed
2dup r@ read-file throw over <> abort" could not read whole file"
\ !! Resource leak here. Memory not freed.
r> close-file throw ;

This makes gforth "slurp-file" implementation not usable in real
applications.

KForth:

1024 1024 * 64 * value MAX_SLURP \ 64 MB limit

0 ptr slurp_buf
variable slurp_fid
variable slurp_size

: slurp-file ( c-addr1 u1 -- c-addr2 u2)
0 slurp_size !
R/O BIN open-file
if -69 throw
then dup slurp_fid !
file-size
if -66 throw
then
0<> over MAX_SLURP > or \ !! U> should be used here.
if 1 throw \ File too large
then dup slurp_size ! allocate
if -59 throw
then to slurp_buf
0 s>d slurp_fid @ reposition-file
if -73 throw
then slurp_buf slurp_size @ slurp_fid @ read-file
if -70 throw
then slurp_size @ over <>
if 2 throw \ Slurp size and read size do not match
then slurp_buf swap
slurp_fid @ close-file
if -62 throw
then ;

1. Bad practice. Global variables.
2. reposition-file -- useless? flie-size shouldn't modify position.
3. Same resource leaks as in gforth version. In theory user can cleanup
in catch handler, but he must clear globals first. Otherwise he may use
garbage left from previous calls.
4. Minor point: numeric values usage should be discouraged. Symbolic
names must be used instead. I.e. ERROR-CLOSE-FILE, instead of -62.
5. Minor point: I don't understand the reason for MAX_SLURP at all.
6. The "close-file if -62 throw then" likes are hiding real error
reasons. Why not just "close-file throw"?

The whole design is radically different from gforths and is much worse.
I.e. one should write something like (not tested):

: gforth-compat-slurp-file ( c-addr u -- addr u )
-1 TO MAX_SLURP \ assuming we use U>
0 slurp_buf ! -1 slurp_fid ! \ clear globals
['] slurp-file catch ?DUP IF
slurp_buf @ 0<> IF slurp_buf @ FREE DROP THEN
slurp_fid @ -1 <> IF slurp_fid @ CLOSE-FILE DROP THEN
THROW
THEN
;

For gforth compatibility.

Now VFX variant from (Examples/Lin32/PowerNet/Services/Pages.fth):

: Slurp-File \ fileid addr -- len ior
\ *G Read the open file into memory, close it and return the
\ ** length read and an ior (0=success).
over file-size 2drop \ -- fileid addr len
rot >r \ -- addr len ; R: -- fileid
tuck r@ read-file nip \ -- len ior ; R: -- fileid
r> close-file drop
;

Just great. No error checking at all... Also, output buffer size is
never checked, so we can easily have buffer overflow.

It should be noted, that in PowerNet v4, this word was removed and
"data-file" was used instead. Which also imposes resource leakage.
This is from "kernel.fth" "data-file" word:

r@ file-size #-414 ?throw drop \ -- size ; R: -- handle
here over r@ read-file #-414 ?throw drop \ -- size ; R: -- handle

I.e. file not closed on error...

As a bonus -- "slurp-fid" from gforth:

: slurp-fid ( fid -- addr u ) \ gforth
\G @var{addr u} is the content of the file @var{fid}
{ fid }
0 0 begin ( awhole uwhole )
\ !! memory leak, extend-mem can throw
dup 1024 + dup >r extend-mem ( anew awhole uwhole R: unew )
\ !! memory leak
rot r@ fid read-file throw ( awhole uwhole uread R: unew )
r> 2dup =
while ( awhole uwhole uread unew )
2drop
repeat
\ !! memory leak
- + dup >r resize throw r> ;

Also, this code uses "old style" implementation, when we read file in
pieces. That was cool 20-30 years ago, but now is almost useless, at
least with such small buffer size.

And I'm not sure that "extend-memory" won't cause performance issues
compared to single "ALLOCATE"...

Second bonus (ruvim code):

: FILE-CONTENT ( h-file -- addr u ior )
\ addr should be freed via FREE on success
>R
R@ FILE-SIZE DUP IF RDROP EXIT THEN DROP
( d-size ) IF 0
-1005 RDROP EXIT THEN \ too big
DUP ALLOCATE DUP IF RDROP EXIT THEN DROP SWAP ( addr u )
2DUP R@ READ-FILE-EXACT DUP IF
( addr u ior )
2>R FREE DROP 0 2R>
THEN ( addr u 0 )
RDROP
;
: FILENAME-CONTENT ( d-txt-filename -- addr u )
R/O OPEN-FILE-SHARED THROW >R
R@ FILE-CONTENT
\ !! memory leak here if close-file fails.
R> CLOSE-FILE SWAP THROW THROW
;

Well, this is the best implementation. Only one leak, which can be
easily fixed.

P.S. Vfx "Lib/FileHacks.fth" is not interesting at all, because it just
aborts on any error.

[Marcel Hendrix]

Allocate/allot memory, set CP/DP/HERE to it, generate :NONAME .
Deallocate / unallot after use. The user initiates the garbage collection
process by hand. There are no new mechanisms to invent but it may
be useful to invest some time to make this as easy and safe as
possible.
In a real Forth one can also the compile the temporary code in a
different thread and COLD / restart that thread after use.

-marcel

[minf...@arcor.de]

F.ex. get rid of the assembler wordlist when you're done with assembling.
This implies that the assembler had not been loaded at HERE to prevent
holes in codespace.

[dxforth]

I was curious what it would take to convert the locals. And since I had SKIP/SCAN ...

: split ( a u c -- a2 u2 a3 u3 )
>r 2dup r> scan 2swap 2 pick - ;

: >lowcase 96 or ;

: intervall \ ad n -- ad+n ad ( usually called BOUNDS)
over + swap ;

: nextword \ a n -- a' n' ad nr

bl skip bl split ;

: cwc \ a n xt -- m conditional word count

0 begin >r over while
>r nextword r@ execute r> r> rot -
repeat drop 2drop r> ;

create vowels ," aeiouy"

: isvowel \ ch -- flag
>lowcase vowels count rot scan nip 0<> ;

[Anton Ertl]

Andy Valencia <van...@vsta.org> writes:
>I'm not a Google user, but I found an archive at:
> https://news.novabbs.com/programming/article-flat.php?id=12344&group=comp.lang.forth

With a search page at

<https://news.novabbs.com/rocksolid/search.php?>

This is nice, because it shows the whole thread (while
<http://al.howardknight.net/> only gives you the article).

[Robert L.]

On 12/24/2021, Andy Valencia wrote:

> I thought of the toy problem
> of counting the number of words in the diction with an "a" in them:
>
> >>> f = open("/usr/share/dict/words", "r")
> >>> na = 0
> >>> for l in f:
> ... if "a" in l:
> ... na += 1
> ...
> >>> print na
> 52849
> >>>
>
> (Python, obviously,

Nonsense. Not everyone is so unfortunate as to know Python.

> and it took .13 CPU seconds on an arm64
> device, the RockPRO64. The file is 102,401 lines.)

Python certainly makes it look difficult.

Awk:

/a/ {na++} END {print na}

Invocation:

awk "/a/ {na++} END {print na}" words

--
archive.org/details/nolies

[Robert L.]

Gauche Scheme:

(use file.util)
(print (count #/a/ (file->string-list "words")))

[Robert L.]

On 12/29/2021, Nickolay Kolchin wrote:

> During first human flights to space a very complex problem raised. Standard
> ink pens don't work in space, due to zero gravity. So NASA spend several
> dozen of millions of dollars and developed ink pen that can work in space
> conditions. Russians just used a pencil.

The 'lead' in a pencil conducts electricity. Pieces of lead
can break off. It is not safe to have tiny conductive pieces
floating around in zero gravity in a spacecraft. They could
cause shorts.

A company named Fisher developed a ballpoint-pen refill that
is pressurized. That company claims that the Russians also
used their pens.

[dxforth]

Considering the amount of space junk they've put up there since,
graphite in pencils would be the least of their problems :)