Highlighting FORTH code in PHP

114 views
Skip to first unread message

Marc Petremann

unread,
Jun 23, 2022, 5:42:35 PMJun 23
to
Preamble

Very early in my long career as a computer programmer, I appreciated the source code editors that highlight source code syntax. Some of these code editors know how to adapt to the programming language used.

Later, by writing articles about programming, on blogs, then on my websites, I used GesHi, a powerful but complex library.

The problem with GesHi is that it does not process the FORTH code. I did well to create scripts adapted to the FORTH language for GesHi, but these scripts had still some flaws.

After analysis, GesHi processes the FORTH code character by character and considers the characters . and - as code operators.

GesHi represents more than 4000 lines of code in PHP. A complex code and little factored. We feel the touch of several programmers who have each added layers.

Complete code is available here:
https://github.com/MPETREMANN11/php/blob/main/Forth.php

A new code analyzer for FORTH

I was faced with several choices:

- leave FORTH GEsHi jobs as is and accept errors from GesHi
- choose another code analyzer and fall back on similar adaptation problems
- spending dozens of hours tracing and trying to improve Geshi....

It turns out that I'm very good at programming in PHP. So I started with do a very simple test: explode a FORTH code with PHP:

/**
* Explode FORTH source code in array of keywords
* @param type $code
*/
private function explodeSource($code) {
$lines = explode( "\n", str_replace("\r", "", $code));
$this->keySource = array();
foreach ($lines AS $line) {
$this->keySource[] = explode(" ", $line);
}
}

The result was just what I was looking for. This function explodeSource splits each row into an array of rows. The second loop slices the contents of each row to table using the space character as a separator. The result is returned in an array $this->keySource.

This is what made me decide to create my own FORTH code analyzer in PHP. I made a FORTH code highlighter in just two evenings....
Define FORTH wordlists

On my website, the FORTH words are stored in a database. Here's how it's done injecting FORTH wordlists into a $this->keywords array:

/**
* fetch all forth words from database
*/
private function getKeywords() {
$Glossaire = new Application_Model_Glossaire;
$this->keywords[0] = $Glossaire->getListeMotsOrdinaires();
$this->keywords[1] = $Glossaire->getListeMotsDefinition();
$this->keywords[2] = $Glossaire->getListeMotsStructure();
$this->keywords[3] = $Glossaire->getVariablesConstantes();
$this->keywords[4] = $Glossaire->getListeMotsVocabulaires();
//$motsDefered = $Glossaire->getListeMotsDefered();
$Registers = new Application_Model_Registers;
$this->keywords[5] = $Registers->getListeRegistres();
}

On your website, you can define your lists in this way if you don't want not operate a database:

/**
* fetch all forth words from database
*/
private function getKeywords() {
$Glossaire = new Application_Model_Glossaire;
$this->keywords[0] = array('drop', 'dup', 'rot', 'over', 'swap');
$this->keywords[1] = array(':', ';', 'constant', 'variable');
$this->keywords[2] = array('if','else','then');
$this->keywords[3] = array('base');
$this->keywords[4] = array('forth');
}

With this method, different categories of FORTH words are defined. this allows to highlight the FORTH code with different styles.
Definition of styles

We define another table containing the different styles to highlight FORTH code:

/**
* define all styles for decoration
* @var array
*/
var $outStyle = array(
0 => 'color: #0000ff;', // normal words
1 => 'color: red; font-weight: bold;', // definitions words
2 => 'color: black; background-color: yellow;', // structure words
3 => 'color: #ff00ff; font-weight: bold;', // constants
4 => 'color: black; font-weight: bold; background-color: #d7d7ff', // vocs
'bin' => 'color: #7f00dd;', // binaries values
'dec' => 'color: #0000dd;', // decimal values
'hex' => 'color: #003ffd;', // hexadecimal values
);

In this code we have three keys bin dec and hex. These keys are used to decorate numeric values in binary, decimal or hexadecimal.
Decoration of a word FORTH

The decoration of a FORTH word works in successive stages:

if the word is in a comment, we don't decorate it
if the word is a binary or decimal or hexadecimal number, we decorate it. It's the role of regular expressions analyzing the structure of the word. For example, 10010011 is parsed as binary value;
if we encounter the word \ we activate the flag marking the comment and we return the decoration marking this comment;
finally, we test whether the word appears in the table of FORTH words to decorate. This is the role of the last loop foreach. If it's a word to decorate, we decorate it.

At each step, exit the function with return. If no condition is met we return the word without decoration, followed by a space character.

/**
* highlight FORTH code
* @param string $word
*/
private function styliseWord($word) {
if ($this->commentFlag == TRUE) {
return $word ." ";
}
// test if $word is a binary value
$re = '/^[0-1]{8}/is';
preg_match($re, $word, $matches, PREG_OFFSET_CAPTURE, 0);
if (!empty($matches)) {
return '<abbr style="' . $this->outStyle['bin'] . '" title="binary byte">'
. $word . '</abbr> ';
}
// test if $word is a decimal value
$re = '/^\d{1,}/is';
preg_match($re, $word, $matches, PREG_OFFSET_CAPTURE, 0);
if (!empty($matches)) {
return '<abbr style="' . $this->outStyle['dec'] . '" title="number">'
. $word . '</abbr> ';
}
// test if $word is a hexadecimal value
$re = '/^\$[a-f0-9]{1,}/is';
preg_match($re, $word, $matches, PREG_OFFSET_CAPTURE, 0);
if (!empty($matches)) {
return '<abbr style="' . $this->outStyle['hex']
. '" title="hexadecimal value">' . $word . '</abbr> ';
}
// if it's the word \ tag comment
if (ord($word) == 92) {
$this->commentFlag = TRUE;
return '<span style="color: #808080; font-style: italic;">' . $word ." ";
}
// test if $word is a FORTH word
foreach ($this->keywords AS $level => $listeMots) {
if (in_array(strtoupper($word), $listeMots)) {
return $this->linkifyWord($word, $level);
}
}
return $word ." ";
}

The decoration of a FORTH word that is defined in the FORTH word list is done this way in PHP:

/**
* array of external links
* @var array
*/
var $outlink = array(
0 => 'help/index-esp32/word/',
1 => 'help/index-esp32/word/',
2 => 'help/index-esp32/word/',
3 => 'help/index-esp32/word/',
4 => 'help/index-esp32/word/',
5 => 'help/reg-esp32/word/',
);

/**
* include FORTH word in an external link
* @param string $word
* @param mixed $level
* @return string
*/
private function linkifyWord($word, $level) {
if ($this->enable_keyword_links) {
return '<a href="' . $this->outlink[$level] . base64_encode($word)
. '" style="' . $this->outStyle[$level] . '">' . $word . '</a> ';
}
return '<abbr style="' . $this->outStyle[$level] . '">' . $word . '</abbr> ';
}

We define a list of links in the $outlink array. Here are internal links specific to my website. It's easy to set links to external sites:

/**
* array of external links
* @var array
*/
var $outlink = array(
0 => 'https://forth-standard.org/standard/core/DUP',
1 => 'https://forth-standard.org/standard/core/DUP',
2 => 'https://forth-standard.org/standard/core/DUP',
3 => 'https://forth-standard.org/standard/core/DUP',
4 => 'https://forth-standard.org/standard/core/DUP',
);

Reconstruction of the decorated FORTH code

To reconstruct our decorated FORTH code, we start from the $this->keySource array. This array was populated by explodeSource. This table is read item by item. If we encounter an empty item, we concatenate a single space. Otherwise, we decorate the item according to the conditions of the styliseWord function.

/**
* Generate highlighted FORTH source code
* @return string
*/
private function generateOutsource() {
$outSource = "";
foreach ($this->keySource AS $line) {
foreach ($line AS $word) {
if (strlen($word)==0) {
$outSource .= " ";
} else {
$outSource .= $this->styliseWord($word);
}
}
if ($this->commentFlag == TRUE) {
$this->commentFlag = FALSE;
$outSource .= "</span>";
}
$outSource .= "\n";
}
return $outSource;
}

Here is how to use PHP code to highlight FORTH code. the PHP code must be loaded beforehand by include or the PHP autoload function:

<?php $forthStr = <<<EOT
DEFINED? --random [if] forget --random [then]
create --random

\ Random number data
$3FF75144 constant RNG_DATA_REG \ Read Only ESP32 register

\ get 32 bits random b=number
: rnd ( -- x )
RNG_DATA_REG L@
;
EOT;
echo $Forth->sourceForView($forthStr); ?>

The result is the expected one:

DEFINED? --random [if] forget --random [then]
create --random

\ Random number data
$3FF75144 constant RNG_DATA_REG \ Read Only ESP32 register

\ get 32 bits random b=number
: rnd ( -- x )
RNG_DATA_REG L@
;

Conclusion

If we refer only to the content of the GesHi code, the code highlighting seems an arduous task. And it is indeed one if we build a gas plant.

Here, with less than 200 lines of code, in PHP, I developed, in a few hours, a FORTH code highlighter that works much better than the gesHi code on which I spent dozens of days trying to understand all the subtleties.

There remains the question of the interest of highlighting the FORTH code...

What interested me, with GesHi, is that natively, if you highlight PHP code, it is possible to activate external documentation links.

It is this aspect of GesHi that seems to me to be of enormous interest to FORTH. Because you will have no matter how well documenting a program, nothing beats direct access to explanations of any FORTH vocabulary word.

In the source codes disseminated here, all the words of ESP32forth that are documented are displayed with syntax highlighting and a hyperlink giving direct access to the documentation of this word.

Because, if we want FORTH to interest programmers, it seems essential to me to facilitate the understanding the code. And, unless I'm mistaken, no other FORTH developer website has yet given such easy access to FORTH code documentation through syntax highlighting.

A word for Peter FORTH:

Peter.

You blocked me from your FORTH2020 group when I presented this method of FORTH code highlighting.

You also blocked me from other bands as soon as I intervened to present FORTH. Your ESP32FORTH blog is needy. Badly built. It would gain in readability to exploit my PHP code.

So put your ego aside. If you really think you are working in the interest of the FORTH language, Stop this childish attitude towards me.

Jurgen Pitaske

unread,
Jun 24, 2022, 3:13:03 PMJun 24
to
Just to put the record staight:

THE PERSON PETER FORTH DOES NOT EXIST
THOUGH HE HAS CREATED A FALSE FACEBOOK ACCOUNT IN THIS NAME

https://www.facebook.com/peter.forth.583

from the German Forth group they clarified:
19 Jul 2021, 11:10:29
to
Just to add some related information:

As Paul Bennett has threatened to delete the whole thread in the Forth facebook group
if there are any more truths posted about Peter Forth there,
I better post it here - a safer place in this case:

This is how it started with a clip from the German magazine:

Peter Minuth alias Peter Forth hat den Finger am Forth–Puls, von Süd– Amerika aus, via Facebook,
so ziemlich in jede Ecke der Welt. Und so konnte er zusammentragen,
was da auf dem Win32Forth so geht. Schaut mal rein bei ihm im Netz. Jostein S

PETER MINUTH alias PETER FORTH has his finger on the Forth Pulse,
from South America, via facebook,
to pretty in every corner of the world.
And so he was able to collect what's going on with Wi32Forth.
Take a look at him on the net

You, in the Forth facebook group kindly posted the link to the 4th Dimension magazine the article is in:
Found at
https://forth-ev.de/wiki/res/lib/exe/fetch.php/vd-archiv:4d2018-03.pdf

Unfortunately all in German. But there is always google translate

At the end of the article there you find:

[Zum Autor: Peter Minuth lebt heute in Sao Paulo, Brasilien. In Argentinien geboren, wuchs er zweisprachig auf, Deutsch ist seine Vatersprache. Forth nutzt er seit vielen Jahren. Im Studium der Astrophysik und später in der professionellen Arbeit bei der CNC Maschinenherstellung — von einfachen Werkzeugwechslern bei Leiterplattenrobotern bis zu complettem CAD/CAM für die LuftfahrtIndustrie. Hobbys: Sternbeobachtung, Tauchen, Fliegen und Bluesmusik.]

[About the author: Peter Minuth now lives in Sao Paulo, Brazil. Born in Argentina, he grew up bilingual, German is his mother tongue.
He's been using Forth for many years.
During the study of astrophysics and later in the professional work in CNC machine production
- from simple tool changers for circuit board robots
to complete CAD / CAM for the aviation industry.
Hobbies: stargazing, diving, flying and blues music.]

Kontakt: eMail: peter4...@gmail.com Facebook: Peter Forth

dxforth

unread,
Jun 25, 2022, 1:35:02 AMJun 25
to
On 25/06/2022 05:13, Jurgen Pitaske wrote:
>
> Just to put the record staight:
>
> THE PERSON PETER FORTH DOES NOT EXIST
> THOUGH HE HAS CREATED A FALSE FACEBOOK ACCOUNT IN THIS NAME
>
> https://www.facebook.com/peter.forth.583
>
> from the German Forth group they clarified:
> 19 Jul 2021, 11:10:29
> to
> Just to add some related information:
>
> As Paul Bennett has threatened to delete the whole thread in the Forth facebook group
> if there are any more truths posted about Peter Forth there,
> I better post it here - a safer place in this case:

Because "The sun, too, shines into cesspools and is not polluted." ?


none albert

unread,
Jun 25, 2022, 5:20:05 AMJun 25
to
In article <41a69a91-f708-447a...@googlegroups.com>,
Jurgen Pitaske <jpit...@gmail.com> wrote:
>
>Just to put the record staight:
>
>THE PERSON PETER FORTH DOES NOT EXIST
>THOUGH HE HAS CREATED A FALSE FACEBOOK ACCOUNT IN THIS NAME
>
>https://www.facebook.com/peter.forth.583

Really? If this is true, Peter Forth risks a removal of Facebook.

Given the reputation of Facebook I tried to sign up using a pseudonym.
It was promptly detected and refused. It was in the rules, only real
names allowed. That was the end of it.

Groetjes Albert
--
"in our communism country Viet Nam, people are forced to be
alive and in the western country like US, people are free to
die from Covid 19 lol" duc ha
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

Jurgen Pitaske

unread,
Jun 25, 2022, 6:18:12 AMJun 25
to
https://forth-ev.de/wiki/res/lib/exe/fetch.php/vd-archiv:4d2018-03.pdf
VD3/2018 on page 4 it clearly states, written by Michael Kalus:

Peter Minuth alias Peter Forth hat den Finger am Forth–Puls, von Süd–
Amerika aus, via Facebook, so ziemlich in jede Ecke der Welt. Und so konnte er
zusammentragen, was da auf dem Win32Forth so geht. Schaut mal rein bei ihm
im Netz.

Quick translation:
Peter Minuth, with the alias of Peter Forth, has his finger on the Forth pulse;
From South America, via facebook in every corner of this planet.
And so he could collect what runs on Win32Forth,
Have a look.

Jurgen Pitaske

unread,
Jun 25, 2022, 6:23:12 AMJun 25
to
And as I just saw, there is actually on page 7 an article that he wrote.

tp

unread,
Jul 15, 2022, 10:46:49 PMJul 15
to
On Thu, 23 Jun 2022 14:42:32 -0700, Marc Petremann wrote:

snip
> https://github.com/MPETREMANN11/php/blob/main/Forth.php
>
> A new code analyzer for FORTH
snip

Hi Marc,
Outstanding! I have been looking for something to highlight my Forth blog
articles for years, and your method looks easy to use and easy to modify.

It will take me a while to experiment with it, but I'll provide feedback
when I'm done.

Cheers,
Terry


--
To me, good Embedded Forth code is all about readability, traceability and
maintainability with respect to the *hardware*.
Maintainer: https://mecrisp-stellaris-folkdoc.sourceforge.io and https://
mecrisp-across-folkdoc.sourceforge.io

Marc Petremann

unread,
Jul 16, 2022, 7:11:41 AMJul 16
to
Le samedi 16 juillet 2022 à 04:46:49 UTC+2, tp a écrit :
> On Thu, 23 Jun 2022 14:42:32 -0700, Marc Petremann wrote:
>
> snip
> > https://github.com/MPETREMANN11/php/blob/main/Forth.php
> >
> > A new code analyzer for FORTH
> snip
>
> Hi Marc,
> Outstanding! I have been looking for something to highlight my Forth blog
> articles for years, and your method looks easy to use and easy to modify.
>
> It will take me a while to experiment with it, but I'll provide feedback
> when I'm done.
>
> Cheers,
> Terry


Thanks,
It's nice to see that my developments can help.
If you have any questions, I'm available.
Cordially
Reply all
Reply to author
Forward
0 new messages