An earnest code review: Postscript interpreter

[luser droog]

I fear to some I may already have cried wolf a time too many.
But, this time, it's not some crazy golfed-up jazz. Rather,
it's the other thing I've been doing wherein I really did
follow all your collective advices.

The project homepage:
http://code.google.com/p/xpost
Clone:
hg clone https://luser...@code.google.com/p/xpost/
There are more details in the INSTALL file, but the short
version is:
cd xpost
./configure && make && make install
which will create the two executable programs
xpost and xpost_client. The client program is
just a demonstration of the library API and it
has a separate review:
http://codereview.stackexchange.com/questions/65061/rfc-ps-rasterizer-library-api

Xpost itself is mostly conformant to the Postscript level-1
standard. The details are listed in the COMPLIANCE file.
An earnest attempt has been made to conform to these coding conventions:
http://code.google.com/p/xpost/wiki/CodingConventions
and to conform to ISO C90 standards and POSIX or Windows where appropriate.
An overview of the design is here:
http://code.google.com/p/xpost/source/browse/doc/NEWINTERNALS
And an older version (with some now-obsolete abbreviated file and
function names) but somewhat more detailed is here:
http://code.google.com/p/xpost/source/browse/doc/INTERNALS

Portions of the code have been previously reviewed, and
are listed in two thread-index posts in:
https://groups.google.com/forum/#!forum/xpost-discuss

Of particular interest to participants here might be
the math operators:
http://code.google.com/p/xpost/source/browse/src/lib/xpost_op_math.c
or type-conversions, like numbers to strings:
http://code.google.com/p/xpost/source/browse/src/lib/xpost_op_type.c
or the graphics devices:
http://code.google.com/p/xpost/source/browse/src/lib/xpost_dev_xcb.c
http://code.google.com/p/xpost/source/browse/src/lib/xpost_dev_win32.c
which both overload a base-class implemented in postscript:
http://code.google.com/p/xpost/source/browse/data/ppmimage.ps

xpost is available under a BSD 3-Clause licence. Permission
to quote portions here for critique is very much permitted and
encouraged.

--
Luser D. Roog

[Malcolm McLean]

On Friday, December 26, 2014 5:05:43 AM UTC, luser droog wrote:
> I fear to some I may already have cried wolf a time too many.
> But, this time, it's not some crazy golfed-up jazz. Rather,
> it's the other thing I've been doing wherein I really did
> follow all your collective advices.
>
> The project homepage:
> http://code.google.com/p/xpost
>

It would be good to have a Postscript-type widget for Baby X.

I've kept most graphics out of Baby X, it just exposes a width * height
rgba surface to draw upon.

[BartC]

On 26/12/2014 05:05, luser droog wrote:

> The project homepage:
> http://code.google.com/p/xpost

So how do you get to the actual source code from here?

Clicking 'Source' tells me how to create a clone, or how to access from
the command line.

Clicking 'Download' gives a long list of assorted files, the nearest
that looks like it might be source is xpost0c.zip "commented postscript
interpreter", but that's dated 2010, and its math.c file is nothing like
the one you directly linked to below.

> http://code.google.com/p/xpost/source/browse/src/lib/xpost_op_math.c

OK, got it, but only because I could use the contents of this link as a
guide. For anyone else having trouble (although it could just be me!):

(1) Click the project link as above
(2) Click 'Source'
(3) In the line below labelled 'Repositoty', click 'Browse'
(4) Under 'Directories' heading, click 'src'
(5) In the list of 'src' subdirectories, click 'lib'

A direct link /might/ be:
https://code.google.com/p/xpost/source/browse/#hg%2Fsrc%2Flib (try with
normal http if https gives trouble; this newsreader doesn't allow me to
preview a post).

--
Bartc

[fir]

i may only say on code lookin, looks clean, not bad (though i seen codes looking better - but judging the style of code is a bit inexact ;/ this seem like a 10% or so 'too long' (inexact feeling ;/)

this im saying, moderately experienced
programmer coding in much chaotic and dirty style ;/

If authar want to say with own words what is this code doing, what was hard part etc i would also like to hear, talking on specyfic projects in depth it is always the thing i like

[luser droog]

On Friday, December 26, 2014 6:16:40 AM UTC-6, Bart wrote:
> On 26/12/2014 05:05, luser droog wrote:
>
> > The project homepage:
> > http://code.google.com/p/xpost
>
> So how do you get to the actual source code from here?
>
> Clicking 'Source' tells me how to create a clone, or how to access from
> the command line.

My bad. I should have add some explanation about that. 'Source' -> 'Browse'
should get you there. C files are in /src/bin and /src/lib and postscript
files are in /data.

> Clicking 'Download' gives a long list of assorted files, the nearest
> that looks like it might be source is xpost0c.zip "commented postscript
> interpreter", but that's dated 2010, and its math.c file is nothing like
> the one you directly linked to below.

Unfortunately, google code as discontinued its download section, so I cannot
add any new files. We're still shopping for a place to host binary releases.

xpost0c.zip is a very old version. from the /doc/INTERNALS file:
[ previous XPost is available in the downloads as a .zip of the source
directory. It was written and tested with Debian Lenny on an Olpc XO-1
laptop. (Well, it's not the /whole/ source directory, but all text files
needed by the makefile) It is no longer being maintained.
for using:
http://code.google.com/p/xpost/downloads/detail?name=xpost2g.zip
or, for just reading:
http://code.google.com/p/xpost/downloads/detail?name=xpost2.pdf ]

http://code.google.com/p/xpost/wiki/EvolutionOfXpost
describes the major versions which the source has traversed.

> > http://code.google.com/p/xpost/source/browse/src/lib/xpost_op_math.c
>
> OK, got it, but only because I could use the contents of this link as a
> guide. For anyone else having trouble (although it could just be me!):
>
> (1) Click the project link as above
> (2) Click 'Source'
> (3) In the line below labelled 'Repositoty', click 'Browse'
> (4) Under 'Directories' heading, click 'src'
> (5) In the list of 'src' subdirectories, click 'lib'
>
> A direct link /might/ be:
> https://code.google.com/p/xpost/source/browse/#hg%2Fsrc%2Flib (try with
> normal http if https gives trouble; this newsreader doesn't allow me to
> preview a post).
>

That's correct. The entire interpreter is in the library, used by a
relatively small "application" source file:
http://code.google.com/p/xpost/source/browse/src/bin/xpost_main.c

[luser droog]

Yes. That would be very cool.
You can email me, or we can use the xpost discussion group.
https://groups.google.com/forum/#!forum/xpost-discuss

[luser droog]

On Friday, December 26, 2014 7:30:53 AM UTC-6, fir wrote:
> W dniu piątek, 26 grudnia 2014 06:05:43 UTC+1 użytkownik luser droog napisał:
> > I fear to some I may already have cried wolf a time too many.
> > But, this time, it's not some crazy golfed-up jazz. Rather,
> > it's the other thing I've been doing wherein I really did
> > follow all your collective advices.
> >
> > The project homepage:
> > http://code.google.com/p/xpost

<snip>

> i may only say on code lookin, looks clean, not bad (though i seen codes looking better - but judging the style of code is a bit inexact ;/ this seem like a 10% or so 'too long' (inexact feeling ;/)
>

A lot of the code has become longer to conform to the coding conventions.
My earlier versions were written in a much more compact style. But this
proved to be distracting to my earlier reviewers.
https://groups.google.com/d/topic/comp.lang.c/l6xjtiIfrfI/discussion

> this im saying, moderately experienced
> programmer coding in much chaotic and dirty style ;/
>
> If authar want to say with own words what is this code doing, what was hard part etc i would also like to hear, talking on specyfic projects in depth it is always the thing i like

Ok. Well, so far it's mostly a clone of ghostscript for simple uses.
If you run `xpost` without arguments, it will open a graphics window
(letter dimensions, 1 point-per-pixel), print some initialization messages
and then a prompt, expecting postscript language commands to be typed.

$ xpost
geom 1 : (null)
geom 2 : -1x-1+0+0
loading init.ps...
DATA_DIR =/usr/local/share/xpost
loading err.ps...
$error installed....error installed...errordict installed...
eof err.ps
eof init.ps
loading graphics.ps...
loading device.ps...
loading image.ps...
loading pgmimage.ps...eof pgmimage.ps
loading ppmimage.ps...eof ppmimage.ps
eof image.ps
loading nulldev.ps...eof nulldev.ps
eof device.ps
loading gstate.ps...eof gstate.ps
loading color.ps...eof color
loading path.ps...eof path.ps
loading clip.ps...eof clip.ps
loading paint.ps...eof paint.ps
loading font.ps...eof font.ps
eof graphics.ps
graphics loaded, calling initgraphics
[ 1 0 0 -1 0 792 ]
Xpost version 0.0.1
PS>

You can see from the init messages that a lot of the graphics is
implemented in the postscript language. So there is a "core"
interpreter which is written in C, and then an upper layer which
is written in postscript. It is my understanding that Forth is
often implemented similarly, with an "inner loop" and an "outer
loop".

The "outer loop" is in /data/init.ps:/executive . It uses the
postscript "special file" %statementedit to read from the keyboard.
The %statementedit special file is implemented in
src/lib/xpost_file.c:statementedit() . It is responsible for
reading lines until a complete "statement" is read, with all
braces and parens (but not square brackets) balanced.

After this, the outer loop uses the sequence `cvx exec` to pass
the file-object representing the statement over to the "inner
loop" which is implemented in /src/lib/xpost_interpreter.c:eval() .

The eval() function executes a file by calling evalfile() which
uses the postscript `token` operator to read the first object
from the file and construct the internal postscript object
which represents it. For more background on the token operator,
see the old clc discussion "embarrassing spaghetti code needs
stylistic advice"
https://groups.google.com/d/topic/comp.lang.c/_m0DEiFqCv0/discussion

The basics of the behavior described so far were all present in
the first version I posted four years ago. The "hard part" has
always been memory management. Postscript has a very different
way of using memory than the C standard library directly
supports. So I had to design my own memory architecture and
the algorithms to support it. The overall story of the memory
management saga is in:
http://code.google.com/p/xpost/wiki/EvolutionOfXpost

So early versions are discussed in
https://groups.google.com/d/topic/comp.lang.c/0r0GyJA1bnA/discussion
https://groups.google.com/d/topic/comp.lang.postscript/tpSevlR1BJ8/discussion (beginning with the message "A Working Prototype VM")
and
https://groups.google.com/d/topic/comp.lang.postscript/VsFzRwnuM_E/discussion (beginning with the message "prototype save-restore mechanism")

From another angle, "how it works" is it describes a sort of "byte code"
or internal representation for all the various types of object it needs to
manipulate. This is described in the same thread as the "prototype save-
restore mechanism" mentioned above, and also in /doc/NEWINTERNALS.
But basically, it squeezes all objects into 8 bytes. So 2 machine words
on a 32bit machine or 1 machine word on a 64bit machine.

Simple objects such as integers, booleans and reals fit into the 8-byte
structure. But composite objects such as strings and arrays and dicts
contain references to data elsewhere in memory. 19 bits are used to
specify an index into the memory table (which contains a pointer to
the actual data) and 16bits are used to specify and offset in that data
and 16bits for the size (in whatever units are appropriate for the type:
bytes for strings, objects for arrays, pairs for dicts).

This is convenient for operators such as `getinterval` which returns
a substring or subarray of a given string or array. It can be implemented
without actually looking into memory, just by manipulating the fields
of the 8-byte object.

After getting all of the above working, a further "hardest part" was
a collaborative effort with a volunteer, Vincent Torri, who wrote
the autotools files and tested building on various systems.

After that, still another "hardest part" was migrating the interpreter
into the library. This involved expanding many of my terse identifiers
into longer, name-spaced names. But it also meant unraveling the
inter-dependencies among my various "modules", as I liked to call them.
And eliminating my use of setjmp and longjmp for error handling,
upon being informed that this was inappropriate for a library.

But of course, there were immediate benefits from all of this work.
Most of the code is pretty solid, though I say so myself (and am
quite biased). But there are some infelicities.

In the operator table, src/lib/xpost_operator.c, I originally
used un-prototyped function pointers to hold pointers to the
various operator functions which may received different numbers
of arguments. The number is tracked separately and a function
is only called with the correct number, but we kept getting
warnings from the compiler. So I went ahead and added a prototype
and lots of casts everywhere. But it feels iffy. Like lying
to the compiler is worse that keeping secrets from it.
I dunno. After the change I haven't really run into it enough
to have accumulated any actual irritation about it.

Umm. I think I forgot the question some time ago.

Hope this made some kind of sense!

--
droog

[fir]

very interesting, congratulations for this project,

could answer more on this but i would
need to think a bit more on this, will see later how yet i could comment it and say more on this; anyway if you have yer more story or remarks on this it is pleasure to read

[luser droog]

On Saturday, December 27, 2014 3:45:56 AM UTC-6, fir wrote:
> W dniu sobota, 27 grudnia 2014 08:27:31 UTC+1 użytkownik luser droog napisał:
> > On Friday, December 26, 2014 7:30:53 AM UTC-6, fir wrote:
> > > W dniu piątek, 26 grudnia 2014 06:05:43 UTC+1 użytkownik luser droog napisał:
> > > > I fear to some I may already have cried wolf a time too many.
> > > > But, this time, it's not some crazy golfed-up jazz. Rather,
> > > > it's the other thing I've been doing wherein I really did
> > > > follow all your collective advices.
> > > >
> > > > The project homepage:
> > > > http://code.google.com/p/xpost

> > <snip...>

> > > If authar want to say with own words what is this code doing, what was hard part etc i would also like to hear, talking on specyfic projects in depth it is always the thing i like
> >
> > Ok. Well, so far it's mostly a clone of ghostscript for simple uses.
> > If you run `xpost` without arguments, it will open a graphics window
> > (letter dimensions, 1 point-per-pixel), print some initialization messages
> > and then a prompt, expecting postscript language commands to be typed.
> >

<snip>

> > Umm. I think I forgot the question some time ago.
> >
> > Hope this made some kind of sense!
> >
> very interesting, congratulations for this project,
>
> could answer more on this but i would
> need to think a bit more on this, will see later how yet i could comment it and say more on this; anyway if you have yer more story or remarks on this it is pleasure to read

One more highlight from the archive:
How does ghostscript make ints and reals equivalent as dictionary keys??!!! 7/23/13
https://groups.google.com/d/topic/comp.lang.postscript/9gWjhaeXjTA/discussion

This thread discusses designing a representation where a float and an int
will yield the same hash value. The trick was to promote both to int,
and ignore some of the extra precision in the mantissa.

Also all of the threads in this comp.lang.postscript list titled "prototype"
are C programs. SSCCEs.
https://groups.google.com/forum/#!topic/xpost-discuss/KJCesIt-AGw

[luser droog]

On Saturday, December 27, 2014 1:52:28 PM UTC-6, luser droog wrote:
> will yield the same hash value. The trick was to promote both to int,

s/int/double/

--
double dammit

[luser droog]

On Thursday, December 25, 2014 at 11:05:43 PM UTC-6, luser droog wrote:
> I fear to some I may already have cried wolf a time too many.
> But, this time, it's not some crazy golfed-up jazz. Rather,
> it's the other thing I've been doing wherein I really did
> follow all your collective advices.
>
> The project homepage:
> http://code.google.com/p/xpost

Browse Code:
http://code.google.com/p/xpost/source/browse
http://code.google.com/p/xpost/source/browse/src/bin C source code
http://code.google.com/p/xpost/source/browse/src/lib C source code
http://code.google.com/p/xpost/source/browse/data PS source code

> Clone:
> hg clone https://luser...@code.google.com/p/xpost/
> There are more details in the INSTALL file, but the short
> version is:
> cd xpost

./autogen.sh # generate the configure script

> xpost is available under a BSD 3-Clause licence. [Quotation of]

> portions here for critique is very much permitted and
> encouraged.
>

Oh, one more thing I forgot to mention, it has doxygen support
although the commentary could be much more extensive. And the
design document is based off of the older internals file.

make doc
cd doc/html
firefox index.html

And on most platforms (except Ms. Windows), there are many
configuration options available from ./configure and make.
Incidentally, the --enable-large-object option needs debugging
help. I think the problem is in the hash function.

I also take responsibility for the that the 'Source' link
takes you to the 'checkout' page rather than the 'browse'
page. That was an option that I selected. But if you join
the project, it takes you to 'changes'!

--
turn and face the strange