To initialize the vm I allocate a fresh, flat stretch of
memory using the POSIX function mmap (although
malloc would work, I'm saving that for more transient
"internals", like the scratch space used by the scanner
for constructing procedures).
Allocating a new composite object involves rounding
up the requested space to an even multiple of the
size of an object (padding out strings so arrays and
dictionary contents stay aligned).
To 'put' data into a composite object involves comparing
the address with the last save boundary. If newer (higher
address), data goes straight in. If older (lower address),
a copy of the composite is created at the current level
and then the new data gets 'put' there.
But I skip all this if it's a string, right?
The complication arises from coordinating all subsequent
'get' operations to find the new data. Because there may
be multiple pointers to the same composite structure,
The links have to be stored in VM itself and not in the
object.
One idea is to overwrite the first object with a double-link
structure, and store the old first element in an extra slot in
the new copy. This way I don't have to allocate the links
for every compoosite, everywhere.
But that seems tricky, and therefore bug-prone.
Would it be so bad to store a header in the vm with
the data? You need one for dictionaries, anyway;
because all duplicates share the same flags.
Hi,
it depends on what you want to do. You should specify, what your VM
allocator should do: either being correct or being efficient. Both go
together, but making it efficient (in whatever direction: speed or
memory usage or...) is another task than simply writing a correct one.
> I'm reimplementing the virtual memory for my interpreter
> and would like to get a sanity-check for what I've decided
> so far.
>
> To initialize the vm I allocate a fresh, flat stretch of
> memory using the POSIX function mmap (although
> malloc would work, I'm saving that for more transient
> "internals", like the scratch space used by the scanner
> for constructing procedures).
Well, internals should not occur in VM - or if they occur, you've a
system behind it. It's not like the scanners in Forth where you see
all the internals in actual state. If you for example use any defined
stack in postscript for internals the error handling gets complicated.
> Allocating a new composite object involves rounding
> up the requested space to an even multiple of the
> size of an object (padding out strings so arrays and
> dictionary contents stay aligned).
I'd call this an optimization. Dont forget to make your new object at
least one byte in length. Otherwise next object has same address.
> To 'put' data into a composite object involves comparing
> the address with the last save boundary. If newer (higher
> address), data goes straight in. If older (lower address),
> a copy of the composite is created at the current level
> and then the new data gets 'put' there.
> But I skip all this if it's a string, right?
I'm not sure what you want with "string". "string" is an array that
has special operators for it's type.
> The complication arises from coordinating all subsequent
> 'get' operations to find the new data. Because there may
> be multiple pointers to the same composite structure,
> The links have to be stored in VM itself and not in the
> object.
Make it a linked list - for example for dictionaries you search down
until you find a fitting key. Start from actual "top" dictionary.
> One idea is to overwrite the first object with a double-link
> structure, and store the old first element in an extra slot in
> the new copy. This way I don't have to allocate the links
> for every compoosite, everywhere.
Basically you could have a "parent" linking field - this is for the
first invocation of the thing. This first invocation should point to
the most recent child. In C this is all very complicated if you do not
do real GC.
> But that seems tricky, and therefore bug-prone.
> Would it be so bad to store a header in the vm with
> the data? You need one for dictionaries, anyway;
> because all duplicates share the same flags.
You need for every thing extra information. Try
123 cvx xcheck =
in a conforming interpreter.
Regards,
-Helmar
> You need for every thing extra information. Try
>
> 123 cvx xcheck =
>
> in a conforming interpreter.
I think that:
123 dup cvx xcheck = =
is even more clear.
Regards,
-Helmar
(should do copy/paste instead of typing the things ;) )
I want this one to be correct without being grossly inefficient.
> > I'm reimplementing the virtual memory for my interpreter
> > and would like to get a sanity-check for what I've decided
> > so far.
>
> > To initialize the vm I allocate a fresh, flat stretch of
> > memory using the POSIX function mmap (although
> > malloc would work, I'm saving that for more transient
> > "internals", like the scratch space used by the scanner
> > for constructing procedures).
>
> Well, internals should not occur in VM - or if they occur, you've a
> system behind it. It's not like the scanners in Forth where you see
> all the internals in actual state. If you for example use any defined
> stack in postscript for internals the error handling gets complicated.
>
> > Allocating a new composite object involves rounding
> > up the requested space to an even multiple of the
> > size of an object (padding out strings so arrays and
> > dictionary contents stay aligned).
>
> I'd call this an optimization. Dont forget to make your new object at
> least one byte in length. Otherwise next object has same address.
Well, I've decided to allocate a one-object-sized header for
everything in VM. So even '0 string' allocates 12 bytes.
This should make GC easier later as everything in VM will have
a tag.
> > To 'put' data into a composite object involves comparing
> > the address with the last save boundary. If newer (higher
> > address), data goes straight in. If older (lower address),
> > a copy of the composite is created at the current level
> > and then the new data gets 'put' there.
> > But I skip all this if it's a string, right?
>
> I'm not sure what you want with "string". "string" is an array that
> has special operators for it's type.
Yes but string contents are immune to save/restore.
Eg.
(abcdef)
save exch
dup 0 (ghi) putinterval
exch restore
==
should print (ghidef).
> > The complication arises from coordinating all subsequent
> > 'get' operations to find the new data. Because there may
> > be multiple pointers to the same composite structure,
> > The links have to be stored in VM itself and not in the
> > object.
>
> Make it a linked list - for example for dictionaries you search down
> until you find a fitting key. Start from actual "top" dictionary.
Hmm, if I make the key-value pairs in a dictionary a list of
2-element arrays, it would be trivial to let them grow.
> > One idea is to overwrite the first object with a double-link
> > structure, and store the old first element in an extra slot in
> > the new copy. This way I don't have to allocate the links
> > for every compoosite, everywhere.
>
> Basically you could have a "parent" linking field - this is for the
> first invocation of the thing. This first invocation should point to
> the most recent child. In C this is all very complicated if you do not
> do real GC.
Do you mean the Boehm GC? Otherwise GC seems more complicated.
> > But that seems tricky, and therefore bug-prone.
> > Would it be so bad to store a header in the vm with
> > the data? You need one for dictionaries, anyway;
> > because all duplicates share the same flags.
>
> You need for every thing extra information. Try
>
> 123 cvx xcheck =
>
> in a conforming interpreter.
Well, yeah, but 123 is an integer. The value is wrapped up
in the object itself. So I've got a set of flags in the
object. But dictionaries are different.
1 dict begin
/foo /bar def
currentdict readonly
/foo /baloney store
should trigger an invalidaccess error.
So the dictionary data-block in VM needs its own set
of flags which override whatever the object contains.
In this example, the dictionary object on the dict stack
still has full access. But reducing the access to the
duplicate on the operand stack should affect all access
to the dictionary data through any duplicate.
<head><string 5><5>
<head><array 2><2*sizeof(object)>
<save>
<head copy><array 2><2*sizeof(object)>
Or do I even need to look at the contents? I think I've thunk myself
confused..
I decided that the way to leave the stacks alone is to LEAVE THE
STACKS ALONE! and just examine the VM contents. But if an array
contains another array, the contained array would get rewound
automatically when encountered whereever it is. So I don't need
to recursively do anything, right? Just nullify any old links to
newer data. Then check the operand stack for bogus references.
#define _GNU_SOURCE
#include <assert.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#define MAXSAVE 15
#define BLOCKSIZE 1000
typedef unsigned long address;
typedef unsigned short size;
typedef bool boolean;
typedef long integer;
typedef double real;
#define AS_BARE(a) a ,
#define AS_STR(a) #a ,
#define ERRORS(_) \
_(noerror) \
_(dictfull) _(dictstackoverflow) _(dictstackunderflow) \
_(execstackoverflow) _(execstackunderflow) \
_(handleerror) \
_(interrupt) \
_(invalidaccess) _(invalidexit) _(invalidfileaccess) \
_(invalidfont) _(invalidrestore) \
_(ioerror) \
_(limitcheck) \
_(nocurrentpoint) \
_(rangecheck) \
_(stackoverflow) _(stackunderflow) \
_(syntaxerror) \
_(timeout) \
_(typecheck) \
_(undefined) _(undefinedfilename) _(undefinedresult) \
_(unmatchedmark) \
_(unregistered) \
_(VMerror) \
enum err { ERRORS(AS_BARE) };
char *errorname[] = { ERRORS(AS_STR) };
void error(integer e) {
fprintf(stderr, "%s\n", errorname[e]);
//exit(e);
}
#define AS_TYPE(a) a ## type ,
#define TYPES(_) \
_(boolean) _(integer) _(mark) _(name) _(null) _(operator) _(real)
_(save) \
_(array) _(dict) _(file) _(gstate) _(packedarray) _(string)
enum type { TYPES(AS_TYPE) };
char *typename[] = { TYPES(AS_STR) };
typedef struct flags {
union {
struct {
unsigned char lit : 1;
unsigned char read : 1;
unsigned char write : 1;
unsigned char exec : 1;
};
unsigned char allflags;
};
} flags;
typedef struct header {
flags flags;
address new, old;
} header;
typedef struct trailer {
address magic;
address head;
size sz;
} trailer;
typedef struct composite {
address a;
size n;
} composite;
typedef struct object {
unsigned char tag;
flags flags;
union {
boolean b;
integer i;
real r;
composite c;
} u;
} object;
object consboolean (boolean b) { return (object) { .tag =
booleantype, .u.b = b }; }
object consint (integer i) { return (object) { .tag =
integertype, .u.i = i }; }
object consreal (real r) { return (object) { .tag =
realtype, .u.r = r }; }
object mark = { .tag = marktype };
object null = { .tag = nulltype };
/* the global interpreter state */
typedef struct state {
unsigned char *vm; /* pointer to "virtual memory" */
address vmsz; /* max memory in use */
address save[MAXSAVE];
size level;
size_t vmmax; /* max memory allocated */
} state;
state sta;
#define VM(a) (st->vm+(a))
#define STR(o) ((char *)(st->vm+o.u.c.a))
#define HDR(o) ( (header *) (((object *)st->vm+o.u.c.a)-1) )
#define ARR(o) ((object *)(st->vm+o.u.c.a))
bool initvm (state *st) {
assert(sizeof(header) <= sizeof(object));
st->vm = mmap(NULL, st->vmmax = BLOCKSIZE*sizeof(object),
PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
if (st->vm == MAP_FAILED) return false;
st->vmsz = 0;
st->level = 0;
return true;
}
void vmdump (state *st) {
address i;
for (i = 0; i < st->vmsz; i++) {
if (i%sizeof(object) == 0) {
puts("");
printf("%05lu: ", i);
}
printf("%02X ", (int)st->vm[i]);
}
puts("");
}
address vmalloc (state *st, size n) {
address p;
printf("vmalloc requested %u\n", n);
/* pad strings so Objects remain aligned */
n += (n % sizeof(object)? sizeof(object) - n % sizeof(object):
0);
printf("vmalloc padded request to %u\n", n);
/* add one for header, one for the trailer */
n += 2*sizeof(object);
printf("vmalloc increased size to %u for header\n", n);
/* grow if necessary */
if (st->vmsz + n > st->vmmax) {
/* pointers into VM are offsets
so we can allow memory to move if desireable
*/
st->vm = mremap(st->vm, st->vmmax,
st->vmmax+BLOCKSIZE*sizeof(object), MREMAP_MAYMOVE);
st->vmmax += BLOCKSIZE*sizeof(object);
}
/* return the current offset */
p = st->vmsz;
printf("vmalloc returning %lu(%lu) for %u\n", p + sizeof(object),
p, n);
/* increase the offset by the allocated size */
st->vmsz += n;
{
object o;
o.u.c.a = p + sizeof(object);
HDR(o)->old = HDR(o)->new = 0;
HDR(o)->flags.allflags = 0;
((trailer *)st->vm+st->vmsz-sizeof(object))->magic =
0xFFFFFFFF;
((trailer *)st->vm+st->vmsz-sizeof(object))->head = p;
((trailer *)st->vm+st->vmsz-sizeof(object))->sz = n -
2*sizeof(object);
return o.u.c.a;
}
/* clear the header */
// ((header *)st->vm+p)->new =
// ((header *)st->vm+p)->old =
// ((header *)st->vm+p)->flags.allflags = 0;
// ((trailer *)st->vm+p+n-sizeof(object))->sz = n - 2*sizeof(object);
/* Since the very first allocation will return sizeof(object)
to make room for the first header, address 0 is not valid
for any vm pointers. So it can be used like a NULL pointer.
*/
//return p + sizeof(object); /* header is at p[-1] */
}
object save (state *st) {
object v = { .tag = savetype };
if (st->level == MAXSAVE) error(limitcheck);
v.u.c.n = st->level; /* stash current level in save object */
st->save[st->level++] = st->vmsz; /* push the current boundary */
v.u.c.a = vmalloc(st, 0); /* allocate just a header */
v.u.c.a -= sizeof(object); /* point at the header */
/* a valid save object will have
v.u.c.n < st->level
st->save[v.u.c.n] == v.u.c.a
*/
return v;
}
void vmrewind (state *st, address a, address bound) {
object o; /* dummy object to hold address of allocation */
o.u.c.a = a;
if (bound == 0) return;
while (HDR(o)->old && o.u.c.a > bound) {
o.u.c.a = HDR(o)->old;
}
HDR(o)->new = 0;
}
void restore (state *st, object v) {
address bound;
if (st->level == 0) error(rangecheck);
if ( (v.u.c.n >= st->level) /* save object refers to nonexistant
level */
|| (st->save[v.u.c.n] != v.u.c.a) ) /* saved boundary does not
match stacked boundary */
error(invalidrestore);
/* rewind each allocation in VM */
for (bound = st->vmsz; bound > 0 && bound >= v.u.c.a; ){
address a;
address t = bound - sizeof(object); /* t points to trailer */
a = t - ((trailer *)VM(t))->sz; /* address of allocation */
vmrewind(st, a, v.u.c.a);
bound = a - sizeof(object); /* subtract header, discarding the
allocation */
}
st->vmsz = st->save[v.u.c.n]; /* pop saved boundary */
st->level = v.u.c.n; /* set level */
}
object consstring (state *st, char *s, size n) {
object o = { .tag = stringtype };
o.u.c.a = vmalloc(&sta, n);
o.u.c.n = n;
memcpy(STR(o), s, n);
return o;
}
void s_put (state *st, object s, integer i, integer c) {
if (i < 0 || i >= s.u.c.n) error(rangecheck);
STR(s)[i] = c;
}
integer s_get (state *st, object s, integer i) {
if (i < 0 || i >= s.u.c.n) error(rangecheck);
return STR(s)[i];
}
void print(state *st, object s) {
integer i;
for (i=0; i < s.u.c.n; i++) {
putchar(STR(s)[i]);
}
}
object consarray (state *st, integer n) {
object o = { .tag = arraytype };
o.u.c.a = vmalloc(st, n*sizeof(object));
o.u.c.n = n;
return o;
}
#define OLDER(o) ( st->level==0? 0: o.u.c.a < st->save[st->level-1] )
object copy (state *st, object o) {
object n = o;
switch (o.tag) {
default: return o;
case stringtype: n.u.c.a = vmalloc(st, n.u.c.n);
memcpy(STR(n), STR(o), n.u.c.n);
break;
case arraytype: n.u.c.a = vmalloc(st, n.u.c.n*sizeof(object));
memcpy(ARR(n), ARR(o),
n.u.c.n*sizeof(object));
break;
}
HDR(o)->new = n.u.c.a;
HDR(n)->old = o.u.c.a;
return n;
}
void a_put (state *st, object a, integer i, object o) {
if (i < 0 || i >= a.u.c.n) error(rangecheck);
while (HDR(a)->new) a.u.c.a = HDR(a)->new; /* find newest copy */
if (OLDER(a)) a = copy(st, a); /* copy older than save */
ARR(a)[i] = o;
}
object a_get (state *st, object a, integer i) {
if (i < 0 || i >= a.u.c.n) error(rangecheck);
while (HDR(a)->new) a.u.c.a = HDR(a)->new; /* find newest copy */
return ARR(a)[i];
}
void a_dump (state *st, object a) {
printf("<%X:%lu:%lu> %lu %d <%u>\n", (unsigned)HDR(a)-
>flags.allflags,
HDR(a)->old, HDR(a)->new,
a.u.c.a, a.u.c.n,
((trailer *)VM(a.u.c.a+a.u.c.n))->sz );
}
bool init (void) {
return initvm(&sta);
}
#define STR_CNT(a) a, sizeof(a)-1
int main (void) {
if (!init()) { return EXIT_FAILURE; }
{
state *st = &sta;
if (1) { /* test save/restore for string */
object v;
object s;
s = consstring(st, STR_CNT("abcdef"));
print(st, s); puts("");
v = save(st);
s_put(st, s, 0, 'g');
print(st, s); puts("");
restore(st, v);
print(st, s); puts("");
}
if (1) { /* test save/restore for array */
object v;
object a;
a = consarray(st, 2);
a_put(st, a, 0, consint(3));
a_put(st, a, 1, consint(5));
printf("[ %d %d ]\n", (int)a_get(st, a, 0).u.i,
(int)a_get(st, a, 1).u.i);
a_dump(st, a);
vmdump(st);
v = save(st);
a_put(st, a, 0, consint(7));
a_put(st, a, 1, consint(9));
printf("[ %d %d ]\n", (int)a_get(st, a, 0).u.i,
(int)a_get(st, a, 1).u.i);
a_dump(st, a);
vmdump(st);
restore(st, v);
printf("[ %d %d ]\n", (int)a_get(st, a, 0).u.i,
(int)a_get(st, a, 1).u.i);
}
}
return 0;
}
Output:
vmalloc requested 6
vmalloc padded request to 12
vmalloc increased size to 36 for header
vmalloc returning 12(0) for 36
abcdef
vmalloc requested 0
vmalloc padded request to 0
vmalloc increased size to 24 for header
vmalloc returning 48(36) for 24
gbcdef
gbcdef
vmalloc requested 24
vmalloc padded request to 24
vmalloc increased size to 48 for header
vmalloc returning 48(36) for 48
[ 3 5 ]
<0:0:0> 48 2 <0>
00000: 00 00 00 00 00 00 00 00 00 00 00 00
00012: 67 62 63 64 65 66 00 00 00 00 00 00
00024: 00 00 00 00 00 00 00 00 00 00 00 00
00036: 00 00 00 00 00 00 00 00 00 00 00 00
00048: 01 00 00 00 03 00 00 00 00 00 00 00
00060: 01 00 00 00 05 00 00 00 00 00 00 00
00072: 00 00 00 00 00 00 00 00 00 00 00 00
vmalloc requested 0
vmalloc padded request to 0
vmalloc increased size to 24 for header
vmalloc returning 96(84) for 24
vmalloc requested 24
vmalloc padded request to 24
vmalloc increased size to 48 for header
vmalloc returning 120(108) for 48
[ 7 9 ]
<0:0:120> 48 2 <0>
00000: 00 00 00 00 00 00 00 00 00 00 00 00
00012: 67 62 63 64 65 66 00 00 00 00 00 00
00024: 00 00 00 00 00 00 00 00 00 00 00 00
00036: 00 00 00 00 00 00 00 00 00 00 00 00
00048: 01 00 00 00 03 00 00 00 00 00 00 00
00060: 01 00 00 00 05 00 00 00 00 00 00 00
00072: 00 00 00 00 00 00 00 00 00 00 00 00
00084: 00 00 00 00 00 00 00 00 00 00 00 00
00096: 00 00 00 00 00 00 00 00 00 00 00 00
00108: 00 00 00 00 00 00 00 00 00 00 00 00
00120: 01 00 00 00 07 00 00 00 00 00 00 00
00132: 01 00 00 00 09 00 00 00 00 00 00 00
00144: 00 00 00 00 00 00 00 00 00 00 00 00
[ 3 5 ]
vmalloc returning 12(0) for 36
vmalloc increased vmsz to 36
abcdef
vmalloc returning 48(36) for 24
vmalloc increased vmsz to 60
gbcdef
gbcdef
vmalloc returning 48(36) for 48
vmalloc increased vmsz to 84
[ 3 5 ]
<0:0:0> 48 2 <0>
00000 00000: 00 00 00 00 00 00 00 00 00 00 00 00
00012 0000C: 67 62 63 64 65 66 00 00 00 00 00 00
00024 00018: FF FF FF FF 00 00 00 00 0C 00 00 00
00036 00024: 00 00 00 00 00 00 00 00 00 00 00 00
00048 00030: 01 00 00 00 03 00 00 00 00 00 00 00
00060 0003C: 01 00 00 00 05 00 00 00 00 00 00 00
00072 00048: FF FF FF FF 24 00 00 00 18 00 00 00
vmalloc returning 96(84) for 24
vmalloc increased vmsz to 108
vmalloc returning 120(108) for 48
vmalloc increased vmsz to 156
[ 7 9 ]
<0:0:120> 48 2 <0>
00000 00000: 00 00 00 00 00 00 00 00 00 00 00 00
00012 0000C: 67 62 63 64 65 66 00 00 00 00 00 00
00024 00018: FF FF FF FF 00 00 00 00 0C 00 00 00
00036 00024: 00 00 00 00 78 00 00 00 00 00 00 00
00048 00030: 01 00 00 00 03 00 00 00 00 00 00 00
00060 0003C: 01 00 00 00 05 00 00 00 00 00 00 00
00072 00048: FF FF FF FF 24 00 00 00 18 00 00 00
00084 00054: 00 00 00 00 00 00 00 00 00 00 00 00
00096 00060: FF FF FF FF 54 00 00 00 00 00 00 00
00108 0006C: 00 00 00 00 00 00 00 00 30 00 00 00
00120 00078: 01 00 00 00 07 00 00 00 00 00 00 00
00132 00084: 01 00 00 00 09 00 00 00 00 00 00 00
00144 00090: FF FF FF FF 6C 00 00 00 18 00 00 00
[ 3 5 ]
<0:0:0> 48 2 <0>
00000 00000: 00 00 00 00 00 00 00 00 00 00 00 00
00012 0000C: 67 62 63 64 65 66 00 00 00 00 00 00
00024 00018: FF FF FF FF 00 00 00 00 0C 00 00 00
00036 00024: 00 00 00 00 00 00 00 00 00 00 00 00
00048 00030: 01 00 00 00 03 00 00 00 00 00 00 00
00060 0003C: 01 00 00 00 05 00 00 00 00 00 00 00
00072 00048: FF FF FF FF 24 00 00 00 18 00 00 00
1142(1)12:06 AM:mark2 0>
Now we're cooking!
> Yes but string contents are immune to save/restore.
I do not see any evidence in PLRM. Did you get this information by
trial and error?
> Eg.
>
> (abcdef)
> save exch
> dup 0 (ghi) putinterval
> exch restore
> ==
>
> should print (ghidef).
Why? It's a composite object in VM an therefore it's restored by
"restore". That's what I see in PLRM. At least 3rd Ed.
Regards,
-Helmar
The PLRM, 3rd ed., states, among other places, restore does:
...
• Resets the values of all composite objects in local VM, except strings, to
their
state at the time of the save
Helge
Wow, implicit at 3.7.3 - the operator definition does not mention it
directly.
That's basically a great disturbance of the overall concept - well, we
expect something like this from Adobe.
What I do not understand with it is, that "strings" are explicit in
VM. How to do pointer-oriented restoring of VM in this case? It has to
be real GC - or I'm wrong?
Regards,
-Helmar
> What I do not understand with it is, that "strings" are explicit in
> VM. How to do pointer-oriented restoring of VM in this case? It has to
> be real GC - or I'm wrong?
I must correct me here. Basically if the string is not converted to
something that should reside outside VM (like names in level > 1),
it's completely possible to implement it without GC.
What I now wonder is
(!OK)
dup 0 32 cvx put
0 get xcheck =
should cause... Is there anything written in PLRM? About this aspect I
do not see the information.
Regards,
-Helmar
I think you should get 'false' here. The executable attribute
is not relevant for integers, and it cannot be stored in the
string. But it wouldn't surprise me if 'get' produced integers
with attributes matching those of the string from which they
came; so you still should get 'false'.
Acknowledged. I'm saving that for later.
> About your implementation: restore does not only cause previous
> values to come back or keys defined after the save to disappear. It
> can also "resuscitate" keys that were undef-ined after the save.
Well, my target is still level 1. But I should definitely start
considering issues like this. My current plan is to treat dictionaries
as flat tables with linear lookup, so restoring undef'ed entries
should be covered. But I'll have to reexamine this when I make the
dictionaries grow-able.