Okay, time to think about this.

We need the ability to layer PMCs. Nothing new, we need something of
the sort for transparent read-only-ness and probably thread-safety
(though we don't have to do it that way) and folks are going to want
to do undoable custom vtable layering. While we don't *have* to let
'em do that, I'm inclined to if only because a good general-purpose
mechanism will make things less error-prone overall.

So, then, the question is... how? I've got two schemes, neither of
which I completely like, so I'm up for some discussion and more ideas.

The first scheme is with simple layered PMCs. You throw a read-only
layer on a PMC and we allocate a new PMC where all the write vtable
entries throw an exception and all the other vtable entries delegate
to the PMC in the next layer below, and we hang that PMC off our
read-only PMC's data pointer.

The second scheme is somewhat more complex. When we mark a PMC as
read-only we allocate a new PMC, copy the contents of the current PMC
to the new PMC, and rejig the internals of the current PMC to have
the read-only vtable. Again, write entries throw an exception, reads
get delegated to the original PMC which is hanging off the data
pointer.

The problem with the first scheme is that anything that has a handle
on the PMC will not get the new layers. Not a good thing.

The problem with the second scheme is that anything that peeks at the
internals of the PMC (like the ultimately wrapped PMC) will either
look at the wrong internals (if it vectors off the PMC pointer) or
will need an extra layer of indirection to get at the internals,
since we'll need to walk down through potentially several layers to
get to the ultimate real PMC.

We could do some sort of merged vtable thing, but besides being
somewhat memory expensive (as we need a new table for each layered
PMC) it also makes it tough to delegate to an inner layer's vtable
function that you've overridden, so that's not good either.

I'm up for opinions or other suggestions.
--
                                         Dan

--------------------------------------"it's like this"-------------------
Dan Sugalski                          even samurai
d...@sidhe.org                         have teddy bears and even
                                       teddy bears get drunk

You are considering read-ony PMC versus others. Another issue
is  properties. Many PMCs "classes" will support properties that will
alter their behavior. But most PMC instances will have no property
attached to them. Or just default values of them if you see it that
way.

To avoid to go back the slow world of perl5 where many things
need to be tested before figuring what to do, performancewise,
there is little choice but to have two versions of each PMC
class, a fast one that is devoid of property support and another
slow one that does support them.

Note to the newcomers: A property is information dynamically
attached to a PMC. At the Perl6 language level, it is for example
a boolean value attached to a Perl integer: '0 but true'. At the
PMC level, a property is set by setprop() that by default adds
info to a hash, but could well setup flags in the pmc if it is
more appropriate for a given property and pmc type.

--
 stef

Turn the vtable field into an array (C-style--an array, not an Array),
and add an INTVAL C<layerno> parameter to all vtable functions.
vtable[0] gets all calls, and can pass them on to vtable[1], etc. etc.
This requires an additional parameter to each vtable function, but keeps
you from having to allocate an entire PMC for each layer, at the cost of
possibly reallocating the array when you want to push a new layer.  (A
smart system would probably allocate a couple extra slots, to cover the
common layering cases.)

A linked list of vtables is also possible, but you'd probably need to
pass the C<next> pointer to all of the vtable methods, so it doesn't
keep you from needing an extra parameter.

If these are untenable, I vote for "allocate a new PMC, copy the old
PMC's guts into it, and set up the layer in the old PMC".  External
references to the PMC are more important than internal ones.

--
Brent "Dax" Royal-Gordon <br...@brentdax.com>
Perl and Parrot hacker

Oceania has always been at war with Eastasia.

What about the current implementation [1]:
* PMCs that have read-only variants have the C<const_too> flag set
* the PMC compiler creates a second set of vtables with all mutating
  vtable slots set to functions that throw exceptions
* P0 = new Const<pmc_name> creates a PMC in the constant PMCs header pool
* but the vtable is the plain one first
* so initialization code can set PMC's contents
* setting the "_ro" property to 1 finally switches in the constant
  vtable

This scheme is basically working. I didn't look, if MMD changes did
break it, though. It's implemented for the SArray PMC for testing.

There are some unimplemented things WRT constant "containers". The
problem is that all items that a constant PMC refers to, have to be
constant too. That's the same problem as with thread-safe, shared PMCs.
I.e. setting the container as read-only is exactly the same operation as
setting a PMC as shared: it has to cover all contained items too.

E.g. Sub PMCs where constructed as constant PMCs for some time. But
setting properties on such a constant Sub PMC causes DOD errors, because
the property hash isn't constructed with constant buffers yet.

We could of course construct constant PMCs in the non-constant PMCs
header pool too, but having long-lived constant items in a separate header
pool would speed up DOD a bit, because these header pools aren't scanned
for dead objects.

Finally user code could have short-lived read-only PMCs too, which would
need construction in the plain PMC header pool.

leo

[1] classes/sarray.{c,pmc}
    t/pmc/sarray.t

We basically have that already. A plain PerlInt doesn't have the PMC_EXT
structure attached. During DOD it's marked in the fast path by just
setting the live bit.

When you attach a property to the PerlInt, it get's "upgraded" to a
"fat" PMC with the PMC_EXT structure appended, where the property hash
is located. DOD marking goes now through the next_for_GC pointer.

The difference is around a factor of 8 for the DOD mark phase and huge
amounts of live PMCs.

Creating two distinct PMC classes isn't necessary. This would only
consume a lot of memory e.g. for the MMD tables.

leo

Combinatorial explosion.  This already requires four variants (normal,
const, threadsafe, const threadsafe), and we haven't even seen what real
users will want to do with PMCs yet.  (I could imagine, for example,
people wanting to push a transcoding layer onto a string or aggregate
PMC, which forces all incoming strings into a certain encoding.  Or a
layer that lowercases incoming keys.  Or, or, or...)

--
Brent "Dax" Royal-Gordon <br...@brentdax.com>
Perl and Parrot hacker

Oceania has always been at war with Eastasia.

Brent 'Dax' Royal-Gordon <br...@brentdax.com> wrote:

Not quite. A const PMC is already thread-safe. But there are basically 3
variants of a PMC, yes. The question is, if these need distinct PMC
types (which would need separate MMD slots), or if we can find a scheme
to just attach a different vtable, that does the right thing.

These layers are implemented at the IO level as I/O layers. If such a
layer should really be PMC specific, it could be implemented as a
distinct object.

leo

At 10:23 PM +0200 5/29/04, Stéphane Payrard wrote:

Yup. Read-only was an example, since having
something concrete to hang this stuff off of
seems to help.

Properties, though, will likely not be done in
this way, at least not too many of them. The way
Larry's got them all spec'd out they'll end up
doing anonymous subclassing things rather than
vtable layering in most cases.

--
                                         Dan

--------------------------------------"it's like this"-------------------
Dan Sugalski                          even samurai
d...@sidhe.org                         have teddy bears and even
                                       teddy bears get drunk

At 11:55 AM +0200 5/30/04, Leopold Toetsch wrote:

Leo, you're missing the point pretty badly here--the read-only bit
was an example, though I really don't want to have to reimplement
read-only checking in every PMC class that ever gets written.

What I want to try and deal with is layering vtables. Whether we use
them for read-only things (which, FWIW, is *not* the same thing as
constant-ness--constant PMCs are certainly read-only, but read-only
PMCs aren't necessarily constant, since the read-only marker can be
added and removed from a PMC at runtime if it's not a constant PMC)
--
                                         Dan

--------------------------------------"it's like this"-------------------
Dan Sugalski                          even samurai
d...@sidhe.org                         have teddy bears and even
                                       teddy bears get drunk

Hmm, good point.  (I guess you can't set properties on a const PMC,
right?  Might we want to, for internal purposes like caching
expensive-to-calculate data?)

Indeed.  I just think that the idea can be extended beyond I/O to all PMCs.

 > If such a

Absolutely.  But if all I want is something that makes any
string-indexed aggregate case-insensitive, why should I have to code a
bunch of brand-new PMC types?  Pushing a layer on top of the existing
types makes more sense.

There are other cases like this, too.  For example, imagine a layer that
transparently compresses strings passed to set() functions, and then
decompresses them when they come out through get() functions.  This
could be useful with any disk-backed PMC, like a DBM hash, or simply
used to save RAM (assuming you have a few very, very large strings).
Once again, why force you to customize a bunch of PMC types when you
could write it once and push it on as a layer?

Can this be done with special subclasses?  Sure.  But if we do it with a
true layering system, we get an incredible amount of power essentially
for free.

--
Brent "Dax" Royal-Gordon <br...@brentdax.com>
Perl and Parrot hacker

Oceania has always been at war with Eastasia.

That sounds suspiciously like Perl 6 roles.

-- c

I like the first scheme. The question that comes up is: when does
something get layered?

That is: if I have code that says:

        new_thread_increment_every_minute(foo)
        become_read_only(foo)

Then, you have two choices of semantic:

1. foo is read-only retroactively and will throw an exception in one
minute
2. You've given the new thread a read-write version of foo, and the
read-only version created after that now has the property of changing
every minute.

I would see this as being very useful for several types of read-only
access to data that DOES change (an accumulator for a random number
entropy pool, for example).

High level languages on the other hand, should probably not expose this
directly. They will create a variable and tag it as read only at the
same time, and to the programmer there's no difference.

If they do allow for run-time read-only-ification, they can always build
their own high-level abstraction around this core PMC.

The only problem I see with this is that high level languages might want
to cache the value of a read-only variable in a typed register. If read
only really is read only, that's valid, but if it's only an interface
restriction it's not.

There you're going to have some semantic boundaries between languages
that might be unfortunate. How much of a problem that is, I'm not sure.

As for threading, I think the simple layering is easiest, and again, you
create the PMC layered if you want that functionality (e.g. for
locking).

--
Aaron Sherman <a...@ajs.com>
Senior Systems Engineer and Toolsmith
"It's the sound of a satellite saying, 'get me down!'" -Shriekback

   The second scheme's major advantage (preserving the address of the logical
(user-code-visible) PMC, so we don't have to run around updating symbol
tables, or worse yet arrays, every time we wrap a PMC) seems IMHO to be pretty
overwhelming.
   AFAICT, it shouldn't really matter much that peeks at the logical PMC's
internals may be incorrect (with one major caveat: if the _vtable_ functions
try to use the PMC's address as an index into some data structure, things will
go wrong since the PMC the base vtable functions get no longer has the same
address as the logical PMC; if external things want to do so, that's fine
though, since the logical PMC's address hasn't changed). After all, AIUI no
one but the vtable functions is supposed to be poking inside the PMC, and the
wrapper PMC (which happens to have the same address as the original PMC) will
of course pass the new wrapped PMC down when it redispatches. (by MMD, I
trust? Note that I think this scheme requires that the wrapper PMC types
dispatch like the _base_ type (the type at the bottom of the layer stack, the
original PMC) that they are wrapping (which means dynamic vtable and
multimethod generation for each new wrapper/base type combo or some such so
that the types work out correctl).)
   This should even work correctly for things like Continuations as long as
the opcodes don't try to be too smart. And I can definitely see
(auto-)wrapping Subs as possibly being a reasonable (not too much
code-duplication) way to implement privilige-domain-crossing checks for a
secure Parrot.

-- BKS

I don't think so. I've described a static scheme in terms of read-only PMCs.

1) static read-only layering
Needs one extra vtable per PMC. All mutating vtable entries point to a
default read-only implementation that throws an exception. Non-mutating
vtable slots point to that of the PMC.
1a) constant read-only is a special case of 1)

2) static thread-safe layering
Needs one extra vtable plus a thread-safe vtable function implementation
for each non thread-safe operation per PMC.

3) dynamic layering
Your second scheme. Needs one wrapper PMC per layer type and a vtable
slot to wrap the PMC.
E.g. The SharedRef PMC class is such a layer. It delegates vtable to the
contained PMC and has LOCK()/UNLOCK() around the call. The wrapping is
done by the vtable->share() function, which has to copy the PMCs data to
the newly created one and swap vtables.
This adds one indirection to get the contained PMC plus one additional
vtable call per layer. It's slower but uses less (mainly code) memory.

4) General purpose dynamic layering
If we really need it, it's basically like 3)

The static scheme especially for read-only PMCs is simple. Its just "put
in a different vtable".

Your scheme number one isn't usable - it would "move" the PMC.

leo

At 12:54 AM -0400 6/3/04, Benjamin K. Stuhl wrote:

Well... I'm not sure that the wrapped PMC is what gets passed on,
which is where things get tricky.

If it does, then access to the internal bits works out just fine. On
the other hand, any vtable redispatch (of which there's a
not-inconsiderable amount) doesn't work out right. For example, for
an Integer PMC with bytecode vtable functions:

     .sub __set_string_native prototyped, method
       .param string value
       $I0 = value
       self = $I0
     .end

if we pass in the inner PMC to that, and someone's thrown a wrapper
that intercepts the integer set then the intercept wouldn't get
triggered, and I think that's probably a bad thing.

Unfortunately doing this makes getting to the guts of the PMC tricky.
It also means you can only have one of each layer type layering a
PMC, since all the non-terminal layers will have to search down from
the top until they find the right PMC, which'll involve doing vtable
type comparisons or something like that, which isn't too good either.

I'm not sure there's a good answer here, and *something* is going to
take a speed hit. (Passing in the base pmc pointer along with the
current pmc pointer has its own costs, since extra parameters aren't
free either) I'm just not sure where we want that hit to be.
--
                                         Dan

--------------------------------------"it's like this"-------------------
Dan Sugalski                          even samurai
d...@sidhe.org                         have teddy bears and even
                                       teddy bears get drunk

All~            

Perhaps I am missing something in this discussion, but wouldn't it be
possible to have the vtable within a PMC be layered rather than the PMC
itself.  This would allow the data portion of PMC's to remain in a fixed
  location, while allowing new vtable layers to be pushed/popped.  If
what I am suggesting is an entirely different beast, than I suppose I
would appreciate an explanation of the differences.  But I would imagine
that one could simply place a new vtable in the PMC and that vtable
functions would have the ability to call the layer below them (possible
after doing something like obtaining a lock).  Or they could throw an
exception and thus prevent certain vtable slots from being accessed.

Hope that makes sense,
Matt

  While this solves the problem of layering the functions, just being able
to call multiple functions isn't that useful if there's nowhere to put
private data. You can't put it in the vtable, since we _really_ want those
to be shared between PMCs, and you can't put it in the PMC, since its data
space is owned by the bottom-most layer (the original, unwrapped PMC). In
perl5, this is solved by chaining MAGIC* layers, which are basically {
vtable, closure, next } triplets (although their actual structure is a bit
more complex and also has some internal bookkeeping: see perl5's mg.h at
  http://public.activestate.com/cgi-bin/perlbrowse?file=mg.h
for the details) - but, as perl5 has shown, that route leads to madness
rather quickly. As could be expected for playing with dark MAGICs. ^_^

-- BKS

Well, there is a way to take care of things reasonably cleanly, for the
low, low cost of only one extra pointer per PMC! :-P

Add an ->toplevel_pmc pointer to all PMCs, and call all your vfuncs on that,
rather than on the PMC itself.

Then you _can_ just pass down the wrapped PMC, and things Should Just
Work: each level can ignore any levels above it and has its own PMC as a
closure,
and it can find both the top of the stack to call vfuncs on and its
wrapped PMC to delegate down to. Pushing or popping a layer just means that
the entire stack's worth of ->toplevel_pmc pointers have to be updated to
point at the new top level.

Speed or space. If the computer's in a good mood, it'll let you choose
one. ^_^

-- BKS

At 3:01 PM -0400 6/4/04, Matt Fowles wrote:

There are two problems with that (though I did think about it for a while):

1) It means that each layering on each PMC needs to have a full
vtable copy, which is somewhat expensive in memory and setup time.

2) It means that each layer can't have its own private data attached
to the PMC. Which isn't a deal-breaker, certainly, but it does limit
the layering somewhat. (Though there are still properties that could
be exploited for this purpose)

--
                                         Dan

--------------------------------------"it's like this"-------------------
Dan Sugalski                          even samurai
d...@sidhe.org                         have teddy bears and even
                                       teddy bears get drunk