Freeze checkin
Dan Sugalski
Just a quick heads-up--I checked in the preliminary patch for freeze/thaw
that Leo sent me for review. It'll change internally a fair amount, and
the vtable/low-level API is going to change, but the op-level interface
will be stable. I wanted it in before things driged any further, though,
so feel free to play but expect it to change rather a lot.
Dan
--------------------------------------"it's like this"-------------------
Dan Sugalski even samurai
d...@sidhe.org have teddy bears and even
teddy bears get drunk
Jeff Clites
> Just a quick heads-up--I checked in the preliminary patch for
> freeze/thaw
> that Leo sent me for review. It'll change internally a fair amount, and
> the vtable/low-level API is going to change, but the op-level interface
> will be stable. I wanted it in before things driged any further,
> though,
> so feel free to play but expect it to change rather a lot.
Two initial concerns:
1) I have a patch which I've been assembling to do ordered destruction.
That needs to use the next_for_GC pointer (and I think any alternate
implementation would need to as well). But Parrot_freeze_at_destruct()
uses next_for_GC. I assume Parrot_freeze_at_destruct is intended to be
called from a destroy() method; if so, there's a problem, and we'd need
2 next_for_GC-ish pointers. We certainly can't have a method intended
to be called only at destruction-time which is incompatible with proper
ordered destruction....
2) For the same patch, I'm using something very similar to the
vtable->visit method used in the new freeze/thaw code (and actually my
method was called vtable->visit as well, which may be where Leo got the
name, since I had mentioned this before...). But the ordered-GC code
needs to know about "children" in the sense of any referenced PObj (ie,
children defined by what needs to be marked in a GC run), whereas the
freeze code needs to know about only "logical children" (ie, only those
which are intended to be frozen when the "parent" is frozen). So I
would modify my ordered GC code to use Leo's vtable->visit, but it has
a different notion of what children to visit, so I think we are either
going to end up with two very-similar-but-different vtable methods, or
we'll need extend vtable->visit to allow it to serve both purposes
(either by calling visit with a flag which tells it which type of
visitation to do, or by instead having a flag which is passed to the
visit_child_function to tell it whether this is a "logical" child or
merely a referenced object). Off the top of my head I think that
"freeze-children" will always be a subset of "mark-children", so
passing a flag to vtable->visit is probably cleaner.
JEff
Leopold Toetsch
> On Nov 19, 2003, at 9:04 AM, Dan Sugalski wrote:
> Two initial concerns:
> 1) I have a patch which I've been assembling to do ordered destruction.
> That needs to use the next_for_GC pointer (and I think any alternate
> implementation would need to as well). But Parrot_freeze_at_destruct()
> uses next_for_GC. I assume Parrot_freeze_at_destruct is intended to be
> called from a destroy() method; if so, there's a problem, and we'd need
> 2 next_for_GC-ish pointers.
I don't know yet, when and from where freeze_at_destruct() is run. It
seems, that objects are already dead then. That could mean, that
destruction ordering is run before.
But, even if these 2 get run at the same time, it could be possible to
share the next_for_GC pointer - ordering could (if possible) skip some
PMCs that don't need ordering.
> ... So I
> would modify my ordered GC code to use Leo's vtable->visit, but it has
> a different notion of what children to visit,
... which, AFAIK isn't carved in stone yet. We don't have a notion for
logical children that need ordered destruction. An array holding some
scalar PMCs can get destructed in any order.
We first should separate destruction (i.e. mainly freeing system memory
or maybe closing other resources) and the more HLL-like finalizing,
which might depend on other objects to get finalized first. This needs
IMHO another vtable, that if present (not default), is a sign for
"destruction ordering" or finalizing.
> ... so I think we are either
> going to end up with two very-similar-but-different vtable methods, or
> we'll need extend vtable->visit to allow it to serve both purposes
> (either by calling visit with a flag which tells it which type of
> visitation to do,
The visit vtable passes all "visited" PMCs on to a callback, which puts
these on some kind of a TODO list. visit, when called via freeze is
different to visit called from thaw. So destruction ordering can setup
just its on visit_todo_list() and work on from there. If that's not
enough, the visit vtable has all the info, that may be needed in the
visit_info structure.
> ... or by instead having a flag which is passed to the
> visit_child_function to tell it whether this is a "logical" child or
> merely a referenced object).
I need for freeze/thaw another vist_info structure member called
possibly "void *extra" anyway, to be able to e.g. call back to the
original PMC to freeze/thaw native data or to handle sparse arrays.
While I'd rather have the visit vtable free of such functionality, it
seems not achievable. So if destruction ordering has to pass some info,
we'll just do it.
> JEff
I'll try to put together some docs tomorrow, how it currently works and
what extensions will be necessary to get freeze/thaw running for all
currently known PMCs. We can have a look then, how destruction ordering
fits in that scheme.
leo
Jeff Clites
> Jeff Clites <jcl...@mac.com> wrote:
>> On Nov 19, 2003, at 9:04 AM, Dan Sugalski wrote:
>
>> Two initial concerns:
>
>> 1) I have a patch which I've been assembling to do ordered
>> destruction.
>> That needs to use the next_for_GC pointer (and I think any alternate
>> implementation would need to as well). But Parrot_freeze_at_destruct()
>> uses next_for_GC. I assume Parrot_freeze_at_destruct is intended to be
>> called from a destroy() method; if so, there's a problem, and we'd
>> need
>> 2 next_for_GC-ish pointers.
>
> I don't know yet, when and from where freeze_at_destruct() is run.
Yeah, that's not clear to me either. (And I think that many other
languages/environments have concluded that trying to archive at
destruction time turns out not to be a good idea, but that's a separate
issue.)
> It seems, that objects are already dead then. That could mean, that
> destruction ordering is run before.
> But, even if these 2 get run at the same time, it could be possible to
> share the next_for_GC pointer - ordering could (if possible) skip some
> PMCs that don't need ordering.
It's possible that they could be made to work in parallel--that for a
particular object we run the freeze right before we run the destroy. I
don't think we could archive after destruction (since destruction by
design will be tearing down the object and may destroy the state we are
trying to archive).
>> ... So I
>> would modify my ordered GC code to use Leo's vtable->visit, but it has
>> a different notion of what children to visit,
>
> ... which, AFAIK isn't carved in stone yet. We don't have a notion for
> logical children that need ordered destruction. An array holding some
> scalar PMCs can get destructed in any order.
I'm not thinking of an array controlling the destruction ordering of
its children--I'm talking about the need to destroy() the array itself
before destroy()ing its children (that's just what "ordered
destruction" means). What I'm really worried about here is the case
where something like an array may have a reference to some PMC which
its using as a cache, but which is not supposed to be archived along
with the array (because it's not part of the "contents" of the array).
The GC traversals need to know about this reference, but freezing does
not.
> We first should separate destruction (i.e. mainly freeing system memory
> or maybe closing other resources) and the more HLL-like finalizing,
> which might depend on other objects to get finalized first. This needs
> IMHO another vtable, that if present (not default), is a sign for
> "destruction ordering" or finalizing.
I've been assuming that these two are tied--that for an HLL object,
whatever PMC is used to implement it would be responsible for calling
the HLL DESTROY() from the PMC's vtable->destroy. If not, then I'm
pretty confused--because any recycling of objects is going to need a
global view of what is reachable, and of what things reference what
other things. But I may be missing something, so I'm anxious to hear
your thoughts about this.
>> ... so I think we are either
>> going to end up with two very-similar-but-different vtable methods, or
>> we'll need extend vtable->visit to allow it to serve both purposes
>> (either by calling visit with a flag which tells it which type of
>> visitation to do,
>
> The visit vtable passes all "visited" PMCs on to a callback, which puts
> these on some kind of a TODO list. visit, when called via freeze is
> different to visit called from thaw. So destruction ordering can setup
> just its on visit_todo_list() and work on from there. If that's not
> enough, the visit vtable has all the info, that may be needed in the
> visit_info structure.
So to take my "cache" example from above, destruction ordering would
need for the visit_child_function to be called on the cache object, and
freeze would need for it to _not_ be. Or, it could always be passed,
but then the visit_child_function would need to be able to know "this
object is referenced but not logically contained". Either way, it's the
vtable->visit which has the information--the question will just be how
it "communicates" it to the visit_child_function.
>> ... or by instead having a flag which is passed to the
>> visit_child_function to tell it whether this is a "logical" child or
>> merely a referenced object).
>
> I need for freeze/thaw another vist_info structure member called
> possibly "void *extra" anyway, to be able to e.g. call back to the
> original PMC to freeze/thaw native data or to handle sparse arrays.
> While I'd rather have the visit vtable free of such functionality, it
> seems not achievable. So if destruction ordering has to pass some info,
> we'll just do it.
As I had originally thought of it, the signature was going to be:
vtable->visit(visit_child_function, void* extra_info)
With this, ordered destruction would pass it's
callback-for-referenced-objects as the visit_child_function (and this
would internally accumulate objects via the next_for_GC list), and NULL
for extra_info. And for freeze, another child callback would be passed,
as well as a pointer to the struct you defined as extra_info. With
this, non-freeze users of visit would not need to set up this struct,
and other uses we haven't thought of yet could pass in whatever sort of
context info they need.
I think this would make the use of visit be a little more general (and
less specific to freeze/thaw), but we can discuss this. In particular,
as I had thought of it vtable->mark would be re-implemented in terms of
vtable->visit, since ordered destruction and mark both care about
_exactly_ the same "child" objects. In that case, visit_child_function
would basically be pobject_lives().
> I'll try to put together some docs tomorrow, how it currently works and
> what extensions will be necessary to get freeze/thaw running for all
> currently known PMCs. We can have a look then, how destruction ordering
> fits in that scheme.
That sounds good. I'll wait for the docs, and then we can discuss it
more.
But I'm pretty optimistic that we've converged on very similar
approaches to two different problems, and I think we'll be able to
arrive at a unified solution to both.
JEff
Leopold Toetsch
> Hey folks.
> Just a quick heads-up--I checked in the preliminary patch for freeze/thaw
> that Leo sent me for review. It'll change internally a fair amount ...
I've put together a description of the current state. I'd like to thank
all people for their contributions and discussion. This nasty piece of
code wouldn't exist without it...
Additions, proposals, changes, money, corrections are all welcome.
> Dan
leo
Leopold Toetsch
> On Nov 19, 2003, at 1:34 PM, Leopold Toetsch wrote:
> ... What I'm really worried about here is the case
> where something like an array may have a reference to some PMC which
> its using as a cache, but which is not supposed to be archived along
> with the array (because it's not part of the "contents" of the array).
> The GC traversals need to know about this reference, but freezing does
> not.
Seems to me like kind of a weak reference. GC would have to mark the
hash and the keys, but not the values. We will need special PMCs for
such cache-like things, where the visit() vtable does the right thing.
>> We first should separate destruction (i.e. mainly freeing system memory
>> or maybe closing other resources) and the more HLL-like finalizing,
>> which might depend on other objects to get finalized first. This needs
>> IMHO another vtable, that if present (not default), is a sign for
>> "destruction ordering" or finalizing.
> I've been assuming that these two are tied--that for an HLL object,
> whatever PMC is used to implement it would be responsible for calling
> the HLL DESTROY() from the PMC's vtable->destroy.
For the HLL case yes. But we are currently "finalizing" e.g. ParrotIO
objects too. We should separate vtable->destroy() (free system memory)
and vtable->destruct() or vtable->finalize(). The latter might have
ordering issues and has to be done always on each interpreter shutdown.
The former hasn't to be done for the last interpreter.
> ... If not, then I'm
> pretty confused--because any recycling of objects is going to need a
> global view of what is reachable, and of what things reference what
> other things. But I may be missing something, so I'm anxious to hear
> your thoughts about this.
We need a notion for a PMC logically referencing another PMC and for an
orderly destruction of such containers. As said, an array holding
PerlInts doesn't impose any ordering constraints. An array of objects
does (or may do).
> ... In particular,
> as I had thought of it vtable->mark would be re-implemented in terms of
> vtable->visit, since ordered destruction and mark both care about
> _exactly_ the same "child" objects. In that case, visit_child_function
> would basically be pobject_lives().
While this is true, it will very probably not happen for performance
reasons. DOD is called thousands of times during normal operation and
has to be as fast as possible. All other functionality using visit()
isn't really that time-critical.
> That sounds good. I'll wait for the docs, and then we can discuss it
> more.
docs/dev/pmc_freeze.pod
> But I'm pretty optimistic that we've converged on very similar
> approaches to two different problems, and I think we'll be able to
> arrive at a unified solution to both.
Yep, that was/is the goal.
> JEff
leo