Library loading and initialization sequence
Dan Sugalski
as to what I'm doing.
Right now, I'm working on loading PMCs, both dynamic and static, and the
initialization sequence they go through. I know we've got some of this
down already, but adding in MMD and the need to initialize tables and
potentially defer things for later class loading means I'm needing to
wedge things in, so the whole process seems like it ought to be
documented. Once any issues with this design are cleared up, we'll do
that. :)
Firstly, we're adding a new method INIT to the .pmc file. *If* it exists
when the .pmc source is parsed, it will be called as the very last action
of the initialization routine. It gets two parameters, the interpreter and
the number assigned to the pmc class. It may do whatever the heck you
want, though generally I expect they'll install MMD functions.
Next, the sequence of loading.
Right now there's a load_pmc op, which goes along with the load_opcode_lib
and loadlib ops. I'd like to unify that--I'm not too thrilled with having
three different "load this external C library and call custom routine X",
so... we're not going to do that. We are, instead, going to toss
load_opcode_lib and load_pmc, and ramp back to a single loadlib opcode.
All loadlib does is use the dynamic loading facilities of the platform to
load in the library.
When we load a library, the bytecode should probe for and, if it exists,
call Parrot_lib_load_%s, where %s is the library name. It takes a single
parameter, the interpreter, and returns a PMC representing the library
that we can query later.
The bytecode should also probe for and, if it exists, call
Parrot_lib_init_%s, where %s is the library name. This takes two
parameters, the interpreter and the PMC that _lib_load_ returned, and
returns a single PMC that represents the per-thread data for the library.
If, and only if, Parrot_lib_init_%s exists, then the bytecode should call
the thread_register op, which I'll add. This takes two parameters, the
_init routine PMC and the _load returned PMC, and stores it away. Whenever
this interpreter is cloned, the function will be called and passed in the
new interpreter and the _load PMC, so the library can do per-thread
initialization.
All registration of opcode libraries, PMCs, and whatnot should be done
from within the _load routine. If a library has a dozen different PMCs,
well... great. Load 'em all in. We don't do any filtering or whatnot here
(I think) just calling it and letting it do its thing. If we want to
define some sort of language conventions, great, we can do that.
I'm definitely up for the argument that this should all be in a single op,
and that loadlib should load up the library and conditionally call the two
functions, as well as doing only-once checking so the libs don't get
multiply loaded. If you want to make a case, well, now'd be a good time
for that.
I'll get to the PMC registration bit in another message.
Dan
Dan Sugalski
> Firstly, we're adding a new method INIT to the .pmc file. *If* it exists
This, by the by, corresponds to the _init method in loading, and is called
on each thread instantiation. We're also going to add a LOAD method, again
optional, which will be called as part of the _load sequence.
Hopefully the PMC-specific parts of this will be more coherent...
Dan
Leopold Toetsch
> Next, the sequence of loading.
> Right now there's a load_pmc op
Gone.
> and loadlib ops. I'd like to unify that
Done.
> When we load a library, the bytecode should probe for and, if it exists,
> call Parrot_lib_load_%s, where %s is the library name.
If it doesn't exist, it assumes a native library, this is what the
sequence loadlib / dlfunc is expecting, --done ...
> , the interpreter, and returns a PMC representing the library
> that we can query later.
... until here. What kind of PMC is e.g a PMC-library intended to
return? As it seems, that the Parrot_lib_init_%s code is called again on
interpreter forking, the PMC has to hold the library handle. Now, when
Parrot_lib_load_%s should construct the PMC, it can't place the handle
into the PMC. Of course, the dynext.c:Parrot_load_lib() could stuff the
handle into the PMC, if the returned PMC is appropriate for that.
I have an optional (currently unused, NULL) 3rd argument to
Parrot_load_lib, which could be an initializer, if needed, so that
loadlib P1, "the_lib", P2
would be possible too.
> Dan
leo
Leopold Toetsch
> Right now there's a load_pmc op, which goes along with the load_opcode_lib
> and loadlib ops. I'd like to unify that
I'll start the opcode load thingy, based on my experimental code I has
posted here mid-March.
The basics are:
- assembler and runtime must see the C<loadlib> "oplib" in the same
sequence, because
- the opcode numbers are assigned dynamically:
When you load an oplib containing 100 ops, they get op numbers
e.g. 1206 ... 1305. Next oplib starts at 1306 ...
- on load all runcores are notified to use the new oplib function
or address table
There are for sure issues e.g. when you run the CGP core and the loaded
lib has functions only, but a default handler in the jump table can
call these functions then. The switched core also need this default
handling.
> ... Parrot_lib_load_%s ...
> ... returns a PMC representing the library
Any hints, what kind of PMC that shall be?
An new Parrot_Lib PMC with the dll handle attached + some flags about
the library and its contents?
> Dan
leo
Dan Sugalski
> - the opcode numbers are assigned dynamically:
> When you load an oplib containing 100 ops, they get op numbers
> e.g. 1206 ... 1305. Next oplib starts at 1306 ...
> - on load all runcores are notified to use the new oplib function
> or address table
I've been thinking about this, and I've decided that loading oplibs and
installing those oplib functions in the current table should be separate
things. The load can do whatever it wants to initialize, but when the
functions are installed in the opcode dispatch table that's when the
interpreter says where the functions should go.
> There are for sure issues e.g. when you run the CGP core and the loaded
> lib has functions only, but a default handler in the jump table can
> call these functions then. The switched core also need this default
> handling.
>
> > ... Parrot_lib_load_%s ...
> > ... returns a PMC representing the library
>
> Any hints, what kind of PMC that shall be?
> An new Parrot_Lib PMC with the dll handle attached + some flags about
> the library and its contents?
At the moment, no. I knew we'd want a thing to hang stuff off of, hence
the PMC mandate. I'm not sure what that stuff should be yet, nor what we
might want to make it do. We can throw it into a ParrotLibrary PMC and
think about it as we go.
Dan
Leopold Toetsch
> On Fri, 10 Oct 2003, Leopold Toetsch wrote:
>> - the opcode numbers are assigned dynamically:
>> When you load an oplib containing 100 ops, they get op numbers
>> e.g. 1206 ... 1305. Next oplib starts at 1306 ...
>> - on load all runcores are notified to use the new oplib function
>> or address table
> I've been thinking about this, and I've decided that loading oplibs and
> installing those oplib functions in the current table should be separate
> things. The load can do whatever it wants to initialize, but when the
> functions are installed in the opcode dispatch table that's when the
> interpreter says where the functions should go.
How or from where is the second stage then done?
Currently it would look like this:
loadlb P1, "myops_ops"
fortytwo I0
C<loadlib> calls F<Parrot_load_lib> which calls F<Parrot_lib_<%s>_load>.
This would call back and run C<oplib_register>. That installs the opcode
dispatch tables.
The assembler must do the same thing (as it does with dynamic PMCs) to
get the same opcode numbers (or class enums respectively).
We could actually defer installing the opcode dispatch table at runtime,
but we must reserve the opcode number range, and the assembler must
at least be able to query the dynamic lib for valid opcode names. When
the first unknown opcode is to be run, then the real functions could be
put in place.
> Dan
leo
Nicholas Clark
> Dan Sugalski <d...@sidhe.org> wrote:
> > On Fri, 10 Oct 2003, Leopold Toetsch wrote:
>
> >> - the opcode numbers are assigned dynamically:
> >> When you load an oplib containing 100 ops, they get op numbers
> >> e.g. 1206 ... 1305. Next oplib starts at 1306 ...
> >> - on load all runcores are notified to use the new oplib function
> >> or address table
What happens if the opcode library is rebuilt with more ops after the
bytecode using it is compiled and frozen to disk?
Even if the new ops are numbered (correctly) after the existing ops,
if the loader just allocates op numbers on loading continuing after
the last op number of the previously loaded library, than as soon as
any op library gains an extra op, anything loading after it goes horribly
wrong.
(Likewise if an op library keeps the same number of ops, but itself
gains a new dependency on an external op library)
> How or from where is the second stage then done?
>
> Currently it would look like this:
>
> loadlb P1, "myops_ops"
> fortytwo I0
>
> C<loadlib> calls F<Parrot_load_lib> which calls F<Parrot_lib_<%s>_load>.
> This would call back and run C<oplib_register>. That installs the opcode
> dispatch tables.
>
> The assembler must do the same thing (as it does with dynamic PMCs) to
> get the same opcode numbers (or class enums respectively).
> We could actually defer installing the opcode dispatch table at runtime,
> but we must reserve the opcode number range, and the assembler must
> at least be able to query the dynamic lib for valid opcode names. When
> the first unknown opcode is to be run, then the real functions could be
> put in place.
Or am I getting confused and all this lookup is done by name?
Nicholas Clark
Leopold Toetsch
> On Fri, Oct 10, 2003 at 05:44:05PM +0200, Leopold Toetsch wrote:
>> >> - the opcode numbers are assigned dynamically:
> What happens if the opcode library is rebuilt with more ops after the
> bytecode using it is compiled and frozen to disk?
Its as invalid as currently bytecode is, when the fingerprint
doesn't match. We must at least assure, that the opcount didn't change.
This whole can of worms is related to version checking. We probably need
a Version PMC that is used for all items loaded at runtime. Its a
generalization of the current PBC fingerprint.
>> We could actually defer installing the opcode dispatch table at runtime,
>> but we must reserve the opcode number range, and the assembler must
>> at least be able to query the dynamic lib for valid opcode names. When
>> the first unknown opcode is to be run, then the real functions could be
>> put in place.
> Or am I getting confused and all this lookup is done by name?
No. Its the same scheme I proposed for the event checker code. When an
event gets scheduled the opfunc dispatch tables get entries that
actually handle the event. When new opcode numbers are reserved, their
function or address tables get entries that install the lib in the
tables. The C<loadlib "oplib"> actually would schedule an event which is
handled when the first loaded opcode is to be run.
> Nicholas Clark
leo