[RFC] Dynamic PMC Classes
Leopold Toetsch
- new subdirectory /dynclasses
- small hack for classes/pmc2c.pl to consider this directory too
- dynclasses/foo.pmc, dynclasses/Makefile (unportable, but short ;-)
That works fine so far, it builds F<foo_pmc.so>.
Now we would need:
- an allocated *Parrot_base_vtables [1]
- the count of available PMCs (current enum_class_max)
- and finally an opcode to load and initialize that
I'm thinking here of the currently unused C<loadext> opcode, which
could be expanded, to load all kinds of extensions.
This opcode would need additionally:
- a flag, stating e.g. a dynamic PMC is loaded
- a signature for the init function to call
[1] per interpreter, because different threads might have different pmc
classes loaded?
Comments welcome,
leo
Dan Sugalski
>I have started looking at dynamic classes. I have currently
>- new subdirectory /dynclasses
>- small hack for classes/pmc2c.pl to consider this directory too
>- dynclasses/foo.pmc, dynclasses/Makefile (unportable, but short ;-)
Here's what I was thinking instead.
We build classes as shared libraries, unless otherwise noted. PMC
libraries get loaded up with an op--load_pmc or something of the
sort. It takes a PMC name and tacks on the library prefix and suffix
as need be. We load in the PMC library, call its setup routine, call
its initialization routine, and let it do its thing. The setup
routine, when called, will at the least register the PMC's class in
the class name hash. There should be some code in the system to
manage this, though it needs mutex protection.
The setup routine is called on library load. The initialization
routine is called when the class is instantiated into a new
interpreter. Setup stuff is done only once, initialization stuff is
potentially done multiple times, though only once per
interpreter--setup is for any initial load things (like linking to
databases or whatever) while the initialization routine is for when a
thread loads up a PMC class for the first time, but that class has
already been loaded by another thread.
What we need to do is:
*) Determine the init and setup routine names
*) Make some standard for library paths and names
*) Figure out how we want to handle versioning
--
Dan
--------------------------------------"it's like this"-------------------
Dan Sugalski even samurai
d...@sidhe.org have teddy bears and even
teddy bears get drunk
Christian Renz
about, but Dan's suggestions made it a lot clearer to me. This seems
like one possible way to attach something like wxWindows to parrot.
I'm wondering about one thing, however:
>We build classes as shared libraries, unless otherwise noted. PMC
Are there any plans to allow PMCs to be implemented in Parrot? Or am I
asking something stoopid :).
Greetings,
Christian
--
cr...@web42.com - http://www.web42.com/crenz/ - http://www.web42.com/
Jos Visser
> Are there any plans to allow PMCs to be implemented in Parrot? Or am I
> asking something stoopid :).
There are currently a lot of PMC types implemented directly in Parrot.
If you look in the classes/ subdirectory of the Parrot distribution you
see lots of files named "perlint.pmc" and so forth. These are the
builtin PMC types...
The fun about dynamically loadable PMC types is that if a language L
needs/wants its own interesting implementation of a datatype T so that
the L compiler writers can generate PIR like:
P1=new LT
then there is no need to extend the core Parrot engine (and clutter all
non-L users with the necessity to carry the LT implementation around).
Instead, one could do
load_ext "LT" # or whatever
P1=new LT
to dynamically load the implementation of the LT data type from a shared
library (or DLL, or LPA load module, or whatever your operating system
groks).
++Jos.es
--
ek is so lug jy vlieg deur my
sonder jou is ek sonder patroon
"Breyten Breytenbach"
Jos Visser
> Thanks for the clarification. Does that mean that a mechanism for
> dynamic PMCs would automatically allow them to be written in Parrot
> also (and not only load binary libs)?
I don't think there are currently plans to allow PMC types to be written
in Parrot. But others are more definitive guru's on that...
Dan Sugalski
>I don't pretend that I fully understand what dynamic classes are
>about, but Dan's suggestions made it a lot clearer to me. This seems
>like one possible way to attach something like wxWindows to parrot.
>
>I'm wondering about one thing, however:
>
>>We build classes as shared libraries, unless otherwise noted. PMC
>
>Are there any plans to allow PMCs to be implemented in Parrot?
Absolutely, yes. PMC classes don't have to be written in C.
Christian Renz
dynamic PMCs would automatically allow them to be written in Parrot
also (and not only load binary libs)?
--
Leopold Toetsch
> At 12:43 +0200 7/30/03, Leopold Toetsch wrote:
>>I have started looking at dynamic classes. I have currently
>>- new subdirectory /dynclasses
>>- small hack for classes/pmc2c.pl to consider this directory too
>>- dynclasses/foo.pmc, dynclasses/Makefile (unportable, but short ;-)
> Here's what I was thinking instead.
> We build classes as shared libraries, unless otherwise noted. PMC
> libraries get loaded up with an op--load_pmc or something of the
> sort. It takes a PMC name and tacks on the library prefix and suffix
> as need be.
So we first need some more Configure support:
- runtime directory
- path separator
- $(SO) extension
(The dynclasses subdir is ment for building these libs, not for loading
them from there [1])
> ... We load in the PMC library, call its setup routine, call
> its initialization routine, and let it do its thing.
The split into C<setup> and <class_init> is necessary for our static
PMCs too. Mainly ParrotIO is using the current class_init hook to work
around initialization order and to setup its method table. [2]
> ... The setup
> routine, when called, will at the least register the PMC's class in
> the class name hash. There should be some code in the system to
> manage this, though it needs mutex protection.
IMHO: The class_setup loads the class into the global
Parrot_base_vtables. The class_init registers the class in a per
interpreter class name hash. Different threads might have different
classes loaded, but the class_enums have to be uniq.
> *) Determine the init and setup routine names
- Parrot_<classname>_class_setup
- Parrot_<classname>_class_init
The class_setup also sets the class_enum i.e vtable->base_type.
> *) Make some standard for library paths and names
- runtime/parrot/pmc
> *) Figure out how we want to handle versioning
brrr, I need a Ponie ;-)
[1] If there are no objections I'll check that part in. It shouldn't be
too hard, to finally have a perl script, that generates a platform
independend Makefile out from %PConfig.
[2] Somehow related: There was some discussion about PMC methods.
What about a standard way to allow methods for (almost) all PMCs:
- have a vtable slot for the method hash
- move find_method into default.pmc
- PMCs used in that process mainly PerlHash should better not have
methods ;-)
leo
Dan Sugalski
> > *) Determine the init and setup routine names
>
>- Parrot_<classname>_class_setup
>- Parrot_<classname>_class_init
>
>The class_setup also sets the class_enum i.e vtable->base_type.
Well... there are versioning issues there. We ought to be able to
have multiple versions of a class loaded, in which case this may not
work. I'm pretty sure that there are some platforms that require
unique names in libraries across the entire process.
>[1] If there are no objections I'll check that part in. It shouldn't be
>too hard, to finally have a perl script, that generates a platform
>independend Makefile out from %PConfig.
Nope, go for it.
>[2] Somehow related: There was some discussion about PMC methods.
>What about a standard way to allow methods for (almost) all PMCs:
>- have a vtable slot for the method hash
>- move find_method into default.pmc
>- PMCs used in that process mainly PerlHash should better not have
> methods ;-)
Yeah, that sounds familiar. I'll have to get back to you on this when
I've got more tha 5 minutes for email. :)
Leopold Toetsch
> At 11:10 +0200 7/31/03, Leopold Toetsch wrote:
>> > *) Determine the init and setup routine names
>>
>>- Parrot_<classname>_class_setup
>>- Parrot_<classname>_class_init
>>
>>The class_setup also sets the class_enum i.e vtable->base_type.
> Well... there are versioning issues there. We ought to be able to
> have multiple versions of a class loaded, in which case this may not
> work. I'm pretty sure that there are some platforms that require
> unique names in libraries across the entire process.
I think we can postpone that for later. Perl5 module versioning +
addons that handle this, have very sophisticated schemes. Though I can
just imagine to find/load a specific version of a pmc class, but not to
have different versions of one module loaded, both providing the class
"Foo".
>>[1] If there are no objections I'll check that part in. It shouldn't be
>>too hard, to finally have a perl script, that generates a platform
>>independend Makefile out from %PConfig.
> Nope, go for it.
Ok. Have checked in that a minute ago.
leo
Gordon Henriksen
> At 12:43 +0200 7/30/03, Leopold Toetsch wrote:
>
>> I have started looking at dynamic classes. I have currently
>> - new subdirectory /dynclasses
>> - small hack for classes/pmc2c.pl to consider this directory too
>> - dynclasses/foo.pmc, dynclasses/Makefile (unportable, but short ;-)
>
> Here's what I was thinking instead.
>
> We build classes as shared libraries, unless otherwise noted. PMC
> libraries get loaded up with an op--load_pmc or something of the sort.
> It takes a PMC name and tacks on the library prefix and suffix as need
> be. We load in the PMC library, call its setup routine, call its
> initialization routine, and let it do its thing. The setup routine,
> when called, will at the least register the PMC's class in the class
> name hash. There should be some code in the system to manage this,
> though it needs mutex protection.
Eek? Forcing a new DLL for every dynamically-loaded PMC class? So many
open()'s and persistent file handles and mmap regions! As if Perl 5's
DynaLoader weren't bad enough already. I can hear the clatter of
keyboards across the world. I can't make out the rest, but it all starts
with su and then echo `perl -e 'print 2**32-1, "\n"'` > /proc/sys/....
Maybe this is one way to go, but please don't make it the only one;
administrators of fork()-based server programs the world over will thank
you from the bottom of their hearts.
I'm speaking from the side-lines, but why not have a DLL register all
the PMCs it contains at load time (placing the onus of setup on the
library author rather than the parrot core)? The library could put into
global variables whatever identifiers/data structures support the
dynamic PMC allocation. Sharing of the PMC class would then be
equivalent to visibility of those globals--equivalent to having linked
with the library containing the PMC definition--and so PMC class
versioning is thereby solved to the same extent that library versioning
is solved by the dynamic linker. Also, if the PMC class global is
visible to user code, then it's already registered and that code doesn't
need to do anything special at all to load it up.
I suppose this screws up in the single process/multiple interprete
(SPMI?) scenario, since the dynamic PMC could need to appear in a
different slot on each interpreter. But if the DLL was itself loaded
only through the interpreter itself, and the interpreter communicated to
the library's initialization routines? (Worst case means of
communication: Giant spinlock around library loading, plus a
current_parrot_interpreter global in the parrot library, which the DLL
is of course linking against.) Then a sufficiently private mapping or
per-interpreter initialization might be possible, depending on dynamic
linker's capabilities.
—
Gordon Henriksen
mali...@mac.com
Leopold Toetsch
> At 11:10 +0200 7/31/03, Leopold Toetsch wrote:
>> > *) Determine the init and setup routine names
>>
>>- Parrot_<classname>_class_setup
>>- Parrot_<classname>_class_init
>>
>>The class_setup also sets the class_enum i.e vtable->base_type.
> Well... there are versioning issues there. We ought to be able to
> have multiple versions of a class loaded, in which case this may not
> work. I'm pretty sure that there are some platforms that require
> unique names in libraries across the entire process.
What about the following (also considering, that we might not like one
handle/mmap... per PMC, as stated in one f'up):
A dynamic PMC library has one init function returning a dyn_pmc_info
structure:
struct { /* pseudo code */
int n_pmcs; /* count of pmcs in this extension file */
struct {
char *class;
version_t version;
(*pmc_setup_func)();
(*pmc_init_func)();
} pmc_info[n_pmcs];
} dyn_pmc_info;
And e.g. these opcodes:
load_pmc "foo" # load $(RUNTIME_LIB)foo$(SO), class Foo
load_pmc sx, sy # load lib sx, class sy
load_pmc sx, sy, sz # load lib sx, class sy, version sz
So a PMC extension could contain e.g. all PMCs needed for one specifc
language including PMCs of same functionality but different versions.
We only need the name of the init func.
More things to consider:
- the assembler has to see the same sequence of B<load_pmc> opcodes as
runtime (the loaded class gets the next available class enum, which
must match the B<new_p_ic> class_enum)
- or we have to put this info into some metadata, so that classes are
loaded in correct order at runtime
- we need global and per interpreter structures to keep track of
loaded extension libs.
Comments welcome,
leo
Dave Whipp
considering, that we might not like one
> structure:
In the past, I've found the "init-fn returns struct" model can become a
burden when, in a few years time, you want to change the info that's
returned. Having the init function call a callback for each PMC that it
wishes to register tends to be more future-proof (because you can define a
different callback for the init functions that know about the extended
registration info).
Dave.
Mattia Barbon
> Dan Sugalski <d...@sidhe.org> wrote:
> > At 11:10 +0200 7/31/03, Leopold Toetsch wrote:
> >> > *) Determine the init and setup routine names
> >>
> >>- Parrot_<classname>_class_setup
> >>- Parrot_<classname>_class_init
> >>
> >>The class_setup also sets the class_enum i.e vtable->base_type.
>
> > Well... there are versioning issues there. We ought to be able to
> > have multiple versions of a class loaded, in which case this may not
> > work. I'm pretty sure that there are some platforms that require
> > unique names in libraries across the entire process.
>
> What about the following (also considering, that we might not like one
> handle/mmap... per PMC, as stated in one f'up):
>
> A dynamic PMC library has one init function returning a dyn_pmc_info
> structure:
Why not do it (more or less) the way Perl5 does it? Have a
"load_and_init" operation which loads a dynamic library and calls some
predefined function in it (just pick a name). That function can then
register new PMCs, opcodes, events, whatever you like using normal Parrot_*
API functions. This can be made to be exactly as efficient as your
proposal, and I think it is more flexible.
Regards
Mattia
Dan Sugalski
>What about the following (also considering, that we might not like one
>handle/mmap... per PMC, as stated in one f'up):
>
>A dynamic PMC library has one init function returning a dyn_pmc_info
>structure:
I think I'd rather have pre-defined names in the PMC library that we
can call like any other function in a dynamically loaded library. We
just pick a pair of names that can be mashed appropriately and are
done with it.
The big issue is one of metadata. The point about having multiple
classes in a single library is well-taken, but there's the issue of
how we keep track of which things are in which files. While there's
still the issue of finding those files in a library tree, if there's
a 1-to-1 relation between classes and files it gets a bit simpler.
We probably want some sort of metadata system on disk, though I'm
loathe to go that way.
Leopold Toetsch
> I think I'd rather have pre-defined names in the PMC library
Yep. I've abandoned the return structure thingy. There is now one fixed
init call to get things running:
for the PMC extension foo_pmc$(SO):
int Parrot_dynext_foo_init(Interp *interp, int action, void *param)
C<action> is SETUP or INIT with an appropriate structure hidden in
C<param>. If we need somewhen an extended setup/init, we can provide
additional interfaces hopefully.
> The big issue is one of metadata.
Its somehow a problem. But the HL compiler (writer) just knows, that
e.g. F<python_pmc.so> provides all Python PMCs. These shared libs have
to be written for a certain language and when some code needs this, the
compiler will just emit the matching C<load_pmc> op.
A bigger issue are IMHO the class enums. Current state is, that the
C<load_pmc> op must be in the same sequence at compile *and* runtime.
This could cause troubles, if a Perl6 program calls a Python lib and
then a Ruby lib, both having a C<load_pmc> in the lib. OTOH there must
be some "use ThisLib;" statements first, which are in one specific
sequence.
> We probably want some sort of metadata system on disk, though I'm
> loathe to go that way.
If we need these, they can be autogenerated by pmc2c.pl.
Anyway, I have committed my current version. Its not a final thing, it
needs more Configure support, base_vtables is still static and so on,
and I really don't like the issues WRT the gloabals, which exist in
parrot and in libparrot twice.
Its simpler to discuss existing code and having big diffs against CVS
isn't really nice ;-)
leo