I have started looking at dynamic classes. I have currently - new subdirectory /dynclasses - small hack for classes/pmc2c.pl to consider this directory too - dynclasses/foo.pmc, dynclasses/Makefile (unportable, but short ;-)
That works fine so far, it builds F<foo_pmc.so>. Now we would need: - an allocated *Parrot_base_vtables [1] - the count of available PMCs (current enum_class_max) - and finally an opcode to load and initialize that I'm thinking here of the currently unused C<loadext> opcode, which could be expanded, to load all kinds of extensions. This opcode would need additionally: - a flag, stating e.g. a dynamic PMC is loaded - a signature for the init function to call
[1] per interpreter, because different threads might have different pmc classes loaded?
>I have started looking at dynamic classes. I have currently >- new subdirectory /dynclasses >- small hack for classes/pmc2c.pl to consider this directory too >- dynclasses/foo.pmc, dynclasses/Makefile (unportable, but short ;-)
Here's what I was thinking instead.
We build classes as shared libraries, unless otherwise noted. PMC libraries get loaded up with an op--load_pmc or something of the sort. It takes a PMC name and tacks on the library prefix and suffix as need be. We load in the PMC library, call its setup routine, call its initialization routine, and let it do its thing. The setup routine, when called, will at the least register the PMC's class in the class name hash. There should be some code in the system to manage this, though it needs mutex protection.
The setup routine is called on library load. The initialization routine is called when the class is instantiated into a new interpreter. Setup stuff is done only once, initialization stuff is potentially done multiple times, though only once per interpreter--setup is for any initial load things (like linking to databases or whatever) while the initialization routine is for when a thread loads up a PMC class for the first time, but that class has already been loaded by another thread.
What we need to do is:
*) Determine the init and setup routine names *) Make some standard for library paths and names *) Figure out how we want to handle versioning
-- Dan
--------------------------------------"it's like this"------------------- Dan Sugalski even samurai d...@sidhe.org have teddy bears and even teddy bears get drunk
I don't pretend that I fully understand what dynamic classes are about, but Dan's suggestions made it a lot clearer to me. This seems like one possible way to attach something like wxWindows to parrot.
I'm wondering about one thing, however:
>We build classes as shared libraries, unless otherwise noted. PMC
Are there any plans to allow PMCs to be implemented in Parrot? Or am I asking something stoopid :).
On Wed, Jul 30, 2003 at 10:44:51PM +0200 it came to pass that Christian Renz wrote:
> Are there any plans to allow PMCs to be implemented in Parrot? Or am I > asking something stoopid :).
There are currently a lot of PMC types implemented directly in Parrot. If you look in the classes/ subdirectory of the Parrot distribution you see lots of files named "perlint.pmc" and so forth. These are the builtin PMC types...
The fun about dynamically loadable PMC types is that if a language L needs/wants its own interesting implementation of a datatype T so that the L compiler writers can generate PIR like:
P1=new LT
then there is no need to extend the core Parrot engine (and clutter all non-L users with the necessity to carry the LT implementation around). Instead, one could do
load_ext "LT" # or whatever P1=new LT
to dynamically load the implementation of the LT data type from a shared library (or DLL, or LPA load module, or whatever your operating system groks).
++Jos.es
-- ek is so lug jy vlieg deur my sonder jou is ek sonder patroon "Breyten Breytenbach"
On Thu, Jul 31, 2003 at 12:32:36AM +0200 it came to pass that Christian Renz wrote:
> Thanks for the clarification. Does that mean that a mechanism for > dynamic PMCs would automatically allow them to be written in Parrot > also (and not only load binary libs)?
I don't think there are currently plans to allow PMC types to be written in Parrot. But others are more definitive guru's on that...
++Jos.es
-- ek is so lug jy vlieg deur my sonder jou is ek sonder patroon "Breyten Breytenbach"
>I don't pretend that I fully understand what dynamic classes are >about, but Dan's suggestions made it a lot clearer to me. This seems >like one possible way to attach something like wxWindows to parrot.
>I'm wondering about one thing, however:
>>We build classes as shared libraries, unless otherwise noted. PMC
>Are there any plans to allow PMCs to be implemented in Parrot?
Absolutely, yes. PMC classes don't have to be written in C. -- Dan
--------------------------------------"it's like this"------------------- Dan Sugalski even samurai d...@sidhe.org have teddy bears and even teddy bears get drunk
Thanks for the clarification. Does that mean that a mechanism for dynamic PMCs would automatically allow them to be written in Parrot also (and not only load binary libs)?
Dan Sugalski <d...@sidhe.org> wrote: > At 12:43 +0200 7/30/03, Leopold Toetsch wrote: >>I have started looking at dynamic classes. I have currently >>- new subdirectory /dynclasses >>- small hack for classes/pmc2c.pl to consider this directory too >>- dynclasses/foo.pmc, dynclasses/Makefile (unportable, but short ;-) > Here's what I was thinking instead. > We build classes as shared libraries, unless otherwise noted. PMC > libraries get loaded up with an op--load_pmc or something of the > sort. It takes a PMC name and tacks on the library prefix and suffix > as need be.
So we first need some more Configure support: - runtime directory - path separator - $(SO) extension
(The dynclasses subdir is ment for building these libs, not for loading them from there [1])
> ... We load in the PMC library, call its setup routine, call > its initialization routine, and let it do its thing.
The split into C<setup> and <class_init> is necessary for our static PMCs too. Mainly ParrotIO is using the current class_init hook to work around initialization order and to setup its method table. [2]
> ... The setup > routine, when called, will at the least register the PMC's class in > the class name hash. There should be some code in the system to > manage this, though it needs mutex protection.
IMHO: The class_setup loads the class into the global Parrot_base_vtables. The class_init registers the class in a per interpreter class name hash. Different threads might have different classes loaded, but the class_enums have to be uniq.
The class_setup also sets the class_enum i.e vtable->base_type.
> *) Make some standard for library paths and names
- runtime/parrot/pmc
> *) Figure out how we want to handle versioning
brrr, I need a Ponie ;-)
[1] If there are no objections I'll check that part in. It shouldn't be too hard, to finally have a perl script, that generates a platform independend Makefile out from %PConfig.
[2] Somehow related: There was some discussion about PMC methods. What about a standard way to allow methods for (almost) all PMCs: - have a vtable slot for the method hash - move find_method into default.pmc - PMCs used in that process mainly PerlHash should better not have methods ;-)
>The class_setup also sets the class_enum i.e vtable->base_type.
Well... there are versioning issues there. We ought to be able to have multiple versions of a class loaded, in which case this may not work. I'm pretty sure that there are some platforms that require unique names in libraries across the entire process.
>[1] If there are no objections I'll check that part in. It shouldn't be >too hard, to finally have a perl script, that generates a platform >independend Makefile out from %PConfig.
Nope, go for it.
>[2] Somehow related: There was some discussion about PMC methods. >What about a standard way to allow methods for (almost) all PMCs: >- have a vtable slot for the method hash >- move find_method into default.pmc >- PMCs used in that process mainly PerlHash should better not have > methods ;-)
Yeah, that sounds familiar. I'll have to get back to you on this when I've got more tha 5 minutes for email. :) -- Dan
--------------------------------------"it's like this"------------------- Dan Sugalski even samurai d...@sidhe.org have teddy bears and even teddy bears get drunk
>>The class_setup also sets the class_enum i.e vtable->base_type. > Well... there are versioning issues there. We ought to be able to > have multiple versions of a class loaded, in which case this may not > work. I'm pretty sure that there are some platforms that require > unique names in libraries across the entire process.
I think we can postpone that for later. Perl5 module versioning + addons that handle this, have very sophisticated schemes. Though I can just imagine to find/load a specific version of a pmc class, but not to have different versions of one module loaded, both providing the class "Foo".
>>[1] If there are no objections I'll check that part in. It shouldn't be >>too hard, to finally have a perl script, that generates a platform >>independend Makefile out from %PConfig. > Nope, go for it.
On Wednesday, July 30, 2003, at 04:28 , Dan Sugalski wrote: > At 12:43 +0200 7/30/03, Leopold Toetsch wrote:
>> I have started looking at dynamic classes. I have currently >> - new subdirectory /dynclasses >> - small hack for classes/pmc2c.pl to consider this directory too >> - dynclasses/foo.pmc, dynclasses/Makefile (unportable, but short ;-)
> Here's what I was thinking instead.
> We build classes as shared libraries, unless otherwise noted. PMC > libraries get loaded up with an op--load_pmc or something of the sort. > It takes a PMC name and tacks on the library prefix and suffix as need > be. We load in the PMC library, call its setup routine, call its > initialization routine, and let it do its thing. The setup routine, > when called, will at the least register the PMC's class in the class > name hash. There should be some code in the system to manage this, > though it needs mutex protection.
Eek? Forcing a new DLL for every dynamically-loaded PMC class? So many open()'s and persistent file handles and mmap regions! As if Perl 5's DynaLoader weren't bad enough already. I can hear the clatter of keyboards across the world. I can't make out the rest, but it all starts with su and then echo `perl -e 'print 2**32-1, "\n"'` > /proc/sys/.... Maybe this is one way to go, but please don't make it the only one; administrators of fork()-based server programs the world over will thank you from the bottom of their hearts.
I'm speaking from the side-lines, but why not have a DLL register all the PMCs it contains at load time (placing the onus of setup on the library author rather than the parrot core)? The library could put into global variables whatever identifiers/data structures support the dynamic PMC allocation. Sharing of the PMC class would then be equivalent to visibility of those globals--equivalent to having linked with the library containing the PMC definition--and so PMC class versioning is thereby solved to the same extent that library versioning is solved by the dynamic linker. Also, if the PMC class global is visible to user code, then it's already registered and that code doesn't need to do anything special at all to load it up.
I suppose this screws up in the single process/multiple interprete (SPMI?) scenario, since the dynamic PMC could need to appear in a different slot on each interpreter. But if the DLL was itself loaded only through the interpreter itself, and the interpreter communicated to the library's initialization routines? (Worst case means of communication: Giant spinlock around library loading, plus a current_parrot_interpreter global in the parrot library, which the DLL is of course linking against.) Then a sufficiently private mapping or per-interpreter initialization might be possible, depending on dynamic linker's capabilities.
>>The class_setup also sets the class_enum i.e vtable->base_type. > Well... there are versioning issues there. We ought to be able to > have multiple versions of a class loaded, in which case this may not > work. I'm pretty sure that there are some platforms that require > unique names in libraries across the entire process.
What about the following (also considering, that we might not like one handle/mmap... per PMC, as stated in one f'up):
A dynamic PMC library has one init function returning a dyn_pmc_info structure:
struct { /* pseudo code */ int n_pmcs; /* count of pmcs in this extension file */ struct { char *class; version_t version; (*pmc_setup_func)(); (*pmc_init_func)(); } pmc_info[n_pmcs]; } dyn_pmc_info;
And e.g. these opcodes:
load_pmc "foo" # load $(RUNTIME_LIB)foo$(SO), class Foo load_pmc sx, sy # load lib sx, class sy load_pmc sx, sy, sz # load lib sx, class sy, version sz
So a PMC extension could contain e.g. all PMCs needed for one specifc language including PMCs of same functionality but different versions.
We only need the name of the init func.
More things to consider: - the assembler has to see the same sequence of B<load_pmc> opcodes as runtime (the loaded class gets the next available class enum, which must match the B<new_p_ic> class_enum) - or we have to put this info into some metadata, so that classes are loaded in correct order at runtime - we need global and per interpreter structures to keep track of loaded extension libs.
"Leopold Toetsch" <l...@toetsch.at> wrote > What about the following (also considering, that we might not like one
> A dynamic PMC library has one init function returning a dyn_pmc_info > structure:
In the past, I've found the "init-fn returns struct" model can become a burden when, in a few years time, you want to change the info that's returned. Having the init function call a callback for each PMC that it wishes to register tends to be more future-proof (because you can define a different callback for the init functions that know about the extended registration info).
> >>The class_setup also sets the class_enum i.e vtable->base_type.
> > Well... there are versioning issues there. We ought to be able to > > have multiple versions of a class loaded, in which case this may not > > work. I'm pretty sure that there are some platforms that require > > unique names in libraries across the entire process.
> What about the following (also considering, that we might not like one > handle/mmap... per PMC, as stated in one f'up):
> A dynamic PMC library has one init function returning a dyn_pmc_info > structure:
Why not do it (more or less) the way Perl5 does it? Have a "load_and_init" operation which loads a dynamic library and calls some predefined function in it (just pick a name). That function can then register new PMCs, opcodes, events, whatever you like using normal Parrot_* API functions. This can be made to be exactly as efficient as your proposal, and I think it is more flexible.
>What about the following (also considering, that we might not like one >handle/mmap... per PMC, as stated in one f'up):
>A dynamic PMC library has one init function returning a dyn_pmc_info >structure:
I think I'd rather have pre-defined names in the PMC library that we can call like any other function in a dynamically loaded library. We just pick a pair of names that can be mashed appropriately and are done with it.
The big issue is one of metadata. The point about having multiple classes in a single library is well-taken, but there's the issue of how we keep track of which things are in which files. While there's still the issue of finding those files in a library tree, if there's a 1-to-1 relation between classes and files it gets a bit simpler.
We probably want some sort of metadata system on disk, though I'm loathe to go that way. -- Dan
--------------------------------------"it's like this"------------------- Dan Sugalski even samurai d...@sidhe.org have teddy bears and even teddy bears get drunk
Dan Sugalski <d...@sidhe.org> wrote: > I think I'd rather have pre-defined names in the PMC library
Yep. I've abandoned the return structure thingy. There is now one fixed init call to get things running:
for the PMC extension foo_pmc$(SO): int Parrot_dynext_foo_init(Interp *interp, int action, void *param)
C<action> is SETUP or INIT with an appropriate structure hidden in C<param>. If we need somewhen an extended setup/init, we can provide additional interfaces hopefully.
> The big issue is one of metadata.
Its somehow a problem. But the HL compiler (writer) just knows, that e.g. F<python_pmc.so> provides all Python PMCs. These shared libs have to be written for a certain language and when some code needs this, the compiler will just emit the matching C<load_pmc> op.
A bigger issue are IMHO the class enums. Current state is, that the C<load_pmc> op must be in the same sequence at compile *and* runtime. This could cause troubles, if a Perl6 program calls a Python lib and then a Ruby lib, both having a C<load_pmc> in the lib. OTOH there must be some "use ThisLib;" statements first, which are in one specific sequence.
> We probably want some sort of metadata system on disk, though I'm > loathe to go that way.
If we need these, they can be autogenerated by pmc2c.pl.
Anyway, I have committed my current version. Its not a final thing, it needs more Configure support, base_vtables is still static and so on, and I really don't like the issues WRT the gloabals, which exist in parrot and in libparrot twice. Its simpler to discuss existing code and having big diffs against CVS isn't really nice ;-)
