[RFC] removing library NCI signatures from core
Peter Lobsinger
opengl_dynamic_nci was to eliminate the need for external projects to
register NCI signatures with core. Once nativecall.pir is part of
parrot's install, I would like to remove all signatures in
config/gen/call_list/misc.in from core. That is, all NCI signatures
previously registered by external projects with parrot core.
I base this proposal on these views:
* parrot core should only contain the minimum necessary to run and be extended
* there will be a new, more flexible (IMHO better) way to get
signature thunks for extenders/embedders
* new signatures should not be registered with core because there's a better way
* old projects are not somehow special, they should use (after a grace
period) the same features made available to new projects
I raised this idea today on #ps, and it was decided that it merited
further discussion.
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
NotFound
Be extended how? Generating and compiling C code is a big NO. That way
NCI is not native at all. It kills the posibility of installing pir
modules that use NCI in machines without a C compiler, forcing you to
use some development system and reinstall parrot or parts of it.
And we don't need a native interface to do that, the extension can
just provide functions with signatures that parrot expects or
provides. Just like perl5 extensions.
> * there will be a new, more flexible (IMHO better) way to get
> signature thunks for extenders/embedders
Fine, but in that case let's keep the current way until we have that
other new way.
I like to be able to access any public function in any dynamic library
of the system from pure pir, hand written or generated, without
compiling and installing anything, even with a parrot buit before that
library existed. That's what I believe a Native interface is.
Otherwise is just an extension mechanism.
--
Salu2
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Peter Lobsinger
>> * parrot core should only contain the minimum necessary to run and be extended
>
> Be extended how? Generating and compiling C code is a big NO. That way
> NCI is not native at all. It kills the posibility of installing pir
> modules that use NCI in machines without a C compiler, forcing you to
> use some development system and reinstall parrot or parts of it.
The 2 major ways to extend parrot, dynpmcs and dynops, already need a
C compiler. For machines without a C compiler, there must be a C
compiler somewhere or else you wouldn't have parrot. Additional static
signatures would not be difficult to add at parrot's compile time -
modify call_list.txt, regenerate src/nci.c, and recompile (what we
have now btw).
But static signatures have a cost of requiring the interpreter to haul
around a hash (~400 elements) of thunks that are hardly ever used.
These bloat both the gc pool and the code size. For users that don't
need it, why have it?
> And we don't need a native interface to do that, the extension can
> just provide functions with signatures that parrot expects or
> provides. Just like perl5 extensions.
perl5 extensions would be great at this point. Our nci system doesn't
even have that yet. There's an important point to perl5 extensions
that we're lacking - a boilerplate generator. And that's what
nativecall.pir provides.
>> * there will be a new, more flexible (IMHO better) way to get
>> signature thunks for extenders/embedders
>
> Fine, but in that case let's keep the current way until we have that
> other new way.
A number of people seem to believe that this better way is a runtime
thunk generator. That won't work everywhere. There will always be some
architecture to which whatever ffi library you use has not yet been
ported.
That's not to say that a runtime thunk mechanism isn't worthwhile,
just that it excludes potential platforms. We should provide a way
that is fully generic, if a bit cumbersome for some.
> I like to be able to access any public function in any dynamic library
> of the system from pure pir, hand written or generated, without
> compiling and installing anything, even with a parrot buit before that
> library existed. That's what I believe a Native interface is.
> Otherwise is just an extension mechanism.
That mis-characterizes what NCI in parrot is right now. NCI in parrot
currently only works for a narrow set of libraries that have been
registered in config/gen/call_list/misc.in. If the library you want
isn't there, and you've created bindings that "work", they only work
by accident.
And the set of currently supported signatures isn't that great either.
I initially got into the nci system because I was trying to embed
parrot and a simple 3 argument signature I was trying to use wasn't
supported. I could have added my project and associated signatures to
config/gen/call_list/misc.in and been on my merry way, but that's a
band-aid solution.
Whether we should call this a native call interface or not doesn't
really matter to me. If it doesn't do what people expect an nci system
to do, we could just as easily call it something else so people don't
get any misleading expectations.
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
NotFound
> C compiler.
Yes. So adding a supposed NCI that accomplish essentially the same
mission is pointless.
> But static signatures have a cost of requiring the interpreter to haul
> around a hash (~400 elements) of thunks that are hardly ever used.
> These bloat both the gc pool and the code size. For users that don't
> need it, why have it?
Yes, static signatures are costly and incomplete. What we need is a
way to generate dynamically, like the old jit tried to provide.
>> Fine, but in that case let's keep the current way until we have that
>> other new way.
> A number of people seem to believe that this better way is a runtime
> thunk generator. That won't work everywhere. There will always be some
> architecture to which whatever ffi library you use has not yet been
> ported.
There are also platforms without symbolic links, but we provide
support for that.
>> I like to be able to access any public function in any dynamic library
>> of the system from pure pir, hand written or generated, without
>> compiling and installing anything, even with a parrot buit before that
>> library existed. That's what I believe a Native interface is.
>> Otherwise is just an extension mechanism.
> That mis-characterizes what NCI in parrot is right now. NCI in parrot
> currently only works for a narrow set of libraries that have been
> registered in config/gen/call_list/misc.in. If the library you want
> isn't there, and you've created bindings that "work", they only work
> by accident.
That's no very exact. We don't need to have any library registered, we
just need the function signatures. And is not an accident, the now
defunct jit system fully allowed that in the platforms that used it.
Andrew Whitworth
> NCI is not native at all. It kills the posibility of installing pir
> modules that use NCI in machines without a C compiler, forcing you to
> use some development system and reinstall parrot or parts of it.
That's how Parrot is now, except when we want to add a new NCI
signature we need to recompile all of Parrot. Because of the Perl5
dependency, it's actually harder to do this than just compile a shared
library.
Compilation of C code libraries is a chore, but it's only one option.
Using libjit, libffi, or other libraries would make the process
significantly easier in the future. These alternatives could be
pluggable once the system is cleaned up and we remove static thunks.
> And we don't need a native interface to do that, the extension can
> just provide functions with signatures that parrot expects or
> provides. Just like perl5 extensions.
Now that's quite an interesting idea, but limiting. We need some kind
of way to construct a call frame in order to provide a thin wrapper
around an external library. If we create a more "thick" wrapper
instead, such as a PMC class to wrap the interface, or a set of ops,
we can do what you suggest. I don't like the idea of restricting what
kinds of add-ons we can support, however. Some libraries don't lend
themselves to being wrapped by a PMC or even a set of PMCs.
>> * there will be a new, more flexible (IMHO better) way to get
>> signature thunks for extenders/embedders
>
> Fine, but in that case let's keep the current way until we have that
> other new way.
As a middle-ground first step, howabout we break the non-necessary NCI
signatures into a dynlib in the repo. That way we still have them in
the repo if needed in the interim, we have a way to run tests on the
mechanism locally, and we prepare for the larger refactors that Peter
is suggesting.
> I like to be able to access any public function in any dynamic library
> of the system from pure pir, hand written or generated, without
> compiling and installing anything, even with a parrot buit before that
> library existed. That's what I believe a Native interface is.
> Otherwise is just an extension mechanism.
That's what we all want, but we cannot do that now. What we do now is
really the worst possible solution, and making loadable libraries of
NCI thunks is a huge improvement over that. In reality, we're probably
not going to have proper dynamic native calls without one of two
things: an existing dynamic interface (using libffi or libjit) or
Lorito.
--Andrew Whitworth
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Peter Lobsinger
<wknig...@gmail.com> wrote:
> Now that's quite an interesting idea, but limiting. We need some kind
> of way to construct a call frame in order to provide a thin wrapper
> around an external library. If we create a more "thick" wrapper
> instead, such as a PMC class to wrap the interface, or a set of ops,
> we can do what you suggest. I don't like the idea of restricting what
> kinds of add-ons we can support, however. Some libraries don't lend
> themselves to being wrapped by a PMC or even a set of PMCs.
As an aside, methods on C-based PMCs use NCI thunks, so unless you can
shoehorn an API into vtables, you're no further ahead by trying to use
PMCs in the current system.
> As a middle-ground first step, howabout we break the non-necessary NCI
> signatures into a dynlib in the repo. That way we still have them in
> the repo if needed in the interim, we have a way to run tests on the
> mechanism locally, and we prepare for the larger refactors that Peter
> is suggesting.
Seems like a good idea to me. I would be willing to do the (small
amount) of work to make that happen.
> That's what we all want, but we cannot do that now. What we do now is
> really the worst possible solution, and making loadable libraries of
> NCI thunks is a huge improvement over that. In reality, we're probably
> not going to have proper dynamic native calls without one of two
> things: an existing dynamic interface (using libffi or libjit) or
> Lorito.
As I understand it, Lorito is supposed to be small. A decent FFI will
need to implement at least a large chunk of the types your C compiler
works with. Because of the increase in scope (and associated
complexity) this causes, I'm not sure that we'd want to push native
interfacing into Lorito. Maybe I'm mistaken, please enlighten me if I
am (the vision for Lorito has always been a little hazy for me).
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
NotFound
> signatures into a dynlib in the repo. That way we still have them in
> the repo if needed in the interim, we have a way to run tests on the
> mechanism locally, and we prepare for the larger refactors that Peter
> is suggesting.
I like that idea. Even better, make it pluggable and selectable at
runtime. This way will also allow to easily control or disable his
usage in security arenas.
Andrew Whitworth
> As an aside, methods on C-based PMCs use NCI thunks, so unless you can
> shoehorn an API into vtables, you're no further ahead by trying to use
> PMCs in the current system.
Is that still the case? I thought that the PCC refactors changed that.
At the very least argument handling is done differently, so it should
be possible to avoid NCI thunks (or, most of them) when calling
C-based methods.
> Seems like a good idea to me. I would be willing to do the (small
> amount) of work to make that happen.
+1
> As I understand it, Lorito is supposed to be small. A decent FFI will
> need to implement at least a large chunk of the types your C compiler
> works with. Because of the increase in scope (and associated
> complexity) this causes, I'm not sure that we'd want to push native
> interfacing into Lorito. Maybe I'm mistaken, please enlighten me if I
> am (the vision for Lorito has always been a little hazy for me).
Lorito is intended to be small, but it's also supposed to have the
same general capabilities as C. This means it should be able to call
arbitrary C functions with it. That may be ambitious, but it sure
sounds good.
Peter Lobsinger
>> As a middle-ground first step, howabout we break the non-necessary NCI
>> signatures into a dynlib in the repo. That way we still have them in
>> the repo if needed in the interim, we have a way to run tests on the
>> mechanism locally, and we prepare for the larger refactors that Peter
>> is suggesting.
>
> I like that idea. Even better, make it pluggable and selectable at
> runtime. This way will also allow to easily control or disable his
> usage in security arenas.
Runtime filtering of available signatures should already be doable:
# warning: untested
.include 'iglobals.pasm'
.sub 'filter_out'
.param string signature
$P0 = getinterp
$P1 = $P0[ .IGLOBALS_NCI_FUNCS ]
delete $P1[ signature ]
.end
Of course, filtering based on signature seems a little coarse to me.
The only really secure solution I can see is to load all permissible
NCI funcs and then completely disallow NCI. But I am not a security
expert.
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Peter Lobsinger
<wknig...@gmail.com> wrote:
> On Wed, Feb 17, 2010 at 11:22 AM, Peter Lobsinger <plob...@gmail.com> wrote:
>> As an aside, methods on C-based PMCs use NCI thunks, so unless you can
>> shoehorn an API into vtables, you're no further ahead by trying to use
>> PMCs in the current system.
>
> Is that still the case? I thought that the PCC refactors changed that.
> At the very least argument handling is done differently, so it should
> be possible to avoid NCI thunks (or, most of them) when calling
> C-based methods.
Seems you're right. pmc2c generates the pcc boilerplate and doesn't
delegate to nci. I was confused by references to pmc methods in
config/gen/call_list/* and the fact that pmc2c calls these functions
"Parrot_${pmc_name}_nci_${method_name}".
What this means (or should mean) then, is that PMC methods should be
more restricted in signature type to avoid complete duplication (or
worse: incorrect duplication) of functionality. We don't want to
maintain 3 C<=>PCC mappings in parallel (we already have nativecall
and nci.pmc). We aren't even managing 2 well (nci.pmc is out of sync
and we don't have tests to complain at us about it).
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Jonathan Leto
On Wed, Feb 17, 2010 at 8:29 AM, NotFound <julian....@gmail.com> wrote:
>> As a middle-ground first step, howabout we break the non-necessary NCI
>> signatures into a dynlib in the repo. That way we still have them in
>> the repo if needed in the interim, we have a way to run tests on the
>> mechanism locally, and we prepare for the larger refactors that Peter
>> is suggesting.
>
> I like that idea. Even better, make it pluggable and selectable at
> runtime. This way will also allow to easily control or disable his
> usage in security arenas.
Since I seem to be the pumpking for PDD18 (security subsystem), I
would say that it is vital that we have the ability to disable NCI
totally, and perhaps even have finer-grained controls, if possible.
Duke
--
Jonathan "Duke" Leto
jona...@leto.net
http://leto.net
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Peter Lobsinger
> On Wed, Feb 17, 2010 at 12:22 PM, Andrew Whitworth
> <wknig...@gmail.com> wrote:
>> On Wed, Feb 17, 2010 at 11:22 AM, Peter Lobsinger <plob...@gmail.com> wrote:
>>> As an aside, methods on C-based PMCs use NCI thunks, so unless you can
>>> shoehorn an API into vtables, you're no further ahead by trying to use
>>> PMCs in the current system.
>>
>> Is that still the case? I thought that the PCC refactors changed that.
>> At the very least argument handling is done differently, so it should
>> be possible to avoid NCI thunks (or, most of them) when calling
>> C-based methods.
>
> Seems you're right. pmc2c generates the pcc boilerplate and doesn't
> delegate to nci. I was confused by references to pmc methods in
> config/gen/call_list/* and the fact that pmc2c calls these functions
> "Parrot_${pmc_name}_nci_${method_name}".
Turns out even this isn't strictly true. pmc2c generates pcc
boilerplate for regular methods, but delegates to NCI for multis. I
don't know why. This should probably be changed at some point to be
consistent one way or the other (I'm biased towards delegating to NCI
for obvious reasons).
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Andrew Whitworth
>> Seems you're right. pmc2c generates the pcc boilerplate and doesn't
>> delegate to nci. I was confused by references to pmc methods in
>> config/gen/call_list/* and the fact that pmc2c calls these functions
>> "Parrot_${pmc_name}_nci_${method_name}".
>
> Turns out even this isn't strictly true. pmc2c generates pcc
> boilerplate for regular methods, but delegates to NCI for multis. I
> don't know why. This should probably be changed at some point to be
> consistent one way or the other (I'm biased towards delegating to NCI
> for obvious reasons).
Well, that's interesting. Personally I think we should try to
standardize on the PCC method for a variety of reasons. Standardizing
on a single code path lets us get rid of a lot of mode-switching cod,
let's us focus optimization efforts on a single hotpath, and takes
some stress off the NCI system that will give us more freedom to
refactor and improve that system.
What might make for a good interim measure would be to examine the
system and see what would be needed to standardize one way or another.
We can make a better decision once we know more.
Peter Lobsinger
pmc2c methods wind up wrapped with NCI objects anyways. This adds
complexity to the NCI pmc in order to support "raw" subs (which handle
their own PCC) as well as "thunked" subs (which delegate their PCC to
NCI thunks). Also, because the wrapping NCI pmc has more information
about thunked subs (nci signature), it can provide more information to
the rest of the system about them (arity, pcc signature).
I agree that we should standardize on a single code path. However, we
need the NCI thunks anyways, whereas the functionality to handle PCC
in pmc2c is a duplication of that found in nci_thunk_gen. If PCC's API
changes, that makes for one more place that can get forgotten to be
changed. If we make all methods thunked, we'll have less code overall
and fewer places for bugs to hide.
Andrew Whitworth
> pmc2c methods wind up wrapped with NCI objects anyways. This adds
> complexity to the NCI pmc in order to support "raw" subs (which handle
> their own PCC) as well as "thunked" subs (which delegate their PCC to
> NCI thunks). Also, because the wrapping NCI pmc has more information
> about thunked subs (nci signature), it can provide more information to
> the rest of the system about them (arity, pcc signature).
I'm not certain I follow all this. In either case, we always need to
have an NCI thunk and NCI PMC associated with the method, though in
the PCC case we only need one NCI thunk with a single signature, which
saves on code bloat. We always need the PCC signature for things like
MMD, but we don't usually need the NCI signature to be available for
any reason besides the NCI dispatcher itself.
Also, does the NCI dispatcher support all the argument options that
PCC supports (:optional and :opt_flag, :named, :slurpy, etc)? Does it
support multiple return values, including return values with option
flags (:flat and :named)? My understanding is that NCI is a gateway
system that converts from Parrot's calling conventions to C's calling
conventions, and is limited to supporting the options that C functions
support, maybe with some massaging to get things into format.
If the question is about complexity in the NCI PMC when
differentiating between subs which are "managed" (using PCC to
pack/unpack args) and "unmanaged" (args must be passed in through the
C function interface), I think the better solution is to have two
different NCI PMC types that each perform a single operation. METHODs
in Pmc2c land could be "ManagedNCI" or "PCCMethod" or something like
that. This gives us simpler code and some possibilities to streamline
certain code paths with only the overhead of registering a new PMC
type. In fact if we were smart about it, a simple PCCMethod PMC could
cut out a lot of the unnecessary logic of the NCI dispatch mechanism.
> I agree that we should standardize on a single code path. However, we
> need the NCI thunks anyways, whereas the functionality to handle PCC
> in pmc2c is a duplication of that found in nci_thunk_gen. If PCC's API
> changes, that makes for one more place that can get forgotten to be
> changed. If we make all methods thunked, we'll have less code overall
> and fewer places for bugs to hide.
Ideally, I think we all can agree that the long term plans are for
nci_thunk_gen to go away in favor of a proper frame builder. Also, as
I alluded to above, I don't think NCI supports all the features that
we need for Pmc2c methods anyway. NCI should stay the system that
provides a gateway to native code libraries, and PCC should stay the
system that allows passing control flow in the Parrot way (with
argument and return options, multiple return values, values stored in
Parrot registers, access to a Parrot context, etc).
Peter Lobsinger
> On Fri, Feb 26, 2010 at 12:59 PM, Peter Lobsinger <plob...@gmail.com> wrote:
>> pmc2c methods wind up wrapped with NCI objects anyways. This adds
>> complexity to the NCI pmc in order to support "raw" subs (which handle
>> their own PCC) as well as "thunked" subs (which delegate their PCC to
>> NCI thunks). Also, because the wrapping NCI pmc has more information
>> about thunked subs (nci signature), it can provide more information to
>> the rest of the system about them (arity, pcc signature).
>
> I'm not certain I follow all this. In either case, we always need to
> have an NCI thunk and NCI PMC associated with the method, though in
> the PCC case we only need one NCI thunk with a single signature, which
> saves on code bloat. We always need the PCC signature for things like
> MMD, but we don't usually need the NCI signature to be available for
> any reason besides the NCI dispatcher itself.
What I'm getting at is that pmc2c hides information from the rest of
the system. AFAICT, you cannot obtain the PCC signature of raw/managed
methods, because the wrapping NCI pmc has no information about them.
When the wrapping NCI pmc knows the NCI signature, it can generate the
PCC signature for the function from the given NCI signature. In fact,
it must generate a pcc signature for a runtime frame builder to work
correctly.
Here's an example using arity:
.sub 'main' :main
# malloc - thunked nci
$P0 = loadlib ''
$P1 = dlfunc $P0, 'malloc', 'pi'
print "malloc type: "
$S0 = typeof $P1
say $S0
print "malloc arity: "
$I0 = $P1.'arity'()
say $I0
# RPA.append - raw nci
$P0 = new 'ResizablePMCArray'
$P1 = find_method $P0, 'append'
print "RPA.append type: "
$S0 = typeof $P1
say $S0
print "RPA.append arity: "
$I0 = $P1.'arity'() # boom
say $I0
.end
> Also, does the NCI dispatcher support all the argument options that
> PCC supports (:optional and :opt_flag, :named, :slurpy, etc)? Does it
> support multiple return values, including return values with option
> flags (:flat and :named)? My understanding is that NCI is a gateway
> system that converts from Parrot's calling conventions to C's calling
> conventions, and is limited to supporting the options that C functions
> support, maybe with some massaging to get things into format.
This is a point I hadn't considered. I did not know that pmc2c methods
were permitted to make use of the full gamut of PCC argument options.
But a quick ack through src/pmc shows that not only is it permitted,
it is in use. I am now convinced that using NCI thunks for methods is
not a good idea.
> If the question is about complexity in the NCI PMC when
> differentiating between subs which are "managed" (using PCC to
> pack/unpack args) and "unmanaged" (args must be passed in through the
> C function interface), I think the better solution is to have two
> different NCI PMC types that each perform a single operation. METHODs
> in Pmc2c land could be "ManagedNCI" or "PCCMethod" or something like
> that. This gives us simpler code and some possibilities to streamline
> certain code paths with only the overhead of registering a new PMC
> type. In fact if we were smart about it, a simple PCCMethod PMC could
> cut out a lot of the unnecessary logic of the NCI dispatch mechanism.
I like the 2 pmc solution solution.
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Andrew Whitworth
> What I'm getting at is that pmc2c hides information from the rest of
> the system. AFAICT, you cannot obtain the PCC signature of raw/managed
> methods, because the wrapping NCI pmc has no information about them.
> When the wrapping NCI pmc knows the NCI signature, it can generate the
> PCC signature for the function from the given NCI signature. In fact,
> it must generate a pcc signature for a runtime frame builder to work
> correctly.
>
> Here's an example using arity:
>
So that's definitely a bug, from where I am standing. The NCI PMC
should contain information about the PCC signature of the method.
> I like the 2 pmc solution solution.
I'm liking it much better too. Combine that with some much-needed
improvements to Pmc2c, and we should be in a much better place.