Extending Parrot NCI callback functionality

1 view
Skip to first unread message

Geoffrey Broadwell

unread,
Jan 16, 2008, 10:39:55 PM1/16/08
to Parrot Porters
Right now, Parrot's support for NCI callbacks (C code calling back into
PIR code) is relatively limited. In particular, there are at least the
following limitations:

1. The return type must be void (C library does not expect a response).

2. The callback must have exactly two arguments (from both the C and PIR
perspectives).

3. One of these arguments must be a "user data" pointer, provided by
the PIR code when it registers the callback with the C library, and
handed back by the C code to the PIR code during every callback.

4. The C code must treat the user data pointer as entirely opaque (no
modification or assumed structure).

5. The other argument's type can vary, but is much more limited than an
arbitrary ("normal") NCI argument.

I am starting to implement a GLUT and OpenGL binding for Parrot. GLUT
is extremely callback-oriented. You essentially can't do anything
beyond open the top-level window until callbacks are functional. You
can't even draw in that window -- the rendered image never actually
appears, and in fact the window itself is simply transparent with just
the chrome showing.

Unfortunately, none of the GLUT callbacks fall within the current
limitations on Parrot NCI callbacks. I was thus beginning to look at
improving the Parrot NCI code accordingly, but soon realized I didn't
know how to get around some pretty fundamental problems (at least, not
in pure C code).

It turns out there's a good reason that limitations #3 and #4 exist.
What actually gets registered with the external C library is a pointer
to a C function in src/inter_cb.c. This function calls a chain of other
functions. First, they check that the "user data" argument has not been
obviously corrupted, and that the interpreter state is still sane.
Next, they pull the Sub PMC and signature for the PIR function that
should be called *from properties of the user data argument*. A little
more sanity checking and a switch statement later, and this PIR Sub is
called using Parrot_runops_fromc_args_event().

And therein lies the problem. The callback entry point is compiled C
code. There could be dozens of callbacks registered by the PIR code,
but only *one* set of C functions that handles all of them. In order to
have a single compiled Parrot function that brokers all these different
callbacks from C to PIR, Parrot cheats. Parrot adds several properties
to the "user data" parameter before handing it off to the C library, and
when the C library calls back, Parrot reads these properties to figure
out which PIR function should get called.

But this cheat doesn't work if the C library's callbacks don't all
support a completely opaque user data parameter. And guess what? Not a
single one of the GLUT callbacks is of this form. In fact, several of
the most important GLUT callbacks have *no parameters at all*, and all
of the callback parameters have defined meanings on the C side.

So ... how can Parrot support callbacks of the types that GLUT uses?
The trick of putting magical properties on a special user data parameter
won't work anymore. I've been tanking on this for a while, and come up
with several possible schemes.

My first idea was that each time a new callback was registered, Parrot
would copy a tiny shim function and then poke the address(es) of the
data it needed to figure out what PIR routine to call directly into the
copied shim code. The shim copy could then be handed off to the C
library callback registration. When a callback came, the shim copy
would then magically call the right PIR function, since it had all the
information it needed "hardcoded".

A variant of this involved having a global registry of callbacks, so
that instead of poking a pile of data into the copied shim, Parrot would
simply need to poke in the callback number (one integer). When the
callback came in, the shim copy would then use this single integer to
look up the info it needed in the global registry.

But I don't know how to portably do either kind of compiled code poking
in pure C. (Not that I know it can't be done -- I just personally don't
know how.)

So my next idea was to have a shim function that did nothing but call
another function, thus putting its own address on the stack. The other
function would then reach up above its own stack frame and grab that
return address, using *that address* to look up the needed information
in a global callback registry. By making copies of this simple shim
every time a new callback was registered, each copy would get its own
address, and everything would magically work.

This was certainly better than the previous ideas, because it didn't
involve poking anything into the shim code, just being able to copy it.

But I still don't know how to do that in pure portable C. My guess is
that you could declare another function (which need not do anything) in
some special way so that it is guaranteed to be compiled into the object
code directly after the magic shim. Subtracting the address of the shim
function from the dummy function would then tell you exactly how many
bytes long the compiled shim is (including perhaps some padding, which
doesn't matter). This would allow you to copy the shim at will.

But this still leaves the problems of:

A. Being able to portably force two functions two be compiled
sequentially in memory. I would hope this is easy, but I don't
recall which set of declarations makes exactly this guarantee.

B. Being able to execute a copy of the shim on architectures that have
non-executable stack and/or heap. I assume this had to be solved
before JIT could work, so I'm guessing this isn't a real problem.

C. Being able to reach up above one's own stack frame and grab the
return address from the shim function's dummy call. Is there a
trick to make this work in pure portable C?

Anyway, there's my brain dump. Any help greatly appreciated, because
right now the GLUT/OpenGL binding is just dead in the water.


-'f


Paul Seamons

unread,
Jan 17, 2008, 12:38:13 AM1/17/08
to perl6-i...@perl.org
> I am starting to implement a GLUT and OpenGL binding for Parrot. GLUT
> is extremely callback-oriented. You essentially can't do anything
> beyond open the top-level window until callbacks are functional. You
> can't even draw in that window -- the rendered image never actually
> appears, and in fact the window itself is simply transparent with just
> the chrome showing.

I started down this path several months ago. The following is the thread on
the topic.

http://tinyurl.com/3crzpu

In that short thread, Allison Randall responded with:

The immediate solution, to get OpenGL working now without waiting for
the implementation of the concurrency scheduler and JITed call/callback
thunks, is to add a few more callback signatures to the current set of
alternatives, and to write a little bit of custom C code for the cases
that can't pass dispatch information in a user data argument.

It appears that NCI is on the roadmap a couple of months out (or more), but
that when it is done it will be done right - for some definition of right.
I would've put more work into a temporary solution, but real life has
interfered.

I'm glad to see other people interested in OpenGL on parrot. When the NCI PDD
is up for drafting, it will be good to have a listing of sample libraries and
required callback signatures available.

I'm excited about the progress Parrot is making.

Paul

Geoffrey Broadwell

unread,
Jan 17, 2008, 4:59:07 PM1/17/08
to Paul Seamons, perl6-i...@perl.org
On Wed, 2008-01-16 at 22:38 -0700, Paul Seamons wrote:
> > I am starting to implement a GLUT and OpenGL binding for Parrot.
>
> I started down this path several months ago. The following is the thread on
> the topic.
>
> http://tinyurl.com/3crzpu

OK, read it now. I think I got a little farther, but I'm blocked now
just as you were, on the same problem.

> It appears that NCI is on the roadmap a couple of months out (or more), but
> that when it is done it will be done right - for some definition of right.

I think that Allison hand-waved over exactly the case that you and I
tried to address first -- no JIT available, and want to create the shims
in portable C instead of hand-coded assembly per platform.

I'll try to take a look at the other PDDs she mentioned in a day or two,
but from what she wrote in your thread, I didn't get the feeling her
solution quite covered the same ground.

> I would've put more work into a temporary solution, but real life has
> interfered.

Yes, and that's exactly my worry. I don't often get concentrated time
to help out with Parrot and Perl 6, but I have some now. If this is
going to be blocked indefinitely, my tuits are likely to evaporate. And
dangit, I don't want to miss out on my window of opportunity!

> I'm glad to see other people interested in OpenGL on parrot. When the NCI PDD
> is up for drafting, it will be good to have a listing of sample libraries and
> required callback signatures available.

I had already made a list of all of the callback signatures needed by
GLUT, which I've included below.

> I'm excited about the progress Parrot is making.

Ditto that! I'm hoping that when this blocking issue is dealt with,
I'll be able to create a fully functioning OpenGL binding, which is
actually for me a bit of yak shaving -- what I really *want* to do is
port some of my OpenGL code to Perl 6, but I need a working binding
first. :-)


-'f

***********

=head1 Needed Callback Signatures

XXX: Currently Parrot does not support the callback signatures needed
by GLUT to function properly (at all, really). The following are the
missing signatures:

=over 4

=item 'v'

C<glutDisplayFunc>, C<glutOverlayDisplayFunc>, C<glutIdleFunc>,
C<glutCloseFunc>, C<glutMenuDestroyFunc>, C<glutWMCloseFunc>(*)

=item 'vi'

C<glutEntryFunc>, C<glutTimerFunc>, C<glutWindowStatusFunc>,
C<glutVisibilityFunc>(*), C<glutMenuStateFunc>(*)

=item 'vii'

C<glutReshapeFunc>, C<glutMotionFunc>, C<glutPassiveMotionFunc>,
C<glutSpaceballButtonFunc>(!), C<glutButtonBoxFunc>(!),
C<glutDialsFunc>(!), C<glutTabletMotionFunc>(!)

=item 'vcii'

C<glutKeyboardFunc>, C<glutKeyboardUpFunc>

=item 'viii'

C<glutSpecialFunc>, C<glutSpecialUpFunc>, C<glutMenuStatusFunc>,
C<glutSpaceballMotionFunc>(!), C<glutSpaceballRotateFunc>(!)

=item 'viiii'

C<glutMouseFunc>, C<glutMouseWheelFunc>, C<glutJoystickFunc>,
C<glutTabletButtonFunc>(!)

=back


Special marks are as follows:

=over 4

=item (*)

Function is deprecated in OpenGLUT in favor of a more modern callback.

=item (!)

Function is unimplemented in freeglut and OpenGLUT (these all support
rarer input devices; in many cases, the input device in question is
SGI-specific and most likely no longer manufactured).

=back

=cut


Geoffrey Broadwell

unread,
Jan 18, 2008, 11:36:57 PM1/18/08
to Allison Randal, Paul Seamons, perl6-i...@perl.org
On Sat, 2008-01-19 at 17:25 +1300, Allison Randal wrote:

> Geoffrey Broadwell wrote:
> > On Wed, 2008-01-16 at 22:38 -0700, Paul Seamons wrote:
> >>> I am starting to implement a GLUT and OpenGL binding for Parrot.
> [...]

> > I don't often get concentrated time
> > to help out with Parrot and Perl 6, but I have some now. If this is
> > going to be blocked indefinitely, my tuits are likely to evaporate. And
> > dangit, I don't want to miss out on my window of opportunity!
>
> It's true that the generalized solution for varying callback signatures
> doesn't exist yet, but it's easy enough to create your own C callback
> layer, with a separate C function for each callback you need. (Note, not
> one-per-signature, but one-per-callback.) The callback functions can
> invoke a Parrot sub that they lookup by name, register a callback sub
> with the concurrency scheduler, or simply directly perform the actions
> needed.

Barring some better option coming up in the mean time, I will probably
end up doing this. At the very least, I'll get more exposure to
Parrot's guts this way.

Still, I'll also file an RT ticket asking for the general solution under
the "twice requested" rule. I'll reference this thread, Paul's old
thread, and our IRC conversation in the ticket.


-'f


Allison Randal

unread,
Jan 18, 2008, 11:25:16 PM1/18/08
to Geoffrey Broadwell, Paul Seamons, perl6-i...@perl.org
Geoffrey Broadwell wrote:
> On Wed, 2008-01-16 at 22:38 -0700, Paul Seamons wrote:
>>> I am starting to implement a GLUT and OpenGL binding for Parrot.
[...]

> I don't often get concentrated time
> to help out with Parrot and Perl 6, but I have some now. If this is
> going to be blocked indefinitely, my tuits are likely to evaporate. And
> dangit, I don't want to miss out on my window of opportunity!

It's true that the generalized solution for varying callback signatures

doesn't exist yet, but it's easy enough to create your own C callback
layer, with a separate C function for each callback you need. (Note, not
one-per-signature, but one-per-callback.) The callback functions can
invoke a Parrot sub that they lookup by name, register a callback sub
with the concurrency scheduler, or simply directly perform the actions
needed.

Allison

Chromatic

unread,
Jan 19, 2008, 1:29:25 AM1/19/08
to parrot-...@perl.org, Allison Randal
On Friday 18 January 2008 20:25:16 Allison Randal wrote:

> It's true that the generalized solution for varying callback signatures
> doesn't exist yet, but it's easy enough to create your own C callback
> layer, with a separate C function for each callback you need. (Note, not
> one-per-signature, but one-per-callback.) The callback functions can
> invoke a Parrot sub that they lookup by name, register a callback sub
> with the concurrency scheduler, or simply directly perform the actions
> needed.

I'm not even sure *that* will work. To invoke a Sub PMC from C, you need to
pass in an Interp as well as the PMC. Unless you know both of those at
compile time, I'm not sure how to make the callback mechanism work.

... although I just had an evil idea regarding memcpy and a hash table.

-- c

Joshua Juran

unread,
Jan 19, 2008, 6:33:33 PM1/19/08
to Geoffrey Broadwell, Parrot Porters
On Jan 16, 2008, at 7:39 PM, Geoffrey Broadwell wrote:

> I am starting to implement a GLUT and OpenGL binding for Parrot. GLUT
> is extremely callback-oriented.
>

> Unfortunately, none of the GLUT callbacks fall within the current
> limitations on Parrot NCI callbacks.

As you've discovered, callbacks are easy enough to implement portably
provided that an opaque pointer-sized value is one of the arguments
-- you register a single C function, which uses information stored in
the opaque structure to dispatch to the real handler -- and it's
unfortunate that GLUT's callbacks don't support this.

> So ... how can Parrot support callbacks of the types that GLUT uses?
> The trick of putting magical properties on a special user data
> parameter
> won't work anymore. I've been tanking on this for a while, and
> come up
> with several possible schemes.
>
> My first idea was that each time a new callback was registered, Parrot
> would copy a tiny shim function and then poke the address(es) of the
> data it needed to figure out what PIR routine to call directly into
> the
> copied shim code.

This requires special knowledge per platform.

> So my next idea was to have a shim function that did nothing but call
> another function, thus putting its own address on the stack. The
> other
> function would then reach up above its own stack frame and grab that
> return address, using *that address* to look up the needed information
> in a global callback registry. By making copies of this simple shim
> every time a new callback was registered, each copy would get its own
> address, and everything would magically work.
>
> This was certainly better than the previous ideas, because it didn't
> involve poking anything into the shim code, just being able to copy
> it.

But it still requires per-platform support, at least for retrieving
the stack frame pointer.

> But I still don't know how to do that in pure portable C. My guess is
> that you could declare another function (which need not do
> anything) in
> some special way so that it is guaranteed to be compiled into the
> object
> code directly after the magic shim. Subtracting the address of the
> shim
> function from the dummy function would then tell you exactly how many
> bytes long the compiled shim is (including perhaps some padding, which
> doesn't matter). This would allow you to copy the shim at will.

I've seen this done (so it 'works') but the underlying assumption
isn't portable.

> But this still leaves the problems of:
>
> A. Being able to portably force two functions two be compiled
> sequentially in memory. I would hope this is easy, but I don't
> recall which set of declarations makes exactly this guarantee.
>
> B. Being able to execute a copy of the shim on architectures that have
> non-executable stack and/or heap. I assume this had to be solved
> before JIT could work, so I'm guessing this isn't a real problem.
>
> C. Being able to reach up above one's own stack frame and grab the
> return address from the shim function's dummy call. Is there a
> trick to make this work in pure portable C?

No. What you're trying to do can't be done portably. Period.

However, once you place per-platform implementation on the table, it
becomes simple (in theory, at least). Rather than teach Parrot to
cope with GLUT callbacks, I suggest a scheme for creating 'GLUT-
prime' callbacks which accept the opaque structure pointer you're
familiar with, which Parrot can already deal with.

For example: Here's a callback interface that includes the opaque
handle:

void generic_handler( void* user_data );

struct foo foo_data;

install_generic_handler( &generic_handler, &foo_data );

Here's one without:

struct bar bar_data;

typedef void (*handler)();

handler bar_handler = new_handler( &generic_handler, bar_data );

install_handler( bar_handler );

The new_handler() function creates a thunk that loads the data
argument and jumps to the generic handler. In 68K, the thunk code
might (if it were written out) look like:

asm void thunk_zero_args()
{
MOVEA.L #$11223344,A0; // load immediate handler address into A0
MOVE.L #$55667788,-(SP); // load immediate data pointer onto stack
JMP (A0)
}

asm void thunk_one_arg( void* arg1 )
{
MOVEA.L #$11223344,A0; // load immediate handler address into A0
MOVE.L #$55667788,-(SP); // load immediate data pointer onto stack
MOVE.L 8(SP),-(SP); // load previous arg 1 onto stack
JMP (A0)
}

asm void thunk_two_args( void* arg1, void* arg2 )
{
MOVEA.L #$11223344,A0; // load immediate handler address into A0
MOVE.L #$55667788,-(SP); // load immediate data pointer onto stack
MOVE.L 12(SP),-(SP); // load previous arg 2 onto stack
MOVE.L 12(SP),-(SP); // load previous arg 1 onto stack
JMP (A0)
}

Etc. Except you'd generate the actual machine code and store it
somewhere, returning the address from new_handler() (after calling
the magic OS routine that makes it executable).

Now that I think about it, this is just machine-level currying.

Actually turning this into working code and implementing this for
other platforms are left as exercises for the reader.


I haven't written viruses either. Not that there's anything *wrong*
with that... oh, wait.

Josh


Allison Randal

unread,
Jan 20, 2008, 3:55:45 AM1/20/08
to chromatic, parrot-...@perl.org

The concurrency scheduler stores the interp, so it's not a problem for
invoking the PIR callback Sub. But, the current way of looking up the
concurrency scheduler is through the interpreter, so that just pushes
the problem back a step.

Really, any scheduler will do, since it can pass a message to all the
other schedulers. So, the C callback function can just grab the 0th
interpreter from the global interp array and send a callback event to
that scheduler. The actual PIR Sub for the callback can be registered as
a handler in the scheduler, and invoked by the scheduler as a handler
for the callback event.

Allison

Reply all
Reply to author
Forward
0 new messages