Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Packed array status?

8 views
Skip to first unread message

Geoffrey Broadwell

unread,
Feb 25, 2007, 3:40:06 PM2/25/07
to perl6-l...@perl.org
What backends support packed native arrays at this point? And what's
the performance like?

Native access to packed arrays is the big thing I'm looking for before I
port a pile of source filtered Perl 5 code to Perl 6. It's a simple 3D
engine, so all of the libraries I need to work with want to work with
pointers to huge arrays of simple and structured C data types -- and I
need the Perl code to be able to directly manipulate those arrays.

In Perl 5 I'm using PDL for this, and for certain operations it's not
too bad, but it's REALLY slow to cross any of the Perl <-> PDL <-> C
borders (even "give me a pointer to the raw data inside a PDL object" is
a slow operation).

Apropos of the recent thread about specification lockdown:

To those wondering why I haven't started the Perl 6 port already, in
anticipation of this feature working: I'm still adding features (and
even some major rewrites) in the Perl 5 version. When I do the port, I
want to be able to cut over from one working code base to another. I
don't want a period of bitrot in which the Perl 6 port is "done" but not
working, and continuously at risk of being out of sync with changes to
the working Perl 5 codebase. I also want to be able to quickly graduate
from a "pure port" design to a colloquial Perl 6 design, which I can't
safely do and still keep syncing updates reasonable.

If I could expect packed arrays to be working at some *known* point in
the future, I could probably aim for that. However, my recollection of
the Perl 6 history so far is that itch-scratching and bootstrapping are
the two forces driving the implementations, so I've no idea when I could
expect the feature to appear if it hasn't already -- and hence no desire
to start now and risk the aforementioned bitrot for an unknown and
possibly long period.

(I'm *NOT* complaining about the state of implementation -- the spec
lockdown thread just reminded me that the core team might want to know
why someone who is very excited about Perl 6 isn't jumping in with both
feet. My gut feeling is that the details may change, but my hesitation
is probably not unusual.)


-'f


Chromatic

unread,
Feb 25, 2007, 6:48:47 PM2/25/07
to perl6-l...@perl.org, Geoffrey Broadwell
On Sunday 25 February 2007 12:40, Geoffrey Broadwell wrote:

> What backends support packed native arrays at this point? And what's
> the performance like?

Parrot does have ManagedStruct and UnManagedStruct PMCs for mapping complex C
structures. The syntax to define them is a little grotty, but once you've
done that, using them is easy--especially with named access.

Performance is good. You don't have to cross the C/HLL boundary to work with
the memory, as Parrot can do that directly.

I don't know if Patrick has using PIR libraries working in Perl 6 yet, but the
last time we talked about it, he said it would take just a bit of work.

Wrapping well-designed libraries for Parrot's NCI is easy. I can show you the
basics in an hour and you'll be well on your way.

-- c

Patrick R. Michaud

unread,
Feb 26, 2007, 9:25:53 AM2/26/07
to chromatic, perl6-l...@perl.org, Geoffrey Broadwell
On Sun, Feb 25, 2007 at 03:48:47PM -0800, chromatic wrote:
> On Sunday 25 February 2007 12:40, Geoffrey Broadwell wrote:
>
> > What backends support packed native arrays at this point? And what's
> > the performance like?
>
> I don't know if Patrick has using PIR libraries working in Perl 6
> yet, but the last time we talked about it, he said it would take
> just a bit of work.

No, I don't have them working yet, but implementing them shouldn't
be too difficult. I just need to have perl6 recognize imported
classnames. (The syntax for making method calls is already in
place and working.)

Pm

Geoffrey Broadwell

unread,
Feb 26, 2007, 11:29:36 AM2/26/07
to Patrick R. Michaud, chromatic, perl6-l...@perl.org

Is there a page where you keep your working / in progress / not working
status information? If so, I can look there to see how close Perl 6 on
Parrot is to what I'll need for the port.

As a simpler case than a full 3D engine port, I have some OpenGL
benchmarks in Perl 5 that I can port -- these have much reduced
requirements. Principly, they need:

1. Basic math and string operators (not grammars)
2. Basic looping and simple control structures
3. Basic subroutine handling
4. Basic I/O: read file, print to STDOUT (printf is a bonus)
5. Read access to command line args
6. Perl scalars, hashes, and arrays, with one level nesting
7. Packed arrays with access to raw data pointer to give to API
8. eval (tight inner loops are generated)
9. High-resolution time
10. Access to SDL & OpenGL constants
11. Procedural calls to OpenGL, with scalar return values
12. Procedural or OO calls to SDL, with scalar (and object, if OO)
return values (no callbacks)

Requirements 3, 4, 5, 8, 9, 10 can all be worked around with some
effort, but the rest would be quite painful to work around. It is
possible to do part of the benchmark without requirement 7 (the one that
started all this) just to make sure the other stuff is functional, but
it's a pretty limited benchmark without the packed arrays.

How is Perl 6 on Parrot doing with regard to the above list?


-'f


Geoffrey Broadwell

unread,
Feb 26, 2007, 1:54:39 PM2/26/07
to aud...@audreyt.org, perl6-l...@perl.org
> Pugs at the moment support all of the above, using the Perl 5 bridge
> for "use perl5:SDL" and "use perl5:OpenGL". So the sole requirement
> seems to be:

Cool beans. I'd had some simple OpenGL code working-with-hacks on Pugs
many months ago, but I did not know the current status after all the
internals churn.

> > 7. Packed arrays with access to raw data pointer to give to API
>

> Is it possible to point us to some use cases of such packed arrays,
> especially the raw data pointer API part?

Are you looking for Perl code that creates such packed arrays and passes
them to OpenGL? Or are you looking for links to manpages for the OpenGL
calls themselves? Or both?

> Also, if you would translate a few such use cases to the syntax in S09
> (http://perlcabal.org/syn/S09.html) and committing them under
> t/data_types/, then it'd be much easier to measure which parts of
> packed arrays needs to be specced/implemented first.

I can write some tests that build packed arrays of some of the types I
need and then go snooping around it checking the contents ... would that
help? Somehow I'm thinking it's a bad thing if data types tests require
OpenGL ... but it seems hard to tell whether the implementation is
actually creating a packed array, or just faking the Perl-side behavior
using an unpacked array, unless we make a call to some C API that can
confirm the results.

Hmmmm ... a quick scan of S09 indicates a packed array of any low-level
type can be treated as a buf, so I might be able to get sneaky with that
equivalence in order to reach a high probability that either the packing
is real or the implementor is insane.


-'f


Geoffrey Broadwell

unread,
Feb 26, 2007, 2:29:23 PM2/26/07
to aud...@audreyt.org, perl6-l...@perl.org
On Tue, 2007-02-27 at 03:02 +0800, Audrey Tang wrote:
> 2007/2/27, Geoffrey Broadwell <ge...@broadwell.org>:

> > Are you looking for Perl code that creates such packed arrays and passes
> > them to OpenGL? Or are you looking for links to manpages for the OpenGL
> > calls themselves? Or both?
>
> The former.

OK, I'll pull some of the interesting routines out of my code and post
them.

> > I can write some tests that build packed arrays of some of the types I
> > need and then go snooping around it checking the contents ... would that
> > help? Somehow I'm thinking it's a bad thing if data types tests require
> > OpenGL ... but it seems hard to tell whether the implementation is
> > actually creating a packed array, or just faking the Perl-side behavior
> > using an unpacked array, unless we make a call to some C API that can
> > confirm the results.
>

> That is correct. However as you noted, our buf should be as good as a
> C-level packed buffer, so you can assume that when writing the tests.
> Alternately, we can assume some fairly simple C FFI calls such as
> strlen() (or some such) that manages the structs we'd like it to
> manage.

I'm being an idiot -- if we can assume Perl 5 connectivity, I can just
pass the buf as a string over to Perl 5, unpack there, return the
unpacked values to Perl 6, and compare.

Does Perl 6 on Parrot have Perl 5 connectivity?


-'f


Chromatic

unread,
Feb 26, 2007, 7:29:55 PM2/26/07
to perl6-l...@perl.org, Geoffrey Broadwell
On Monday 26 February 2007 11:29, Geoffrey Broadwell wrote:

> Does Perl 6 on Parrot have Perl 5 connectivity?

Not until Perl 6 can use PIR code. After that, it depends on what you want to
do with the two.

If you can get Parrot::Embed compiled and running on your machine, Perl 5 can
have Parrot connectivity. (As far as I know, Windows is the only broken
platform now, but I already knew that.)

-- c

Audrey Tang

unread,
Feb 26, 2007, 12:31:55 PM2/26/07
to Geoffrey Broadwell, perl6-l...@perl.org
2007/2/27, Geoffrey Broadwell <ge...@broadwell.org>:

> As a simpler case than a full 3D engine port, I have some OpenGL
> benchmarks in Perl 5 that I can port -- these have much reduced
> requirements. Principly, they need:
>
> 1. Basic math and string operators (not grammars)
> 2. Basic looping and simple control structures
> 3. Basic subroutine handling
> 4. Basic I/O: read file, print to STDOUT (printf is a bonus)
> 5. Read access to command line args
> 6. Perl scalars, hashes, and arrays, with one level nesting
> 8. eval (tight inner loops are generated)
> 9. High-resolution time
> 10. Access to SDL & OpenGL constants
> 11. Procedural calls to OpenGL, with scalar return values
> 12. Procedural or OO calls to SDL, with scalar (and object, if OO)
> return values (no callbacks)

Pugs at the moment support all of the above, using the Perl 5 bridge


for "use perl5:SDL" and "use perl5:OpenGL". So the sole requirement
seems to be:

> 7. Packed arrays with access to raw data pointer to give to API

Is it possible to point us to some use cases of such packed arrays,


especially the raw data pointer API part?

Also, if you would translate a few such use cases to the syntax in S09


(http://perlcabal.org/syn/S09.html) and committing them under
t/data_types/, then it'd be much easier to measure which parts of
packed arrays needs to be specced/implemented first.

Cheers,
Audrey

Audrey Tang

unread,
Feb 26, 2007, 2:02:27 PM2/26/07
to Geoffrey Broadwell, perl6-l...@perl.org
2007/2/27, Geoffrey Broadwell <ge...@broadwell.org>:

> > > 7. Packed arrays with access to raw data pointer to give to API
> >
> > Is it possible to point us to some use cases of such packed arrays,
> > especially the raw data pointer API part?
>
> Are you looking for Perl code that creates such packed arrays and passes
> them to OpenGL? Or are you looking for links to manpages for the OpenGL
> calls themselves? Or both?

The former.

> > Also, if you would translate a few such use cases to the syntax in S09
> > (http://perlcabal.org/syn/S09.html) and committing them under
> > t/data_types/, then it'd be much easier to measure which parts of
> > packed arrays needs to be specced/implemented first.
>
> I can write some tests that build packed arrays of some of the types I
> need and then go snooping around it checking the contents ... would that
> help? Somehow I'm thinking it's a bad thing if data types tests require
> OpenGL ... but it seems hard to tell whether the implementation is
> actually creating a packed array, or just faking the Perl-side behavior
> using an unpacked array, unless we make a call to some C API that can
> confirm the results.

That is correct. However as you noted, our buf should be as good as a


C-level packed buffer, so you can assume that when writing the tests.
Alternately, we can assume some fairly simple C FFI calls such as
strlen() (or some such) that manages the structs we'd like it to
manage.

Thanks,
Audrey

Geoffrey Broadwell

unread,
Feb 26, 2007, 8:12:35 PM2/26/07
to chromatic, perl6-l...@perl.org

OK, between chromatic and Audrey, it sounds like the paths forward are:

1. Add tests to Pugs for packed data handling.
2. Pugs implements functionality for said tests.
3. Port of OpenGL code proceeds assuming use of SDL/OpenGL via Perl 5.
4. Perl 6 on Parrot implements PIR calls.
5. Perl 6 on Parrot gets native packed arrays.

Then either:

6. Perl 6 on Parrot gets Perl 5 connectivity.
7. Perl 6 on Parrot uses #3 as written.

or:

6. SDL/OpenGL are wrapped directly in PIR.
7. #3 is rewritten to use native Parrot modules.

Of course, *both* paths will probably eventually be followed, it's just
a question of which one works first. :-)

In the mean time, I need to try to find my commit details for the Pugs
repo so that I can check out a copy and start on step 1.


-'f


0 new messages