XS in Perl 6
Aaron Sherman
things at SOME point.... It would be so nice if Perl 6's XS was part of
the language, rather than an external pre-processor.
Something like:
module somesuch;
use External (language=>"C");
sub chdir(string $path //= $ENV{HOME}) is external(returns=>'int');
Then you could extract the C source like so:
perl -MExternal -e 'extract "chdir"'
Which would generate something along the very vague lines of:
#include<parrot_external.h>
/* modulename.c */
void perlexternal_chdir(parrot_sv *path) {
char* param_path = parrot_sv_to_char_STAR(path);
int RETVAL = chdir(param_path);
parrot_stackpush(parrot_int_to_iv(RETVAL));
}
The more complex case:
module getpw;
use External(language=>"C",PREP=>q{
#include<pwd.h>
#include<sys/types.h>});
class uid_t := class real; # Proposed class alias syntax (see below)
class gid_t := class real;
class passwd {
my string $.pw_name; # Proposed pseudo-classes "string" and "real"
my string $.pw_passwd; # are required to determine how to convert
my int $.pw_uid; # the value to C
my int $.pw_gid;
my string $.pw_gecos;
my string $.pw_dir;
my string $.pw_shell;
method storage { # storage returns a string name of the C type
# or a listref to the field names if it's a
# struct. If it's a scalar type in C, then
# this class must contain a $.internal of the
# appropriate Perl pseudo-class type (string,
# int or real).
return
[qw{
pw_name pw_passwd pw_uid pw_gid
pw_gecos pw_dir pw_shell
}]
}
}
sub getpwuid(uid_t $uid //= $_) is external(returns=>'passwd');
So, to recap:
All types passed to an externally defined function must be classes which
have well defined type behavior.
A subroutine is declared external by using "is external" which takes
argument pairs as parameters.
One of external's parameters is "returns" which take a class name as its
parameter. The function will return an object of the given class.
Classes can be aliased using the C<:=> construct
The following pseudo-classes are defined, which force a parrot-internal
storage and define the mechanism by which the value is translated to
external representation (these are all scalar types):
int - signed integer value
real - floating point
string - A character string
utf8string - A string which is never converted to ASCII
bytes - A string which may contain binary data including nul
each of these will implicitly have a method called "storage" which
returns their class name, for consistency.
There are some limitations to this very abstract approach, but most of
those can be overcome by defining glue functions in External's "PREP"
value. For example, if your target language is C++, and the default way
of implementing "is rw" in C++ is to expect the target function to take
a reference, then you could define your own glue function:
module a;
use External (language=>'C++', PREP=>q{
int _b(int& c) { aab(&c) }
});
sub b(int $c is rw) is external(returns=>'int', call=>'_b');
This lets us do the quick-and-dirty conversion automatically and the
fine-grained things in the target language.
The big advantage to this mechanism is that it does not require a user
to know the internals of Perl or Parrot in order to create linkage to
external programs.
Thoughts?
--
Aaron Sherman X137
a...@itasoftware.com
"We had some good machines, but they don't work no more."
-"Faded Flowers" / Shriekback
David Whipp
> I'm thinking XS thoughts because we're going to need a few external
> things at SOME point.... It would be so nice if Perl 6's XS
> was part of the language, rather than an external pre-processor.
> Thoughts?
Its good to start thinking about this early; but should we be thinking
"Parrot's" XS; not "Perl 6's"?
Parrot has to solve the problem of binding multiple languages together
within a unified framework: Ruby calls a python method, passing a
Perl object as a parameter. I would hope that the solution to that
challenge will be sufficiently powerful to solve the problem of
describing/generating an interface to C, C++, Java, Eiffel, and
INTERCAL.
Given that parrot will solve it in a language-neutral way, I am
not sure that we should focus an a Perl6 XS, "rather than an
external pre-processor". We may want a Perl6 module that hides
the complexity of the Parrot stuff; but that depends on how complex
the Parrot stuff turns out to be.
Dave.
Brent Dax
# I'm thinking XS thoughts because we're going to need a few
# external things at SOME point.... It would be so nice if Perl
# 6's XS was part of the language, rather than an external
# pre-processor.
#
# Something like:
#
# module somesuch;
# use External (language=>"C");
# sub chdir(string $path //= $ENV{HOME}) is
# external(returns=>'int');
I prefer:
module System::FS is version(1.0.0);
use Parrot::XS 'load';
sub chdir(string $path //= $ENV{HOME}) is external returns(int);
sub curdir() is external returns(string);
# 'returns' would be Perl's normal way to declare return
# value, not a special thing for external functions only.
load module => System::FS, version => 1.0;
But that's just the shell around the real XS file:
parrot_xs {
module = System::FS;
pversion = 0.0.8;
xsversion = 0.0.1;
modversion = 1.0.0;
}
#include <chdir's_header.h>
xsub extern int chdir(char*);
xsub char* curdir() {
...
}
/* Creates a wrapper and argument converter like:
Parrot_PMC * Parrot_XS_System__FS_dispatch(Parrot_String func,
Parrot_PMC obj, Parrot_PMC retto, Parrot_PMC args) {
if(Parrot_string_eq_cstr(func, "chdir")) {
Parrot_pmc_set_integer(
retto,
chdir(
Parrot_string_to_cstr(
Parrot_pmc_get_string(
Parrot_pmc_get_keyed_integer(
args, 0
)
)
)
)
);
}
else if(Parrot_string_eq_cstr(func, "curdir")) {
Parrot_pmc_set_string(
retto,
curdir()
);
}
else {
Parrot_xs_die(Parrot_sprintf_cstr("Function %S
not found!", func));
}
}
except with all the missing references to 'interpreter'.
*/
Of course, if you wanted to, you could use something like Inline.pm:
module System::FS is version(1.0);
use Parrot::XS::Inline;
inline C => qq{
#include <chdir's_header.h>
};
sub chdir(string $path //= $ENV{HOME} is external('C', 'int',
'char*') returns(int);
sub curdir() is external('C', 'char*') returns(string) {
/* C code here */
...
}
--Brent Dax <bren...@cpan.org>
@roles=map {"Parrot $_"} qw(embedding regexen Configure)
"In other words, it's the 'Blow up this Entire Planet and Possibly One
or Two Others We Noticed on our Way Out Here' operator."
--Damian Conway
Aaron Sherman
> Aaron Sherman [mailto:a...@itasoftware.com] wrote:
> > I'm thinking XS thoughts because we're going to need a few external
> > things at SOME point.... It would be so nice if Perl 6's XS
> > was part of the language, rather than an external pre-processor.
> > [ some interesting stuff]
> > Thoughts?
>
> Its good to start thinking about this early; but should we be thinking
> "Parrot's" XS; not "Perl 6's"?
I disagree. Parrot will have a way to access runtime-loadable libraries
and call them from parrot code, but the argument marshaling and type
conversion specification will all happen in the way that the target
language specifies, since that language will be constrained in how it
can take advantage of Parrot and the native library it's accessing.
Yes, Parrot's facility for external binding must be established, but
it's almost 100% tangential to the discussion of how Perl 6 will
interface through that facility.
> Parrot has to solve the problem of binding multiple languages together
> within a unified framework: Ruby calls a python method, passing a
> Perl object as a parameter.
Good point. However, Perl6 will call Ruby in different ways than Python
will. Certainly a function that returns a list in Perl would be called
differently form Scheme or BASIC, since Scheme supports list types
albeit very different ones from Perl, where BASIC does not.
No, Parrot will make it easier for these languages to call external
binary objects or PBC modules from other languages, but the solution
will ultimately involve a heavy amount of custom work per-language.
--
Aaron Sherman <a...@ajs.com>
http://www.ajs.com/~ajs
Aaron Sherman
> Aaron Sherman:
> # I'm thinking XS thoughts
> # Something like:
> #
> # module somesuch;
> # use External (language=>"C");
> # sub chdir(string $path //= $ENV{HOME}) is
> # external(returns=>'int');
>
> I prefer:
>
> module System::FS is version(1.0.0);
> use Parrot::XS 'load';
>
> sub chdir(string $path //= $ENV{HOME}) is external returns(int);
> sub curdir() is external returns(string);
> # 'returns' would be Perl's normal way to declare return
> # value, not a special thing for external functions only.
I like "returns" as a property. Much cleaner, and I really like the
ability to specify a return type generally.
You do need to be able to specify the target language, though.
> load module => System::FS, version => 1.0;
I would think that this version number and the one in the module
statement before would always be the same (since they're in the same
file, and the XS is generated by this file). What's more, this whole
statement could be considered "implied" by the use of C<Parrot::XS>.
> #include <chdir's_header.h>
Too magical. You need to have some way to specify an arbitrary string in
your module that will be exported to the target language ala my "PREP"
parameter. That C<#include> statement would go in the string along with
any glue functions required.
So, to re-write both of our suggestions as one:
module System::FS is version(1.0.0);
use Parrot::XS language => 'C', prep => q{
#include <unistd.h>
};
sub chdir(string $path //= $ENV{HOME}) is external returns(class int);
sub curdir() is external returns(class string);
I stuck the "class" keyword in with the returns property because
otherwise I think it's ambiguous.
However, that was my simple example. Here again (and slightly re-written
based on your suggestions) is my more complex example:
module getpw is version(1.0.0);
use Parrot::XS language => 'C', prep => q{
#include<pwd.h>
#include<sys/types.h>
};
class uid_t := class real; # Proposed class alias syntax (see below)
class gid_t := class real;
class passwd {
my string $.pw_name; # Proposed pseudo-classes "string" and "real"
my string $.pw_passwd; # are required to determine how to convert
my int $.pw_uid; # the value to C
my int $.pw_gid;
my string $.pw_gecos;
my string $.pw_dir;
my string $.pw_shell;
method storage { # storage returns a string name of the C type
# or a listref to the field names if it's a
# struct. If it's a scalar type in C, then
# this class must contain a $.internal of the
# appropriate Perl pseudo-class type (string,
# int or real).
return
[qw{
pw_name pw_passwd pw_uid pw_gid
pw_gecos pw_dir pw_shell
}]
}
}
sub getpwuid(uid_t $uid //= $_) is external returns(class passwd);
Brent Dax
# On Thu, 2002-09-12 at 21:10, Brent Dax wrote:
# > Aaron Sherman:
# > # I'm thinking XS thoughts
#
# > # Something like:
# > #
# > # module somesuch;
# > # sub chdir(string $path //= $ENV{HOME}) is
# > # external(returns=>'int');
# >
# > I prefer:
# >
# > module System::FS is version(1.0.0);
# > use Parrot::XS 'load';
# >
# > sub chdir(string $path //= $ENV{HOME}) is external returns(int);
# > sub curdir() is external returns(string);
# > # 'returns' would be Perl's normal way to declare return
# > # value, not a special thing for external functions only.
#
# I like "returns" as a property. Much cleaner, and I really
# like the ability to specify a return type generally.
#
# You do need to be able to specify the target language, though.
I don't think so. As long as it's something that C<load> can load in,
why should you have to specify the language?
# > load module => System::FS, version => 1.0;
#
# I would think that this version number and the one in the
# module statement before would always be the same (since
# they're in the same file, and the XS is generated by this
# file). What's more, this whole statement could be considered
# "implied" by the use of C<Parrot::XS>.
Except that I might want to implement this with a number of
platform-specific modules, to make sure I don't try to pull in another
platform's module:
given($*OS) { #Or whatever $^O becomes
#Make sure I don't try to use something designed for
# a different platform
when /Solaris/ { load module => System::FS::Solaris,
version => 1.0 }
when /Linux/ { load module => System::FS::Linux ,
version => 1.0 }
...
}
Since the loaded module need not have the same name as the surrouding
module, the version number is needed.
# > #include <chdir's_header.h>
#
# Too magical. You need to have some way to specify an
# arbitrary string in your module that will be exported to the
# target language ala my "PREP" parameter. That C<#include>
# statement would go in the string along with any glue
# functions required.
I consider this more generic--the C<inline> command is for any arbitrary
code.
# So, to re-write both of our suggestions as one:
#
# module System::FS is version(1.0.0);
# use Parrot::XS language => 'C', prep => q{
# #include <unistd.h>
# };
#
# sub chdir(string $path //= $ENV{HOME}) is external
# returns(class int);
# sub curdir() is external returns(class string);
#
# I stuck the "class" keyword in with the returns property
# because otherwise I think it's ambiguous.
Why do you need your own class? Presumably XS would know how to convert
between C's C<int> and Perl's C<int>...
# However, that was my simple example. Here again (and slightly
# re-written based on your suggestions) is my more complex example:
#
# module getpw is version(1.0.0);
# use Parrot::XS language => 'C', prep => q{
# #include<pwd.h>
# #include<sys/types.h>
# };
#
# class uid_t := class real; # Proposed class alias syntax
# (see below)
# class gid_t := class real;
# class passwd {
# my string $.pw_name; # Proposed pseudo-classes
# "string" and "real"
# my string $.pw_passwd; # are required to determine
# how to convert
# my int $.pw_uid; # the value to C
C<string> is already a built-in type, and presumably XS would understand
how to convert a C<char *> into a C<string>. But why use C<real>? Are
C<int> and C<num> not enough? (Or are you using C<real> instead of
C<num> because you don't know about C<num>? Or does it say somewhere
that it's C<real> and I'm incorrect in thinking it's C<num>?)
# my int $.pw_gid;
# my string $.pw_gecos;
# my string $.pw_dir;
# my string $.pw_shell;
# method storage { # storage returns a string name of the C type
# # or a listref to the field names if it's a
# # struct. If it's a scalar type in C, then
# # this class must contain a $.internal of the
# # appropriate Perl pseudo-class type (string,
# # int or real).
# return
# [qw{
# pw_name pw_passwd pw_uid pw_gid
# pw_gecos pw_dir pw_shell
# }]
# }
# }
Why not:
class passwd is xs_struct(
qw(pw_name pw_passwd pw_uid pw_gid pw_gecos pw_dir
pw_shell)
) {
...
And for simpler types:
class Parrot::String is xs_type('Parrot_String' => $.ptr) {
my Parrot::XS::Pointer $.ptr;
...
# sub getpwuid(uid_t $uid //= $_) is external returns(class passwd);
Once again, why the C<class>?
Dan Sugalski
>Aaron Sherman [mailto:a...@itasoftware.com] wrote:
>> I'm thinking XS thoughts because we're going to need a few external
>> things at SOME point.... It would be so nice if Perl 6's XS
>> was part of the language, rather than an external pre-processor.
>> [ some interesting stuff]
>> Thoughts?
>
>Its good to start thinking about this early; but should we be thinking
>"Parrot's" XS; not "Perl 6's"?
Yes, we should. Perl 6 XS is Apocalypse 21. That's likely rather a ways off...
--
Dan
--------------------------------------"it's like this"-------------------
Dan Sugalski even samurai
d...@sidhe.org have teddy bears and even
teddy bears get drunk
Aaron Sherman
> Aaron Sherman:
> # You do need to be able to specify the target language, though.
>
> I don't think so. As long as it's something that C<load> can load in,
> why should you have to specify the language?
We were talking past each other. In reviewing, I see that you went back
to the Perl 5 way of having a separate XS and PM file. I was trying to
escape that, and just have the whole thing in Perl. In fact, if you look
over your "real XS file", you will see that most of it is redundant with
the information in your Perl module. Why not combine them? By adding a
couple of lines to your Perl code, we can do away with this separate
language called XS entirely.
> # > load module => System::FS, version => 1.0;
> #
> # I would think that this version number and the one in the
> # module statement before would always be the same (since
> # they're in the same file, and the XS is generated by this
> # file). What's more, this whole statement could be considered
> # "implied" by the use of C<Parrot::XS>.
>
> Except that I might want to implement this with a number of
> platform-specific modules, to make sure I don't try to pull in another
> platform's module:
Again, you're assuming external XS. If you internalize this in Perl, it
suddenly makes sense to have a C<System::FS> which detects platform and
calls the Perl module C<System::FS::$Config{os}>
> # Too magical. You need to have some way to specify an
> # arbitrary string in your module that will be exported to the
> # target language ala my "PREP" parameter. That C<#include>
> # statement would go in the string along with any glue
> # functions required.
>
> I consider this more generic--the C<inline> command is for any arbitrary
> code.
In my other example, I put a C function in the PREP block, so I don't
think I ever had any intention of limiting its usability.
> # sub curdir() is external returns(class string);
> #
> # I stuck the "class" keyword in with the returns property
> # because otherwise I think it's ambiguous.
>
> Why do you need your own class? Presumably XS would know how to convert
> between C's C<int> and Perl's C<int>...
C<int> in Perl 6 is essentially a class name (grammatically, at least)
even though it is slightly more magical than that. If you want to put a
class name inside those parentheses, you have to disambiguate it somehow
(Perl 6 does not have barewords). So, I added the C<class> keyword to
allow the parser to identify that the next token should be a
pre-declared class name (the same way we disambiguate C<my Dog $spot>
because of the C<my> keyword).
> C<string> is already a built-in type, and presumably XS would understand
> how to convert a C<char *> into a C<string>. But why use C<real>? Are
> C<int> and C<num> not enough? (Or are you using C<real> instead of
> C<num> because you don't know about C<num>? Or does it say somewhere
> that it's C<real> and I'm incorrect in thinking it's C<num>?)
I asked this list a couple of weeks ago, and I thought the consensus
reply was "real". I too had assumed "num" before.
> Why not:
>
> class passwd is xs_struct(
> qw(pw_name pw_passwd pw_uid pw_gid pw_gecos pw_dir
> pw_shell)
> ) {
Actually I think you can combine the two for simpler code than I was
suggesting. You do need to specify types for the members, though:
class a is xs_struct(qw(b d)) {
my int $.b;
my c $.d;
}
class c is xs_struct(qw(a b)) {
my string $.a;
my string $.b;
}
Then you should be able to say:
my a $foo = some_xs_function();
print $foo.c.a;
Assuming auto-accessors, of course.
> class Parrot::String is xs_type('Parrot_String' => $.ptr) {
> my Parrot::XS::Pointer $.ptr;
> ...
I think that should be internal to Parrot.
You just say C<sub foo(string $a)> and XS knows what Parrot code to emit
for handling the type conversions.
Aaron Sherman
> At 5:09 PM -0700 9/12/02, David Whipp wrote:
> >Its good to start thinking about this early; but should we be thinking
> >"Parrot's" XS; not "Perl 6's"?
>
> Yes, we should. Perl 6 XS is Apocalypse 21. That's likely rather a ways off...
We'll ignore my previous response to David's question for a second. Just
taking on your answer, I'm perplexed. The Perl compiler is in rough
shape, but it's moving along nicely. I've been able to at least parse
all of the builtin functions I've written (though I have no idea what to
do with them until the parameter-passing bugs are fixed by Sean or
myself, whoever gets there first). Are we saying that we're going to
wait for 16 more Apocalypses before we write code that allows C<chdir()>
to call the C library function?! If that's the case, I'm going to go
back to programming in Perl 5, and wait for 5 years or so to come back
and continue work on Perl 6. :(
I think what we should be doing is agreeing generally on prototype
features, implementing them, and then letting that serve as ground-work
for the network notes from Larry.
Brent Dax
# We were talking past each other. In reviewing, I see that you
# went back to the Perl 5 way of having a separate XS and PM
# file. I was trying to escape that, and just have the whole
# thing in Perl. In fact, if you look over your "real XS file",
# you will see that most of it is redundant with the
# information in your Perl module. Why not combine them? By
# adding a couple of lines to your Perl code, we can do away
# with this separate language called XS entirely.
Because Python and Ruby and Scheme might have different front-end
signatures for the same back-end C code. Because some C functions will
want C<char *> and others C<Parrot_String> and still others a
C<Parrot_PMC> for a Perl-level C<string>. Need I continue?
# Again, you're assuming external XS. If you internalize this
# in Perl, it suddenly makes sense to have a C<System::FS>
# which detects platform and calls the Perl module
# C<System::FS::$Config{os}>
Except that C<use> is compile-time and can't be given a string.
# > I consider this more generic--the C<inline> command is for any
# > arbitrary code.
#
# In my other example, I put a C function in the PREP block, so
# I don't think I ever had any intention of limiting its usability.
But you can't intersperse your XSUBs with normal C code. (For, say,
C<#define> nonsense.)
# C<int> in Perl 6 is essentially a class name (grammatically,
# at least) even though it is slightly more magical than that.
# If you want to put a class name inside those parentheses, you
# have to disambiguate it somehow (Perl 6 does not have
# barewords). So, I added the C<class> keyword to allow the
# parser to identify that the next token should be a
# pre-declared class name (the same way we disambiguate C<my
# Dog $spot> because of the C<my> keyword).
I think that, since there's no other possible meaning for C<int> in that
context, it becomes a 'class object' of some sort. (Or maybe the
C<returns> property would be prototyped to take a class name. *shrugs*)
# > C<string> is already a built-in type, and presumably XS would
# > understand how to convert a C<char *> into a C<string>.
# But why use
# > C<real>? Are C<int> and C<num> not enough? (Or are you
# using C<real>
# > instead of C<num> because you don't know about C<num>? Or
# does it say
# > somewhere that it's C<real> and I'm incorrect in thinking it's
# > C<num>?)
#
# I asked this list a couple of weeks ago, and I thought the
# consensus reply was "real". I too had assumed "num" before.
OK. I'll drop this point, then. :^)
# > Why not:
# >
# > class passwd is xs_struct(
# > qw(pw_name pw_passwd pw_uid pw_gid pw_gecos pw_dir
# > pw_shell)
# > ) {
#
# Actually I think you can combine the two for simpler code
# than I was suggesting. You do need to specify types for the
# members, though:
I wasn't suggesting that you didn't need to give types. That class
block was left incomplete because I was lazy.
# class a is xs_struct(qw(b d)) {
# my int $.b;
# my c $.d;
# }
# class c is xs_struct(qw(a b)) {
# my string $.a;
# my string $.b;
# }
#
# Then you should be able to say:
#
# my a $foo = some_xs_function();
# print $foo.c.a;
#
# Assuming auto-accessors, of course.
I don't see how this is different from what I was talking about...
# > class Parrot::String is xs_type('Parrot_String' => $.ptr) {
# > my Parrot::XS::Pointer $.ptr;
# > ...
#
# I think that should be internal to Parrot.
I think that the C<Parrot::Internals> module should be able to create
C<Parrot::String>, so you can say
C<$parrot_str.transcode($Parrot::Internals::CurInterp, 'utf8')> if you
want. ;^)
If you mean that I shouldn't be able to store a pointer in a Perl
object, then there are a lot of C structures you'll have a hard time
wrapping portably.
Nicholas Clark
> The following pseudo-classes are defined, which force a parrot-internal
> storage and define the mechanism by which the value is translated to
> external representation (these are all scalar types):
>
> int - signed integer value
> real - floating point
> string - A character string
> utf8string - A string which is never converted to ASCII
> bytes - A string which may contain binary data including nul
I find this vague and open to misinterpretation as a way of binding to C.
To bind to C sanely I believe that we must be clear *what* C types we're
binding to. Hence for the numeric types
int - C's int (signed)
uint - C's unsigned int
real - C's double
(and probably also types for long (signed and unsigned), short (signed and
unsigned), long long (signed and unsigned), size_t, float and long double, as
well as integers signed and unsigned of exactly 8, 16, 32 and 64 bits, and a
second set of at least 8, 16, 32 and 64. (with some sort of exception if you
try to use a size that isn't supported, such as exactly 16 or 32 on a Cray,
or long long on a real C89 system)
the parrot names are not important, but having the mapping well defined and
utterly unambiguous is
just saying "it's an int" is open to misinterpretation, particularly when
parrot may well be compiled with INTVAL as some type larger than int.
I confess I've not really understood the rest of the thread, mainly because
my brain feeling exhausted (and likely to stay that way until after YAPC::EU)
but I would hope that for simple things the wrapping interface to parrot and
perl6 works like Inline.
[plus avoiding Inline's current niggles, by being able to cope with function
prototypes that are (void), and parameters that are const and unsigned]
Also, I'd want to be able to process C structs just by defining them as a
struct in C. Or by saying "it's a variable of type struct stat, defined in
<sys/stat.h>" and just have parrot parse the relevant include files.
Nicholas Clark
--
Even better than the real thing: http://nms-cgi.sourceforge.net/
Tim Bunce
> On Fri, 2002-09-13 at 00:10, Brent Dax wrote:
> > Aaron Sherman:
>
> > # You do need to be able to specify the target language, though.
> >
> > I don't think so. As long as it's something that C<load> can load in,
> > why should you have to specify the language?
>
> We were talking past each other. In reviewing, I see that you went back
> to the Perl 5 way of having a separate XS and PM file. I was trying to
> escape that, and just have the whole thing in Perl. In fact, if you look
> over your "real XS file", you will see that most of it is redundant with
> the information in your Perl module. Why not combine them? By adding a
> couple of lines to your Perl code, we can do away with this separate
> language called XS entirely.
I'm thowing this quote from Larry into the thread just as a reminder
to everyone what he is (or was) thinking on the Perl6 side:
: The hope is to extend Perl's subroutine declaration syntax (via types
: and attributes) to the point where a "forward" declaration in Perl of a
: C, Java, or C# routine can supply all the glue information formerly
: supplied by XS. While this will undoubtedly give us some rather
: strange looking Perl, I'd rather look at potentially strange Perl than
: certainly strange XS.
And then for those who really can't let go of XS there's always the
option of just embedding it into your perl code:
{
use XS;
bool_t
rpcb_gettime(host,timep)
char *host
time_t &timep
OUTPUT:
timep
}
Tim.
Sean O'Rourke
> (though I have no idea what to do with them until the
> parameter-passing bugs are fixed by Sean or myself, whoever gets there
> first).
"sub foo(@required, *@slurp)" should now behave itself.
/s
Aaron Sherman
> Aaron Sherman:
> # We were talking past each other. In reviewing, I see that you
> # went back to the Perl 5 way of having a separate XS and PM
> # file. I was trying to escape that, and just have the whole
> # thing in Perl. In fact, if you look over your "real XS file",
> # you will see that most of it is redundant with the
> # information in your Perl module. Why not combine them? By
> # adding a couple of lines to your Perl code, we can do away
> # with this separate language called XS entirely.
>
> Because Python and Ruby and Scheme might have different front-end
> signatures for the same back-end C code. Because some C functions will
> want C<char *> and others C<Parrot_String> and still others a
> C<Parrot_PMC> for a Perl-level C<string>. Need I continue?
There are some great reasons for there to be an IMC, but I should not
have to learn IMC to write Perl. Nor should anyone have to learn XS to
make Perl talk to C or C++ or even Python, for that matter.
So yes, you do need to continue, but not by naming languages and types.
What I was asking is why I should have to learn XS in order to make Perl
call chdir()?
> # Again, you're assuming external XS. If you internalize this
> # in Perl, it suddenly makes sense to have a C<System::FS>
> # which detects platform and calls the Perl module
> # C<System::FS::$Config{os}>
>
> Except that C<use> is compile-time and can't be given a string
..
use Config;
BEGIN { eval "use System::FS::%Config{os}" }
C'mon, that's not even a challenge. I do that sort of thing all the time
for getting at IO::Socket::INET in Perl 5, since it lives inside
IO::Socket in some versions and on its own in others. :)
> # > I consider this more generic--the C<inline> command is for any
> # > arbitrary code.
> #
> # In my other example, I put a C function in the PREP block, so
> # I don't think I ever had any intention of limiting its usability.
>
> But you can't intersperse your XSUBs with normal C code. (For, say,
> C<#define> nonsense.)
Nor should you. If you have something complex to do in C, do it in an
external C<.c> file (certainly enough existing XS modules for Perl 5 do
this). XS should be for getting access to simple functions and type
translations, not for writing complex C or C++ programs and libraries.
Granted, you could put the entire Linux kernel in a PREP block, so
you're not really very limited, but there's no reason to start letting
people throw C code all over their Perl modules.
> I think that, since there's no other possible meaning for C<int> in that
> context, it becomes a 'class object' of some sort. (Or maybe the
> C<returns> property would be prototyped to take a class name. *shrugs*)
property returns(class class $class) { ... }
?! Yikes. :-)
> I wasn't suggesting that you didn't need to give types. That class
> block was left incomplete because I was lazy.
Ah ok, then yes I agree.
[...]
> # Then you should be able to say:
> #
> # my a $foo = some_xs_function();
> # print $foo.c.a;
> #
> # Assuming auto-accessors, of course.
>
> I don't see how this is different from what I was talking about...
Cool, then. I thought you were being more limiting. If not, great! :)
> # > class Parrot::String is xs_type('Parrot_String' => $.ptr) {
> # > my Parrot::XS::Pointer $.ptr;
> # > ...
> #
> # I think that should be internal to Parrot.
>
> I think that the C<Parrot::Internals> module should be able to create
> C<Parrot::String>, so you can say
> C<$parrot_str.transcode($Parrot::Internals::CurInterp, 'utf8')> if you
> want. ;^)
I would never disagree with you there. More power == good. However, I
don't think that 99% of external interfaces should ever have to go into
the Parrot guts to get anything done.
> If you mean that I shouldn't be able to store a pointer in a Perl
> object, then there are a lot of C structures you'll have a hard time
> wrapping portably.
I agree, though I failed to note that as a type.... There are going to
be many opaque types that Perl will want access to, so:
sub gtk_file_selection_new(string $name) is external, returns(Gtk::Widget) { ...}
I would think that if you pass an object whose class does not define
Aaron Sherman
> On Thu, Sep 12, 2002 at 05:36:34PM -0400, Aaron Sherman wrote:
>
> > The following pseudo-classes are defined, which force a parrot-internal
> > storage and define the mechanism by which the value is translated to
> > external representation (these are all scalar types):
> >
> > int - signed integer value
> > real - floating point
> > string - A character string
> > utf8string - A string which is never converted to ASCII
> > bytes - A string which may contain binary data including nul
>
> I find this vague and open to misinterpretation as a way of binding to C.
It's supposed to be vague. Basically, it tells Perl everything it needs
to know to convert a basic Perl type to a C type (or C++ or Ruby, etc).
On the topic of uint, I don't think we should support it. The problem is
that since Perl ints are signed, a C unsigned int will overflow. You
need to put a C unsigned int (e.g. a uid_t) into a real.
Brent Dax
# There are some great reasons for there to be an IMC, but I
# should not have to learn IMC to write Perl. Nor should anyone
# have to learn XS to make Perl talk to C or C++ or even
# Python, for that matter.
I'm not suggesting that you have to learn XS--I'm suggesting that you
should learn C. :^) If you don't specify what format you want a string
to come in as, that's a huge ambiguity. And don't just assume the user
wants C<char *>--not only is it not always correct, but the conversion
is lossy.
# So yes, you do need to continue, but not by naming languages
# and types. What I was asking is why I should have to learn XS
# in order to make Perl call chdir()?
In the simple case, it's just:
module System::FS;
#Shiny new shorthand format
use Parrot::XS::Inline qq{
#include <chdir's_header.h>
};
sub chdir(string $dir) is external returns(int);
sub curdir() is inline('C': 'int':) returns(int) {
...
}
In the hypothetical (Parrot-level) C-, that might be:
namespace System::FS {
#use Parrot::XS::Inline ["#include <chdir's_header.h>"]
xsub('external') int chdir(str dir);
xsub('C', 'int', []) str curdir() {
...
}
}
I'd like the XS facility to be generic enough to handle that with little
knowledge of C-; it should only need to know how to map between C- and C
types, and have a façade compatible with C-. It should *not* have to be
rewritten from scratch.
# use Config;
# BEGIN { eval "use System::FS::%Config{os}" }
#
# C'mon, that's not even a challenge. I do that sort of thing
# all the time for getting at IO::Socket::INET in Perl 5, since
# it lives inside IO::Socket in some versions and on its own in
# others. :)
But you shouldn't have to resort to stupid tricks like that. If we
think that loading in platform-specific bits of XS in the way I
described will be common--or other cases where we load in different XS
based on run-time tests--we shouldn't require an C<eval>, since that'll
kill optimization instantly.
Remember, optimization-killers were okay for Perl 5, but now that we
have a Real VM(tm), we're trying to make things fast. :^)
# > But you can't intersperse your XSUBs with normal C code.
# (For, say,
# > C<#define> nonsense.)
#
# Nor should you. If you have something complex to do in C, do
....
# block, so you're not really very limited, but there's no
# reason to start letting people throw C code all over their
# Perl modules.
I would argue that there's no reason to *stop* people from throwing C
code all over their Perl modules. Last I checked, Perl isn't a
thought-control language. Whether we like it or not, there will be some
situations in which it's better to scatter C<#define>s
throughout--especially if we don't allow that to happen, if Murphy is to
be believed. :^)
# > I think that, since there's no other possible meaning for C<int> in
# > that context, it becomes a 'class object' of some sort.
# (Or maybe the
# > C<returns> property would be prototyped to take a class name.
# > *shrugs*)
#
# property returns(class class $class) { ... }
#
# ?! Yikes. :-)
property returns(class $class)
#Since it's already expecting a type in a parameter list
Although I'm not convinced that a 'property' keyword is better than a
class:
class returns is property {
method INIT(CODE $sub: class $class) {
$sub.prop{returns}=$class;
}
method operator:() (CODE $sub: ARRAY::ArgList $args) {
# ARRAY::ArgList magically preserves colons and such
return new $sub.prop{returns}($sub($args));
}
}
# > I think that the C<Parrot::Internals> module should be able
# to create
# > C<Parrot::String>, so you can say
# > C<$parrot_str.transcode($Parrot::Internals::CurInterp,
# 'utf8')> if you
# > want. ;^)
#
# I would never disagree with you there. More power == good.
# However, I don't think that 99% of external interfaces should
# ever have to go into the Parrot guts to get anything done.
But that 1% is going to be a killer.
# I would think that if you pass an object whose class does not define
It looks like you forgot to finish this sent
Nicholas Clark
> On Fri, 2002-09-13 at 17:36, Nicholas Clark wrote:
> > On Thu, Sep 12, 2002 at 05:36:34PM -0400, Aaron Sherman wrote:
> >
> > > The following pseudo-classes are defined, which force a parrot-internal
> > > storage and define the mechanism by which the value is translated to
> > > external representation (these are all scalar types):
> > >
> > > int - signed integer value
> > > real - floating point
> > > string - A character string
> > > utf8string - A string which is never converted to ASCII
> > > bytes - A string which may contain binary data including nul
> >
> > I find this vague and open to misinterpretation as a way of binding to C.
>
> It's supposed to be vague. Basically, it tells Perl everything it needs
> to know to convert a basic Perl type to a C type (or C++ or Ruby, etc).
without precision perl6 would be unable to warn the programmer of any
overflow on output.
> On the topic of uint, I don't think we should support it. The problem is
> that since Perl ints are signed, a C unsigned int will overflow. You
> need to put a C unsigned int (e.g. a uid_t) into a real.
If I understand you correctly, then "int" here means a perl INTVAL?
So it could well be a 64 bit integer. So declaring something as "real" on
the way out could cause an intermediate stage with a conversion to a NUMVAL?
Which could be a 64 bit double?
This sounds like a recipe to silently lose low order bits.
What's the odd byte between friends on a seek to file offset greater than
18014398509481984?
Dan Sugalski
>On Fri, 2002-09-13 at 04:26, Dan Sugalski wrote:
>> At 5:09 PM -0700 9/12/02, David Whipp wrote:
>
>> >Its good to start thinking about this early; but should we be thinking
>> >"Parrot's" XS; not "Perl 6's"?
>>
>> Yes, we should. Perl 6 XS is Apocalypse 21. That's likely rather a
>>ways off...
>
>We'll ignore my previous response to David's question for a second. Just
>taking on your answer, I'm perplexed. The Perl compiler is in rough
>shape, but it's moving along nicely. I've been able to at least parse
>all of the builtin functions I've written (though I have no idea what to
>do with them until the parameter-passing bugs are fixed by Sean or
>myself, whoever gets there first). Are we saying that we're going to
>wait for 16 more Apocalypses before we write code that allows C<chdir()>
>to call the C library function?!
I'm not sure what you're talking about here. You certainly don't need
any Perl XS stuff, if perl even has an XS system separate from
Parrot's, to do that. chdir is, or should be, a parrot function
independent of perl. At best you'd need the perl 6 compiler to
generate the appropriate interface code, but it's likely going to be
a single opcode in the file access opcode library for parrot.
>I think what we should be doing is agreeing generally on prototype
>features, implementing them, and then letting that serve as ground-work
>for the network notes from Larry.
I think that would be ill-advised. Larry has on many occasions showed
himself more than willing to toss out prototype solutions for what he
considers a more appropriate solution. It won't in the least surprise
me to find that he decides that the solution you might work out and
spent several months hashing out is flawed in some way and toss it.
If you want to work on Parrot's interface, that's fine. Do it on
perl6-internals, and then convince me you have the right answer. (Or
right enough answer, at least) I don't think you'll find it
particularly productive to work on perl 6's interface, though.
Dan Sugalski
>On Fri, 2002-09-13 at 17:36, Nicholas Clark wrote:
>> On Thu, Sep 12, 2002 at 05:36:34PM -0400, Aaron Sherman wrote:
>>
>> > The following pseudo-classes are defined, which force a parrot-internal
>> > storage and define the mechanism by which the value is translated to
>> > external representation (these are all scalar types):
>> >
>> > int - signed integer value
>> > real - floating point
>> > string - A character string
>> > utf8string - A string which is never converted to ASCII
>> > bytes - A string which may contain binary data including nul
>>
>> I find this vague and open to misinterpretation as a way of binding to C.
>
>It's supposed to be vague. Basically, it tells Perl everything it needs
>to know to convert a basic Perl type to a C type (or C++ or Ruby, etc).
Vague is bad for interfaces. Exact sizes are the way to go, specified
as exactly as the underlying C lets us.
>On the topic of uint, I don't think we should support it. The problem is
>that since Perl ints are signed, a C unsigned int will overflow.
Says who, exactly? Are you *sure* that perl won't have unsigned ints?
Really sure?
> You
>need to put a C unsigned int (e.g. a uid_t) into a real.
Not necessarily. A bigint would work fine. As would an unsigned int.
Nicholas Clark
> Aaron Sherman:
> # There are some great reasons for there to be an IMC, but I
> # should not have to learn IMC to write Perl. Nor should anyone
> # have to learn XS to make Perl talk to C or C++ or even
> # Python, for that matter.
>
> I'm not suggesting that you have to learn XS--I'm suggesting that you
> should learn C. :^) If you don't specify what format you want a string
> to come in as, that's a huge ambiguity. And don't just assume the user
> wants C<char *>--not only is it not always correct, but the conversion
> is lossy.
> In the hypothetical (Parrot-level) C-, that might be:
>
> namespace System::FS {
> #use Parrot::XS::Inline ["#include <chdir's_header.h>"]
>
> xsub('external') int chdir(str dir);
>
> xsub('C', 'int', []) str curdir() {
> ...
> }
> }
>
> I'd like the XS facility to be generic enough to handle that with little
> knowledge of C-; it should only need to know how to map between C- and C
> types, and have a façade compatible with C-. It should *not* have to be
> rewritten from scratch.
But this all sounds like progress backwards. I can *already* do all this
in perl5. In 1 line. In real C:
$ perl5.00503 -MCwd -lwe 'use Inline C => "void my_chdir (char *dir) {chdir (dir);}"; print "Currently ", cwd; my_chdir ".."; print "Now ", cwd'
Currently /stuff/blead
Now /stuff
And that was 5.005_03. Not 5.8. And it's a real perl not patched in any way.
Inline::C is just a standard CPAN install.
Sorry if this sounds arrogant, but everyone arguing in this thread who has
not experimented with Inline::C should hold fire, go away now and
perl -MCPAN -e 'install Inline'
play with it a bit, and only carry on discussing what an interfacing system
should do when they've seen what Inline::C has already achieved.
Fine, no-one is saying that how Inline::C is actually implemented is perfect,
how it reports errors when you mess up could never be improved, or how it
handles some of the C functions you want to embed is always correct. But the
style of interface it gives you for embedding C -
just writing it in C
- is about as clean you're conceivably going to get.
Nicholas Clark
Brent Dax
# But this all sounds like progress backwards. I can *already*
# do all this in perl5. In 1 line. In real C:
#
# $ perl5.00503 -MCwd -lwe 'use Inline C => "void my_chdir
# (char *dir) {chdir (dir);}"; print "Currently ", cwd;
# my_chdir ".."; print "Now ", cwd'
I'm aware of Inline.pm, and I think it's a model worth emulating.
However, I'm not sure if it'll mesh perfectly with some of Perl 6's new
features. How do we know whether to prototype the generated Perl stub?
How do we know if we should type those parameters? How do we know if we
want a unary star on something? How does it interact with ':' and such?
Tim Bunce
> Nicholas Clark:
> # But this all sounds like progress backwards. I can *already*
> # do all this in perl5. In 1 line. In real C:
> #
> # $ perl5.00503 -MCwd -lwe 'use Inline C => "void my_chdir
> # (char *dir) {chdir (dir);}"; print "Currently ", cwd;
> # my_chdir ".."; print "Now ", cwd'
>
> I'm aware of Inline.pm, and I think it's a model worth emulating.
> However, I'm not sure if it'll mesh perfectly with some of Perl 6's new
> features. How do we know whether to prototype the generated Perl stub?
> How do we know if we should type those parameters? How do we know if we
> want a unary star on something? How does it interact with ':' and such?
Assuming we go that way then perhaps reasonable defaults could be
provided and where those don't match what's needed then the application
can use <<a "forward" declaration in Perl of a C routine can supply
all the glue information formerly supplied by XS>> [to paraphrase Larry].
But remember that it should be possible to use those forward
declarations in Perl to talk to any *existing* shared library
(or java, C#, .net thingy etc) and support the appropriate calling
conventions. Ideally without the declaration having to contain any
procedural interface logic, it should just be declarative.
So for a function that takes a "char **" parameter the forward
declaration needs to be be able to express how that's actually
used, including its use for input and output, and any memory
management issues (may need to free the pointed-to memory).
Now assuming you accept that, then it's only a small step to allow
the forward declarations to also specify some additional/alternate
glue code.
Seems to me that we should be thinking about the forward declaration
syntax and semantics for using exisiting libraries at this stage.
I suspect that it'll then become clear how to add extra code in
a simple and natural way.
Tim.
Tim Bunce
>
> that since Perl ints are signed
Perl5 supports unsigned ints to some extent: SvUV(sv)
Tim.
Aaron Sherman
> >On the topic of uint, I don't think we should support it. The problem is
> >that since Perl ints are signed, a C unsigned int will overflow.
>
> Says who, exactly? Are you *sure* that perl won't have unsigned ints?
> Really sure?
If it does, it does. I'm working with what I've seen discussed and
what's in the current prototype.
Aaron Sherman
> I'm not sure what you're talking about here. You certainly don't need
> any Perl XS stuff, if perl even has an XS system separate from
> Parrot's, to do that. chdir is, or should be, a parrot function
Ok, this has gone on too long, and since I think that fundamentally,
it's not a disagreement so much as a difference in definitions, I'm
going to dictatorially define a bunch of them :)
XP -- Parrot's external interface. Can access C, C++, etc *or* other
Parrot-based languages.
XN -- Native external interface used from a Parrot-based language to
gain access to an XP. An XN might auto-generate its own XP or use
one that already exists.
XG -- Interface glue code written in the same language as the target
library or function (possibly in a pre-processable form).
Now, in Perl 5, XP+XN+XG made up XS. In Perl 6, it no longer makes sense
to speak of all of these as one whole.
For example, Parrot may provide an XP/XG package for POSIX, but that
doesn't mean that any Parrot-based languages *have* to use it.
Also, Perl 6 may have a CPAN module that provides the XN for calling NFS
RPCs, but the XP portion (which would presumably be auto-generated)
would not have an analog in Python.
For large subsystems that many languages will have to access, I imagine
that there would normally be a separate module that contains the XP/XG.
And then the XN would depend on its existence. For example, Parrot will
probably have a Gtk+ module at some point so that all Parrot-based
languages can use Gtk+ and glib. On the other hand, I might write a
bunch of C code to speed up my Perl module, and I don't see why Perl
should force me to program in Parrot or some other external syntax just
to gain access to it.
Here's the fun part: if the XG and XP are auto-generated by the
compiler, then someone who writes their module for Perl can decide later
that they want to break it out as a standalone module for Parrot and let
other languages use it. Or that same author could simply say: you can
write your XN for this from Python or Scheme, but you'll have to
download, compile and install my Perl module to do so.
Either way works, and the author of the Perl module never had to know
how Parrot types work, or what the calling conventions are in
Python/Parrot.
So, going back to my proposal, I would like to think about Perl 6 XS in
terms of the XN, and leave the XP and XG for later discussion. If the XP
and XG can be wholly generated form the XN we are free to change the
former two at any time.
Aaron Sherman
> On Sat, Sep 14, 2002 at 10:15:11AM -0700, Brent Dax wrote:
> > Nicholas Clark:
> > # But this all sounds like progress backwards. I can *already*
> > # do all this in perl5. In 1 line. In real C:
> provided and where those don't match what's needed then the application
> can use <<a "forward" declaration in Perl of a C routine can supply
> all the glue information formerly supplied by XS>> [to paraphrase Larry].
Putting C or C++ or Python or Parrot or Scheme, etc code into Perl 6 in
order to describe an external interface seems very klunky to me. Call me
silly, but I think I should be able to tell Perl how to access external
resources in Perl. Perl should be able to determine how to glue my Perl
code to whatever external language I'm targeting. When the interface is
too complicated for the defaults to work out, THEN I should have to step
in to clarify.
Aaron Sherman
If Perl 6 will have an unsigned integer type (either standalone or as a
property of the pseudo-class int), then you can ignore my comments on
the topic, and all will be right in the world :)
Tim Bunce
> On Sat, 2002-09-14 at 17:33, Tim Bunce wrote:
> > On Sat, Sep 14, 2002 at 10:15:11AM -0700, Brent Dax wrote:
> > > Nicholas Clark:
> > > # But this all sounds like progress backwards. I can *already*
> > > # do all this in perl5. In 1 line. In real C:
> [...]
> > Assuming we go that way then perhaps reasonable defaults could be
> > provided and where those don't match what's needed then the application
> > can use <<a "forward" declaration in Perl of a C routine can supply
> > all the glue information formerly supplied by XS>> [to paraphrase Larry].
>
> Putting C or C++ or Python or Parrot or Scheme, etc code into Perl 6 in
> order to describe an external interface seems very klunky to me. Call me
> silly, but I think I should be able to tell Perl how to access external
> resources in Perl.
Me too. That wasn't what I was suggesting. Larry seems to be saying that
an external interface can be described in sufficient detail by extending
the "sub foo;" forward declaration. No code, just declarative.
Perhaps something, at a minimum, like:
sub chdir(str $dir) is external("C","libc");
and/or maybe:
package mylibc is external("C","libc") {
sub int chdir(str $dir);
sub int getpid();
sub str perror(int $errno);
sub str fgets(str $buffer is length($size), $size, $fh is external_type("FILE"));
}
> Perl should be able to determine how to glue my Perl
> code to whatever external language I'm targeting. When the interface is
> too complicated for the defaults to work out, THEN I should have to step
> in to clarify.
I agree.
Just to (maybe) clarify, the paragraph you quoted was a reply to
the previous post, but the rest of my email was trying to make a
different point: That we should work out how to talk to exiting
code before we worry too much about glue code.
But the whole area is strewn with confusion. Hopefully your XP/XN/XG
split will help.
Tim.