I have patched Storable-2.09 to use
PERL_NO_GET_CONTEXT (w/pTHX_/aTHX_).
Test results are good. I am using the
patched module w/o problems on my Apache
2.0.48/modperl 1.9913 server using
perl 5.8.3 in Linux 2.4.25 (Sorcerer).
Would you like me to send the patch to you?
Aloha => Beau;
Of course, Beau. But there were several changes since 2.09 (I don't know why
2.10 wasn't released yet). You need to checkout blead-perl and submit the
patch against ext/Storable/
Here is how to get blead perl:
rsync -acvz --delete --force rsync://ftp.linux.activestate.com/perl-current/
./perl-current
cd perl-current/ext/Storable/
# this is the latest Storable...
Thanks.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:st...@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Stas:
I have the ithreads/multiplicity patch for Storeable based
on the current blead at:
ftp://ftp.beaucox.com/pub/XS/Storable/Storable-blead-multiplicity-patch
with a README in that same location.
All tests OK for perl 5.8.x, but...
The patched storable DOES NOT WORK for 5.6.1, so I need some
coaching:
Storable uses the c function 'qsort' to sort an array;
but 'qsort' calls back a sort compare routine passing
two elements. There is no way I can figure out how to
get the context, aTHX_, passed to the sort compare with
the standard 'qsort' and I should think trying to fool
with a global would prove to be very unsafe.
Well, I found a perl built-in, 'sortsv', and it works
great in 5.8.x. I was feeling pretty good, until I tested
with 5.6.1 and, guess what, 'sortsv' is not implemented
in 5.6.1. I tried to tie in to 5.6.1's 'qsortsv', but
could not get that working.
Can you think of anything I can do to implement Storable
for 5.6.1?
Aloha => Beau;
First of all, I think Storable has to support older perls as well. Probably at
least 5.005_03. But since there are no ithreads before 5.6, it's not an issue
(just something to test).
Backporting sortsv is probably going to be too much pain. Try to do the thing
that xs was doing w/o PERL_NO_GET_CONTEXT defined, i.e.:
static int
sortcmp(const void *a, const void *b)
{
dTHX;
return sv_cmp(*(SV * const *) a, *(SV * const *) b);
}
...
static int store_hash(stcxt_t *cxt, HV *hv)
{
...
{
#ifdef USE_ITHREADS
PerlInterpreter *orig_perl = PERL_GET_CONTEXT;
PERL_SET_CONTEXT(aTHX);
#end
qsort((char *) AvARRAY(av), len, sizeof(SV *), sortcmp);
#ifdef USE_ITHREADS
PERL_SET_CONTEXT(orig_perl);
#end
}
So we have no choice but to use dTHX in sortcmp, but we restore the previous
context whatever it was. This is untested, but you get the idea.
The code above has the same effect as when passing aTHX directly, though it's
slower of course.
You may want to convert this extra code into macros to make the code more
readable:
{
OVERRIDE_PERL_CONTEXT;
qsort((char *) AvARRAY(av), len, sizeof(SV *), sortcmp);
RESTORE_PERL_CONTEXT;
Should it use SAVE* functions (scope.c) to make that secure against
the sort code croaking and leaving the wrong context in effect?
Tim.
Perl_sv_cmp (called from sortcmp) doesn't seem to croak on its own, but could
croak via internal calls. So I guess you are right, Tim. Should it be one of
SAVESPTR(s) or SAVEPPTR(p) (perlguts.pod)?
ftp://ftp.beaucox.com/pub/XS/Storable/Storable-blead-multiplicity-patch
with a README in that same location.
All tests OK for perl 5.6.1 - 5.8.3.
Thank you (and Tim) for the qsort help. I have not implemented
the SAVExPTR(x) yet. I need some help with that one.
The qsort was handled with a macro:
/*
* sort (used in store_hash) - conditionally use qsort when
* sortsv is not available ( <= 5.6.1 ).
*/
#if (PATCHLEVEL <= 6)
#if defined(MULTIPLICITY)
#define STORE_HASH_SORT \
PerlInterpreter *orig_perl = PERL_GET_CONTEXT; \
PERL_SET_CONTEXT(aTHX); \
qsort((char *) AvARRAY(av), len, sizeof(SV *), sortcmp); \
PERL_SET_CONTEXT(orig_perl);
#else /* ! MULTIPLICITY */
#define STORE_HASH_SORT \
qsort((char *) AvARRAY(av), len, sizeof(SV *), sortcmp);
#endif /* MULTIPLICITY */
#else /* PATCHLEVEL > 6 */
#define STORE_HASH_SORT \
sortsv(AvARRAY(av), len, Perl_sv_cmp_locale);
#endif /* PATCHLEVEL <= 6 */
You see I use the faster sortsv for > 5.6.
The sortcmp routine is conditionally compiled:
#if (PATCHLEVEL <= 6)
/*
* sortcmp
*
* Sort two SVs
* Borrowed from perl source file pp_ctl.c, where it is used by pp_sort.
*/
static int
sortcmp(const void *a, const void *b)
{
#if defined(MULTIPLICITY)
dTHX;
#endif /* MULTIPLICITY */
return sv_cmp(*(SV * const *) a, *(SV * const *) b);
}
#endif /* PATCHLEVEL <= 6 */
And finally, the sort is called via:
STORE_HASH_SORT;
Aloha => Beau;
Whichever description fits best:
=item C<SAVESPTR(s)>
=item C<SAVEPPTR(p)>
These macros arrange things to restore the value of pointers C<s> and
C<p>. C<s> must be a pointer of a type which survives conversion to
C<SV*> and back, C<p> should be able to survive conversion to C<char*>
and back.
I dunno. The docs should explain how to choose. I guess alignment
is the issue here.
You'll want ENTER/LEAVE macros as well.
Tim.
p.s. The valuable service offered by the many SAVE* scope macros are not
mentioned at all in perlxs*.pod. ENTER/LEAVE are only mentioned in passing.
Tim -
I tried the following:
#define STORE_HASH_SORT \
ENTER; \
PerlInterpreter *orig_perl = PERL_GET_CONTEXT; \
SAVESPTR(aTHX); \
PERL_SET_CONTEXT(aTHX); \
qsort((char *) AvARRAY(av), len, sizeof(SV *), sortcmp); \
PERL_SET_CONTEXT(orig_perl); \
LEAVE;
and use this macro for the sort. Got the following warning:
Storable.xs: In function `store_hash':
Storable.xs:2230: warning: address of register variable `my_perl' requested
This warning also was given when SAVEPPTR() was used.
Tests all OK in 5.6.1 (but, of course, nothing croaked).
Am I saving the right thing (aTHX)? Would one of the
integer saves work? Do I really need this? What is the
meaning of life?
Aloha => Beau;
ps: I hope I'm not being a PITA.
How do we know whether PerlInterpreter * survives SV* or char* conversions? (I
suppose it talks about casting into those types and then recasting them back.
Are there any examples in the perl source we can copy from?
>>You'll want ENTER/LEAVE macros as well.
Yup.
> I tried the following:
>
> #define STORE_HASH_SORT \
> ENTER; \
> PerlInterpreter *orig_perl = PERL_GET_CONTEXT; \
> SAVESPTR(aTHX); \
No, you want to save orig_perl, not my_perl (aTHX==my_perl)
> PERL_SET_CONTEXT(aTHX); \
> qsort((char *) AvARRAY(av), len, sizeof(SV *), sortcmp); \
> PERL_SET_CONTEXT(orig_perl); \
> LEAVE;
and you want to restore *after* LEAVE. Or actually you probably don't need to
restore it, since LEAVE will restore it for you. In perl that will look like:
{ # ENTER
local *ctx = current_ctx; # PerlInterpreter *orig_perl = PERL_GET_CONTEXT;
# SAVESPTR(orig_perl);
# PERL_SET_CONTEXT(aTHX);
do_something; # qsort((char *...
} # LEAVE;
why MULTIPLICITY? You probably need to use USE_ITHREADS, no?
I couldn't find any perl source actually saving PerlInterpreter *
using SAVExPTR(); but lots of 'similar' objects seem to be savable.
Given that, I am using SAVESPTR(). I will try to construct a simple
test to prove it is valid.
>
> >>You'll want ENTER/LEAVE macros as well.
>
> Yup.
>
> > I tried the following:
> >
> > #define STORE_HASH_SORT \
> > ENTER; \
> > PerlInterpreter *orig_perl = PERL_GET_CONTEXT; \
> > SAVESPTR(aTHX); \
>
> No, you want to save orig_perl, not my_perl (aTHX==my_perl)
>
> > PERL_SET_CONTEXT(aTHX); \
> > qsort((char *) AvARRAY(av), len, sizeof(SV *), sortcmp); \
> > PERL_SET_CONTEXT(orig_perl); \
> > LEAVE;
>
> and you want to restore *after* LEAVE. Or actually you probably don't need
> to restore it, since LEAVE will restore it for you. In perl that will look
> like:
>
> { # ENTER
> local *ctx = current_ctx; # PerlInterpreter *orig_perl =
> PERL_GET_CONTEXT; # SAVESPTR(orig_perl);
> # PERL_SET_CONTEXT(aTHX);
> do_something; # qsort((char *...
> } # LEAVE;
OK. Here is my latest attempt:
#if (PATCHLEVEL <= 6)
#if defined(USE_ITHREADS)
#define STORE_HASH_SORT \
ENTER; \
PerlInterpreter *orig_perl = PERL_GET_CONTEXT; \
SAVESPTR(orig_perl); \
PERL_SET_CONTEXT(aTHX); \
qsort((char *) AvARRAY(av), len, sizeof(SV *), sortcmp); \
LEAVE;
#else /* ! USE_ITHREADS */
#define STORE_HASH_SORT \
qsort((char *) AvARRAY(av), len, sizeof(SV *), sortcmp);
#endif /* USE_ITHREADS */
#else /* PATCHLEVEL > 6 */
#define STORE_HASH_SORT \
sortsv(AvARRAY(av), len, Perl_sv_cmp_locale);
#endif /* PATCHLEVEL <= 6 */
All test successful 5.6.1 - 5.8.3. The patch is at
ftp://ftp.beaucox.com/pub/XS/Storable/Storable-blead-multiplicity-patch
Please let me know if I should do anything else.
Oh, by the way, I changed MULTIPLICITY to USE_ITHREADS, but I
remember reading somewhere - I can't find the reference now -
that you can't have one without the other.
Aloha => Beau;
PS: I now hold the world's speed record for typing aTHX_ and
pTHX_ :)
Looks like you use tabs. Please avoid using them (even if the original source
has used them), they are evil at least when you need to communicate patches
over email. Use spaces instead. (I wish perl source was to be cleaned of tabs
and was properly aligned.)
> #else /* ! USE_ITHREADS */
>
> #define STORE_HASH_SORT \
> qsort((char *) AvARRAY(av), len, sizeof(SV *), sortcmp);
>
> #endif /* USE_ITHREADS */
>
> #else /* PATCHLEVEL > 6 */
>
> #define STORE_HASH_SORT \
> sortsv(AvARRAY(av), len, Perl_sv_cmp_locale);
>
> #endif /* PATCHLEVEL <= 6 */
Looks cool! Is sortsv equivalent to what qsort did before?
> All test successful 5.6.1 - 5.8.3.
I'd test at least with 5.005_03 and 5.6.0. Also you need to test with threaded
and unthreaded versions. I have the following perl builds:
5.005_03/ 5.8.1-nouseperlio/
5.005_04/ 5.8.2/
5.6.0/ 5.8.2-ithread/
5.6.0-ithread/ 5.8.2-ithread-nouseshrplib/
5.6.1/ 5.8.2-nouseperlio/
5.6.1-ithread/ 5.8.3/
5.6.2/ 5.8.3-ithread/
5.6.2-ithread/ 5.8.3-ithread-nouseshrplib/
5.6.2-perlio/ 5.8.3-ithread-pool/
5.8.0/ 5.8.3-nouseperlio/
5.8.0-ithread/ blead/
5.8.1/ blead-ithread/
5.8.1-ithread/ blead-multi/
5.8.1-ithread-nouseshrplib/ makelinks
5.8.1-nodebug/
I also have a script that accepts a list of commands and smokes them for me.
Let me know if you are interested to try it. I use it to smoke my modules
against all the above perls.
> The patch is at
>
> ftp://ftp.beaucox.com/pub/XS/Storable/Storable-blead-multiplicity-patch
>
> Please let me know if I should do anything else.
>
> Oh, by the way, I changed MULTIPLICITY to USE_ITHREADS, but I
> remember reading somewhere - I can't find the reference now -
> that you can't have one without the other.
I guess so, (i.e. I haven't verified that), but I'm biased as I use
USE_ITHREADS everywhere ;)
> PS: I now hold the world's speed record for typing aTHX_ and
> pTHX_ :)
Thank you Beau for working on fixing all these modules. I'm sure that there
quite a few others that may need your expertise. ;)
OK - will de-tab, I use emacs; I thought there was a option to
fill tabs with spaces, but I can't find it. Do you happen to
know what it is (if it exists)?
[...]
> >
> > #define STORE_HASH_SORT \
> > sortsv(AvARRAY(av), len, Perl_sv_cmp_locale);
> >
> > #endif /* PATCHLEVEL <= 6 */
>
> Looks cool! Is sortsv equivalent to what qsort did before?
Er, I think so - I will check the 'locale' cmp again.
>
> > All test successful 5.6.1 - 5.8.3.
>
> I'd test at least with 5.005_03 and 5.6.0. Also you need to test with
> threaded and unthreaded versions. I have the following perl builds:
>
[...]
Building additional perls now.
>
> I also have a script that accepts a list of commands and smokes them for
> me. Let me know if you are interested to try it. I use it to smoke my
> modules against all the above perls.
Sure, looks like fun.
[...]
> Thank you Beau for working on fixing all these modules. I'm sure that there
> quite a few others that may need your expertise. ;)
>
Happy to help. I will be happy to convert more modules as
time permits. You folks just let me know what's next.
Aloha => Beau;
Can't find 5.005_04 on perl.org or CPAN. Do you know where
it is hiding?
Aloha => Beau;
PS: looks like yout table above has TABS! ;)
Didn't you know that anything that you ever needed can be found at
perl.apache.org? ;)
http://perl.apache.org/docs/2.0/devel/core/coding_style.html#Coding_Style_Guide
'(indent-tabs-mode nil)
>>I also have a script that accepts a list of commands and smokes them for
>>me. Let me know if you are interested to try it. I use it to smoke my
>>modules against all the above perls.
>
>
> Sure, looks like fun.
I'll send it to you off list.
>>Thank you Beau for working on fixing all these modules. I'm sure that there
>>quite a few others that may need your expertise. ;)
> Happy to help. I will be happy to convert more modules as
> time permits. You folks just let me know what's next.
Any popular CPAN module that has XS code and you can't grep for
PERL_NO_GET_CONTEXT. I'd start with the modules that live in the core (like
Storable that you have handled already).
There was a talk of making PERL_NO_GET_CONTEXT enabled by default by 5.10 or
5.12, so converting things early will help ease the transition. Not talking
about the speedup.
One more nit. This:
>> I tried the following:
>>
>> #define STORE_HASH_SORT \
>> ENTER; \
>> PerlInterpreter *orig_perl = PERL_GET_CONTEXT; \
won't compile on most compilers besides recent gcc's, as you have declarations
after code. So it needs to be:
ENTER;
{
PerlInterpreter *orig_perl = PERL_GET_CONTEXT;
...
it's not there yet. There is only RC2
http://use.perl.org/articles/04/02/20/1536210.shtml
But testing with 5.005_03 is good enough.
> PS: looks like yout table above has TABS! ;)
heh, that's the output of 'ls'
As I mentioned before PERL_NO_GET_CONTEXT isn't documented in perlxs
or perlxstut at all. It ought to be.
Tim.
> p.s. The valuable service offered by the many SAVE* scope macros are not
> mentioned at all in perlxs*.pod. ENTER/LEAVE are only mentioned in passing.
On Fri, Feb 27, 2004 at 01:23:26PM +0000, Tim Bunce wrote:
> As I mentioned before PERL_NO_GET_CONTEXT isn't documented in perlxs
> or perlxstut at all. It ought to be.
For that specific macro, I don't remember where, and it wasn't in this thread.
Patches welcome from anyone. [I realise that Tim is always overbusy already.
Pesky DBI being popular and wanting TLC :-)]
Nicholas Clark
It's documented in perlguts, along with a couple of examples on how
to use it. perlguts is mentioned right at the beginning of perlxstut
and also a couple of times in perlxs.
Marcus
--
It's easier to fix a broken spec than 10,000 broken programs.
--Larry Wall
perlguts is mentioned, but PERL_NO_GET_CONTEXT isn't.
Many extension writers will only dig into perlguts to find out how
to do something they *know* they want to do. People just don't know
that they should use PERL_NO_GET_CONTEXT because no one has told them.
Tim.
What about adding another example to perlxstut?
----8<-----------------------------------------------------------------
=head2 EXAMPLE 10 - Speeding up your XS module
One of the reasons for writing an XS module is speed. If you care about
speed and your module uses lots of Perl API calls, you can probably give
your module an additional performance boost by using C<PERL_NO_GET_CONTEXT>.
It won't help if you're using a non-threaded perl binary, but it will help
everyone who's using your module with a threaded perl, even if they're not
using threads at all.
Here's some XS code that returns a reference to an array that holds the
first C<n> Fibonacci numbers:
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
static SV *S_fibonacci(IV n)
{
AV *a = newAV();
IV i, fib0 = 0, fib1 = 1, fib2 = 0;
for (i = 0; i <= n; i++)
{
av_push(a, newSViv(fib0));
fib2 = fib1;
fib1 = fib0;
fib0 = fib1 + fib2;
}
return newRV_noinc((SV*)a);
}
MODULE = Fibonacci PACKAGE = Fibonacci
SV *
fibonacci(IV n)
CODE:
RETVAL = S_fibonacci(n);
OUTPUT:
RETVAL
Using the C<Benchmark> module, one can figure out that a non-threaded perl
binary can run C<fibonacci(42)> about 75188 times per second. A threaded
perl binary will only run it about 50251 times per second, which is a
significant difference.
Some of that difference is caused by the threads overhead in the perl
binary itself. However, a reasonable amount of time is spent in the XS
module determining the interpreter context each time an API function is
called. So the interpreter context is determined several times, even
though it never changes.
The solution to speed up your XS module is to explicitly pass the context
to all functions that use API calls, and to make perl aware of the fact
that it doesn't have to get the context for each API call. This is done by
defining C<PERL_NO_GET_CONTEXT> before including F<XSUB.h> and passing the
context using the C<aTHX> (argument) and C<pTHX> (prototype) macros. The
source looks only slightly different:
#define PERL_NO_GET_CONTEXT
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
static SV *S_fibonacci(pTHX_ IV n)
{
AV *a = newAV();
IV i, fib0 = 0, fib1 = 1, fib2 = 0;
for (i = 0; i <= n; i++)
{
av_push(a, newSViv(fib0));
fib2 = fib1;
fib1 = fib0;
fib0 = fib1 + fib2;
}
return newRV_noinc((SV*)a);
}
MODULE = Fibonacci PACKAGE = Fibonacci
SV *
fibonacci(IV n)
CODE:
RETVAL = S_fibonacci(aTHX_ n);
OUTPUT:
RETVAL
This code will run at exactly the same speed as our first version on a
non-threaded perl. However, on a threaded perl it will run more than
20 percent faster (executing C<fibonacci(42)> about 60976 times per
second).
You most probably wondered about the underscores in C<pTHX_> and C<aTHX_>.
These underscores represent commas. Both C<pTHX> and C<aTHX> may actually
be defined as an empty string when building the extension with a
non-threaded perl. If you had written
static SV *S_fibonacci(pTHX, IV n)
this would be expanded by the C preprocessor as
static SV *S_fibonacci(, IV n)
which would be a syntax error. If you use
static SV *S_fibonacci(pTHX_ IV n)
instead, it will be correct for threaded and non-threaded perl binaries.
For more information on C<PERL_NO_GET_CONTEXT>, see L<perlguts>.
----8<-----------------------------------------------------------------
I'll apply it if nobody has objections.
Marcus
--
party, n.:
A gathering where you meet people who drink
so much you can't even remember their names.
We need to decide if we want all extensions to use PERL_NO_GET_CONTEXT.
I think we probably do. If so *all* the examples should be rewritten to use it.
Tim.
Do you think that we should re-write all the examples without mentioning what
it means until later on through the tutorial?
Or that we should work the explanation of PERL_NO_GET_CONTEXT into the first
example that uses a perl API function, so that it is explained at first use?
While I would agree that encouraging the use of PERL_NO_GET_CONTEXT is good,
it does complicate the code, which isn't going to help people to learn XS
from this tutorial document.
Nicholas Clark
My progress on PERL_NO_GET_CONTEXT for Storable:
1) TABS in my patches removed.
2) ENTER; { ... } LEAVE; brackets added.
3) Lots of perls tested, results so far:
Sat Feb 28 17:02:20 HST 2004
/home/test/src/XS/Storable/Storable-blead-no_get_context
------------------------------------------------------------------------
5.005_03 - OK
All tests successful, 8 tests and 1 subtest skipped.
Files=25, Tests=1527, 1 wallclock secs ( 0.86 cusr + 0.08 csys = 0.94 CPU)
------------------------------------------------------------------------
5.005_04 - OK
All tests successful, 7 tests and 1 subtest skipped.
Files=25, Tests=1535, 1 wallclock secs ( 0.84 cusr + 0.10 csys = 0.94 CPU)
------------------------------------------------------------------------
5.6.0 - OK
All tests successful, 6 tests skipped.
Files=25, Tests=1530, 1 wallclock secs ( 1.10 cusr + 0.13 csys = 1.23 CPU)
------------------------------------------------------------------------
5.6.0-ithread - OK
All tests successful, 6 tests skipped.
Files=25, Tests=1530, 2 wallclock secs ( 1.64 cusr + 0.17 csys = 1.81 CPU)
------------------------------------------------------------------------
5.6.1 - OK
All tests successful, 6 tests skipped.
Files=25, Tests=1530, 1 wallclock secs ( 1.04 cusr + 0.17 csys = 1.21 CPU)
------------------------------------------------------------------------
5.6.1-ithread - OK
All tests successful, 6 tests skipped.
Files=25, Tests=1530, 2 wallclock secs ( 1.30 cusr + 0.19 csys = 1.49 CPU)
------------------------------------------------------------------------
5.6.2 - OK
All tests successful, 6 tests skipped.
Files=25, Tests=1530, 1 wallclock secs ( 1.07 cusr + 0.10 csys = 1.17 CPU)
------------------------------------------------------------------------
5.6.2-ithread - OK
All tests successful, 6 tests skipped.
Files=25, Tests=1530, 2 wallclock secs ( 1.21 cusr + 0.14 csys = 1.35 CPU)
------------------------------------------------------------------------
5.6.2-perlio - OK
All tests successful, 6 tests skipped.
Files=25, Tests=1530, 1 wallclock secs ( 1.06 cusr + 0.12 csys = 1.18 CPU)
------------------------------------------------------------------------
5.8.0 - FAILED exit code = 2
DIED. FAILED tests 9-50
Failed 42/50 tests, 16.00% okay
t/retrieve............ok
t/store...............ok
t/threads.............skipped
all skipped: no threads
t/tied................ok
t/tied_hook...........ok
t/tied_items..........ok
t/utf8................ok
t/utf8hash............ok
Failed 1/25 test scripts, 96.00% okay. 42/1947 subtests failed, 97.84% okay.
Failed Test Stat Wstat Total Fail Failed List of Failed
-------------------------------------------------------------------------------
t/restrict.t 2 512 50 42 84.00% 9-50
2 tests skipped.
make: *** [test_dynamic] Error 29
------------------------------------------------------------------------
5.8.0-ithread - FAILED exit code = 2
DIED. FAILED tests 9-50
Failed 42/50 tests, 16.00% okay
t/retrieve............ok
t/store...............ok
t/threads.............ok
t/tied................ok
t/tied_hook...........ok
t/tied_items..........ok
t/utf8................ok
t/utf8hash............ok
Failed 1/25 test scripts, 96.00% okay. 42/1949 subtests failed, 97.85% okay.
Failed Test Stat Wstat Total Fail Failed List of Failed
-------------------------------------------------------------------------------
t/restrict.t 2 512 50 42 84.00% 9-50
1 test skipped.
make: *** [test_dynamic] Error 29
------------------------------------------------------------------------
5.8.1 - OK
All tests successful, 2 tests skipped.
Files=25, Tests=1947, 3 wallclock secs ( 2.13 cusr + 0.17 csys = 2.30 CPU)
------------------------------------------------------------------------
5.8.1-ithread - OK
All tests successful, 1 test skipped.
Files=25, Tests=1949, 4 wallclock secs ( 2.89 cusr + 0.19 csys = 3.08 CPU)
------------------------------------------------------------------------
5.8.1-ithread-nouseshrplib - OK
All tests successful, 1 test skipped.
Files=25, Tests=1949, 3 wallclock secs ( 2.41 cusr + 0.23 csys = 2.64 CPU)
------------------------------------------------------------------------
5.8.1-nouseperlio - OK
All tests successful, 2 tests skipped.
Files=25, Tests=1947, 3 wallclock secs ( 2.18 cusr + 0.23 csys = 2.41 CPU)
------------------------------------------------------------------------
5.8.2 - OK
All tests successful, 2 tests skipped.
Files=25, Tests=1947, 2 wallclock secs ( 2.14 cusr + 0.16 csys = 2.30 CPU)
------------------------------------------------------------------------
5.8.2-ithread - OK
All tests successful, 1 test skipped.
Files=25, Tests=1949, 3 wallclock secs ( 2.85 cusr + 0.22 csys = 3.07 CPU)
------------------------------------------------------------------------
5.8.2-ithread-nouseshrplib - OK
All tests successful, 1 test skipped.
Files=25, Tests=1949, 3 wallclock secs ( 2.44 cusr + 0.19 csys = 2.63 CPU)
------------------------------------------------------------------------
5.8.2-nouseperlio - OK
All tests successful, 2 tests skipped.
Files=25, Tests=1947, 2 wallclock secs ( 2.05 cusr + 0.27 csys = 2.32 CPU)
------------------------------------------------------------------------
5.8.3 - OK
All tests successful, 2 tests skipped.
Files=25, Tests=1947, 3 wallclock secs ( 2.12 cusr + 0.19 csys = 2.31 CPU)
------------------------------------------------------------------------
5.8.3-ithread - OK
All tests successful, 1 test skipped.
Files=25, Tests=1949, 4 wallclock secs ( 2.92 cusr + 0.18 csys = 3.10 CPU)
------------------------------------------------------------------------
5.8.3-ithread-nouseshrplib - OK
All tests successful, 1 test skipped.
Files=25, Tests=1949, 3 wallclock secs ( 2.46 cusr + 0.19 csys = 2.65 CPU)
------------------------------------------------------------------------
5.8.3-nouseperlio - OK
All tests successful, 2 tests skipped.
Files=25, Tests=1947, 3 wallclock secs ( 2.14 cusr + 0.19 csys = 2.33 CPU)
------------------------------------------------------------------------
blead - OK
All tests successful, 2 tests skipped.
Files=25, Tests=1947, 2 wallclock secs ( 2.23 cusr + 0.12 csys = 2.35 CPU)
------------------------------------------------------------------------
blead-ithread - OK
All tests successful, 1 test skipped.
Files=25, Tests=1949, 3 wallclock secs ( 2.92 cusr + 0.20 csys = 3.12 CPU)
------------------------------------------------------------------------
blead-multi - OK
All tests successful, 2 tests skipped.
Files=25, Tests=1947, 3 wallclock secs ( 2.18 cusr + 0.15 csys = 2.33 CPU)
------------------------------------------------------------------------
As you can see, everthing passes the test suite except perl 5.8.0
and 5.8.0-ithread. I have carefully re-installed both versions to
no avail.
Does anyone remember any quirk in 5.8.0 that may help me find out
what is happening? It seems strange that both the non-threaded and
threaded versions fail.
For now, I am tracing and backing out changes until I can isolate
the problem further. When I isolate it, I may need further help.
Aloha => Beau;
I have started a new thread for my Storable PERL_NO_GET_CONTEXT
project, because:
Storable-blead (ext/Storable in the current rsynced perl blead)
- untouched by me - does NOT pass tests for perl 5.8.0.
Here is what I did:
1) rsynced the current perl
2) extracted ext/Storable
3) set perl to 5.8.0
4) perl Makefile.PL; make; make test
The results:
perl -V
-------
Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
Platform:
osname=linux, osvers=2.4.25, archname=i686-linux
uname='linux cathy.beaucox.com 2.4.25 #1 fri feb 20 11:05:53 hst 2004 i686
unknown unknown gnulinux '
config_args='-Dprefix=/home/test/perl/5.8.0 -d -e -Duseshrplib'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef
usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-fno-strict-aliasing -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
optimize='-O3',
cppflags='-fno-strict-aliasing'
ccversion='', gccversion='3.3.3', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lnsl -ldb -ldl -lm -lc -lcrypt -lutil
perllibs=-lnsl -ldl -lm -lc -lcrypt -lutil
libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version='2.3.2'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic
-Wl,-rpath,/home/test/perl/5.8.0/lib/5.8.0/i686-linux/CORE'
cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'
Characteristics of this binary (from libperl):
Compile-time options: USE_LARGE_FILES
Built under linux
Compiled at Feb 28 2004 23:31:27
@INC:
/home/test/perl/5.8.0/lib/5.8.0/i686-linux
/home/test/perl/5.8.0/lib/5.8.0
/home/test/perl/5.8.0/lib/site_perl/5.8.0/i686-linux
/home/test/perl/5.8.0/lib/site_perl/5.8.0
/home/test/perl/5.8.0/lib/site_perl
.
perl Makefile.PL
----------------
Processing hints file hints/linux.pl
Checking if your kit is complete...
Looks good
Writing Makefile for Storable
make
----
cp Storable.pm blib/lib/Storable.pm
AutoSplitting blib/lib/Storable.pm (blib/lib/auto/Storable)
/home/test/perl/5.8.0/bin/perl /home/test/perl/5.8.0/lib/5.8.0/ExtUtils/xsubpp
-typemap /home/test/perl/5.8.0/lib/5.8.0/ExtUtils/typemap Storable.xs >
Storable.xsc && mv Storable.xsc Storable.c
cc -c -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2
-DVERSION=\"2.10\" -DXS_VERSION=\"2.10\" -fpic
"-I/home/test/perl/5.8.0/lib/5.8.0/i686-linux/CORE" Storable.c
Running Mkbootstrap for Storable ()
chmod 644 Storable.bs
rm -f blib/arch/auto/Storable/Storable.so
LD_RUN_PATH="" cc -shared -L/usr/local/lib Storable.o -o
blib/arch/auto/Storable/Storable.so
chmod 755 blib/arch/auto/Storable/Storable.so
cp Storable.bs blib/arch/auto/Storable/Storable.bs
chmod 644 blib/arch/auto/Storable/Storable.bs
make test
---------
PERL_DL_NONLAZY=1 /home/test/perl/5.8.0/bin/perl "-MExtUtils::Command::MM"
"-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/blessed.............ok
t/canonical...........ok
t/code................ok
t/compat06............ok
t/croak...............ok
t/dclone..............ok
t/downgrade...........ok
t/forgive.............ok
t/freeze..............ok
t/integer.............ok
t/interwork56.........skipped
all skipped: Your IVs are no larger than your longs
t/just_plain_nasty....ok
t/lock................ok
t/malice..............ok
t/overload............ok
t/recurse.............ok
t/restrict............Object #1 should have been retrieved already at
t/restrict.t line 111.
dubious
Test returned status 2 (wstat 512, 0x200)
DIED. FAILED tests 9-50
Failed 42/50 tests, 16.00% okay
t/retrieve............ok
t/store...............ok
t/threads.............skipped
all skipped: no threads
t/tied................ok
t/tied_hook...........ok
t/tied_items..........ok
t/utf8................ok
t/utf8hash............ok
Failed 1/25 test scripts, 96.00% okay. 42/1947 subtests failed, 97.84% okay.
Failed Test Stat Wstat Total Fail Failed List of Failed
-------------------------------------------------------------------------------
t/restrict.t 2 512 50 42 84.00% 9-50
2 tests skipped.
make: *** [test_dynamic] Error 29
This is the same problem I get in my modified Storable for
PERL_NO_GET_CONTEXT. The same error occurs for these
configurations of perl 5.8.0:
config_args='-Dprefix=/home/test/perl/5.8.0 -d -e -Duseshrplib'
config_args='-Dprefix=/home/test/perl/5.8.0-ithread -d -e -Dusethreads
-Duseshrplib'
config_args='-Dprefix=/home/test/perl/5.8.0-nosueshrplib -d -e
-Uuseshrplib'
config_args='-Dprefix=/home/test/perl/5.8.0-ithread-nouseshrplib -d -e
-Dusethreads -Uuseshrplib'
All the other perls I have setup test fine:
5.005_03
5.005_04
5.6.0
5.6.0-ithread
5.6.1
5.6.1-ithread
5.6.2
5.6.2-ithread
5.6.2-perlio
5.8.1
5.8.1-ithread
5.8.1-ithread-nouseshrplib
5.8.1-nouseperlio
5.8.2
5.8.2-ithread
5.8.2-ithread-nouseshrplib
5.8.2-nouseperlio
5.8.3
5.8.3-ithread
5.8.3-ithread-nouseshrplib
5.8.3-nouseperlio
blead
blead-ithread
blead-multi
BUT, the latest Storable on CPAN (Storable-2.09) tests OK
for all perl 5.8.0s;
Here is my plan:
1) diff Storable-2.09 Storable-blead
2) back off changes until 5.8.0 works.
If I get is working, 'I'll be back' ((C)Copyright 1980 A.S.)
and show you what seems to be causing the problem.
To be continued...
Aloha => Beau;
Like I said, we need to decide if we want all extensions to use
PERL_NO_GET_CONTEXT. It's a strategic issue.
Tim.
I think trying to explain PERL_NO_GET_CONTEXT too early in this tutorial
is likely to be more harm than help - the approach in Marcus' patch
of adding a new 'how to optimise' example makes sense to me.
I don't think it is generically harmful for people to write XS code
without PERL_NO_GET_CONTEXT up until the point that speed becomes
an issue for the author or users of the module.
However, if we end up bringing Inline in to the core and recommending
it as the first resort for accessing C code from perl, it would be
useful then to think again about what sort of reasons people might
still need to use XS, and how we best teach them enough to do so.
Hugo
But isn't there a wider issue for mod_perl? Stas?
> However, if we end up bringing Inline in to the core and recommending
> it as the first resort for accessing C code from perl, it would be
> useful then to think again about what sort of reasons people might
> still need to use XS, and how we best teach them enough to do so.
True. Especially as (I imagine) it'll be much easier to port Inline
to Perl6.
Tim.
>>I don't think it is generically harmful for people to write XS code
>>without PERL_NO_GET_CONTEXT up until the point that speed becomes
>>an issue for the author or users of the module.
Harmfull, not. But as threads become used more and more it's better to start
writing the module to pass THX around from the very beginning. It doesn't take
too much effort. I'd try to enable it as a default in 5.9.x, once we have a
tutorial ready. Of course we will need to introduce PERL_DO_GET_CONTEXT or
similar to allow modules to continue to run with 5.9.x unmodified (after that
defined will be added).
> But isn't there a wider issue for mod_perl? Stas?
No, there is no impact of PERL_NO_GET_CONTEXT to mod_perl, besides speed.
The only issue is the bug in 5.8.2 which affects any XS module running under
threads. PERL_NO_GET_CONTEXT makes this bug irrelevant for the XS module that
uses that define, since it always passes the right context around.
ALso we still need to fix the core internal API to pass THX around and not use
GET_CONTEXT (like safefree and others).
I've CC'ed Abhijit on this one.
Please send the output verbose testing of the test that fails.
% PERL_DL_NONLAZY=1 /home/test/perl/5.8.0/bin/perl "-MExtUtils::Command::MM"
"-e" "test_harness(1, 'blib/lib', 'blib/arch')" t/restrict
Abhijit was planning to release 2.10 this weekend. I'm not sure whether he has
tested it with 5.8.0.
Otherwise, since you get the failure w/ and w/o your patch and all other perls
look good, I think your patch is ready to go in. Please post the final version
url here and someone will put it in. Mark it as [PATCH] in the subject so it
won't be missed in the noise of the thread.
Good work, Beau!!!
Working on this now.
>
> Abhijit was planning to release 2.10 this weekend. I'm not sure whether he
> has tested it with 5.8.0.
>
> Otherwise, since you get the failure w/ and w/o your patch and all other
> perls look good, I think your patch is ready to go in. Please post the
> final version url here and someone will put it in. Mark it as [PATCH] in
> the subject so it won't be missed in the noise of the thread.
Will have it ready overnight.
>
> Good work, Beau!!!
For general information, I now have a web site and ftp site
for PERL_NO_GET_CONTEXT:
and
<ftp://ftp.beaucox.com/pub/pngx/>
Everything I do will end up there.
Aloha => Beau;
See below.
> [...]
>
> Abhijit was planning to release 2.10 this weekend. I'm not sure whether he
> has tested it with 5.8.0.
The Storable test (t/restrict.t) fails under perl 5.8.0. However, this
test fails in the same way w/o my PERL_NO_GET_CONTEXT patch, so
perhaps it is a problem the the current Storable (about to be released as
2.10). I have documented the problem (with my patch in place):
The verbose test log of t/restrict.t:
<http://pngx.beaucox.com/Storable/5.8.0-restrict-verbose.log>
A trace of t/restrict.t using the module's 'DEBUGME' and 'TRACEME'
facilities:
<http://pngx.beaucox.com/Storable/5.8.0-restrict-verbose.trace>
Complete test results are available at:
<http://pngx.beaucox.com/Storable/index.html>
Aloha => Beau;
from #p5p:
<Nicholas> I assume Beau is seeing what I see:
<Nicholas> ok 8 - value for key 'undef' is undefined
<Nicholas> Object #1 should have been retrieved already at t/restrict.t line 111.
<Nicholas> No idea where that error actually comes from within the guts of
Storable, but I will try to have a look at it tomorrow
<Nicholas> Bah humbug:
<Nicholas> http://public.activestate.com/cgi-bin/perlbrowse?file=hv.c&blame=1
<Nicholas> gives errors about p4
<Nicholas> argh: possibly applying change 22281 from blead to a clean 5.8.0
source will fix bug, in which case it's a core bug, not a Storable bug
though it looks like perlbrowse is down at the moment...