I have created a PAR file which contains my script, and the included
modules include an XS module.
However if I try to run the script, the XS module fails to load with
DynaLoader, as the system-installed .so file has been loaded, rather
than the one included with the PAR file.
To give an example:
test.pl contains:
#!/usr/bin/perl
use strict;
use warnings;
use Template;
my $foo = Template->new({STASH => 'Template::Stash::XS'});
I create a par like so:
pp -p -o test.par test.pl
I verify that the contents of the PAR contain Template/Stash/XS.pm and
Template/Stash/XS.so, and that they are version 2.20.
Now I attempt to run the PAR on a machine with version 2.19 of the above
installed, and get:
$ parl test.par test.pl
Failed to initialise Template: failed to create context: failed to
create context: failed to load Template/Stash/XS.pm: Couldn't load
Template::Stash::XS 2.20:
Template::Stash::XS object version 2.19 does not match bootstrap
parameter 2.20 at /usr/lib/perl/5.10.0/DynaLoader.pm line 219.
I may be making a mistake here of course!
This is tested on:
Perl 5.10.0
PAR 0.984
PAR::Packer 0.982
AutoLoader 5.68
PAR::Dist 0.43
Sorry, jumping back into this bug again after a long break.
The workaround I was using was simply to ensure the target system didn't
install much at all into the system libraries, thus ensuring no
conflicts, and then forgot about the issue.
I've noticed the problem still seems to exist, even with the PAR being
built on an identical system, so I'm sure the shared objects should be
compatible. In fact, the PAR and parl executables were built on one
system, and installed on the other, seeming to indicate binary
compatibility between the two.
The system-installed XS module's shared-library is loaded in preference
to the one included in the PAR, thus causing DynaLoader to die.
(in this case, it's Digest::SHA1 and Digest/SHA1.so)
Is there anything else I can check in the meantime, that would be
preventing the par-included SHA1.so to be loaded? I have verified it is
included in the PAR.
thanks.
On Thu Feb 04 03:03:13 2010, TJC wrote:
> Sorry, jumping back into this bug again after a long break.
Have you re-checked with the latest and greatest version of
PAR::Packer?
If so, does the minimal "hello world"
$ pp -o hello -e 'print "hello world\n"'
demonstrate the problem?
On Thu Feb 04 09:32:52 2010, RSCHUPP wrote:
> On Thu Feb 04 03:03:13 2010, TJC wrote:
> > Sorry, jumping back into this bug again after a long break.
>
> Have you re-checked with the latest and greatest version of
> PAR::Packer?
>
> If so, does the minimal "hello world"
>
> $ pp -o hello -e 'print "hello world\n"'
>
> demonstrate the problem?
Sorry, it does not.
Small test case. Note that on the building system, File::Temp is at 0.22
(current) whereas the target system, File::Temp is, erm, slightly older
(standard debian lenny) and doesn't have the newdir method.
#!/usr/bin/perl
use strict;
use warnings;
use File::Temp;
my $dir = File::Temp->newdir;
On Thu Feb 04 18:12:00 2010, TJC wrote:
> Small test case. Note that on the building system, File::Temp is at 0.22
> (current) whereas the target system, File::Temp is, erm, slightly older
> (standard debian lenny) and doesn't have the newdir method.
>
> #!/usr/bin/perl
> use strict;
> use warnings;
> use File::Temp;
> my $dir = File::Temp->newdir;
Can't reproduce this here - strace'ing the packed version of above
shows that the system File::Temp is NOT used.
Did you build PAR::Packer on the same system you're doing
the packing? Or did you build it somewhere else (or got it
from a distro)?
On Sat Feb 06 08:35:17 2010, RSCHUPP wrote:
> Did you build PAR::Packer on the same system you're doing
> the packing? Or did you build it somewhere else (or got it
> from a distro)?
While we're at it: if your.exe is the packed executable from
the example above, please run
$ strings -a your.exe
and search for the string "File/Temp.pm" in the output.
You should be looking at something like:
6b3c9548/File/Temp.pm
9#line 1 "/usr/share/perl/5.10/File/Temp.pm"
package File::Temp;
#line 138
# 5.6.0 gives us S_IWOTH, S_IWGRP, our and auto-vivifying filehandls
# People would like a version on 5.004 so give them what they want :-)
use 5.004;
use strict;
use Carp;
use File::Spec 0.8;
use File::Path qw/ rmtree /;
...
That's actually a copy of File/Temp.pm - scroll down a bit and look for
$VERSION = ...
If that's the same version as on your target system (or at least
older than that on your build system) I have an idea what's
going on :)
Cheers, Roderich
To demonstrate the other/related problem, involving XS modules, create
sha.pl as:
#!/usr/bin/perl
use strict;
use warnings;
use Digest::SHA1;
my $digest = Digest::SHA1->new;
Create a PAR with:
pp -p -o sha.par sha.pl
(Do this on a machine with version 2.12 of Digest::SHA1, eg. from CPAN)
Then copy to new machine (which contains 2.11 of Digest::SHA1, eg. from
Debian repository), and run it:
tobyc@arya-tmp$ perl -MPAR sha.par
Digest::SHA1 object version 2.11 does not match bootstrap parameter 2.12
at /usr/lib/perl/5.10/DynaLoader.pm line 219.
Compilation failed in require at script/shatest.pl line 4.
BEGIN failed--compilation aborted at script/shatest.pl line 4.
BEGIN failed--compilation aborted.
On Sat Feb 06 08:35:17 2010, RSCHUPP wrote:
> On Thu Feb 04 18:12:00 2010, TJC wrote:
> > Small test case. Note that on the building system, File::Temp is at
0.22
> > (current) whereas the target system, File::Temp is, erm, slightly
older
> > (standard debian lenny) and doesn't have the newdir method.
> >
> > #!/usr/bin/perl
> > use strict;
> > use warnings;
> > use File::Temp;
> > my $dir = File::Temp->newdir;
>
> Can't reproduce this here - strace'ing the packed version of above
> shows that the system File::Temp is NOT used.
I build the par file with:
pp -p -o testfile.par testfile.pl
(where testfile.pl contains the above text only)
If I extract the par with `unzip testfile.par` and then view the
extracted file, I see it does contain File/Temp.pm, and it is the
version from the source system, ie. 0.22.
If I copy testfile.par to the remote system, and then execute either
`parl testfile.par` or `par -MPAR testfile.par` I receive the error:
tobyc@arya-tmp:~$ perl -MPAR filetest.par
Can't locate object method "newdir" via package "File::Temp" at
script/filetest.pl line 5.
BEGIN failed--compilation aborted.
(Note that the target system already has File::Temp 0.18 available in
the system, but that version doesn't support the newdir() method.)
> Did you build PAR::Packer on the same system you're doing
> the packing? Or did you build it somewhere else (or got it
> from a distro)?
PAR::Packer was built on the source machine, and turned into a debian
package, which was then installed on the target machine.
Thanks for your help so far,
Toby
On Sun Feb 07 21:37:08 2010, TJC wrote:
> tobyc@arya-tmp$ perl -MPAR sha.par
Sorry, I misread your report until now - you're talking
about a .par file, not a packed executable.
I think what happens is a kind of chicken and egg problem:
- your .par contains different versions of File::Temp etc
than installed on the target machine
- conceptually we would like "perl -MPAR foo.par"
to behave like
- extract file.par into temporary directory /tmp/foo123
- run "perl -I/tmp/foo123 /tmp/foo123/foo.pl"
(i.e. "/tmp/foo123" is at the start of @INC)
- in order to get to the modules in the .par file, PAR.pm
needs some modules, e.g. File::Temp
- hence it already has loaded the target system's File::Temp
before it even gets to consider the (different) copy included
in the .par
All modules cited in your report are in this category
"required by PAR.pm to extract stuff from a .par file".
(Digest::SHA1 is a special case that's even weirder.)
Sorry, I see no sane way to make this work.
Hi Toby, hi Roderich,
RSCHUPP via RT wrote:
> Queue: PAR
> Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=42986 >
> I think what happens is a kind of chicken and egg problem:
>
> - your .par contains different versions of File::Temp etc
> than installed on the target machine
>
> - conceptually we would like "perl -MPAR foo.par"
> to behave like
> - extract file.par into temporary directory /tmp/foo123
> - run "perl -I/tmp/foo123 /tmp/foo123/foo.pl"
> (i.e. "/tmp/foo123" is at the start of @INC)
>
> - in order to get to the modules in the .par file, PAR.pm
> needs some modules, e.g. File::Temp
>
> - hence it already has loaded the target system's File::Temp
> before it even gets to consider the (different) copy included
> in the .par
>
> All modules cited in your report are in this category
> "required by PAR.pm to extract stuff from a .par file".
> (Digest::SHA1 is a special case that's even weirder.)
>
> Sorry, I see no sane way to make this work.
I agree. No sane way. But you may try to do something like this. I would
probably not do it in production.
- Pack Class::Unload into your .par.
- Load the .par file (using system File::Temp)
- Load Class::Unload.
- Extract File/Temp.pm from the .par file into the appropriate location
of the temporary storage location. Just have a look, the layout is
basically a normal lib/ dir. PAR.pm has hooks for loading the contents
of a file from any loaded .par file.
- Run Class::Unload->unload('File::Temp')
- require File::Temp
This may or may not work. It's certainly one of the uglier hacks I've
suggested in the past.
Best regards,
Steffen
Quoth bug...@rt.cpan.org:
> On Sun Feb 07 21:37:08 2010, TJC wrote:
> > tobyc@arya-tmp$ perl -MPAR sha.par
>
> Sorry, I misread your report until now - you're talking
> about a .par file, not a packed executable.
>
> I think what happens is a kind of chicken and egg problem:
>
> - your .par contains different versions of File::Temp etc
> than installed on the target machine
>
> - conceptually we would like "perl -MPAR foo.par"
> to behave like
> - extract file.par into temporary directory /tmp/foo123
> - run "perl -I/tmp/foo123 /tmp/foo123/foo.pl"
> (i.e. "/tmp/foo123" is at the start of @INC)
>
> - in order to get to the modules in the .par file, PAR.pm
> needs some modules, e.g. File::Temp
>
> - hence it already has loaded the target system's File::Temp
> before it even gets to consider the (different) copy included
> in the .par
>
> All modules cited in your report are in this category
> "required by PAR.pm to extract stuff from a .par file".
> (Digest::SHA1 is a special case that's even weirder.)
>
> Sorry, I see no sane way to make this work.
How about doing the unpacking, and then re-execing perl with exactly
that command-line? A little slow, perhaps, but PAR's startup (at least
when it has to do unpacking) isn't exactly fast anyway.
Ben
On Wed Feb 10 01:44:53 2010, TJC wrote:
> Should this issue still occur with executable packed archives?
> I ask because I am having a shot at that method instead, but still seem
> to be having the same problem.
Basically it's the same problem, but at least you can do
something about it :)
The packed executable has the same chicken-and-egg problem
as the .par archive. As a first approximation, a packed
executable is just a custom built perl interpreter with
a .par bolted on. In order to get at the stuff in the .par
you need a minimal set of Perl modules. A .par takes these
from the Perl environment where it is run (and doesn't
care if different versions of these modules are included in
the .par). The packed executable takes these modules from its
own executable file (NOT from the appended .par). You may find
them extracted below /tmp/par-username/cache-xxxx/, but you
probably won't recognize them, because their file names have
been mangled (some-checksum.pm). If you find, say, a File::Temp
in the cache area that actually is called File/Temp.pm then this
is the one extracted from the .par, but as in the .par case,
it's not the one that gets loaded.
See one of my earlier comments how to find out about
the "built-in" modules in a packed executable.
Which version of a specific module is "built-in" depends
on the environment that was current when PAR::Packer was built.
(That's why I asked where you got your PAR::Packer from.)
If you rebuild PAR::Packer on your building system
it will pick up the current versions of these modules.
Should this issue still occur with executable packed archives?
I ask because I am having a shot at that method instead, but still seem
to be having the same problem.
If i look in the unpacked /tmp/par-username/cache-xxxx/ directory, I see
that I have version 2.024 of the .pm and .so files, but when I attempt to
run the executable, I am told that it can't as object version 2.008 does
not match bootstrap parameter 2.024. (The source system has 2.024, the
target system has 2.008 installed).
Running "strings -a my_executeable" and looking for Compress::Raw::Zlib
in the output shows that 2.024 (the newer version) appears to be packed
in. (At least for the pm version.. How do I check for the .so that's
packed?)
On Wed Feb 10 20:21:10 2010, TJC wrote:
> Running "strings -a my_executeable" and looking for Compress::Raw::Zlib
> in the output shows that 2.024 (the newer version) appears to be packed
> in. (At least for the pm version.. How do I check for the .so that's
> packed?)
Run the attached script as
$ perl extract-embedded.pl EXECUTABLE DIR
it will extract all stuff (modules and shared libs) embedded
in EXECUTABLE into DIR (using their original names).
To check the version of .so I suggest to compare md5sums against
possible candidates.
> I think there might be a problem with the @INC order..
> The old, system version of Compress::Raw::Zlib still exists, in
> /usr/lib/perl/5.10.0, and the new version is in
> /usr/local/lib/perl/5.10.0.
> The latter version is higher in @INC and is the version picked
> up by modules.. However it seems like pp is picking
> up the older version for some reason?
Unlikely. pp will use the order as shown by
$ perl -e 'print "@INC\n"'
or does /usr/local/lib/perl/5.10.0 get prepended because
you have PERL5LIB set in your environment?
Chers, Roderich
I note that b3cf5856 = Compress/Raw/Zlib.pm
and ce08cac2 = auto/Compress/Raw/Zlib/Zlib.so
I tried 'strace'ing the executable, and noticed:
644: stat64("/tmp/par-tobyc/cache-
038134f668a1529182c945464ecd15936dab8d95/ce08cac2.so", {st_mode=S_IFREG|0755,
st_size=331208, ...}) = 0
(no corresponding open() that I see)
1018: stat64("/tmp/par-tobyc/cache-
038134f668a1529182c945464ecd15936dab8d95/b3cf5856.pm", {st_mode=S_IFREG|0644,
st_size=15076, ...}) = 0
1019: open("/tmp/par-tobyc/cache-
038134f668a1529182c945464ecd15936dab8d95/b3cf5856.pm", O_RDONLY|O_LARGEFILE) = 7
1062: stat64("CODE(0x9795f00)/auto/Compress/Raw/Zlib", 0x968f0c0) = -1 ENOENT (No
such file or directory)
1069: stat64("/usr/lib/perl/5.10/auto/Compress/Raw/Zlib/Zlib.so",
{st_mode=S_IFREG|0644, st_size=104100, ...}) = 0
1070: stat64("/usr/lib/perl/5.10/auto/Compress/Raw/Zlib/Zlib.bs", 0x968f0c0) = -1
ENOENT (No such file or directory)
1071: open("/usr/lib/perl/5.10/auto/Compress/Raw/Zlib/Zlib.so", O_RDONLY) = 7
I attach the full strace.
On Thu Feb 11 21:28:56 2010, TJC wrote:
> I tried 'strace'ing the executable, and noticed:
Yeah, the strace shows that .so's (but not corresponding .pm's)
are still searched using the built-in @INC.
This rings a bell, but I can't pin it yet.
This is Perl 5.10.0, right?
After using extract-embedded.pl on your executable
please post the extracted XSLoader.pm and PAR/Heavy.pm
Cheers, Roderich
On Thu Feb 11 05:25:53 2010, RSCHUPP wrote:
> Run the attached script as
>
> $ perl extract-embedded.pl EXECUTABLE DIR
>
> it will extract all stuff (modules and shared libs) embedded
> in EXECUTABLE into DIR (using their original names).
> To check the version of .so I suggest to compare md5sums against
> possible candidates.
Checking the original executable:
$ ./blah
Compress::Raw::Zlib object version 2.008 does not match bootstrap
parameter 2.024 at /usr/local/lib/perl/5.10.0/Compress/Raw/Zlib.pm line
97.
Compilation failed in require at /home/tobyc/perl/lib/Archive/Zip.pm
line 12.
BEGIN failed--compilation aborted at /home/tobyc/perl/lib/Archive/Zip.pm
line 12.
Compilation failed in require at -e line 358.
$ ./extract-embedded.pl blah extract/
...
$ cd extract
$ find .|grep Zlib
./IO/Compress/Zlib
./IO/Compress/Zlib/Extra.pm
./Compress/Zlib.pm
./Compress/Raw/Zlib.pm
./auto/Compress/Raw/Zlib
./auto/Compress/Raw/Zlib/autosplit.ix
./auto/Compress/Raw/Zlib/Zlib.so
./auto/Compress/Zlib
./auto/Compress/Zlib/autosplit.ix
$ grep VERSION ./Compress/Raw/Zlib.pm
$VERSION = '2.024';
$ md5sum ./auto/Compress/Raw/Zlib/Zlib.so
04c2dc8c8fe0ac9606ef29aad5c959d4
# This matches the md5sum of version 2.024 on the build machine.
So it seems like the executable PAR file contains both .pm and .so of
version 2.024, yet when executed is still mysteriously trying to load
the .so from the target machine.
On Fri Feb 12 05:41:41 2010, RSCHUPP wrote:
> On Thu Feb 11 21:28:56 2010, TJC wrote:
> > I tried 'strace'ing the executable, and noticed:
>
> Yeah, the strace shows that .so's (but not corresponding .pm's)
> are still searched using the built-in @INC.
> This rings a bell, but I can't pin it yet.
>
> This is Perl 5.10.0, right?
Correct.
> After using extract-embedded.pl on your executable
> please post the extracted XSLoader.pm and PAR/Heavy.pm
Attached.
On Wed Feb 10 18:14:24 2010, TJC wrote:
> On Wed Feb 10 04:59:03 2010, RSCHUPP wrote:
> > If you rebuild PAR::Packer on your building system
> > it will pick up the current versions of these modules.
>
> I'm fairly sure that PAR::Packer was built after those modules, since
it
> (and PAR) were the modules that required these things to be updated
from
> the default-installed versions. However I will rebuild it again and
see.
Hmm, nope, same problem after rebuilding PAR::Packer.
I think there might be a problem with the @INC order..
The old, system version of Compress::Raw::Zlib still exists, in
/usr/lib/perl/5.10.0, and the new version is in
/usr/local/lib/perl/5.10.0.
The latter version is higher in @INC and is the version picked up by
modules.. However it seems like pp is picking up the older version for
some reason?
.. Actually, I have deleted those copies, then rebuild PAR and
PAR::Packer, then rebuild my test script (pp -o blah blah.pl) and I am
*still* getting the error regarding versions on the target system.
On Wed Feb 10 04:59:03 2010, RSCHUPP wrote:
> If you rebuild PAR::Packer on your building system
> it will pick up the current versions of these modules.
I'm fairly sure that PAR::Packer was built after those modules, since it
(and PAR) were the modules that required these things to be updated from
the default-installed versions. However I will rebuild it again and see.
ta,
Toby
On Sat Feb 13 03:37:38 2010, TJC wrote:
> > This is Perl 5.10.0, right?
>
> Correct.
I finally remembered what's going on :(
PAR::Heavy interposes the loading of shared libraries
by tricking DynaLoader into always calling DynaLoader::dl_expandspec
(which PAR::Heavy has sneakily replaced by its own version).
DynaLoader usually first tries to locate the shared library
by explicity checking along @INC; only if this fails it
will call dl_expandspec.)
Unfortunately this trick (setting $DynaLoader::do_expand = 1)
doesn't work in Perl 5.10.0 (look into DynaLoader.pm, $do_expand
is defined, but nowhere used). It works in all versions before
and after.
So this explains what you see: the .pm part of a XS based module
is correctly resolved from the packed executable, but the
corresponding shared library is resolved from the target system.
If the two happen to be of different versions, Perl aborts.
Looking at the versions of DynaLoader.pm in Perl 5.10.0 and
5.10.1 (on Linux) I think that you may savely replace the
former with the latter. If you do that remember to rebuild PAR::Packer
afterwards so that packed executables will have the "fixed"
DynaLoader.pm embedded.
On Sun Feb 14 12:44:51 2010, RSCHUPP wrote:
> On Sat Feb 13 03:37:38 2010, TJC wrote:
> > > This is Perl 5.10.0, right?
> >
> > Correct.
>
> I finally remembered what's going on :(
>
> PAR::Heavy interposes the loading of shared libraries
> by tricking DynaLoader into always calling DynaLoader::dl_expandspec
> (which PAR::Heavy has sneakily replaced by its own version).
> DynaLoader usually first tries to locate the shared library
> by explicity checking along @INC; only if this fails it
> will call dl_expandspec.)
> Unfortunately this trick (setting $DynaLoader::do_expand = 1)
> doesn't work in Perl 5.10.0 (look into DynaLoader.pm, $do_expand
> is defined, but nowhere used). It works in all versions before
> and after.
>
> So this explains what you see: the .pm part of a XS based module
> is correctly resolved from the packed executable, but the
> corresponding shared library is resolved from the target system.
> If the two happen to be of different versions, Perl aborts.
>
> Looking at the versions of DynaLoader.pm in Perl 5.10.0 and
> 5.10.1 (on Linux) I think that you may savely replace the
> former with the latter. If you do that remember to rebuild PAR::Packer
> afterwards so that packed executables will have the "fixed"
> DynaLoader.pm embedded.
Thanks for your help on this.
Rather than backport Dynaloader to 5.10.0, I ended up making my own
installation of 5.10.1 + modules; I was considering using it to build
PARs but then decided to ship the entire tree with the application. It
blows the size out quite a bit, but it's a lot simpler than trying to
get Module::ScanDeps to locate everything.
Cheers,
Toby
Hello,
Thank you for contacting NetZero.
This is an automated reply and no further response will be sent. Please do not reply to this email.
If you are looking for help here are some useful links:
* Online NetZero Help Center: http://www.netzero.com/support
* Online Billing and Account Center: https://account.netzero.net
* Reset/Retrieve your Password: http://my.netzero.com/s/resetpassword
* Lookup Access Numbers in your area: http://my.netzero.com/s/numbers
* Order/Download NetZero Software: http://www.netzero.com/support/start/download-index.html
Thank you for choosing NetZero.
Sincerely,
NetZero Member Services