Option to use and create "unique" library names

[Brian Fraser]

Background:
Android's linker has some bugg^Wunusual behavior, in that it caches loaded
libraries, but only uses the basename in the cache.  That means that,
as far as its dlopen() is concerned, the libraries for Hash::Util and List::Util,
both of which are named Util.so, are the same.

What we did in the android branch was introduce an option, d_libname_unique,

that when defined, has Makemaker create an "unique" libname for each module,
and teaches XSLoader and Dynaloader to look for those; So for example,
Hash/Util/Util.so becomes Hash/Util/Perl_Hash_Util.so.

The changes for this are in http://perl5.git.perl.org/perl.git/shortlog/refs/heads/hugmeir/d_libname_unique and can be tested by compiling perl with -Dd_libname_unique.

Any objections to this going in?

[H.Merijn Brand]

On Wed, 20 Nov 2013 03:41:55 -0200, Brian Fraser <fras...@gmail.com>
wrote:

The change is by far complete, as it is missing
• documentation in Porting/Glossaary
• documentation in config_h.SH (might not be required in this case)
• entries in Porting/config*
• entry in uconfig.h

run Porting/checkcfgvar.pl

> Any objections to this going in?

Not when all other issues are resolved :)

--
H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/
using perl5.00307 .. 5.19 porting perl5 on HP-UX, AIX, and openSUSE
http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/
http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/

[Brian Fraser]

On Wed, Nov 20, 2013 at 6:09 AM, H.Merijn Brand <h.m....@xs4all.nl> wrote:

On Wed, 20 Nov 2013 03:41:55 -0200, Brian Fraser <fras...@gmail.com>
wrote:

> Background:
> Android's linker has some bugg^Wunusual behavior, in that it caches loaded
> libraries, but only uses the basename in the cache.  That means that,
> as far as its dlopen() is concerned, the libraries for Hash::Util and
> List::Util,
> both of which are named Util.so, are the same.
>
> What we did in the android branch was introduce an option, d_libname_unique,
> that when defined, has Makemaker create an "unique" libname for each module,
> and teaches XSLoader and Dynaloader to look for those; So for example,
> Hash/Util/Util.so becomes Hash/Util/Perl_Hash_Util.so.
>
> The changes for this are in
> http://perl5.git.perl.org/perl.git/shortlog/refs/heads/hugmeir/d_libname_uniqueand
> can be tested by compiling perl with -Dd_libname_unique.

The change is by far complete, as it is missing
• documentation in Porting/Glossaary

That file says that it's generated by metaconfig? https://github.com/castaway/metaconfig/commit/8aca8823163546bf8589275d60d3a4742c7c888c has the changes for metaconfig, including the description, although the branch I pushed has a slightly reworded message for Configure, to avoid ruffling people's feathers.

 

• documentation in config_h.SH (might not be required in this case)

Not necessary, it's not a C-visible symbol.

 

• entries in Porting/config*

Should I only add it to Porting/config.sh, or to all the canned config files?

 

• entry in uconfig.h 

run Porting/checkcfgvar.pl

I did this, both before and after modifying Porting/config.sh, and with perls with/without -Dd_libname_unique, and nothing happened. --tap says that it's all good. I imagine that's not actually true, since the other canned config files are missing the value?

[H.Merijn Brand]

On Wed, 20 Nov 2013 07:43:11 -0200, Brian Fraser <fras...@gmail.com>

wrote:

> On Wed, Nov 20, 2013 at 6:09 AM, H.Merijn Brand <h.m....@xs4all.nl> wrote:
>
> > On Wed, 20 Nov 2013 03:41:55 -0200, Brian Fraser <fras...@gmail.com>
> > wrote:
> >
> > > Background:
> > > Android's linker has some bugg^Wunusual behavior, in that it caches
> > > loaded libraries, but only uses the basename in the cache. That
> > > means that, as far as its dlopen() is concerned, the libraries for
> > > Hash::Util and List::Util, both of which are named Util.so, are the
> > > same.
> > >
> > > What we did in the android branch was introduce an option,
> > > d_libname_unique, that when defined, has Makemaker create an
> > > "unique" libname for each module, and teaches XSLoader and
> > > Dynaloader to look for those; So for example, Hash/Util/Util.so
> > > becomes Hash/Util/Perl_Hash_Util.so.
> > >
> > > The changes for this are in
> > > http://perl5.git.perl.org/perl.git/shortlog/refs/heads/hugmeir/d_libname_uniqueand
> > > can be tested by compiling perl with -Dd_libname_unique.
> >
> > The change is by far complete, as it is missing
> > • documentation in Porting/Glossaary
>
> That file says that it's generated by metaconfig?
> https://github.com/castaway/metaconfig/commit/8aca8823163546bf8589275d60d3a4742c7c888chas

Yes, but the snippet you want to add in Configure comes from the same
meta-unit that holds that documentation.

If you want that code in Configure, I will create you a meta-unit that
not only holds that code, but also holds the documentation I referred
to. Both Configure and Porting/Glossary will be regenerated, but *only*
contain the new code if there is a reference to that code (the
regeneration skips unused meta-units)

> the changes for metaconfig, including the description, although the
> branch I pushed has a slightly reworded message for Configure, to avoid
> ruffling people's feathers.
>
> > • documentation in config_h.SH (might not be required in this case)
>
> Not necessary, it's not a C-visible symbol.
>
> > • entries in Porting/config*
>
> Should I only add it to Porting/config.sh, or to all the canned config
> files?

run the script, and it'll show you what to add where :)

> > • entry in uconfig.h
>
> > run Porting/checkcfgvar.pl
>
> I did this, both before and after modifying Porting/config.sh, and with
> perls with/without -Dd_libname_unique, and nothing happened. --tap says
> that it's all good. I imagine that's not actually true, since the other
> canned config files are missing the value?

Nothing happened, as there is no entry in any of the used files to the
new variable. First check if it is set in config.sh after Configure,
then check if it is set in uconfig.sh and/or Porting/config.sh

[Craig A. Berry]

On Tue, Nov 19, 2013 at 11:41 PM, Brian Fraser <fras...@gmail.com> wrote:
> Background:
> Android's linker has some bugg^Wunusual behavior, in that it caches loaded
> libraries, but only uses the basename in the cache. That means that,
> as far as its dlopen() is concerned, the libraries for Hash::Util and
> List::Util,
> both of which are named Util.so, are the same.
>
> What we did in the android branch was introduce an option, d_libname_unique,
> that when defined, has Makemaker create an "unique" libname for each module,
> and teaches XSLoader and Dynaloader to look for those; So for example,
> Hash/Util/Util.so becomes Hash/Util/Perl_Hash_Util.so.

We've done this since forever on VMS; the actual shareable image
(dynamic library) names come out as PL_HASH__UTIL.EXE and
PL_LIST__UTIL.EXE. There is a function called DynaLoader::mod2fname
that gets called from XSLoader::load. Here's what it does:

$ perl -MDynaLoader -e "@a=('Module','SubModule','SubSubModule');
print DynaLoader::mod2fname(\@a);"
PL_Module__SubModule__SubSubModule

So I think you can get what you want by simply implementing a
mod2fname function (ours is in vms/vms.c, but dl_xxx.xs might be a
better place for a new one). And I think there's a good chance you
wouldn't need any MakeMaker modifications as mod2fname, if it exists,
is already called from MM_Unix::init_main().

Plus it seems less than optimal to introduce an entirely new mechanism
to do something for which there is already an existing mechanism.

[Leon Timmermans]

On Wed, Nov 20, 2013 at 4:04 PM, Craig A. Berry <craig....@gmail.com> wrote:

We've done this since forever on VMS; the actual shareable image
(dynamic library) names come out as PL_HASH__UTIL.EXE and
PL_LIST__UTIL.EXE.  There is a function called DynaLoader::mod2fname
that gets called from XSLoader::load.  Here's what it does:

$ perl -MDynaLoader -e "@a=('Module','SubModule','SubSubModule');
print DynaLoader::mod2fname(\@a);"
PL_Module__SubModule__SubSubModule

So I think you can get what you want by simply implementing a
mod2fname function (ours is in vms/vms.c, but dl_xxx.xs might be a
better place for a new one).  And I think there's a good chance you
wouldn't need any MakeMaker modifications as mod2fname, if it exists,
is already called from MM_Unix::init_main().

Plus it seems less than optimal to introduce an entirely new mechanism
to do something for which there is already an existing mechanism.

Actually, it seems ExtUtils::CBuilder is currently not doing this correctly (it only passes the basename to mod2fname). Module::Build contains a workaround that does do the right thing, which is why no one noticed this before.

I *really* hate how just about any feature in ExtUtils::CBuilder that isn't used by Module::Build can be assumed to be broken or unusable on some non-Linux-like platform (unless someone fixed it in the past couple of years), if not all of them.

Leon

[Craig A. Berry]

On Tue, Nov 26, 2013 at 6:20 PM, Leon Timmermans <faw...@gmail.com> wrote:
> On Wed, Nov 20, 2013 at 4:04 PM, Craig A. Berry <craig....@gmail.com>
> wrote:
>>
>> We've done this since forever on VMS; the actual shareable image
>> (dynamic library) names come out as PL_HASH__UTIL.EXE and
>> PL_LIST__UTIL.EXE. There is a function called DynaLoader::mod2fname
>> that gets called from XSLoader::load. Here's what it does:
>>
>> $ perl -MDynaLoader -e "@a=('Module','SubModule','SubSubModule');
>> print DynaLoader::mod2fname(\@a);"
>> PL_Module__SubModule__SubSubModule
>>
>> So I think you can get what you want by simply implementing a
>> mod2fname function (ours is in vms/vms.c, but dl_xxx.xs might be a
>> better place for a new one). And I think there's a good chance you
>> wouldn't need any MakeMaker modifications as mod2fname, if it exists,
>> is already called from MM_Unix::init_main().
>>
>> Plus it seems less than optimal to introduce an entirely new mechanism
>> to do something for which there is already an existing mechanism.
>
>
> Actually, it seems ExtUtils::CBuilder is currently not doing this correctly
> (it only passes the basename to mod2fname). Module::Build contains a
> workaround that does do the right thing, which is why no one noticed this
> before.

Ouch. You're right. ExtUtils::CBuilder::Platform::VMS::lib_file() is
definitely wrong. I don't really see how to fix it as that interface
assumes you can create the correct loadable library name from the name
of a single object file, but you can't.

It looks like ExtUtils::CBuilder::Platform::os2::_do_link may do
things right with regard to mod2fname as it only uses the output of
lib_file() as a fallback. But I don't think any of this should be in
platform-specific overrides. The definedness of DynaLoader::mod2fname
is what determines whether it's used at run time, so the same thing
should determine whether it's used at build time.

[Brian Fraser]

Yeah, we were discussing exactly that on irc yesterday. Overriding _do_link is dodgy though, since it causes tests to fail, and means that
$b->link(lib_file => $b->lib_file($object)) ne $b->link().
I think that I have the least worst solution, though. We can have ->lib_file() accept an optional module_name parameter, and if that's defined, we can use mod2fname there. Only two modules on CPAN use lib_file directly, and only one would need changing, and the change is fully backwards compatible, so I think it might be the way to go.

With that change, all tests pass on linux with d_libname_unique defined; since the commits also remove the handling of mod2fname in the platform-specific files, this will need smoking on VMS (and OS2, I guess, but that's unlikely)

Proposed changes are now on http://perl5.git.perl.org/perl.git/shortlog/refs/heads/hugmeir/d_libname_unique

[Craig A. Berry]

On Thu, Nov 28, 2013 at 12:18 PM, Brian Fraser <fras...@gmail.com> wrote:
> On Thu, Nov 28, 2013 at 2:55 PM, Craig A. Berry <craig....@gmail.com>
> wrote:
>>
>> On Tue, Nov 26, 2013 at 6:20 PM, Leon Timmermans <faw...@gmail.com> wrote:
>> > On Wed, Nov 20, 2013 at 4:04 PM, Craig A. Berry

>> It looks like ExtUtils::CBuilder::Platform::os2::_do_link may do
>> things right with regard to mod2fname as it only uses the output of
>> lib_file() as a fallback. But I don't think any of this should be in
>> platform-specific overrides. The definedness of DynaLoader::mod2fname
>> is what determines whether it's used at run time, so the same thing
>> should determine whether it's used at build time.
>
>
> Yeah, we were discussing exactly that on irc yesterday. Overriding _do_link
> is dodgy though, since it causes tests to fail, and means that
> $b->link(lib_file => $b->lib_file($object)) ne $b->link().
> I think that I have the least worst solution, though. We can have
> ->lib_file() accept an optional module_name parameter, and if that's
> defined, we can use mod2fname there. Only two modules on CPAN use lib_file
> directly, and only one would need changing, and the change is fully
> backwards compatible, so I think it might be the way to go.
>
> With that change, all tests pass on linux with d_libname_unique defined;
> since the commits also remove the handling of mod2fname in the
> platform-specific files, this will need smoking on VMS (and OS2, I guess,
> but that's unlikely)
>
> Proposed changes are now on
> http://perl5.git.perl.org/perl.git/shortlog/refs/heads/hugmeir/d_libname_unique
>

I've built that on VMS and the report is at
<http://www.nntp.perl.org/group/perl.daily-build.reports/2013/12/msg154855.html>.
t/op/sub.t is an unrelated problem that I think has been fixed in
blead since your last rebase, ext/Pod-Html/t/cache.t is flaky and
fails sporadically on almost every platform, and lib/perl5db.t is a
long-standing set of problems I've never gotten around to. So I
think you're good to go as far as the tests can tell us.

And to be clear, what I'm testing is the case where
$Config{d_libname_unique} is *not* defined, but using the ancient
mod2fname definition in vms/vms.c. And so we know what you're doing
doesn't collide with that. But I'm now inclined to ditch the old XS
version of mod2fname and use yours, so I may give that a try.

I did note that in your _mod2fname_libname_unique function you have this line:

my $path_max = 1024; # No easy way to get this here

Maybe we can't load POSIX in DynaLoader, but I think what you really
want here is NAME_MAX, not PATH_MAX. What your'e assembling is the
name of an individual file, which may have directory portions
prepended in order to make up a complete path. If we can't call POSIX
directly, we can at least take a straw poll; this on Darwin:

$ perl -e "use POSIX; print POSIX::NAME_MAX . qq/\n/;"
255

So I would rename $path_max to $name_max and make it 255 if a better
number doesn't show up.

[Leon Timmermans]

On Thu, Nov 28, 2013 at 7:18 PM, Brian Fraser <fras...@gmail.com> wrote:

Yeah, we were discussing exactly that on irc yesterday. Overriding _do_link is dodgy though, since it causes tests to fail, and means that
$b->link(lib_file => $b->lib_file($object)) ne $b->link().
I think that I have the least worst solution, though. We can have ->lib_file() accept an optional module_name parameter, and if that's defined, we can use mod2fname there. Only two modules on CPAN use lib_file directly, and only one would need changing, and the change is fully backwards compatible, so I think it might be the way to go.

It makes sense from a backwards compatabilty POV, though a module_name method might be better for the future.

 

With that change, all tests pass on linux with d_libname_unique defined; since the commits also remove the handling of mod2fname in the platform-specific files, this will need smoking on VMS (and OS2, I guess, but that's unlikely)

Proposed changes are now on http://perl5.git.perl.org/perl.git/shortlog/refs/heads/hugmeir/d_libname_unique

Some details of the Module::Build patch need some tweaking, but that does look almost ready to apply :-).

Leon

[Irek Szczesniak]

Yes, I do have one. I think this should be fixed in Android and not be
the burden of every damn platform or language to fix. There are more
than perl which will stumble over this, and IMHO the Google staff who
invented this should get the whips where it hurts.

If we work around this issue for perl than Google is likely never
fixing their broken linker.

Irek

[Zefram]

On Wed, Nov 20, 2013 at 6:41 AM, Brian Fraser <fras...@gmail.com> wrote:
>What we did in the android branch was introduce an option, d_libname_unique,
>that when defined, has Makemaker create an "unique" libname for each module,
>and teaches XSLoader and Dynaloader to look for those; So for example,
>Hash/Util/Util.so becomes Hash/Util/Perl_Hash_Util.so.

So Hash::Util and Hash_Util will get the same `unique' name? (There are
two clashes of thus-transformed module names in the current CPAN index;
none of the involved modules is XS.) Mangle ftw.

-zefram

[Leon Timmermans]

No, Hash::Util would collide with the much less likely Hash__Util.

Leon

[Brian Fraser]

*shrug*

Idealism won't get perl working. We already do the libname fixup for other two platforms. We jump through a monstrous amount of hoops for most platforms that we support. Why you think that drawing the line on this one would be meaningful is beyond me.
Moreso, I'd rather have perl working on every version of Android out there *now*, rather than sitting around, twiddling my thumbs until -- if ever -- google fixes bionic's linker.
But even that's useless. I can write an Android app using perl with this workaround (...and some others). By your approach, I'd have to wait for one hypothetical version that won't get any significant market share in at least a couple of years.

Google already knows about the bug. You're free to nag them about it. In fact, Android is open source, so you're free to file a patch.

 

If we work around this issue for perl than Google is likely never
fixing their broken linker.

What do you think this would accomplish? You'd be "telling" Google that a language that they don't support and doesn't quite work on their system... Still doesn't work?

[Brian Fraser]

To follow up on this, the revised version is now on blead, but I did not merge the changes to Module::Build and MakeMaker, so currently compiling with -Dd_libname_unique will fail two of Module::Build's tests; the necessary fixes are already in the respective githubs[0][1], so the tests with -Dd_libname_unique will start passing once those get merged back to blead.

[0] https://github.com/Perl-Toolchain-Gang/Module-Build/commit/ff5b2f970325e7dfbb3caf57a37e6b4807978b97
[1] https://github.com/Perl-Toolchain-Gang/ExtUtils-MakeMaker/commit/60a8ef2f5110732f5689d05b591aab37195a2ebf

[bulk88]

I'm wondering a little bit if Win32 Perl needs this. Win32's runtime
linker does the same basename matching unless you pass an absolute path
(Dynaloader passes absolute paths to Win32's runtime linker). A big
problem results, if a XS DLL has the same filename as non-XS DLL, and
another DLL tries to link to the filename, and linker finds the XS DLL
instead, which only exports the boot function, not dozens of non-Perl C
functions. Here is one with version::'s version.dll conflicting with
MS's version.dll (MS version.dll is a metadata library for PE files's
"version resource")
https://rt.cpan.org/Ticket/Display.html?id=88458#txn-1308331 . I also
recently ran into this problem with building
http://search.cpan.org/~pjacklam/Math-BigInt-GMP-1.37/lib/Math/BigInt/GMP.pm
which generated a GMP.dll, which linked to gmp.dll, and couldn't be
loaded into any process due to missing functions (GMP.dll the XS lib's
imported functions were attempted to be resolved with GMP.dll the XS
lib's exported function table by the Win32 runtime linker, instead of
searching for another gmp.dll using the default collection of folders).
I had to rename the C function exporting gmp.dll to something else and
recompile to make it work.

[Leon Timmermans]

On Tue, Jan 7, 2014 at 7:52 AM, bulk88 <bul...@hotmail.com> wrote:

I'm wondering a little bit if Win32 Perl needs this. Win32's runtime linker does the same basename matching unless you pass an absolute path (Dynaloader passes absolute paths to Win32's runtime linker). A big problem results, if a XS DLL has the same filename as non-XS DLL, and another DLL tries to link to the filename, and linker finds the XS DLL instead, which only exports the boot function, not dozens of non-Perl C functions. Here is one with version::'s version.dll conflicting with MS's version.dll (MS version.dll is a metadata library for PE files's "version resource") https://rt.cpan.org/Ticket/Display.html?id=88458#txn-1308331 . I also recently ran into this problem with building http://search.cpan.org/~pjacklam/Math-BigInt-GMP-1.37/lib/Math/BigInt/GMP.pm which generated a GMP.dll, which linked to gmp.dll, and couldn't be loaded into any process due to missing functions (GMP.dll the XS lib's imported functions were attempted to be resolved with GMP.dll the XS lib's exported function table by the Win32 runtime linker, instead of searching for another gmp.dll using the default collection of folders). I had to rename the C function exporting gmp.dll to something else and recompile to make it work.

Yeah. that sounds like a good idea.

Leon

[Leon Timmermans]

On Tue, Jan 7, 2014 at 6:37 AM, Brian Fraser <fras...@gmail.com> wrote:

To follow up on this, the revised version is now on blead, but I did not merge the changes to Module::Build and MakeMaker, so currently compiling with -Dd_libname_unique will fail two of Module::Build's tests; the necessary fixes are already in the respective githubs[0][1], so the tests with -Dd_libname_unique will start passing once those get merged back to blead.

[0] https://github.com/Perl-Toolchain-Gang/Module-Build/commit/ff5b2f970325e7dfbb3caf57a37e6b4807978b97
[1] https://github.com/Perl-Toolchain-Gang/ExtUtils-MakeMaker/commit/60a8ef2f5110732f5689d05b591aab37195a2ebf

A fix for Module::Build was released to cpan yesterday and imported into blead so everything should work, but I haven't verified that.

Leon

[Brian Fraser]

I haven't actually tried running this in Windows, but I don't see why it wouldn't work -- try adding a line with d_libname_unique='define' to the config files in win32?