Using 32bit binaries on 64bit arch - ld.so fails to preload tup-ldpreload.so

[Anatol Pomozov]

Hi,

I am trying to build my project that uses custom gcc version.
Currently gcc is compiled for 32 bit architecture:

$ file path/to/gcc
ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically
linked (uses shared libs), for GNU/Linux 2.6.15, not stripped

From other side I use 64 bit Ubuntu, and tup is also compiled for 64 bit
$ file tup
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically
linked (uses shared libs), for GNU/Linux 2.6.15, not stripped

And when I run my build process it fails

[ tup ] Executing Commands...
[ 0/1174 ] path/to/gcc
ERROR: ld.so: object '/home/anatol/bin/tup-ldpreload.so' from
LD_PRELOAD cannot be preloaded: ignored.
Error: Expected to write to file 'foo.o' from cmd 297798 but didn't
*** Command 297798 failed: some log output....

The output *.o file is actually present, so gcc was able to compile
it. What happen is that ld.so was not able to preload 64 bit *.so file
for 32 bit executable.

What is the any workaround for it? Or the only way is to build gcc for 64 arch?

[Anatol Pomozov]

Hi

Ok, I solved my problem. Now I can run both 32 and 64 bit executables from tup.

I created 2 versions of tup-ldpreload.so file, one for 32 bits arch
another for 64, and put them into /usr/lib32 and /usr/lib64
accordingly. And it works. I do not completely understand how ld.so
chooses correct .so version, but I suspect that it just skips .so
files with incorrect architecture.

Here are my changes if anyone has the same issue on x86_64

[Mike Shal]

On Wed, Jan 26, 2011 at 12:12 PM, Anatol Pomozov

Thanks, I will take a look at this. I was also considering trying to
use LD_PRELOAD_32 and LD_PRELOAD_64 if those are supported instead of
forcing the libraries to be installed in /usr.

-Mike

[Mike Shal]

Well it seems LD_PRELOAD_32 and LD_PRELOAD_64 are a Solaris thing, so
that doesn't work on Linux. In Linux I tried export
LD_LIBRARY_PATH="%s/$LIB", (where %s is the normal tup working
directory), and $LIB is apparently some ld.so magic (see:
https://bugzilla.redhat.com/show_bug.cgi?id=249256). Though I tried
that on a 64-bit machine, and $LIB expanded to x86_64 instead of lib64
like that link suggests. I'd like to be able to just have lib/ and
lib64/ directories if possible, rather than a bunch of arch-specific
directories. I'd also like to avoid having to install in the top-level
/lib directories. Anyone have any suggestions?

-Mike

[Anatol Pomozov]

I have the same issue on MacOSX 64 bit. I need to be able to run both
32 and 64 bit applications. And I cannot find any way to do this.
Right now when I try to run 32 bit apps I have following error:

dyld: could not load inserted library: /Users/anatol/bin/tup-ldpreload.so

On a related note: I almost ready to try ptrace instead of ldpreload.
In theory it should work fine on MacOSX and *nix. What is the state of
this implementation? I see that the branch has not been changed/merged
for months.

[Mike Shal]

On Fri, Feb 4, 2011 at 4:15 AM, Anatol Pomozov <anatol....@gmail.com> wrote:
> I have the same issue on MacOSX 64 bit. I need to be able to run both
> 32 and 64 bit applications. And I cannot find any way to do this.
> Right now when I try to run 32 bit apps I have following error:
>
> dyld: could not load inserted library: /Users/anatol/bin/tup-ldpreload.so
>
> On a related note: I almost ready to try ptrace instead of ldpreload.
> In theory it should work fine on MacOSX and *nix. What is the state of
> this implementation? I see that the branch has not been changed/merged
> for months.

I have no idea if this is feasible, but is it possible to build a fat
binary that is 32-bit and 64-bit? Or does OSX only do that for
PPC/x86? I'm wondering if tup-ldpreload.so can be built as both at the
same time, and hope the loader knows what to do with it.

As for ptrace, it is not sufficient on OSX:
http://uninformed.org/index.cgi?v=4&a=3&p=14

In particular for tup, it would need to be able to support automatic
tracing on fork, as well as a stop-at-next-syscall feature. That
doesn't mean there can't be OSX-specific tracing functionality in tup,
but it won't be based on ptrace() (unless that has changed recently).

-Mike

[Abdulla Kamar]

I have no idea if this is feasible, but is it possible to build a fat

binary that is 32-bit and 64-bit? Or does OSX only do that for
PPC/x86? I'm wondering if tup-ldpreload.so can be built as both at the
same time, and hope the loader knows what to do with it.

You should be able to do that using lipo on Mac OS.

--
Abdulla

[Anatol Pomozov]

Hi

On Fri, Feb 4, 2011 at 7:04 PM, Abdulla Kamar <abdull...@gmail.com> wrote:

I have no idea if this is feasible, but is it possible to build a fat

binary that is 32-bit and 64-bit? Or does OSX only do that for
PPC/x86? I'm wondering if tup-ldpreload.so can be built as both at the
same time, and hope the loader knows what to do with it.

I just tried and it works. Thanks to universal binaries support on Macosx [1]. The change is very simple, see it attached to this message.

I see that Linux/Unix has something similar to universal binaries, its called FatELF [2]. But as I understand it is not supported by Linux loader yet.

1] http://en.wikipedia.org/wiki/Fat_binary

2] http://icculus.org/fatelf/

[Mike Shal]

Does it work if you add the -arch flags to the FPIC variable in
macosx.tup? Or do they need to go at the end of the linker line?

Also, what happens if you are supplying the -arch x86_64 flag on a
32-bit osx machine?

Thanks!
-Mike

[Anatol Pomozov]

Hi

I do not see any reason to put it to the root file. The universal library is only needed for *.so file to allow to run both 32 and 64 bit apps from tup. And the universal library is only needed for Intel platform where you have 2 architectures (i386 and x86_64), on other platforms (ppc) you need only *.so file for ppc architecture.

As for the 'tup' binary - I would prefer to use non-universal binaries. Users will compile tup for their own archs.

Also, what happens if you are supplying the -arch x86_64 flag on a
32-bit osx machine?

It should work. XCode can generate output for other platforms (such as ppc on my x86_64). It should allow to generate 64 code on 32 (assuming that XCode is recent enough).

Ok, let's continue 32vs64 saga on MacOSX. Last couple of days I've learn a lot about MacOSX. One thing is so called Symbol Variants [1]. Many functions in libSystem library (libc) have several variants. For example open() function has 2 variants for x86_64:

$ nm -arch x86_64 /usr/lib/libSystem.dylib | grep _open

000000000000b8b4 T _open

0000000000002ee4 T _open$NOCANCEL

and 3 variants on i386 platform

$ nm -arch i386 /usr/lib/libSystem.dylib | grep _open

000d781b T _open

00001400 T _open$NOCANCEL$UNIX2003

0000c3f8 T _open$UNIX2003

And depending on how do you compile you project, on what platform and with what compiler keys your open() function will be bound to some of these variants. For example ldpreload.so compiled for 32bit is bound to _open$UNIX2003 while on 64 bit platforms it bound to _open

See [2] it contains header file that constructs variant names (such as #define __DARWIN_ALIAS)

I assume that tup-ldperload should wrap all variants on both on 32 and 64 platform. Right?

My question is what is the best way to do this. One thing is how to avoid code duplications. All 5 open() functions will be the same (except the variant suffix). Another question is how to tell compiler not to use __asm() from header. I want to tell compiler that open() should be bound exactly to _open on 32 bit platform, not to another name.

There is some progress in tup + mac [3] it works for 64 bit apps and some of the 32 bit apps.

[1] http://developer.apple.com/library/mac/#releasenotes/Darwin/SymbolVariantsRelNotes/

[2] http://www.google.com/codesearch/p?hl=en#J-o3djqejfs/bsd/sys/cdefs.h&q=__DARWIN_ALIAS_C%20file:cdefs.h&sa=N&cd=1&ct=rc

[3] https://github.com/anatol/tup/commit/90aacc94a32c0208bfebcaee06c5d41ec6fed3d7

[Mike Shal]

On Thu, Feb 10, 2011 at 8:29 PM, Anatol Pomozov

What is your argument against putting it in macosx.tup? That's the
place where flags specific to macosx goes, so you don't have to have
ifeq (@(TUP_PLATFORM),macosx) everywhere. The FPIC variable is only
used to build ldpreload, so it seems like a good place for the -arch
flags.

I would think so, yes. I'm certainly no expert on OSX's symbol
variants, but if there's a way for a client program to call a
particular version of open(), then ldpreload should wrap it so tup
knows about it.

> My question is what is the best way to do this. One thing is how to avoid
> code duplications. All 5 open() functions will be the same (except the
> variant suffix). Another question is how to tell compiler not to use __asm()
> from header. I want to tell compiler that open() should be bound exactly to
> _open on 32 bit platform, not to another name.
> There is some progress in tup + mac [3] it works for 64 bit apps and some of
> the 32 bit apps.

Some of the open() code could certainly be pulled out into a separate
function, but as I understand it there has to be a function definition
for each in order to get the symbol into ldpreload.

In your macosx branch, why does tup need the -macosx-version-min=10.5 flag?

Thanks!
-Mike

[Anatol Pomozov]

Hi

There is no particular reason, it is ok to put it to macosx.tup. But the varname FPIC is confusing for it. The arch is not related to position independent code and it needed only for ldpreload. Maybe var with name LDPRELOAD_ARCH is better?

I think I made it work on MacOSX. I'll send an update in a separate thread.

 

> My question is what is the best way to do this. One thing is how to avoid
> code duplications. All 5 open() functions will be the same (except the
> variant suffix). Another question is how to tell compiler not to use __asm()
> from header. I want to tell compiler that open() should be bound exactly to
> _open on 32 bit platform, not to another name.
> There is some progress in tup + mac [3] it works for 64 bit apps and some of
> the 32 bit apps.

Some of the open() code could certainly be pulled out into a separate
function, but as I understand it there has to be a function definition
for each in order to get the symbol into ldpreload.

In your macosx branch, why does tup need the -macosx-version-min=10.5 flag?

As of  macosx-version-min - it was a local experiment. I tried to understand how and when the symbol names changed. I just removed it from my local branch.

[Anatol Pomozov]

Hi

With @(TUP_ARCH) it is possible to compile universal tup-ldpreload.so
file on 64 bits platform.

Here are 2 changes in my upstream branch:
1) For macosx. It is very simple
https://github.com/anatol/tup/commit/d74c5c529fd5d484a3b0177694b358d189f755f2
2) For Linux/Solaris the change looks uglier
https://github.com/anatol/tup/commit/912b29e139d10c66abdc0e1542a725e3d4e03ff2

The idea with 64 bits Linux is that we generate two *.so files, one
for 32 bits and another for 64 bits. And instead of using
LD_PRELOAD=absolute_path we use pair of LD_PRELOAD=tup-ldpreload.so +
LD_LIBRARY_PATH, where latter is set to <tup_dir>:<tup_dir>/lib32.

If you compile tup on 64 bits Linux, you'll have 2 files:
tup-ldpreload.so and tup-ldpreload.32.so. tup-ldpreload.so should be
copied to <tup_dir> (dir where the binary lives) and
tup-ldpreload.32.so should be copied to
<tup_dir>/lib32/tup-ldpreload.32.so.

Tested both on 64 bit Linux Ubuntu and 64 bit MacOSX.

[Mike Shal]

These should both be merged. I fiddled with the second commit a
little, so it just creates lib32/tup-ldpreload.so directly instead of
having a tup-ldpreload.32.so that needs to be copied. It seems to work
in 32-bit native, 64-bit native, and 32-bit under 64-bit.

Thanks for implementing it!

-Mike