Extract an archive member to satisfy a DSO undef

[Fangrui Song]

[Also posted to https://sourceware.org/pipermail/binutils/2021-August/117688.html ,

but this list has many folks from non-Linux backgrounds or having experience with other binary formats.]

ELF linkers extract an archive member to satisfy an undefined symbol from a

shared object. (Do Unix linkers have this behavior?) Here is an example:

  echo '.globl _start; _start: call bb' > a.s

  echo '.globl bb; bb: call cc' > b.s

  echo '.globl cc; cc: nop' > c.s

  cc -c a.s b.s c.s

  rm -f c.a && ar rc c.a c.o

  ld -shared -z defs c.o -o c.so

  ld -shared -z defs b.o ./c.so -o b.so

  

  ld -rpath=. a.o b.so c.a

a.out defines cc. ld.bfd, gold, and ld.lld have the same behavior.

This behavior intrigued me because link.exe (PE/COFF) and ld64 (Mach-O) don't

inspect undefined symbols in DLL/dylib for symbol resolution.

  # On macOS

  echo '.globl _main; _main: call bb' > a.s

  echo '.globl bb; bb: call cc' > b.s

  echo '.globl cc; cc: nop' > c.s

  clang --target=x86_64-apple-darwin -c a.s b.s c.s

  rm -f c.a && ar rc c.a c.o

   

  clang -dynamiclib c.o -o c.dylib

  clang -dynamiclib b.o c.dylib -o b.dylib

  clang a.o b.dylib c.a

  # a.out doesn't define or reference cc

Does anyone know the design consideration of using (or not using) this behavior?

Below has some thoughts of mine.

You probably want to think for a few minutes before looking at mine:)

I want different thoughts.

---

We know that an undef from a DSO can interact with a .o definition.  The .o

definition needs to be exported to .dynsym in case the DSO relies on symbols in

the executable. (PE/COFF and Mach-O don't need/have the behavior.)

This does not necessarily generalize to the interaction between a DSO and an

archive.

If a DSO linked into the final executable has incomplete DT_NEEDED entries, I

can imagine that not extracting archive members to satisfy a DSO undef can cause

ld.so "unresolved symbol" errors. (--no-allow-shlib-undefined can flag similar

fragile links.) However, if every DSO is linked with -z defs (--no-undefined),

not extracting archive members appears to be very robust. (My summary of [1].)

---

In my `ld -rpath=. a.o b.so c.a` ELF example, if we make c.so DF_SYMBOLIC,

a.out will have definitions trying to interpose c.so definitions in vain.

Linkers seem to have no abilibity detect such one definition rule violation.

(One definition rule is C++'s, but we can reuse the concept discussing

symbol resolution in a robust system.)

---

In [2], I raised the point that it can be difficult to make --no-undefined happy

because all linkers as I know don't support an option listing allowed undefined

symbols.

Related discussions:

[1] https://bugs.llvm.org/show_bug.cgi?id=43554

[2] https://reviews.llvm.org/D108006 (propose --no-search-static-libs-for-shlib-undefined to ld.lld)

[Ali Bahrami]

I'm not sure what distinction you're drawing
between Unix linkers, and ELF linkers. The first
ELF linkers were all on SYSVr4 derived Unix. There
are of course other unix formats (a.out, coff, ecoff, ...),
but it's been awhile since they were dominant.

Speaking of ELF generally, I think that any object that
doesn't specify all of its dependencies is broken. A well
formed library has full symbol closure --- every symbol
is resolved by the object itself, or by a direct dependency
(DT_NEEDED), or for libraries possibly from the parent
object. Indirect dependencies (dependencies of dependencies)
don't count --- those scenarios are common, and can be made
to work, but those objects are not well formed, and are just
getting "lucky", by intent, or otherwise.

The ELF Forefathers made a mistake (my opinion) in making
full closure only the default for executables, but not for
libraries, and and this encouraged a practice of creating
incomplete libraries, so it's pretty common, and unlikely
to go away fully. I don't think the benefits of full
closure are well understood generally, and programmers
don't give it the attention it deserves.

In Solaris, the core OS is built with -zdefs so that the
libraries we ship do have full closure. Many third parties
we see don't do that, but at least they sit on a solid
foundation.

Anyway, getting down to the actual question, I don't see
an issue with the link-editor trying to pull code out
of an archive to satisfy any outstanding symbol. Archive
extraction is based on unsatisfied references, but the
source of the reference (library, executable, etc) is
not part of the equation. Hence, the behavior you're
asking about seems fine to me --- that's how it's
supposed to work, for better or worse. It's much
too late to change that now, even if one wanted to.
Fix the library to have full closure, and you don't
have to care about it.

Different systems evolved differently, and have different
rules. One could debate which is best, but ultimately,
you have to play by house rules. Full symbol closure
makes life simpler everywhere, so I think it's good advice
generally, not just for ELF.

- Ali

> --
> You received this message because you are subscribed to the Google Groups "Generic System V Application Binary Interface" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to generic-abi...@googlegroups.com <mailto:generic-abi...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/generic-abi/490a74d0-b286-42e6-8bd0-452f3af926ffn%40googlegroups.com
> <https://groups.google.com/d/msgid/generic-abi/490a74d0-b286-42e6-8bd0-452f3af926ffn%40googlegroups.com?utm_medium=email&utm_source=footer>.