PN_XNUM extension for program headers

[Fangrui Song]

Solaris has an ELF extension for executable files with more than 65534 program headers (e_phnum limit).

https://docs.oracle.com/cd/E53394_01/html/E54813/man-eh.html#scrolltoc

> e_phnum
> The number of entries in the program header table. The product of e_phentsize and e_phnum gives the table's size in bytes. If a file has no program header table, e_phnum holds the value zero.
>
> If the number of program headers is greater than 65534, this member has the value PN_XNUM (0xffff). The actual number of program header table entries is contained in the sh_info field of the section header at index 0. Otherwise, the sh_info member of the initial section header entry contains the value zero. See Extended Section Header.

GNU Binutils adopted the PN_XNUM feature in 2010.
https://sourceware.org/pipermail/binutils/2010-January/065151.html ([RFC][PATCH 0/3] readelf and libbfd: Add Extended Numbering Support for ELF)

Executables rarely exceed 10 program headers, but the extended limit could be useful for core dumps. (FreeBSD test https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285547)
Core dump files use program headers to describe memory areas; therefore, a process with many thread stacks can generate a core dump with a significantly large number of headers.

Should this PN_XNUM extension for e_phnum be supported by a generic ABI?

[Roland McGrath]

I hadn't even realized that PN_XNUM hadn't been formally added to the generic ABI originally.  I consider it a necessary part of full ELF support, definitely.  We should certainly formalize that.  It's unavoidably necessary in ET_CORE files, as you say.

It's reasonable enough for program loaders (including dynamic linkers) to impose limits such that they should never expect to deal with the case, and linkers can thus reasonably impose the limit that they won't produce it.  Nonetheless, I think it's better to leave that to the implementations rather than formally specifying that using it in ET_EXEC or ET_DYN files is invalid.

I do think it would be worth formalizing that it's invalid to use PN_XNUM with a sh_info < 65534, such that no program loader or the like that would refuse a large e_phnum is obliged to decode PN_XNUM format just to determine that the actual value is within its limit after all.

[Ali Bahrami]

I have always assumed that this was a gABI feature, and
not purely a Sun addition. Extended section indexes were added
as part of a second round of gABI additions under our

PSARC/2001/332 ELF gABI updates - round II

but the extended program headers were added independently
in 2005, right about when I started at Sun:

PSARC/2005/689 ELF Extended Program Headers

The first sentence in that case is:

To conform to a revised ELF standard, we're introducing
a new constant in <sys/elf.h>.

and later:

This is needed to support core files (which themselves
are in the ELF format) with more than 0xffff mappings.

The gABI is not mentioned specifically, but is the only
"ELF standard" this could be referring to. I have no direct
memory of it, but there must have been a discussion with
other gABI members that settled the details, which then
fell through the gaps and didn't make it to the gABI
document. I also can't recall any discussions about having
actually hit the 0xffff limit in actual core files back
then. That's a lot of mappings, but maybe?

Fortunately, we all seem to have implemented the same thing,
so it would make sense to update the new gABI doc to cover
it and heal the rift.

- Ali

[Cary Coutant]

The first sentence in that case is:

     To conform to a revised ELF standard, we're introducing
     a new constant in <sys/elf.h>.

and later:

     This is needed to support core files (which themselves
     are in the ELF format) with more than 0xffff mappings.

The gABI is not mentioned specifically, but is the only
"ELF standard" this could be referring to. I have no direct
memory of it, but there must have been a discussion with
other gABI members that settled the details, which then
fell through the gaps and didn't make it to the gABI
document. I also can't recall any discussions about having
actually hit the 0xffff limit in actual core files back
then. That's a lot of mappings, but maybe?

This is the first I've seen about this on this mailing list, and I don't recall any side discussions about it.

Fortunately, we all seem to have implemented the same thing,
so it would make sense to update the new gABI doc to cover
it and heal the rift.

 Well, if there's consensus—and it seems there is—I can add this to the v4.3 draft. I'll open an issue in the github project.

-cary