Critical program headers and dynamic tags

[Florian Weimer]

I think the GNU toolchain largely ignores unsupported dynamic tags and
program headers. If you want failures because new features are used
that can't be ignored, you have to use special symbols or new relocation
types, or EI_ABIVERSION.

I do not know if this is standard behavior across ELF implementations.
If it is, I'd like to propose three new dynamic tags:

DT_CRITICAL_PHDR
An implementation should signal an error if it does not support
the program header type denoted by the dynamic tag value.

DT_CRITICAL_LOAD_DT
A program interpreter should signal an error if it does not support
the dynamic tag specified by the value of this dynamic tag.

DT_CRITICAL_LINK_DT
An ELF consumer that is not a program interpreter should signal an
error if it does not support the dynamic tag specified by the value
of this dynamic tag.

In each case, actual presence of matching program headers dynamic tags
is not required for the failure.

There is only one new tag for program header types because I think link
editors do not use program headers, but section headers.

With the new tags, a one-time EI_ABIVERSION bump can be used to make
these compatibility markers completely effective even for older ELF
consumers.

The “critical” terminology follows X.509 and OpenPGP, which use the term
in the same way for extensions that require processing.

Thanks,
Florian
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

[Cary Coutant]

> I think the GNU toolchain largely ignores unsupported dynamic tags and
> program headers. If you want failures because new features are used
> that can't be ignored, you have to use special symbols or new relocation
> types, or EI_ABIVERSION.

It's a pain either way -- when the toolchain rejects an unknown new
feature, it slows the adoption of that feature, but if the new feature
requires that the toolchain understand it, we need a way to flag that,
too.

> I do not know if this is standard behavior across ELF implementations.

In my experience, we've seen it both ways.

> If it is, I'd like to propose three new dynamic tags:
>
> DT_CRITICAL_PHDR
> An implementation should signal an error if it does not support
> the program header type denoted by the dynamic tag value.

We addressed this for section types long ago, by adding the
SHF_OS_NONCONFORMING flag. I'd suggest a similar approach for phdrs.

> DT_CRITICAL_LOAD_DT
> A program interpreter should signal an error if it does not support
> the dynamic tag specified by the value of this dynamic tag.
>
> DT_CRITICAL_LINK_DT
> An ELF consumer that is not a program interpreter should signal an
> error if it does not support the dynamic tag specified by the value
> of this dynamic tag.
>
> In each case, actual presence of matching program headers dynamic tags
> is not required for the failure.

The dynamic tags are harder to deal with, in that we don't have a
supply of flag bits. (We did adopt a convention at one point where the
even-numbered tags implied the use of d_un.d_ptr, while odd-numbered
tags implied the use of d_un.d_val, in effect carving out a single bit
as a flag, but that's not easily extensible.)

We could define a special range of DT_ values that require loader support.

> There is only one new tag for program header types because I think link
> editors do not use program headers, but section headers.

In theory, linkers, when reading a shared library, could go straight
to the program headers and the dynamic table, without depending on the
section table at all. There was a discussion in the binutils mailing
list a while back about whether the linker should be able to read a
shared library whose section table had been stripped. The spec isn't
crystal clear on that, and there were differing opinions. My personal
opinion is that the section table ought to be truly optional in both
ET_DYN and ET_EXEC files. In practice, though, I think it's true that
linkers I'm aware of use the section headers. But I'd design for the
general case, not for the specific cases we have in existence today.

That said, I'm curious what features you think are worthy of flagging
a loader error, but not a linker error -- why do you think we need to
make this distinction?

> With the new tags, a one-time EI_ABIVERSION bump can be used to make
> these compatibility markers completely effective even for older ELF
> consumers.

You're basically talking about new features that break compatibility.
This is what EI_OSABI and EI_ABIVERSION are there for. I don't think
we should be so reluctant to use the mechanism that was already
designed for this.

-cary

[Ali Bahrami]

Hi Florian,

I'd rather not see these tags in the gABI, because they
facilitate a model of development that we (Solaris) don't
support. The merit of Cary's NONCOMFORMING suggestion is that
it's far easier for abstaining implementations to opt out. Of
of course, there's always the OSABI arena, which I think might
be the better fit here.

Regarding the OSABI, EI_ABIVERSION is defined relative to
EI_OSABI, meaning that it has a different meaning depending
on which OSABI the object specifies. I could see bumping it
helping a specific ABI (e.g. OSABI_GNU), but I don't know that
it has an impact on generic features.

EI_ABIVERSION
Byte e_ident[EI_ABIVERSION] identifies the version
of the ABI to which the object is targeted. This
field is used to distinguish among incompatible
versions of an ABI. The interpretation of this
version number is dependent on the ABI identified
by the EI_OSABI field. If no values are specified
for the EI_OSABI field by the processor supplement
or no version values are specified for the ABI
determined by a particular value of the EI_OSABI byte,
the value 0 shall be used for the EI_ABIVERSION byte;
it indicates unspecified.


-----

Longer, more details that you probably care less about...

I don't believe that we document a required behavior
for this. It's one of those unspecified implementation details.
think we probably inherited our behavior from SVR4.

I didn't know the answer off hand for Solaris, so I did a quick
experiment, by editing an object to set phdr and dyn items to
unknown types. It seems that Solaris does ignore unsupported
program headers and dynamic tags. The link-editor (ld) on the
other hand, objects to unknown section types. On the surface,
this might seem inconstant, but it really doesn't matter,
because we don't want to support that model. We intend for
the linkers to always understand everything they are given.
If that's not the case, it indicates a "time traveling" object
built by newer linkers, something we explicitly rule out of bounds
for support.

Our approach (which is part of the Solaris backward compatibility
guarantee) is that we support old objects running on new systems,
but explicitly rule out the reverse (new object, old system).
The user is required to build on the oldest system they want
to support, and we guarantee that it will work on anything newer.
In this model, the scenario where we encounter unknown types
just doesn't happen except by accident, and the answer is always
"don't do that". You might think that people dislike this, and
perhaps some individuals do run afoul of it and resent having
to change their process, but overall, it's been pretty drama free
over many decades of SunOS. It has the merit of being easy to
explain and understand.

Easier said than done of course.

- Ali

[Florian Weimer]

* Cary Coutant:

>> If it is, I'd like to propose three new dynamic tags:
>>
>> DT_CRITICAL_PHDR
>> An implementation should signal an error if it does not support
>> the program header type denoted by the dynamic tag value.
>
> We addressed this for section types long ago, by adding the
> SHF_OS_NONCONFORMING flag. I'd suggest a similar approach for phdrs.

Okay, this might work for program headers.

> That said, I'm curious what features you think are worthy of flagging
> a loader error, but not a linker error -- why do you think we need to
> make this distinction?

Link editors rarely care about the specific behavior of the various
callback functions we have. Say when ELF constructors are called, and
what their exact prototype is. The GNU gABI has additional callback
functions, and we retroactively added additional arguments there on some
targets. There is also an idea to delay the activation of PT_GNU_RELRO
after ELF constructors have run. I want to introduce additional
callbacks similar to DT_FINI_ARRAY. All these extensions require loader
support, and older loaders will produce obscure failures (typically
hard-to-diagnose crashes).

>> With the new tags, a one-time EI_ABIVERSION bump can be used to make
>> these compatibility markers completely effective even for older ELF
>> consumers.
>
> You're basically talking about new features that break compatibility.
> This is what EI_OSABI and EI_ABIVERSION are there for. I don't think
> we should be so reluctant to use the mechanism that was already
> designed for this.

But this is a pretty big hammer. If we bump EI_OSABI, then all linkers
and loaders have to be updated, or they won't be able to process the
objects, right?

This is a pretty big hammer for something that is just an internal
implementation detail of a shared object (e.g., how it deallocates
thread-local data).

[Florian Weimer]

* Ali Bahrami:

> I didn't know the answer off hand for Solaris, so I did a quick
> experiment, by editing an object to set phdr and dyn items to
> unknown types. It seems that Solaris does ignore unsupported
> program headers and dynamic tags. The link-editor (ld) on the
> other hand, objects to unknown section types. On the surface,
> this might seem inconstant, but it really doesn't matter,
> because we don't want to support that model. We intend for
> the linkers to always understand everything they are given.
> If that's not the case, it indicates a "time traveling" object
> built by newer linkers, something we explicitly rule out of bounds
> for support.

It's not a feasible model for us because we have too many supported
implementations out there. I see rare valuabe if we can upgrade one
linker/loader combination first, so that the dynamic loader can use new
run-time features (as directed by the linker), without impacting other
linkers that merely consume the same shared objects.

> Our approach (which is part of the Solaris backward compatibility
> guarantee) is that we support old objects running on new systems,
> but explicitly rule out the reverse (new object, old system).
> The user is required to build on the oldest system they want
> to support, and we guarantee that it will work on anything newer.
> In this model, the scenario where we encounter unknown types
> just doesn't happen except by accident, and the answer is always
> "don't do that". You might think that people dislike this, and
> perhaps some individuals do run afoul of it and resent having
> to change their process, but overall, it's been pretty drama free
> over many decades of SunOS. It has the merit of being easy to
> explain and understand.

That policy is quite similar to ours, except that we no longer have just
one unified GNU toolchain. Release schedules between the ELF consumers
and producers vary, so we need to increasingly watch out for two-way
compatibility.

As far as I understand it, if we bump OSABI because the shared object
uses a new run-time-only feature, we immediately lock out link editors
that don't know about that OSABI level.