Re: Garbage collection of SHT_GROUP sections

[Rafael Espíndola]

Looks like the same problem I had with my previous email.

H.j.lu, could you please take a look?

Thanks,

Rafael
On Wednesday, 11 January 2017, Evgenii Stepanov <eug...@google.com> wrote:

(trying to make the message appear in the google groups web ui)

On Wed, Jan 11, 2017 at 12:00 PM, Evgenii Stepanov <eug...@google.com> wrote:
> Hi,
>
> ELF spec is not clear on whether the GC is allowed to discard
> individual sections from a group, or only the entire group at once. In
> other words, the question is if section groups have the semantics of
> "associative" comdats in COFF terminology (see
> IMAGE_COMDAT_SELECT_ASSOCIATIVE) or not.
>
> It looks like BFD and Gold disagree on this issue. BFD discards entire
> section groups, and Gold may discard individual sections.
>
> Our use case requires the BFD behavior. What we are trying to do is,
> for each global, emit a "global descriptor", which is another global
> that points to the original and is located in a specially named
> section. Then we use __start_$section and __stop_$section symbols to
> iterate over all descriptors at  runtime. We don't want the
> descriptors to affect the liveness of globals in any way. Currently,
> this approach defeats linker GC in Gold.
>
> Previous discussion, along with a patch for LLD that implements the
> BFD-like behavior, can be found at the link:
> http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20170109/417985.html

[Evgenii Stepanov]

[Evgenii Stepanov]

[Rafael Avila de Espindola]

Evgenii Stepanov <eug...@google.com> writes:

> On Wed, Jan 11, 2017 at 12:00 PM, Evgenii Stepanov <eug...@google.com> wrote:
>> Hi,
>>
>> ELF spec is not clear on whether the GC is allowed to discard
>> individual sections from a group, or only the entire group at once. In
>> other words, the question is if section groups have the semantics of
>> "associative" comdats in COFF terminology (see
>> IMAGE_COMDAT_SELECT_ASSOCIATIVE) or not.

Yes, unfortunately garbage collection is not defined in the spec. It is
not clear if it should specify everything, but at least having a notion
of required sections (gc roots) and dependencies would be useful.

>> It looks like BFD and Gold disagree on this issue. BFD discards entire
>> section groups, and Gold may discard individual sections.
>>
>> Our use case requires the BFD behavior. What we are trying to do is,
>> for each global, emit a "global descriptor", which is another global
>> that points to the original and is located in a specially named
>> section. Then we use __start_$section and __stop_$section symbols to
>> iterate over all descriptors at runtime. We don't want the
>> descriptors to affect the liveness of globals in any way. Currently,
>> this approach defeats linker GC in Gold.

I don't think it requires the bfd behavior. It requires the linker
knowing more about the dependencies.

For GC, each linker has a somewhat convoluted set of rules for finding
the roots and the dependencies:

* Sections like .ctor are roots. Sections whose names are valid C
identifiers are roots, etc.
* Most sections with relocations to another section imply a dependency
on that direction.
* For some special sections (arm unwind, .eh_frame), the direction of
the dependency is inverted.

Given this my first idea was to have a SHF_METADATA section flag that
would indicate that relocations from that section imply a reverse
dependency. Like SHF_INFO_LINK, not all such sections would need it. It
would still be valid for the linker to know that .eh_frame is one such
section without the flag.

In the discussion on the lld list Peter Collingbourne noted that the
flag could say, in addition, that a section is not a gc root. That way a
compiler could tag a global constructor that is known to not have side
effects other than initializing the variable. If the variable is
dropped, so could the constructor.

Cheers,
Rafael

[Cary Coutant]

> ELF spec is not clear on whether the GC is allowed to discard
> individual sections from a group, or only the entire group at once. In
> other words, the question is if section groups have the semantics of
> "associative" comdats in COFF terminology (see
> IMAGE_COMDAT_SELECT_ASSOCIATIVE) or not.

The ELF spec (aka gABI) does not discuss GC at all. Garbage collection
is offered by ld and gold as an optional feature beyond what is
specified by the gABI. It has nothing to do with section groups.

The ELF spec provides for any number of different kinds of section
groups, but currently defines only one kind: the COMDAT group. This
kind of group was designed for C++ template instantiations and
out-of-line inlined functions. We discard whole groups (keeping
exactly one per unique signature) as part of the COMDAT group
mechanism, but once we've chosen which group to keep from among the
duplicates, the sections within that group can be treated as ordinary
sections, and may or may not be subject to garbage collection.

Other kinds of section groups are certainly possible, but ELF does not
currently specify any.

> It looks like BFD and Gold disagree on this issue. BFD discards entire
> section groups, and Gold may discard individual sections.

I'm assuming you mean that BFD will *not* discard unreferenced
sections from within a group. (Both linkers discard entire groups as
part of COMDAT processing.) I don't know why it wouldn't do that -- I
can't find any basis for that in the documentation.

> Our use case requires the BFD behavior. What we are trying to do is,
> for each global, emit a "global descriptor", which is another global
> that points to the original and is located in a specially named
> section. Then we use __start_$section and __stop_$section symbols to
> iterate over all descriptors at runtime. We don't want the
> descriptors to affect the liveness of globals in any way. Currently,
> this approach defeats linker GC in Gold.

It sounds like (if I understand correctly) you want a kind of
"reverse" reference for the purposes of GC. In other words, you have a
reference from section A into section B, and you want to establish a
reverse dependency arc from B to A as a result, so that if section B
is referenced, then section A should also be treated as referenced.
This isn't a new use case -- we'd want the same kind of thing for
debug sections and EH frame sections, for example [1] -- but there
really isn't a way to model that in ELF at the moment.

Rather than use section groups for this, I'd suggest something like a
new section flag SHF_ASSOCIATED, which tells the linker to reverse the
dependency arcs for any references made from that section into other
sections. It might even make sense to extend the meaning of the
existing SHF_LINK_ORDER flag, since its current purpose is really just
another aspect of the same condition: a section that is associated
with another section.

-cary

[1] Some static constructors, too, as sometimes we only need a static
constructor to run if the things it references are not themselves
garbage collected. (I'm thinking here of the static constructors used
to implement Google's command-line flags.)

[Rafael Avila de Espindola]

Cary Coutant <ccou...@gmail.com> writes:

> It sounds like (if I understand correctly) you want a kind of
> "reverse" reference for the purposes of GC. In other words, you have a
> reference from section A into section B, and you want to establish a
> reverse dependency arc from B to A as a result, so that if section B
> is referenced, then section A should also be treated as referenced.
> This isn't a new use case -- we'd want the same kind of thing for
> debug sections and EH frame sections, for example [1] -- but there
> really isn't a way to model that in ELF at the moment.
>
> Rather than use section groups for this, I'd suggest something like a
> new section flag SHF_ASSOCIATED, which tells the linker to reverse the
> dependency arcs for any references made from that section into other
> sections. It might even make sense to extend the meaning of the
> existing SHF_LINK_ORDER flag, since its current purpose is really just
> another aspect of the same condition: a section that is associated
> with another section.

So have SHF_LINK_ORDER be allowed with a 0 sh_link with the meaning that
there is a reverse dependency in the relocations?

I will try to prototype that in MC/LLD.

Thanks,
Rafael

[Rafael Avila de Espindola]

Rafael Avila de Espindola <rafael.e...@gmail.com> writes:

>> really isn't a way to model that in ELF at the moment.
>>
>> Rather than use section groups for this, I'd suggest something like a
>> new section flag SHF_ASSOCIATED, which tells the linker to reverse the
>> dependency arcs for any references made from that section into other
>> sections. It might even make sense to extend the meaning of the
>> existing SHF_LINK_ORDER flag, since its current purpose is really just
>> another aspect of the same condition: a section that is associated
>> with another section.
>
> So have SHF_LINK_ORDER be allowed with a 0 sh_link with the meaning that
> there is a reverse dependency in the relocations?
>
> I will try to prototype that in MC/LLD.

I implemented a prototype of that with lld. One problem we hit is that
BFD ld will warn on SHF_ASSOCIATED with no sh_Link, which makes it
inconvenient to write .o files that work both with the new gc logic and
BFDs old behavior.

SHF_ASSOCIATED works better. What I did was

* Assign 0x1000 to SHF_ASSOCIATED.
* In the .section directive in .s files, map the 'o' flag to
SHF_ASSOCIATED.
* In lld reverse the gc edges for SHF_ASSOCIATED sections.

One interesting corner case is what should happen when a relocation
points to a SHF_MERGE section. We probably don't want to keep the
SHF_ASSOCIATED section alive if any part of SHF_MERGE is.

What is the next step? How does one send a patch to the spec?

Cheers,
Rafael

[Suprateeka R Hegde]

On 19-Jan-2017 02:35 AM, Rafael Avila de Espindola wrote:
> I implemented a prototype of that with lld. One problem we hit is that
> BFD ld will warn on SHF_ASSOCIATED with no sh_Link, which makes it
> inconvenient to write .o files that work both with the new gc logic and
> BFDs old behavior.
>
> SHF_ASSOCIATED works better. What I did was
>
> * Assign 0x1000 to SHF_ASSOCIATED.
> * In the .section directive in .s files, map the 'o' flag to
> SHF_ASSOCIATED.

If mapping to 'o' flag is OS specific, then the value 0x1000 is
incorrect. The value should be under SHF_MASKOS range.

ON HP-UX, we handle this internally (since entire stack is proprietary
and only for a specific target). The GC edges and dependencies do not
use any of such flags.

If there is no benefit of this flag to any other SVR4 platforms, may be
it is better to call it under GNU-gABI and name it SHF_GNU_ASSOCIATED?

BTW, does it make sense to rename SHF_EXCLUDR to SHF_GNU_EXCLUDE?

> * In lld reverse the gc edges for SHF_ASSOCIATED sections.

What about the semantics to mark a section as not a GC root? That was
also a requirement (in addition)?

>
> One interesting corner case is what should happen when a relocation
> points to a SHF_MERGE section. We probably don't want to keep the
> SHF_ASSOCIATED section alive if any part of SHF_MERGE is.

To handle a similar situation, on HP-UX, we use SHF_HP_MERGE and solve
it in a OS specific way. But what you are suggesting also makes sense. I
dont see any conflicts.

>
> What is the next step?

Usually if there is consensus, all SVR4 compliant UNIX should at least
detect this flag and not emit unknown flag. If there is no consensus, it
is better to put it under GNU-gABI.

(I do not have any conflict on HP-UX to add this SHF_ASSOCIATED flag, at
least for detection through our elfdump. I will consider implementing in
our linker/tools later in time)

> How does one send a patch to the spec?

This is a repeated question. John Wolfe is no longer maintaining the
SPEC I believe.

JLW asked us to be patient till things get reorganized :-)

--
Supra

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

[Ali Bahrami]

Speaking for Solaris, our unused section elimination (what I think
you mean by garbage collection) doesn't require such a flag either. The
approach is to track referenced sections, by noting the relocations against
them, and to discard the ones that aren't referenced. This all happens after
all section processing has occurred, and so, after COMDAT processing. At that
point, the notion of a "group" has outlived its usefulness, and any unreferenced
section from a group is tossed, like any other.

A small number of section types needed to be exempt from this culling (whitelisted),
but it's a small unchanging list, and we haven't seen any issues with this
in the several years its been in the wild. It would be nice if those sections
were explicitly tagged, but as we support multiple compilers, old and new,
we knew we could never really depend on the compilers, so we solve the problem
in a way that doesn't require it.

Perhaps I'm missing something, but but I think we'd prefer these additions
to be part of the GNU ABI rather than the generic ABI as well. As Supra notes,
the gABI is currently not very easy to modify, so making your proposal part of
the GNU ABI would be easier for you in that way as well.

- Ali

[Rafael Avila de Espindola]

Ali Bahrami <Ali.B...@Oracle.COM> writes:

> Speaking for Solaris, our unused section elimination (what I think
> you mean by garbage collection) doesn't require such a flag either. The
> approach is to track referenced sections, by noting the relocations against
> them, and to discard the ones that aren't referenced. This all happens after
> all section processing has occurred, and so, after COMDAT processing. At that
> point, the notion of a "group" has outlived its usefulness, and any unreferenced
> section from a group is tossed, like any other.

Correct. That is exactly what we mean by garbage collection, and is what gold and
lld implement.

> A small number of section types needed to be exempt from this culling (whitelisted),
> but it's a small unchanging list, and we haven't seen any issues with this
> in the several years its been in the wild. It would be nice if those sections
> were explicitly tagged, but as we support multiple compilers, old and new,
> we knew we could never really depend on the compilers, so we solve the problem
> in a way that doesn't require it.

Similar for lld. Section like .ctor are always roots and and sections
like the arm unwind info have reverse edges.

We now got a "customer" (address sanitizer) request for a way to have
their own metadata about a section and keep that metadata only if the
section it refers to is kept.

So we wouldn't replace the existing logic as we need to support old
compilers too, but would expose the flag for users that want the same
behavior with other sections.

> Perhaps I'm missing something, but but I think we'd prefer these additions
> to be part of the GNU ABI rather than the generic ABI as well. As Supra notes,
> the gABI is currently not very easy to modify, so making your proposal part of
> the GNU ABI would be easier for you in that way as well.

That is a pity, as the flag could be used in other systems too (for
supporting address sanitizer for example), but I understand the
hesitation.

Cary, to add this under SHF_MASKOS, should I just start a discussion on
the binutils list? Would you be OK with implementing a similar change on gold?

Cheers,
Rafael

[Rafael Avila de Espindola]

Suprateeka R Hegde <hegdes...@gmail.com> writes:

> If mapping to 'o' flag is OS specific, then the value 0x1000 is
> incorrect. The value should be under SHF_MASKOS range.

The intention at least was for it to be generic.

>> * In lld reverse the gc edges for SHF_ASSOCIATED sections.
>
> What about the semantics to mark a section as not a GC root? That was
> also a requirement (in addition)?

Sections with the flag are never considered gc roots.

>> How does one send a patch to the spec?
>
> This is a repeated question. John Wolfe is no longer maintaining the
> SPEC I believe.
>
> JLW asked us to be patient till things get reorganized :-)

:-(

Thanks,
Rafael

[Rafael Avila de Espindola]

Reid Kleckner <r...@google.com> writes:

> Can you elaborate on what the use cases are for comdat groups in ELF if we
> aren't putting metadata (debug info, .eh_frame, asan global metadata, etc)
> in the group? I'm aware of GCC's destructor alias optimization involving
> D5, but that doesn't seem like an important enough use case to guide the
> design and usage of ELF comdat groups.

The use case is to allow fast and simple elimination of multiple
related sections. For example, given

inline int *foo() {
static int bar;
return &bar;
}
void zed() {
foo();
}

There will be multiple related sections. One text section, one data
section for bar. If bar was more complicated, what section it was in
(ro/rw) could depend on the translation unit and optimization levels.

Comdats make it possible for the linker to always select a paired foo
and bar. And it doesn't even have to read the sections of unselected
comdats.

> It's possible that I am alone in imagining the rough equivalence between
> ELF comdat groups and COFF associative comdats, but it seemed pretty clear
> to me that ELF comdat groups were intended to be used for metadata the way
> that COFF associative comdat sections are.

Yes, if you have a metadata for the function foo above, it is a good
practice to put that in a section in the same comdat. But garbage
collection is an unrelated feature.

> In fact, the more I've learned about the way things work on ELF, the more
> I've been surprised at how much duplicate metadata remains after the final
> link step. For example, both clang and gcc seem to construct monolithic
> .eh_frame and .debug_info sections full of information describing inline
> functions that may be discarded.

That is a very unfortunate case indeed. LLD has special logic to
optimize .eh_frame, but not .debug_info.

I would much prefer if something like the arm unwind tables were used in
other platforms.

> Going forward, if DWARF producers wanted
> to try to eliminate this metadata from the final link, do you think we
> should use comdat groups or this new flag? I think whatever ELF uses to
> drop duplicate DWARF would also apply to our asan global metadata use case.

Debug info is already in one of the gc special cases (non alloc sections
don't keep alloc sections alive), but it would probably be cleaner to
add this flag to any new addition.

Cheers,
Rafael

[Rafael Avila de Espindola]

Reid Kleckner <r...@google.com> writes:

> On Mon, Jan 23, 2017 at 12:24 PM, Rafael Avila de Espindola <
> rafael.e...@gmail.com> wrote:
>
>> The use case is to allow fast and simple elimination of multiple
>> related sections. For example, given
>>
>> inline int *foo() {
>> static int bar;
>> return &bar;
>> }
>> void zed() {
>> foo();
>> }
>>
>> There will be multiple related sections. One text section, one data
>> section for bar. If bar was more complicated, what section it was in
>> (ro/rw) could depend on the translation unit and optimization levels.
>>
>> Comdats make it possible for the linker to always select a paired foo
>> and bar. And it doesn't even have to read the sections of unselected
>> comdats.
>
>

> I'm not aware of any C++ compilers that put foo and bar into a comdat
> group.

Sorry about the confusion, yes, they are in different groups:

.section .text._Z3foov,"axG",@progbits,_Z3foov,comdat
.section .bss._ZZ3foovE3bar,"aGw",@nobits,_ZZ3foovE3bar,comdat

> If you want to support inlining foo, you need to keep bar in its own
> comdat group. In fact, there are tons of ABI compatibility bugs around
> this, precisely because different TUs make different decisions about the
> readonly-ness of bar.
>
> I don't think comdat groups solve this problem directly. I can imagine that
> comdat group *priorities* might, but that's a separate issue.

Things that right now end up in the same group: relocation sections and
arm's unwind info.

> I don't see any reason why GC needs to be treated separately. What we need
> is a way to represent that one section is metadata describing another
> section, and if it is discarded for any reason (comdat resolution or GC),
> it should be discarded as well.

That happens. The fact that gc is independent just means that if a
section gets gced (in addition to not being selected initially) is
independent of it being in a comdat or not.

> I guess I'm hesitant to discard the comdat group representation in favor of
> a new flag. Multiple people have come on to say that section GC is
> orthogonal to comdat resolution, but to me they both discard sections. We
> should have one representation for section metadata so that both section
> discarding tasks can discard metadata when they discard a section.

But the semantics are different. For comdat the rule is to just keep one
set of sections. With gc, a section can be discarded that was not even
in a comdat in the first place.

Cheers,
Rafael

[Cary Coutant]

> Can you elaborate on what the use cases are for comdat groups in ELF if we
> aren't putting metadata (debug info, .eh_frame, asan global metadata, etc)
> in the group? I'm aware of GCC's destructor alias optimization involving D5,
> but that doesn't seem like an important enough use case to guide the design
> and usage of ELF comdat groups.

It was always intended that a whole collection of sections associated
with the function would be in the comdat group -- it was essentially
supposed to be an "object file within an object file". The idea was
that associated code, data, unwind, debug, etc., sections would all be
in the one group, and could be kept or discarded as a whole group.
(You talk about "metadata", but it's more than just metadata.) This
was an improvement over prior template instantiation schemes, which
involved either (a) a post-link step to identify missing
instantiations, compile them, then relink, (b) using weak symbols to
suppress the multiple-definition complaints, but otherwise living with
duplicate code (possibly relying on garbage collection to clean things
up), or (c) using "link-once" sections but no grouping.

Unfortunately, due to the nature of DWARF debug information, and the
way most compilers generated it, it wasn't practical to put the debug
info in the group -- at least in GCC and other compilers I'm familiar
with -- and we had to live with the duplication (which could get quite
unwieldy in large projects, sadly). I think GCC has always been a bit
primitive, compared to some other vendor compilers, with regards to
including other associated sections in the comdat groups, and I
suspect this is related to the simple conversion from the old
"link-once" sections that GCC had used prior to comdat groups.

> I'm not sure what you mean about how they were intended to be used for
> template instantiation. Was the idea that all of an explicit
> 'std::vector<int>' instantiation would live in a single comdat group? That's
> definitely not the way things seem to operate today.

No -- all the sections for each individual function were meant to be
part of a single group, but each function gets its own group.

> It's possible that I am alone in imagining the rough equivalence between ELF
> comdat groups and COFF associative comdats, but it seemed pretty clear to me
> that ELF comdat groups were intended to be used for metadata the way that
> COFF associative comdat sections are.

I'm not familiar enough with PE-COFF to give you a good answer. We
definitely borrowed the idea (and the name) of "COMDAT" from
Microsoft, but we designed it specifically for the needs of template
instantiation rather than copying the exact specification.

> In fact, the more I've learned about the way things work on ELF, the more
> I've been surprised at how much duplicate metadata remains after the final
> link step. For example, both clang and gcc seem to construct monolithic
> .eh_frame and .debug_info sections full of information describing inline

> functions that may be discarded. Going forward, if DWARF producers wanted to

> try to eliminate this metadata from the final link, do you think we should
> use comdat groups or this new flag? I think whatever ELF uses to drop
> duplicate DWARF would also apply to our asan global metadata use case.

Yes, it's regrettable. I think some of the vendor compilers (e.g.,
Sun, HP, IBM) may do better, but I don't think anyone has yet solved
the DWARF issues. The DWARF spec might need some small enhancements to
make it practical, and I've had that on my list of ideas to work on
for years now.

>> Comdats make it possible for the linker to always select a paired foo
>> and bar. And it doesn't even have to read the sections of unselected
>> comdats.
>
> I'm not aware of any C++ compilers that put foo and bar into a comdat group.

> If you want to support inlining foo, you need to keep bar in its own comdat
> group. In fact, there are tons of ABI compatibility bugs around this,
> precisely because different TUs make different decisions about the
> readonly-ness of bar.

No, you don't need to put bar in its own comdat group, as long as you
always generate the out-of-line instance of foo along with it. You do
need to globalize the name of bar, so that it can be referenced from
any inlined code outside of the comdat group.

The requirement that each instance of a comdat group must define the
same set of global symbols is critical, and failure to follow that
rule has been the cause of many mysterious "undefined symbol" errors
and other bugs.

-cary

[Cary Coutant]

>> Perhaps I'm missing something, but but I think we'd prefer these additions
>> to be part of the GNU ABI rather than the generic ABI as well. As Supra notes,
>> the gABI is currently not very easy to modify, so making your proposal part of
>> the GNU ABI would be easier for you in that way as well.
>
> That is a pity, as the flag could be used in other systems too (for
> supporting address sanitizer for example), but I understand the
> hesitation.
>
> Cary, to add this under SHF_MASKOS, should I just start a discussion on
> the binutils list? Would you be OK with implementing a similar change on gold?

This is the place to try to convince the other gABI participants that
the feature would be generally useful. If the consensus is that it's
not, then, yes, a Gnu-specific feature would be best discussed on the
binutils list (I'd include the gcc list as well). [1] When/if you
start a new discussion there, you should link to this discussion.

Sure, I'll be willing to add support for this flag in gold.

-cary

[1] Too bad, as we have 9 available bits in the generic set, but only
6 in the Gnu set.

[Suprateeka R Hegde]

On 24-Jan-2017 06:21 AM, Cary Coutant wrote:

> [1] When/if you
> start a new discussion there, you should link to this discussion.

...and add gnu-gabi AT sourceware.org also to the list.

> [1] Too bad, as we have 9 available bits in the generic set, but only
> 6 in the Gnu set.

With support for so many targets, with ever expanding feature set, such
ELF based flag space is bound to exhaust and is not scalable. Its very
easy for feature-rich GNU based toolchain to reach saturation in that
direction.

Just like H.J's GNU_PROPERTY_X86_ISA_* is a scalable solution as
compared to ELF based EF_machine_flags, it may make sense to add a new
GNU property something like GNU_PROPERTY_SHF_ATTRIBS. May not be
immediately, but when/if we run out of space.

[Reid Kleckner]

On Mon, Jan 23, 2017 at 12:24 PM, Rafael Avila de Espindola <rafael.e...@gmail.com> wrote:

The use case is to allow fast and simple elimination of multiple
related sections. For example, given

inline int *foo() {
  static int bar;
  return &bar;
}
void zed() {
  foo();
}

There will be multiple related sections. One text section, one data
section for bar. If bar was more complicated, what section it was in
(ro/rw) could depend on the translation unit and optimization levels.

Comdats make it possible for the linker to always select a paired foo
and bar. And it doesn't even have to read the sections of unselected
comdats.

I'm not aware of any C++ compilers that put foo and bar into a comdat group. If you want to support inlining foo, you need to keep bar in its own comdat group. In fact, there are tons of ABI compatibility bugs around this, precisely because different TUs make different decisions about the readonly-ness of bar.

I don't think comdat groups solve this problem directly. I can imagine that comdat group *priorities* might, but that's a separate issue.

 

> It's possible that I am alone in imagining the rough equivalence between

> ELF comdat groups and COFF associative comdats, but it seemed pretty clear
> to me that ELF comdat groups were intended to be used for metadata the way
> that COFF associative comdat sections are.

Yes, if you have a metadata for the function foo above, it is a good
practice to put that in a section in the same comdat. But garbage
collection is an unrelated feature.

I don't see any reason why GC needs to be treated separately. What we need is a way to represent that one section is metadata describing another section, and if it is discarded for any reason (comdat resolution or GC), it should be discarded as well.

 

> In fact, the more I've learned about the way things work on ELF, the more

> I've been surprised at how much duplicate metadata remains after the final
> link step. For example, both clang and gcc seem to construct monolithic
> .eh_frame and .debug_info sections full of information describing inline
> functions that may be discarded.

That is a very unfortunate case indeed. LLD has special logic to
optimize .eh_frame, but not .debug_info.

I would much prefer if something like the arm unwind tables were used in
other platforms.

I agree, from looking at LLVM MC, it appears to use comdat groups. :)

------

[Reid Kleckner]

(Re-sending through the web UI. Maybe generic-abi requires you to be a subscriber to post. We'll see if I am now.)

Can you elaborate on what the use cases are for comdat groups in ELF if we aren't putting metadata (debug info, .eh_frame, asan global metadata, etc) in the group? I'm aware of GCC's destructor alias optimization involving D5, but that doesn't seem like an important enough use case to guide the design and usage of ELF comdat groups.

I'm not sure what you mean about how they were intended to be used for template instantiation. Was the idea that all of an explicit 'std::vector<int>' instantiation would live in a single comdat group? That's definitely not the way things seem to operate today.

It's possible that I am alone in imagining the rough equivalence between ELF comdat groups and COFF associative comdats, but it seemed pretty clear to me that ELF comdat groups were intended to be used for metadata the way that COFF associative comdat sections are.

In fact, the more I've learned about the way things work on ELF, the more I've been surprised at how much duplicate metadata remains after the final link step. For example, both clang and gcc seem to construct monolithic .eh_frame and .debug_info sections full of information describing inline functions that may be discarded. Going forward, if DWARF producers wanted to try to eliminate this metadata from the final link, do you think we should use comdat groups or this new flag? I think whatever ELF uses to drop duplicate DWARF would also apply to our asan global metadata use case.

[Ali Bahrami]

We're prepared to reconsider, and Cary's implication that we
should is a strong motivation to do so. The simple truth is that
we end up having to reverse engineer and adapt to lots of GNUish
things. Pushing this off to the GNU ABI doesn't mean that we won't
end up having to figure out how to deal with it when gcc starts
producing sections that use it. So let's continue the discussion a bit...

I've been rereading the messages in this thread, and I understand
that the intent is to supply a way to indicate reverse relationships
between 2 sections. We already have what might be called forward
links, where one section conveys that it uses another, via the
sh_link, or sh_info (with SHF_INFO_LINK). Here, you're proposing a
flag, SHF_ASSOCIATED, that conveys the reverse: A section with
SHF_ASSOCIATED is saying "I'm needed by some other section".

That might be useful, but generally, we know from the type of
section whether or not that's true. For instance, we know that
.eh_frame sections don't stand alone, and that they are always
paired with some other (text) section. A flag saying so isn't
adding information. The more useful thing would be for this new
mechanism to also tell us which section is doing the pointing.
I can't tell from reading this thread whether that's part of the
proposal or not.

A lot has been said already, and the proposal has shifted a bit,
so it might help to do a reset and start fresh. Would you consider
giving us the diffs for the changes you would want to see in the
gABI for SHF_ASSOCIATED, as you currently understand it it?

Thanks...

- Ali

[Cary Coutant]

> We're prepared to reconsider, and Cary's implication that we
> should is a strong motivation to do so. The simple truth is that
> we end up having to reverse engineer and adapt to lots of GNUish
> things. Pushing this off to the GNU ABI doesn't mean that we won't
> end up having to figure out how to deal with it when gcc starts
> producing sections that use it. So let's continue the discussion a bit...

Sorry, I didn't mean to imply that I think you should reconsider --
I'm on the fence here, especially since I no longer represent a
vendor. I only meant to suggest to Rafael that perhaps he hadn't fully
made his case yet.

(It could be that Rafael doesn't much care one way or the other, and
I'm the one trying to make a case! So maybe I'm not truly on the
fence. Hmmm. Let's just say that I'm looking to make sure we do
consider the idea fairly.)

> I've been rereading the messages in this thread, and I understand
> that the intent is to supply a way to indicate reverse relationships
> between 2 sections. We already have what might be called forward
> links, where one section conveys that it uses another, via the
> sh_link, or sh_info (with SHF_INFO_LINK). Here, you're proposing a
> flag, SHF_ASSOCIATED, that conveys the reverse: A section with
> SHF_ASSOCIATED is saying "I'm needed by some other section".
>
> That might be useful, but generally, we know from the type of
> section whether or not that's true. For instance, we know that
> .eh_frame sections don't stand alone, and that they are always
> paired with some other (text) section. A flag saying so isn't
> adding information. The more useful thing would be for this new
> mechanism to also tell us which section is doing the pointing.
> I can't tell from reading this thread whether that's part of the
> proposal or not.

Yes, for most cases, we do in fact know by the type of section.
Sometimes, however, the section type is simply PROGBITS, and we know
by the section name (e.g., ".eh_frame", ".gcc_except_table",
".debug_info").

In the past, we have added gABI flags to formalize relationships that
had previously been implied by the section type; namely, SHF_INFO_LINK
and SHF_LINK_ORDER. We did this so that additional sections could be
added with similar semantics without having to add hard-coded logic
into each linker. It seems to me that this is another example: we have
existing sections that have this particular quality, and we now have a
new example that perhaps justifies making it an orthogonal flag.

On the other hand, this flag is really kind of a special purpose flag,
whose effect is limited to link-time garbage collection. The
relationships between sections aren't derived from an explicit section
attribute like sh_info or sh_link, but instead by relocations between
sections.

> A lot has been said already, and the proposal has shifted a bit,
> so it might help to do a reset and start fresh. Would you consider
> giving us the diffs for the changes you would want to see in the
> gABI for SHF_ASSOCIATED, as you currently understand it it?

I think all we'd need in the gABI is a new paragraph for
SHF_ASSOCIATED that says something like: "This section contains
metadata associated with the contents of another section. For the
purposes of link-time garbage collection, relocations for this section
that refer to another section should be treated as reverse
dependencies."

I could go for SHF_METADATA as an alternative name for the flag.

-cary

[Suprateeka R Hegde]

On 25-Jan-2017 03:00 AM, Ali Bahrami wrote:
> We're prepared to reconsider, and Cary's implication that we
> should is a strong motivation to do so. The simple truth is that
> we end up having to reverse engineer and adapt to lots of GNUish
> things. Pushing this off to the GNU ABI doesn't mean that we won't
> end up having to figure out how to deal with it when gcc starts
> producing sections that use it.

Thats exactly what I had in mind, when I said I do not have any
conflicts and consider implementing.

On HP-UX also, we have a port of GCC and that uses our HP-UX linker and
runtime. So when GCC starts producing this either as GNU ABI or Generic
ABI, the linker and tools have to support this.

Just because HP-UX has this model, I didnt want to agree on behalf of
others in gABI who dont care what GNU features are.

Since there is no comment from any other participants, we do seem to
have a consensus here.

On 25-Jan-2017 03:34 AM, Cary Coutant wrote:

> On the other hand, this flag is really kind of a special purpose flag,
> whose effect is limited to link-time garbage collection.

That seems to narrow down the claim/benefits. I am not explicit about
this SHF_ASSOCIATED, but the ELF flag space here (9 remaining?) is
really expensive. And these being fancy features of a feature-rich
linker, does not really go inline with the "Format" essence of EL"F".

So lets make a bigger story. Lets not bring in garbage collection as the
primary requirement here. Instead, lets put this as enhancing section
associations in addition to relocs/sh_link.

And then garbage collection (in HP-UX terms, it is dead procedure
elimination) is one of the use cases.

> The
> relationships between sections aren't derived from an explicit section
> attribute like sh_info or sh_link,

Do you mean to say we are not reaping the benefits of existing mechanism
itself completely? And hence try that instead of SHF_ASSOCIATED?

> but instead by relocations between
> sections.

True with respect to Generic ABI. But in a OS specific way, we do have
lot of other algo to establish various relationships between sections.

> I think all we'd need in the gABI is a new paragraph for
> SHF_ASSOCIATED that says something like: "This section contains
> metadata associated with the contents of another section.

Lets not confine it to METADATA. Lets just say association. There are
implementations that splits a section, for say, reordering based on
cache locality. Such implementations may also benefit from associations.

> For the
> purposes of link-time garbage collection, relocations for this section
> that refer to another section should be treated as reverse
> dependencies."

These wordings make it very specific to garbage collection feature only.
I somehow feel that we have to maker this flag more useful and beneficial.

I would say, lets introduce arcs/edges (need not be acyclic) in ELF.
That would be scalable and all dependency relations can be looked-up in
one place instead of relocs+flags+info+etc.

OR

If it just for the sake of GC, lets call it SHF_METADATA instead of
SHF_ASSOCIATED, and use it just for reversing dependencies for sake of
GC. But personally, reiterating, this is a very narrow benefit to spend
a Generic SHF slot in ELF.

That said, I am OK with whatever the consensus is, since, after all it
is a new flag and not a change in sematics of an existing flag.

--
Supra

[Ali Bahrami]

On 01/24/17 15:04, Cary Coutant wrote:
> Sorry, I didn't mean to imply that I think you should reconsider --
> I'm on the fence here, especially since I no longer represent a
> vendor. I only meant to suggest to Rafael that perhaps he hadn't fully
> made his case yet.
>
> (It could be that Rafael doesn't much care one way or the other, and
> I'm the one trying to make a case! So maybe I'm not truly on the
> fence. Hmmm. Let's just say that I'm looking to make sure we do
> consider the idea fairly.)

Either way, I didn't fully understand what was being proposed,
and you nudged me to give it another look, so that's good.

> I think all we'd need in the gABI is a new paragraph for
> SHF_ASSOCIATED that says something like: "This section contains
> metadata associated with the contents of another section. For the
> purposes of link-time garbage collection, relocations for this section
> that refer to another section should be treated as reverse
> dependencies."
>
> I could go for SHF_METADATA as an alternative name for the flag.
>

Thanks, that helps, and I think I understand now. The thing that
I was missing before is the relocations.

Here's a concrete example:

% cc -m64 hello.c -c
% elfdump -r hello.o
...
Relocation Section: .rela.eh_frame
index type offset value addend section symbol
[0] R_AMD64_64 0x28 0 0x10 .eh_frame .text (section)

This eh_frame section has a relocation to .text, but
nothing explicitly depends on the eh_frame itself. Naive
unused processing would throw the eh_frame away, even though
it is necessary. Today, we have special code in ld that prevents
this from happening.

What's being proposed is that if eh_frame had SHF_ASSOCIATED set,
that we'd add a link to our graph saying that .text depends on
the eh_frame, and that would eliminate the need for the special
case code.

I think that makes sense, and I also think it might protect us
from being broken by future GNU changes that use it. I also see
why Cary likes SHF_METADATA better than SHF_ASSOCIATED, since
sections like eh_frame really are extra information related
directly to the section they buddy with. Section association
to me is more along the lines of how sections point at each
other via sh_link (and sometimes sh_info), or COMDAT groups.

Another way to look at it is that if 2 names seem to be equally
good, then the shorter one ought to win. Given that, maybe it
could be reduced to just SHF_META.

If folks come to the agreement that such a flag is valuable enough to
have, then we (Solaris) are OK with having it be in the generic range.
I guess the biggest concern is exhaustion of the flag bitspace,
something I hadn't given much thought to before.

Speaking of the available flags, I have a couple of questions:

- Does anyone recall the history behind the current gap in
the generic SHF_ assignments? Why did we skip over 0x00000008?

- Does anyone understand SHF_OS_NONCONFORMING and how it
might be actually used? Given that ELF has OS-defined ranges
for nearly everything, I'd really expect "nonconforming"
things to end up buried in that vendor's personal space,
so I wonder what purpose SHF_OS_NONCONFORMING serves.
Could we reclaim that bit for a different use, like SHF_META?

- Ali

[Rafael Avila de Espindola]

Ali Bahrami <Ali.B...@Oracle.COM> writes:


> A lot has been said already, and the proposal has shifted a bit,
> so it might help to do a reset and start fresh. Would you consider
> giving us the diffs for the changes you would want to see in the
> gABI for SHF_ASSOCIATED, as you currently understand it it?

Sure. Where is the source of of the gABI stored?

Thanks,
Rafael

[Rafael Avila de Espindola]

Cary Coutant <ccou...@gmail.com> writes:

>> We're prepared to reconsider, and Cary's implication that we
>> should is a strong motivation to do so. The simple truth is that
>> we end up having to reverse engineer and adapt to lots of GNUish
>> things. Pushing this off to the GNU ABI doesn't mean that we won't
>> end up having to figure out how to deal with it when gcc starts
>> producing sections that use it. So let's continue the discussion a bit...
>
> Sorry, I didn't mean to imply that I think you should reconsider --
> I'm on the fence here, especially since I no longer represent a
> vendor. I only meant to suggest to Rafael that perhaps he hadn't fully
> made his case yet.
>
> (It could be that Rafael doesn't much care one way or the other, and
> I'm the one trying to make a case! So maybe I'm not truly on the
> fence. Hmmm. Let's just say that I'm looking to make sure we do
> consider the idea fairly.)

Sorry for being slow to reply, but I would also prefer to have a flag in
the gABI if possible. I do think that supporting metadata like the one
asan uses is generally useful.

Cheers,
Rafael

[Ali Bahrami]

As Supra aluded to earlier, the source was last in the hands of
SCO (aka xinuos), but it's maintainer (John Wolfe) left the company,
and none of us are really sure where the source is or if anyone is
is actively maintaining it. The hardest part about updating the gABI
at this moment is knowing who is taking care of it.

I'm not finding a current copy via google at the moment, which
is a problem. The Solaris Linker and Libraries Guild is based
on the same original AT&T document, and as we've been maintaining
it consistently over the years, should contain all the same information
(plus ELFOSABI_SOLARIS additions). I understand that it is not
an adequate community replacement, but it is a stable source for
the same information.

So I don't really know. However, if you agree with Cary's
recent message as to how it would be defined, then I consider
my immediate question to be answered.

- Ali

[Suprateeka R Hegde]

On 26-Jan-2017 11:01 PM, Ali Bahrami wrote:
> On 01/26/17 08:49, Rafael Avila de Espindola wrote:
>> Ali Bahrami <Ali.B...@Oracle.COM> writes:
>>
>>
>>> A lot has been said already, and the proposal has shifted a bit,
>>> so it might help to do a reset and start fresh. Would you consider
>>> giving us the diffs for the changes you would want to see in the
>>> gABI for SHF_ASSOCIATED, as you currently understand it it?
>>
>> Sure. Where is the source of of the gABI stored?
>>
>> Thanks,
>> Rafael
>>
>
> As Supra aluded to earlier, the source was last in the hands of
> SCO (aka xinuos), but it's maintainer (John Wolfe) left the company,
> and none of us are really sure where the source is or if anyone is
> is actively maintaining it. The hardest part about updating the gABI
> at this moment is knowing who is taking care of it.
>
> I'm not finding a current copy via google at the moment, which
> is a problem. The Solaris Linker and Libraries Guild is based
> on the same original AT&T document, and as we've been maintaining
> it consistently over the years, should contain all the same information
> (plus ELFOSABI_SOLARIS additions). I understand that it is not
> an adequate community replacement, but it is a stable source for
> the same information.

That is the same case in here too (on HP-UX). May be for now, Ali can
add the diff to the Solaris base. I need sometime to sync up as we have
not updated our copy of ELF since almost 3 years.

And anyway, this google group is there for reference when the official
SCO one gets to life again.

> I also see
> why Cary likes SHF_METADATA better than SHF_ASSOCIATED, since
> sections like eh_frame really are extra information related
> directly to the section they buddy with.

Thats exactly what I said. If it is just for the sake of GC and
specially for sections like debug/unwind, then METADATA is a better word.

> If folks come to the agreement that such a flag is valuable enough to
> have, then we (Solaris) are OK with having it be in the generic range.

I am OK.

> I guess the biggest concern is exhaustion of the flag bitspace,
> something I hadn't given much thought to before.

That is the reason why I said, lets make it a bigger story and a more
useful flag than just for reversing dependency (for procedure elim). And
then we can have SHF_ASSOCIATED.

But this looks like something of nobody's interest and we may have to
settle down with just a narrow set of use case.

> - Does anyone recall the history behind the current gap in
> the generic SHF_ assignments? Why did we skip over 0x00000008?

In our copy, it says "reserved", but also says "...deprecated value for
SHF_HP_COMDAT". And thats true, as SHF_HP_COMDAT. I am not able to dig
further in the history, but looks like something HP wanted to claim, but
later didnt, and by that time further bits were assigned and this
0x00000008 came back to reserved pool.

(Just my guess)

>
> - Does anyone understand SHF_OS_NONCONFORMING and how it
> might be actually used?

Looks like a helper flag to inform linker that the section with this
flag is not conforming to any OS (Not even OSABI_NONE) and the linker
need not spend time processing other conforming flags even if they are set.

Again just my guess based on a comment I have in our ELF copy. I cant
think of any other semantics.

> Given that ELF has OS-defined ranges
> for nearly everything, I'd really expect "nonconforming"
> things to end up buried in that vendor's personal space,
> so I wonder what purpose SHF_OS_NONCONFORMING serves.
> Could we reclaim that bit for a different use, like SHF_META?

Thats something interesting. Thats a good idea, but how do we "really"
broadcast this?

--
Supra

[Ali Bahrami]

On 01/26/17 11:12, Suprateeka R Hegde wrote:
> Looks like a helper flag to inform linker that the section with this flag is not conforming to any OS (Not even OSABI_NONE) and the linker need not spend time processing other conforming flags even if they are set.
>
> Again just my guess based on a comment I have in our ELF copy. I cant think of any other semantics.
>
>> Given that ELF has OS-defined ranges
>> for nearly everything, I'd really expect "nonconforming"
>> things to end up buried in that vendor's personal space,
>> so I wonder what purpose SHF_OS_NONCONFORMING serves.
>> Could we reclaim that bit for a different use, like SHF_META?
>
> Thats something interesting. Thats a good idea, but how do we "really" broadcast this?

NONCONFORMING seems incomplete and a bit worthless. If that's
so, and there is no use of it in the world, then I think we'd
all just agree to change the gABI.

We should probably not hold up this SHF_(ASSOCIATED|METADATA|META)
discussion over it though.

- Ali

[Suprateeka R Hegde]

On 26-Jan-2017 11:50 PM, Ali Bahrami wrote:
> On 01/26/17 11:12, Suprateeka R Hegde wrote:
>> Looks like a helper flag to inform linker that the section with this
>> flag is not conforming to any OS (Not even OSABI_NONE) and the linker
>> need not spend time processing other conforming flags even if they are
>> set.
>>
>> Again just my guess based on a comment I have in our ELF copy. I cant
>> think of any other semantics.
>>
>>> Given that ELF has OS-defined ranges
>>> for nearly everything, I'd really expect "nonconforming"
>>> things to end up buried in that vendor's personal space,
>>> so I wonder what purpose SHF_OS_NONCONFORMING serves.
>>> Could we reclaim that bit for a different use, like SHF_META?
>>
>> Thats something interesting. Thats a good idea, but how do we "really"
>> broadcast this?
>
>
> NONCONFORMING seems incomplete and a bit worthless.

Ah! A nice statement. I like the pun on the word "bit" :-)

If that's
> so, and there is no use of it in the world, then I think we'd
> all just agree to change the gABI.

OK. (Lets wait for a day for any comments?)

>
> We should probably not hold up this SHF_(ASSOCIATED|METADATA|META)
> discussion over it though.

Agree.

--
Supra

[Rafael Avila de Espindola]

Ali Bahrami <Ali.B...@Oracle.COM> writes:
> So I don't really know. However, if you agree with Cary's
> recent message as to how it would be defined, then I consider
> my immediate question to be answered.

I agree with it and would have a small preference for SHF_METADATA.

Thanks,
Rafael

[Cary Coutant]

> As Supra aluded to earlier, the source was last in the hands of
> SCO (aka xinuos), but it's maintainer (John Wolfe) left the company,
> and none of us are really sure where the source is or if anyone is
> is actively maintaining it. The hardest part about updating the gABI
> at this moment is knowing who is taking care of it.
>
> I'm not finding a current copy via google at the moment, which
> is a problem. The Solaris Linker and Libraries Guild is based
> on the same original AT&T document, and as we've been maintaining
> it consistently over the years, should contain all the same information
> (plus ELFOSABI_SOLARIS additions). I understand that it is not
> an adequate community replacement, but it is a stable source for
> the same information.

The current copy is still here:

http://www.sco.com/developers/gabi/latest/contents.html

Given Xinous' seeming disinterest in owning this, I'm thinking it
might be time to try to form a non-profit organization to own this
spec. If we can't obtain the IP from Xinous, we might be able to use
the HP documentation (which I wrote and is free of any SCO IP). I had
obtained approval to put an MIT-like open-source license on the
documents, but left HP before I got around to doing that (sorry!).
Perhaps Suprateeka would be willing to pursue that avenue if
necessary.

I'm willing to check into the legal aspects and costs of setting up a
non-profit organization, and I'm hoping we could do this with some
minimal contribution from member companies to cover legal fees and
ongoing website hosting.

-cary

[Rafael Avila de Espindola]

Cary Coutant <ccou...@gmail.com> writes:

> The current copy is still here:
>
> http://www.sco.com/developers/gabi/latest/contents.html
>
> Given Xinous' seeming disinterest in owning this, I'm thinking it
> might be time to try to form a non-profit organization to own this
> spec. If we can't obtain the IP from Xinous, we might be able to use
> the HP documentation (which I wrote and is free of any SCO IP). I had
> obtained approval to put an MIT-like open-source license on the
> documents, but left HP before I got around to doing that (sorry!).
> Perhaps Suprateeka would be willing to pursue that avenue if
> necessary.
>
> I'm willing to check into the legal aspects and costs of setting up a
> non-profit organization, and I'm hoping we could do this with some
> minimal contribution from member companies to cover legal fees and
> ongoing website hosting.

That would be awesome. Thanks for taking the initiative.

Cheers,
Rafael

[Joseph Myers]

On Thu, 26 Jan 2017, Cary Coutant wrote:

> The current copy is still here:
>
> http://www.sco.com/developers/gabi/latest/contents.html

e_machine values have been assigned that do not appear on that list.

https://lists.iovisor.org/pipermail/iovisor-dev/2016-June/000266.html

(which gives Michael.Brinke-Engel <mikebren at xinuos.com> as the person
who responded to that e_machine request).

--
Joseph S. Myers
jos...@codesourcery.com

[Cary Coutant]

>> So I don't really know. However, if you agree with Cary's
>> recent message as to how it would be defined, then I consider
>> my immediate question to be answered.
>
> I agree with it and would have a small preference for SHF_METADATA.

I've thought about this a bit more, and had a conversation with Jim
Dehnert yesterday about it. I think the intended use for the
SHF_LINK_ORDER flag is still very, very, close to what we're talking
about, and I think we should simply rename it rather than define a new
flag. I don't think the flag is in widespread use, and the change
we're proposing wouldn't actually invalidate any existing treatment of
it.

Here's the current definition:

--------------------------
SHF_LINK_ORDER

This flag adds special ordering requirements for link editors. The
requirements apply if the sh_link field of this section's header
references another section (the linked-to section). If this section is
combined with other sections in the output file, it must appear in the
same relative order with respect to those sections, as the linked-to
section appears with respect to sections the linked-to section is
combined with.

NOTE: A typical use of this flag is to build a table that references
text or data sections in address order.
--------------------------

As written, it doesn't actually apply to any PROGBITS sections, since
those sections do not use the sh_link field, and therefore the
requirements do not apply. It was originally written to formalize the
rule for SHT_IA_64_UNWIND sections, so that linkers could process them
properly without understanding the section type (and thus eliminating
the need to set the NON_CONFORMING flag on the unwind sections). I
would also shift the requirement of maintaining relative section
ordering to the psABI.

Here's what I propose as a replacement for SHF_LINK_ORDER:

--------------------------
SHF_METADATA

This flag indicates that the section contains metadata describing
another section, identified by the sh_link field of this section's
header. If the referenced section is discarded by the linker for any
reason (e.g., garbage collection of unreachable sections), this
section should also be discarded. Furthermore, any relocations from
this section into the referenced section should not be used to
establish reachability of the referenced section. A psABI may also
require that the metadata sections remain in the same relative order
as the referenced sections in the event that a linker performs any
section reordering.

This flag may not be used for a section type where the sh_link field
is already defined for a different purpose (see Table 4.14).

NOTE: Metadata sections typically would contain unwind information,
exception handling information, or some other auxiliary information
about the contents of other sections. In these cases, the referenced
sections do not contain any direct references to the metadata section;
rather, the metadata sections may contain pointers to address ranges
in the referenced sections. The metadata sections are usually
aggregated together and reachable by "start" and "stop" bracketing
symbols added either by the linker or by startup object files. This
may defeat link-time garbage collection by failing to identify that
the metadata sections are reachable, or, if the linker treats them as
"root" sections, may force the referenced sections to be considered
reachable even if there are no other references.
--------------------------

It would probably also be a good idea to add a row for SHT_PROGBITS to
Table 4.14:

--------------------------
sh_type | sh_link | sh_info
SHT_PROGBITS | If SHF_METADATA is set, the section header index of the
section to which the metadata applies | 0
--------------------------

-cary

[Rafael Avila de Espindola]

Cary Coutant <ccou...@gmail.com> writes:

> Here's what I propose as a replacement for SHF_LINK_ORDER:
>
> --------------------------
> SHF_METADATA
>
> This flag indicates that the section contains metadata describing
> another section, identified by the sh_link field of this section's
> header. If the referenced section is discarded by the linker for any
> reason (e.g., garbage collection of unreachable sections), this
> section should also be discarded. Furthermore, any relocations from
> this section into the referenced section should not be used to
> establish reachability of the referenced section.

But relocation to other sections, would, correct? The case I have in
mind in an hypothetical split .eh_frame. The relocations to the
corresponding .text should not keep the .text alive, but the relocation
to the personality function should keep the personality function alive.

I like the idea. Since .sh_link would be set bfd would not warn on
it. It also solves the ambiguity of which relocations should point in
the opposite direction. Now it is clear that only the relocations along
the sh_link edge are inverted.

I will try prototyping this too. BTW, have you put any thought on what
the assembly would look like? Maybe something like

.section .meta,"moG",@progbits,group,comdat,metadata,local_symbol

where local_symbol is used to find the section that will be in the
sh_link. The symbol can be just a section name (".text"), but is defined
as a symbol to handle the case of multiple sections with the same name.

Cheers,
Rafael

[Cary Coutant]

> But relocation to other sections, would, correct? The case I have in
> mind in an hypothetical split .eh_frame. The relocations to the
> corresponding .text should not keep the .text alive, but the relocation
> to the personality function should keep the personality function alive.

Yes, I deliberately worded it so as not to exclude such relocations.

> I like the idea. Since .sh_link would be set bfd would not warn on
> it. It also solves the ambiguity of which relocations should point in
> the opposite direction. Now it is clear that only the relocations along
> the sh_link edge are inverted.

We don't even need to invert those edges, since the sh_link field
already explicitly defines the reachability of the metadata section
based on the associated section. All the proposal needs to say now is
that those relocations can be discounted for the purposes of
establishing reachability.

> I will try prototyping this too. BTW, have you put any thought on what
> the assembly would look like? Maybe something like
>
> .section .meta,"moG",@progbits,group,comdat,metadata,local_symbol
>
> where local_symbol is used to find the section that will be in the
> sh_link. The symbol can be just a section name (".text"), but is defined
> as a symbol to handle the case of multiple sections with the same name.

LGTM. Not sure I understand which tokens there are literal -- what are
"comdat" and "metadata" for?

-cary

[Evgenii Stepanov]

On Thu, Jan 26, 2017 at 12:26 PM, Cary Coutant <ccou...@gmail.com> wrote:
>> But relocation to other sections, would, correct? The case I have in
>> mind in an hypothetical split .eh_frame. The relocations to the
>> corresponding .text should not keep the .text alive, but the relocation
>> to the personality function should keep the personality function alive.
>
> Yes, I deliberately worded it so as not to exclude such relocations.
>
>> I like the idea. Since .sh_link would be set bfd would not warn on
>> it. It also solves the ambiguity of which relocations should point in
>> the opposite direction. Now it is clear that only the relocations along
>> the sh_link edge are inverted.
>
> We don't even need to invert those edges, since the sh_link field
> already explicitly defines the reachability of the metadata section
> based on the associated section. All the proposal needs to say now is
> that those relocations can be discounted for the purposes of
> establishing reachability.

Could this not talk about relocations at all, and instead say that
start/stop symbols should not prevent discarding of SHF_LINK_ORDER
sections? That way, relocations from metadata into the sh_link section
would not affect it's liveness (assuming that nothing else uses the
metadata section).

>
>> I will try prototyping this too. BTW, have you put any thought on what
>> the assembly would look like? Maybe something like
>>
>> .section .meta,"moG",@progbits,group,comdat,metadata,local_symbol
>>
>> where local_symbol is used to find the section that will be in the
>> sh_link. The symbol can be just a section name (".text"), but is defined
>> as a symbol to handle the case of multiple sections with the same name.
>
> LGTM. Not sure I understand which tokens there are literal -- what are
> "comdat" and "metadata" for?
>
> -cary
>

> --
> You received this message because you are subscribed to the Google Groups "Generic System V Application Binary Interface" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to generic-abi...@googlegroups.com.
> To post to this group, send email to gener...@googlegroups.com.
> Visit this group at https://groups.google.com/group/generic-abi.
> For more options, visit https://groups.google.com/d/optout.

[Rafael Avila de Espindola]

Cary Coutant <ccou...@gmail.com> writes:

>> I will try prototyping this too. BTW, have you put any thought on what
>> the assembly would look like? Maybe something like
>>
>> .section .meta,"moG",@progbits,group,comdat,metadata,local_symbol
>>
>> where local_symbol is used to find the section that will be in the
>> sh_link. The symbol can be just a section name (".text"), but is defined
>> as a symbol to handle the case of multiple sections with the same name.
>
> LGTM. Not sure I understand which tokens there are literal -- what are
> "comdat" and "metadata" for?

"comdat" and "metadata" are literal. comdat already exists, and is just
to say that this is part of a comdat (the only grouping available).

"metadata" just marks the next thing as being the symbol that will
determine which section will be in the sh_info.

Cheers,
Rafael

[Reid Kleckner]

On Thu, Jan 26, 2017 at 11:18 AM, Cary Coutant <ccou...@gmail.com> wrote:

>> So I don't really know. However, if you agree with Cary's
>> recent message as to how it would be defined, then I consider
>> my immediate question to be answered.
>
> I agree with it and would have a small preference for SHF_METADATA.

I've thought about this a bit more, and had a conversation with Jim
Dehnert yesterday about it. I think the intended use for the
SHF_LINK_ORDER flag is still very, very, close to what we're talking
about, and I think we should simply rename it rather than define a new
flag. I don't think the flag is in widespread use, and the change
we're proposing wouldn't actually invalidate any existing treatment of
it.

+1, I like the way this uses sh_link. Thanks for proposing this!

[Suprateeka R Hegde]

On 27-Jan-2017 12:20 AM, Cary Coutant wrote:
>> I'm not finding a current copy via google at the moment, which
>> is a problem. The Solaris Linker and Libraries Guild is based
>> on the same original AT&T document, and as we've been maintaining
>> it consistently over the years, should contain all the same information
>> (plus ELFOSABI_SOLARIS additions). I understand that it is not
>> an adequate community replacement, but it is a stable source for
>> the same information.
>
> The current copy is still here:
>
> http://www.sco.com/developers/gabi/latest/contents.html
>
> Given Xinous' seeming disinterest in owning this, I'm thinking it
> might be time to try to form a non-profit organization to own this
> spec. If we can't obtain the IP from Xinous, we might be able to use
> the HP documentation (which I wrote and is free of any SCO IP). I had
> obtained approval to put an MIT-like open-source license on the
> documents, but left HP before I got around to doing that (sorry!).
> Perhaps Suprateeka would be willing to pursue that avenue if
> necessary.

I can do that. I shall reply back on this and the timeline by Next
Wednesday. I am on a vacation till then.

>
> I'm willing to check into the legal aspects and costs of setting up a
> non-profit organization, and I'm hoping we could do this with some
> minimal contribution from member companies to cover legal fees and
> ongoing website hosting.

Makes sense, but cant confirm right now. Need to work with org and
confirm. But looks like feasible.

--
Supra

[Ali Bahrami]

On 01/26/17 12:18, Cary Coutant wrote:
> I've thought about this a bit more, and had a conversation with Jim
> Dehnert yesterday about it. I think the intended use for the
> SHF_LINK_ORDER flag is still very, very, close to what we're talking
> about, and I think we should simply rename it rather than define a new
> flag. I don't think the flag is in widespread use, and the change
> we're proposing wouldn't actually invalidate any existing treatment of
> it.

...

>
> As written, it doesn't actually apply to any PROGBITS sections, since
> those sections do not use the sh_link field, and therefore the
> requirements do not apply. It was originally written to formalize the
> rule for SHT_IA_64_UNWIND sections, so that linkers could process them
> properly without understanding the section type (and thus eliminating
> the need to set the NON_CONFORMING flag on the unwind sections). I
> would also shift the requirement of maintaining relative section
> ordering to the psABI.
>
> Here's what I propose as a replacement for SHF_LINK_ORDER:
>
> --------------------------
> SHF_METADATA
>
> This flag indicates that the section contains metadata describing
> another section, identified by the sh_link field of this section's
> header. If the referenced section is discarded by the linker for any
> reason (e.g., garbage collection of unreachable sections), this
> section should also be discarded. Furthermore, any relocations from
> this section into the referenced section should not be used to
> establish reachability of the referenced section. A psABI may also
> require that the metadata sections remain in the same relative order
> as the referenced sections in the event that a linker performs any
> section reordering.
>
> This flag may not be used for a section type where the sh_link field
> is already defined for a different purpose (see Table 4.14).
>

I'm concerned about the idea that the ordering implied by SHF_LINK_ORDER
becomes an optional psABI matter. The Solaris Studio compilers use
SHF_LINK_ORDER for that effect on sparc and on x86, for purposes
that don't have much to do with platform details.

The part that I like, other than the effort to preserve flag bits, is
making the meta connection to the section explicit via sh_link. I also
do agree that in the cases I know about, the ordering and meta behaviors
are compatible (the sections are sorted, *and* meta, to the same parent
section).

One way to resolve this would be to take this latest proposal,
but to keep the requirement for sorting in the gABI. In other words,
to overload the existing SHF_LINK_ORDER, by adding the "meta"
meaning. That would preclude unsorted meta sections. I'm not sure
if that's a problem.

Another way is to add a new flag for meta, and leave LINK_ORDER
alone. In that case, we could say that when META and LINK_ORDER are
both set, that the sh_link applies to both flags.

Cary, I asked earlier about NONCONFORMING --- where did it come from,
and has anything ever used it. I think you're saying above that it came
with IA-64. In that case, if it's never been used elsewhere, and if
there's no expected use for it, can we release that flag bit for other use?

In the Solaris sources, there are no uses of SHF_OS_NONCONFORMING at
all, so we wouldn't know what to do if we encountered it. It's in
the <sys/elf.h> header, and our display code recognizes and prints it,
but that's all.

- Ali

[Suprateeka R Hegde]

I didnt understand why we want to remove "ordering" from gABI. May be I
am missing something here.

How about the inverse?
"A psABI may relax the ordering requirement and hence metadata sections
*need not* remain in the same relative order as the referenced sections

in the event that a linker performs any section reordering"

> The Solaris Studio compilers use
> SHF_LINK_ORDER for that effect on sparc and on x86, for purposes
> that don't have much to do with platform details.
> The part that I like, other than the effort to preserve flag bits, is
> making the meta connection to the section explicit via sh_link. I also
> do agree that in the cases I know about, the ordering and meta behaviors
> are compatible (the sections are sorted, *and* meta, to the same parent
> section).
>
> One way to resolve this would be to take this latest proposal,
> but to keep the requirement for sorting in the gABI. In other words,
> to overload the existing SHF_LINK_ORDER, by adding the "meta"
> meaning. That would preclude unsorted meta sections. I'm not sure
> if that's a problem.

Overloading LINK_ORDER with META, in my opinion, can be termed in a
better way as "ASSOCIATED". To mean that section is associated with
another in possibly more than one way (currently, either as a META data
holder or in a ordering relationship or both)

But will there ever be a possibility that such a Section-A
(SHF_ASSOCIATED) contains META data related to Section-B, but ordering
relationship relative to Section-C? In that case we have only one
sh_link. I cant think of any such situations.

>
> Another way is to add a new flag for meta, and leave LINK_ORDER
> alone. In that case, we could say that when META and LINK_ORDER are
> both set, that the sh_link applies to both flags.
>
> Cary, I asked earlier about NONCONFORMING --- where did it come from,
> and has anything ever used it. I think you're saying above that it came
> with IA-64.

I am not sure if Cary meant NONCONFORMING came with IA-64.

Had the LINK_ORDER not been there, it would require linkers to
special-case-understand vendor specific sh_type (Eg.: SHT_IA_64_UNWIND)
and process it as per the psABI. And the ELF way to tell linkers to do
such special things, is to set SHF_OS_NONCONFORMING.

Since LINK_ORDER was introduced to handle SHT_IA_64_UNWIND, the use of
SHF_OS_NONCONFORMING (as described above) is eliminated.

So what came with IA-64 is only the LINK_ORDER.

Thats what I understand from what Cary wrote.

> In that case, if it's never been used elsewhere, and if
> there's no expected use for it, can we release that flag bit for other use?
>
> In the Solaris sources, there are no uses of SHF_OS_NONCONFORMING at
> all, so we wouldn't know what to do if we encountered it. It's in
> the <sys/elf.h> header, and our display code recognizes and prints it,
> but that's all.

Thats the same case on HP-UX too. It is not used anywhere except to
recognize and print.

--
Supra

[Ali Bahrami]

Hi Supra,

On 02/ 3/17 03:31 AM, Suprateeka R Hegde wrote:
> I didnt understand why we want to remove "ordering" from gABI. May be I am missing something here.

In case your "missing something" comment is directed at me,
wondering where I got that idea, the proposal states:

On 01/26/17 12:18 PM, Cary Coutant wrote:
> I
> would also shift the requirement of maintaining relative section
> ordering to the psABI.

This is a concern because I have some existing practice that depends
on the ordering across all platforms. In my case, it's more of an OSABI
matter than a psABI one, and I wouldn't want the psABI to get in the way.

I'm sure Cary knows that, so I'm wondering what was really
being proposed here.

>
> How about the inverse?
> "A psABI may relax the ordering requirement and hence metadata sections *need not* remain in the same relative order as the referenced sections in the event that a linker performs any section reordering"

I don't have a concern with the idea that the psABI, or the OSABI,
might relax the ordering requirement, as long as the details of that
exception were so tightly specified that no one could possibly be
confused. However, I wonder if this needs to be stated in the gABI,
as the OSABI and gABI can always add additional rules for anything
they want, within reason.

>
> But will there ever be a possibility that such a Section-A (SHF_ASSOCIATED) contains META data related to Section-B, but ordering relationship relative to Section-C? In that case we have only one sh_link. I cant think of any such situations.

I don't know, but it hasn't happened yet, and we've been making
objects for a long time. I'd rather not complicate things to
support the fully general case of N things depending on each other
in special ways. Let's wait until we have a real problem to solve
with it.

>> Cary, I asked earlier about NONCONFORMING --- where did it come from,
>> and has anything ever used it. I think you're saying above that it came
>> with IA-64.
>
> I am not sure if Cary meant NONCONFORMING came with IA-64.
>
> Had the LINK_ORDER not been there, it would require linkers to special-case-understand vendor specific sh_type (Eg.: SHT_IA_64_UNWIND) and process it as per the psABI. And the ELF way to tell linkers to do such special things, is to set SHF_OS_NONCONFORMING.
>
> Since LINK_ORDER was introduced to handle SHT_IA_64_UNWIND, the use of SHF_OS_NONCONFORMING (as described above) is eliminated.
>
> So what came with IA-64 is only the LINK_ORDER.
>
> Thats what I understand from what Cary wrote.
>

You're probably right.

The use of NONCONFORMING, to tell link-editors that they don't understand
something, doesn't seem useful to me. They already know that, and the
flag doesn't help them produce a better outcome than the one they'll
produce without it.

My real question though is, does SHF_OS_NONCONFORMING have any
actual purpose and/or existing use, and if not, can we get rid of it?
And I ask this, because much of this conversation has been along
the lines of "I hate to waste a precious flag bit on META", and
so, perhaps we should reclaim this, for use as META, or otherwise.

- Ali

[Rafael Espíndola]

> But relocation to other sections, would, correct? The case I have in
> mind in an hypothetical split .eh_frame. The relocations to the
> corresponding .text should not keep the .text alive, but the relocation
> to the personality function should keep the personality function alive.
>
> I like the idea. Since .sh_link would be set bfd would not warn on
> it. It also solves the ambiguity of which relocations should point in
> the opposite direction. Now it is clear that only the relocations along
> the sh_link edge are inverted.
>
> I will try prototyping this too. BTW, have you put any thought on what
> the assembly would look like? Maybe something like
>
> .section .meta,"moG",@progbits,group,comdat,metadata,local_symbol
>
> where local_symbol is used to find the section that will be in the
> sh_link. The symbol can be just a section name (".text"), but is defined
> as a symbol to handle the case of multiple sections with the same name.

Sorry for the long delay, but I finally have a prototype. It works
really nice and there is no ambiguity about which edges are reversed.

Since other than the flag name change this is just an extension of
existing practices, I will probably propose a similar change llvm/lld
using the SHF_LINK_ORDER name while the spec discussion happens is
parallel.

Cheers,
Rafael

[Ali Bahrami]

This discussion seems to have died away without anything
definite having been decided. I asked some questions at the
end of the thread that didn't get much of a response. I'm not
sure where matters stand, and I hesitate to draw any conclusions.
So, let me tell you where we stand.

I think that the most recent proposal was to rename SHF_LINK_ORDER
to SHF_METADATA, and to remove the sorting requirement from the
gABI, leaving it to the psabi to specify whether or not sorting
should also be done:

A psABI may also
require that the metadata sections remain in the same relative order

as the referenced sections in the event that a linker performs any

section reordering.

That's a problem for us. The Solaris Studio compilers have been
generating SHF_LINK_ORDER specifically for the sorting behavior,
almost from the moment the gABI created it (15 years?), on both
sparc and x86. The sole meaning of SHF_LINK_ORDER over that time has
been to get sorting. We have lots of existing objects in the field
with this flag. For example:

Section Header[14]: sh_name: .exception_ranges
sh_addr: 0 sh_flags: [ SHF_WRITE SHF_ALLOC SHF_LINK_ORDER ]
sh_size: 0x50 sh_type: [ SHT_PROGBITS ]
sh_offset: 0x6d0 sh_entsize: 0
sh_link: 3 sh_info: 0
sh_addralign: 0x4

This is a PROGBITS (not a psABI defined type) section that needs
sorting. One could argue that this is a Solaris only thing that the
OSABI can define rules for, but we prefer for things like that to
have section types in the OSABI range, and in this case, that ship
has sailed.

We'd therefore like SHF_LINK_ORDER to retain it's name, and the
current gABI meaning. However, we don't see a problem with overloading
it to also encompass the meta data meaning. In the existing uses
of SHF_LINK_INFO that I know about, the sections being ordered are
also meta data for the sections they use as sort keys, as Cary
has pointed out previously, and it wouldn't hurt our existing uses
for it to mean both things.

The sole drawback to that is that meta data sections that don't
want sorting would get it anyway. If that's a problem, then we'd
we'd also be OK with creating a separate SHF_METADATA flag. If
LINK_ORDER and METADATA are both set for a single section, then the
sh_link would apply to both of them, but they can be set independently
to get sorting-only, or metadata-only.

If there is to be a new SHF_METADATA flag, then I'd vote to
remove SHF_OS_NONCONFORMING and reuse that bit value, since
no one has given an example of how it is actually used, or of
why defining an OSABI specific section type of flag wouldn't
be the better answer to something like that. Is there any existing
practice?

I'm happy to drop this here, as the status quo works for us,
but also happy to continue the discussion if there's interest.

Thanks.

- Ali

[Rafael Avila de Espindola]

I don't know of any case that requires the metadata to not be sorted, so
I am OK with keeping that requirement.

Cheers,
Rafael

[Cary Coutant]

>> This is a PROGBITS (not a psABI defined type) section that needs
>> sorting. One could argue that this is a Solaris only thing that the
>> OSABI can define rules for, but we prefer for things like that to
>> have section types in the OSABI range, and in this case, that ship
>> has sailed.
>>
>> We'd therefore like SHF_LINK_ORDER to retain it's name, and the
>> current gABI meaning. However, we don't see a problem with overloading
>> it to also encompass the meta data meaning. In the existing uses
>> of SHF_LINK_INFO that I know about, the sections being ordered are
>> also meta data for the sections they use as sort keys, as Cary
>> has pointed out previously, and it wouldn't hurt our existing uses
>> for it to mean both things.
>
> I don't know of any case that requires the metadata to not be sorted, so
> I am OK with keeping that requirement.

Well, it sounds like everyone's not only OK with retaining the sorting
requirement, but in favor of it. I proposed dropping it down to the
psABI level only because I thought it might not be so acceptable.

I think I'd still like to rename the flag, but all existing behavior
associated with the flag would remain, so I don't see that as a
compatibility issue.

So I'd revise my proposal by simply dropping the "psABI may require"
bit. Revised proposal below.

-cary

--------------------------
SHF_METADATA

This flag indicates that the section contains metadata describing
another section, identified by the sh_link field of this section's
header. If the referenced section is discarded by the linker for any
reason (e.g., garbage collection of unreachable sections), this
section should also be discarded. Furthermore, any relocations from
this section into the referenced section should not be used to
establish reachability of the referenced section.

In the event that a linker performs any section reordering, the
metadata sections must remain in the same relative order
as the referenced sections.

This flag may not be used for a section type where the sh_link field
is already defined for a different purpose (see Table 4.14).

NOTE: Metadata sections typically would contain unwind information,
exception handling information, or some other auxiliary information
about the contents of other sections. In these cases, the referenced
sections do not contain any direct references to the metadata section;
rather, the metadata sections may contain pointers to address ranges
in the referenced sections. The metadata sections are usually
aggregated together and reachable by "start" and "stop" bracketing
symbols added either by the linker or by startup object files. This
may defeat link-time garbage collection by failing to identify that
the metadata sections are reachable, or, if the linker treats them as
"root" sections, may force the referenced sections to be considered
reachable even if there are no other references.
--------------------------

Add a row for SHT_PROGBITS to Table 4.14:

[Ali Bahrami]

On 03/01/17 16:52, Cary Coutant wrote:
> Well, it sounds like everyone's not only OK with retaining the sorting
> requirement, but in favor of it. I proposed dropping it down to the
> psABI level only because I thought it might not be so acceptable.
>
> I think I'd still like to rename the flag, but all existing behavior
> associated with the flag would remain, so I don't see that as a
> compatibility issue.
>
> So I'd revise my proposal by simply dropping the "psABI may require"
> bit. Revised proposal below.

I think we're fine with this.

I don't want to prolong this discussion, but we have many years
of SHF_LINK_ORDER use, and our assemblers have #link_order
directives, so we really can't completely drop the name. If it were
to be renamed, we'd have to maintain both names in our headers
as aliases in our headers and documentation. That's easy enough, but
it's not that much of a win relative to the churn.

We'd be happy to leave the name as is, but are prepared to deal with
it if the consensus is to rename.

---

How about SHF_OS_NONCONFORMING. Can we nuke that?

- Ali

[Suprateeka R Hegde]

I also would like to keep the LINK_ORDER name as is and may be rename
OS_NON_CONFORMING to METADATA. There are some ugly legacy scripts
relying on the string "LINK_ORDER" output by our elfdump.

However, if we really want to rename LINK_ORDER, I am OK to handle these
problems we have under some compatibility options.

--
Supra

[Evgenii Stepanov]

I've uploaded a patch implementing this in gold to
https://sourceware.org/ml/binutils/2017-04/msg00000.html

[Ali Bahrami]

On 04/ 4/17 05:30 PM, 'Evgenii Stepanov' via Generic System V Application Binary Interface wrote:
> I've uploaded a patch implementing this in gold to
> https://sourceware.org/ml/binutils/2017-04/msg00000.html

I guess all that leaves is to agree on the words for the gABI.

I went back to Cary's original SHF_METADATA proposal from this thread,
on January 26th:

https://groups.google.com/d/msg/generic-abi/_CbBM6T6WeM/yCv9xmctBgAJ

and put together the following by merging the metadata ideas into
the existing SHF_LINK_ORDER description:

SHF_LINK_ORDER

This flag adds special ordering requirements for link editors. The

requirements apply to the referenced section identified by the sh_link
field of this section's header. If this section is combined with other
sections in the output file, the section must appear in the same
relative order with respect to those sections, as the referenced
section appears with respect to sections the referenced section is
combined with.

A typical use of this flag is to build a table that references
text or data sections in address order.

In addition to adding ordering requirements, SHF_LINK_ORDER indicates
that the section contains metadata describing the referenced section.
If the referenced section is discarded by the link editor for any reason,
this section must also be discarded. Furthermore, when performing unused
section elimination, relocations from this section into the referenced
section should not be taken as evidence that the referenced section
should be retained.

If that is acceptable to everyone, I think we can call this done.

- Ali

[Evgenii Stepanov]

Should it not be the other way around? "If the referenced section is
retained by the link editor, this section must be retained as well".

> Furthermore, when performing
> unused
> section elimination, relocations from this section into the referenced
> section should not be taken as evidence that the referenced section
> should be retained.
>
> If that is acceptable to everyone, I think we can call this done.
>
> - Ali
>
>

[Ali Bahrami]

On 04/ 5/17 03:53 PM, 'Evgenii Stepanov' via Generic System V Application Binary Interface wrote:
>> If the referenced section is discarded by the link editor for any
>> reason,
>> this section must also be discarded.
> Should it not be the other way around? "If the referenced section is
> retained by the link editor, this section must be retained as well".
>

Now that you mention it, I'd say that it's really both ways.
It's wrong to discard either of these without also discarding
the other.

How about this version (only the last paragraph is changed):

SHF_LINK_ORDER

This flag adds special ordering requirements for link editors. The
requirements apply to the referenced section identified by the sh_link
field of this section's header. If this section is combined with other
sections in the output file, the section must appear in the same
relative order with respect to those sections, as the referenced
section appears with respect to sections the referenced section is
combined with.

A typical use of this flag is to build a table that references
text or data sections in address order.

In addition to adding ordering requirements, SHF_LINK_ORDER indicates
that the section contains metadata describing the referenced section.

When performing unused section elimination, the link editor should
ensure that both the section and the referenced section are retained
or discarded together. Furthermore, relocations from this section

into the referenced section should not be taken as evidence that the
referenced section should be retained.

- Ali

[Evgenii Stepanov]

This version looks fine to me.

[Suprateeka R Hegde]

On 06-Apr-2017 03:49 AM, Ali Bahrami wrote:
>
> SHF_LINK_ORDER
>
> This flag adds special ordering requirements for link editors. The
> requirements apply to the referenced section identified by the
> sh_link
> field of this section's header. If this section is combined with
> other
> sections in the output file, the section must appear in the same
> relative order with respect to those sections, as the referenced
> section appears with respect to sections the referenced section is
> combined with.
>
> A typical use of this flag is to build a table that references
> text or data sections in address order.
>
> In addition to adding ordering requirements, SHF_LINK_ORDER indicates
> that the section contains metadata describing the referenced section.
> When performing unused section elimination, the link editor should
> ensure that both the section and the referenced section are retained
> or discarded together. Furthermore, relocations from this section
> into the referenced section should not be taken as evidence that the
> referenced section should be retained.

Looks good to me. I have updated our documentation too with this. I
assume this is now complete.

(The idea of removal of OS_NONCONFORMING and hence reuse of that slot
has, however, ended with an anti-climax)

--
Supra