Proposed gABI wording change for SHT_SYMTAB_SHNDX

[Ali Bahrami]

We've recently become aware of a divergence between
the interpretation of SHT_SYMTAB_SHNDX in the Solaris
and GNU toolchain that I'd like to see fixed.

The gABI can be interpreted as saying that a given object
can only have a single SHT_SYMTAB_SHNDX section:

http://www.sco.com/developers/gabi/latest/ch4.sheader.html

SHT_SYMTAB_SHNDX
This section is associated with a section of
type SHT_SYMTAB and is required if any of the
section header indexes referenced by that symbol
table contain the escape value SHN_XINDEX.

and we see that binutils/BFD enforce that, as witnessed
by this bug that the illumos guys filed a while back:

https://sourceware.org/bugzilla/show_bug.cgi?id=15835

Meanwhile, the Solaris Linker and Libraries Manual, which
is where we have our version of the gABI information, says:

SHT_SYMTAB_SHNDX

Identifies a section containing extended section
indexes, that are associated with a symbol table.
If any section header indexes referenced by a symbol
table, contain the escape value SHN_XINDEX, an
associated SHT_SYMTAB_SHNDX is required.

Note the change from "SHT_SYMTAB" to the generic "symbol table",
which accommodates SHT_DYNSYM, and the Solaris specific
SHT_SUNW_LDYNSYM, along with any other symbol tables that may
be invented in the future.

We were part of the gABI definition, and our historical records
show that we started with the same words as in the gABI, but it
seems that at some point, probably while implementing the extended
section stuff, we modified that, and in an object with more than
SHN_LORESERVE sections, we will produce an SHT_SYMTAB_SHNDX for
all symbol tables, and not just .symtab. I think this makes sense:

- Because the SHNDX section elements are paired
1:1, each symbol table requires its own.

- The intent to support multiple SHNDX is clear in
the fact that the sh_link field is used to make
the association explicit.

- It is standard and common for a given executable
or shared object to have both a .symtab, and a
.dynsym, and if you want their st_shndx fields to
be accurate, each needs an SHNDX.

Hence, the need, and possibility, for multiple SHNDX sections.
Although what we did seems reasonable, I'm unable to explain
why we never fed that information back to the wider community.
I think we just dropped that ball, something that I'd like to
rectify now.

Overall, I think that multiple SHNDX should be allowed, but not
required, at the option of the link-editor, and that BFD ought
to support that. Clarifying the gABI on this point seems like
the right place to start that discussion. I asked Cary about the
GNU linkers off list, and he confirms that they can in fact
generate multiple SHNDX:

> As far as the gABI is concerned, I think it's mostly that we only
> anticipated a need for huge section tables in relocatable object
> files. The upper bound became an issue only after we added COMDAT
> sections to handle C++, and after linking, I think it would be
> exceedingly unlikely to need so many sections in an executable file or
> a shared object.
>
> Nevertheless, gold will create a .dynsym_shndx section if necessary,
> alongside the .symtab_shndx section, both with type SHT_SYMTAB_SHNDX,
> and we use the sh_link field to point to the associated SHT_SYMTAB
> section. I doubt we've ever exercised this feature, though.

FWIW, both of his points (only really used in relocatable objects,
not something that gets commonly exercised) match our experiences.

Are there any objections to changing the gABI wording on this
feature to match those from the Solaris documentation, and make it
clear that multiple SHT_SYMTAB_SHNDX sections are allowed?

- Ali

[Cary Coutant]

> For what it is worth, I tried bfd ld and gold in the attached
> testcase. With bfd I got
>
> ld-new: test.so: Too many sections: 65281 (>= 65280)
>
> With gold I got a crash (backtrace attached).

The crash is because gold has an incorrect test for whether or not
there are more than SHN_LORESERVE sections -- it tests
allocated_section_count(), which only counts allocated sections. But
we need to create the XINDEX section if the *total* section count
exceeds SHN_LORESERVE.

Anyway, if you add an "a" flag to the .section directives in your test
case, gold seems to generate a good output file.

-cary

[Mark Wielaard]

Hi,

On Tue, 2015-07-07 at 11:35 -0600, Ali Bahrami wrote:
> Are there any objections to changing the gABI wording on this
> feature to match those from the Solaris documentation, and make it
> clear that multiple SHT_SYMTAB_SHNDX sections are allowed?

No objection. It does make sense.

In elfutils (libdwfl) we do make the assumption that SHT_SYMTAB_SHNDX
should be paired with a SHT_SYMTAB. We don't look for, and don't expect,
a SHT_SYMTAB_SHNDX for a SHT_DYNSYM. So that should be fixed.

There seem to be two reasons for that. First is indeed the literal
interpretation of "This section is associated with a section of type
SHT_SYMTAB". The second is because we might arrive at the (dynsym)
symbol table through the dynamic section/segment. In that case we follow
the DT_SYMTAB and there is no DT_SYMTAB_SHNDX, so we assume that for
symbol tables found through DT_SYMTAB there is no extended section index
array.

If we are cleaning this up then it would be good to also define
DT_SYMTAB_SHNDX. That would help code to not have to fall back to
scanning the section headers (which might be stripped) when a symbol
table is found through the dynamic section/segment. It makes sense to
have this in the dynamic table since the section already has to be
allocated if the corresponding (dynsym) symbol table is and ends up in
the dynamic table.

Cheers,

Mark

[Mark Wielaard]

On Tue, 2015-07-07 at 15:49 -0700, Cary Coutant wrote:
> > For what it is worth, I tried bfd ld and gold in the attached
> > testcase.

For some reason I didn't see the testcase through the mailinglist.
But I am interested in test files that use special symbol table section
index arrays. elfutils has code to handle them, but that code is not
really tested at the moment. So if someone has some example files and
could post them somewhere or forward them to me that would be
appreciated.

Thanks,

Mark

[Ali Bahrami]

For what it's worth, I never saw the original mail with the test case either. I didn't mention that because the gist of it was clear.

- Ali

[Ali Bahrami]

I'm not sure if anything in the runtime linker cares about st_shndx, but DT_SYMTAB_SHNDX makes sense to me, if only for completeness.

- Ali

[John Wolfe]

Ali,

I think that your recommendation to replace the first sentence of the SHT_SYMTAB_INDEX description with the two sentences from the Solaris documentation clears up any ambiguity and is in keeping with current needs.

Likewise the addition of a DT_SYMTAB_SHNDX makes sense.   That change would be along the lines of:

DT_SYMTAB_SHNDX

34

d_ptr

optional

optional

....
DT_SYMTAB_SHNDX

This element holds the address of the SHT_SYMTAB_SHNDX section associated with the dynamic symbol table referenced by the DT_SYMTAB element.

The selection of the tag 34 was chosen for two reasons:

• It is the next number available
• It must be an even number for a pointer versus and odd positive number for a value.

Looking at the Solaris documentation, I see that tag 34 has been defined as

DT_MAXPOSTAGS

The number of positive dynamic array tag values.

This appears to be a tag range check.    If we define DT_SYMTAB_SHNDX as 34 in the gABI:

• does this cause problems with existing tools and if so what is the recourse?
• do other OS or Arch ABIs have any problems with this definition?
  

-- John

[Ali Bahrami]

On 07/22/15 16:09, John Wolfe wrote:
> DT_SYMTAB_SHNDX
>
> This element holds the address of the SHT_SYMTAB_SHNDX section associated with the dynamic symbol table referenced by the DT_SYMTAB element.
>
>
> The selection of the tag 34 was chosen for two reasons:
>

> * It is the next number available
> * It must be an even number for a pointer versus and odd positive number for a value.

>
> Looking at the Solaris documentation, I see that tag 34 has been defined as
>
> DT_MAXPOSTAGS
>
> The number of positive dynamic array tag values.
>
> This appears to be a tag range check. If we define DT_SYMTAB_SHNDX as 34 in the gABI:
>

> * does this cause problems with existing tools and if so what is the recourse?
> * do other OS or Arch ABIs have any problems with this definition?


Hi John,

I've just been consulting with Rod, and we don't believe that
DT_MAXPOSTAGS has any actual ELF meaning. A search of our code
base reveals that the line in <sys/link.h> is the only instance
to it in Solaris. It seems that in older releases, we invented it
in order to size a table of string constants used to map the small
integer DT_ constants to their ASCII strings. We've since changed
that code to map the values to strings differently, and DT_MAXPOSTAGS
isn't used.

Google seems to support that as well: All the hits end up pointing
back at us. Since we don't use it, and it has no gABI definition,
it looks to be cruft.

So please do assign DT_SYMTAB_SHNDX the value 34. When we
add it to our implementation, we'll remove DT_MAXPOSTAGS
at the same time, and clean up our documentation.

Thanks for your attention to detail on this.

- Ali

[John Wolfe]

After some additional discussion with Ali off-line, I have completed the update to the latest gABI document.

http://www.sco.com/developers/gabi/latest/contents.html

The only difference from what was presented in this topic thread was that the existing first sentence of the SHT_SYMTAB_SHNDX description was revise to be a generic symbol table section reference, eliminating a specific reference to the SHT_SYMTAB section.

This section is associated with a symbol table section and is required if any of the section header indexes referenced by that symbol table contain the escape value SHN_XINDEX.

-- John

[Cary Coutant]

> Likewise the addition of a DT_SYMTAB_SHNDX makes sense. That change would
> be along the lines of:
>
> DT_SYMTAB_SHNDX
> 34
> d_ptr
> optional
> optional
>
> ....
> DT_SYMTAB_SHNDX
>
> This element holds the address of the SHT_SYMTAB_SHNDX section associated
> with the dynamic symbol table referenced by the DT_SYMTAB element.

While I don't have a strong objection to adding DT_SYMTAB_SHNDX, I do
agree with Ali's earlier comment that it's unnecessary: No one reading
the dynamic table should care about section indexes; if they do, the
value is readily available from the section table.

-cary

[Mark Wielaard]

On Mon, Jul 13, 2015 at 07:00:05AM -0700, Rafael Espíndola wrote:
> > For some reason I didn't see the testcase through the mailinglist.
> > But I am interested in test files that use special symbol table section
> > index arrays. elfutils has code to handle them, but that code is not
> > really tested at the moment. So if someone has some example files and
> > could post them somewhere or forward them to me that would be
> > appreciated.
>

> Attached the original .s that crashes gold and the test.so that gold
> produces when the section is made allocatable.

Thanks. It showed several issues in our handling of shndx, even though
the basics worked.

- We weren't expecting indexes except in ET_REL files.
I don't know where that restriction came from.
- We would only search for and SHT_SYMTAB_SHNDX for SHT_SYMTAB as
reported before. Which this proposal clarifies.
- And we would flag the opposite as error, an SHT_SYMTAB_SHNDX that
didn't have a SHT_SYMTAB as sh_link target.
This is worded correctly already in Figure 4-14: sh_link and sh_info
Interpretation, so we were just too strict.

binutils ld seems to not handle/create shndx. It gives an error:
Too many sections: 65298 (>= 65280). And gold seems to have an bug
if the sections aren't allocated. So I assume there aren't many files
using shndx in the wild. Has anybody seen them for non-artificial
testcases?

Cheers,

Mark

[Ali Bahrami]

On 07/23/15 11:54, Mark Wielaard wrote:
> binutils ld seems to not handle/create shndx. It gives an error:
> Too many sections: 65298 (>= 65280). And gold seems to have an bug
> if the sections aren't allocated. So I assume there aren't many files
> using shndx in the wild. Has anybody seen them for non-artificial
> testcases?

No, they shouldn't occur in executables or shared objects,
because the link-editor should be merging sections.

In the case I got pulled into, the Solaris ld used with
LLVM/Clang was generating executables with 122K sections
due to the use of -ffunction-sections coupled with our
ld not understanding the GNU rules for folding these
sections back together in a final link. I put together
a quick mapfile to merge them, using a list of section names
that Cary sent us, and was able to get the same object down
to a svelte 32 sections. A real fix for that is on my list.

So yes, pretty artificial, but these bugs should probably be
fixed as a low priority matter just for correctness.

- Ali

[Mark Wielaard]

On Sun, Jul 12, 2015 at 10:02:02PM -0600, Ali Bahrami wrote:
> I'm not sure if anything in the runtime linker cares about
> st_shndx, but DT_SYMTAB_SHNDX makes sense to me, if only for
> completeness.

It really only came up because we have some generic code to get
at the symbol table either through the phdrs and dynamic segment
or through the shdrs, whatever happened to be available. I would
have to audit the code a bit more to see if anything would get
confused or error out if it then saw a symbol with SHN_XINDEX but
without having a shndx table. So it would mostly be convenient
to keep such code generic.

Since gabi requires the shndx table to be allocated it seems a
waste to not be able to access it at runtime. But if we decide
not to support something like DT_SYMTAB_SHNDX then we could maybe
change gabi to say that SHT_SYMTAB_SHNDX is never an allocated
section.

Cheers,

Mark

[Mark Wielaard]

On Wed, Jul 22, 2015 at 06:09:21PM -0400, John Wolfe wrote:
> Likewise the addition of a DT_SYMTAB_SHNDX makes sense. That change would
> be along the lines of:
>
> |DT_SYMTAB_SHNDX
> | | 34|
> |d_ptr
> | optional
> optional
>
> ....
> DT_SYMTAB_SHNDX
>
> This element holds the address of the SHT_SYMTAB_SHNDX section
> associated with the dynamic symbol table referenced by the DT_SYMTAB
> element.
>
>
> The selection of the tag 34 was chosen for two reasons:
>

> * It is the next number available
> * It must be an even number for a pointer versus and odd positive

> number for a value.
>
> Looking at the Solaris documentation, I see that tag 34 has been defined as
>
> DT_MAXPOSTAGS
>
> The number of positive dynamic array tag values.
>
> This appears to be a tag range check. If we define DT_SYMTAB_SHNDX as 34
> in the gABI:
>

> * does this cause problems with existing tools and if so what is the
> recourse?
> * do other OS or Arch ABIs have any problems with this definition?

GNU elf.h defines DT_NUM as 34. As far as I can tell DT_NUM is like
DT_MAXPOSTAGS and would just be incremented to 35 once DT_SYMTAB_SHNDX
is added.

Cheers,

Mark

[Ali Bahrami]

I don't think anyone really disagrees --- it's just an observation
that we don't know of anything that currently needs that information
at runtime. It fits the existing precedents for sections related to
the .dynsym, and I see no harm to having it.

And, John's update this morning already added DT_SYMTAB_SHNDX, so it
currently *is* in the gABI. :-)

- Ali

[John Wolfe]

I concur with Ali's assessment, but remember it is only a document and
it can be revised as needed.

That is why this "conclave" was formed, was it not?

-- John