SHT_NOBITS clarification

[ben.dunbobb...@gtempaccount.com]

Hi All,

I was looking at a problem with .bss recently and have a couple of queries about SHT_NOBITS sections.

The current specification states:

SHT_NOBITS
A section of this type occupies no space in the file but otherwise resembles SHT_PROGBITS. Although this section contains no bytes, the sh_offset member contains the conceptual file offset.

What does the specification mean by the "conceptual" file offset here? I believe that the specification should be more precise, something like:

For sections of type SHT_NOBITS the sh_offset member is meaningful only for executable and shared objects where it has the file offset that the link editor would have assigned the section if the type had been SHT_PROGBITS.

The specification also contains the following note:

As ``Sections'' in Chapter 4 describes, the .bss section has the type SHT_NOBITS. Although it occupies no space in the file, it contributes to the segment's memory image. Normally, these uninitialized data reside at the end of the segment, thereby making p_memsz larger than p_filesz in the associated program header element.

Given that the above note hints that the link editor may have to allocate space in the file for SHT_NOBITS sections I think that we could improve the specification here, something like:

As ``Sections'' in Chapter 4 describes, in a relocatable object file the .bss section has the type SHT_NOBITS. In common with other SHT_NOBITS sections, although it occupies no space in the file, it contributes to the segment's memory image. Normally, this uninitialized data resides at the end of the segment, thereby making p_memsz larger than p_filesz in the associated program header element. Otherwise the link editor must change the section type to SHT_PROGBITS and will have to allocate space in the output file for the contents of the .bss sections.

The spec also has a section on "special sections" that constrains the type of the .bss sections:

Figure 4-16: Special Sections
Name Type Attributes
.bss SHT_NOBITS SHF_ALLOC+SHF_WRITE

Perhaps we should change the spec to say that in a shared object or executable file the type could also be SHT_PROGBITS, if the linker has had to allocate file space for the section?

[Ali Bahrami]

On 07/25/18 08:16 AM, ben.dunbobbin%sony.com via Generic System V Application Binary Interface wrote:
> Perhaps we should change the spec to say that in a shared object or executable file the type could also be SHT_PROGBITS, if the linker has had to allocate file space for the section?

I would expect that space to be allocated in .data, not .bss.

- Ali

[ben.dunbobb...@gtempaccount.com]

Ali,

The scenario I had in mind was where the placement was forced in some way (e.g by an linker script).

Example:

PHDRS {

  ph_text PT_LOAD FLAGS (0x5);

  ph_data PT_LOAD FLAGS (0x6);

}

 

SECTIONS {

  .text : { *(.text) } : ph_text

  . = ALIGN (0x4000);

  .bss  : { *(.bss) } : ph_data

  .nobits : { *(.nobits) } : ph_data

  .data : { *(.data) } : ph_data

}

 

.text

.global _start

.type _start,@function

_start:

    ret

 

.bss

.global bss

.type bss,@object

bss:

    .quad 0

    .quad 0

 

.section .nobits,"aw",@nobits

.global nobits

.type nobits,@object

nobits:

    .quad 0

 

.data

.global data

.type data,@object

data:

    .quad 0xdeadbeefdeadbeef

[Ali Bahrami]

On 07/25/18 09:44 AM, ben.dunbobbin%sony.com via Generic System V Application Binary Interface wrote:
> The scenario I had in mind was where the placement was forced in some way (e.g by an linker script).

Hi Ben,

One can do an arbitrary number of gABI violating things from
a linker script. I don't think the gABI should be in the business
of listing these things case by case, because it's hopelessly
infinite, and because the gABI is already quite long and not always
as readable or comprehensible as it might be as a result.

In this case, it doesn't seem to me that the fix is to say that
a section named .bss can be PROGBITS. Rather, the linker script
should to rename the section to .data at the same time that it
converted it to non-BSS. But it doesn't feel necessary for the
gABI to say that explicitly.

I'm also not really sure that this case violates the ELF spec
as much as it violates expectations. That object with a non-bss
section named .bss would work just fine when loaded on a machine,
won't it?

- Ali

[Cary Coutant]

> I was looking at a problem with .bss recently and have a couple of queries
> about SHT_NOBITS sections.
>
> The current specification states:
>
>> SHT_NOBITS
>> A section of this type occupies no space in the file but otherwise
>> resembles SHT_PROGBITS. Although this section contains no bytes, the
>> sh_offset member contains the conceptual file offset.
>
>
> What does the specification mean by the "conceptual" file offset here? I
> believe that the specification should be more precise, something like:
>
>> For sections of type SHT_NOBITS the sh_offset member is meaningful only
>> for executable and shared objects where it has the file offset that the link
>> editor would have assigned the section if the type had been SHT_PROGBITS.

It means pretty much what you suspect it means -- it's what the offset
would be if the section did indeed have bits.

Rather than tighten up the spec, I'd prefer to loosen it up -- the
sh_offset member has no real meaning for SHT_NOBITS sections, and it
should say that.

> The specification also contains the following note:
>
>> As ``Sections'' in Chapter 4 describes, the .bss section has the type
>> SHT_NOBITS. Although it occupies no space in the file, it contributes to the
>> segment's memory image. Normally, these uninitialized data reside at the end
>> of the segment, thereby making p_memsz larger than p_filesz in the
>> associated program header element.
>
> Given that the above note hints that the link editor may have to allocate
> space in the file for SHT_NOBITS sections I think that we could improve the
> specification here, something like:
>
>> As ``Sections'' in Chapter 4 describes, in a relocatable object file the
>> .bss section has the type SHT_NOBITS. In common with other SHT_NOBITS
>> sections, although it occupies no space in the file, it contributes to the
>> segment's memory image. Normally, this uninitialized data resides at the end
>> of the segment, thereby making p_memsz larger than p_filesz in the
>> associated program header element. Otherwise the link editor must change the
>> section type to SHT_PROGBITS and will have to allocate space in the output
>> file for the contents of the .bss sections.

Just because it has to allocate space for those zeroes in the case of
a NOBITS section in the middle of a segment, doesn't mean that the
section can't still be SHT_NOBITS.

I don't believe any additional wording is necessary here.

> The spec also has a section on "special sections" that constrains the type
> of the .bss sections:
>
>> Figure 4-16: Special Sections
>> Name Type Attributes
>> .bss SHT_NOBITS SHF_ALLOC+SHF_WRITE
>
> Perhaps we should change the spec to say that in a shared object or
> executable file the type could also be SHT_PROGBITS, if the linker has had
> to allocate file space for the section?

I don't think this is necessary. This section of the spec really
belongs as part of the gABI, not as part of the ELF spec, and is more
advisory in nature (i.e., "here are some typical sections you might
find in an object file"). From the point of view of the ELF spec,
section names are arbitrary -- the section type and flags are meant to
convey all the information needed.

-cary

[Rod Evans]

On Wed, Jul 25, 2018 at 11:35 AM, Cary Coutant <ccou...@gmail.com> wrote:

Just because it has to allocate space for those zeroes in the case of
a NOBITS section in the middle of a segment, doesn't mean that the
section can't still be SHT_NOBITS.

I've always thought of NOBITS in terms of elf_getdata() returning an Elf_Data

buffer with a NULL d_buf pointer, because there is no "backing storage" for

the section.  If there was backing storage, of allocated space filled with zeros,

I'd expect the section to *not* be NOBITS.  I think some tools might get

confused. 

--

Rod.

[Cary Coutant]

>> Just because it has to allocate space for those zeroes in the case of
>> a NOBITS section in the middle of a segment, doesn't mean that the
>> section can't still be SHT_NOBITS.
>
> I've always thought of NOBITS in terms of elf_getdata() returning an
> Elf_Data
> buffer with a NULL d_buf pointer, because there is no "backing storage" for
> the section. If there was backing storage, of allocated space filled with
> zeros,
> I'd expect the section to *not* be NOBITS. I think some tools might get
> confused.

What's there to be confused about? There may be zeroes in the file,
but that's only because it's part of the loadable segment. What tool
is going to be trying to match up every last byte in a segment with a
PROGBITS section? Only a broken tool, I'd think.

-cary

[Alan Modra]

On Wed, Jul 25, 2018 at 11:35:27AM -0700, Cary Coutant wrote:
> Rather than tighten up the spec, I'd prefer to loosen it up -- the
> sh_offset member has no real meaning for SHT_NOBITS sections, and it
> should say that.

I agree with Cary. The "conceptual placement" idea is nonsense. The
net effect of that phrase encourages people to write validation tools
that check sh_offset for SHT_NOBITS sections, forcing linkers to
comply. There is no other utility that I can see.

For example, you can't use sh_offset alone to reliably determine
section to PT_LOAD segment mapping for SHT_NOBITS sections, as you can
for non-zero sized SHT_PROGBITS sections. With two tightly packed
PT_LOAD segments sh_offset for a SHT_NOBITS section belonging to the
first segment will correspond to p_offset values in the second segment
as well, and as the discussion about converting SHT_NOBITS to
SHT_PROGBITS show, it is quite possible to have a SHT_NOBITS section
amongst SHT_PROGBITS sections in the second segment.

[snip]

> Just because it has to allocate space for those zeroes in the case of
> a NOBITS section in the middle of a segment, doesn't mean that the
> section can't still be SHT_NOBITS.

Yes, and again the only likely problem this will cause is with buggy
validation tools that find this unexpected. Fix the tools rather than
adding silly requirements to the specification!

--
Alan Modra
Australia Development Lab, IBM

[ben.dunbobb...@gtempaccount.com]

On Thursday, 26 July 2018 02:57:43 UTC+1, Alan Modra wrote:

On Wed, Jul 25, 2018 at 11:35:27AM -0700, Cary Coutant wrote:
> Rather than tighten up the spec, I'd prefer to loosen it up -- the
> sh_offset member has no real meaning for SHT_NOBITS sections, and it
> should say that.

I agree with Cary.  The "conceptual placement" idea is nonsense.  The
net effect of that phrase encourages people to write validation tools
that check sh_offset for SHT_NOBITS sections, forcing linkers to
comply.  There is no other utility that I can see.

Thanks for the feedback.

Replacing "conceptual placement" with a statement that the sh_offset of an SHT_NOBITS section has no
meaning LGTM. From experimentation, the current tools do not strictly comply with the "conceptual
placement" constraint so this change should have the nice effect of making current tools more
compliant with the standard. It does leave a mystery of why this wording was chosen originally,
possibly this was just standardizing the behaviour of toolchains at the time?

We did have a problem in our binary stripping tool where BSS sections were failing a sanity check
that the file offset of sections should not overlap. Making it explicit that the sh_offset is
meaningless could have helped the tool writer avoid this.

Interestingly, if it is important for binary tools to account for the file space allocated for BSS
sections when they are placed in the middle of segments, then I think the linker would be forced to
change the section type in the output to SHT_PROGBITS to indicate that the section offsets are
meaningful.