Allow SHF_ALLOC | SHF_COMPRESSED sections

[Fangrui Song]

https://www.sco.com/developers/gabi/latest/ch4.sheader.html says

> SHF_COMPRESSED - This flag identifies a section containing compressed data. SHF_COMPRESSED applies only to non-allocable sections, and cannot be used in conjunction with SHF_ALLOC. In addition, SHF_COMPRESSED cannot be applied to sections of type SHT_NOBITS.

I find that the incompatibility with SHF_ALLOC is unnecessary.

In the case of linker input (relocatable object files), having both the SHF_ALLOC and SHF_COMPRESSED flags should be acceptable, as long as the linker is capable of decompressing the content.
It is also important to ensure that non-linker consumers are compatible with the section, which is relatively easy to confirm.

Similarly, for linker output (executable or shared object), having both the SHF_ALLOC and SHF_COMPRESSED flags should be permissible.
The section's address may be insignificant at runtime, as it might encode "relocatable" metadata for purposes such as PGO, coverage, sanitizers, etc.
The runtime can locate the section using encapsulation symbols (_start$sectionname, _stop$sectionname) and determine whether it needs to be decompressed by parsing the optional ElfNN_Chdr header.
To differentiate between compressed and uncompressed content, the uncompressed content may begin with 4 zero bytes, as the ch_type field of ElfNN_Chdr is non-zero (typically, 1 or 2 for zlib or zstd compression).

Therefore, I propose that we omit the statement about the incompatibility of the SHF_COMPRESSED flag with SHF_ALLOC:

> SHF_COMPRESSED - This flag identifies a section containing compressed data. SHF_COMPRESSED cannot be applied to sections of type SHT_NOBITS.


---

Related:

In 2021 I made a feature request to GNU ld: Support compressing arbitrary sections (generalized --compress-debug-sections=)
https://sourceware.org/bugzilla/show_bug.cgi?id=27452

I am now considering implementing a lld option to this stuff.

BTW, http://www.linker-aliens.org/blogs/ali/entry/elf_section_compression/ has a nice summary of the history and design of this section flag.
It mentions "non-allocable" in only one place.

[Roland McGrath]

I think SHF_ALLOC|SHF_COMPRESSED has clear uses and obvious semantics in the case of ET_REL files.  I think any notion of SHF_COMPRESSED surviving into SHF_ALLOC sections in final-linked files needs a great deal more consideration and discussion and it seems unlikely to me that this is something whose details should be specified generically.  I think it makes sense to separate the two proposals and consider each one separately.

--
You received this message because you are subscribed to the Google Groups "Generic System V Application Binary Interface" group.
To unsubscribe from this group and stop receiving emails from it, send an email to generic-abi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/generic-abi/d5bce64d-8148-402b-b39b-fd02be9e672fn%40googlegroups.com.

[ali_e...@emvision.com]

Hi,

On 6/26/23 8:56 PM, 'Fangrui Song' via Generic System V Application Binary Interface wrote:
> https://www.sco.com/developers/gabi/latest/ch4.sheader.html says
>
> > SHF_COMPRESSED - This flag identifies a section containing compressed data. SHF_COMPRESSED applies only to non-allocable sections, and cannot be used in conjunction with SHF_ALLOC. In addition,
> SHF_COMPRESSED cannot be applied to sections of type SHT_NOBITS.
>
> I find that the incompatibility with SHF_ALLOC is unnecessary.
>
> In the case of linker input (relocatable object files), having both the SHF_ALLOC and SHF_COMPRESSED flags should be acceptable, as long as the linker is capable of decompressing the content.
> It is also important to ensure that non-linker consumers are compatible with the section, which is relatively easy to confirm.

I don't see that as really beneficial: If an alloc section
is so big that compression is wanted, that's going to be a runtime
issue as well. An early warning of problems to come.

I agree that it could be allowed for ET_REL, because only the
link-editor is involved, and is in a position to uncompress the
input. I won't stand in the way, but it seems a dubious win to me,
and I think it would be better not to encourage it.

>
> Similarly, for linker output (executable or shared object), having both the SHF_ALLOC and SHF_COMPRESSED flags should be permissible.
> The section's address may be insignificant at runtime, as it might encode "relocatable" metadata for purposes such as PGO, coverage, sanitizers, etc.
> The runtime can locate the section using encapsulation symbols (_start$sectionname, _stop$sectionname) and determine whether it needs to be decompressed by parsing the optional ElfNN_Chdr header.
> To differentiate between compressed and uncompressed content, the uncompressed content may begin with 4 zero bytes, as the ch_type field of ElfNN_Chdr is non-zero (typically, 1 or 2 for zlib or zstd
> compression).
>
> Therefore, I propose that we omit the statement about the incompatibility of the SHF_COMPRESSED flag with SHF_ALLOC:
>
> > SHF_COMPRESSED - This flag identifies a section containing compressed data. SHF_COMPRESSED cannot be applied to sections of type SHT_NOBITS.

I disagree with this, particularly as part of the gABI.
I have conceptual and practical concerns.

Sections are a link-time concept, not a runtime one. Runtime
linkers deal in segments, not sections. The section headers
aren't even in memory at load time. While symbols could be
used, it feels more like a workaround than a design. And
requiring the runtime linker to construct segments, filling in
the holes, early in startup, is problematic. ELF features should
be widely useful, rather than trying to cater to absolutely
every scenario,

It also subverts the mapping model that's core to how ELF is
intended to work. The runtime linker is supposed to be a small
piece of code that does an mmap (or equivalent) for each PT_LOAD,
does relocation and some other lightweight work, and gets out
of the way. This pulls in a lot of code, a fair amount of complexity.
and it undermines the VMs ability to share readonly pages. Such
sharing would seem to be particularly important for large sections.

Please note the use of "supposed to be" rather than "are"
in the above. I know that they're already big and complex,
but adding more isn't the answer to that. :-).

Applying COMPRESS to segments rather than sections, makes more
conceptual sense, but it doesn't address the basic problems.

I think that if an allocable section is too large to be in a
final object (EXEC or DYN), then it's also too big to be loaded
onto memory wholesale. Given modern systems, a section has to be
*really* large for that to be true. That sort of data should be
accessed in some other way, outside of linking.

> Related:
>
> In 2021 I made a feature request to GNU ld: Support compressing arbitrary sections (generalized --compress-debug-sections=)
> https://sourceware.org/bugzilla/show_bug.cgi?id=27452
>
> I am now considering implementing a lld option to this stuff.
>
> BTW, http://www.linker-aliens.org/blogs/ali/entry/elf_section_compression/ has a nice summary of the history and design of this section flag.
> It mentions "non-allocable" in only one place.

I appreciate your finding that.

The blog mentioned non-allocable, only to disqualify it without
getting into the details, largely to avoid derailing the discussion
of compression with a long negative digression.

It was definitely discussed carefully in house, and when going
through the process of adding COMPRESS to the ABI. ABIs don't
usually do deep dives on the reasons why something is disallowed
though. Describing what is allowed, accurately, and concisely is
hard enough.

Thanks.

- Ali

[Fangrui Song]

Thank you for sharing your thoughts. Perhaps I should have provided a clearer
description of the use cases for executables/DSOs.

I am considering the inclusion of a SHF_ALLOC|SHF_COMPRESSED output section that
will serve as metadata for certain compiler instrumentations. This section will
be part of a PT_LOAD segment.

At load time, the runtime loader does not require any special handling for the
SHF_COMPRESSED flag. The compressed section would be treated the same as an
opaque .rodata or .eh_frame section.

When the program is executed, the runtime library responsible for the metadata
section will allocate memory and decompress the section's content. For example,
the section content may contain references to .text, but there will be no
references from .text to the decompressed section.

I want to emphasize that this decompression and "relocation" operation imposes
certain limitations on use cases. For instance, it would not be possible to
define a symbol relative to an input section if its absolute address is
significant, as the "relocation" of the section would nullify the absolute
address. However, differences in labels within the output section would still be
permissible.

The SHF_ALLOC|SHF_COMPRESSED output section should not contain dynamic
relocations, as the offset would have an ambiguous meaning.

Now, you may ask: why do we have both a compressed section (part of the file)
and a decompressed section (on the heap) within the process? Well, the user may
have a strong concern about the file size and the tradeoff may favor making the
file smaller (similar to compressed debug sections, but here the runtime library
loads the section).

The decompressed section may be writable and it will take many memory pages
anyway.

 

> Related:
>
> In 2021 I made a feature request to GNU ld: Support compressing arbitrary sections (generalized --compress-debug-sections=)
> https://sourceware.org/bugzilla/show_bug.cgi?id=27452
>
> I am now considering implementing a lld option to this stuff.
>
> BTW, http://www.linker-aliens.org/blogs/ali/entry/elf_section_compression/ has a nice summary of the history and design of this section flag.
> It mentions "non-allocable" in only one place.

I appreciate your finding that.

The blog mentioned non-allocable, only to disqualify it without
getting into the details, largely to avoid derailing the discussion
of compression with a long negative digression.

It was definitely discussed carefully in house, and when going
through the process of adding COMPRESS to the ABI. ABIs don't
usually do deep dives on the reasons why something is disallowed
though. Describing what is allowed, accurately, and concisely is
hard enough.

Thanks.

- Ali

I think similarly, the ABI doesn't extensively restrict or disallow specific
features. If there is a valid and useful use case, I believe maintaining
a certain level of permissiveness is beneficial.

 

--
You received this message because you are subscribed to a topic in the Google Groups "Generic System V Application Binary Interface" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/generic-abi/HUVhliUrTG0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to generic-abi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/generic-abi/890f6057-d583-5877-c1aa-f8f0e057753f%40emvision.com.

--

宋方睿

[Michael Matz]

Hello,

On Tue, 27 Jun 2023, 'Fangrui Song' via Generic System V Application
Binary Interface wrote:

> > I agree that it could be allowed for ET_REL, because only the
> > link-editor is involved, and is in a position to uncompress the
> > input. I won't stand in the way, but it seems a dubious win to me,
> > and I think it would be better not to encourage it.

I agree with Ali here: SHF_ALLOC|SHF_COMPRESSED in ET_REL: sure. Dubious
but doesn't have many issues. In ET_EXEC or ET_DYN: no.

> Thank you for sharing your thoughts. Perhaps I should have provided a
> clearer description of the use cases for executables/DSOs.
>
> I am considering the inclusion of a SHF_ALLOC|SHF_COMPRESSED output
> section that will serve as metadata for certain compiler
> instrumentations. This section will be part of a PT_LOAD segment.

So, as far as the link editor and dynamic linker is concerned, it's simply
a blob of opaque data that the process is then somehow using. I don't see
a reason why the ELF file should somehow special case this data blob from
any other random data blob (say embedded jpeg images).

A good mental model for ET_EXEC and ET_DYN is that sections simply don't
exist. Any generic ELF proposal needs to be measured against that: you
can assume that you don't _have_ section headers at runtime. But your
proposal would need such section headers.

> At load time, the runtime loader does not require any special handling
> for the SHF_COMPRESSED flag. The compressed section would be treated the
> same as an opaque .rodata or .eh_frame section.

So, why then do you want to mark it specially? Everything of your idea
works just as fine if secion headers don't exist, or if they exist aren't
loaded, or if they are loaded don't have the flag set. That doesn't seem
to be a good case for allowing the flag to be set :-)


Ciao,
Michael.

[Ali Bahrami]

Thanks for the explanation.

In this scenario, SHF_COMPRESS has a different
meaning for ALLOC and non-ALLOC sections, and the
link-editor (ld) is expected to act on it, but the
runtime linker is not.

If I'm understanding that correctly, it seems confusing
to have this sort of structure. Using ELF metadata to signal
information to a runtime library, while the runtime linker
is expected to ignore it, feels odd to me. It would be
better to use a different way to label data like this.

I also have to wonder about a user who is concerned about
the disk space taken by these sections, but is unconcerned
about the overhead of having both the compressed and uncompressed
copies in memory at runtime. That's an odd tradeoff --- memory
is the more valuable resource.

Reading between the lines, I think that you're hoping to
leverage the code in libelf that knows how to decompress these
sections, and I understand the desire to not reinvent that wheel
separately. Is that a fair guess?

Most debuggers locate the on-disk image for the object,
and then retrieve non-ALLOC data from them to support
debugging activities (e.g. .symtab, rather than .dynsym).
Is there a reason that this section couldn't be a compressed
non-ALLOC section? If so, you could easily write a small
function that uses libelf to inflate it into memory at runtime,
using the existing libelf support.

- Ali

[Fangrui Song]

I think it's probably unfair to say that SHF_COMPRESSED has a different meaning for ALLOC and non-ALLOC sections.

We know that the dynamic linker can simply ignore section headers, so there is no distinction from this aspect.

The linker can be expected to act on many section flags that the dynamic linker ignores, such as:

* SHF_X86_64_LARGE: linkers can use this flag to place the section away from regular code and data sections to alleviate relocation overflow pressure
* SHF_ARM_PURECODE: linkers may place the section in a PT_LOAD segment without the PF_R flag.
* There are some MIPS flags belonging to this category that I am not familiar with.

SHF_COMPRESSED is not significantly different in this regard.

What's different on the linker side? Well, SHF_ALLOC sections may have non-zero addresses, and the linker
needs to consider that compression changes the section size and may affect the addresses of following sections.
That said, this is all normal code, not much different from some address-affecting features like SHT_RELR/range extension thunks/CPU errata workarounds.

In fact, I have implemented a prototype layouting SHF_ALLOC|SHF_COMPRESSED sections with just ~40 lines of new code.

 

I also have to wonder about a user who is concerned about
the disk space taken by these sections, but is unconcerned
about the overhead of having both the compressed and uncompressed
copies in memory at runtime. That's an odd tradeoff --- memory
is the more valuable resource.

The tradeoff between compressed debug sections and using SHF_ALLOC|SHF_COMPRESSED is quite similar.
When a symbolizer or debugger loads the compressed debug information, it needs to allocate a memory chunk to hold the decompressed content instead of memory mapping the content from the disk.

Why do some people accept this tradeoff? Well, they may prioritize file size and consider debugging as an infrequent operation, or they simply accept this inefficiency.

I understand that SHF_ALLOC|SHF_COMPRESSED sections create an additional copy in the memory image, which can be seen as wasteful.
However, this portion is read-only and accessed on-demand. It's not significantly different from when a program has an internal symbolizer that performs introspection (opening itself, parsing section headers, finding debug sections); sanitizers support such an internal symbolizer. 

 

Reading between the lines, I think that you're hoping to
leverage the code in libelf that knows how to decompress these
sections, and I understand the desire to not reinvent that wheel
separately. Is that a fair guess?

Sorry, but I am not familiar with libelf (elfutils/libelf? I might be influenced by GNU).
It appears that elf_compress_gnu refuses to compress a SHF_ALLOC section. Perhaps the restriction could be lifted for ET_REL files.
In any case, for this discussion, my focus is on the linker (specifically, lld), runtime libraries, and ELF compliance.
If I contribute something, it's likely on the llvm-project side.

Perhaps mentioning why I started this thread will be helpful.

I have a patch for ld.lld --compress-sections <section-glob>=[zlib|zstd] (https://github.com/MaskRay/llvm-project/pull/1) to compress arbitrary sections.
A natural question arises: if <section-glob> matches a SHF_ALLOC section, what should lld do?
I find that the linker can simply ignore the distinction between SHF_ALLOC and non-SHF_ALLOC, and the result makes sense to me.
However, I am not comfortable with the potential ELF compliance issue, which is why I started this thread.

 

Most debuggers locate the on-disk image for the object,
and then retrieve non-ALLOC data from them to support
debugging activities (e.g. .symtab, rather than .dynsym).
Is there a reason that this section couldn't be a compressed
non-ALLOC section? If so, you could easily write a small
function that uses libelf to inflate it into memory at runtime,
using the existing libelf support.

- Ali

I believe that SHF_ALLOC has two primary use cases:

* replace runtime introspection (opening its own file, parsing section headers, parsing section content) with inspecting the content between the encapsulation symbols
* prevent strip/llvm-strip from stripping the sections

Metadata sections have gained more uses in recent years. Some designers may want to implement compression within the format.
If multiple metadata sections adopt this approach, it would lead to duplicated code, considering that we have a generic feature at the object file format level.
It would be unfortunate if the ABI wording simply states that SHF_ALLOC|SHF_COMPRESSED is not allowed.

 

[Michael Matz]

Hello,

On Wed, 28 Jun 2023, 'Fangrui Song' via Generic System V Application Binary Interface wrote:

> In fact, I have implemented a prototype layouting SHF_ALLOC|SHF_COMPRESSED
> sections with just ~40 lines of new code.

Noone is worried about implementation complexity for this. Of course it's
relatively trivial. But not everything that can be done should be done.
IMHO this is one of these cases.

> > I also have to wonder about a user who is concerned about
> > the disk space taken by these sections, but is unconcerned
> > about the overhead of having both the compressed and uncompressed
> > copies in memory at runtime. That's an odd tradeoff --- memory
> > is the more valuable resource.
>
> The tradeoff between compressed debug sections and using
> SHF_ALLOC|SHF_COMPRESSED is quite similar.
> When a symbolizer or debugger loads the compressed debug information, it
> needs to allocate a memory chunk to hold the decompressed content instead
> of memory mapping the content from the disk.

Not at all. The compressed non-alloc parts will be thrown away after
decompression. Parts of loadable segments are not thrown away. (Let's
ignore kernel effects under memory pressure unmapping pages behind
processes backs).

> Perhaps mentioning why I started this thread will be helpful.
>
> I have a patch for ld.lld --compress-sections <section-glob>=[zlib|zstd]
> (https://github.com/MaskRay/llvm-project/pull/1) to compress arbitrary
> sections.
> A natural question arises: if <section-glob> matches a SHF_ALLOC section,
> what should lld do?

It should refuse to do it. But one question back to you: _why_ have you
implemented the above? I guess not merely because you can, so what made
you do it?

> I find that the linker can simply ignore the distinction between SHF_ALLOC
> and non-SHF_ALLOC, and the result makes sense to me.

Of course you can give that combination some meaning easily. But again,
that's not the isse, the issues are in what is implied by that and which
usecases it furthers, and how those usecases could be implemented in a
better way.

> > Most debuggers locate the on-disk image for the object,
> > and then retrieve non-ALLOC data from them to support
> > debugging activities (e.g. .symtab, rather than .dynsym).
> > Is there a reason that this section couldn't be a compressed
> > non-ALLOC section? If so, you could easily write a small
> > function that uses libelf to inflate it into memory at runtime,
> > using the existing libelf support.
>

> I believe that SHF_ALLOC has two primary use cases:
>
> * replace runtime introspection (opening its own file, parsing section
> headers, parsing section content) with inspecting the content between the
> encapsulation symbols

SHF_ALLOC actually has only one meaning: sections in ET_REL files so
marked are placed in PT_LOAD segments. That's it. You can't replace
runtime introspection with that flag, on the contrary: that flag
_requires_ runtime introspection as you need to load section headers
yourself. Inspecting content between symbols is actually less runtime
introspection.

> * prevent strip/llvm-strip from stripping the sections

That's a follow-up from above only.

> Metadata sections have gained more uses in recent years. Some designers may
> want to implement compression within the format.
> If multiple metadata sections adopt this approach, it would lead to
> duplicated code, considering that we have a generic feature at the object
> file format level.

I think this is going in the wrong direction. You _don't_ have that
generic feature, simply because sections are optional. If you want to
have a useful feature in ELF that conveys the idea of metadata blobs then
design such. It possibly will have quite some similarity with section
headers, but there will be more details than just allowing a random flag
combination in unavailable section headers.

For instance you haven't answered what exactly should happen with multiple
SHF_COMPRESSED input sections: will they remain separate? Will they have
to be decompressed, then concatenated, then re-compressed? Will they be
concatenated in compressed form? You lightly touched the topic of
relocations saying they shouldn't exist, but what if they do?

> It would be unfortunate if the ABI wording simply states that
> SHF_ALLOC|SHF_COMPRESSED is not allowed.

Let me ask another question (again): why do you think you need the
SHF_COMPRESSED flag in ET_DYN/EXEC section headers? What exactly can you
do in the presence of that flag that you can't also do without it? (Do
note: you need to parse the first couple of bytes of a so-marked section
anyway to determine what, if any, compression was applied).

So, what will you loose in potential consumers if you simply don't set the
SHF_COMPRESSED flag on SHF_ALLOC sections in your lld patch?


Ciao,
Michael.

[Fangrui Song]

Because (a) there are use cases, (b) the extension is natural, and (c) it's easy to implement.

I know an interested LLVM contributor from code coverage side and interested contributors for sanitizer metadata.

I can think of use cases like .nv_fatbin that embeds ELF file(s) into a SHF_ALLOC section.

We just provide one linker option, and it will serve all these users.

 

> I find that the linker can simply ignore the distinction between SHF_ALLOC
> and non-SHF_ALLOC, and the result makes sense to me.

Of course you can give that combination some meaning easily. But again,
that's not the isse, the issues are in what is implied by that and which
usecases it furthers, and how those usecases could be implemented in a
better way.

Arm's proprietary linker has a feature compressing all Read-Write SHF_ALLOC, SHT_PROGBITS sections:

https://discourse.llvm.org/t/rfc-compress-arbitrary-sections-with-ld-lld-compress-sections/71674/2 

> In an embedded system where the program executed directly from read-only flash, with the initial contents of RW data copied from flash (LMA) to RAM (VMA). When the size of the compressed data + size of the decompressor is < uncompressed data size it can be worth compressing the data.

 

> > Most debuggers locate the on-disk image for the object,
> > and then retrieve non-ALLOC data from them to support
> > debugging activities (e.g. .symtab, rather than .dynsym).
> > Is there a reason that this section couldn't be a compressed
> > non-ALLOC section? If so, you could easily write a small
> > function that uses libelf to inflate it into memory at runtime,
> > using the existing libelf support.
>
> I believe that SHF_ALLOC has two primary use cases:
>
> * replace runtime introspection (opening its own file, parsing section
> headers, parsing section content) with inspecting the content between the
> encapsulation symbols

SHF_ALLOC actually has only one meaning: sections in ET_REL files so
marked are placed in PT_LOAD segments. That's it. You can't replace
runtime introspection with that flag, on the contrary: that flag
_requires_ runtime introspection as you need to load section headers
yourself. Inspecting content between symbols is actually less runtime
introspection.

As explained, SHF_ALLOC|SHF_COMPRESSED doesn't force parsing the section headers.

 

> * prevent strip/llvm-strip from stripping the sections

That's a follow-up from above only.

> Metadata sections have gained more uses in recent years. Some designers may
> want to implement compression within the format.
> If multiple metadata sections adopt this approach, it would lead to
> duplicated code, considering that we have a generic feature at the object
> file format level.

I think this is going in the wrong direction. You _don't_ have that
generic feature, simply because sections are optional. If you want to
have a useful feature in ELF that conveys the idea of metadata blobs then
design such. It possibly will have quite some similarity with section
headers, but there will be more details than just allowing a random flag
combination in unavailable section headers.

For instance you haven't answered what exactly should happen with multiple
SHF_COMPRESSED input sections: will they remain separate? Will they have
to be decompressed, then concatenated, then re-compressed? Will they be
concatenated in compressed form? You lightly touched the topic of
relocations saying they shouldn't exist, but what if they do?

In GNU ld, gold, and ld.lld, input section compression and output section compression are separate decisions.

If there are multiple SHF_COMPRESSED input sections, the linker should decompress them.

Whether the output section uses compression will be decided by --compress-sections <section-glob>=[zlib|zstd].

Your idea implementing a heavy-weight representation reminds me of 

'Proposal for new ELF extension - "Symbol meta-information", but I am not sure we need this kind of stuff.

SHF_ALLOC|SHF_COMPRESSED works without sabotaging the ELF ABI.

 

> It would be unfortunate if the ABI wording simply states that
> SHF_ALLOC|SHF_COMPRESSED is not allowed.

Let me ask another question (again): why do you think you need the
SHF_COMPRESSED flag in ET_DYN/EXEC section headers? What exactly can you
do in the presence of that flag that you can't also do without it? (Do
note: you need to parse the first couple of bytes of a so-marked section
anyway to determine what, if any, compression was applied).

So, what will you loose in potential consumers if you simply don't set the
SHF_COMPRESSED flag on SHF_ALLOC sections in your lld patch?


Ciao,
Michael.

SHF_COMPRESSED in ET_DYN/EXEC section headers tell tools that the described section is compressed.

The runtime library using the SHF_ALLOC|SHF_COMPRESSED section is recommended to locate the section with a symbol (e.g., parse the header at __start_<sectionname> to get the actual size, not using __stop_<sectionname>) instead of parsing the section headers.

 

[Cary Coutant]

Hi,

I agree with Michael and Ali. I don't see a good reason to disallow SHF_ALLOC | SHF_COMPRESSED on ET_REL files, as the linker can quite naturally decompress the section, apply relocations, and throw it into an output section. But to use it in an executable really does change its meaning. We've been using it for non-allocatable sections (e.g., .debug_*), which works because those sections are read by tools outside the runtime environment, and it's reasonable to require a section table in order for that kind of metadata to be found.

Using it for any sort of runtime data is asking for trouble. As Michael pointed out, anything in an allocatable PROGBITS section is just a blob of data to be interpreted by the executing program. Any special encoding or compression needs to be understood out-of-band from ELF. There can be dynamic table entries to help the runtime loader understand special tables, or ABI-specific program header table entries (e.g., PT_*_UNWIND), but I see no value and only potential headaches in communicating information to the runtime via the section table.

As far as embedded systems go, sure, I see the value of compressing templates for ROM storage that are meant to be decompressed and copied into RAM by a bootstrap routine or loader, but those should be handled in the program header table. Going way back, I've known systems that use a simple run-length encoding scheme for the code in ROM, but in that case, the compression scheme was simply understood by the bootstrap loader.

-cary

[Fangrui Song]

On Thursday, July 13, 2023 at 11:55:22 AM UTC-7 Cary Coutant wrote:

Hi,

I agree with Michael and Ali. I don't see a good reason to disallow SHF_ALLOC | SHF_COMPRESSED on ET_REL files, as the linker can quite naturally decompress the section, apply relocations, and throw it into an output section.

But to use it in an executable really does change its meaning. We've been using it for non-allocatable sections (e.g., .debug_*), which works because those sections are read by tools outside the runtime environment, and it's reasonable to require a section table in order for that kind of metadata to be found.

Using it for any sort of runtime data is asking for trouble. As Michael pointed out, anything in an allocatable PROGBITS section is just a blob of data to be interpreted by the executing program. Any special encoding or compression needs to be understood out-of-band from ELF. There can be dynamic table entries to help the runtime loader understand special tables, or ABI-specific program header table entries (e.g., PT_*_UNWIND), but I see no value and only potential headaches in communicating information to the runtime via the section table.

The SHF_ALLOC | SHF_COMPRESSED section can be a metadata section or an embedded ELF file (certain GPU users).

Some runtime data use cases allow for streaming decompression. Technically the runtime performance can even improve if the disk read overhead outweighs decompression overhead.

For some cases the user may build an in-memory data structure from the decompressed section content and can then discard the decompressed section. With this style of use, the decompressed buffer overhead is no longer an issue.

For the linker side, I acknowledge that there is a fair amount of complexity: https://maskray.me/blog/2023-07-07-compressed-arbitrary-sections