How does libelf handle SHF_COMPRESSED sections?

393 views
Skip to first unread message

H.J.

unread,
Apr 15, 2015, 8:52:00 AM4/15/15
to gener...@googlegroups.com
In particular how do elf_getdata () and elf_newdata () work for SHF_COMPRESSED
sections? Does the compression happen transparently (and does elf_rawdata provide
the compressed data) or is this completely up to the user (which would be a pain and
wouldn't really help with better compressed section support since it would mean to
audit all elf section data accesses in all tools and other libraries based on libelf).

H.J.

Ali Bahrami

unread,
Apr 15, 2015, 4:42:28 PM4/15/15
to gener...@googlegroups.com, Rainer Orth, Rod Evans
Hi H.J.,

It's undesirable for elf_getdata() and elf_newdata() to expand
or compress data of their own volition. There are a couple of reasons:

1) Many libelf tools don't want to read the data from these
sections, and don't want to pay for the inflate/deflate dance.

2) Automatic compression takes the decision of what to compress
away from the tools, which I believe the tools (ld mainly)
should be in the drivers seat for that.

As such, elf_getdata()/elf_newdata() don't do that. If an existing section
is compressed, that's what the user gets, along with SHF_COMPRESSED being set.
When the user creates a section (elf_newdata), then they need to decide whether
or not to compress it, and to explicitly cause the compression to happen if
they do.

A centralized implementation is still wanted though, and libelf is the
obvious place for it to live. You're absolutely right that it's a terrible
idea for individual applications to be doing the compression/decompression
on their own.

- Different implementations mean different bugs, and an
impossible maintenance situation.

- Adding support for new compression formats becomes impossible,
since you have to worry about all those applications catching
up before you can use it. (Note that I'm not a fan of adding
more compression options, I just don't like to burn bridges).

The Solaris libelf has support in libelf for section compression:

/*
* Varieties of section compression.
*/
typedef enum {
ELF_ZSEC_T_NONE = 0, /* No compression applied. must be first, 0 */
ELF_ZSEC_T_ZLIB, /* ZLIB, using generic ELF ABI format */
ELF_ZSEC_T_ZLIB_GNU /* ZLIB, using GNU-style format */
} Elf_ZSec_Type;

elf{32,64}_zsec_identify()
Identify a section that is compressed, and which can be decompressed,
and return the type of compression employed.

elf{32,64}_zsec_compressible()
Determine if a given section can be compressed using a proposed
compression format.

elf{32,64}_zsec_compress()
Compress a section.

elf_zsec_decompress()
Decompress a compressed section.

The compress and decompress operations rely on an opaque
state block that allows the underlying compression code
to allocate memory, used between calls or maintain any other
persistent state they require. For instance, ZLIB has
data structures that need to be allocated and maintained.
This is an Elf_ZSec_State, which is allocated and released with:

elf_zsec_state_alloc()
elf_zsec_state_free()

This shipped in Solaris 11.2. I kept the details out of the
header files and manpages in order to have some soak time in
order to have flexibility to change details based on experience,
but it seems fine, and it's been a couple of years and I was
planning to make them public sometime soon. I think your question
is the push I need to get off my tail and make that happen. I've
been meaning to blog about this stuff as well, and will try to
get that done as well.

Is this something you're planning to work on in the very near
future?

- Ali

H.J. Lu

unread,
Apr 15, 2015, 4:52:39 PM4/15/15
to Generic System V Application Binary Interface, Rainer Orth, Rod Evans
Thanks for the information.

> Is this something you're planning to work on in the very near
> future?
>

I just checked support for

--compress-debug-sections=[none|zlib|zlib-gnu|zlib-gabi]
Compress DWARF debug sections using zlib

into GNU assembler and linker. We are working on libelf
support:

https://bugzilla.redhat.com/show_bug.cgi?id=807053

--
H.J.

Ali Bahrami

unread,
Apr 21, 2015, 4:46:51 PM4/21/15
to Mark Wielaard, gener...@googlegroups.com, Rainer Orth, Rod Evans
Hi Mark,

Sorry for the delay in answering. I've been slammed
with other work, and I wanted to be able to give a useful
and carefully considered reply.

First, here is all the public information I know of about
this subject. You've probably seen it all, but if not,
then it might be useful.

1) My initial proposal from July 2012:

https://groups.google.com/forum/#!topic/generic-abi/dBOS1H47Q64

2) The final definition that was added to the gABI, which was
slightly adjusted from (1), but essentially the same:

http://www.sco.com/developers/gabi/latest/ch4.sheader.html

3) The Solaris Linker and Libraries Guide

gABI format: http://docs.oracle.com/cd/E36784_01/html/E36857/section_compression.html#scrolltoc
GNU-format: http://docs.oracle.com/cd/E36784_01/html/E36857/makehtml-id-7.html#scrolltoc
Using Info: http://docs.oracle.com/cd/E36784_01/html/E36857/compressed_debug_sections.html#scrolltoc


As general background, I should say that I had 4 issues with
the original format:

1) The reliance on section name, specifically those starting
with .debug.

2) The use of a 64-bit size field, in a hardwired
byte order, ignoring the content of the ELF header.
Very un-ELF.

3) The naming explosion that would follow should we want to
support formats other than ZLIB (.bzdebug, etc ?).

4) It doesn't account for the different alignment requirements
of the compressed and inflated data.

(1) was the one that caused me to actually propose the gABI format.
The others are unfortunate, but I probably wouldn't have bothered.
We have debug sections that don't use the .debug prefix, and I really
didn't want to extend the .zXXX naming scheme to non-"debug" sections.

I was sorely tempted to only support the gABI format for Solaris,
as we had no history of existing .zdebug objects, and that old
format is a significant complication. Ultimately, I did add it
in, to support gcc, and in recognition that you can't get very
far in this business by ignoring everything you don't like... :-)

The above is relevant to this discussion because if we didn't
want to support the old format, the libelf API for all of this
would not have to worry about the possibility that a given section's
name can change based on compress/decompress happening.

On to the actual questions...


> To help existing programs it might be helpful to introduce a new flag,
> ELF_F_COMPRESSED, that a program can use with elf_flagelf on an Elf to
> get the automatic compression/decompression. That way a program can
> easily support compressed sections without having to adjust all their
> existing elf_getdata/newdata calls. But if we keep it off by default
> then there are no surprises for existing programs.

I thought about that when I did the Solaris work, but
it presumes that most code will want to always decompress
everything. I found that I never wanted that. Here are my
main use cases for decompression:

1) debugger: A debugger will generally want to defer decompression
of debug data until it needs to read it (the debugger
versions of lazy loading).

2) ld: You might think that ld would always want to decompress
all input, because you can't concatenate compressed
sections. However, our ld has options that cause certain
debug sections to be stripped from the output object.
By deferring decompression until after we've decided what
sections to strip, we do less work.

On the compress side, if libelf decides of its own volition
based on ELF_F_COMPRESSED, then it has to decide which format
to compress too. Today, there's only OriginalGNUFlavor, and
gABI ZLIB. In the future, there could be others. This seems
like a complication not worth solving. Let the application call
the compress routine and just specify what it wants.

I see the need for 4 operations, which map fairly directly to
the functions I listed before:

read_query - Is this section compressed, and in
which format?

write_query - Can this section be compressed in a
specified format? (this provides a place to deal
with the naming limitations of the original GNU
format, which can only deal with sections named
.debug and .zdebug

compress - Compress this section, and give me back the
modified section header (set SHF_COMPRESS if the gABI
style) and the resulting section name (returns the
original name for SHF_COMPRESS, or the .zdebugXXX
name for the old style).

decompress - Decompress this section, and give me back
the modified section header (remove SHF_COMPRESS if
the gABI style) and the resulting section name (returns
the original name for SHF_COMPRESS, or the .debugXXX
name for the old style).

With these, it's easy for a program to quickly decide what it's
looking at, and to create what it wants. This has been working
for us --- I haven't touched that code since I wrote it, until
last week.


>
> Who is responsible for setting the SHF_COMPRESSED flag? Do the above
> functions set/unset it when appropriate? And should the flag also be set
> for sections using the traditional GNU-style format?
>
> It would be convenient if the traditional .zdebug GNU style sections
> also got SHF_COMPRESSED set, but it looks like that is not what GNU
> binutils does at the moment.


The tool creating the section sets SHF_COMPRESSED if it has
produced data compressed in the gABI style. In terms of the above
primitives, compress and decompress do it as described above.

The presence of SHF_COMPRESSED identifies a section compressed
in the gABI style. This has 2 benefits:

1) It lets us distinguish between the old style, and
the new one.

2) It allows the section name to be unchanged, and to
handle debug sections of arbitrary name.

Please don't propose setting SHF_COMPRESSED for the .zdebug
GNU style. That imposes a significant ambiguity on code for
deciding whether it's looking at gABI or old compression,
and defeats a big part of the benefit of the flag. That change
would certainly break our existing code.

Another angle: Let's work to phase out the old style, rather than
improving it. I think the gABI style is better, and so similar
structurally, that there's no real reason not to drive the old
stuff into the sea.



>>
>> Is this something you're planning to work on in the very near
>> future?
>
> There is no hurry, but if you could publish some more background on the
> above proposal I can try implementing it also for elfutils libelf and
> test against the current GNU binutils support.
>


Here's the part where I become an impediment...

I'd really like for us to unify this stuff between Solaris
and GNU if we can. You'll note though that I omitted all the
argument data in the functions I listed in my previous message.
The reason is that when I looked at them, I realized that while
they're OK for internal APIs, I would not be happy to see them
go public in their current form. I spotted some things that
should be better, and I need some time to properly think it through.

If you can allow me a month or so to carefully consider this,
I'd be happy to put something forward as a starting point.
If it turns out to be OK, or can be made OK through discussion,
then maybe we could both code to that interface.

And if not, then we can agree to use _sunw_ and _gnu_
in our function names, as we do for the vendor ranges
in <sys/elf.h>, in order to avoid namespace conflicts.

Thanks...

- Ali

-------------------------------------------------------------------

Because this mail is already way too long, here's some more
info that might be interesting. In this example, I'll use
hello.c to produce debug sections, and show the effect of the
various compression choices on one of them:

Non-compressed:

% cc -g hello.c
% elfdump -cN.debug_abbrev a.out

Section Header[25]: sh_name: .debug_abbrev
sh_addr: 0 sh_flags: 0
sh_size: 0xa5 sh_type: [ SHT_PROGBITS ]
sh_offset: 0x155c sh_entsize: 0
sh_link: 0 sh_info: 0
sh_addralign: 0x1

Compressed, old GNU:

% cc -g hello.c -zcompress-sections=zlib-gnu
% elfdump -cN.zdebug_abbrev a.out

Section Header[25]: sh_name: .zdebug_abbrev
sh_addr: 0 sh_flags: 0
sh_size: 0x88 sh_type: [ SHT_PROGBITS ]
sh_offset: 0x1532 sh_entsize: 0
sh_link: 0 sh_info: 0
sh_addralign: 0x1
ch_size: 0xa5 ch_type: ZLIB (GNU format)

Compressed, gABI ZLIB:

% cc -g hello.c -zcompress-sections=zlib
% elfdump -cN.debug_abbrev a.out

Section Header[25]: sh_name: .debug_abbrev
sh_addr: 0 sh_flags: [ SHF_COMPRESSED ]
sh_size: 0x88 sh_type: [ SHT_PROGBITS ]
sh_offset: 0x1534 sh_entsize: 0
sh_link: 0 sh_info: 0
sh_addralign: 0x4
ch_size: 0xa5 ch_type: [ ELFCOMPRESS_ZLIB ]
ch_addralign: 0x1

Mark Wielaard

unread,
Apr 21, 2015, 4:53:01 PM4/21/15
to gener...@googlegroups.com
[Apologies for any duplicate messages. Seems the mailinglist bounced my
first attempt because it believed I was not yet subscribed.]

Hi Ali,

I would like to see compressed section support in elfutils libelf and
would like to make sure it is source compatible with the Solaris libelf
if possible.

On Wed, 2015-04-15 at 14:42 -0600, Ali Bahrami wrote:
> It's undesirable for elf_getdata() and elf_newdata() to expand
> or compress data of their own volition. There are a couple of reasons:
>
> 1) Many libelf tools don't want to read the data from these
> sections, and don't want to pay for the inflate/deflate dance.
>
> 2) Automatic compression takes the decision of what to compress
> away from the tools, which I believe the tools (ld mainly)
> should be in the drivers seat for that.
>
> As such, elf_getdata()/elf_newdata() don't do that. If an existing section
> is compressed, that's what the user gets, along with SHF_COMPRESSED being set.
> When the user creates a section (elf_newdata), then they need to decide whether
> or not to compress it, and to explicitly cause the compression to happen if
> they do.

To help existing programs it might be helpful to introduce a new flag,
ELF_F_COMPRESSED, that a program can use with elf_flagelf on an Elf to
get the automatic compression/decompression. That way a program can
easily support compressed sections without having to adjust all their
existing elf_getdata/newdata calls. But if we keep it off by default
then there are no surprises for existing programs.

> The Solaris libelf has support in libelf for section compression:
>
> /*
> * Varieties of section compression.
> */
> typedef enum {
> ELF_ZSEC_T_NONE = 0, /* No compression applied. must be first, 0 */
> ELF_ZSEC_T_ZLIB, /* ZLIB, using generic ELF ABI format */
> ELF_ZSEC_T_ZLIB_GNU /* ZLIB, using GNU-style format */
> } Elf_ZSec_Type;
>
> elf{32,64}_zsec_identify()
> Identify a section that is compressed, and which can be decompressed,
> and return the type of compression employed.
>
> elf{32,64}_zsec_compressible()
> Determine if a given section can be compressed using a proposed
> compression format.
>
> elf{32,64}_zsec_compress()
> Compress a section.
>
> elf_zsec_decompress()
> Decompress a compressed section.

Who is responsible for setting the SHF_COMPRESSED flag? Do the above
functions set/unset it when appropriate? And should the flag also be set
for sections using the traditional GNU-style format?

It would be convenient if the traditional .zdebug GNU style sections
also got SHF_COMPRESSED set, but it looks like that is not what GNU
binutils does at the moment.

> The compress and decompress operations rely on an opaque
> state block that allows the underlying compression code
> to allocate memory, used between calls or maintain any other
> persistent state they require. For instance, ZLIB has
> data structures that need to be allocated and maintained.
> This is an Elf_ZSec_State, which is allocated and released with:
>
> elf_zsec_state_alloc()
> elf_zsec_state_free()
>
> This shipped in Solaris 11.2. I kept the details out of the
> header files and manpages in order to have some soak time in
> order to have flexibility to change details based on experience,
> but it seems fine, and it's been a couple of years and I was
> planning to make them public sometime soon. I think your question
> is the push I need to get off my tail and make that happen. I've
> been meaning to blog about this stuff as well, and will try to
> get that done as well.
>
> Is this something you're planning to work on in the very near
> future?

There is no hurry, but if you could publish some more background on the
above proposal I can try implementing it also for elfutils libelf and
test against the current GNU binutils support.

Thanks,

Mark

Mark Wielaard

unread,
Apr 29, 2015, 9:33:30 AM4/29/15
to gener...@googlegroups.com, Rainer Orth, Rod Evans
Hi Ali,

On Tue, 2015-04-21 at 14:49 -0600, Ali Bahrami wrote:
> Sorry for the delay in answering. I've been slammed
> with other work, and I wanted to be able to give a useful
> and carefully considered reply.

No worries. My reply to your reply took even longer...
Also it is good to take some time. From your answers I see my
assumptions about the goals/semantics for SHF_COMPRESSED were slightly
off. So lets take some time to think and make sure we aren't
miscommunicating.

> First, here is all the public information I know of about
> this subject. You've probably seen it all, but if not,
> then it might be useful.
>
> 1) My initial proposal from July 2012:
>
> https://groups.google.com/forum/#!topic/generic-abi/dBOS1H47Q64
>
> 2) The final definition that was added to the gABI, which was
> slightly adjusted from (1), but essentially the same:
>
> http://www.sco.com/developers/gabi/latest/ch4.sheader.html
>
> 3) The Solaris Linker and Libraries Guide
>
> gABI format: http://docs.oracle.com/cd/E36784_01/html/E36857/section_compression.html#scrolltoc
> GNU-format: http://docs.oracle.com/cd/E36784_01/html/E36857/makehtml-id-7.html#scrolltoc
> Using Info: http://docs.oracle.com/cd/E36784_01/html/E36857/compressed_debug_sections.html#scrolltoc

Thanks. I didn't know about the Solaris documentation. The description
of the datastructes is very helpful.

> As general background, I should say that I had 4 issues with
> the original format:
>
> 1) The reliance on section name, specifically those starting
> with .debug.
>
> 2) The use of a 64-bit size field, in a hardwired
> byte order, ignoring the content of the ELF header.
> Very un-ELF.
>
> 3) The naming explosion that would follow should we want to
> support formats other than ZLIB (.bzdebug, etc ?).
>
> 4) It doesn't account for the different alignment requirements
> of the compressed and inflated data.
>
> (1) was the one that caused me to actually propose the gABI format.
> The others are unfortunate, but I probably wouldn't have bothered.
> We have debug sections that don't use the .debug prefix, and I really
> didn't want to extend the .zXXX naming scheme to non-"debug" sections.

Yes. This was also why I was advocated adopting SHF_COMPRESSED. The
current approach is at the wrong level. At least in elfutils (libdw, the
DWARF processing library) we currently have the problem that the above
scheme is very awkward when having to process ELF files that need
relocations in the debug sections (like linux kernel modules). None of
the code that handles the lower level ELF file data is name based, only
the debug DWARF layer cares about section names.

> I was sorely tempted to only support the gABI format for Solaris,
> as we had no history of existing .zdebug objects, and that old
> format is a significant complication. Ultimately, I did add it
> in, to support gcc, and in recognition that you can't get very
> far in this business by ignoring everything you don't like... :-)
>
> The above is relevant to this discussion because if we didn't
> want to support the old format, the libelf API for all of this
> would not have to worry about the possibility that a given section's
> name can change based on compress/decompress happening.

To be honest I hope we can deprecate the old (name based) format. Not
many tools handle it at the moment and some of those that do get it
wrong when having to handle relocations. To promote progress I wouldn't
mind saying the old format is only supported for DWARF debug sections
for which no relocations are needed.

> > To help existing programs it might be helpful to introduce a new flag,
> > ELF_F_COMPRESSED, that a program can use with elf_flagelf on an Elf to
> > get the automatic compression/decompression. That way a program can
> > easily support compressed sections without having to adjust all their
> > existing elf_getdata/newdata calls. But if we keep it off by default
> > then there are no surprises for existing programs.
>
> I thought about that when I did the Solaris work, but
> it presumes that most code will want to always decompress
> everything. I found that I never wanted that.

In my experience most programs using elfutils libelf are read-only tools
for inspecting some specific data sections. Often just to quickly
inspect some phdrs, the dynamic array, notes, symbol tables or DWARF
debug data. Only a small amount of tools use it to manipulate or create
new ELF data.

> Here are my
> main use cases for decompression:
>
> 1) debugger: A debugger will generally want to defer decompression
> of debug data until it needs to read it (the debugger
> versions of lazy loading).
>
> 2) ld: You might think that ld would always want to decompress
> all input, because you can't concatenate compressed
> sections. However, our ld has options that cause certain
> debug sections to be stripped from the output object.
> By deferring decompression until after we've decided what
> sections to strip, we do less work.

In both those cases you don't need to decompresses unless you actually
call elf_getdata () don't you? Given that elf_getdata () can cause the
on disk data to be converted to the in memory representation I assume
such tools will defer calling elf_getdata () normally till they actually
need to access the section contents.

> On the compress side, if libelf decides of its own volition
> based on ELF_F_COMPRESSED, then it has to decide which format
> to compress too. Today, there's only OriginalGNUFlavor, and
> gABI ZLIB. In the future, there could be others. This seems
> like a complication not worth solving. Let the application call
> the compress routine and just specify what it wants.

I had assumed there was only one compression format to support (gABI
ZLIB). For existing sections that have ELF_F_COMPRESSED set we already
know the flavor. But yes, for new sections if the user sets
ELF_F_COMPRESSED they should also explicitly set the compression if they
don't want the default (but we could just define a default).

> I see the need for 4 operations, which map fairly directly to
> the functions I listed before:
>
> read_query - Is this section compressed, and in
> which format?
>
> write_query - Can this section be compressed in a
> specified format? (this provides a place to deal
> with the naming limitations of the original GNU
> format, which can only deal with sections named
> .debug and .zdebug
>
> compress - Compress this section, and give me back the
> modified section header (set SHF_COMPRESS if the gABI
> style) and the resulting section name (returns the
> original name for SHF_COMPRESS, or the .zdebugXXX
> name for the old style).
>
> decompress - Decompress this section, and give me back
> the modified section header (remove SHF_COMPRESS if
> the gABI style) and the resulting section name (returns
> the original name for SHF_COMPRESS, or the .debugXXX
> name for the old style).
>
> With these, it's easy for a program to quickly decide what it's
> looking at, and to create what it wants. This has been working
> for us --- I haven't touched that code since I wrote it, until
> last week.

Trying to deal with the old GNU format seems to make this a bit more
complicated than necessary IMHO. The section renaming also feels like it
doesn't really belong in libelf.

> > Who is responsible for setting the SHF_COMPRESSED flag? Do the above
> > functions set/unset it when appropriate? And should the flag also be set
> > for sections using the traditional GNU-style format?
> >
> > It would be convenient if the traditional .zdebug GNU style sections
> > also got SHF_COMPRESSED set, but it looks like that is not what GNU
> > binutils does at the moment.
>
> The tool creating the section sets SHF_COMPRESSED if it has
> produced data compressed in the gABI style. In terms of the above
> primitives, compress and decompress do it as described above.
>
> The presence of SHF_COMPRESSED identifies a section compressed
> in the gABI style. This has 2 benefits:
>
> 1) It lets us distinguish between the old style, and
> the new one.
>
> 2) It allows the section name to be unchanged, and to
> handle debug sections of arbitrary name.
>
> Please don't propose setting SHF_COMPRESSED for the .zdebug
> GNU style. That imposes a significant ambiguity on code for
> deciding whether it's looking at gABI or old compression,
> and defeats a big part of the benefit of the flag. That change
> would certainly break our existing code.

OK. My suggestion is based on the idea that you don't want to have to
deal with section name mangling in libelf. IMHO we should just make the
user responsible for using the "correct" legacy section naming if they
use the non-standard compression scheme.

> Another angle: Let's work to phase out the old style, rather than
> improving it. I think the gABI style is better, and so similar
> structurally, that there's no real reason not to drive the old
> stuff into the sea.

Agreed. But if we can do that without adding extra complications for
libelf implementations that would be nice. The few tools that currently
handle the GNU style already do it without using libelf (at least in the
GNU toolchain).

> >> Is this something you're planning to work on in the very near
> >> future?
> >
> > There is no hurry, but if you could publish some more background on the
> > above proposal I can try implementing it also for elfutils libelf and
> > test against the current GNU binutils support.
> >
>
> Here's the part where I become an impediment...
>
> I'd really like for us to unify this stuff between Solaris
> and GNU if we can. You'll note though that I omitted all the
> argument data in the functions I listed in my previous message.
> The reason is that when I looked at them, I realized that while
> they're OK for internal APIs, I would not be happy to see them
> go public in their current form. I spotted some things that
> should be better, and I need some time to properly think it through.
>
> If you can allow me a month or so to carefully consider this,
> I'd be happy to put something forward as a starting point.
> If it turns out to be OK, or can be made OK through discussion,
> then maybe we could both code to that interface.

That is a fine time-table. I still have to think a bit about whether it
really makes sense to try to "transparently" support non-standard
compression format/section naming. Support for the standard
SHF_COMPRESSED sections was only added to GNU binutils ld earlier this
month and the support for gold is still being worked on. So it will be
some time before people will be able to use this.

Thanks,

Mark

Ali Bahrami

unread,
Apr 29, 2015, 1:27:37 PM4/29/15
to gener...@googlegroups.com, Mark Wielaard, Rainer Orth, Rod Evans
Hi Mark,

Your reply is timely, as I spent part of yesterday wrestling
with how to deal with the section name issue in these public
functions, and feeling generally unhappy about the whole thing.
I like your idea of pushing the responsibility for the section
renaming for the old format out of libelf, and onto the code calling
it. That helps quite a bit, and I'm going to run with it.


>
> To be honest I hope we can deprecate the old (name based) format. Not
> many tools handle it at the moment and some of those that do get it
> wrong when having to handle relocations. To promote progress I wouldn't
> mind saying the old format is only supported for DWARF debug sections
> for which no relocations are needed.

That sounds good to me. As a simplification, I don't think that the
libelf implementation should check for such relocations --- it should
just be understood that compressing such a section isn't going to work.


>
>> > To help existing programs it might be helpful to introduce a new flag,
>>> ELF_F_COMPRESSED, that a program can use with elf_flagelf on an Elf to
>>> get the automatic compression/decompression. That way a program can
>>> easily support compressed sections without having to adjust all their
>>> existing elf_getdata/newdata calls. But if we keep it off by default
>>> then there are no surprises for existing programs.
>>
>> I thought about that when I did the Solaris work, but
>> it presumes that most code will want to always decompress
>> everything. I found that I never wanted that.
>
> In my experience most programs using elfutils libelf are read-only tools
> for inspecting some specific data sections. Often just to quickly
> inspect some phdrs, the dynamic array, notes, symbol tables or DWARF
> debug data. Only a small amount of tools use it to manipulate or create
> new ELF data.
...
>> Here are my
>> main use cases for decompression:
...
> In both those cases you don't need to decompresses unless you actually
> call elf_getdata () don't you? Given that elf_getdata () can cause the
> on disk data to be converted to the in memory representation I assume
> such tools will defer calling elf_getdata () normally till they actually
> need to access the section contents.


I think you're correct about that, with a couple of caveats. One is that
in the original GNU format, you cannot safely identify a compressed
section without knowing its name, and libelf generally doesn't interpret
the .shstrtab. That makes the old format problematic for ELF_F_COMPRESSED.
Of course, you could explicitly not support ELF_F_COMPRESSED for
the old GNU style. That's probably going to be surprising and
undesirable for those dealing with a legacy of old objects and just
wanting things to "work".

The other caveat is that when you decompress, some aspects of the section
header are altered:

- SHF_COMPRESSED is cleared
- sh_size is updated with the inflated size
- sh_addralign is modified to reflect data alignment rather than
the alignment requirements of the compressed stuff.

I think a program that sets ELF_F_COMPRESSED would want to be completely
shielded from any knowledge of underlying compression, which complicates
elf{32,64}_getshdr() if called before elf_getdata() cooks things:

- Should they cook the data in order to make sure the section
header reflects uncompressed state? That likely defeats our
desire to decompress only on demand.

- Should they "lie" to the user and give them a section header
that is already updated to show the decompressed state, rather
than the actual current state? I presume so, in which case libelf
will have to retain the true size/alignment somewhere internally
for use in cooking later.

Having the user code explicitly do

scn = elf_{get,ndx,next}scn(elf, ...);
if (is_compressed(scn, scn_name))
decompress(scn);

allows us to duck all of those questions.



> I had assumed there was only one compression format to support (gABI
> ZLIB). For existing sections that have ELF_F_COMPRESSED set we already
> know the flavor. But yes, for new sections if the user sets
> ELF_F_COMPRESSED they should also explicitly set the compression if they
> don't want the default (but we could just define a default).

Ignoring the old GNU format, you're right that there's only one
format to support today. Personally, I'm hoping it stays that way,
because I think the big win from compression comes from doing it
at all, and not necessarily from being able to use some incrementally
more or less effective compression algorithm. However, in proposing the
gABI format, I felt that the format wouldn't be long lived if it
didn't provide for the possibility of other formats. Given that
it's possible, and therefore probably inevitable, it seems best to
figure out now how ELF_F_COMPRESSED would work in a world with more
than one choice.

The thing about defaults is that they're generally right when they're
picked, but can become very wrong down the road, and changing defaults
is usually a bad thing. I can imagine a future where everyone always
explicitly calls the compression function, because the old default
is a bad one. In arguing against ELF_F_COMPRESSED, I'm proposing that
we just embrace that now and keep the plumbing simple. The trade off is
for slightly more work for the programmer in the common case, weighed
against the conceptual benefit of forcing them to weigh more options.
The extra work is essentially:

if (can_compress(scn, scn_name, compress_type))
compress(scn, compress_type);

Alternatively, we can provide a function for changing the compression
format used by ELF_F_COMPRESSED, and assume ZLIB unless otherwise instructed.

Overall, my opening bid on ELF_F_COMPRESSED is to admit that it's possible,
but to question the cost/benefit. Suppose we table this for a bit, and
then discuss it once we have a straw proposal for the core functions?

- Ali

Ali Bahrami

unread,
Apr 30, 2015, 12:43:33 AM4/30/15
to gener...@googlegroups.com, Mark Wielaard, Rainer Orth, Rod Evans
On 4/29/15 11:27 AM, Ali Bahrami wrote:
> The trade off is
> for slightly more work for the programmer in the common case, weighed
> against the conceptual benefit of forcing them to weigh more options.

I missed a "not" in the above. It should have been "benefit of
not forcing". In other words, you have to write an extra statement
in all of your section processing code, but it's a simple statement,
and in return, you don't have to worry about picking between different
options for doing it --- there's just the one way.

- Ali

Mark Wielaard

unread,
May 1, 2015, 6:12:59 PM5/1/15
to gener...@googlegroups.com, Ali Bahrami, Rainer Orth, Rod Evans
On Wed, Apr 29, 2015 at 11:27:30AM -0600, Ali Bahrami wrote:
> The other caveat is that when you decompress, some aspects of the section
> header are altered:
>
> - SHF_COMPRESSED is cleared
> - sh_size is updated with the inflated size
> - sh_addralign is modified to reflect data alignment rather than
> the alignment requirements of the compressed stuff.
>
> I think a program that sets ELF_F_COMPRESSED would want to be completely
> shielded from any knowledge of underlying compression, which complicates
> elf{32,64}_getshdr() if called before elf_getdata() cooks things:
>
> - Should they cook the data in order to make sure the section
> header reflects uncompressed state? That likely defeats our
> desire to decompress only on demand.
>
> - Should they "lie" to the user and give them a section header
> that is already updated to show the decompressed state, rather
> than the actual current state? I presume so, in which case libelf
> will have to retain the true size/alignment somewhere internally
> for use in cooking later.
>
> Having the user code explicitly do
>
> scn = elf_{get,ndx,next}scn(elf, ...);
> if (is_compressed(scn, scn_name))
> decompress(scn);
>
> allows us to duck all of those questions.

I don't think it is necessary to change the section header at all.
If we don't do section name changing then all "different state"
(size and addralign) can be expressed as part of the Elf_Data.
And I think compression/decompression should not be done on the
section, but when creating the Elf_Data of the section.

So assuming we have something like ELF_F_COMPRESSED elf_getdata/rawdata
would just return Elf_Data with d_size and d_align set from the
Elf[32|64]_Chdr and elf_newdata would add the uncompressed data which
elf_update would compress if SHF_COMPRESSED is set. And if we want to
have explicit functions we could have elf_(get|raw)data_uncompressed
functions that return the Elf_Data (with d_size and d_align set to the
uncompressed values from the Elf(32|64)_Chdr) and returns the ch_type
(with zero to indicate the original section wasn't compressed).
And we could then have a elf_newdata_uncompressed function that takes
a ch_type to indicate which compression format the section should use.

So the section sh_size and sh_align would always be the compressed
values and the uncompressed values would be in the Elf_Data d_size
and d_align. Compression is done by elf_update, unless the
Elf_Data was already created compressed, and the decompression is
done by elf_(get|raw)data[_uncompressed].

Cheers,

Mark

Ali Bahrami

unread,
May 8, 2015, 6:50:57 PM5/8/15
to Mark Wielaard, gener...@googlegroups.com, Rod Evans, Rainer Orth
On 04/20/15 06:46 AM, Mark Wielaard wrote:
> There is no hurry, but if you could publish some more background on the
> above proposal I can try implementing it also for elfutils libelf and
> test against the current GNU binutils support.

Hi,

Thanks for your patience with this. In the last
couple of weeks, I've gone back and blown the dust off
our non-public 2 year old libelf compression code. The
existing code was fine as an internal implementation, but
a bit rough and complex to be a good public API. It's been
nice to have a chance to simplify it and make it presentable
before exposing it. In hindsight, I probably should have
done this earlier.

The basic structure is unchanged, there are still 4 primitives
(identify, compressible, compress, decompress), but it's
now in a form that that makes sense on a manpage, and which can
be used in a reasonably short demo.

The <libelf.h> header file gains a new enum for
identifying compression types:

typedef enum {
ELF_ZSCN_T_NONE = 0, /* No compression applied. must be first */
ELF_ZSCN_T_ZLIB_GNU, /* ZLIB (GNU-style, deprecated) */
ELF_ZSCN_T_ZLIB, /* ZLIB (ELF gABI) */
ELF_ZSCN_T_NUM /* must be last */
} Elf_ZScn_Type;

And libelf exports 4 new public functions:

/* Identify the compression status of a section */
int elf_zscn_identify(Elf_Scn *scn,
const char *scn_name);

/* Determine if a section can be compressed */
int elf_zscn_compressible(Elf_Scn *scn,
Elf_ZScn_Type ztype,
const char *scn_name);

/* Compress a section */
int elf_zscn_compress(Elf_Scn *scn,
Elf_ZScn_Type ztype,
const char *scn_name);

/* Decompress a section */
int elf_zscn_decompress(Elf_Scn *scn,
const char *scn_name);

I have attached 4 manpages for this to this message.
I hope attachments work on this list --- if they
don't, please let me know.

You'll note that these are written from the standpoint of
a Solaris implementation --- they list specific section
names and types that are Solaris implementation details. I
would expect that the GNU versions would list their own
names and types. Note also that having elf_zscn_compressible
embody that information, rather than individual programs,
should be a boon for writing portable code that just
compresses the right thing on any platform, without having
their own #ifdef'd lists.

I have also attached a demo program, ec.c, that can
be used to exercise these APIs. This is all working code
(not just a thought experiment). I have it running on
my desktop. I believe that you could use ec as an
initial proof of concept for your implementation.

% cc ec.c -o ec -lelf

Here's a little demo of putting ec through its paces.
I'll use it on a hello world program, built with -g.

% cc -g ~/hello.c

These are the debug sections in the resulting hello
a.out. I could have used the ld compression option to
create them in compressed formats, but since this is
an ec demo, I've left them uncompressed to start.
Of course, ld contains similar logic to ec for doing
that stuff.

% elfdump -I23:26 -I28 -c a.out

Section Header[23]: sh_name: .debug_info
sh_addr: 0 sh_flags: 0
sh_size: 0x1b4 sh_type: [ SHT_PROGBITS ]
sh_offset: 0x1397 sh_entsize: 0
sh_link: 0 sh_info: 0
sh_addralign: 0x1

Section Header[24]: sh_name: .debug_line
sh_addr: 0 sh_flags: 0
sh_size: 0x66 sh_type: [ SHT_PROGBITS ]
sh_offset: 0x154b sh_entsize: 0
sh_link: 0 sh_info: 0
sh_addralign: 0x1

Section Header[25]: sh_name: .debug_abbrev
sh_addr: 0 sh_flags: 0
sh_size: 0xa5 sh_type: [ SHT_PROGBITS ]
sh_offset: 0x15b1 sh_entsize: 0
sh_link: 0 sh_info: 0
sh_addralign: 0x1

Section Header[26]: sh_name: .stab.index
sh_addr: 0 sh_flags: 0
sh_size: 0x84 sh_type: [ SHT_PROGBITS ]
sh_offset: 0x1658 sh_entsize: 0xc (11 entries)
sh_link: 29 sh_info: 0
sh_addralign: 0x4

Section Header[28]: sh_name: .annotate
sh_addr: 0 sh_flags: [ SHF_SUNW_NODISCARD ]
sh_size: 0x7c sh_type: [ SHT_SUNW_ANNOTATE ]
sh_offset: 0x17f0 sh_entsize: 0
sh_link: 0 sh_info: 0
sh_addralign: 0x4

I can compress everything in the preferred gABI ZLIB style:

% ./ec -t zlib a.out

deflate: [23] .debug_info: none => zlib
deflate: [24] .debug_line: none => zlib
deflate: [25] .debug_abbrev: none => zlib
deflate: [26] .stab.index: none => zlib
deflate: [28] .annotate: none => zlib

Note that the lack of a .debug name didn't prevent
elf_zscn_compressible from identifying the stabs and
annotate sections, which on Solaris are also debug
sections. I won't list all of the headers again, but
here's the first one:

% elfdump -N.debug_info -c a.out

Section Header[23]: sh_name: .debug_info
sh_addr: 0 sh_flags: [ SHF_COMPRESSED ]
sh_size: 0x15f sh_type: [ SHT_PROGBITS ]
sh_offset: 0x1398 sh_entsize: 0
sh_link: 0 sh_info: 0
sh_addralign: 0x4
ch_size: 0x1b4 ch_type: [ ELFCOMPRESS_ZLIB ]
ch_addralign: 0x1

Now, I will convert the file to the old GNU style:

% ./ec -t zlib_gnu a.out
inflate: [23] .debug_info: zlib => none
deflate: [23] .debug_info: none => zlib_gnu
inflate: [24] .debug_line: zlib => none
deflate: [24] .debug_line: none => zlib_gnu
inflate: [25] .debug_abbrev: zlib => none
deflate: [25] .debug_abbrev: none => zlib_gnu
inflate: [26] .stab.index: zlib => none
inflate: [28] .annotate: zlib => none
rename: [23] .debug_info => .zdebug_info
rename: [24] .debug_line => .zdebug_line
rename: [25] .debug_abbrev => .zdebug_abbrev

It's decompressed everything, and then recompressed in
the old format, with necessary name changes. Two of
the sections were left uncompressed because their names
preclude compression in this style. Note that it's
possible for each section to use a separate compression
format, as it's per-section, not per file. Hence,
.stab.index and .annotate could have been left
in gABI ZLIB style.

Here is the debug_info section header in this format,
under it's new .zdebug name:

% elfdump -N.zdebug_info -c a.out

Section Header[23]: sh_name: .zdebug_info
sh_addr: 0 sh_flags: 0
sh_size: 0x15f sh_type: [ SHT_PROGBITS ]
sh_offset: 0x1397 sh_entsize: 0
sh_link: 0 sh_info: 0
sh_addralign: 0x1
ch_size: 0x1b4 ch_type: ZLIB (GNU format)

I can easily go back:

% ./ec -t zlib a.out
inflate: [23] .zdebug_info: zlib_gnu => none
deflate: [23] .zdebug_info: none => zlib
inflate: [24] .zdebug_line: zlib_gnu => none
deflate: [24] .zdebug_line: none => zlib
inflate: [25] .zdebug_abbrev: zlib_gnu => none
deflate: [25] .zdebug_abbrev: none => zlib
deflate: [26] .stab.index: none => zlib
deflate: [28] .annotate: none => zlib
rename: [23] .zdebug_info => .debug_info
rename: [24] .zdebug_line => .debug_line
rename: [25] .zdebug_abbrev => .debug_abbrev

Or decompress everything:

% ./ec -t none a.out
inflate: [23] .debug_info: zlib => none
inflate: [24] .debug_line: zlib => none
inflate: [25] .debug_abbrev: zlib => none
inflate: [26] .stab.index: zlib => none
inflate: [28] .annotate: zlib => none

- Ali
elf_zscn_identify.3ELF
elf_zscn_compressible.3ELF
elf_zscn_compress.3ELF
elf_zscn_decompress.3ELF
ec.c

Ali Bahrami

unread,
May 8, 2015, 6:53:04 PM5/8/15
to Mark Wielaard, gener...@googlegroups.com, Rainer Orth, Rod Evans
Hi Mark,

I intentionally kept my head down and kept working on my
straw proposal, rather than replying right away, because I
wanted to give myself time to think about it carefully. I
didn't mean for it to take a full week, but overall it was
good, as it gave me time to go back and forth on the idea
at the same time that I had my head in our libelf code.

I've just sent you a separate message with that straw proposal,
and it is section based rather than data descriptor based. I agree
that your proposal above is workable in the single data buffer
case. However, the section approach seems like the right conceptual
level to me.

My reasons:

- I believe that it is quite common for ELF code to assume
that the section header and data descriptor are in agreement
about size and alignment. I know that in our code, there are
cases where the size in the section header is used to drive
the code that reads the data. An API that can be dropped
into existing code in a targeted way without disturbing those
assumptions has advantages. In contrast, I don't see the
reverse case where something is simplified by not doing that.

- If libelf does not modify the section header in concert
with the data, then programs that wish to convert from
compressed to uncompressed are required to handle those
details. As you'll see in my test program (ec.c) from
the straw proposal, having libelf handle that simplifies
the caller.

- I found in producing the straw proposal, that having
libelf manage the section header in concert with the
data wasn't a significant complication, and adds no
important performance overhead. Not doing it doesn't
win much.

- ELF allows a section to have multiple data descriptors.
In this scenario, compression is a problem. It doesn't
make sense to compress or decompress any one of these
in isolation from the others. It really is the section
that's being compressed (hence SHF_COMPRESSED), and if
there are multiple buffers, they need to be treated as
a single logical concatenation, with the compression header
at the front of the first one.

I should say that our implementation of compression for libelf
does not implement the multiple buffer case today, and probably
won't unless a need emerges, but I wouldn't want to
preclude it.

- API increase: I see a need for 4 primitives, as detailed
in the proposal (identify, compressible, compress, decompress).
If I follow correctly, a data descriptor oriented version
would need at least 5: identify, compressible, get_uncompressed,
raw_uncompressed, and at least one for creating compressed
data.

Thanks...

- Ali

Mark Wielaard

unread,
May 15, 2015, 4:17:38 PM5/15/15
to gener...@googlegroups.com, Rod Evans, Rainer Orth
On Fri, 2015-05-08 at 16:52 -0600, Ali Bahrami wrote:
> The basic structure is unchanged, there are still 4 primitives
> (identify, compressible, compress, decompress)

I don't fully understand why you need these 4.
I would only have two primitives, get and set compression type for the
section.

> The <libelf.h> header file gains a new enum for
> identifying compression types:
>
> typedef enum {
> ELF_ZSCN_T_NONE = 0, /* No compression applied. must be first */
> ELF_ZSCN_T_ZLIB_GNU, /* ZLIB (GNU-style, deprecated) */
> ELF_ZSCN_T_ZLIB, /* ZLIB (ELF gABI) */
> ELF_ZSCN_T_NUM /* must be last */
> } Elf_ZScn_Type;

I think it would be convenient to match up with the ch_types and make
ELF_ZSCN_T_ZLIB = 1 and set ELF_ZSCN_T_ZLIB_GNU = 0x80000000.

> And libelf exports 4 new public functions:
>
> /* Identify the compression status of a section */
> int elf_zscn_identify(Elf_Scn *scn,
> const char *scn_name);
>
> /* Determine if a section can be compressed */
> int elf_zscn_compressible(Elf_Scn *scn,
> Elf_ZScn_Type ztype,
> const char *scn_name);
>
> /* Compress a section */
> int elf_zscn_compress(Elf_Scn *scn,
> Elf_ZScn_Type ztype,
> const char *scn_name);
>
> /* Decompress a section */
> int elf_zscn_decompress(Elf_Scn *scn,
> const char *scn_name);

I think the section name and some of the other extra checks described
really don't belong in the libelf interface/implementation.

I can see how enforcing policy that might be different across
implementations or platforms might be convenient in some situations, but
I think that should be outside libelf. If I want to use GNU style
decompression on a ".foobar" named section or compress a SHT_SYMTAB
section why should libelf deny that?

This might be a difference in how we see the function of libelf. The
elfutils libelf implementation is just one way to interpret and
manipulate ELF images on the GNU platform. So it IMHO doesn't have the
authority to set policy on what valid ELF constructs are allowed.
Especially not which section names can or cannot be used.

I would reduce this interface to just two functions:

int elf_scn_get_compression (Elf_Scn *scn, Elf_Zscn_Type *ztype);
int elf_scn_set_compression (Elf_Scn *scn, Elf_ZScn_Type ztype);

The first sets ztype to ELF_ZSCN_T_ZLIB if SHF_COMPRESSED is set and the
section data starts with a valid Chdr with ZLIB as ch_type (if there is
no valid Chdr an error is returned). If SHF_COMPRESSED isn't set and the
section data starts with the magic "ZLIB", 8 bytes size and a valid ZLIB
header, then ztype is set to ELF_ZSCN_T_ZLIB_GNU. Otherwise it is set to
ELF_ZSCN_T_NONE.

The second can be called with ELF_ZSCN_T_NONE to decompress,
ELF_ZSCN_T_ZLIB to compress in ZLIB format with Chdr or
ELF_ZSCN_T_ZLIB_GNU to compress in the GNU format.

As currently documented it seems to imply that the update immediately
marks the Elf_Scn as ELF_F_DIRTY. Although it doesn't directly say that,
so I might misinterpret. I think it might be good to be explicit about
whether or not the Elf_Scn ELF_F_DIRTY flag is changed. And what effects
elf_flagscn () has when the user explicitly sets or clears ELF_F_DIRTY
on a section which has been (de)compressed.

Instead of saying the result of elf(32|64)_getshdr (Elf_Scn) is
invalidated it would IMHO better to say that it will be updated and be
explicit about which fields will be updated (sh_flags to clear/set
SHF_COMPRESSED, sh_size and sh_addralign). And again, whether the Shdr
is explicitly marked DIRTY or not.

It seems more convenient to leave this up to the user. So it isn't
marked ELF_F_DIRTY automatically. Which means, don't write this section
change back to the file when I do an elf_update (ELF_C_WRITE). That way
you can read uncompressed data from one section and only change another
section without having to explicitly recompress the section again.

Finally we should be explicit about what happens with ELF_F_DIRTY
Elf_Data like created with elf_newdata when a Elf_Scn is compressed.
Currently we just say that the result of elf_getdata (Elf_Scn) will be
invalidated. But we should be explicit about whether that changed
Elf_Data (or that created with elf_newdata) for the Elf_Scn will be
compressed or just discarded. I don't think it should just be discarded,
but compressed (and then discarded). Because otherwise, unless I am
missing something, you would be unable to add any (uncompressed) data to
a section that then needs to be compressed.

Cheers,

Mark

Mark Wielaard

unread,
May 15, 2015, 5:07:49 PM5/15/15
to gener...@googlegroups.com, Rainer Orth, Rod Evans
On Fri, 2015-05-08 at 16:55 -0600, Ali Bahrami wrote:
> I've just sent you a separate message with that straw proposal,
> and it is section based rather than data descriptor based. I agree
> that your proposal above is workable in the single data buffer
> case. However, the section approach seems like the right conceptual
> level to me.

You have working code and I don't, so you win on practicality at
least :) I went over your proposal and thought about how to implement
it. That did make me feel less negative about the section based approach
(though I still very much disagree with the section name filtering). But
it is the multi data buffer case (especially the interaction with
elf_newdata) that still isn't fully clear to me.

> My reasons:
>
> - I believe that it is quite common for ELF code to assume
> that the section header and data descriptor are in agreement
> about size and alignment. I know that in our code, there are
> cases where the size in the section header is used to drive
> the code that reads the data. An API that can be dropped
> into existing code in a targeted way without disturbing those
> assumptions has advantages. In contrast, I don't see the
> reverse case where something is simplified by not doing that.

I hadn't considered that indeed. The counter argument is that the
sh_size == d_size case is only true when not manipulating the Elf_Data
(but often you only read indeed). And this will only happen for new code
that explicitly handles (de)compressed data.

> - If libelf does not modify the section header in concert
> with the data, then programs that wish to convert from
> compressed to uncompressed are required to handle those
> details. As you'll see in my test program (ec.c) from
> the straw proposal, having libelf handle that simplifies
> the caller.

I didn't write a counter example, so you win on points again :)
But what was missing from that example was the case of modifying the
(decompressed) data. Which seems to be the tricky case, whether section
or data based.

> - I found in producing the straw proposal, that having
> libelf manage the section header in concert with the
> data wasn't a significant complication, and adds no
> important performance overhead. Not doing it doesn't
> win much.

The elegant thing is that all meta data seems in sync. But what I found
somewhat confusing in the straw proposal was knowing whether the meta
data represented the in memory view or the elf image/file view. Having
explicit different data getter/setters for compressed/decompressed data
might make that clearer (but again I didn't write an actual straw man,
so this is theory only).

> - ELF allows a section to have multiple data descriptors.
> In this scenario, compression is a problem. It doesn't
> make sense to compress or decompress any one of these
> in isolation from the others. It really is the section
> that's being compressed (hence SHF_COMPRESSED), and if
> there are multiple buffers, they need to be treated as
> a single logical concatenation, with the compression header
> at the front of the first one.
>
> I should say that our implementation of compression for libelf
> does not implement the multiple buffer case today, and probably
> won't unless a need emerges, but I wouldn't want to
> preclude it.

I think this one is easy, you can only have Elf_Data buffers associated
with a Elf_Scn that have the same compressed state (all are uncompressed
or all are compressed in the same format).
elf_(get|raw|new)data[_decompressed] () just has to check the requested
compression format matches previous requested formats.

> - API increase: I see a need for 4 primitives, as detailed
> in the proposal (identify, compressible, compress, decompress).
> If I follow correctly, a data descriptor oriented version
> would need at least 5: identify, compressible, get_uncompressed,
> raw_uncompressed, and at least one for creating compressed
> data.

Yes, the interface would be slightly larger. And I even think you can
have an even smaller interface for the section based approach (although
I get the feeling I am missing something, because I don't understand why
you think we need 4 primitives).

For a data level interface you would need at least 4 functions. 3
functions that duplicate elf_getdata (), elf_rawdata () and elf_newdata
() to request the decompression data. And one function to mark the
Elf_Scn as dirty (including all Elf_Data associated with the section)
plus the (de)compress format to use when elf_update is called.

Cheers,

Mark

Ali Bahrami

unread,
May 18, 2015, 7:24:44 PM5/18/15
to gener...@googlegroups.com, Mark Wielaard, Rainer Orth, Rod Evans
Hi Mark,

Some answers are inline below.

- Ali



On 05/15/15 15:07, Mark Wielaard wrote:
> I hadn't considered that indeed. The counter argument is that the
> sh_size == d_size case is only true when not manipulating the Elf_Data
> (but often you only read indeed). And this will only happen for new code
> that explicitly handles (de)compressed data.

It's true in any code where you have only a single
ELF_Data for a given section. I think that's most code.
Of course, once the programmer starts modifying the data,
then keeping these things in sync, or not, becomes their
problem.


>
>> - If libelf does not modify the section header in concert
>> with the data, then programs that wish to convert from
>> compressed to uncompressed are required to handle those
>> details. As you'll see in my test program (ec.c) from
>> the straw proposal, having libelf handle that simplifies
>> the caller.
>
> I didn't write a counter example, so you win on points again :)
> But what was missing from that example was the case of modifying the
> (decompressed) data. Which seems to be the tricky case, whether section
> or data based.

I don't think it's much trickier than the normal case
of modifying data. You can't sensibly modify a compressed
stream of bytes, since each value depends on the history
of bytes before it. So the model is:

- Decompress the data if it isn't already decompressed.

- Modify in the usual way

- Compress the result if desired

Noting that compression is mainly for debug sections, I'm not
sure how much modification there will be. I think that there
is creation (compiler, linker) and consumption (debuggers,
other observability), but not a lot of need to modify existing
data, Nonetheless, I think it's pretty straightforward.




>
>> - I found in producing the straw proposal, that having
>> libelf manage the section header in concert with the
>> data wasn't a significant complication, and adds no
>> important performance overhead. Not doing it doesn't
>> win much.
>
> The elegant thing is that all meta data seems in sync. But what I found
> somewhat confusing in the straw proposal was knowing whether the meta
> data represented the in memory view or the elf image/file view.

It's the memory view. libelf xlates the Chdr fields just as it
xlates the Shdr. I don't think any of this compression support is
interesting for the raw view, and so, don't think anything beyond
the existing elf_rawdata is needed for that. In the raw view, you
see whatever is on disk, compressed or not.


> Having
> explicit different data getter/setters for compressed/decompressed data
> might make that clearer (but again I didn't write an actual straw man,
> so this is theory only).

I'll reserve judgment, but I don't see yet how different getter/setters
simplifies things. Anyone who wants to actually read the data, will
generally want it decompressed first. The number of programs that want
to interpret the data in compressed form is probably limited to a handful
of ELF utility programs.


> I think this one is easy, you can only have Elf_Data buffers associated
> with a Elf_Scn that have the same compressed state (all are uncompressed
> or all are compressed in the same format).
> elf_(get|raw|new)data[_decompressed] () just has to check the requested
> compression format matches previous requested formats.

I mainly agree: The data for a section is the logical concatenation
of its Elf_Data buffers. There is however, no per-data compression.
There is one compression header, at the front of the data, and all
the data goes with it. However, as above, I don't think it makes sense
to modify compressed data without first decompressing it, and so,
a compressed section with more than one buffer seems unnecessary,
or even wrong.

So I think libelf should probably make it an error to add a second
buffer to a compressed section. The right model is that you decompress
the section, add whatever buffers you want, and then re-compress if
desired. I also think that if you compress a multi-data section,
that it would be reasonable to replace the list of ELF_Data with
a single buffer.

- Ali

Ali Bahrami

unread,
May 18, 2015, 7:24:51 PM5/18/15
to gener...@googlegroups.com, Mark Wielaard, Rod Evans, Rainer Orth
Hi Mark,

Here, I'll reply specifically to the questions asked,
but please read the separate message "Compressed Section
Name Filtering", for general thoughts about some of
these issues.

Thanks...

- Ali



On 05/15/15 14:17, Mark Wielaard wrote:
> On Fri, 2015-05-08 at 16:52 -0600, Ali Bahrami wrote:
>> The basic structure is unchanged, there are still 4 primitives
>> (identify, compressible, compress, decompress)
>
> I don't fully understand why you need these 4.
> I would only have two primitives, get and set compression type for the
> section.

It's largely to protect the programmer from having to deal
with the old format, and to ensure that different programs
make the right choices. In a gABI only world, identify boils
down to SHF_COMPRESSED, and compressible is roughly

SHF_ALLOC not set and not some section type
that should be left alone

both of which are simple enough not to necessarily
require helper functions, though I do think that a
helper function is still useful.



>
>> The <libelf.h> header file gains a new enum for
>> identifying compression types:
>>
>> typedef enum {
>> ELF_ZSCN_T_NONE = 0, /* No compression applied. must be first */
>> ELF_ZSCN_T_ZLIB_GNU, /* ZLIB (GNU-style, deprecated) */
>> ELF_ZSCN_T_ZLIB, /* ZLIB (ELF gABI) */
>> ELF_ZSCN_T_NUM /* must be last */
>> } Elf_ZScn_Type;
>
> I think it would be convenient to match up with the ch_types and make
> ELF_ZSCN_T_ZLIB = 1 and set ELF_ZSCN_T_ZLIB_GNU = 0x80000000.

Yes, and you've zeroed in on the only substantial difference between
the ch_types (ELFCOMPRESS_) and Elf_ZScn_Type. So sure, OK.

On the other hand, it doesn't seem like the programmer
should be making any assumptions about the integer values
assigned to these names, so I don't know how much it matters.


>
> I think the section name and some of the other extra checks described
> really don't belong in the libelf interface/implementation.

Explained to some degree in the "Compressed Section Name Filtering"
message.


>
> I can see how enforcing policy that might be different across
> implementations or platforms might be convenient in some situations, but
> I think that should be outside libelf. If I want to use GNU style
> decompression on a ".foobar" named section or compress a SHT_SYMTAB
> section why should libelf deny that?

The GNU style format is only possible for sections named .debug*.
Unless I'm over-interpreting the rules for GNU style, there's no
option for compressing a section that doesn't follow that naming.
Hence, libelf might reasonably reject an attempt to compress .foobar,
because it would create an invalid or undefined section.

The question about compressing SHT_SYMTAB is more interesting. On one
hand, I think it's going to break a lot of consumers to compress
the .symtab, but I don't necessarily feel that libelf needs to
enforce that. (more on this is below)


>
> This might be a difference in how we see the function of libelf. The
> elfutils libelf implementation is just one way to interpret and
> manipulate ELF images on the GNU platform. So it IMHO doesn't have the
> authority to set policy on what valid ELF constructs are allowed.
> Especially not which section names can or cannot be used.

I think most programs want to follow the established rules, so
providing them with some guidance is a benefit. And in the case
of the old GNU format, there's a correctness issue as I mentioned
above.

Otherwise, I largely agree. That's why elf_zscn_compressible and
elf_zscn_compressible both allow allows the scn_name argument to
be NULL for non-GNU style:

For other compression types, it is convention that
SHT_PROGBITS sections are only compressed if they have
names that start with .compcom, .debug, .line, or .stab.
if a non-NULL scn_name is provided, elf_zscn_compressible
will check for these names. If scn_name is NULL, then
then any non-allocable section with a valid section type
is accepted.

If you pass a NULL name to them, they'll happily compress a PROGBITS
section with any name. But maybe it doesn't go far enough. As it stands,
attempts to compress a symbol table will currently be rejected. I'd be
open to changing things as follows:

- elf_zscn_compressible evaluates the name and section type
in order to provide guidance.

- elf_zscn_compress is willing to compress anything, even if it's
really wrong to do so.

That ought to give both camps what they want.


>
> I would reduce this interface to just two functions:
>
> int elf_scn_get_compression (Elf_Scn *scn, Elf_Zscn_Type *ztype);
> int elf_scn_set_compression (Elf_Scn *scn, Elf_ZScn_Type ztype);
>
> The first sets ztype to ELF_ZSCN_T_ZLIB if SHF_COMPRESSED is set and the
> section data starts with a valid Chdr with ZLIB as ch_type (if there is
> no valid Chdr an error is returned). If SHF_COMPRESSED isn't set and the
> section data starts with the magic "ZLIB", 8 bytes size and a valid ZLIB
> header, then ztype is set to ELF_ZSCN_T_ZLIB_GNU. Otherwise it is set to
> ELF_ZSCN_T_NONE.
>
> The second can be called with ELF_ZSCN_T_NONE to decompress,
> ELF_ZSCN_T_ZLIB to compress in ZLIB format with Chdr or
> ELF_ZSCN_T_ZLIB_GNU to compress in the GNU format.


get_compression is what I called "identify" minus the scn_name
argument.

set_compression is my compress, decompress, and compressible, all
rolled into one, without the scn_name argument.

Modulo settling the "Compressed Section Name Filtering" question,
I only have one concern here. Flexibility is good, but I think most
programs want to follow the generally accepted rules for what
to compress, and would be better off not having their own
policies for that. The lack of an explicit "compressible"
function forces those programs to deal with decisions they'd
rather delegate..


>
> As currently documented it seems to imply that the update immediately
> marks the Elf_Scn as ELF_F_DIRTY. Although it doesn't directly say that,
> so I might misinterpret. I think it might be good to be explicit about
> whether or not the Elf_Scn ELF_F_DIRTY flag is changed. And what effects
> elf_flagscn () has when the user explicitly sets or clears ELF_F_DIRTY
> on a section which has been (de)compressed.

You're correct --- in my current implementation, ELF_F_DIRTY
is set for both the data and section. (more below)


> Instead of saying the result of elf(32|64)_getshdr (Elf_Scn) is
> invalidated it would IMHO better to say that it will be updated and be
> explicit about which fields will be updated (sh_flags to clear/set
> SHF_COMPRESSED, sh_size and sh_addralign). And again, whether the Shdr
> is explicitly marked DIRTY or not.

All fair points: I was trying to allow for memory to be reallocated,
but I don't think that can happen for Shdr, and I don't have a problem
with spelling out the changes on a per-field basis.

>
> It seems more convenient to leave this up to the user. So it isn't
> marked ELF_F_DIRTY automatically. Which means, don't write this section
> change back to the file when I do an elf_update (ELF_C_WRITE). That way
> you can read uncompressed data from one section and only change another
> section without having to explicitly recompress the section again.

My feeling was that there is no reason not to mark those things as
dirty, because unless you do an elf_update(), it doesn't hurt anything.
But your example here shows the hole in my reasoning --- I agree that
it's more flexible for the caller to explicitly dirty the section
themselves if they want it done.


>
> Finally we should be explicit about what happens with ELF_F_DIRTY
> Elf_Data like created with elf_newdata when a Elf_Scn is compressed.
> Currently we just say that the result of elf_getdata (Elf_Scn) will be
> invalidated. But we should be explicit about whether that changed
> Elf_Data (or that created with elf_newdata) for the Elf_Scn will be
> compressed or just discarded. I don't think it should just be discarded,
> but compressed (and then discarded). Because otherwise, unless I am
> missing something, you would be unable to add any (uncompressed) data to
> a section that then needs to be compressed.

As I mentioned in the reply I've just sent about "Data level compression
vs Section", I think that on compression it would be reasonable for all
ELF_Data descriptors for the section to be replaced with a single one.
If that's to be possible, then libelf needs to be able to free the old
descriptors and create a new one (or reuse one of the others), and in any
of these cases, we'd want the programmer to assume that any pointers to
an ELF_Data should be dropped and reacquired.

Ali Bahrami

unread,
May 18, 2015, 7:25:22 PM5/18/15
to gener...@googlegroups.com, Mark Wielaard, Rainer Orth, Rod Evans
Hi Mark,

Thanks for your replies on Friday...

As I read your various questions and issues about the
API I proposed, I believe that they largely boil down
to the section name arguments, the reasons for having
them in the API, and the various things that follow from
that decision. So perhaps this is a good time to have a
higher level discussion about that.

I included that stuff mainly to accommodate what I imagined
were the GNU community requirements to support the old
format. If I'm wrong about that, then I'm very happy to take
a knife to what I've proposed and simplify it down.

To review, I proposed a new enum, and 4 functions:

typedef enum {
ELF_ZSCN_T_NONE = 0, /* No compression applied. must be first */
ELF_ZSCN_T_ZLIB_GNU, /* ZLIB (GNU-style, deprecated) */
ELF_ZSCN_T_ZLIB, /* ZLIB (ELF gABI) */
ELF_ZSCN_T_NUM /* must be last */
} Elf_ZScn_Type;

int elf_zscn_identify(Elf_Scn *scn, const char *scn_name);
int elf_zscn_compressible(Elf_Scn *scn, Elf_ZScn_Type ztype,
const char *scn_name);
int elf_zscn_compress(Elf_Scn *scn, Elf_ZScn_Type ztype,
const char *scn_name);
int elf_zscn_decompress(Elf_Scn *scn, const char *scn_name);

As you correctly point out, Elf_ZScn_Type is really the same
thing as the ch_type ELFCOMPRESS_ constants from the gABI, with one
added wart (ELF_ZSCN_T_ZLIB_GNU) for handing the old GNU style.
It is of course, ugly and unfortunate that we need Elf_ZScn_Type.

The scn_name arguments support 2 different things:

1) In the case of compressing PROGBITS, different platforms
may have different conventions based on the section name,
for what constitutes a debug section. It's convenient to
have that defined in a single place, and good for writing
cross platform code. This is a triviality, and not a show
stopper for me to drop, but the facility seems useful for
most programs.

2) You cannot safely support the GNU style compression without
knowing the section name, because the definition
of GNU compression is that both of the following hold:

a) Name starts with .zdebug
b) The data starts with a magic number ('ZLIB').

You need both pieces of information. It's wrong to
assume that all .zdebug are compressed, and also
wrong to assume that any section with data starting
with 'ZLIB' is compressed. This later point is
especially important -- any uncompressed section
might start with those bytes.

Hence, the idea of supporting ZLIB_GNU, but not having the
names, exposes libelf to an ambiguity that seems like a
problem to me.

We might explore the possibility of having libelf not
support the old format at all. Since all the debatable
baggage is related to that, we could get some nice wins
for the API:

- Without the need to support GNU-style, there's no
need for Elf_ZScn_Type. The ELFCOMPRESS_xxx constants
would be used.

- Without GNU-style, there's relatively little need for
the identify operation, since SHF_COMPRESSED is all that
most programs care about, and the remaining few can
read the Chdr.

- The compressible operation becomes less important, but
I think it still has a role. Flexibility is good, but most
programs just want to compress the things that convention
says should be compressed, and an API that tells them
what to do is useful.

- There would be no need for scn_name arguments.

In an world without the GNU style we might simply have:

int elf_zscn_compress(Elf_Scn *scn, int ztype);
int elf_zscn_decompress(Elf_Scn *scn);
int elf_zscn_compressible(Elf_Scn *scn);

where ztype is an ELFCOMPRESS_xxx constant. You could even
go super-purist, and have just:

int elf_zscn_set_compression(Elf_Scn *scn, int ztype);
int elf_zscn_compressible(Elf_Scn *scn);

allowing ztype to be ELFCOMPRESS_NONE to decompress, as you've
suggested .

That's definitely nicer --- simple and clear. However, it
doesn't provide a single framework for supporting both
the old and new formats, so either that will have to be
provided somewhere else, or a conscious decision to kill
off the old format will need to be taken.

Help me better understand what your (GNU community) requirements
are, and then we can move forward.

In the meantime, I'll reply to your mails from Friday
to cover the specific questions.

Thanks...

- Ali

Mark Wielaard

unread,
Oct 21, 2015, 8:59:12 PM10/21/15
to gener...@googlegroups.com, Ali Bahrami, Rod Evans, Rainer Orth
Hi,

I finally posted a prototype implementation for handling compressed ELF
sections for elfutils libelf:
https://lists.fedorahosted.org/pipermail/elfutils-devel/2015-October/005335.html

At first I had tried to implement a proposal based on data level access
with variants of the elf_(get|raw|new)_[z]data functions. That does make
handling mixed compressed/uncompressed more flexible because you could
mix accessing/adding compressed/decompressed data without having to put
the section in a compressed or decompressed mode. But it did cause an
explosion of different accessor/setter functions that were not
immediately intuitive for the user. And it made the implementation
somewhat more complex because you have to keep track of the different
compressed/decompressed data chunks. In the end mixing access didn't
actually seem to happen a lot. (Also I lost that prototype in a silly
typo accident. And yeah, I didn't have backups, sigh.)

So this prototype is more like what Ali proposed initially. But changed
a bit based on how I found things a bit more natural given how I used it
in different programs using libelf. Please let me know which changes
do/don't make sense.

The Elf types and structures are obviously the same:

#define SHF_COMPRESSED (1 << 11) /* Section with compressed data. */

/* Section compression header. Used when SHF_COMPRESSED is set. */

typedef struct
{
Elf32_Word ch_type; /* Compression format. */
Elf32_Word ch_size; /* Uncompressed data size. */
Elf32_Word ch_addralign; /* Uncompressed data alignment. */
} Elf32_Chdr;

typedef struct
{
Elf64_Word ch_type; /* Compression format. */
Elf64_Word ch_reserved;
Elf64_Xword ch_size; /* Uncompressed data size. */
Elf64_Xword ch_addralign; /* Uncompressed data alignment. */
} Elf64_Chdr;

The thing I am missing here is the entsize of the (uncompressed) data in
the section. Currently I am assuming the sh_entsize is set for the
uncompressed case and should be ignored when SHF_COMPRESSED is set. We
might want to clarify that in gabi. Or maybe it isn't too late and we
can just add it as a field to the header/struct. This matters for
example when compressing a symtab symbol table section. You cannot set
the sh_entsize to 1 when compressing because you then don't have a way
to get the original value back. Originally elfutils rejected sections
where the sh_size wasn't a multiple of sh_entsize, but with compressed
sections that can happen now.

/* Legal values for ch_type (compression algorithm). */
#define ELFCOMPRESS_ZLIB 1 /* ZLIB/DEFLATE algorithm. */
#define ELFCOMPRESS_LOOS 0x60000000 /* Start of OS-specific. */
#define ELFCOMPRESS_HIOS 0x6fffffff /* End of OS-specific. */
#define ELFCOMPRESS_LOPROC 0x70000000 /* Start of processor-specific. */
#define ELFCOMPRESS_HIPROC 0x7fffffff /* End of processor-specific. */

Since these give the algorithm used I changed the meaning of the
libelf.h compression type enum slightly:

/* Different compression types a section can have. */

typedef enum
{
ELF_ZSCN_T_NONE, /* No compression. */
ELF_ZSCN_T_GNU, /* Old GNU style (Always uses ZLIB, no real Chdr). */
ELF_ZSCN_T_ELF, /* ELF style (with Chdr giving compression type). */
ELF_ZSCN_T_NUM /* Number of different compression types. */
} Elf_ZScn_Type;

So they type is just the compression style/header (GNU or ELF) used and
the actual compression type is independent from that. It doesn't really
matter a lot in practice there is just one compression type anyway. But
it felt slightly cleaner/clearer.

Instead of the identify/compressible functions I found it more useful to
have accessors for the Chdr and Elf_Zscn_Type used. Also I think libelf
shouldn't have section name based and/or platform specific section type
functions if it can be avoided. To easily handle the 32/64 variants
there are 3 functions like for other such structures.

/* Returns compression header for a section if section data is
compressed. Returns the compression type (Elf_ZScn_Type) in
TYPE. If the type is ELF_ZSCN_T_GNU then the returned
compression header is artificial (since the old GNU style
compressed section data didn't contain a real header). The ch_type
of such an artificial header is always ELFCOMPRESS_ZLIB and the
ch_addralign equals the sh_addralign of the Shdr of the section.
If an error occurs NULL is returned and TYPE will be -1.
Allocated or no bits sections are never compressed. Requesting the
Chdr for a section that isn't compressed returns NULL and sets
TYPE to ELF_ZSCN_T_NONE, elf_errno will be set to indicate the
section wasn't compressed. */
Elf32_Chdr *elf32_getchdr (Elf_Scn *scn, int *type);
Elf64_Chdr *elf64_getchdr (Elf_Scn *scn, int *type);

typedef Elf64_Chdr GElf_Chdr;
/* Get compression header and compression type of section data, if
any. Returns NULL and sets TYPE to -1 on failure. When the
section data isn't compressed TYPE is set to ELF_ZSCN_T_NONE and
NULL is returned. Otherwise DEST is returned and TYPE is
set. */
GElf_Chdr *gelf_getchdr (Elf_Scn *scn, GElf_Chdr *dest, int *type);

I merged the elf_zscn_(de)compress function into one by giving it an
argument to use for compression/decompression so you don't need two
different functions. I also made it more like the above getchdr function
by returning the updated shdr. Which means there are again 3 variants
for 32/64/gelf. I found it more natural to return the updated Shdr
because in practice programs often have multiple passes over the
sections. First they examine the section headers only and then based on
the information collected they make a second pass over the sections and
copy/adjust data depending on the results of the first pass. Only in a
later pass does the program have to switch the section to being
uncompressed. It then always gets the updated Shdr again so it makes
sense to just return it from elf_zscn_compress. Especially when using
the gelf_getshdr function it might otherwise not immediately be clear
one has to update their copy anyway.

/* When type is ELF_ZSCN_T_NONE then ch_type is ignored and the raw
section data is decompressed if the section was compressed before.
The returned section header will have an updated sh_size, sh_addralign
and will have SHF_COMPRESSED cleared from the sh_flags. sh_entsize
isn't changed.
If the section wasn't compressed, or an error occurred while
decompressing the section date NULL is returned and elf_err is set.

When type is ELF_ZSCN_T_GNU or ELF_ZSCN_T_ELF type must be a valid
ELFCOMPRESS algorithm. Currently the only valid value is
ELFCOMPRESS_ZLIB. If the section wasn't compressed yet then the
Elf_Data is converted to the raw format of the Elf image and
compressed using the given algorithm. The correct compression format
header will be added at the start of the new raw data for the section.
The returned section header will have an updated sh_size,
sh_addralign and if type was ELF_ZSCN_T_ELF sh_flags will have
SHF_COMPRESSED set. sh_entsize isn't changed.
If the section data was already compressed, or an unknown compression
type or algorithm was given or if an error occurred while compressing
NULL is returned and elf_err is set.

Too switch between GNU and ELF compression types or to use another
compression algorithm one has to uncompress the section first and
then compress using the other type/algorithm.

All previous returned Elf_Data buffers are invalidated by this call
and should no longer be accessed.

Note that although this changes the header and data it doesn't mark
the section as dirty. To keep the changes when calling elf_update the
section has to be flagged ELF_F_DIRTY. */
Elf32_Shdr *elf32_zscn_compress (Elf_Scn *scn, int type, int ch_type);
Elf64_Shdr *elf64_zscn_compress (Elf_Scn *scn, int type, int ch_type);

GElf_Shdr * gelf_zscn_compress (Elf_Scn *scn, GElf_Shdr *dest, int type,
int ch_type);

The above implies that the compression/decompression happens immediately
and any failure can be immediately observed (which is how I implemented
it in my prototype). Maybe we can reformulate it a bit so an
implementation can be more lazy and return success here even though a
later call to elf_(new|raw|get)data or elf_update might fail because the
conversion wasn't valid after all.

It might be good to be explicit about the d_type of compressed sections
returned by elf_getdata. Which is always ELF_T_BYTE and isn't matched
from the section type in case sh_flags has SHF_COMPRESSED set.

elf_strptr should always return the string at the index of the
uncompressed section data, even if the section is still compressed. This
shouldn't change the compression state of the string section nor should
it update the Shdr.

Cheers,

Mark

Ali Bahrami

unread,
Oct 22, 2015, 1:31:50 AM10/22/15
to gener...@googlegroups.com, Rod Evans
Hi Mark,

It's going to take me a few days to blow the dust of
my old workspace and get back to speed on what I had
proposed, and then see how it relates to what you're
putting forward here. It seems like you've moved things
pretty far down the field since we last exchanged mail,
which is great.

I'll provide some cursory feedback now, and ask a few questions,
and then go away to study things before coming back for a deeper
discussion.


> At first I had tried to implement a proposal based on data level access
> with variants of the elf_(get|raw|new)_[z]data functions. That does make
> handling mixed compressed/uncompressed more flexible because you could
> mix accessing/adding compressed/decompressed data without having to put
> the section in a compressed or decompressed mode. But it did cause an
> explosion of different accessor/setter functions that were not
> immediately intuitive for the user. And it made the implementation
> somewhat more complex because you have to keep track of the different
> compressed/decompressed data chunks. In the end mixing access didn't
> actually seem to happen a lot. (Also I lost that prototype in a silly
> typo accident. And yeah, I didn't have backups, sigh.)

We've discussed this, so I guess you know that I prefer the
simpler section based API. My reason is that I just can't see
a practical use for all that rope, and this stuff is complicated
enough as is. I really think Section is the right level to be
applying compression, rather than Data.

I am sorry you lost your work though --- we've all been there.
Hopefully you learned what you needed before it was lost.


>
> The thing I am missing here is the entsize of the (uncompressed) data in
> the section. Currently I am assuming the sh_entsize is set for the
> uncompressed case and should be ignored when SHF_COMPRESSED is set. We
> might want to clarify that in gabi. Or maybe it isn't too late and we
> can just add it as a field to the header/struct. This matters for
> example when compressing a symtab symbol table section. You cannot set
> the sh_entsize to 1 when compressing because you then don't have a way
> to get the original value back. Originally elfutils rejected sections
> where the sh_size wasn't a multiple of sh_entsize, but with compressed
> sections that can happen now.

First things first: It is definitely late to adding fields to
the compression header. We've been shipping product based around
the gABI compression for a couple of years now, so we have objects
in the field.

I can tell you though that we discussed the need for a separate sh_entsize
for the compressed and uncompressed modes, and found that it wasn't
necessary. sh_entsize is always the entsize for the uncompressed data.
There really is no entsize for the compressed form. Compressed data
consists of a compression header, followed by a stream of uninterpreted
bytes that are private to the compression engine.

I should note, tangentially, that individual compression algorithms are
free to define their own headers --- everything in the data that follows
the ELF_Chdr header belongs to the algorithm. Hence, if some particular
form of compression has a need for an entsize like piece of metadata,
it has a place to put that. The addition of ch_addralign was really to
support this --- those extra headers might have strict alignment
requirements. The generic part of libelf would have to know the
alignment, but it would have no need for the rest of that data.

I think the gABI is already clear about sh_entsize, but documentation
can always be improved.



> Since these give the algorithm used I changed the meaning of the
> libelf.h compression type enum slightly:
>
> /* Different compression types a section can have. */
>
> typedef enum
> {
> ELF_ZSCN_T_NONE, /* No compression. */
> ELF_ZSCN_T_GNU, /* Old GNU style (Always uses ZLIB, no real Chdr). */
> ELF_ZSCN_T_ELF, /* ELF style (with Chdr giving compression type). */
> ELF_ZSCN_T_NUM /* Number of different compression types. */
> } Elf_ZScn_Type;
>
> So they type is just the compression style/header (GNU or ELF) used and
> the actual compression type is independent from that. It doesn't really
> matter a lot in practice there is just one compression type anyway. But
> it felt slightly cleaner/clearer.

I'll have to think about this, but at first blush, this defeats my
purpose for defining Elf_ZScn_Type.

I wanted one type that could unify the GNU and gABI compression
options --- a single value that would capture all the possibilities.
That was needed for the identify operation in particular.
Hence, I proposed

typedef enum {
ELF_ZSCN_T_NONE = 0, /* No compression applied. must be first */
ELF_ZSCN_T_ZLIB_GNU, /* ZLIB (GNU-style, deprecated) */
ELF_ZSCN_T_ZLIB, /* ZLIB (ELF gABI) */
ELF_ZSCN_T_NUM /* must be last */
} Elf_ZScn_Type;

The idea being that additional gABI types would go
after ELF_ZSCN_T_ZLIB.

The version you're proposing simply defines GNU and ELF,
but since I don't expect that there will ever be a third
variant at that level now that we have a standard gABI
format, I don't see how this is useful. A simple boolean
type could capture that.

I'll have to study your proposed interfaces in some detail
before I can say more on this. I suspect that how much I
end up caring about this is inversely proportional to how
much I buy into your proposal to eliminate the identify
and compressible operations. Let me go figure out where
I stand on that, and then we can pick this thread back up.

> I merged the elf_zscn_(de)compress function into one by giving it an
> argument to use for compression/decompression so you don't need two
> different functions.

I'm not sure two different functions are bad, but let me
think about it

> When type is ELF_ZSCN_T_GNU or ELF_ZSCN_T_ELF type must be a valid
> ELFCOMPRESS algorithm. Currently the only valid value is
> ELFCOMPRESS_ZLIB.
...
Elf32_Shdr *elf32_zscn_compress (Elf_Scn *scn, int type, int ch_type);
Elf64_Shdr *elf64_zscn_compress (Elf_Scn *scn, int type, int ch_type);

I think you mean "ch_type must be a valid ELCOMPRESS algorithm",
and not "Type must be".

If we were to add another gABI compression type, it would have no
valid GNU form, so an error would have to be returned. That's an
error case that wouldn't exist if Elf_ZScn_Type were defined as
I originally had it, and we would only need 2 arguments:

elf32_zscn_compress (Elf_Scn *scn, Elf_ZScnType type);

Why are type and ch_type defined as int, rather than Elf_ZScnType
and Elf32_Word/Elf64_Word respectively?


>
> Too switch between GNU and ELF compression types or to use another
> compression algorithm one has to uncompress the section first and
> then compress using the other type/algorithm.

Yes.

>
> All previous returned Elf_Data buffers are invalidated by this call
> and should no longer be accessed.

Yes.

>
> Note that although this changes the header and data it doesn't mark
> the section as dirty. To keep the changes when calling elf_update the
> section has to be flagged ELF_F_DIRTY. */
> Elf32_Shdr *elf32_zscn_compress (Elf_Scn *scn, int type, int ch_type);
> Elf64_Shdr *elf64_zscn_compress (Elf_Scn *scn, int type, int ch_type);
>
> GElf_Shdr * gelf_zscn_compress (Elf_Scn *scn, GElf_Shdr *dest, int type,
> int ch_type);

I think we discussed this before, and I agree.


>
> The above implies that the compression/decompression happens immediately
> and any failure can be immediately observed (which is how I implemented
> it in my prototype). Maybe we can reformulate it a bit so an
> implementation can be more lazy and return success here even though a
> later call to elf_(new|raw|get)data or elf_update might fail because the
> conversion wasn't valid after all.

I'd prefer the simpler immediate behavior. An application shouldn't
call these APIs if it doesn't intend to use the results, and immediate
errors are always better, especially for debugging.

>
> It might be good to be explicit about the d_type of compressed sections
> returned by elf_getdata. Which is always ELF_T_BYTE and isn't matched
> from the section type in case sh_flags has SHF_COMPRESSED set.

I'll have to look at our implementation, but I think the type might
actually be the type of the uncompressed data, and so, not
necessarily ELF_T_BYTE.

However, most of the sections to which compression should be
applied are going to be SHT_PROGBITS, and so, ELF_T_BYTE.

>
> elf_strptr should always return the string at the index of the
> uncompressed section data, even if the section is still compressed. This
> shouldn't change the compression state of the string section nor should
> it update the Shdr.

As we discussed previously, and I agree.

Of course, libelf really does have to decompress that section
internally to do this, but it would hide that from the user
until the user performed an explicit decompress.

Thanks...

- Ali

Ali Bahrami

unread,
Oct 22, 2015, 1:45:24 AM10/22/15
to gener...@googlegroups.com, Rod Evans
On 10/21/15 06:58 PM, Mark Wielaard wrote:
> I finally posted a prototype implementation for handling compressed ELF
> sections for elfutils libelf:
> https://lists.fedorahosted.org/pipermail/elfutils-devel/2015-October/005335.html


I see that in that posting, you said

I like to add a new program that lets
you [un]compress sections and/or convert between different compression types
next. That would also be a good testcase for the implementation.

That's definitely a useful utility to have, both for testing, and
as a general tool. We have elfcompress, as I've mentioned earlier,
and I consider it to be essential:

https://docs.oracle.com/cd/E36784_01/html/E36870/elfcompress-1.html

If you like it well enough, then I'd invite you to implement the
same command, which would help unify Linux and Solaris in a minor
but useful way.

If you don't like it, then I understand, but please consider giving
it a different name, to avoid the inevitable confusion.

Thanks...

- Ali

Ali Bahrami

unread,
Oct 23, 2015, 7:52:26 PM10/23/15
to Mark Wielaard, gener...@googlegroups.com, Rod Evans, Rainer Orth
Hi Mark,

I didn't hear anything back from my earlier comments from
Wednesday, but having looked at this proposal in a bit more detail,
I think I wasn't too far off in my assumptions. I'm now ready to have
a deeper conversation about this.

I find that both of our proposals are similar. I regret to say that I
don't love either proposal very much. I don't regret the effort, as I
think we needed to go through the exercise of defining them in order to
explore the issues. I also think that the problems are all about syntax,
and so, the code we've written will not be wasted.

The problem with both is that they wrestle somewhat unsuccessfully with
how to integrate support for the original GNU format. We believe that the
gABI format will eclipse the older one, and that the old one will fade away.
However, our proposed APIs cement the support for that old stuff into
core libelf routines in a way that will persist. In the best case, we'll
end up with options that no one uses.

It's been months, so let's start with a quick review. I proposed this:

typedef enum {
ELF_ZSCN_T_NONE = 0, /* No compression applied. must be first */
ELF_ZSCN_T_ZLIB_GNU, /* ZLIB (GNU-style, deprecated) */
ELF_ZSCN_T_ZLIB, /* ZLIB (ELF gABI) */
ELF_ZSCN_T_NUM /* must be last */
} Elf_ZScn_Type;


int elf_zscn_identify(Elf_Scn *scn,
const char *scn_name);

int elf_zscn_compressible(Elf_Scn *scn,
Elf_ZScn_Type ztype,
const char *scn_name);

int elf_zscn_compress(Elf_Scn *scn,
Elf_ZScn_Type ztype,
const char *scn_name);

int elf_zscn_decompress(Elf_Scn *scn,
const char *scn_name);

Note that these are ELFCLASS neutral, and don't require 32/64/gelf variants.

You are countering with this:

typedef enum {
ELF_ZSCN_T_NONE, /* No compression. */
ELF_ZSCN_T_GNU, /* Old GNU style (Always uses ZLIB, no real Chdr) */
ELF_ZSCN_T_ELF, /* ELF style (with Chdr giving compression type) */
ELF_ZSCN_T_NUM /* Number of different compression types. */
} Elf_ZScn_Type;


Elf32_Chdr *elf32_getchdr (Elf_Scn *scn, int *type);
Elf64_Chdr *elf64_getchdr (Elf_Scn *scn, int *type);
GElf_Chdr *gelf_getchdr (Elf_Scn *scn, GElf_Chdr *dest,
int *type);


Elf32_Shdr *elf32_zscn_compress (Elf_Scn *scn, int type, int ch_type);
Elf64_Shdr *elf64_zscn_compress (Elf_Scn *scn, int type, int ch_type);
GElf_Shdr * gelf_zscn_compress (Elf_Scn *scn, GElf_Shdr *dest, int type,
int ch_type);

You also mentioned some internal implementation details (ELF_F_DIRTY,
elf_strptr) that we have already discussed and agreed upon. To me, it
looks like we are in conceptual agreement, but are still dancing around
issues of syntax, and again, largely trying to decide what to do with
the old GNU format.

I'd summarize your changes, along with my initial reactions, as follows:

(1) Combine compress and decompress into 1 routine

Assuming this can work out cleanly, I think it's OK.

(2) Return a section header from compress operations

This might be a convenience, but it requires ELFCLASS-specific
routines, which I think isn't worth the other convenience.

(3) Specify compression as 2 values: (1) Style (GNU or gABI) and
algorithm (none, zlib, etc...).

A couple of things:

a) It suggests that additional styles, in addition to
ELF_ZSCN_T_GNU and ELF_ZSCN_T_ELF, might be possible.
I want to discourage that.

b) 2 arguments to specify a type seems like 1 too many.

This seems like the first issue we need to focus on, as it
ripples through a lot of things. See below.

(4) Eliminate the compressible query routine.

You've made the point before that libelf should not be in the
position of policing what gets compressed, and what cannot. I
agree with that principle. However, I am not content to force
individual developers to make these decisions, because I think
that will result in unnecessary confusion and chaos. Most
developers just want to do the standard correct thing, and would
value some guidance. If we don't provide it here, who will?

I think the original GNU format support is really at the root
of this concern as well. We wouldn't need to provide guidance
if things were simpler.

(5) Express the identify operation differently, with routines,
elfXX_getchdr(), that return the compression header, and give
the style (GNU vs gABI).

Returning the Chdr is a convenience, but note that in a gABI
compressed section, it's at the top of the data, and easily
accessible. There are 2 things about this routine that I would
rather avoid:

(a) the need to allocate and manage the memory for
a fake Chdr struct in the GNU format case.
(b) The need for 3 ELFCLASS-specific variants.

At a deeper level, this is really a symptom of the same problem
we face in (3).

It's clear that the original GNU format is the main complication
that is causing otherwise simple APIs to mutate unpleasantly, and the
need for Elf_ZScn_Type lies at the heart of this. In my proposal, we're
mapping the ELFCOMPRESS types, with an extra GNU type, into yet another
"unified" type. That's an unwelcome indirection.

In your proposal, you have avoided the ugliness I accepted, but have
accepted a different one by requiring 2 types to be specified (GNU vs gABI,
and the type of compression algorithm).

Maybe we're trying too hard to unify this stuff. Let's pause, and consider
what an API that supports only the gABI form of compression might be. We
might start with a stripped down version of my original proposal, which
no longer needs section names, nor an invented new Elf_ZScn_Type:

int elf_zscn_identify(Elf_Scn *scn);

int elf_zscn_compressible(Elf_Scn *scn, int type);

int elf_zscn_compress(Elf_Scn *scn, int type);

int elf_zscn_decompress(Elf_Scn *scn);

The type argument in both cases is now simply one of the gABI ELFCOMPRESS_*
constants. At this time, that means ELFCOMPRESS_ZLIB, since that's the only
one we've defined:

/* Legal values for ch_type (compression algorithm). */
#define ELFCOMPRESS_ZLIB 1 /* ZLIB/DEFLATE algorithm. */

Note that while I didn't define an ELFCOMPRESS_NONE code when I proposed
the gABI format, I did reserve the value of 0 for that meaning. We could
amend the gABI to formalize that:

#define ELFCOMPRESS_NONE 0 /* Uncompressed data */
#define ELFCOMPRESS_ZLIB 1 /* ZLIB/DEFLATE algorithm */

The reason I didn't define NONE in the first place is that I didn't want
to see objects creating SHF_COMPRESSED sections with a Chdr, and then
find ELFCOMPRESS_NONE being specified the specified type. Let's assume
however, that no one would be dumb enough to do that. Then, we could take
your idea and have just one routine for compress and decompress.

In this simpler scenario, my concerns about identifying the compression
type go away: It's just looking for SHF_COMPRESSED, followed by looking
at the compression header at the top of the Data. We no longer need an
identify function.

Similarly, my concern about giving developers guidance on what to compress
and what not to become rather less important. The general advice that
"you should compress non-SHF_ALLOC with names starting with .debug" seems
good enough. We can dispense with "compressible".

Put it all together and you end up with this compression API:

int elf_compress(Elf_Scn *scn, int type);

That's genuinely simple. I really like this far more than
any option we've considered so far. What do you think?

-----

What about the old GNU format?

I wish we could pretend that it never happened. On Solaris, we almost
can, as we don't have an installed base of compressed objects in that
format. On GNU, perhaps we could, but it feels a bit more aggressive
than we might like. My decision 2-3 years ago to support that format
on Solaris reflects my judgment that while we don't want to encourage
it, and we hope it dies out, we don't want to force anyone off a cliff.

So, I think we have to handle it. And, I think that each of us should want
to have the code that does that in libelf, for 2 reasons:

- The implementation of ZLIB for the GNU and gABI forms is 98%
the same code.

- Centralized location

Solaris already has that code in libelf, but its all private, and used
only by our ld and elfcompress utilities.

We could agree to each solve this our own way. Standardizing the gABI
is very important to us. Standardizing support for the old GNU format
is considerably less so.

On the other hand, it's clearly good overall to reduce arbitrary
differences that don't matter, so maybe we should try. We could agree
to a small purpose-built API to handle the old format. Perhaps this
would suffice?

int elf_gnu_zsec_identify(Elf_Scn *scn, *name);

Return True (1) if the given section, with its name
given by name, is compressed in the original GNU ZLIB
format, and False (0) otherwise.

int elf_gnu_zsec_compress(Elf_Scn *scn, int inflate);

Set inflate to True (1) to compress, and False (0) to
decompress. The caller is responsible for ensuring that
scn has a name starting with .debug when uncompressed,
and .zdebug when compressed. The caller is also responsible
for changing the name when the compression status is changed.


Supporting the GNU format this way has 2 benefits:

- Allows support for the gABI format to consist of one
function.

- Provides the support for the old GNU format with a couple
of routines that can fade from view as that old format dies,
and which might even be deprecated some day.


Let me know what you think. If this seems promising, I'll try
to implement it, as should you, and we can compare notes. Unlike
my previous proposal, this is just a paper exercise so far, so we're
going to want to try it before committing fully. It should be a
short leap from where we are now to this, so that shouldn't take long.

Thank you for your work on this.

- Ali

H.J. Lu

unread,
Oct 23, 2015, 8:00:19 PM10/23/15
to Generic System V Application Binary Interface, Mark Wielaard, Rod Evans, Rainer Orth
On Fri, Oct 23, 2015 at 4:52 PM, Ali Bahrami <Ali.B...@oracle.com> wrote:
> Hi Mark,
>
> I didn't hear anything back from my earlier comments from
> Wednesday, but having looked at this proposal in a bit more detail,
> I think I wasn't too far off in my assumptions. I'm now ready to have
> a deeper conversation about this.
>
> I find that both of our proposals are similar. I regret to say that I
> don't love either proposal very much. I don't regret the effort, as I
> think we needed to go through the exercise of defining them in order to
> explore the issues. I also think that the problems are all about syntax,
> and so, the code we've written will not be wasted.
>
> The problem with both is that they wrestle somewhat unsuccessfully with
> how to integrate support for the original GNU format. We believe that the
> gABI format will eclipse the older one, and that the old one will fade away.
> However, our proposed APIs cement the support for that old stuff into
> core libelf routines in a way that will persist. In the best case, we'll
> end up with options that no one uses.
>

I am perfectly fine without GNU format support. Given that Mark only
has partial support for GNU format, there is no big loss. In any case,
we can always run objcopy to convert GNU format to gABI format if
needed.

--
H.J.

Mark Wielaard

unread,
Oct 26, 2015, 3:05:15 PM10/26/15
to gener...@googlegroups.com, Rod Evans
On Wed, 2015-10-21 at 23:34 -0600, Ali Bahrami wrote:
> We've discussed this, so I guess you know that I prefer the
> simpler section based API. My reason is that I just can't see
> a practical use for all that rope, and this stuff is complicated
> enough as is. I really think Section is the right level to be
> applying compression, rather than Data.
>
> I am sorry you lost your work though --- we've all been there.
> Hopefully you learned what you needed before it was lost.

It was probably good, because I was slightly stuck with the old
implementation. This at least got me unstuck and I learned the section
based approach is slightly simpler to implement, but slightly harder to
reason about (because you have to make decisions about what holds in
which (un)compressed state of the section). IMHO.

> > The thing I am missing here is the entsize of the (uncompressed) data in
> > the section. Currently I am assuming the sh_entsize is set for the
> > uncompressed case and should be ignored when SHF_COMPRESSED is set. We
> > might want to clarify that in gabi. Or maybe it isn't too late and we
> > can just add it as a field to the header/struct. This matters for
> > example when compressing a symtab symbol table section. You cannot set
> > the sh_entsize to 1 when compressing because you then don't have a way
> > to get the original value back. Originally elfutils rejected sections
> > where the sh_size wasn't a multiple of sh_entsize, but with compressed
> > sections that can happen now.
>
> First things first: It is definitely late to adding fields to
> the compression header. We've been shipping product based around
> the gABI compression for a couple of years now, so we have objects
> in the field.

Yeah, I was afraid of that.

> I can tell you though that we discussed the need for a separate sh_entsize
> for the compressed and uncompressed modes, and found that it wasn't
> necessary. sh_entsize is always the entsize for the uncompressed data.
> There really is no entsize for the compressed form. Compressed data
> consists of a compression header, followed by a stream of uninterpreted
> bytes that are private to the compression engine.

Yes, that does make sense.
But there is a slight risk that there is some code out there (like I
found) which doesn't know about SHF_COMPRESSED and could have handled
the data as if fine, but does a sanity check to see whether the
sh_entsize makes sense (that sh_size is a multiple of sh_entsize). It is
easy to correct such assumptions though.

> I should note, tangentially, that individual compression algorithms are
> free to define their own headers --- everything in the data that follows
> the ELF_Chdr header belongs to the algorithm. Hence, if some particular
> form of compression has a need for an entsize like piece of metadata,
> it has a place to put that. The addition of ch_addralign was really to
> support this --- those extra headers might have strict alignment
> requirements. The generic part of libelf would have to know the
> alignment, but it would have no need for the rest of that data.

I rather avoid defining new compression algorithms/extended headers.
Lets just go with sh_entsize always applies to the uncompressed
sh/ch_size.

> I think the gABI is already clear about sh_entsize, but documentation
> can always be improved.

We could make the SHF_COMPRESSED description a bit more clear. Currently
it only calls out relocations against compressed section data. But don't
say anything about other references or properties related to the
(uncompressed) section data. It might be more clear to explicitly say
which properties don't apply against the uncompressed data. Maybe
something like:

"For a SHF_COMPRESSED section all section properties, except sh_size and
sh_addralign, and all indexes into the section data, like table indexes,
relocations, etc. apply to the uncompressed data."
So, how would you encode a variant that does have sh_entsize added to
the compression header/data for example? Would you just say that is the
ELF gABI format with a different ch_type? The reason I am asking is
because ch_type is explicitly defined as specifying the compression
algorithm. I hadn't immediately thought of it as specifying a different
format/header.

I might have over-designed things. But I was thinking we might want the
same structure/info with different algorithms (ZLIB, BZIP2, LZMA,
Brotli, etc.) and some way to indicate different headers/properties/data
(GNU, ELF, ELF+entsize, etc.). It seemed cleaner because the first is an
ELF/gabi concept, while the second is a libelf concept.

But I agree that it is unlikely that any of these will ever see another
alternative value. So using one type/argument for both seems fine.

> > When type is ELF_ZSCN_T_GNU or ELF_ZSCN_T_ELF type must be a valid
> > ELFCOMPRESS algorithm. Currently the only valid value is
> > ELFCOMPRESS_ZLIB.
> ...
> Elf32_Shdr *elf32_zscn_compress (Elf_Scn *scn, int type, int ch_type);
> Elf64_Shdr *elf64_zscn_compress (Elf_Scn *scn, int type, int ch_type);
>
> I think you mean "ch_type must be a valid ELCOMPRESS algorithm",
> and not "Type must be".

Yes. That was a typo indeed.

> If we were to add another gABI compression type, it would have no
> valid GNU form, so an error would have to be returned. That's an
> error case that wouldn't exist if Elf_ZScn_Type were defined as
> I originally had it, and we would only need 2 arguments:
>
> elf32_zscn_compress (Elf_Scn *scn, Elf_ZScnType type);
>
> Why are type and ch_type defined as int, rather than Elf_ZScnType
> and Elf32_Word/Elf64_Word respectively?

Honestly simply because no other libelf interface uses enums.
There is also a slight abi risk with using enums in interfaces since
some compilers/arches might change the size depending on the number of
elements in the enumeration/bits needed to encode the enum values (see
e.g. -fshort-enums for gcc, which for some reason is the default on some
arm configurations). So in general I try to avoid them as function
arguments. Also since int is signed it is convenient, like in the
getchdr case, to use it when returning a value to indicate an error (a
value outside the enumeration).

> > The above implies that the compression/decompression happens immediately
> > and any failure can be immediately observed (which is how I implemented
> > it in my prototype). Maybe we can reformulate it a bit so an
> > implementation can be more lazy and return success here even though a
> > later call to elf_(new|raw|get)data or elf_update might fail because the
> > conversion wasn't valid after all.
>
> I'd prefer the simpler immediate behavior. An application shouldn't
> call these APIs if it doesn't intend to use the results, and immediate
> errors are always better, especially for debugging.

Agreed. But then there should be a simple way to get at all the other
information that isn't inside the compressed data. Which is why I added
the getchdr functions.

> > It might be good to be explicit about the d_type of compressed sections
> > returned by elf_getdata. Which is always ELF_T_BYTE and isn't matched
> > from the section type in case sh_flags has SHF_COMPRESSED set.
>
> I'll have to look at our implementation, but I think the type might
> actually be the type of the uncompressed data, and so, not
> necessarily ELF_T_BYTE.

I think that is confusing, if the (raw) uncompressed data is returned it
should IMHO always be ELF_T_BYTE. In any case it should be clearly
documented.

Thanks,

Mark

Ali Bahrami

unread,
Oct 26, 2015, 4:22:17 PM10/26/15
to gener...@googlegroups.com
Hi Mark,


> I rather avoid defining new compression algorithms/extended headers.
> Lets just go with sh_entsize always applies to the uncompressed
> sh/ch_size.

I agree --- in fact there's nothing today for us to define, as
ZLIB, our only algorithm, doesn't need anything like that.

I mentioned that only to note that there is a way for something
like this to be added later for an algorithm that might need
it in a backward compatible manner.


>
>> I think the gABI is already clear about sh_entsize, but documentation
>> can always be improved.
>
> We could make the SHF_COMPRESSED description a bit more clear. Currently
> it only calls out relocations against compressed section data. But don't
> say anything about other references or properties related to the
> (uncompressed) section data. It might be more clear to explicitly say
> which properties don't apply against the uncompressed data. Maybe
> something like:
>
> "For a SHF_COMPRESSED section all section properties, except sh_size and
> sh_addralign, and all indexes into the section data, like table indexes,
> relocations, etc. apply to the uncompressed data."


The gABI describes the fields that have an alternative meaning when
SHF_COMPRESSED is set. It seems clear to me that if no alternative
meaning is defined for a given field, then that field doesn't have
one.

That said, I've no problem with the above recap.



>>
>> The version you're proposing simply defines GNU and ELF,
>> but since I don't expect that there will ever be a third
>> variant at that level now that we have a standard gABI
>> format, I don't see how this is useful. A simple boolean
>> type could capture that.
>
> So, how would you encode a variant that does have sh_entsize added to
> the compression header/data for example? Would you just say that is the
> ELF gABI format with a different ch_type? The reason I am asking is
> because ch_type is explicitly defined as specifying the compression
> algorithm. I hadn't immediately thought of it as specifying a different
> format/header.


I would not choose to have another variant. Chdr is intended to
provide a least common denominator base, and individual algorithms
are expected to add their own additional metadata following the Chdr.

As discussed above, we don't need ch_entsize: Compression as we know it is
a byte stream, so there is no entry size to worry about. If an algorithm
of the future did need something like that, it could have it in its
own algorithm-specific header. I'd give the same answer for any
section header field.


>
> I might have over-designed things. But I was thinking we might want the
> same structure/info with different algorithms (ZLIB, BZIP2, LZMA,
> Brotli, etc.) and some way to indicate different headers/properties/data
> (GNU, ELF, ELF+entsize, etc.). It seemed cleaner because the first is an
> ELF/gabi concept, while the second is a libelf concept.
>
> But I agree that it is unlikely that any of these will ever see another
> alternative value. So using one type/argument for both seems fine.

You'll have seen my proposal to throw away Elf_ZScn_Type completely,
so perhaps this is all moot?

The decision made when we added this stuff to the gABI was that the
Chdr would be fixed and standard, and algorithms would add their own additional
metadata. That was basically a bet that algorithms needing more would be
rare, and the complexity should be paid for by them. I think (and hope)
that this turns out to be true. I also hope that there won't be an
explosion of additional algorithms, as I think fewer would be better,
and ZLIB is fine.

Whether what you're proposing is cleaner or not comes down to whether or
not that bet is right.





> Honestly simply because no other libelf interface uses enums.
> There is also a slight abi risk with using enums in interfaces since
> some compilers/arches might change the size depending on the number of
> elements in the enumeration/bits needed to encode the enum values (see
> e.g. -fshort-enums for gcc, which for some reason is the default on some
> arm configurations). So in general I try to avoid them as function
> arguments. Also since int is signed it is convenient, like in the
> getchdr case, to use it when returning a value to indicate an error (a
> value outside the enumeration).

I think C defines enums as int, but I wasn't thinking about -fshort-enums.
That's a dubious option for gcc to apply to system headers, from an
ABI point of view --- If I felt strongly about it, I think we might debate
whether that was a compiler bug. :-)

I don't feel strongly about it though. int works, and keeping existing
code working is important.


> Agreed. But then there should be a simple way to get at all the other
> information that isn't inside the compressed data. Which is why I added
> the getchdr functions.

Outside of diagnostic tools and libelf itself, I don't think anything really
has a need to look in the Chdr. If they do though, ELF guarantees that it's
always at offset 0 of the data, so:

Elf32_Chdr *chdr = data;


Putting the compression header in the data wouldn't be my choice if we
were inventing ELF from scratch, but given that it's there, I don't see
why a function is needed to do this cast.


>>> It might be good to be explicit about the d_type of compressed sections
>>> returned by elf_getdata. Which is always ELF_T_BYTE and isn't matched
>>> from the section type in case sh_flags has SHF_COMPRESSED set.
>>
>> I'll have to look at our implementation, but I think the type might
>> actually be the type of the uncompressed data, and so, not
>> necessarily ELF_T_BYTE.
>
> I think that is confusing, if the (raw) uncompressed data is returned it
> should IMHO always be ELF_T_BYTE. In any case it should be clearly
> documented.


Yes, no question, it should be stated clearly.
I vote for compression to not alter d_type.

In my opinion, compression is primarily for debug data, and particularly,
is a tool to tame DWARF verbosity. Debug are all SHT_PROGBITS, so they're
always ELF_T_BYTE. In this degenerate case, both views yield the same result,
which is convenient.

Outside of PROGBITS, I'd find it more confusing to see, for instance,
an SHT_SYMTAB section reporting ELF_T_BYTE rather than ELF_T_SYM. If you
don't agree with this, then I'd ask why does ELF_T_BYTE seem better?
Aren't they really opaque compressed data, rather than BYTES? Consider
compressed debug data --- wouldn't ELF_T_BYTE make a compression-unaware
program think that there's valid data to be read, and then fall over
trying? Wouldn't we really need a special new type (ELF_T_ZSTREAM?),
in order to avoid that? Of course, this old code doesn't know about
ELF_T_ZSTREAM either, so that could also end badly.

I think that compression unaware code that doesn't read debug data will have
no problems with SHF_COMPRESSED sections, but any such code that actually
reads these sections will have to be updated to learn to decompress them
before access. That seems unavoidable, whatever we decide about elf_getdata().
Fortunately, this is mainly debuggers, and there aren't many of them. We'll
have to update them as we would any for any new ELF feature, and already
have for the most part.

Thanks...

- Ali

Mark Wielaard

unread,
Oct 26, 2015, 4:48:26 PM10/26/15
to Ali Bahrami, gener...@googlegroups.com, Rod Evans, Rainer Orth
Hi,

I should have read this message first before responding to your other
message. I agree almost 99% with your analysis, and the other 1% isn't
worth it to nitpick over. So...

On Fri, Oct 23, 2015 at 05:52:13PM -0600, Ali Bahrami wrote:
> Put it all together and you end up with this compression API:
>
> int elf_compress(Elf_Scn *scn, int type);
>
> That's genuinely simple. I really like this far more than
> any option we've considered so far. What do you think?

Awesome. Lets just pick this.

I am slightly sad there isn't a convenience function to get the
chdr or uncompressed data properties. But you are right that one
can get that by just peeking at the start of the compressed data
section. Having a 1 function api however is so nice that I won't
press for anything more.

> What about the old GNU format?
>
> I wish we could pretend that it never happened. On Solaris, we almost
> can, as we don't have an installed base of compressed objects in that
> format. On GNU, perhaps we could, but it feels a bit more aggressive
> than we might like. My decision 2-3 years ago to support that format
> on Solaris reflects my judgment that while we don't want to encourage
> it, and we hope it dies out, we don't want to force anyone off a cliff.

On some GNU/Linux distros people have been using the format
and elfutils does support it. But not in libelf, which makes it less
transparent, and it doesn't really handle any relocations in ET_REL
files that use it. Some other tools binutils/gdb also support it.
So having some support for it would be nice. Having the support in
libelf would be somewhat convenient, but not really essential IMHO.

> On the other hand, it's clearly good overall to reduce arbitrary
> differences that don't matter, so maybe we should try. We could agree
> to a small purpose-built API to handle the old format. Perhaps this
> would suffice?
>
> int elf_gnu_zsec_identify(Elf_Scn *scn, *name);
>
> Return True (1) if the given section, with its name
> given by name, is compressed in the original GNU ZLIB
> format, and False (0) otherwise.
>
> int elf_gnu_zsec_compress(Elf_Scn *scn, int inflate);
>
> Set inflate to True (1) to compress, and False (0) to
> decompress. The caller is responsible for ensuring that
> scn has a name starting with .debug when uncompressed,
> and .zdebug when compressed. The caller is also responsible
> for changing the name when the compression status is changed.

Lets go for as simple as possible here too.
Given that elf_gnu_zsec_identify is basically just
return strncmp(".zdebug_", name, strlen (".zdebug_")) == 0;
I think we can just drop that function and assume the user can/will do
that themselves, just like they would check whether the section has
SHF_COMPRESSED set. Then we would again have a simple 1 function API with
just elf_gnu_zsec_compress.

> Let me know what you think. If this seems promising, I'll try
> to implement it, as should you, and we can compare notes. Unlike
> my previous proposal, this is just a paper exercise so far, so we're
> going to want to try it before committing fully. It should be a
> short leap from where we are now to this, so that shouldn't take long.

I'll adapt my code to this new interface. Should be fun.

Cheers,

Mark

Mark Wielaard

unread,
Oct 26, 2015, 5:26:33 PM10/26/15
to gener...@googlegroups.com
On Mon, Oct 26, 2015 at 02:22:11PM -0600, Ali Bahrami wrote:
> >Agreed. But then there should be a simple way to get at all the other
> >information that isn't inside the compressed data. Which is why I added
> >the getchdr functions.
>
> Outside of diagnostic tools and libelf itself, I don't think anything really
> has a need to look in the Chdr. If they do though, ELF guarantees that it's
> always at offset 0 of the data, so:
>
> Elf32_Chdr *chdr = data;
>
>
> Putting the compression header in the data wouldn't be my choice if we
> were inventing ELF from scratch, but given that it's there, I don't see
> why a function is needed to do this cast.

Because what libelf users probably want is GElf_Chdr, so they can work
with the ELF file independent from its class. So it really would be just
to avoid a couple of lines a code to do the correct cast. And to get at
the data with libelf you basically have to read the whole section data
to get the Elf_Data first. If you really are only interested in the Chdr,
and not use any of the actually compressed data then libelf could be only
lazily read the necessary data from disk. But given that I really like the
1 function api I won't try to convince you.

> >>>It might be good to be explicit about the d_type of compressed sections
> >>>returned by elf_getdata. Which is always ELF_T_BYTE and isn't matched
> >>>from the section type in case sh_flags has SHF_COMPRESSED set.
> >>
> >>I'll have to look at our implementation, but I think the type might
> >>actually be the type of the uncompressed data, and so, not
> >>necessarily ELF_T_BYTE.
> >
> >I think that is confusing, if the (raw) uncompressed data is returned it
> >should IMHO always be ELF_T_BYTE. In any case it should be clearly
> >documented.
>
> Yes, no question, it should be stated clearly.
> I vote for compression to not alter d_type.
>
> In my opinion, compression is primarily for debug data, and particularly,
> is a tool to tame DWARF verbosity. Debug are all SHT_PROGBITS, so they're
> always ELF_T_BYTE. In this degenerate case, both views yield the same result,
> which is convenient.

I feel it isn't so interesting for debug data sections to be honest.
Those are, at least on GNU/Linux mostly put into a separate file in the
first place. If you do that, then it makes more sense to just compress
that whole separate debug file instead of individual sections. In general
I think "pure" debug data should be kept as separate as possible from the
"real" ELF data. There are some efforts like GNU Debug Fission, which might
be adopted by DWARFv5, that make this even easier.
https://gcc.gnu.org/wiki/DebugFissionDWP

I think it is actually more interesting to compress things like the
symtab/strtab which people do sometimes want to keep in the main ELF file
even though they aren't used by the runtime loader.

> Outside of PROGBITS, I'd find it more confusing to see, for instance,
> an SHT_SYMTAB section reporting ELF_T_BYTE rather than ELF_T_SYM. If you
> don't agree with this, then I'd ask why does ELF_T_BYTE seem better?
> Aren't they really opaque compressed data, rather than BYTES?

Yes, they are opaque data. Which is why I believe they should be
ELF_T_BYTE. So programs that just copy some data between ELF files
(which might not yet know about SHF_COMPRESSED) should just threat it
as ELF_T_BYTE and not try to interpret the data at all. Unless you
explicitly uncompress the section, the data returned really isn't
ELF_T_SYM.

> Consider
> compressed debug data --- wouldn't ELF_T_BYTE make a compression-unaware
> program think that there's valid data to be read, and then fall over
> trying? Wouldn't we really need a special new type (ELF_T_ZSTREAM?),
> in order to avoid that? Of course, this old code doesn't know about
> ELF_T_ZSTREAM either, so that could also end badly.

Yes, that is a problem. But anything except ELF_T_BYTE would make the
issue worse IMHO.

Cheers,

Mark

Ali Bahrami

unread,
Oct 26, 2015, 5:51:42 PM10/26/15
to Mark Wielaard, gener...@googlegroups.com, Rod Evans, Rainer Orth
On 10/26/15 14:48, Mark Wielaard wrote:
> Hi,
>
> I should have read this message first before responding to your other
> message. I agree almost 99% with your analysis, and the other 1% isn't
> worth it to nitpick over. So...
>
> On Fri, Oct 23, 2015 at 05:52:13PM -0600, Ali Bahrami wrote:
>> Put it all together and you end up with this compression API:
>>
>> int elf_compress(Elf_Scn *scn, int type);
>>
>> That's genuinely simple. I really like this far more than
>> any option we've considered so far. What do you think?
>
> Awesome. Lets just pick this.


Great, let's try it and see what happens!


>
> I am slightly sad there isn't a convenience function to get the
> chdr or uncompressed data properties. But you are right that one
> can get that by just peeking at the start of the compressed data
> section. Having a 1 function api however is so nice that I won't
> press for anything more.


Had you pressed, I wouldn't have fought too hard, but yeah,
the power of 1 is pretty compelling.



>> On the other hand, it's clearly good overall to reduce arbitrary
>> differences that don't matter, so maybe we should try. We could agree
>> to a small purpose-built API to handle the old format. Perhaps this
>> would suffice?
>>
>> int elf_gnu_zsec_identify(Elf_Scn *scn, *name);
>>
>> Return True (1) if the given section, with its name
>> given by name, is compressed in the original GNU ZLIB
>> format, and False (0) otherwise.
>>
>> int elf_gnu_zsec_compress(Elf_Scn *scn, int inflate);
>>
>> Set inflate to True (1) to compress, and False (0) to
>> decompress. The caller is responsible for ensuring that
>> scn has a name starting with .debug when uncompressed,
>> and .zdebug when compressed. The caller is also responsible
>> for changing the name when the compression status is changed.
>
> Lets go for as simple as possible here too.
> Given that elf_gnu_zsec_identify is basically just
> return strncmp(".zdebug_", name, strlen (".zdebug_")) == 0;
> I think we can just drop that function and assume the user can/will do
> that themselves, just like they would check whether the section has
> SHF_COMPRESSED set. Then we would again have a simple 1 function API with
> just elf_gnu_zsec_compress.


Sure, that makes sense. The value is all in reusing the same
compression code --- the identify part isn't a big concern.

Given that we no longer have any other "zsec" functions for this
one to cluster with, should we perhaps rename it from
elf_gnu_zsec_compress() to elf_compress_gnu()? Making the
"_gnu" be a suffix means that elf_compress and elf_compress_gnu
will sort together lexically in manpage lists and other documentation.


> I'll adapt my code to this new interface. Should be fun.

Be sure to save a copy of the current code first! :-)

I'll do the same, and will plan on comparing notes with you next
week, assuming that's agreeable.

I'll also write a quick summary of what I think the full proposal
is, and send it as a separate message today, so that you can sanity
check it before we get too far into this.

Thanks...

- Ali

Ali Bahrami

unread,
Oct 26, 2015, 6:54:26 PM10/26/15
to gener...@googlegroups.com
> I feel it isn't so interesting for debug data sections to be honest.
> Those are, at least on GNU/Linux mostly put into a separate file in the
> first place. If you do that, then it makes more sense to just compress
> that whole separate debug file instead of individual sections. In general
> I think "pure" debug data should be kept as separate as possible from the
> "real" ELF data. There are some efforts like GNU Debug Fission, which might
> be adopted by DWARFv5, that make this even easier.
> https://gcc.gnu.org/wiki/DebugFissionDWP
>
> I think it is actually more interesting to compress things like the
> symtab/strtab which people do sometimes want to keep in the main ELF file
> even though they aren't used by the runtime loader.

We also have a separate debug file facility:

https://blogs.oracle.com/ali/entry/ancillary_objects_separate_debug_elf

I find that compression and ancillary objects are compatible concepts, and
combining them means that you can deliver smaller files, which is good
for network based packaging, and that the debugger an only decompress
what it actually wants to read, which is often not everything.

You could be right about compressing symbol tables being valuable.


>> Consider
>> compressed debug data --- wouldn't ELF_T_BYTE make a compression-unaware
>> program think that there's valid data to be read, and then fall over
>> trying? Wouldn't we really need a special new type (ELF_T_ZSTREAM?),
>> in order to avoid that? Of course, this old code doesn't know about
>> ELF_T_ZSTREAM either, so that could also end badly.
>
> Yes, that is a problem. But anything except ELF_T_BYTE would make the
> issue worse IMHO.


Perhaps anything else would be worse, but that minimizes the problem
rather than solving it, if indeed a solution is needed:

- Does code outside of libelf really use d_type this way, and if
so, is that something we need to support? Surely external code
is using the section header type (sh_type) to make these decisions?

As far as we know, d_type is mainly for libelf to use in
making xlate decisions. I tend to see d_type as a libelf
implementation detail. Our <libelf.h> has this:

/*
* Translation types
*/
typedef enum {
ELF_T_BYTE = 0, /* must be first, 0 */
ELF_T_ADDR,
...
ELF_T_NUM /* must be last */
} Elf_Type;

- Using ELF_T_BYTE for compressed data fails to stop the
problem for debug --- the largest category of compressed sections.

I still tend to think that its simpler overall to have d_type and
compression be orthogonal to each other, and that having d_type reflect
the type of the uncompressed data is OK. If that's not good enough, then
I think we should talk about adding a new code, ELF_T_ZDATA, to represent
it rather than mapping everything to ELF_T_BYTE.

Our Elf_Type has undoubtedly diverged from those of other systems, but as
long as we agree that this is an enum, and that no implementation is allowed
to assume a specific integer being assigned to any one of the items in that
enum, I think it would be OK.

Before we do that though, are you able to determine a rough sense of
how much d_type usage there is outside of libelf? We haven't hit anything
on Solaris yet, though that might be due to compression not being widely
used, rather than anything more fundamental. I know that our core linkers
code uses the section header sh_type rather than d_type in its decisions.

- Ali

Ali Bahrami

unread,
Oct 26, 2015, 8:29:41 PM10/26/15
to gener...@googlegroups.com
On 10/26/15 15:51, Ali Bahrami wrote:
> I'll also write a quick summary of what I think the full proposal
> is, and send it as a separate message today, so that you can sanity
> check it before we get too far into this.


I have attached a rough draft manpage for the 2 functions,
elf_compress(), and elf_compress_gnu(). I think it catches all
of the issues except:

- The special behavior of elf_strptr() when faced with
a compressed string table.

I agree that we can have elf_strptr() do an implicit decompress.
However, can we limit that to the gABI format, based on
SHF_COMPRESSED? Given the lack of name information, I'm not
comfortable with having libelf implicitly try to do a decompress
for GNU style. I think that this is what you intended, and is also
a non-issue, as string tables are not named with a .debug prefix,
but I thought I should ask.

- The d_type question. I've punted on this for the moment.

- Ali

elf_compress.3ELF

Mark Wielaard

unread,
Oct 27, 2015, 7:31:20 AM10/27/15
to gener...@googlegroups.com
On Mon, 2015-10-26 at 16:54 -0600, Ali Bahrami wrote:
> > I feel it isn't so interesting for debug data sections to be honest.
> > Those are, at least on GNU/Linux mostly put into a separate file in the
> > first place. If you do that, then it makes more sense to just compress
> > that whole separate debug file instead of individual sections. In general
> > I think "pure" debug data should be kept as separate as possible from the
> > "real" ELF data. There are some efforts like GNU Debug Fission, which might
> > be adopted by DWARFv5, that make this even easier.
> > https://gcc.gnu.org/wiki/DebugFissionDWP
> >
> > I think it is actually more interesting to compress things like the
> > symtab/strtab which people do sometimes want to keep in the main ELF file
> > even though they aren't used by the runtime loader.
>
> We also have a separate debug file facility:
>
> https://blogs.oracle.com/ali/entry/ancillary_objects_separate_debug_elf

O nice. I didn't know that. I should read and learn from that. Thanks.

> I find that compression and ancillary objects are compatible concepts, and
> combining them means that you can deliver smaller files, which is good
> for network based packaging, and that the debugger an only decompress
> what it actually wants to read, which is often not everything.

I do see your point about only having to decompress individual sections.
That is better than compressing the whole file if you assume only a few
sections are needed. In general it is a tradeoff between storing as
little (compressed) data as possible and quick access to the data. DWARF
consumers will often not even use the whole section data. gold/GDB for
example make sure there is a index into the DWARF DIE tree to quickly
get at the relevant debug info entries. That way GDB can simply mmap the
data and jump right to what they need without needing to read through
and parse all data beforehand. elfutils libdw likewise has ways to
traverse debug data by touching as little data as possible. If the
section (or whole file indeed) is compressed these techniques aren't
effective for quick access because first the whole data has to be
decompressed.

> >> Consider
> >> compressed debug data --- wouldn't ELF_T_BYTE make a compression-unaware
> >> program think that there's valid data to be read, and then fall over
> >> trying? Wouldn't we really need a special new type (ELF_T_ZSTREAM?),
> >> in order to avoid that? Of course, this old code doesn't know about
> >> ELF_T_ZSTREAM either, so that could also end badly.
> >
> > Yes, that is a problem. But anything except ELF_T_BYTE would make the
> > issue worse IMHO.
>
> Perhaps anything else would be worse, but that minimizes the problem
> rather than solving it, if indeed a solution is needed:
>
> - Does code outside of libelf really use d_type this way, and if
> so, is that something we need to support? Surely external code
> is using the section header type (sh_type) to make these decisions?

The code that I have seen uses d_type as an opaque value they just have
to copy. Assuming that just carrying it over from one Elf_Data to the
next will do the right thing. The simplest example is a program that
transforms some sections, but leaves everything else in tact. So they
read an Elf file, and copy over what is needed into a new Elf file
preserving the structure of the old file except for the sections
added/deleted/modified. A quick search shows the following pattern in a
couple of programs for section data they don't want to change:

Elf_Data *olddata = elf_getdata (oldscn, NULL);
Elf_Data *newdata = elf_newdata (newscn);
*newdata = *olddata;

This works fine as long as elf_getdata set up the d_type correctly for
the new data structure to be translated from memory to file
representation. But if the memory and file representation happen to be
different and the above section has SHF_COMPRESSED set then when writing
out the newdata will cause the compressed data to be "translated" into
the on disk representation... oops.

Of course what the programmer should have used if they are not
interested in the actual data of the section is elf_rawdata, which would
return an uninterpreted Elf_Data with d_type set to ELF_T_BYTE Elf_Data.
But elf_getdata/elf_newdata really is used as above. More than we will
like. Sorry.

So IMHO to not break such programs SHF_COMPRESSED section data should
have a d_type of ELF_T_BYTE. Otherwise we will have to document
exceptions for elf_update ELF_C_WRITE and/or elf*_xlatetom/xlatetof in
the case the Elf_Data belongs to a compressed section. But I think it is
clearer to just document that elf_getdata acts like elf_rawdata and sets
d_type to ELF_T_BYTE if the section is compressed.

> I think we should talk about adding a new code, ELF_T_ZDATA, to represent
> it rather than mapping everything to ELF_T_BYTE.

I think ELF_T_BYTE is already fine since it means uninterpreted data
that has the same memory and file representation. I am not sure what
extra meaning ELF_T_ZDATA adds.

Cheers,

Mark

Ali Bahrami

unread,
Oct 27, 2015, 9:57:10 PM10/27/15
to gener...@googlegroups.com
> On Oct 27, 2015, at 4:30 AM, Mark Wielaard <ma...@klomp.org> wrote:
>
> I think ELF_T_BYTE is already fine since it means uninterpreted data
> that has the same memory and file representation. I am not sure what
> extra meaning ELF_T_ZDATA adds.

I think you're right --- BYTE really just means "don't xlate". Let's do as you suggest. Sorry for missing the point
before.

Thanks...

- Ali

Ali Bahrami

unread,
Nov 18, 2015, 11:51:45 AM11/18/15
to gener...@googlegroups.com, Mark Wielaard
On 10/26/15 03:26 PM, Mark Wielaard wrote:
>> >Consider
>> >compressed debug data --- wouldn't ELF_T_BYTE make a compression-unaware
>> >program think that there's valid data to be read, and then fall over
>> >trying? Wouldn't we really need a special new type (ELF_T_ZSTREAM?),
>> >in order to avoid that? Of course, this old code doesn't know about
>> >ELF_T_ZSTREAM either, so that could also end badly.
> Yes, that is a problem. But anything except ELF_T_BYTE would make the
> issue worse IMHO.

A brief update, and then a question for Mark below...

Mark and I discussed this further offline, and came to the
conclusion that d_type should be set to ELF_T_CHDR for compressed
sections. On decompression, that gets changed to whatever other
ELF_T_* code applies to the section type in question. For debug
sections, that's generally going to be ELF_T_BYTE.

We also agreed to resurrect the three functions for getting
access to the compression header: elf32_getchdr(), elf64_getchdr(),
and gelf_getchdr(). These are not really necessary, as previously
discussed, but they are useful basic convenience functions, and
easily provided.

I've implemented the above, and it's working well. It required
me to do some significant refactoring, which was momentarily
inconvenient, but the result is simpler and I'm very happy
with it.

-----

Mark, I have a couple of questions for you, regarding
elf_compress() and elf_compress_gnu(). These are currently
defined as:

int elf_compress(Elf_Scn *scn, int type);
int elf_compress_gnu(Elf_Scn *, int inflate);

returning 0 for success, and -1 on error.

My first question is whether the type argument to elf_compress()
shouldn't really be uint_t rather than int? I don't see a use
for negative values here, and type maps internally to an ELF
Word, which is unsigned.

My second question is about adding another argument to these
functions to support a missing feature.

My internal Solaris implementation for compression supports a mode
in which when you compress a section, it will be compressed if the
result would be smaller than the uncompressed original data, and
which leaves the section uncompressed otherwise. We use this in
both ld, and the elfcompress utility. Since the main purpose of
compression is to make things smaller, I think this is a really
nice ability, but our current definition for elf_compress() and
elf_compress_gnu() don't support it, forcing compression even
when the result is larger. This seems seems unfortunate, and
is a reason why I wouldn't currently move off the internal stuff.

I'd like to make this mode of operation the default for these
compress routines, and provide a flags argument that allows for
compression to be forced, so that we can support both modes.

int elf_compress(Elf_Scn *scn, uint_t type, uint_t flags);
int elf_compress_gnu(Elf_Scn *, uint_t inflate, uint_t flags);

Where the valid values for flags are

#define ELF_CHF_FORCE 0x1

defined as:

By default, elf_compress() and elf_compress_gnu() will only
compress the specified section if the resulting section is
thereby reduced in size. Set ELF_CHF_FORCE to require that
the section always be compressed.

And the return value for these functions would be modified to:

-1 error
0 success: no compression done, section is unchanged
1 success: compression done, section is modified

How does that sound?

Thanks...

- Ali

Mark Wielaard

unread,
Nov 20, 2015, 11:20:14 AM11/20/15
to Ali Bahrami, gener...@googlegroups.com
On Wed, 2015-11-18 at 09:51 -0700, Ali Bahrami wrote:
> Mark and I discussed this further offline, and came to the
> conclusion that d_type should be set to ELF_T_CHDR for compressed
> sections. On decompression, that gets changed to whatever other
> ELF_T_* code applies to the section type in question. For debug
> sections, that's generally going to be ELF_T_BYTE.

Which means that the Chdr at the start of the elf_getdata Elf_Data
buffer is always translated to the in-memory representation. And the
[g]elf[32|64]xlateto(f|m) functions will translate the same Elf_Data
interpreting it as an Chdr that might need translation followed by
compression data (that doesn't need to be translated).

> We also agreed to resurrect the three functions for getting
> access to the compression header: elf32_getchdr(), elf64_getchdr(),
> and gelf_getchdr(). These are not really necessary, as previously
> discussed, but they are useful basic convenience functions, and
> easily provided.
>
> I've implemented the above, and it's working well. It required
> me to do some significant refactoring, which was momentarily
> inconvenient, but the result is simpler and I'm very happy
> with it.

I also needed some refactoring and the code is simpler. And I like that
it also makes more sense conceptually. You do have to watch out for some
unnecessary copying/xlating between rawdata and in-memory data though.
That is mostly an implementation detail though.

> Mark, I have a couple of questions for you, regarding
> elf_compress() and elf_compress_gnu(). These are currently
> defined as:
>
> int elf_compress(Elf_Scn *scn, int type);
> int elf_compress_gnu(Elf_Scn *, int inflate);
>
> returning 0 for success, and -1 on error.
>
> My first question is whether the type argument to elf_compress()
> shouldn't really be uint_t rather than int? I don't see a use
> for negative values here, and type maps internally to an ELF
> Word, which is unsigned.

Yes, using unsigned int should be fine. I assume that is what you call
uint_t, which isn't a standard type as far as I can see.

> My second question is about adding another argument to these
> functions to support a missing feature.
>
> My internal Solaris implementation for compression supports a mode
> in which when you compress a section, it will be compressed if the
> result would be smaller than the uncompressed original data, and
> which leaves the section uncompressed otherwise. We use this in
> both ld, and the elfcompress utility. Since the main purpose of
> compression is to make things smaller, I think this is a really
> nice ability, but our current definition for elf_compress() and
> elf_compress_gnu() don't support it, forcing compression even
> when the result is larger. This seems seems unfortunate, and
> is a reason why I wouldn't currently move off the internal stuff.

I had some fun fixing corner cases uncompressing data that turned out
the be zero sized, which might happen even if we don't allow it :)

> I'd like to make this mode of operation the default for these
> compress routines, and provide a flags argument that allows for
> compression to be forced, so that we can support both modes.
>
> int elf_compress(Elf_Scn *scn, uint_t type, uint_t flags);
> int elf_compress_gnu(Elf_Scn *, uint_t inflate, uint_t flags);
>
> Where the valid values for flags are
>
> #define ELF_CHF_FORCE 0x1
>
> defined as:
>
> By default, elf_compress() and elf_compress_gnu() will only
> compress the specified section if the resulting section is
> thereby reduced in size. Set ELF_CHF_FORCE to require that
> the section always be compressed.

To be honest I rather not add more complexity. Where do we stop? Should
the flags be extended to include the compression level (speed,
compression), the compression strategy (filtered, huffman, rle), window
size, mem level? IMHO if the user wishes to follow a specific
zlib/defalte compression strategy that isn't the default they should
just create their own Chdr and compressed data directly.

For the "makes data bigger" case. What is the real harm?
Are you using an heuristic to determine whether to compress? In which
case why not document that heuristic so the user can do it themselves
before calling elf_compress. Or are you actually compressing and then
after checking tossing away the resulting data buffer? Then again, why
not let the user check the size themselves and just toss the result away
themselves?

BTW. I am not really against it. It is just extra complexity in the
interface that I rather avoid. But I can see that it is something useful
that people would probably use. And implementing it is simple enough. So
if you insists I will just roll over and add the flag :)

> And the return value for these functions would be modified to:
>
> -1 error
> 0 success: no compression done, section is unchanged
> 1 success: compression done, section is modified
>
> How does that sound?

I would swap the zero and one case since normally zero is returned for
an libelf function if it was successful. "no compression done" doesn't
really sound like success. But this is just nitpicking. As long as -1
indicates error I am fine.

Cheers,

Mark

Ali Bahrami

unread,
Nov 20, 2015, 1:58:30 PM11/20/15
to Mark Wielaard, gener...@googlegroups.com
Hi Mark,


On 11/20/15 09:20, Mark Wielaard wrote:
> On Wed, 2015-11-18 at 09:51 -0700, Ali Bahrami wrote:
>> My first question is whether the type argument to elf_compress()
>> shouldn't really be uint_t rather than int? I don't see a use
>> for negative values here, and type maps internally to an ELF
>> Word, which is unsigned.
>
> Yes, using unsigned int should be fine. I assume that is what you call
> uint_t, which isn't a standard type as far as I can see.

Yes, that's what I meant. Sorry --- uint_t is widely used
in my world, but I see that only the explicit uintXX_t types
are standard.

So 'unsigned int'.



>
>> I'd like to make this mode of operation the default for these
>> compress routines, and provide a flags argument that allows for
>> compression to be forced, so that we can support both modes.
>>
>> int elf_compress(Elf_Scn *scn, uint_t type, uint_t flags);
>> int elf_compress_gnu(Elf_Scn *, uint_t inflate, uint_t flags);
...
>
> To be honest I rather not add more complexity. Where do we stop? Should
> the flags be extended to include the compression level (speed,
> compression), the compression strategy (filtered, huffman, rle), window
> size, mem level? IMHO if the user wishes to follow a specific
> zlib/defalte compression strategy that isn't the default they should
> just create their own Chdr and compressed data directly.

I know. There is no simple answer to where we stop. I guess we figure
that out, case by case, in discussions much like this one. I would
certainly be against any of the examples you've listed, and would rather
not have those discussions. :-)

>
> For the "makes data bigger" case. What is the real harm?
> Are you using an heuristic to determine whether to compress? In which
> case why not document that heuristic so the user can do it themselves
> before calling elf_compress. Or are you actually compressing and then
> after checking tossing away the resulting data buffer? Then again, why
> not let the user check the size themselves and just toss the result away
> themselves?

The real harm is probably negligible. I think it's true that any
compression algorithm has pathological cases where things end up bigger
than if compression hadn't happened. How likely that is, or how bad the
bloat, are statistical questions I can't really answer. I've seen it in
my small experiments, but of course, they're small --- compression isn't
important for them anyway.

The way this actually works in my implementation isn't a heuristic.
In fact, I have a hard time thinking of a heuristic that could work
well, since it's hard to second guess how well something will compress
without doing it. Rather, we just start the compression in the normal
manner, but stop and throw away the work if the job isn't done before
the allowed max size is hit. That's easy for libelf to do, but not
for the user above. The best they can do is compress, check the size,
and decompress back if they don't like it. That's not so bad, though
it does require an unnecessary allocation of a buffer that's immediately
freed.


>
> BTW. I am not really against it. It is just extra complexity in the
> interface that I rather avoid. But I can see that it is something useful
> that people would probably use. And implementing it is simple enough. So
> if you insists I will just roll over and add the flag :)
>

I appreciate your giving me that option, but I don't feel strongly
enough about it to force things if you're not on board. As I mentioned
before, our libelf does have this ability in its underlying code, so
I'm not giving up much by dropping this. I'll give you the last word,
but I'm fine with dropping this, and letting experience tell us if we
should add some way to do it later.

-----

I have now finished my work with elf_strptr(), and believe I have a full
implementation of what we've discussed. I'm going to turn to polishing
and testing, but I think we may be nearly done.

Thanks...

- Ali

Mark Wielaard

unread,
Nov 26, 2015, 10:56:02 AM11/26/15
to Ali Bahrami, gener...@googlegroups.com
On Fri, 2015-11-20 at 11:58 -0700, Ali Bahrami wrote:
> On 11/20/15 09:20, Mark Wielaard wrote:
> > On Wed, 2015-11-18 at 09:51 -0700, Ali Bahrami wrote:
> The real harm is probably negligible. I think it's true that any
> compression algorithm has pathological cases where things end up bigger
> than if compression hadn't happened. How likely that is, or how bad the
> bloat, are statistical questions I can't really answer. I've seen it in
> my small experiments, but of course, they're small --- compression isn't
> important for them anyway.
>
> The way this actually works in my implementation isn't a heuristic.
> In fact, I have a hard time thinking of a heuristic that could work
> well, since it's hard to second guess how well something will compress
> without doing it. Rather, we just start the compression in the normal
> manner, but stop and throw away the work if the job isn't done before
> the allowed max size is hit. That's easy for libelf to do, but not
> for the user above. The best they can do is compress, check the size,
> and decompress back if they don't like it. That's not so bad, though
> it does require an unnecessary allocation of a buffer that's immediately
> freed.

OK, I am convinced. And not having a flag argument means we have to add
a new function if we would like to add any options in the future. So
better to add it from the start. I assume the "allowed max" would be
something implementation specific.

I would also like to swap the return value as said in my previous email.
-1 for error, 0 for compressed, 1 for not compressed (because not
beneficial). Because IMHO zero should always mean the function did what
was expected.

Should elf_compress also set elf_errno if it didn't actually compress so
the user can get a reason from elf_errmsg?

> I have now finished my work with elf_strptr(), and believe I have a full
> implementation of what we've discussed. I'm going to turn to polishing
> and testing, but I think we may be nearly done.

I posted my latest implementation for elfutils here:
https://lists.fedorahosted.org/archives/list/elfutils-devel%
40lists.fedorahosted.org/thread/XPEPWF2KGWN63LSEKT2KDBQBU4TJOV7M/
It doesn't yet include the type change to unsigned int and the addition
of the flag. But it should be close to the final implementation.

Cheers,

Mark

Ali Bahrami

unread,
Nov 28, 2015, 6:11:40 PM11/28/15
to Mark Wielaard, gener...@googlegroups.com
Hi Mark,

Funny, you had completely talked me out of this. :-)


On 11/26/15 08:55 AM, Mark Wielaard wrote:
> OK, I am convinced. And not having a flag argument means we have to add
> a new function if we would like to add any options in the future. So
> better to add it from the start. I assume the "allowed max" would be
> something implementation specific.

I think the "allowed max" should aways be one less than the uncompressed
d_size of the section, inclusive of all compression header overhead. The
intent is to only compress if the result would be smaller than the
non-compressed original, and to reject an equal or larger result.



>
> I would also like to swap the return value as said in my previous email.
> -1 for error, 0 for compressed, 1 for not compressed (because not
> beneficial). Because IMHO zero should always mean the function did what
> was expected.

I just did a quick survey of the libelf manpages we ship with Solaris,
and I find that the vast majority of libelf routines return 0 or NULL
for failure, and a nonzero value on success. Zero almost always means
that the function did the opposite of what was expected.

The -1 return for failure is used in the small number of cases where
zero needs to convey a valid result. There are 2 routines that return
-1 for error, and 0 for success:

elf_cntl
elf_update

And 4 that return -1 for error and (value >= 0) for success,
which is the club that elf_compress/elf_compress_gnu are joining.

elf_getbase
Zero is no more or less valid than any other
positive value.
elf_getphdrnum
elf_getshdrnum
elf_getshdrstrndx
Zero is a valid value, but it isn't what the
programmer normally expects, since it means
that the object lacks phdrs, shdrs, or a section
name string table.

That's a pretty small (6) subset of routines, and we could quibble about
whether zero is an expected value for the last three. To my eye, the choice
of what 0 and 1 mean for elf_compress is somewhat arbitrary, rather than
about conforming to a wider libelf pattern used by lots of functions.
Either way, a programmer cannot use elf_compress() or elf_compress_gnu()
successfully without reading the manpage and coding against what it says.
I had a small reason for the order I proposed: It orders things from the
weakest result (-1: fail) to the strongest (1: compressed), with the
halfway case (0: maybe compressed, maybe not) lying in between.

Having read the above, if you still feel strongly that we should
swap the meaing of 0 and 1, or if I've made some blunder in my
reasoning above, then I'm open to swapping these, but I think the
originally proposed order also makes sense.


>
> Should elf_compress also set elf_errno if it didn't actually compress so
> the user can get a reason from elf_errmsg?

I say no, because this isn't an error. 0 and 1 are both successful
outcomes, though the caller does need to treat them differently,
as they would for instance with elf_getphdrnum.

The return value from elf_compress() tells the user everything
that libelf can reasonably know about the situation:

-1 error (and sets errno)
0 success, but not compressed for "compression algorithm reasons"
1 success, and compressed

If you did want to set an errno value for "didn't actually compress",
what extra information would it convey? All we're really in a position
to know is that the result was bigger than the original, and that
compression algorithms can do that.

Thanks...

- Ali

Ali Bahrami

unread,
Dec 18, 2015, 4:15:19 PM12/18/15
to gener...@googlegroups.com, johnl...@comcast.net, H.J. Lu, Mark Wielaard
Just in time for the holidays, a gift I've been wanting for a
few years: I believe that I'm done with my end of this project.
I checked in the Solaris implementation earlier this week. I know
from offline email that Mark is also closing in on completion. I'm
very much looking forward to seeing code that does ELF section
compression build seamlessly across platforms, and just work. I
probably wouldn't have bet big on that 3 years ago.

When I first put forward the gABI proposal for compression, I had a
sinking feeling that perhaps I was just solving my own problem, and
that it might not get much traction. The power of an existing "good
enough" solution can be hard to overcome. It didn't turn out that way,
and I'd like to thank all of you for that.

In Particular...

Thank you John, for taking the original request seriously, and then
working on your own time to help polish it into a proper gABI feature,
even while recovering from the flooding of your home and office.

Thank you H.J, for driving this stuff into binutils this year.
Timing is everything, and your timing was great.

And particularly, thank you Mark, for being interested in solving
this in a platform neutral way, for reaching out to do that, and then
for all the hard work of iterating over the various proposals, and
working to whittle it down to a simpler and better form. In my
experience, few programmers have the skills or patience to do that.
It was a pleasure to team up with you on this.

As with any real software, it's not perfect, but it fits the existing
libelf structure, and it has the merit of simplicity. I'm very happy to
see it done. (If it turns out that we're not really quite done yet,
then you know where to find me.)

Happy Holidays, and thanks again...

- Ali

Mark Wielaard

unread,
Jan 13, 2016, 8:06:03 AM1/13/16
to gener...@googlegroups.com, johnl...@comcast.net, H.J. Lu
On Fri, 2015-12-18 at 14:15 -0700, Ali Bahrami wrote:
> Just in time for the holidays, a gift I've been wanting for a
> few years: I believe that I'm done with my end of this project.
> I checked in the Solaris implementation earlier this week. I know
> from offline email that Mark is also closing in on completion. I'm
> very much looking forward to seeing code that does ELF section
> compression build seamlessly across platforms, and just work. I
> probably wouldn't have bet big on that 3 years ago.

elfutils 0.165 was just released and includes an implementation:
https://lists.fedorahosted.org/archives/list/elfutils-devel%
40lists.fedorahosted.org/message/2NVGNR5MV3S2ZO6QKNSQFVJ2Q3MEUABC/

I also wrote a blog entry on some of the details and discussions we had:
https://gnu.wildebeest.org/blog/mjw/2016/01/13/elf-libelf-compressed-sections-and-elfutils/

Thanks all,

Mark

Ali Bahrami

unread,
Jan 13, 2016, 11:26:23 PM1/13/16
to gener...@googlegroups.com
This is a big deal, congratulations, and thanks again!

- Ali

Reply all
Reply to author
Forward
0 new messages