Question about 64bit ELF (upx)

[Miki Tebeka]

Greetings,

Can someone from the Go team (or who knows this stuff) answer the question about 64bit ELF
at http://sourceforge.net/tracker/?func=detail&atid=102331&aid=3408066&group_id=2331?

Thanks,
--
Miki

[Russ Cox]

On Mon, Sep 12, 2011 at 12:44, Miki Tebeka <miki....@gmail.com> wrote:
> Can someone from the Go team (or who knows this stuff) answer the question
> about 64bit ELF
> at
> http://sourceforge.net/tracker/?func=detail&atid=102331&aid=3408066&group_id=2331?

An ELF PT_LOAD segment is allowed to specify an
address that is not aligned according to the alignment
it requests. In that case, the loader rounds the va
and file offset down to the nearest boundary and
increases the total size up by the same amount.
Similarly, the total size is then rounded up to the
nearest boundary.

It sounds like this upx program does not know about
that rule.

Russ

[Miki Tebeka]

Thanks Russ.

[reise...@gmail.com]

The upx program knows ALL about ELF.  There is no doubt that the specific executable conforms to ELF with regard to (.p_vaddr - .p_offfset) being divisible by PAGE_SIZE, and what this implies about the mapped address space.  The question is _why_ does PT_LOAD[0].p_offset==0xc00 ?  Many files which agree with the ELF spec with respect to execve() have other properites which can make other processing either easier or more difficult.  Setting PT_LOAD[0].p_offset non-zero is something which makes other processing more difficult. _Why_ does Go prefer 0xc00 instead of 0 ?  (The mapped address space under execve() will be the same.)

[Russ Cox]

On Mon, Sep 12, 2011 at 13:24, <reise...@gmail.com> wrote:
> _Why_ does Go prefer 0xc00 instead of 0 ?

Because that's where the Go binary's text segment begins.
The stuff before it is the ELF header, and strictly speaking
does not need to be loaded. If it is, fine.

That said, I had to change the ELF headers we write out
to do the rounding ourselves, because there are buggy
qemu binaries floating around that I wanted to use with
Go ARM binaries. So in a release or two you won't have
to worry about this case tripping up upx.

Russ

[reise...@gmail.com]

There are other reasons besides qemu for choosing PT_LOAD[0].p_offset==0.  During execve(), it is permitted for the operating system to initialize only the address space that is described by the union of all [PT_LOAD.p_vaddr, +PT_LOAD.p_memsiz), using [base, +length) notation.  In particular, the kernel may zero out the page fragment which lies below .p_vaddr.  For the "hw" example executable with PT_LOAD[0].p_vaddr==0x400c00, this includes the PT_PHDR at 0x400040.  In other words, "hw" has a bad pointer in its description of its layout.  This pointer will appear in auxv{AT_PHDR}, where it can wreak havoc with the runtime linker/loader.  In particular, dl_iterate_phdr() probably won't work for the main executable.  Other forms of runtime introspection, including by the "user-written" code itself, also may have trouble.  Similarly, a coredump might not include the original Ehdr, and this can be nasty.  Trying to be minimalist, or to strive for  .p_vaddr==origin(.text), leads to problems.

The question  "Why 0xc00?" persists.  The space for Shdrs need not be covered by PT_LOAD, and this might matter for reducing the number of mapped pages.  Also, if there are 0xc00/64 Shdrs then they won't all fit below 0xc00.  The Phdrs must reside within the first 0x200 bytes (32-bit) or 0x400 bytes (64-bit) for convenience and efficiency of execve().  It looks like the high end of PT_INTERP abuts 0xc00; why is it preceded by a large region of zero bytes?  In short, 0xc00 looks like an unjustified whim [where is the documentation?]

[Russ Cox]

> The question  "Why 0xc00?" persists.

The 0xc00 value comes from the ELFRESERVE
constant in http://golang.org/src/cmd/ld/elf.h.

Originally we packed the ELF hdr, phdrs, shdrs,
string data, and interp string into a fixed-size chunk
at the beginning of the file, just because it was easy.
The string data kept growing, though, so we had to
keep making the chunk bigger, and then the ELF layout
code had various assumptions that the chunk was
smaller than a page, so rather than track them all down,
I stopped growing the chunk when it hit 3k and took
the time to move the string data into the read-only data.
Now that the string data is gone the 3k could probably
be reduced back to something like 1k, but I haven't
bothered.

The interp string is at the end of the chunk just because
it was an easy address to calculate without knowing
exactly how many phdr/shdrs there were.

It's all a bit sloppy, but it's the kind of thing that
is hard enough to test on all the various platforms,
so once it works, I just back quietly away.

So 0xc00 is definitely not an unjustified whim, but
it's also more something that evolved than something
that was intelligently designed.

Russ