RFC: Implementation of ELF sharable sections
H. J. Lu
http://groups-beta.google.com/group/generic-abi/browse_thread/thread/bca08f6560f61b0d
Several people have expressed interests. I post it here for comments.
I used OS specific values. If we get consensus, I can change those
values to generic range.
H.J.
Roland McGrath
the object-file format is necessary to accomplish the real world goals.
What's wrong with using an independent library API to designate memory you
want to share?
H. J. Lu
> supports this type of memory. Which I doubt it will anytime soon as
> the concept is rather oddly defined and no one posted a kernel-level
> design and implementation yet.
Kernel support may be useful, but not strictly required. Consider
shared libraries with sharable segment, they are loaded by dynamic
linker, not kernel.
Another thing, a sharabale segment isn't a PT_LOAD segment. Dynamic
linker doesn't have to know it for applications to run correctly. So
it is backward compatible.
H.J.
Hoeflinger, Jay P
Adding this feature is not necessary for correctness, only for higher
performance. Certainly allocation of sharable memory can be done at run
time (and our software is doing that now). But doing it at runtime
forces run-time allocation, plus indirection for all accesses to the
memory, adding to program overhead.
If we can mark object file sections directly as sharable, then they can
be mapped at load time and accessed directly at run-time, saving
significant runtime overhead. Our measurements show a reduction of up
to 15% of running time for some codes.
Jay
Ulrich Drepper
> Adding this feature is not necessary for correctness, only for higher
> performance. Certainly allocation of sharable memory can be done at run
> time (and our software is doing that now).
our, out, our. This is support for crappy proprietary software. None
of this is used so far in free software (I understand gomp would be the
target). Until such a free implementation exists there is no way to
know a) the design is necessary in the first place, b) the design meets
all the requirements, c) the design is the best possible and not just
what you want.
There is no way I'm going to support anything which isn't proven to be
of benefit in a free software implementation. Support for optimizing
proprietary code is no priority.
--
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Roland McGrath
image if you want to avoid indirection costs in using them. I don't see
how the setup cost is any different from having the dynamic linker know
something magical instead of application initialization code.
Hoeflinger, Jay P
special way? We don't want to make everything sharable - only the data
that needs to be shared. Mapping everything kills any performance
advantage direct mapping might have gained for you.
Jay
-----Original Message-----
From: gener...@googlegroups.com [mailto:gener...@googlegroups.com]
On Behalf Of Roland McGrath
Sent: Friday, January 05, 2007 4:42 PM
To: gener...@googlegroups.com
Roland McGrath
> special way?
You don't rely on magic. Your code uses normal initializers to do its setup.
Hoeflinger, Jay P
Of course we use normal initializers.
"Sharable" is an attribute of the data in a section. It means that
certain data needs to be maintained consistently across processes
running on different machines. This is a general and new characteristic
of data made useful by clusters, and deserves to be tagged on the
sections that need to be kept consistent.
Of course, there are all kinds of ways I could imagine implementing
this, but most of them require users to do a lot of extra work, such as
forcing users or compilers to generate special .init sections for each
of their sharable sections. Our proposal is an alternative to this. It
eliminates all this extra effort by a standardized tagging mechanism.
Jay
-----Original Message-----
From: gener...@googlegroups.com [mailto:gener...@googlegroups.com]
On Behalf Of Roland McGrath
Sent: Friday, January 05, 2007 4:52 PM
To: gener...@googlegroups.com
Subject: RE: RFC: Implementation of ELF sharable sections
Hoeflinger, Jay P
On the other hand, I'd be happy to discuss the merits of the ideas.
Jay
-----Original Message-----
From: gener...@googlegroups.com [mailto:gener...@googlegroups.com] On Behalf Of Ulrich Drepper
Sent: Friday, January 05, 2007 4:17 PM
To: gener...@googlegroups.com
Ian Lance Taylor
> Of course we use normal initializers. "Sharable" is an attribute of
> the data in a section. It means that
> certain data needs to be maintained consistently across processes
> running on different machines. This is a general and new characteristic
> of data made useful by clusters, and deserves to be tagged on the
> sections that need to be kept consistent. Of course, there are all
> kinds of ways I could imagine implementing
> this, but most of them require users to do a lot of extra work, such as
> forcing users or compilers to generate special .init sections for each
> of their sharable sections. Our proposal is an alternative to this. It
> eliminates all this extra effort by a standardized tagging
> mechanism. Jay
I think I would find this proposal somewhat clearer if you outlined
how you would expect a dynamic linker to implement it.
Is it just that the dynamic linker would pass MAP_SHARED to mmap at
program load time?
It seems that your proposal doesn't have any provision for putting the
shared section at the same address in different processes, and indeed
I don't see how that could be described or implemented. If that is
so, then it seems like a significant restriction on the types of data
which may be placed in the shared section, and it seems to me that
this restriction must be documented.
Ian
Lu, Hongjiu
>
>I think I would find this proposal somewhat clearer if you outlined
>how you would expect a dynamic linker to implement it.
>
>Is it just that the dynamic linker would pass MAP_SHARED to mmap at
>program load time?
>
This proposal is designed with backward compatibility in mind, the new
segment isn't PT_LOAD and dynamic linker support is desirable, but
optional. Here is the segment dump of a binary with sharable sections:
[hjl@gnu-13 sharable-8]$ readelf -l foo
Elf file type is EXEC (Executable file)
Entry point 0x4006d0
There are 10 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
PHDR 0x0000000000000040 0x0000000000400040
0x0000000000400040
0x0000000000000230 0x0000000000000230 R E 8
INTERP 0x0000000000000270 0x0000000000400270
0x0000000000400270
0x000000000000001c 0x000000000000001c R 1
[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
LOAD 0x0000000000000000 0x0000000000400000
0x0000000000400000
0x0000000000000a5c 0x0000000000000a5c R E 200000
LOAD 0x0000000000000a60 0x0000000000600a60
0x0000000000600a60
0x00000000003ff5a0 0x00000000007ff5a0 RW 200000
DYNAMIC 0x0000000000000a90 0x0000000000600a90
0x0000000000600a90
0x00000000000001e0 0x00000000000001e0 RW 8
NOTE 0x000000000000028c 0x000000000040028c
0x000000000040028c
0x0000000000000020 0x0000000000000020 R 4
GNU_SHR 0x0000000000200000 0x0000000000800000
0x0000000000800000
0x0000000000200000 0x0000000000200000 R 200000
GNU_SHR 0x0000000000400000 0x0000000000c00000
0x0000000000c00000
0x0000000000000000 0x0000000000200000 R 200000
GNU_EH_FRAME 0x0000000000000988 0x0000000000400988
0x0000000000400988
0x000000000000002c 0x000000000000002c R 4
GNU_STACK 0x0000000000000000 0x0000000000000000
0x0000000000000000
0x0000000000000000 0x0000000000000000 RWE 8
Section to Segment mapping:
Segment Sections...
00
01 .interp
02 .interp .note.ABI-tag .gnu.hash .dynsym .dynstr .gnu.version
.gnu.version_r .rela.got .rela.sharable_bss .rela.plt .init .plt .text
.fini .rodata .eh_frame_hdr .eh_frame
03 .init_array .ctors .dtors .jcr .dynamic .got .got.plt .data
.sharable_data .bss .sharable_bss
04 .dynamic
05 .note.ABI-tag
06 .sharable_data
07 .sharable_bss
08 .eh_frame_hdr
09
[hjl@gnu-13 sharable-8]$
Sharable segments are within a PT_LOAD segment. To get around the lack
of dynamic linker support, I added section start/end symbols in my
implementation:
61: 0000000000e00000 0 NOTYPE LOCAL HIDDEN 28
__sharable_bss_end
62: 0000000000a00000 0 NOTYPE LOCAL HIDDEN 26
__sharable_data_end
66: 0000000000800000 0 NOTYPE LOCAL HIDDEN 26
__sharable_data_start
67: 0000000000c00000 0 NOTYPE LOCAL HIDDEN 28
__sharable_bss_start
You can put a function in .init_array section to process the sharable
sections at load time even if there is no dynamic linker support.
>It seems that your proposal doesn't have any provision for putting the
>shared section at the same address in different processes, and indeed
>I don't see how that could be described or implemented. If that is
>so, then it seems like a significant restriction on the types of data
>which may be placed in the shared section, and it seems to me that
>this restriction must be documented.
Jay can better explain it than me.
H.J.
Lu, Hongjiu
>Sent: Monday, January 08, 2007 7:33 AM
>To: gener...@googlegroups.com
>Subject: RE: RFC: Implementation of ELF sharable sections
For dynamic linker support, we can add a new DT_ entry, DT_GNU_SHR_INIT,
which contains the address of a function
extern void sharable_init (const char *, const char *);
If DT_GNU_SHR_INIT exists and isn't NULL, dynamic linker will call it
with start and end of PT_GNU_SHR segment, for each of them.
Jay, please make sure that our sharable init functions can be called
more than once for a given address range.
H.J.
Hoeflinger, Jay P
Thanks.
Jay
Hoeflinger, Jay P
> >It seems that your proposal doesn't have any provision for putting
the
> >shared section at the same address in different processes, and indeed
> >I don't see how that could be described or implemented. If that is
> >so, then it seems like a significant restriction on the types of data
> >which may be placed in the shared section, and it seems to me that
> >this restriction must be documented.
> Jay can better explain it than me.
> H.J.
This is true. We currently leave the problem of getting data at the
same address in different processes to the runtime system. It's not a
restriction, but a note about this would be appropriate in the right
place. I'm not sure where that would be, though.
Jay
Ian Lance Taylor
> >Is it just that the dynamic linker would pass MAP_SHARED to mmap at
> >program load time?
> >
>
> This proposal is designed with backward compatibility in mind, the new
> segment isn't PT_LOAD and dynamic linker support is desirable, but
> optional.
I don't really understand. If dynamic linker support is optional, and
you are actually doing the initialization with a .init_array entry,
then what do we gain from this proposal? This seems like something
that you can do entirely within the linker script.
Ian
Lu, Hongjiu
>To: gener...@googlegroups.com
>Subject: Re: RFC: Implementation of ELF sharable sections
Linker script change isn't enough. We need to deal with putting common
symbols into sharable sections, merging sharable symbols with non
sharable symbols, as well as supporting sharable sections in dynamic
executables/shared libraries, which requires SHN_GNU_SHARABLE_COMMON
among other things. And some linker implementation details may be very
specific to GNU linker.
Since we have to extend gABI anyway, we made our proposal linker
independent with dynamic linker support as an optional feature so that
other platforms can benefit from it as well. Without dynamic linker
support, we have to depend on linker provided symbols.
H.J.
Cary Coutant
It doesn't really look like we've got a consensus. In order to
support a gABI extension for this, I'd want to see it done "right",
and I'd like to see a general agreement that it would be useful to
multiple vendors as proposed.
In general, I think the proposal has evolved in the right direction,
but I would prefer to ditch the bracketing symbols and require loader
support. I'd also prefer using a PF_SHARABLE flag on a PT_LOAD
segment, but that's probably only because that's the way we did it on
HP-UX.
HP-UX added support for two kinds of sharable memory many years ago
for our ccNUMA PA-RISC systems -- we added near-shared and far-shared
memory in much the same way -- (SHF_ flags, special SHN_ indexes for
common symbols, and PF_ flags, but no new DT entries). More recently,
for IPF systems, we've reused this mechanism for a single type of
shared memory. We've used vendor-specific values throughout.
The fact that we've done this twice, with slightly different needs
each time, and that Intel's needs seem to be slightly different
still, makes me think that this is a feature best left as a vendor
extension.
--
Cary Coutant
HP-UX Runtime Architect