Request to extend symbol visibilities (st_other)

[Rod Evans]

We'd like to extend the meaning of the ELF symbol ST_VISIBILITY
bits. These aliases are all I can dig up from past collaborations.
Our main goal is not to clash with any ELF extensions that may
already exist, or are in the works by other vendors.

Comments welcome. Perhaps just responding to:

gener...@googlegroups.com ?

We'd like to add:

#define STV_DEFAULT 0
#define STV_INTERNAL 1
#define STV_HIDDEN 2
#define STV_PROTECTED 3
#define STV_SINGLETON 4 <--- New
#define STV_ELIMINATE 5 <--- New

#define ELF32_ST_VISIBILITY(o) ((o)&0x7) <--- Changed from 0x3
#define ELF64_ST_VISIBILITY(o) ((o)&0x7) <--- Changed from 0x3

Symbol visibility is currently represented by the least significant
2 bits of a symbol's st_other field. Therefore, adding these two
additional values implies a widening of the visibility part of
st_other to three bits (as encapsulated by the ELF[32|64]_ST_VISIBLITY
macros). We feel that it is safe to take another bit from st_other,
as the bit is currently unused and always 0. Old linkers will fail
to understand the new bits, but will continue to understand the
original 4 values. This behavior is to be expected, and is not a
problem, as new applications that use these features will be built
by new linkers.

STV_SINGLETON:

We have a need to ensure that all references to certain symbols bind
to only one instance of a definition. In the old days (when processes
and linking capabilities were much simpler), this requirement was
typically met with simple interposition. For example, if A.so and B.so
both defined the symbol foo(), and a process loaded the objects in the
following order:

main -> A.so -> B.so ....

then any reference to foo() from within the process would bind to the
first definition provided by A.so.

However, life has gotten more complex. Some compiler implementations
instantiate the same template code within multiple dynamic objects.
Sometimes this template code uses private static data, or provides
link-list pointers or such. These duplicates expect to interpose on
one-another, so that all references bind to *one* instance of a
definition. This expectation can be circumvented in a number of ways:

i. dlopen hierarchies can establish local groups that result in
different definitions being bound to from within each group.

ii. scoping technologies that define interface definitions for
dynamic objects can result in "interposing" symbols being
reduced to locals.

iii. technologies such as direct-binding seek to allow objects to
bind to different definitions of the same symbols.

The bottom line is that an "interposing" expectation generated by a
compiler implementation can be compromised by the user's attempt to
control the runtime visibility of a symbol.

Explicitly identifying a symbols visibility as STV_SINGLETON is
intended to ensure that the symbol's global attribute isn't compromised,
and to instruct the runtime environment that only one definition of a
singleton can be bound to from any reference within a process.

This new visibility attribute stands out from the existing definitions
(STV_DEFAULT, STV_PROTECTED, STV_HIDDEN and STV_INTERNAL) in that the
most restrictive visibility can not be applied to any symbol resolution.

STV_ELIMINATE:

We have a need to eliminate symbols from the final dynamic executable
or shared object. We've actually been doing this for some time with
an augmented versioning attribute (VER_NDX_ELIMINATE) which gets
associated with a symbol through a mapfile (eliminate) definition.

We'd like to promote this capability by using the st_other field.
This makes it easier for compilers and third party relocatable object
generators to assign the attribute to a symbol. This capability also
allows non-global symbols to be identified for elimination, which can
not be done via our mapfile technology. Hence this new visibility makes
our present elimination easier, and also enables more cases to be handled.

STV_SINGLETON and STV_ELIMINATE are not directly related. However,
it would be efficient for us to address them both at the same time,
as they are both changes to st_other.

------------------------------------------------------------------------

The ELF ABI document contains an ELF Symbol Visibility table. I
suggest the following update, and associated description:

Name Value
....

STV_SINGLETON 4
STV_ELIMINATE 5

STV_SINGLETON
This visibility attribute ensures that a symbol remains global,
and that a single instance of the symbol definition is bound to
by all references within a process. An STV_SINGLETON can not be
directly bound to.

STV_ELIMINATE
This visibility attribute extends STV_HIDDEN. A symbol that
is defined in the current component as eliminate is not
visible to other components. The symbol is not written to the
symbol table of a dynamic executable or shared object from
which the component is used.

The STV_SINGLETON visibility attribute can affect the resolution of
symbols within an executable or shared object during link-editing.
A STV_SINGLETON can be combined with a STV_DEFAULT visibility
attribute, with the STV_SINGLETON taking precedence. A STV_SINGLETON
can not be combined with any other visibility attribute. Such an
event is deemed fatal to the link-edit.

--

Rod.

[Dave Prosser]

First some questions.

- Which (or all) instances of the same-spelled names are expected
to be marked as STV_SINGLETON?

- What's the conceptual RTLD operation model intended to handle this
visibility? Is it such that there's expected to be a (smallish?)
table of STV_SINGLETON names against which every newly looked-up
name is expected to be checked against? (It seems like something
like this is required if STV_SINGLETON isn't required to be present
on every instance of the potential duplicates.)

- How is STV_ELIMINATE functually any different from STV_HIDDEN or
STV_INTERNAL? Isn't the existing visibility effect of these cause
names not included in the runtime symbol table?

Some comments.

We've been using STV_EXPORTED (4) for years now to round out the
existing visibilities. This permits code to use the complete
range of visibility #pragma's as the programmer desires. And thus
we've had the ELFxx_ST_VISIBILITY() masking 3 bits as well. Since
you've stated you don't want to clash with existing art, we'll need
to choose different values for other STV_*s. I suppose that if we
do end up extending the visibility list, I'd expect that we should
go ahead and include STV_EXPORTED.

I'm also a bit skittish about STV_SINGLETON in another way. The
other visibility controls are for practical purposes only effective
at a.out and library creation link time. They aren't noticed at all
by runtime linking except for one special case.*** If I understand
what you're proposing STV_SINGLETON is thus really a different beast
altogether. It has no effect at all at a.out or library creation
link time, only having an effect during runtime linking (where we
want to keep the overhead as low as possible).

So I guess I remain unconvinced that these two new visibilities
are worth adding, at least as of yet.

*** The special case is where there is already a special case made
for runtime linking symbol lookup/binding. When the a.out is not
built PIC, we already have special case handling for undefined
functions and copy relocated data. We make sure that any requests
for the address of undefined functions that happen to be called by
the a.out use the a.out's corresponding PLT entry instead of the
actual function's address. For copy relocated data symbols, we
find the potentially initialized definition of the data symbol
elsewhere and copy it into the a.out's space. (There's size
compatibility checking.)

For both of these, the situation's made a bit more complicated by
STV_PROTECTED. When a shared library has a symbol marked as
STV_PROTECTED during RTLD's relocation pass we need to check to
see whether there's a matching symbol that's one of the above
special cases in the a.out. If so and "the next" symbol found is
the one we're trying to process for relocation, we need to use the
a.out's address instead of the one in the shared library that was
marked as STV_PROTECTED. (Ugh!)

--
Dave Prosser d...@sco.com (908)790-2358 The SCO Group, Murray Hill, NJ
SCO is a leading provider of UNIX-based solutions and mobile services.

[Lowell, Randy]

Rod,

At HP we're only using 2-bits of st_other, so the proposed definitions
do not clash with our current implementation, however I am also interested
in hearing the proposed runtime model for handling STV_SINGLETON. A
model that requires examination of every definition instance is less
efficient than one that only requires locating the "first" definition.
How do you handle a dlopen that introduces a new STV_SINGLETON instance
when there are already multiple bindings to other instances of the same
symbol?

Our compiler makes certain assumptions about symbols that it marks as
STV_PROTECTED and STV_HIDDEN. Based on those assumptions it can
optimize references to those symbols. STV_SINGLETON could not safely
override limited visibilities, because it would break those assumptions.
Do your compilers take advantage of symbol visibility in any way?
If so, how do you plan to address this issue?

Randy Lowell, HP

[H.J. Lu]

Hi Rod,

GNU binutils has

alpha.h:#define STO_ALPHA_NOPV 0x80
alpha.h:#define STO_ALPHA_STD_GPLOAD 0x88
m68hc11.h:#define STO_M68HC12_FAR 0x80
m68hc11.h:#define STO_M68HC12_INTERRUPT 0x40
mips.h:#define STO_MIPS16 0xf0
mips.h:#define STO_OPTIONAL (1 << 2)
sh.h:#define STO_SH5_ISA32 (1 << 2)

H.J.

[Rod Evans]

Dave Prosser wrote:

> First some questions.
>
> - Which (or all) instances of the same-spelled names are expected
> to be marked as STV_SINGLETON?

All of them, even the references, read on.

> - What's the conceptual RTLD operation model intended to handle this
> visibility? Is it such that there's expected to be a (smallish?)
> table of STV_SINGLETON names against which every newly looked-up
> name is expected to be checked against? (It seems like something
> like this is required if STV_SINGLETON isn't required to be present
> on every instance of the potential duplicates.)

Possibly, but this might just be necessary for "abnormal" scenarios,
again read on.

> - How is STV_ELIMINATE functually any different from STV_HIDDEN or
> STV_INTERNAL? Isn't the existing visibility effect of these cause
> names not included in the runtime symbol table?

STV_ELIMINATE = STV_HIDDEN/STV_INTERNAL + don't produce a symbol table
entry. BTW, I never have know what STV_INTERNAL implied - we've just
considered it another form of STV_HIDDEN.

This might seem trivial, in that if a symbol is hidden then its symbol
is only written to the .symtab and that can be stripped. But, as
diagnosability (?) has become a central theme for us, we want o keep
as much symbol table information as possible - striping isn't so common
anymore. But, we do have lots of "auxiliary" symbols that are just
needed to aid the relocation process (ie. tie one piece of code to
another), and after which they have no use. Removing this "scaffolding"
clutter is what we're trying to do. We've also added more local
symbol information to the runtime environment, again to aid runtime
observability - removing any clutter is useful.

> Some comments.
>
> We've been using STV_EXPORTED (4) for years now to round out the
> existing visibilities. This permits code to use the complete
> range of visibility #pragma's as the programmer desires. And thus
> we've had the ELFxx_ST_VISIBILITY() masking 3 bits as well. Since
> you've stated you don't want to clash with existing art, we'll need
> to choose different values for other STV_*s. I suppose that if we
> do end up extending the visibility list, I'd expect that we should
> go ahead and include STV_EXPORTED.

Ok, so we could have:

#define STV_DEFAULT 0
#define STV_INTERNAL 1
#define STV_HIDDEN 2

#define STV_PROTECT 3
#define STV_EXPORTED 4 <--- New to some
#define STV_SINGLETON 5 <--- New
#define STV_ELIMINATE 6 <--- New

Care to elaborate on STV_EXPORT - I'd assume it's a way of ensuring a
GLOB symbol remains global (ie. doesn't get demoted to local by any
user means). This would also imply that STV_EXPORTED is incompatible
with STV_INTERNAL/STV_HIDDEN/STV_PROTECTED, right? Read on.

> I'm also a bit skittish about STV_SINGLETON in another way. The
> other visibility controls are for practical purposes only effective
> at a.out and library creation link time. They aren't noticed at all
> by runtime linking except for one special case.*** If I understand
> what you're proposing STV_SINGLETON is thus really a different beast
> altogether. It has no effect at all at a.out or library creation
> link time, only having an effect during runtime linking (where we
> want to keep the overhead as low as possible).

It has an effect at link-edit time and runtime.

With the previous visibilities (STV_DEFAULT/STV_INTERNAL/STV_HIDDEN
and STV_PROTECTED), any symbol resolution between input relocatable
objects that assigned different visibilities could assign the
resolution as the most restrictive of the visibilities. This was
pretty simple to implement.

The symbol resolution when a STV_SINGLETON is present is a little
different. A STV_SINGLETON and STV_DEFAULT resolution is fine, and the
singleton will persist. But, as the compilers may have optimized
code that references STV_INTERNAL/STV_HIDDEN and STV_PROTECTED,
the resolution of a STV_SINGLETON with any of these must be fatal.
(I assume this is the same case with STV_EXPORT if my guess at
its meaning is correct).

Not that I'd expect this conflict condition to occur normally. As
the STV_SINGLETON is targeted for compiler implementation details,
I'd expect the compiler to put out consistent definitions for
singletons throughout all the modules it generates.

Another attribute of the link-editing I'd expect is that a
reference to a singleton would be tagged as singleton itself -
either because the compiler generated such a reference or
because ld(1) assigned the reference when it bound to a singleton
definition. This singleton reference is very useful to the
runtime environment.

When the runtime linker processes a relocation it looks up the
symbol reference (we already do this to determine whether its WEAK
of not), but now we'll determine whether the reference is a singleton.
Hence, under normal conditions I'd expect the runtime linker to
know its searching for a singleton up front, and thus search each loaded
object to find the first definition.

So, under normal conditions:

i. the compilers would consistently tag singletons
ii. the compilers/link-editor would tag singleton references
iii. the runtime linker would trigger on a singleton reference
to do the appropriate symbol search

Thus, not too complicated and not much overhead.

But then we come to unraveling the abnormal conditions. What
happens if the singleton tagging isn't consistent, or references
aren't tagged? I think this is where we get into "implementation
details", and vendors might chose to do more or less work in this
area. I can think of ways of handling certain scenarios, ie. a
standard symbol search binds to a singleton, the runtime linker
says "Oh Sh.., I didn't know that was a singleton, lets start
the search again to look through each loaded object". But how
much work we do in this area seems to be up to a vendor and
what "abnormal" situations they wish to accommodate.

Don't forget, today we have a problem with multiple instances
of the same symbol existing which should act as singletons but because
of user applied binding techniques have resulted in multiple instances
of the symbol being used. Sometimes the app works fine (as it never
throws the exception which would be compromised by the multiple
binding :-). Sometimes the app fails. The addition of the singleton
visibility isn't going to break these apps - they're broken already.
The visibility is going to help us prevent this breakage, and
give us better insight into the intent of the compiler code to
help diagnose problems.

> So I guess I remain unconvinced that these two new visibilities
> are worth adding, at least as of yet.

Actually, I'm not trying to convince you. You didn't try to convince
me that STV_EXPORTED was worth adding :-) Although if it does follow
what I'd expect, I could see it being useful too. We have a problem
that we'd like to help solve, and the STV_SINGLETON model is what
we've come up with. Plus, the use of st_other is much easier for
compilers and other object generators than inventing more private
ELF meta-data. So, what I'm really asking for is the st_other
name space.

Now, if our problem/solution looks useful to others, then great.
And if a discussion of "abnormal" handling is useful I'll love to
discuss what we're thinking along these lines. Or, if anyone has
solved this problem another way, I'd be interested in hearing of
that too.

> *** The special case is where there is already a special case made
> for runtime linking symbol lookup/binding. When the a.out is not
> built PIC, we already have special case handling for undefined
> functions and copy relocated data. We make sure that any requests
> for the address of undefined functions that happen to be called by
> the a.out use the a.out's corresponding PLT entry instead of the
> actual function's address. For copy relocated data symbols, we
> find the potentially initialized definition of the data symbol
> elsewhere and copy it into the a.out's space. (There's size
> compatibility checking.)

Yeah, we do this too, because 99% of our apps are non-pic. Bloody
awful isn't it.

> For both of these, the situation's made a bit more complicated by
> STV_PROTECTED. When a shared library has a symbol marked as
> STV_PROTECTED during RTLD's relocation pass we need to check to
> see whether there's a matching symbol that's one of the above
> special cases in the a.out. If so and "the next" symbol found is
> the one we're trying to process for relocation, we need to use the
> a.out's address instead of the one in the shared library that was
> marked as STV_PROTECTED. (Ugh!)

Cool. You do more work than we do. We let ld(1) warn you a copy
relocation has been created to a protected symbol, and the runtime
linker just runs with the first definition. Haven't had any issues
yet :-). But this does support my position on "abnormal" events -
each vendor can select their own criteria for what's abnormal,
and how, or if to handle it.

Thank you for your comments.

--
Rod

[Rod Evans]

Lowell, Randy wrote:
> Rod,
>
> At HP we're only using 2-bits of st_other, so the proposed definitions
> do not clash with our current implementation, however I am also interested
> in hearing the proposed runtime model for handling STV_SINGLETON. A
> model that requires examination of every definition instance is less
> efficient than one that only requires locating the "first" definition.
> How do you handle a dlopen that introduces a new STV_SINGLETON instance
> when there are already multiple bindings to other instances of the same
> symbol?

I think I covered most of this in my response to Dave Prosser.

A dlopen, or any new module loaded, can contribute a singleton
Any reference to a singleton will follow a simple search of each
loaded object, regardless of how the object was loaded. Thus, if
a dlopen'd object referenced a singleton I'd expect it to jump
out of the dlopen hierarchy and follow the simple search.

Another way to view this might be to think of interposers. Folks
having been using techniques like LD_PRELOAD to interpose on
existing symbols for years. All references to symbols defined in
an LD_PRELOAD object are bound to the preloaded object. Singletons
provide a similar capability where the binding requirements are an
attribute of the symbol and are controlled by the code generators
rather than the user having to set some environment variable or such.

In a process where multiple instances of a singleton symbol exist,
all references to these symbols bind to the first definition - interposition!

> Our compiler makes certain assumptions about symbols that it marks as
> STV_PROTECTED and STV_HIDDEN. Based on those assumptions it can
> optimize references to those symbols. STV_SINGLETON could not safely
> override limited visibilities, because it would break those assumptions.

Correct. It would be a fatal linking error to come across the same symbol
name with a STV_SINGLETON and any other visibility besides STV_DEFAULT.
Although this resolution differs from what we presently handle (ie.
the most restrictive visibility is taken), catching a flagrant
incomparability isn't any different from the link-edit already
catching "foo()" in one object and "foo[]" in another
object as questionable :-)

[Rod Evans]

Hmmm, looks like st_other has been a free-for-all.

--
Rod

[Dave Prosser]

Rod Evans wrote:

> Dave Prosser wrote:
>>- How is STV_ELIMINATE functually any different from STV_HIDDEN or
>> STV_INTERNAL? Isn't the existing visibility effect of these cause
>> names not included in the runtime symbol table?
>
> STV_ELIMINATE = STV_HIDDEN/STV_INTERNAL + don't produce a symbol table
> entry. BTW, I never have know what STV_INTERNAL implied - we've just
> considered it another form of STV_HIDDEN.

As I recall, STV_INTERNAL was requested by H-P when we
were putting this gABI feature together. I don't really
recall what it does beyond STV_HIDDEN. I know we take
them as the same for practical purposes. I'd suggest
sending a query to Cary Coutant (cary at cup.hp.com),
assuming he's still at H-P, as he was that company's
representative back then.

> This might seem trivial, in that if a symbol is hidden then its symbol
> is only written to the .symtab and that can be stripped. But, as
> diagnosability (?) has become a central theme for us, we want o keep
> as much symbol table information as possible - striping isn't so common
> anymore. But, we do have lots of "auxiliary" symbols that are just
> needed to aid the relocation process (ie. tie one piece of code to
> another), and after which they have no use. Removing this "scaffolding"
> clutter is what we're trying to do. We've also added more local
> symbol information to the runtime environment, again to aid runtime
> observability - removing any clutter is useful.

Given a request for an enforced absence through the
visibility channel, I'd think that the existing
(essentially noise to most of us) STV_INTERNAL
visibility might well be the way to go, instead of
creating a new name/value. I'm assuming here that
STV_ELIMINATE has otherwise the same effect as
STV_HIDDEN (or STV_INTERNAL) in that after "glueing
together" the collected object files, such names are
no longer to be visible for linking.

I like this in that it'd given a tangible meaning for
STV_INTERNAL that can be expressed in the gABI. I
don't think it'd even be a clash with whatever H-P's
using it for. If it is, we can say that STV_INTERNAL
is then a request for the symbol to be elided from the
resulting symbol table, but not a hard requirement.
At least as far as I can recall, whether local symbols
are present in symbol tables is an implementation
choice since it cannot affect linking.

>>Some comments.
>>
>>We've been using STV_EXPORTED (4) for years now to round out the
>>existing visibilities. This permits code to use the complete
>>range of visibility #pragma's as the programmer desires. And thus
>>we've had the ELFxx_ST_VISIBILITY() masking 3 bits as well. Since
>>you've stated you don't want to clash with existing art, we'll need
>>to choose different values for other STV_*s. I suppose that if we
>>do end up extending the visibility list, I'd expect that we should
>>go ahead and include STV_EXPORTED.
>
>
> Ok, so we could have:
>
> #define STV_DEFAULT 0
> #define STV_INTERNAL 1
> #define STV_HIDDEN 2
> #define STV_PROTECT 3
> #define STV_EXPORTED 4 <--- New to some
> #define STV_SINGLETON 5 <--- New
> #define STV_ELIMINATE 6 <--- New
>
> Care to elaborate on STV_EXPORT - I'd assume it's a way of ensuring a
> GLOB symbol remains global (ie. doesn't get demoted to local by any
> user means). This would also imply that STV_EXPORTED is incompatible
> with STV_INTERNAL/STV_HIDDEN/STV_PROTECTED, right? Read on.

Pretty much, and yes, the STV_INTERNAL...STV_EXPORTED
continuum are all intended to be mutually exclusive.
They are an enumeration and not a collection of flag
bits. Their values are also intended to reflect how
restricted an access is implied by that visibility.

Back when we were discussing the addition of symbol
visibility in the gABI, we also included a discussion
of visibility #pragma's, as an extension from the
already in-use one for weak.

#pragma <kind> <symbol>
#pragma <kind> <symbol> = <name>

where <kind> is weak, internal, hidden, protected,
or (for us) exported. The first form just flags the
named symbol, the second form defines the symbol as
being equivalent to the second name.

Once one has the ability to mark the visibility of
names in the source and object file, you no longer
need to rely on external lists of symbol names.

ELF linkers these days usually provide a way to control
the default visibility of names, hidden or exported.
Some linkers also base their defaults on whether one
is creating an a.out or a shared library.

Providing the most flexible library/application symbol
visibility requires permitting linking with either a
default of hidden or exported. When choosing to link
with a default of exported, the means to mark a symbol
as hidden is to use #pragma hidden. When choosing to
link with a default of hidden, you need something like
#pragma exported.

With the full range of visibility choices, you give
the developer/maintainer the flexibility to use what
works best for their code. One can, for example, make
the choice based on whether most names in the library
source are to be externally visible or most are only
internal. Or, possibly, choose based on what set of
names tends to be the most stable.

I'm going to need more time to think on this to be sure
I'm understanding this completely.

>>So I guess I remain unconvinced that these two new visibilities
>>are worth adding, at least as of yet.
>
> Actually, I'm not trying to convince you. You didn't try to convince
> me that STV_EXPORTED was worth adding :-) Although if it does follow
> what I'd expect, I could see it being useful too. We have a problem
> that we'd like to help solve, and the STV_SINGLETON model is what
> we've come up with. Plus, the use of st_other is much easier for
> compilers and other object generators than inventing more private
> ELF meta-data. So, what I'm really asking for is the st_other
> name space.
>
> Now, if our problem/solution looks useful to others, then great.
> And if a discussion of "abnormal" handling is useful I'll love to
> discuss what we're thinking along these lines. Or, if anyone has
> solved this problem another way, I'd be interested in hearing of
> that too.

Ah, but you DO need to convince folks of the utility
and reasonable commonality of need for things to be
included in the gABI, which is the reflector to which
you've posted your suggestion. If what you've proposed
isn't judged to be worth the carving out of ELF symbol
table st_other space for all target architectures, then
it won't make it into the gABI.

This is one reason why I'm asking why STV_INTERNAL can't
be used by you for your STV_ELIMINATE need, and whether the
intended STV_SINGLETON behavior is really something that's
substantially distinct from the other visibilities, enough
so that maybe it belongs elsewhere, and whether it's
something that is addressing an issue that's common.

I don't recall why STV_EXPORTED wasn't pushed way back
when. I didn't make a case for it in my reply to your
proposal as we were discussing your suggestions. As it
happened to use the same value as we already had in
place for a visibility encoding that really is in the
same flavor as those in the gABI (and because you said
that you didn't want to use space that others were
currently using) I included it. I've added more about it
above since you asked.

[Cary Coutant]

On Aug 15, 2007, at 11:09 am, Dave Prosser wrote:
> As I recall, STV_INTERNAL was requested by H-P when we
> were putting this gABI feature together. I don't really
> recall what it does beyond STV_HIDDEN. I know we take
> them as the same for practical purposes. I'd suggest
> sending a query to Cary Coutant (cary at cup.hp.com),
> assuming he's still at H-P, as he was that company's
> representative back then.

Nope, it was an SGI request (Jim Dehnert, are you still on this
list?). I'll see if I can dig up the original description that Jim
wrote up.

I'm no longer at HP -- I'll be starting at Google in a couple of
weeks (which is also where Jim is now).

-cary

[Lowell, Randy]

> -----Original Message-----
> From: ia64-abi...@linux.intel.com
> [mailto:ia64-abi...@linux.intel.com] On Behalf Of Dave Prosser
> Sent: Wednesday, August 15, 2007 2:09 PM
> To: gener...@googlegroups.com
> Cc: dis...@x86-64.org; ia64...@linux.intel.com
> Subject: [Ia64-abi] Re: Request to extend symbol visibilities
> (st_other)
>

> As I recall, STV_INTERNAL was requested by H-P when we
> were putting this gABI feature together. I don't really
> recall what it does beyond STV_HIDDEN. I know we take
> them as the same for practical purposes. I'd suggest
> sending a query to Cary Coutant (cary at cup.hp.com),
> assuming he's still at H-P, as he was that company's
> representative back then.

I represent HP now (randy....@hp.com).

According to the documentation STV_INTERNAL symbols are hidden
symbols that are not accessed outside of the module. STV_HIDDEN
symbols aren't exported, but they can still be accessed if
their addresses are given away by external interfaces.

At HP, we treat hidden and internal identically.

-Randy

[Cary Coutant]

>> Care to elaborate on STV_EXPORT - I'd assume it's a way of ensuring a
>> GLOB symbol remains global (ie. doesn't get demoted to local by any
>> user means). This would also imply that STV_EXPORTED is incompatible
>> with STV_INTERNAL/STV_HIDDEN/STV_PROTECTED, right? Read on.
>
> Pretty much, and yes, the STV_INTERNAL...STV_EXPORTED
> continuum are all intended to be mutually exclusive.
> They are an enumeration and not a collection of flag
> bits. Their values are also intended to reflect how
> restricted an access is implied by that visibility.

I don't understand how your STV_EXPORTED is different from STV_DEFAULT.

> Back when we were discussing the addition of symbol
> visibility in the gABI, we also included a discussion
> of visibility #pragma's, as an extension from the
> already in-use one for weak.
>
> #pragma <kind> <symbol>
> #pragma <kind> <symbol> = <name>
>
> where <kind> is weak, internal, hidden, protected,
> or (for us) exported. The first form just flags the
> named symbol, the second form defines the symbol as
> being equivalent to the second name.

At HP, we used #pragma default_binding to round out the set of
pragmas; it would set the visibility to STV_DEFAULT, which means a
normal exported symbol. (Being the default, it was only necessary to
override a blanket setting via the -B command-line options.)

-cary

[Cary Coutant]

Rod Evans wrote:

> STV_ELIMINATE = STV_HIDDEN/STV_INTERNAL + don't produce a symbol table
> entry. BTW, I never have know what STV_INTERNAL implied - we've just
> considered it another form of STV_HIDDEN.
>
> This might seem trivial, in that if a symbol is hidden then its symbol
> is only written to the .symtab and that can be stripped. But, as
> diagnosability (?) has become a central theme for us, we want o keep
> as much symbol table information as possible - striping isn't so
> common
> anymore. But, we do have lots of "auxiliary" symbols that are just
> needed to aid the relocation process (ie. tie one piece of code to
> another), and after which they have no use. Removing this
> "scaffolding"
> clutter is what we're trying to do. We've also added more local
> symbol information to the runtime environment, again to aid runtime
> observability - removing any clutter is useful.

I'm not saying I'm opposed to STV_ELIMINATE (yet), but if you don't
want the symbols in the symbol table in the first place, why not just
convert any relocations to use the section symbol when generating
the .o file? No need for the linker to eliminate them if they're not
in the symbol table to begin with.

-cary

[H.J. Lu]

BTW, please use ia64...@googlegroups.com instead of
ia64...@linux.intel.com for ia64 psABI discussion.

On Wed, Aug 15, 2007 at 12:22:04PM -0700, Cary Coutant wrote:
> >> Care to elaborate on STV_EXPORT - I'd assume it's a way of ensuring a
> >> GLOB symbol remains global (ie. doesn't get demoted to local by any
> >> user means). This would also imply that STV_EXPORTED is incompatible
> >> with STV_INTERNAL/STV_HIDDEN/STV_PROTECTED, right? Read on.
> >
> > Pretty much, and yes, the STV_INTERNAL...STV_EXPORTED
> > continuum are all intended to be mutually exclusive.
> > They are an enumeration and not a collection of flag
> > bits. Their values are also intended to reflect how
> > restricted an access is implied by that visibility.
>
> I don't understand how your STV_EXPORTED is different from STV_DEFAULT.
>

On Linux, a global symbol in executable won't be available to
dlopened dynamic object unless it is referenced by a dynamic object
at link-time. However, it is desirable to make such symbol dynamic.
Will STV_EXPORTED make a global symbol dynamic without any side
effects?

H.J.

[Dave Prosser]

Cary Coutant wrote:
> I don't understand how your STV_EXPORTED is different from STV_DEFAULT.
>

> At HP, we used #pragma default_binding to round out the set of
> pragmas; it would set the visibility to STV_DEFAULT, which means a
> normal exported symbol. (Being the default, it was only necessary to
> override a blanket setting via the -B command-line options.)

First off, sorry about my failing memory regarding
the origin of STV_INTERNAL and also my failing to pay
attention to the email address for Randy.

STV_DEFAULT is the compatibility choice. It says to
handle the symbol's visibility just like you did with
ELF symbols before we added visibility attributes.

So, if your linker builds a.outs with the default
choice being hidden, only including those names in
the dynamic symbol table that are found to come from
the shared libraries its linked with, STV_DEFAULT
certainly doesn't mean export. In contrast, if you
have a symbol marked as STV_EXPORTED in an object
file used to create the a.out, that name will end up
in the dynamic symbol table, regardless.

If I'm making a shared library out of a pile of object
files and I only want one or a few names visible, for
example if I'm building an NSS module and I only want
the nss_module_register() routine exported, I can get
this result most readily by having that one name marked
as STV_EXPORTED and not to anything special for all the
other nonlocal names in the collection of object files
that comprise the shared library and link -Bhide. This
also handles things down the road as I fiddle around
with the implementation as I'll never accidentally end
up making other names visible (which could easily happen
if I need to mark *every* nonlocal name except for one
as STV_HIDDEN).

Maybe if we'd defined ELF from the start with symbol
visibility in place, then there wouldn't be a need for
a distinction between STV_EXPORTED and STV_DEFAULT,
but there's a real difference in our world.

A #pragma default_binding directive would have no
effect (other than forcing a clash with a conflicting
differing visibility directive) in our world, since
STV_DEFAULT is the visibility you get automatically to
match what was done from the earliest ELF days.

[Dave Prosser]

H.J. Lu wrote:
> BTW, please use ia64...@googlegroups.com instead of
> ia64...@linux.intel.com for ia64 psABI discussion.

Alright, but so far I don't think any of this topic has
been anything outside of generic.

That's the way it works in our implementation, as I just
described (in more detail) in reply to Cary's question.

[Rod Evans]

We have a need for *.o's to reference information from other .o's.

--

Rod.

[H.J. Lu]

On Wed, Aug 15, 2007 at 03:50:09PM -0400, Dave Prosser wrote:
>
> > On Wed, Aug 15, 2007 at 12:22:04PM -0700, Cary Coutant wrote:
> >
> >>>>Care to elaborate on STV_EXPORT - I'd assume it's a way of ensuring a
> >>>>GLOB symbol remains global (ie. doesn't get demoted to local by any
> >>>>user means). This would also imply that STV_EXPORTED is incompatible
> >>>>with STV_INTERNAL/STV_HIDDEN/STV_PROTECTED, right? Read on.
> >>>
> >>>Pretty much, and yes, the STV_INTERNAL...STV_EXPORTED
> >>>continuum are all intended to be mutually exclusive.
> >>>They are an enumeration and not a collection of flag
> >>>bits. Their values are also intended to reflect how
> >>>restricted an access is implied by that visibility.
> >>
> >>I don't understand how your STV_EXPORTED is different from STV_DEFAULT.
> >
> > On Linux, a global symbol in executable won't be available to
> > dlopened dynamic object unless it is referenced by a dynamic object
> > at link-time. However, it is desirable to make such symbol dynamic.
> > Will STV_EXPORTED make a global symbol dynamic without any side
> > effects?
>
> That's the way it works in our implementation, as I just
> described (in more detail) in reply to Cary's question.
>

We found STV_PROTECTED is too expensive. It will be nice to have
something like STV_RESOLVED. That is similar to STV_PROTECTED,
but it doesn't require special handling for symbol addresss.

H.J.

[Cary Coutant]

>> I'm not saying I'm opposed to STV_ELIMINATE (yet), but if you don't
>> want the symbols in the symbol table in the first place, why not just
>> convert any relocations to use the section symbol when generating
>> the .o file? No need for the linker to eliminate them if they're not
>> in the symbol table to begin with.
>
> We have a need for *.o's to reference information from other .o's.

OK, that makes sense.

-cary

[Jim Dehnert]

On Aug 14, 9:06 pm, Rod Evans <Rod.Ev...@sun.com> wrote:

> Dave Prosser wrote:
>
> > - How is STV_ELIMINATE functually any different from STV_HIDDEN or
> > STV_INTERNAL? Isn't the existing visibility effect of these cause
> > names not included in the runtime symbol table?
>
> STV_ELIMINATE = STV_HIDDEN/STV_INTERNAL + don't produce a symbol table
> entry.

That should always be a legitimate optimization of STV_HIDDEN and
STV_INTERNAL in the linker.

> This might seem trivial, in that if a symbol is hidden then its symbol
> is only written to the .symtab and that can be stripped. But, as
> diagnosability (?) has become a central theme for us, we want o keep
> as much symbol table information as possible - striping isn't so common
> anymore. But, we do have lots of "auxiliary" symbols that are just
> needed to aid the relocation process (ie. tie one piece of code to
> another), and after which they have no use. Removing this "scaffolding"
> clutter is what we're trying to do. We've also added more local
> symbol information to the runtime environment, again to aid runtime
> observability - removing any clutter is useful.

This is certainly a reasonable concern. But given that it is a
debugging
concern, I wonder if it wouldn't be more appropriate to deal with it
in the
debugging information instead. I don't feel strongly about it,
though.

> With the previous visibilities (STV_DEFAULT/STV_INTERNAL/STV_HIDDEN
> and STV_PROTECTED), any symbol resolution between input relocatable
> objects that assigned different visibilities could assign the
> resolution as the most restrictive of the visibilities. This was
> pretty simple to implement.

This is a little tricky, depending on what you mean by "most
restrictive."
These were intended as guarantees to the compiler, as well as allowing
users better control over symbol visibility. Restricting
STV_PROTECTED
to STV_HIDDEN is OK -- its implication that an inside reference won't
be mapped to an outside definition remains true for STV_HIDDEN.
But you probably shouldn't map STV_HIDDEN to STV_INTERNAL --
there may be different values because the compiler could guarantee
no leakage in one .o but not in the other, and the conservative
combination is STV_HIDDEN. (Of course that last point doesn't
matter if you're treating STV_INTERNAL like STV_HIDDEN, and
none of it matters if you don't have linktime interprocedural
optimization going on -- the compiler in that case has already
drawn its conclusions.)

Jim

[Rod Evans]

Thanks for the input Jim ... comments in-line ...

Jim Dehnert wrote:
> On Aug 14, 9:06 pm, Rod Evans <Rod.Ev...@sun.com> wrote:
>> Dave Prosser wrote:
>>
>>> - How is STV_ELIMINATE functually any different from STV_HIDDEN or
>>> STV_INTERNAL? Isn't the existing visibility effect of these cause
>>> names not included in the runtime symbol table?
>> STV_ELIMINATE = STV_HIDDEN/STV_INTERNAL + don't produce a symbol table
>> entry.
>
> That should always be a legitimate optimization of STV_HIDDEN and
> STV_INTERNAL in the linker.

Although the linker could take this approach, it doesn't do us much good.
We wish to distinguish between "useful" local symbols (function/data
addresses) and "not-so-useful" symbols (stuff used to stitch objects
together).

>> This might seem trivial, in that if a symbol is hidden then its symbol
>> is only written to the .symtab and that can be stripped. But, as
>> diagnosability (?) has become a central theme for us, we want o keep
>> as much symbol table information as possible - striping isn't so common
>> anymore. But, we do have lots of "auxiliary" symbols that are just
>> needed to aid the relocation process (ie. tie one piece of code to
>> another), and after which they have no use. Removing this "scaffolding"
>> clutter is what we're trying to do. We've also added more local
>> symbol information to the runtime environment, again to aid runtime
>> observability - removing any clutter is useful.
>
> This is certainly a reasonable concern. But given that it is a
> debugging
> concern, I wonder if it wouldn't be more appropriate to deal with it
> in the
> debugging information instead. I don't feel strongly about it,
> though.

Hmm, good point, however it wouldn't do us much use. Our linker doesn't
even look at debugging information (it might relocate it, but that's all).
The compilers create the information, we "concatenate" it together and
put in the output file, and debuggers use it. This has given the
traditional debuggers and compilers a great deal of flexibility to
control their information without ld(1) getting involved.

But we have a growing set of other diagnostic tools, things that revolve
around /proc, DTrace, etc, and these guys typically want to match pc's
to symbolic addresses. These tools could benefit from removing the
"not-so-useful" symbols, and don't know the details of any compiler
generated debugging information. It is this family of tools that are
requesting the eliminate attribute.

>> With the previous visibilities (STV_DEFAULT/STV_INTERNAL/STV_HIDDEN
>> and STV_PROTECTED), any symbol resolution between input relocatable
>> objects that assigned different visibilities could assign the
>> resolution as the most restrictive of the visibilities. This was
>> pretty simple to implement.
>
> This is a little tricky, depending on what you mean by "most
> restrictive."
> These were intended as guarantees to the compiler, as well as allowing
> users better control over symbol visibility. Restricting
> STV_PROTECTED
> to STV_HIDDEN is OK -- its implication that an inside reference won't
> be mapped to an outside definition remains true for STV_HIDDEN.
> But you probably shouldn't map STV_HIDDEN to STV_INTERNAL --
> there may be different values because the compiler could guarantee
> no leakage in one .o but not in the other, and the conservative
> combination is STV_HIDDEN. (Of course that last point doesn't
> matter if you're treating STV_INTERNAL like STV_HIDDEN, and
> none of it matters if you don't have linktime interprocedural
> optimization going on -- the compiler in that case has already
> drawn its conclusions.)

According the ABI I know of:

The attributes, ordered from least to most constraining, are:
STV_PROTECTED, STV_HIDDEN and STV_INTERNAL.

And this is all we've followed - but then we've taken hidden and
internal as being equivalent. Perhaps for your implementation,
the words should be tightened up to convey what should happen
if different visibility attributes are specified for distinct
references to or definitions of STV_HIDDEN/STV_INTERNAL symbols.

Folks have suggested that STV_INTERNAL should perhaps be given the
STV_ELIMINATE capabilities I've been proposing. If everyone agreed
to this I'd be happy, but it sounds like you have a implementation
where STV_INTERNAL and STV_HIDDEN differ, and thus I think we
should not redefine STV_INTERNAL. Whether others will implement
STV_INTERNAL as you describe, who knows. I'll probably continue
to view STV_INTERNAL/STV_HIDDEN the same - but we shouldn't steal
the name for another purpose.

So to sum up, I think we've discovered/proposed a few things.

i. STV_INTERNAL can provide a unique class of visibility.
Perhaps Jim could provide the description that should
be defined in the ABI (part of the Symbol Table description).

ii. STV_EXPORT (or visibility 4), even though it's not in the ABI,
is already in use, so we should carve out this name space to
avoid future conflicts. Perhaps Dave could provide
the description that should be defined in the ABI (part of
the Symbol Table description).

iii. STV_ELIMINATE (my proposal).

iv. STV_SINGLETON (my proposal).

And to clarify another point in regards to where I'm coming from.
I don't believe that defining an ABI namespace, and the implementation
behind that namespace dictates that everyone must provide the
associated capability. There are a number of things in the ABI
that we don't implement. But we shouldn't steal the namespace for
a different/conflicting purpose. Hence I wouldn't want to use
visibility 4 for some other purpose than STV_EXPORT now that I know
STV_EXPORT exists. And, I'm also not comfortable taking STV_INTERNAL
for a new purpose (unless Jim gave his blessing :-).

So, although I'd like STV_ELIMINATE/STV_SINGLETON, I don't think
it necessary that anyone else implement these visibilities.
Perhaps others do see some value (just as I think STV_EXPORT could
be useful in our environments) and can follow at their own pace.

If there were enough bits in st_other, I'd suggest carving out a
platform/vendor specific range as we've done with other ABI namepaces,
but we're somewhat limited in the symbol table area.

--

Rod.

[Cary Coutant]

On 8/16/07, Rod Evans <Rod....@sun.com> wrote:

> But we have a growing set of other diagnostic tools, things that revolve
> around /proc, DTrace, etc, and these guys typically want to match pc's
> to symbolic addresses. These tools could benefit from removing the
> "not-so-useful" symbols, and don't know the details of any compiler
> generated debugging information. It is this family of tools that are
> requesting the eliminate attribute.

I'm starting to think that your STV_ELIMINATE would better be served
by the symbol type, rather than the visibility attribute. From your
description, it really sounds like these "auxiliary" symbols are a
whole different type of symbol -- not STT_OBJECT or STT_ENTRY, but
perhaps STT_SUNW_AUXILIARY.

Doing this would address the issues we've been discussing regarding
the orthogonality of this with respect to STV_HIDDEN and STV_INTERNAL,
and also would avoid the problem of how to order the visibilities from
least to most constraining.

> If there were enough bits in st_other, I'd suggest carving out a
> platform/vendor specific range as we've done with other ABI namepaces,
> but we're somewhat limited in the symbol table area.

It also helps with this issue. There's still the problem of how to
reconcile allocating another bit to the visibility field when some
implementations are already using that bit for another purpose.

-cary

[Roland McGrath]

> I'm starting to think that your STV_ELIMINATE would better be served
> by the symbol type, rather than the visibility attribute. From your
> description, it really sounds like these "auxiliary" symbols are a
> whole different type of symbol -- not STT_OBJECT or STT_ENTRY, but
> perhaps STT_SUNW_AUXILIARY.

I'm liking that idea a lot. For the uses Rod has described, STV_HIDDEN
plus STT_INTERNAL or suchlike seems like the right thing. The visibility
behavior wanted really is no different from "hidden". The new thing that's
wanted is a marker that ld can use as a hint that a given hidden symbol is
or is not worthwhile to preserve for the user's edification in debugging.
A type of "symbol users don't need to know about" fits just right.

Thanks,
Roland

[Rod Evans]

Cary Coutant wrote:
> On 8/16/07, Rod Evans <Rod....@sun.com> wrote:
>
>> But we have a growing set of other diagnostic tools, things that revolve
>> around /proc, DTrace, etc, and these guys typically want to match pc's
>> to symbolic addresses. These tools could benefit from removing the
>> "not-so-useful" symbols, and don't know the details of any compiler
>> generated debugging information. It is this family of tools that are
>> requesting the eliminate attribute.
>
> I'm starting to think that your STV_ELIMINATE would better be served
> by the symbol type, rather than the visibility attribute. From your
> description, it really sounds like these "auxiliary" symbols are a
> whole different type of symbol -- not STT_OBJECT or STT_ENTRY, but
> perhaps STT_SUNW_AUXILIARY.
>
> Doing this would address the issues we've been discussing regarding
> the orthogonality of this with respect to STV_HIDDEN and STV_INTERNAL,
> and also would avoid the problem of how to order the visibilities from
> least to most constraining.

But this obscures the symbols real type - what if I need to know the
symbol type (OBJT, FUNC, TLS, etc) to validate or perform things like
relocation? What type should I assume STT_SUNW_AUXILIARY is?

With STV_ELIMINATE and STV_SINGLETON I was trying to provide a generic
means of describing a new type of visibility without obscuring any of the
other symbol information we already use.

--
Rod

[Jim Dehnert]

And the problem is that it does turn out to obscure other symbol information,
although most of us aren't using that bit. I think Cary's got the right idea,
even if the solution isn't right. This _feels_ like a visibility
attribute, but it's
different from what's there. The current visibility field controls visibility
between modules from the point of view of the linker(s), whereas this
is aimed at global visibility from the point of view of the user/debugger.

You're trying to get the linker to do a bit of cleanup for you. This sounds
a lot like strip, except that you want the default to be removal. Is there
some inherent reason that segregating these symbols into a separate
symtab that can be removed/ignored won't work just as well? (That
may be a dumb question, as I haven't looked at this stuff for years.)

Jim

--
--
Jim Dehnert
deh...@gmail.com
dehn...@acm.org

[Rod Evans]

Jim Dehnert wrote:
>>
>> With STV_ELIMINATE and STV_SINGLETON I was trying to provide a generic
>> means of describing a new type of visibility without obscuring any of the
>> other symbol information we already use.
>
> And the problem is that it does turn out to obscure other symbol information,
> although most of us aren't using that bit. I think Cary's got the right idea,
> even if the solution isn't right. This _feels_ like a visibility
> attribute, but it's
> different from what's there. The current visibility field controls visibility
> between modules from the point of view of the linker(s), whereas this
> is aimed at global visibility from the point of view of the user/debugger.

Ok, so do I conclude that everyones view of the STV visibility field is now
something that is only meaningful to the link-editing phase? If so, then
my two proposed symbols do lie outside of this new categorization.

> You're trying to get the linker to do a bit of cleanup for you. This sounds
> a lot like strip, except that you want the default to be removal. Is there
> some inherent reason that segregating these symbols into a separate
> symtab that can be removed/ignored won't work just as well? (That
> may be a dumb question, as I haven't looked at this stuff for years.)

No, not dumb, and not impossible. But it does make the implementation of
the producer more complex. Not only does another symbol table have to be
created, but sections that might need to reference these symbols (like
relocations) must also be separated (a relocation sections sh_link
points to the symbol table to be used).

Personally, I still think the symbol attributes I'm proposing are
visibilities, but I'll agree they now extend the notion of link-editor
visibilities.

--
Rod

[Rod Evans]

Cary Coutant wrote:

>> If there were enough bits in st_other, I'd suggest carving out a
>> platform/vendor specific range as we've done with other ABI namepaces,
>> but we're somewhat limited in the symbol table area.
>
> It also helps with this issue. There's still the problem of how to
> reconcile allocating another bit to the visibility field when some
> implementations are already using that bit for another purpose.

But the only bits in the ABI are INTERNAL/HIDDEN/PROTECTED.

We've heard of some other bits, whether they're all still in use who
knows, but none are in the ABI. Perhaps this type of conversation
isn't fruitful anymore? In that we should each just go off any carve
out our own use for any remaining ELF name space regardless of what
anyone else is doing?

--
Rod

[Lu, Hongjiu]

Given that some st_other bits have been used by various psABIs, can we
make the new visilibity bits optional, which are only enabled with a
different EI_ABIVERSION value? Each psABI can decide if it wants to
support the new optional bits.

H.J.
hongj...@intel.com

[Rod Evans]

It's been a while, and the conversations have died down.
There's been a few suggestions, but I don't feel anything
has really come forward as an alternative to my original
request.

So, I think I'm left with the proposal of three new additions
to the visibility name-space:

#define STV_DEFAULT 0
#define STV_INTERNAL 1
#define STV_HIDDEN 2
#define STV_PROTECT 3

#define STV_EXPORTED 4 <--- New (from Dave)

#define STV_SINGLETON 5 <--- New
#define STV_ELIMINATE 6 <--- New

#define ELF32_ST_VISIBILITY(o) ((o)&0x7) <--- Changed from 0x3

#define ELF64_ST_VISIBILITY(o) ((o)&0x7) <--- Changed from 0x3

With descriptions:

Name Value
....

STV_EXPORTED 4
STV_SINGLETON 5
STV_ELIMINATE 6

STV_EXPORTED
This visibility attribute ensures that a symbol remains global.
Unlike STV_DEFAULT symbols, whose visibility can be affected by
other visibility requests, such as from a command line option,
mapfile definition, or resolution against another symbol of the same
name, the STV_EXPORTED attribute ensures that the visibility of the
symbol is not reduced by any other visibility request.

STV_SINGLETON
This visibility attribute ensures that a symbol remains global,
and that a single instance of the symbol definition is bound to
by all references within a process. An STV_SINGLETON can not be
directly bound to.

STV_ELIMINATE
This visibility attribute extends STV_HIDDEN. In addition to the
properties of STV_HIDDEN, a symbol that is defined in the current

component as eliminate is not visible to other components. The symbol
is not written to the symbol table of a dynamic executable or shared
object from which the component is used.

The STV_SINGLETON visibility attribute can affect the resolution of
symbols within an executable or shared object during link-editing.

When a symbol with STV_SINGLETON visibility is combined with a symbol
that has STV_DEFAULT visibility, the resulting symbol has STV_SINGLETON
visibility. A STV_SINGLETON can not be combined with any other visibility

attribute. Such an event is deemed fatal to the link-edit.

--

Rod.

[Lu, Hongjiu]

What do you mean by "global"? A symbol in an executable can be global,
but not visible
outside of the exucutable unless it is also dynamic.

H.J.

[Rod Evans]

Lu, Hongjiu wrote:

> What do you mean by "global"? A symbol in an executable can be global,
> but not visible outside of the exucutable unless it is also dynamic.

I'm not sure I follow.

The term global is already in use in the ABI, for example:

STV_DEFAULT

The visibility of symbols with the STV_DEFAULT attribute is as
specified by the symbol's binding type. Global symbols and weak
symbols are visible outside of their defining component, the
executable file or shared object. Local symbols are hidden.
Global symbols and weak symbols can also be preempted. These
symbols can by interposed by definitions of the same name in
another component.

What do you mean by "unless it is also dynamic"?

--

Rod.

[Lu, Hongjiu]

>-----Original Message-----
>From: gener...@googlegroups.com
>[mailto:gener...@googlegroups.com] On Behalf Of Rod Evans
>Sent: Tuesday, August 28, 2007 11:55 AM
>To: gener...@googlegroups.com
>Subject: Re: Request to extend symbol visibilities (st_other)
>
>

I am talking about run-time behavior. On Linux, not all global symbols
in an
executable are in SHT_DYNSYM. If a global symbol isn't in SHT_DYNSYM, it
isn't
visible to dlopened shared libraries. Will STV_EXPORTED force a symbol
into SHT_DYNSYM.

H.J.

[Rod Evans]

Lu, Hongjiu wrote:

> I am talking about run-time behavior. On Linux, not all global symbols
> in an
> executable are in SHT_DYNSYM. If a global symbol isn't in SHT_DYNSYM, it
> isn't
> visible to dlopened shared libraries. Will STV_EXPORTED force a symbol
> into SHT_DYNSYM.

I would presume so, as the user has explicitly define it to be "exported".

As a mild curiosity, if you don't want a symbol exported (or made visible
to other objects), why don't you simply demote it to local?

--

Rod.

[H.J.]

It isn't we don't want to. By default, in executable, we put a global
symbol into
SHT_DYNSYM only if it is referenced by a DSO.

H.J.

[Rod Evans]

So, if you build an app against a number of DSO's, you only place in
SHT_DYNSYM any globals that the app references, right? How do you
control the addition of other symbols, ie,. one's that might be needed
by dlopen'ed DSO's?

Anyway, sounds like one of your implementation details that falls within
the lines of the ABI definition. *I'd* assume that STV_EXPORTED would
force the symbol into the SHT_DYNSYM. Perhaps Dave could confirm, as
this is really his definition :-)

--

Rod.

[H.J.]

On Aug 28, 2:34 pm, Rod Evans <Rod.Ev...@sun.com> wrote:
> H.J. wrote:
> > On Aug 28, 2:18 pm, Rod Evans <Rod.Ev...@sun.com> wrote:
> >> Lu, Hongjiu wrote:
> >>> I am talking about run-time behavior. On Linux, not all global symbols
> >>> in an
> >>> executable are in SHT_DYNSYM. If a global symbol isn't in SHT_DYNSYM, it
> >>> isn't
> >>> visible to dlopened shared libraries. Will STV_EXPORTED force a symbol
> >>> into SHT_DYNSYM.
> >> I would presume so, as the user has explicitly define it to be "exported".
>
> >> As a mild curiosity, if you don't want a symbol exported (or made visible
> >> to other objects), why don't you simply demote it to local?
>
> > It isn't we don't want to. By default, in executable, we put a global
> > symbol into
> > SHT_DYNSYM only if it is referenced by a DSO.
>
> So, if you build an app against a number of DSO's, you only place in
> SHT_DYNSYM any globals that the app references, right? How do you
> control the addition of other symbols, ie,. one's that might be needed
> by dlopen'ed DSO's?

We have a linker switch to put all global symbols in SHT_DYNSYM.

> Anyway, sounds like one of your implementation details that falls within
> the lines of the ABI definition. *I'd* assume that STV_EXPORTED would
> force the symbol into the SHT_DYNSYM. Perhaps Dave could confirm, as
> this is really his definition :-)
>

It should be explicitly specified.

H.J.

[Mats Wichmann]

Is there anything that needs to be said about this
combination, which looks like a shared object in
that e_type=ET_DYN but also has the properties of
an executable? Is there any place this has been
documented in detail?

-- mats

[Michael Eager]

Shared objects are (or can be) executables. Try
/lib/ld-linux.so.2 /bin/echo "hello"

On the other hand, labeling a position-independent
executable as a shared library simply because both
it and the PIE do not require relocation seems wrong.

--
Michael Eager ea...@eagercon.com
1960 Park Blvd., Palo Alto, CA 94306 650-325-8077

[Roland McGrath]

In ELF terms, a PIE is just an ET_DYN file following the normal spec.
The only deep intrinsic ELF meaning of ET_DYN vs ET_EXEC is whether the
p_vaddr addresses are absolute or can be offset at load time. (It's
always been possible to execute an ET_DYN file directly if it was linked
with its e_entry address set to entry code that works.) So on that
mark, a PIE is nothing new or special.

The special thing about linking a PIE is that it's an ET_DYN file whose
PT_DYNAMIC (.dynamic section) contains a DT_DEBUG element. An ordinary
DSO does not have DT_DEBUG, while a dynamically-linked ET_EXEC always does.

Thanks,
Roland