What is a position independent and position dependent code. How does it is different from relocatable code?

[acidla...@gmail.com]

I had this idea that when we link dynamic libraries, then wherever a call to a function thats defined inside dynamic library is, replaced by some address. That address was calculated using some relocation algorithm.
Now I came to know some switches in gcc such as -fpic. So if I don't -fpic does my shared library code is always position dependent, i.e loading at same virtual address?

[acidla...@gmail.com]

On Sunday, March 26, 2017 at 10:29:09 PM UTC+5:30, acidla...@gmail.com wrote:
> I had this idea that when we link dynamic libraries, then wherever a call to a function thats defined inside dynamic library is, replaced by some address. That address was calculated using some relocation algorithm.
> Now I came to know some switches in gcc such as -fpic. So if I don't -fpic does my shared library code is always position dependent, i.e loading at same virtual address?

I will try to give a example of my understanding.
I work as a freelancer. I may not be available onsite at an absolute address, but I can perform my task from some offset(anywhere from the world as long as I have connectivity to company). This is dynamic loading of a shared/dynamic library. Isn't it? Or is it position independent property?
Position dependent may mean that I have to be available in 30 km vicinity of the company to be able to work. This is position dependent.

[Jean-Marc Bourguet]

If you compile with -fpic, the code generated can be loaded at any address
but will be slightly slower. If you compile without -fpic, the code
generated will have to be adjusted for the address at which it is loaded
but will be slightly quicker. The advantage of using -fpic is that 1/ the
code may be shared even if it is loaded at different address in different
process (that sharing of relatively big libraries used by a lot of program
-- for instance the system one, or those of X -- save memory and in the
times were computers were more constrained by memory, avoid swapping which
makes everything slow) 2/ there is no need to patch the code to adjust it
for the loading address, this patching takes also some time and that fixed
cost may offset the speed advantage of not using -fpic.

Yours,

--
Jean-Marc

[Robert Wessel]

On Sun, 26 Mar 2017 09:59:02 -0700 (PDT), acidla...@gmail.com
wrote:

>I had this idea that when we link dynamic libraries, then wherever a call to a function thats defined inside dynamic library is, replaced by some address. That address was calculated using some relocation algorithm.
>Now I came to know some switches in gcc such as -fpic. So if I don't -fpic does my shared library code is always position dependent, i.e loading at same virtual address?

Position independent code can load at any location without the
relocations (usually) performed by the loader. Position dependent
code can only run at one location, unless relocated by the loader,
which can only happen if the executable image includes the required
relocation information (which is what's usually meant by relocatable
code). In addition, an executable can have a preferred address, where
it can load without any relocations being applied.

For example, in Windows, .EXEs (assuming no ASLR) are by default
linked to load at 0x20000, and linked *without* relocation
information. So that's the only place they *can* load. DLLs (shared
libraries), usually have a default location for which they're linked
(0x1000000 if not otherwise specified), but *do* include relations. If
the loader can load the module at the preferred address it's faster,
and it can just memory-map the code from the executable, without
having to generate program text pages with the relocations applied.
So you can improve load times of an application that loads multiple
DLLs by basing them at different addresses (obviously only a single
DLL could load at 0x10000000). And you can change the base and
whether relocations are included for any executable (both EXEs and
DLLs).

In any event, once the code is loaded, it generally won't move. So if
you load a shared library, and a particular entry point resolves as
0x12345678, that's where it'll stay. The PIC/relocatedable/etc. thing
only impacts where/how the loader will load the code.

[acidla...@gmail.com]

Can anybody give example such as
"a man was going for a walk...."

[acidla...@gmail.com]

One more thing is there any difference in shared library and dynamic library? Or its just same.

[acidla...@gmail.com]

I am coming to the conclusion that PIC code and relocatable code are same.
quoting from a document
fpic, -FPIC, -KPIC, When these options are
present during compilation, the compiler generates code that
accesses symbols through a collection of indirection registers
and global offset tables. The primary benefit of PIC code is
that the text segment of the resulting library (containing ma-
chine instructions) can quickly relocate to any memory address
without code patches.

So does this mean if we use the term relocatable, the shared code will everytime at a new location but cannot be anywhere in the ram. In case of PIC code, the shared code can be loaded anywhere in the RAM. Now this code is loaded anywhere in the RAM thats why performance is degraded.

[Malcolm McLean]

On Sunday, March 26, 2017 at 6:54:05 PM UTC+1, acidla...@gmail.com wrote:
>
> Can anybody give example such as
> "a man was going for a walk...."
>

The absolute addressing said "go to the war memorial, turn left, cross the bridge, go onto the canal towpath" the relative addressing said, "Go forwards 200 years, turn left, another 200 years, left again and downhill, then and other three hundred yards". Relocateable addressing said "do a three legged walk, forwards to a local landmark, turn left, forwards to another landmark, then go along a feature, so if we were doing it from here - to the war memorial, then the bridge, then along the canal."

[Robert Wessel]

On Sun, 26 Mar 2017 11:01:43 -0700 (PDT), acidla...@gmail.com

Shared library is the more generic and Unix-y term, DLLs tend to imply
Windows, although the term is used on other platforms as well.

[Robert Wessel]

On Sun, 26 Mar 2017 11:21:37 -0700 (PDT), acidla...@gmail.com

No, they're not. PIC does not need to be relocated by the loader,
it's effectively "self-relocating". Relocatable code must be fixed up
by an external agent, typically the OS's loader.

>quoting from a document
>fpic, -FPIC, -KPIC, When these options are
>present during compilation, the compiler generates code that
>accesses symbols through a collection of indirection registers
>and global offset tables.

Implementation specific, but not an uncommon approach.

>The primary benefit of PIC code is
>that the text segment of the resulting library (containing ma-
>chine instructions) can quickly relocate to any memory address
>without code patches.

Yes. Although "*loaded* to any memory address without code patches"
would be more accurate.

>So does this mean if we use the term relocatable, the shared code will everytime at a new location but cannot be anywhere in the ram. In case of PIC code, the shared code can be loaded anywhere in the RAM. Now this code is loaded anywhere in the RAM thats why performance is degraded.

I'm not sure I follow this. The code is not moved once it's loaded.
It might be loaded into a different position every time it is loaded,
however. PIC or ordinary relocatable makes no difference. In the
typical use case, where you load a share library once (either
implicitly because of a dependency resolved by the loader - IOW an
"import"), or with an explicit load in your program (dlopen,
LoadLibrary), it will not move until it's actually unloaded (which
would happen at program shutdown in the case of an implicit load,
dlclose or FreeLibrary if you're doing it explicitly). Every time you
call it, it will be in the same location, unless you do an
unload/reload sequence (which you normally wouldn't, unless you
actually wanted to change which module was loaded).

And as I mentioned, there are program objects that are not relocatable
at all.

[Manfred]

On 3/26/2017 8:21 PM, acidla...@gmail.com wrote:
> On Sunday, March 26, 2017 at 11:31:59 PM UTC+5:30, acidla...@gmail.com wrote:

<snip>

>
> I am coming to the conclusion that PIC code and relocatable code are same.

IIRC they are not the same.
There is a relevant difference about security.
Relocatable code requires that some memory area be both executable and
writable during load, which poses a security vulnerability.
PIC code enables the library to be loaded at an arbitrary address,
without the need for any executable memory to be writable, so it does
not have this vulnerability. This comes at the cost of an additional jmp
instruction at function call (also IIRC).

[Robert Wessel]

On Mon, 27 Mar 2017 00:59:42 +0200, Manfred <inv...@invalid.add>
wrote:

>On 3/26/2017 8:21 PM, acidla...@gmail.com wrote:
>> On Sunday, March 26, 2017 at 11:31:59 PM UTC+5:30, acidla...@gmail.com wrote:
><snip>
>>
>> I am coming to the conclusion that PIC code and relocatable code are same.
>IIRC they are not the same.
>There is a relevant difference about security.
>Relocatable code requires that some memory area be both executable and
>writable during load, which poses a security vulnerability.

If the memory is not writable during the load, the OS is not going to
be able to load the program object, PIC or not! Nor do most OS's (at
least those that routinely make program text areas un-writable),
actually execute any of the *program* code while loading and
performing the relocations.

>PIC code enables the library to be loaded at an arbitrary address,
>without the need for any executable memory to be writable, so it does
>not have this vulnerability. This comes at the cost of an additional jmp
>instruction at function call (also IIRC).

Exactly how internal calls, external calls and entry points get
handled, is highly implemetation dependent. For example, for Win32,
normal (non-PIC) executable modules do most of their external calls
(calls to other program objects) via a set of trampolines - and it's
only the table of trampolines which gets modified during the load to
fix up external references.

[Robert Wessel]

On Sun, 26 Mar 2017 17:40:43 -0500, Robert Wessel
<robert...@yahoo.com> wrote:

>No, they're not. PIC does not need to be relocated by the loader,
>it's effectively "self-relocating". Relocatable code must be fixed up
>by an external agent, typically the OS's loader.

The choice of the term "self-relocating" was somewhat unfortunate in
the above. Self-relocating code usually refers to code that contains
its own fixup code (rather than relying on the loader), PIC code
really doesn't need that.

[Gordon Burditt]

Depending on the situation, it may be *IMPOSSIBLE* to load all the
shared libraries at standard but globally unique addresses, as there
probably isn't a master registry of every version of every shared
library ever made for the platform, and they probably all wouldn't
fit simultaneously in the limited address space. What probably
happens is that the library gets loaded at a different address (with
different relocation patches), and that copy can't be shared with
programs using the library at the "standard" address. This can be
tuned in favor of putting the current-version heavily-used system
libraries at standard places.

If you use -fpic, you can use a single copy of the code in memory
for 87 programs using it simultaneously at 29 different load
addresses. If you don't use -fpic, you need at least 29 different
copies because the linker/loader needs to put different addresses
in the different copies.

It is possible that if you don't use -fpic, the system won't even
TRY to share it. The recipe to build a shared library may *require*
-fpic. Otherwise you have a non-shared library which is linked
(copied) into the executable.

> If you compile with -fpic, the code generated can be loaded at any address
> but will be slightly slower.

Here are some examples in a hypothetical assembly language that probably
matches no real machine but is close to a lot of them.

If the program can use the Program Counter (it might have a different
name, like Instruction Counter, for various CPUs) as an index
register, the code will likely contain instructions like:
jp <some_constant>(PC)
rather than
jp <relocatable_address>

If it can't, -fpic will probably start off the code with something like:
load PC, R8172 ; the destiation for "load" is on the right
; in this assembly language.
where Register 8172 will be loaded with an address relative to the
base address for the code segment. (It may need to save and restore
Register 8172 for the caller, also). Branches within the code
look like like:

jp <some_constant>(R8172)

Data references may be made relative to the program counter or the
base address register also.

The use of an index register typically slows down the instruction
by one or a few clock cycles compared to not using an index register.
So do instructions used to set up registers needed only for
position-independence. An added complication: if it is possible
to use an instruction with an index register and a short offset
(say, 16 bits instead of 64 bits), not loading the other 6 bytes
from an instruction may cancel out the cost of an index register,
but only if the library is small enough to have all the branches
"reach" with a 16-bit offset.

There's a trade-off here:

Using -fpic:
- Allows sharing all the copies in use simultaneously.
- The code is slightly slower.

Not using -fpic:
- Does not allow sharing the copies, which requires more
memory, which may involve more paging/swapping, which
slows things down.
- Loading more copies also requires more CPU.
- The code needed to apply relocation patches to the copies
requires time to execute, and possibly more disk I/O to
read this information from the executable. This may end
up slower than -fpic.

Which is better may depend on the library (the C library, used by
almost every program, vs. the Egyptian Hieroglyphics to Klingon
translation library, which is hardly ever used at all, vs. a graphics
library which is heavily used by the X server on the one and only
set of display hardware on this system (so more than one copy is
never used).

[supe...@casperkitty.com]

On Sunday, March 26, 2017 at 9:25:53 PM UTC-5, robert...@yahoo.com wrote:
> If the memory is not writable during the load, the OS is not going to
> be able to load the program object, PIC or not! Nor do most OS's (at
> least those that routinely make program text areas un-writable),
> actually execute any of the *program* code while loading and
> performing the relocations.

On systems which do not write-protect code memory, it is possible on many
platforms to have a program which can be loaded at any address as a sequence
of bytes and executed without the loader having to know anything about its
structure. If C code knows the size of the code, it could literally do
something like:

typedef int (*zDriverFunction)(int op, void *params);
unsigned char codeBuff[4096];
zDriverFunction theFunc;

fread(codeBuff, 1, 4096, codeFile);
theFunc = (zDriverFunction)codeBuff;

and then call theFunc like any other function. This approach was used a
fair amount on both MS-DOS and the Macintosh, and likely many other systems
as well. Some things that were loaded that way used PC-relative addressing
for everything, while others had a small position-independent fixup routine
coupled with a list of patch-points.

[Gordon Burditt]

>>One more thing is there any difference in shared library and dynamic library? Or its just same.
>
>
> Shared library is the more generic and Unix-y term, DLLs tend to imply
> Windows, although the term is used on other platforms as well.

Modern versions of UNIX also have two different levels of using
libraries. Well, three, if you include the entirely static-linked
case.

Normal shared libraries are loaded at program startup before main()
gets called. They stay loaded while the program runs. The set of
libraries to link is fixed at link time (some implementations let
libraries "drag in" other libraries they require), but the specific
paths used are not. The program code does not need to know it's
using shared libraries vs. being entirely static-linked.

The dlopen() call and friends (dlclose(), dlsym(), etc.) allow the
program to load and unload modules that weren't even designed at
the time the executable was created, using names obtained from
configuration files, the command line, user input, or searching the
filesystem. (The function call signatures are fixed, though, and
designing the interface carefully is highly desirable). The shared
objects used as a target of dlopen() are often called "modules" and
they do not have to be designed to work as a shared library. The
code *does* have to know that it is expected to call dlopen(). Some
people consider only the dlopen() case to be "dynamic linking" and
the usual dynamic-link to be "using shared libraries". I don't
agree with this, but there's nothing deep here, only the naming
convention.

I've never seen it done, but it is possible for a program to write,
compile, link a module, load, run, and test it, then unload it and
try again with different source code, all with the original program
still running.

In practice, you configure programs like Apache httpd, PHP, and the
GIMP by setting up a configuration file to load modules you need,
and commands that get handled by modules that register handlers for
them. Sometimes certain modules are mutually exclusive. Unused
optional code takes up *NO* code space.

The FreeBSD kernel (also Linux and other BSD kernels) also work
this way so if you don't screw up too badly, you can load a device
driver, test it, unload it, fix the driver, and repeat, without
rebooting. (Well, that doesn't work if you cause a kernel panic).
Lots of (userland) programs will quietly load a kernel module if
you do something that needs one (e.g. mounting a particular filesystem
type that needs one).

[acidla...@gmail.com]

I will again try to update my understanding according to ubuntu operating system.

I created a file foo.c which contain a function foo() to be called by a main.c file.
1) What should be my choice of compilation here
I did this
gcc -c -Wall -Werror -fpic foo.c
renamed this object file as foo_fpic.o
then
gcc -c -Wall -Werror foo.c
renamed this object file as foo_nonfpic.o

readelf -h foo_fpic.o
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: REL (Relocatable file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x0
Start of program headers: 0 (bytes into file)
Start of section headers: 320 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 0 (bytes)
Number of program headers: 0
Size of section headers: 64 (bytes)
Number of section headers: 13
Section header string table index: 10

readelf -h foo_nonfpic.o
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: REL (Relocatable file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x0
Start of program headers: 0 (bytes into file)
Start of section headers: 312 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 0 (bytes)
Number of program headers: 0
Size of section headers: 64 (bytes)
Number of section headers: 13
Section header string table index: 10


I see the type in both cases
Type: REL (Relocatable file)
start of section headers: 312 (bytes into file)//non fpic
start of section headers: 320 (bytes into file)//fpic

nm -r foo_fpic.o
U puts
U _GLOBAL_OFFSET_TABLE_ //something extra created inside object file
0000000000000000 T foo

nm -r foo_nonfpic.o
U puts
0000000000000000 T foo

So I can see that in object file creation there is some difference. Can I think that in linking with -fpic the object code of foo.c will contain relevant information where is each and every variable will be at what offset to text.This is the GOT.Again these are offsets from text segment and are not absolute address.

In non fpic code this information is not present so a burden is at loader to find proper places to to load this shared code. Also it have to calculate itself GOT at that time itself.

Is the above understanding correct?

If yes then,
both code actually in the end have to be relocated to somewhere in RAM. The GOT only contain information about internal offsets. Same situation in relocateble code.
So I say the only difference is PIC contain GOT, is relocatable. The GOT is missing when we use nonfpic compilation.

[acidla...@gmail.com]

Pardon me if I am putting logic in some fashion that is totally bullshit. I started reading

https://www.iecc.com/linker/

To understand the underlying requirements and different approaches in linkers and loaders.

[Manfred]

On 3/27/2017 4:25 AM, Robert Wessel wrote:
> On Mon, 27 Mar 2017 00:59:42 +0200, Manfred <inv...@invalid.add>
> wrote:
>
>> On 3/26/2017 8:21 PM, acidla...@gmail.com wrote:
>>> On Sunday, March 26, 2017 at 11:31:59 PM UTC+5:30, acidla...@gmail.com wrote:
>> <snip>
>>>
>>> I am coming to the conclusion that PIC code and relocatable code are same.
>> IIRC they are not the same.
>> There is a relevant difference about security.
>> Relocatable code requires that some memory area be both executable and
>> writable during load, which poses a security vulnerability.
>
>
> If the memory is not writable during the load, the OS is not going to
> be able to load the program object, PIC or not! Nor do most OS's (at
> least those that routinely make program text areas un-writable),
> actually execute any of the *program* code while loading and
> performing the relocations.
>

"How To Write Shared Libraries" by Ulrich Drepper:

https://software.intel.com/sites/default/files/m/a/1/e/dsohowto.pdf

Section 1.4:
"Modern operating systems and processors can protect mem-
ory regions to allow and disallow reading, writing, and
executing separately for each page of memory.
It is preferable to mark as many pages as possible not writable
since this means that the pages can be shared between
processes which use the same application or DSO the
page is from. Write protection also helps to detect and
prevent unintentional or malignant modifications of data
or even code."

Section 1.5.5:
"... The call to bar requires two relocations: one to load the
value of foo and another one to find the address of bar.
...
This would encode the addresses of foo and bar as part
of the instruction in the text segment. If the address is
only known to the dynamic linker the text segment would
have to be modified at run-time. *According to what we
learned above this must be avoided*."
...
"Therefore the code generated for DSOs, i.e., when using
-fpic or -fPIC, looks like this:
...
The address of the variable foo is now not part of the in-
struction. Instead it is loaded from the GOT."...

Section 2:
"The most important recommendation is to always use -fpic or
-fPIC when generating code which ends up in DSOs. This applies to data
as well as code. Code which is not compiled this way almost certainly
will contain text relocations. For these there is no excuse"

Section 2.6 "Increasing Security":
...
"A completely different issue, but still worth mentioning
in the context of security, are text relocations. Generat-
ing DSOs so that text relocations are necessary (see sec-
tion 2) means that the dynamic linker has to make mem-
ory pages, which are otherwise read-only, temporarily
writable. The period in which the pages are writable is
usually brief, only until all non-lazy relocations for the
object are handled. But even this brief period could be
exploited by an attacker. In a malicious attack code re-
gions could be overwritten with code of the attacker’s
choice and the program will execute the code blindly if it
reaches those addresses."
...
"For this reason creating DSOs with text relocation means unnecessarily
increasing the security problems of the system"

[Manfred]

The following goes into detail about this:

"How To Write Shared Libraries" by Ulrich Drepper
https://software.intel.com/sites/default/files/m/a/1/e/dsohowto.pdf

It is also worth mentioning that according to:
https://en.wikipedia.org/wiki/Position-independent_code#Windows_DLLs

"Dynamic-link libraries (DLLs) in Microsoft Windows do not use position
independent code; instead they use relocatable code."