How To: Fail a load of an SO?
Jeffrey Walton
How does one non-destructively fail a load of an SO file? The code
below definitely fails to load, but it takes out the hosting
executable also. I've also tried wrapping the call to dlopen() in a
try/catch.
I'm also looking for two behaviors if initialization fails:
(1) the loader unmaps the SO
(2) the loader returns NULL to the EXE from dlopen()
Thanks,
Jeff
// Hosting Executable
void main() {
// testlib is on path for the loader
void* plib = dlopen("testlib.so", RTLD_LAZY);
....
}
// Shared Object
void LibInitialize() __attribute__((constructor(255)));
...
void LibInitialize()
{
// simulate failed initialization
// Is there another way to indicate failure
// other than throwing?
throw std::exception();
}
David Schwartz
Just implement a 'load okay' function in the library. Have the program
that loads it call that function, and if it returns an error, unload
the library.
DS
Jeffrey Walton
I don't really want to require the program using the lib to call
Initialize(). There's a few of reasons:
* The programmer must actually remember to call it
* I can't enforce that the program will unmap the lib if
Initialization fails
* Possible concurrency issue when two instances are initializing at
the same time
If I can fail initialization through the loader,the programmer does
not have to explicitly call it; and I'm 'ensured a "NO means NO"
policy (ie, unload on failure). I also expect that the OS has locks in
place, so I would get the third item for free.
Jeff
David Schwartz
> I don't really want to require the program using the lib to call
> Initialize(). There's a few of reasons:
> * The programmer must actually remember to call it
Any interface between a program and a shared library will have
requirements the programmer must remember to meet.
> * I can't enforce that the program will unmap the lib if
> Initialization fails
So? If the application is hostile, it can map the library without ever
executing it.
> * Possible concurrency issue when two instances are initializing at
> the same time
You have the same issue with calling 'dlopen' or with any initializer.
> If I can fail initialization through the loader,the programmer does
> not have to explicitly call it; and I'm 'ensured a "NO means NO"
> policy (ie, unload on failure). I also expect that the OS has locks in
> place, so I would get the third item for free.
You can't compel unload on failure against a malicious application. A
library never needs to compel an application to do anything because
the relationship is necessarily one of trust. A sufficiently clever
programmer can map your library and call its functions without ever
running your initializer if he was of a mind to.
DS
Jeffrey Walton
> On Dec 18, 9:32 am, Jeffrey Walton <noloa...@gmail.com> wrote:
>
> > I don't really want to require the program using the lib to call
> > Initialize(). There's a few of reasons:
> > * The programmer must actually remember to call it
>
> Any interface between a program and a shared library will have
> requirements the programmer must remember to meet.
resort :). I can actually make it easier on the programmer (which I
don't mind doing), but I need help from the OS.
>
> > * I can't enforce that the program will unmap the lib if
> > Initialization fails
>
> So? If the application is hostile, it can map the library without ever
> executing it.
I'm not sure what you are getting at here. If the hostile program open
the SO as READ (opposed to READ_EXECUTE), then the SO is inert since
it cannot be executed. If the adversary want to load for execution, I
can take reasonable steps to ensure integrity. But if the integrity
checks fail, I need help from the OS to get the SO out of memory.
Again, I don't trust a naive (and surely not a malicious) programmer
to unmap the SO.
> > * Possible concurrency issue when two instances are initializing at
> > the same time
>
> You have the same issue with calling 'dlopen' or with any initializer.
Agreed. My hopes were that the OS would provide the proper locks
during load. I should probably ask, does Linux lock the SO during
static and global object initialization.
>
> > If I can fail initialization through the loader,the programmer does
> > not have to explicitly call it; and I'm 'ensured a "NO means NO"
> > policy (ie, unload on failure). I also expect that the OS has locks in
> > place, so I would get the third item for free.
>
> You can't compel unload on failure against a malicious application. A
> library never needs to compel an application to do anything because
> the relationship is necessarily one of trust.
I think "... is necessarily one of trust" is the philosophical problem
here. I'm modeling for a hostile environment, and the shared object
features of the OS model for a benign environment. In fact, it appears
that the OS expects that all [initialization] functions never FAIL,
even in a benign environment. (Perhaps I'm incorrectly reading the
'inability to convey initialization failure' here).
Jeff
Rainer Weikusat
> On Dec 18, 1:16�pm, David Schwartz <dav...@webmaster.com> wrote:
[...]
>> You can't compel unload on failure against a malicious application. A
>> library never needs to compel an application to do anything because
>> the relationship is necessarily one of trust.
> I think "... is necessarily one of trust" is the philosophical problem
> here. I'm modeling for a hostile environment, and the shared object
> features of the OS model for a benign environment.
I'd say its rather that you are accustomed to 'a hostile environment'
and have some troubles adjusting to a cooperative one.
Jeffrey Walton
David Schwartz
> I think "... is necessarily one of trust" is the philosophical problem
> here. I'm modeling for a hostile environment, and the shared object
> features of the OS model for a benign environment.
Correct. Using shared libraries is *inappropriate* for a hostile
environment. Multiple processes must be used.
> In fact, it appears
> that the OS expects that all [initialization] functions never FAIL,
> even in a benign environment. (Perhaps I'm incorrectly reading the
> 'inability to convey initialization failure' here).
An initialization function can fail, and it can report that failure by
having a "report initialization status" function report that failure.
But this is fundamentally a cooperative environment. The failure is a
cooperative report.
If you do not trust the calling process to follow the API, you should
not be using a shared library.
You will not have fun trying to cram a square peg in a round hole.
Change the hole or change the peg.
DS