interposing library calls - general development
phil-new...@ipal.net
particular for libc, but possibly for any library. Google gives a number of
articles and forum posts, but all seem to be narrow for particular needs and
tend to be dominated by that need. What I really want is information about
doing the interposing in general, including descriptions of the various ways
to deal with issues such as a function that has to, in turn, call the function
of the same name in the original library. Other issues would include how to
force interposing system-wide (by the admin, of course) withing using LD_PRELOAD.
I'd prefer coverage of doing this for all Unix and Unix-like systems, but my
development will be on Linux for now and BSD and/or Solaris later (but I don't
want to paint myself into the non-portable corner). I'm looking for such
documents in order to avoid re-inventing the various solutions myself where
others have come up with better ideas.
Documents that focus on specific reasons to do library interposing are not
what I am looking for. The right document would be describing how to do the
interposing in various ways without regard to the reasons one might want to.
If you want to ask why I would want to do library interposing, please start
a different thread for that.
--
|WARNING: Due to extreme spam, googlegroups.com is blocked. Due to ignorance |
| by the abuse department, bellsouth.net is blocked. If you post to |
| Usenet from these places, find another Usenet provider ASAP. |
| Phil Howard KA9WGN (email for humans: first name in lower case at ipal.net) |
Johannes Bauer
> I'm looking for any documents on developing interposing calls for libraries,
> particular for libc, but possibly for any library.
I've written some (GPLed) software which has not been released yet which
deals with this topic. Also I'm writing a paper about that - how soon do
you need the information? Are you working on a publication?
Regards,
Johannes
--
"Wer etwas kritisiert muss es noch lange nicht selber besser können. Es
reicht zu wissen, daß andere es besser können und andere es auch
besser machen um einen Vergleich zu bringen." - Wolfgang Gerber
in de.sci.electronics <47fa8447$0$11545$9b62...@news.freenet.de>
phil-new...@ipal.net
| phil-new...@ipal.net schrieb:
|> I'm looking for any documents on developing interposing calls for libraries,
|> particular for libc, but possibly for any library.
|
| I've written some (GPLed) software which has not been released yet which
| deals with this topic. Also I'm writing a paper about that - how soon do
| you need the information? Are you working on a publication?
It's not for a publication, although I'd be willing to host any documents
that help people do this kind of thing. Instead, it is for development
that I am contemplating. This is something I can figure out. It's just
that I want to cheat and use other people errors instead of my own errors
for the trial and error part to figure out all the details and nuances
that would be involved. If you paper is general purpose regarding the
interposing of library calls, I'm interested. That doesn't rule out real
life examples; it just shouldn't focus on them as some other tools do
(for example one that tracks library call statistics).
William Ahern
<snip>
> Documents that focus on specific reasons to do library interposing are not
> what I am looking for. The right document would be describing how to do the
> interposing in various ways without regard to the reasons one might want to.
man 3 dlsym
The handle you want supported is RTLD_NEXT. It's a Sun invention, so Solaris
and Linux support it. FreeBSD, OpenBSD, and OS X should support this. You'll
have to confirm for NetBSD and others Unices, but it's been picked up in
POSIX/SUSv3, so it's officially standardized (technically as an extension).
This particular interface doesn't exist on Win32, of course.
Note that linking order is important (as usually on Unix). If you want to
interpose a non-libc library from another library, you'll have to link it
first (specify it first when calling the linker). If you're interposing from
the main application, you're fine. If you can't control linking look for
something like LD_PRELOAD support. LD_PRELOAD is an ENV variable that
specifies a library to load and link into the main application ahead of
other shared libraries; for instance, on OS X it's call DYLD_PRELOAD or some
such.
You may run into issues with interposing system calls. This used to be (and
perhaps may still be) an issue in OpenBSD, for example. Also static linking
and statically scoped functions might (or will, in the latter case) cause
problems.
If you're looking for a magic solution that works around all the caveats I
listed, yet is reasonably portable... you have my condolences.
phil-new...@ipal.net
I'm looking NOT for a few little fundamental basics. These I already have.
What I am looking for is a document that covers all the complexities and
pitfalls, and how to get around them, including any platform specific issues.
I don't care about Win32.
I mostly want to interpose at run time. But coverage of how to do this at
link time would be an interesting topic, too.
I do expect issues in interposing suid-root programs. End users should never
be able to do that. But the system administrator should be able to on a system
wide basis.
David Schwartz
> I'm looking for any documents on developing interposing calls for libraries,
> particular for libc, but possibly for any library. Google gives a number of
> articles and forum posts, but all seem to be narrow for particular needs and
It sounds like 'ltrace' does exactly what you want.
http://freshmeat.net/projects/ltrace/
That may be too heavy though. Is this meant as an invasive debug/
troubleshooting aid? Or is meant as a mostly-transparent system
enhancement?
DS
William Ahern
> I'm looking NOT for a few little fundamental basics. These I already have.
> What I am looking for is a document that covers all the complexities and
> pitfalls, and how to get around them, including any platform specific issues.
You're coming at this w/ the wrong perspective (and attitude). The document
is the dlsym manual page. Did you read it?
> I don't care about Win32.
> I mostly want to interpose at run time. But coverage of how to do this at
> link time would be an interesting topic, too.
It's part-and-parcel. (Unless you literally want to be able to arbitrarily
decide, at run-time, to interpose a particular call for which you do not
know the name during link-time, in which case there is no single solution
that would even work on the same box.)
Read the dlsym man page. If you want a tutorial, here you go. On OpenBSD I
can compile w/ `make interpose`; "interpose.c" as the source, and
"interpose" as resultant binary executable. On Linux you may need to do
`make interpose LDFLAGS="-ld"`.
#include <stdarg.h> /* va_list va_start va_arg va_end */
#include <stdio.h> /* FILE fopen(3) fprintf(3) */
#include <assert.h> /* assert(3) */
#include <sys/types.h> /* mode_t */
#include <fcntl.h> /* O_CREAT open(2) */
#include <dlfcn.h> /* RTLD_NEXT dlsym(3) */
int open(const char *path, int flags, ...) {
int (*sym)();
va_list ap;
mode_t mode;
int fd;
assert(0 != (sym = dlsym(RTLD_NEXT, "open")));
if (flags & O_CREAT) {
va_start(ap, flags);
mode = va_arg(ap, int);
va_end(ap);
} else
mode = 0;
fd = sym(path, flags, mode);
fprintf(stderr, "open(%s) => %d\n", path, fd);
return fd;
}
int main(void) {
/* Prove that we can interpose libc's open(2) call. */
FILE *fp = fopen("/dev/null", "w");
return 0;
}
John Reiser
> /* Prove that we can interpose libc's open(2) call. */
> FILE *fp = fopen("/dev/null", "w");
That example works, but there are others that do not satisfy
the intuitive goal of "interposing open(2) means that I see
all pathnames that are presented to the operating system via
the SYS_open system call."
glibc also has symbols such as:
3785: 005eb32c 205 FUNC LOCAL HIDDEN 11 __GI___open64
4354: 005eb2b0 122 FUNC LOCAL DEFAULT 11 __GI_open
4660: 005eb2b0 122 FUNC LOCAL DEFAULT 11 __GI___open
4670: 0058536a 180 FUNC LOCAL DEFAULT 11 __fopen_internal
4696: 005eb32c 205 FUNC LOCAL DEFAULT 11 __libc_open64
4966: 005eb2b0 122 FUNC LOCAL DEFAULT 11 __libc_open
5395: 005eb2b0 122 FUNC LOCAL DEFAULT 11 __GI___libc_open
which invoke SYS_open without the benefit of a visible runtime
resolution of the name "open". The presented code does not see these.
If the goal is to log and/or modify all pathnames that are passed
to any SYS_open, then the presented code fails in cases such as these.
The only way to capture all the pathnames is to use a facility
such as strace, or an instruction-by-instruction emulator.
--
William Ahern
> William Ahern wrote:
> > /* Prove that we can interpose libc's open(2) call. */
> > FILE *fp = fopen("/dev/null", "w");
> That example works, but there are others that do not satisfy
> the intuitive goal of "interposing open(2) means that I see
> all pathnames that are presented to the operating system via
> the SYS_open system call."
Indeed. The purpose of the example, however, was merely to show that the
scope of the interposed open() implementation was not an inadvertent trick.
> glibc also has symbols such as:
> 3785: 005eb32c 205 FUNC LOCAL HIDDEN 11 __GI___open64
> 4354: 005eb2b0 122 FUNC LOCAL DEFAULT 11 __GI_open
> 4660: 005eb2b0 122 FUNC LOCAL DEFAULT 11 __GI___open
> 4670: 0058536a 180 FUNC LOCAL DEFAULT 11 __fopen_internal
> 4696: 005eb32c 205 FUNC LOCAL DEFAULT 11 __libc_open64
> 4966: 005eb2b0 122 FUNC LOCAL DEFAULT 11 __libc_open
> 5395: 005eb2b0 122 FUNC LOCAL DEFAULT 11 __GI___libc_open
> which invoke SYS_open without the benefit of a visible runtime
> resolution of the name "open". The presented code does not see these.
And in fact, the _reason_ that those functions exist is specifically so that
we can play shenanigans w/ the global "open" symbol. Thus, glibc is not
written to intentionally subvert syscall interposition; quite the opposite.
Last time I was interested in interposition (several years ago), OpenBSD
lacked this sort of indirection, and it was a royal PITA to interpose system
calls (and nigh impossible for library symbols, because OpenBSD lacked
support for RTLD_NEXT). OpenBSD at the time merely relied on GCC's weak
aliasing feature when declaring system calls, which allowed libraries like
GNU libpth to work (albeit w/ much #ifdef'd code). But not much else was
possible.
> If the goal is to log and/or modify all pathnames that are passed
> to any SYS_open, then the presented code fails in cases such as these.
> The only way to capture all the pathnames is to use a facility
> such as strace, or an instruction-by-instruction emulator.
Only the emulator. Any other system requires kernel modifications to
guarantee consistency. Linux ptrace(2) is insufficient, for instance. See
this paper for how Systrace (and all similarly composed systems) was broken:
Exploiting Concurrency Vulnerabilities in System Call Wrappers
http://www.usenix.org/events/woot07/tech/full_papers/watson/watson.pdf
So, in short, dlsym(3) gives all the guarantees, more-or-less, that one
could ever reasonably expect from the system.
phil-new...@ipal.net
| In article <g41aj...@news5.newsguy.com> you wrote:
|> I'm looking NOT for a few little fundamental basics. These I already have.
|> What I am looking for is a document that covers all the complexities and
|> pitfalls, and how to get around them, including any platform specific issues.
|
| You're coming at this w/ the wrong perspective (and attitude). The document
| is the dlsym manual page. Did you read it?
The Debian server I have (it's a rented one, I didn't install the OS) does not
even have that man page (so I'll have to hunt down what package it's in and
add that). But my Slackware machines do have it (latest installed is 12.0).
But, that dlsym man page is NOT what I want to know now. I do understand that
what it does say is a good starting point. But it doesn't cover experiences
in doing this.
Like I say, it (dlsym docs) is a starting point. I am past the starting point.
I'm looking for stuff the covers the whole picture. I can accept that it does
not (yet) exist. In which case I'll have to do the rest on my own and I will
consider writing a document on how to do this.
Tell me what perspective/attitude you think I should have? I assume you have
that which you would promote; could you explain how it works well for you?
|> I mostly want to interpose at run time. But coverage of how to do this at
|> link time would be an interesting topic, too.
|
| It's part-and-parcel. (Unless you literally want to be able to arbitrarily
| decide, at run-time, to interpose a particular call for which you do not
| know the name during link-time, in which case there is no single solution
| that would even work on the same box.)
I want to interpose on calls to various library functions, mostly in libc,
that will affect dynamically linked programs that are already completely
linked as much as a dynamic executable would be (e.g. the final linking to
the dynamic libraries happens at run time by ld.so).
It's a nice simple example. My projects aren't all that simple. Some will
be doing things like forking additional processes. Maybe I want that new
process to also be interposed, or maybe not at all. I want to know how that
will interfere with interposing. Again, I'm not looking for the basic how
to set it up ... I'm looking for the big picture of issues with interposing
and the kinds of solutions people have already worked out.
phil-new...@ipal.net
Potentially a need to interpose every open could interpose these other
function entry points as well. Maybe. But I'm sure you will point out
that this doesn't guarantee 100% interposing.
It is issues like these I'm wanting to read about. And if that means the
solution ultimately needs to be an OS syscall level intercept like ptrace
in Linux, than I think the document would need to cover that, too (and how
on at least a few major platforms like BSD, Linux, and Solaris).
phil-new...@ipal.net
I remember that. One solution I read about involed a couple layers of
interposing with a pseudo-API inserted in between to use different symbols.
One interposing library was somehow linked to libc so it would not be
interposed upon, and another library called this first library's alternate
symbols, and did the interposing. There are some details I don't remember
that may not have even been in what I read.
phil-new...@ipal.net
| On Jun 26, 10:28?am, phil-news-nos...@ipal.net wrote:
|> I'm looking for any documents on developing interposing calls for libraries,
|> articles and forum posts, but all seem to be narrow for particular needs and
|
| It sounds like 'ltrace' does exactly what you want.
| http://freshmeat.net/projects/ltrace/
|
| That may be too heavy though. Is this meant as an invasive debug/
| troubleshooting aid? Or is meant as a mostly-transparent system
| enhancement?
Are you asking about ltrace or asking me what my interposing projects are?
My interposing is meant to be transparent production. That is, once it is
implemented and deployed, it will be left in place. It is not for debugging,
but that might be done as well for other things.
William Ahern
<snip>
> |> I mostly want to interpose at run time. But coverage of how to do this at
> |> link time would be an interesting topic, too.
> |
> | It's part-and-parcel. (Unless you literally want to be able to arbitrarily
> | decide, at run-time, to interpose a particular call for which you do not
> | know the name during link-time, in which case there is no single solution
> | that would even work on the same box.)
> I want to interpose on calls to various library functions, mostly in libc,
> that will affect dynamically linked programs that are already completely
> linked as much as a dynamic executable would be (e.g. the final linking to
> the dynamic libraries happens at run time by ld.so).
Then you want dlsym.
<snip>
> It's a nice simple example.
It's both simple _and_ complete.
> My projects aren't all that simple. Some will be doing things like
> forking additional processes. Maybe I want that new process to also be
> interposed, or maybe not at all. I want to know how that will interfere
> with interposing.
Forking won't interfere. You want dlsym.
> Again, I'm not looking for the basic how to set it up I'm looking for the
> ... big picture of issues with interposing and the
> kinds of solutions people have already worked out.
What I'm hearing then, is not that you want to simply learn how to
interpose. Rather, you wish to understand how linking works. Interposition
is a relatively high-level mechanism; it's implicit that the
interpositioning interface will (or should) abstract the gory linking
details, and do-what-you-want.
Your evident frustration stems from asking the wrong question(s). That's
what happens when you start out saying, "don't ask me why...".
There's no single document that will clearly lay out how linking works on
every platform, or even the different mechanisms on a single platform. It
couldn't hurt to read the ELF specification, a particular spec for dynamic
linking which covers most of the bases. I've never read the ELF spec, but
you can work backwards from this specification for implementing thread-local
storage within ELF:
http://people.redhat.com/drepper/tls.pdf
Or you could just start at Wikipedia. In any event, Usenet is not the best
place for these sorts of more general questions. Google is a superior tool
for that task.
The final answer, however, is still most likely "dlsym". It couldn't hurt to
use it. If and when it fails you, then you'll have much better context for
understanding what it's doing, and what exactly you're trying to do (beyond
a mere wish). Answers don't make much sense w/o the proper question.
phil-new...@ipal.net
| <snip>
|> |> I mostly want to interpose at run time. But coverage of how to do this at
|> |> link time would be an interesting topic, too.
|> |
|> | It's part-and-parcel. (Unless you literally want to be able to arbitrarily
|> | decide, at run-time, to interpose a particular call for which you do not
|> | know the name during link-time, in which case there is no single solution
|> | that would even work on the same box.)
|
|> I want to interpose on calls to various library functions, mostly in libc,
|> that will affect dynamically linked programs that are already completely
|> linked as much as a dynamic executable would be (e.g. the final linking to
|> the dynamic libraries happens at run time by ld.so).
|
| Then you want dlsym.
I know that dlsym is a component of the things that need to be done.
But just saying this alone is extreme over-simplification of what is
involved.
|> It's a nice simple example.
|
| It's both simple _and_ complete.
But it's not relevant to _all_ issues. It is relevant to one particular
need to do library interposing and in that sense it does appear to be
complete and is probably correct as well.
|> My projects aren't all that simple. Some will be doing things like
|> forking additional processes. Maybe I want that new process to also be
|> interposed, or maybe not at all. I want to know how that will interfere
|> with interposing.
|
| Forking won't interfere. You want dlsym.
What I wonder about is why yoy are thinking I might be looking for
something other than dlsym. You're still only giving me the basic
situation.
|> Again, I'm not looking for the basic how to set it up I'm looking for the
|> ... big picture of issues with interposing and the
|> kinds of solutions people have already worked out.
|
| What I'm hearing then, is not that you want to simply learn how to
| interpose. Rather, you wish to understand how linking works. Interposition
| is a relatively high-level mechanism; it's implicit that the
| interpositioning interface will (or should) abstract the gory linking
| details, and do-what-you-want.
|
| Your evident frustration stems from asking the wrong question(s). That's
| what happens when you start out saying, "don't ask me why...".
I do not want to dwell on ONE SINGLE EXAMPLE of what to do with interposing.
I have many YEARS of experience knowing how Usenet people respond to such
things. They get deep into that particular example beyond the questions being
asked, and entirely off topic.
I have MORE THAN ONE thing I want to do that will, or for some cases may,
need to use library interposing. I'm not going to post them all because for
one thing it will look like I'm trying to get others to do the work I should
be doing. But posting any SUBSET of cases doesn't cover the whole picture,
either.
| There's no single document that will clearly lay out how linking works on
| every platform, or even the different mechanisms on a single platform. It
| couldn't hurt to read the ELF specification, a particular spec for dynamic
| linking which covers most of the bases. I've never read the ELF spec, but
| you can work backwards from this specification for implementing thread-local
| storage within ELF:
I'm only interested in the SUBSET of information focused only on interposing.
Anything else is a waste of my current time. It could well be that other
information may be useful for other projects in the future. But there is no
point in learning what I need for project B when project A needs to be done
before project B and I need information for project A.
| http://people.redhat.com/drepper/tls.pdf
At least this is something. I'll go download it and read it and see if it
says anything along the lines of what I want to do.
| Or you could just start at Wikipedia. In any event, Usenet is not the best
| place for these sorts of more general questions. Google is a superior tool
| for that task.
I was not asking for people to write posts to explain it. I was asking for
pointers to existing documents that I was unable to find via Google. Google
usually is a superiod tool. But it doesn't always work as expected. One
possible answer that sometimes gets posted in Usenet or some forum is an idea
for alternative keywords to use with Google, because some other term is used
to describe the activity. What is wrong with searching for "interposing"
along with terms like "library", "run time", "BSD", "Linux", and "dlsym"?
| The final answer, however, is still most likely "dlsym". It couldn't hurt to
| use it. If and when it fails you, then you'll have much better context for
| understanding what it's doing, and what exactly you're trying to do (beyond
| a mere wish). Answers don't make much sense w/o the proper question.
It's not an answer to my question. My question is for documentation about the
whole general effort. If it doesn't exist (and I suspect that being the case
if Google can't find any), then I'm OK with that (not like I have a choice).
I will go read "http://people.redhat.com/drepper/tls.pdf".
If I asked how to write a program that would list a directory tree of files,
would your answer be "readdir"? There's a lot MORE to listing a tree of files
than just knowing to use "readdir". While it wouldn't be a whole book, it
could be the better part of a chapter. It could be a substantial web page
document or possibly more than one. I've already done such development in
more than one way. And maybe I should have written up about it for the benefit
of others so they don't have to stumble through the trials and errors I did on
the parts beyond just doing "readdir".
Bernie Ohls
> I'm looking for any documents on developing interposing calls for libraries,
> particular for libc, but possibly for any library.
I have been down this same road, using interposition to develop
production-quality SW. Unfortunately, though interposition is actually
quite stable, powerful, and (nowadays) pretty portable, it is still
generally used only as a one-off debugging aid or as a workaround for a
bug in a particular OS or app (typically something like overriding the
uname() call to allow a program to run on a system it wasn't designed
for). There appears to be no document describing how to use it in a
"grown-up" way wrt portability, thread-safety and re-entrancy, avoiding
deadlocks in intra-library calls, etc.
In particular I know of no reliable, automated way of determining which
symbols need to be intercepted. But these things can generally be worked
out via nm and elfdump and so on, and an individual platform's ABI
commitments usually ensure that they won't change (additions are a
different story of course).
So I hope you or someone finds and posts such a document but I don't
hold out much hope. In the meantime I would recommend the Sun Linker and
Libraries Guide (http://docs.sun.com/app/docs/doc/817-1984). Although it
only speaks officially for Sun, it's really about ELF/SysV linking and
most other extant systems have worked from the same spec. Linux and *BSD
for sure.
My situation has been simplified by the fact that I'm not interested in
intra- or inter-library calls but only in explicit calls from the
application. In fact ignoring those library calls can be a feature. For
instance, thinking of opened files, using truss or strace reveals a
number of file opens done by libraries which you may find uninteresting
(such as locale files). Sticking to the app layer ignores all that stuff
and tells us only what the application does explicitly. In other words
it reveals more about the app logic and less about the system
implementation. But interposition can also be used to grab those calls
if you choose to.
You've said that you don't want simple examples and tips but I will
throw in a couple:
1. There's a real risk of deadlock of course; even reporting an error
can lock up if it uses one of the symbols you're interposing on. I've
had pretty good luck playing games with the C preprocessor here. I.e.
interpose on "open" to get a symbol "my_open", then "#define open
my_open". This way any code doing a simple-minded 'open("foo")' won't
get into trouble. Otherwise it becomes mind-bending trying to keep track
of what symbol means what when. For a while I was actually interposing
on dlopen() - that led to some situations which made the old bean hurt.
2. Despite what others have said, process creation is painful. There's
no problem with fork() as long as you're careful. The problem is vfork.
Here's a quote from the Sun man page:
The vfork() function can normally be used the same way as fork().
The procedure that called vfork(), however, should not return
while running in the child's context, since the eventual return
from vfork() would be to a stack frame that no longer exists.
You see the problem; it's impossible to wrap vfork since the wrapper is
not allowed to return. I've "solved" this by intercepting vfork() calls
and turning them into a real fork(). This works most of the time but is
not ideal. On a given platform and CPU it may be possible to work around
this with handcrafted assembler code but I haven't tried yet. If you
solve this I'd appreciate it if you'd post the solution.
Bernie O.
John Reiser
> The problem is vfork.
[snip]
> it's impossible to wrap vfork since the wrapper is
> not allowed to return. I've "solved" this by intercepting vfork() calls
> and turning them into a real fork(). This works most of the time but is
> not ideal. On a given platform and CPU it may be possible to work around
> this with handcrafted assembler code but I haven't tried yet. If you
> solve this I'd appreciate it if you'd post the solution.
You already discovered the solution: replace vfork with fork.
Everyone has agreed that vfork is a performance optimization only.
If replacing vfork with real fork creates any semantic difference,
then that is a certified bug in the code which [mis-]uses vfork.
--
David Schwartz
> My interposing is meant to be transparent production. That is, once it is
> implemented and deployed, it will be left in place. It is not for debugging,
> but that might be done as well for other things.
Then I wouldn't use the method that 'ltrace' uses. I would use the
dlsym LD_PRELOAD mechanism discussed elsewhere. The 'ltrace' mechanism
is not really suitable for a transparent production use.
DS
Bernie Ohls
There's a lot of history around vfork, in c.u.p as well as other places.
I don't want to go over it, and it's not topical here anyway, but the
short version is that when a system is running low on virtual memory a
vfork/exec sequence might succeed where a fork/exec would fail. This is
why I say my solution is not ideal. Admittedly it's a corner case.
Bernie O.
Rainer Weikusat
Assuming the system is really 'running low on virtual memory', it is
conceivable that vfork succeeds, but exec fails :->.
phil-new...@ipal.net
| You've said that you don't want simple examples and tips but I will
| throw in a couple:
Thoe documents that only have simple examples tend to be incomplete with
respect to the whole picture that I'm looking for a single document to
explain. But, a collection of simple examples would be good towards me
writing such a document. Given that I have not found a single big
picture document, it looks like this is what I will have to work with.
| 1. There's a real risk of deadlock of course; even reporting an error
| can lock up if it uses one of the symbols you're interposing on. I've
| had pretty good luck playing games with the C preprocessor here. I.e.
| interpose on "open" to get a symbol "my_open", then "#define open
| my_open". This way any code doing a simple-minded 'open("foo")' won't
| get into trouble. Otherwise it becomes mind-bending trying to keep track
| of what symbol means what when. For a while I was actually interposing
| on dlopen() - that led to some situations which made the old bean hurt.
I can see where that would be painful.
| 2. Despite what others have said, process creation is painful. There's
| no problem with fork() as long as you're careful. The problem is vfork.
| Here's a quote from the Sun man page:
|
| The vfork() function can normally be used the same way as fork().
| The procedure that called vfork(), however, should not return
| while running in the child's context, since the eventual return
| from vfork() would be to a stack frame that no longer exists.
|
| You see the problem; it's impossible to wrap vfork since the wrapper is
| not allowed to return. I've "solved" this by intercepting vfork() calls
| and turning them into a real fork(). This works most of the time but is
| not ideal. On a given platform and CPU it may be possible to work around
| this with handcrafted assembler code but I haven't tried yet. If you
| solve this I'd appreciate it if you'd post the solution.
Since vfork() is not supposed to return to the parent until the child calls
execve() or _exit(), it might seem like vfork() can be wrapped. The problem
is that the wrapped context exists in the child, too. When the child returns
it modifies the stack frame both child and parent share. And the child does
need to have the wrapper return since the program will be coded under the
assumption it will be doing the execve() call after vfork(). The solution
I see is very non-portable, which requires the wrapper to manipulate the
stack frame and make a copy before the systcall, and restore it afterwards.
But who knows what all the pitfalls of this will be.
If I write the big picture document, I'll try to describe why vfork() is one
of the syscall stubs that should not be interposed by function call wrapping.
phil-new...@ipal.net
Or if it is even lower on virtual memory, everything can fail. I'd call it
more of a boundary case than a corner case. Correct programs can fail for
reasons like this. They simply need to handle errors in as correct a way
as possible.
Paul Pluzhnikov
> In comp.unix.programmer John Reiser <jre...@bitwagon.com> wrote:
...
> | glibc also has symbols such as:
> | 3785: 005eb32c 205 FUNC LOCAL HIDDEN 11 __GI___open64
> | 4354: 005eb2b0 122 FUNC LOCAL DEFAULT 11 __GI_open
...
> Potentially a need to interpose every open could interpose these other
> function entry points as well.
Except you can't interpose them without "extreme hackery": the
symbols are local, which means they don't go through "normal"
lazy binding. Rather, there are direct 'call 0x5eb2b0' instructions
inside glibc, and if you want to "steal" them, you have to patch
.text, or place breakpoints (ala "ltrace").
> It is issues like these I'm wanting to read about. And if that means the
> solution ultimately needs to be an OS syscall level intercept like ptrace
> in Linux, than I think the document would need to cover that, too (and how
> on at least a few major platforms like BSD, Linux, and Solaris).
I have been doing large amount of libc interposing on Linux, Solaris,
HP-UX and AIX for the last 10 years, and have not seen a document
you are seeking.
From my experience, the issues you are likely to encounter are:
1. Partial interception (e.g. interposing mmap() does not get you
intra-glibc calls to mmap() from e.g. fopen()).
This is the issue John raised with open(); there are many more
such "internal" calls in glibc, and some in Solaris libc as well.
2. Startup dependencies: libc startup is a complicated dance,
where things work only partially while libc is starting itself up,
and calling things in the "wrong" order often leads to a crash
either due to NULL pointer dereference (some global pointer
is expected to have been set up, but hasn't been set yet),
or infinite recursion (discover that some global pointer hasn't
been set up yet, try to set it, hit interposer, discover that
the global pointer hasn't been set up yet, spin into the ground ...).
In addition, libc startup of threaded programs is even more
complicated, as various pthread_* routines expect parts of libc
they depend on to have been initialized beforehand, and
interposing can severely change the order of initialization.
3. Interposing symbols that are called during dlsym(): on Linux,
for programs linked against libpthread, dlsym() calls calloc().
If you interpose calloc(), you can't find the "original" glibc
calloc() via 'dlsym(RTLD_NEXT, "calloc")' because that will
result in infinite recursion.
4. Symbol versioning: there are two incompatible versions of
e.g. pthread_cond_init() inside glibc, and if you have 3rd
party libraries in the picture, it's quite possible for both of
these versions to be actually used. Correctly interposing both
symbols requires some tricks, and the use of dlvsym().
Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.
phil-new...@ipal.net
| phil-new...@ipal.net writes:
|
|> In comp.unix.programmer John Reiser <jre...@bitwagon.com> wrote:
| ...
|> | glibc also has symbols such as:
|> | 3785: 005eb32c 205 FUNC LOCAL HIDDEN 11 __GI___open64
|> | 4354: 005eb2b0 122 FUNC LOCAL DEFAULT 11 __GI_open
| ...
|> Potentially a need to interpose every open could interpose these other
|> function entry points as well.
|
| Except you can't interpose them without "extreme hackery": the
| symbols are local, which means they don't go through "normal"
| lazy binding. Rather, there are direct 'call 0x5eb2b0' instructions
| inside glibc, and if you want to "steal" them, you have to patch
| .text, or place breakpoints (ala "ltrace").
Ah, yes, they would be statically linked within the linking of the
library itself. So this means that library interposing might have
to be limited to API scope. Beyond that is ptrace() intercepting
(or whatever is equivalent to that Linux call on other systems).
|> It is issues like these I'm wanting to read about. And if that means the
|> solution ultimately needs to be an OS syscall level intercept like ptrace
|> in Linux, than I think the document would need to cover that, too (and how
|> on at least a few major platforms like BSD, Linux, and Solaris).
|
| I have been doing large amount of libc interposing on Linux, Solaris,
| HP-UX and AIX for the last 10 years, and have not seen a document
| you are seeking.
OK. I've pretty much ended the search, anyway.
| From my experience, the issues you are likely to encounter are:
|
| 1. Partial interception (e.g. interposing mmap() does not get you
| intra-glibc calls to mmap() from e.g. fopen()).
|
| This is the issue John raised with open(); there are many more
| such "internal" calls in glibc, and some in Solaris libc as well.
One would have to be wanting to interpose only what the application
is directly calling for this to be useful or clean, then.
| 2. Startup dependencies: libc startup is a complicated dance,
| where things work only partially while libc is starting itself up,
| and calling things in the "wrong" order often leads to a crash
| either due to NULL pointer dereference (some global pointer
| is expected to have been set up, but hasn't been set yet),
| or infinite recursion (discover that some global pointer hasn't
| been set up yet, try to set it, hit interposer, discover that
| the global pointer hasn't been set up yet, spin into the ground ...).
|
| In addition, libc startup of threaded programs is even more
| complicated, as various pthread_* routines expect parts of libc
| they depend on to have been initialized beforehand, and
| interposing can severely change the order of initialization.
|
| 3. Interposing symbols that are called during dlsym(): on Linux,
| for programs linked against libpthread, dlsym() calls calloc().
|
| If you interpose calloc(), you can't find the "original" glibc
| calloc() via 'dlsym(RTLD_NEXT, "calloc")' because that will
| result in infinite recursion.
A new call to calloc() via the interposers call ... a nice gotcha to
know about. One would have to be sure they are not interposing the
library itself (e.g. calls inside the library don't get interposed).
Not sure how that is done.
| 4. Symbol versioning: there are two incompatible versions of
| e.g. pthread_cond_init() inside glibc, and if you have 3rd
| party libraries in the picture, it's quite possible for both of
| these versions to be actually used. Correctly interposing both
| symbols requires some tricks, and the use of dlvsym().
|
| Cheers,
These are the kinds of things I was looking for. Thanks for the starter clues.
