[9fans] dtrace for plan 9
ron minnich
mods I made to 8l and friends and use these as a way to build dtrace
for plan 9.
Had a fun talk with someone from Sun today and for at least part of
dtrace functionality the 8l mods get us part of the way there.
What they can do with dtrace is pretty impressive.
ron
Devon H. O'Dell
> I realize it is early but a neat GSOC project would be to take the
> mods I made to 8l and friends and use these as a way to build dtrace
> for plan 9.
Getting CTF working in Plan 9 would be difficult since it's not ELF.
It was annoying enough to get working in FreeBSD. As that's a sort of
first requirement, what would you suggest for starting that?
> Had a fun talk with someone from Sun today and for at least part of
> dtrace functionality the 8l mods get us part of the way there.
>
> What they can do with dtrace is pretty impressive.
Yes, it is.
(In before this thread degenerates into how DTrace is bigger than Plan
9 and how retarded it is)
> ron
--dho
erik quanstrom
> I realize it is early but a neat GSOC project would be to take the
> mods I made to 8l and friends and use these as a way to build dtrace
> for plan 9.
a more centrally plan 9 project would be to build a coverage
analysis tool. that may require some help from the compiler.
- erik
ron minnich
module which understands the dtrace language. Maybe an alternative
plan 9 approach is a kernel driver which understands acid.
ron
Jeff Sickel
some unfortunate programmer(s) with having to write hundreds of
thousands of probes instead of following the more acid based approach.
dtrace has it's place. And as you've said, eye candy wins. But I
still think there's a way to use acid and the linker to provide the
kinds of hooks you want for debugging.
-jas
dav...@mac.com
not quite total dissimilarity to awk,
I think even acid's intemperate lingo is preferable.
OTOH, the idea of chucking ANY language interpreter into a kernel
seems wrong too.
Yes, dtrace dtrace/D provide great capabilities (trust me: I
administer hairy systems and dtrace has definitely helped us),
and some of the ideas are good, but the whole thing is a mess.
Maybe I'm missing something,
but wouldn't be slightly nicer to have something like a set of dynamic
probes
which queue up blobs of data up for userland code to do the hairy
lifting on?
Also, you don't want a debugger, you want a data analyser: awk, maybe?
Dave.
ron minnich
> but wouldn't be slightly nicer to have something like a set of dynamic
> probes
> which queue up blobs of data up for userland code to do the hairy lifting
> on?
yeah. You are right. Is there any reason that D couldn't always run in
user mode, having read from a device which delivers events to it?
I'm not really that familiar with dtrace, as you can tell; I've just
read the papers, not really used it.
ron
Roman Shaposhnik
The whole point of D is to be able to execute scripts in kernel mode (and
sometime in tricky places like interrupt handlers, etc). The main reason of
doing that is to be able to filter in place what is it that you want to send
to userland /dev/dtrace and what you just want to discard. That gives you
a huge reduction in event traffic at a price of restricting D. Lack of looping
is the prime example of such a restriction.
As for your question -- I have to ask you back: why would you want to? IOW,
why would you still want to restrict yourself in user mode? If you simply want
to pipe all events to an aggregation engine won't:
# dtrace -n 'whatever spec' | awk 'do postprocessing here'
be what you really want?
Thanks,
Roman.
erik quanstrom
> sometime in tricky places like interrupt handlers, etc).
that's a scary idea. i wouldn't run anything like
that outside of testing. and so why wouldn't i just
write a bit of custom code?
- erik
matt
hmm, I'm always writing the similar debugging code for the kernel, I
know I'll write a bit of custom code.
... some time later ....
hmm, I'm always writing the similar post processing scripts, I know I'll
add a callback before each print
erik quanstrom
> know I'll write a bit of custom code.
>
> ... some time later ....
>
> hmm, I'm always writing the similar post processing scripts, I know I'll
> add a callback before each print
we have acid for a reason. dtrace isn't a debugger.
in case this has slipped by you, you can sic acid on a
running kernel.
- erik
Devon H. O'Dell
script, and the script goes through some sanity checking in the D
compiler. The bytecode is sent to the kernel to execute. There are
some in-kernel safety guarantees -- for instance, a D script causing a
nil ptr deref in kernel obviously shouldn't (and does not) cause the
system to panic.
--dho
erik quanstrom
that wasn't my main concern.
there are many interrupt routines that depend on
the exact sequence of events and probing is likely
to wreck havoc.
how do you prevent probes from reading read-to-clear
registers?
how do you know a probe in an irq routine won't set
off a latent race?
- erik
ron minnich
D is very similar in this sense to how the berkeley packet filter (the
original) worked.
ron
Nathaniel W Filardo
> On Sun Nov 1 11:55:47 EST 2009, devon...@gmail.com wrote:
> > Also, D is not compiled in kernel. The dtrace utility compiles the D
> > script, and the script goes through some sanity checking in the D
> > compiler. The bytecode is sent to the kernel to execute. There are
> > some in-kernel safety guarantees -- for instance, a D script causing a
> > nil ptr deref in kernel obviously shouldn't (and does not) cause the
> > system to panic.
>
> that wasn't my main concern.
>
> there are many interrupt routines that depend on
> the exact sequence of events and probing is likely
> to wreck havoc.
D is not able (AFAIK, anyway... the design seems like it oughtn't be) to
change the externally observable sequence of events to hardare. It may of
course change the timing of those events, but that's often less of an issue.
If you don't disable safety (as root, you can, AIUI) DTrace writes only to
its own logs, even if it can read almost everything.
> how do you prevent probes from reading read-to-clear
> registers?
DTrace has a notion of "toxic" regions of memory -- lists provided by
drivers -- which D is not allowed to access. It is possible to get these
wrong, and that's a bug, just like it would be possible to get the driver
itself wrong.
> how do you know a probe in an irq routine won't set
> off a latent race?
AIUI since D bytecode is loop-free (it permits only forward branches, making
it trivially terminating), hooks run in non-preemptable context can safely
inherit non-preemption from their hooked context.
If you've got a race between CPUs that's tickled by DTrace, it could also be
tickled by changing your compiler, or dynamic CPU clocking, or a higher
priority IRQ, or ... take your pick. That's a bug and it's not DTrace's
fault.
All that said, DTrace is not perfect and probably has some corner cases that
may cause trouble. I'm sure the Solaris developers would love to hear from
you if you find one.
--nwf;
Tim Newsham
> that outside of testing. and so why wouldn't i just
> write a bit of custom code?
Some people find the idea of writing their own kernel code "scary".
To those who don't I imagine d-trace has less appeal.
> - erik
Tim Newsham
http://www.thenewsh.com/~newsham/
erik quanstrom
> To those who don't I imagine d-trace has less appeal.
sure thing. plan 9, by being simple, makes the kernel
a much less scary place to be.
by the way, many of the same people have now decided
that library code is scary. :-)
- erik
David Leimbach
Iruata Souza
if i remember correctly, in 2006 i took Sun's SA-327-S10 course on dtrace.
after learning some D, i tried the obvious thing: enabling all probes.
something strange has happened after that. here i try to reproduce
from memory the steps i took and the strange thing i found (if anyone
is still able to reproduce the thing, i'm curious)
on a solaris 10 system:
1. log in to a CDE session
2. open two ksh terminal windows (w1 and w2 from now on, respectively)
3. type some commands at random in w1
4. type some commands at random in w2
5. with the up arrow key, check the command history on both windows
6. run dtrace enabling all probes in w2 (the system may become slow)
7. interrupt dtrace
8. check the command history of the windows
i was told dtrace was non-intrusive at the time, but w2 would show the
command history from w1.
iru
Lyndon Nerenberg
> command history from w1.
More likely this is ksh sharing a history file.
Iruata Souza
at the time i couldn't reproduce it with other shells. anyway, iirc,
no such side-effect was seen without running dtrace (that still
doesn't say anything directly against dtrace, of course).
Ethan Grammatikidis
For example, look at the updated LoopbackFS example file system: the
logging code from it has been removed because better tracing information
can be had using DTrace. Consequently, the file system code is shorter
and easier to read.
I don't know anything specific about DTrace, but I'm thinking a clear,
consistent interface for logging and tracing kernel operations sounds
like a good thing.
ron minnich
> I don't know anything specific about DTrace, but I'm thinking a clear,
> consistent interface for logging and tracing kernel operations sounds
> like a good thing.
I have seen several use cases for DTrace and one thing that I would
find useful are even more examples.
Of course, plan 9 file servers are in user space, and the fact that
DTrace can integrate kernel and user mode tracing is very handy.
ron
dav...@mac.com
<insert-obvious-and-boring-sarcastic-comment-here/>
> but I'm thinking a clear,
> consistent interface for logging and tracing kernel operations sounds
> like a good thing.
So am I, but how does this relate to dtrace?
D
dav...@mac.com
feechure of YA UNIX variant is that
one of the reasons for DTrace's complexities (hierarchical namespace,
in-kernel interpreter)
is the complexity of the Solaris kernel.
When they're trying to work out how a thread in a proc in a zone in an
ldom got to spin on a lock,
you *need* awk in your kernel to work out wtf is going on.
(In fact, for the serious cases, to me,
D feels too limited to be useful because of it's deliberate
restrictions,
yet too dangerous due to its possible unbounded runtime
).
In reality, for many cases, something simpler does the job better:
other people use DTrace to do what I would do with truss(1) and a few
joined-up braincells.
The latter uses less time and effort and doesn't Heisenberging your
kernel nearly so much.
On plan9 this is even more the case.
I'd be happy to see a BPF-style filter in the kernel for filtering
events before chucking them up to userland,
but a language interpreter and all the associated mess, overheads and
potential security holes?
No thanks.
D
hiro
> nearly so much.
I can also recommend Newtoning from a high cliff.
ron minnich
> I'd be happy to see a BPF-style filter in the kernel for filtering events
> before chucking them up to userland,
Point taken. I could pretty easily put a simple filter like that in
devtrace. It's just a question of what's needed.
ron
erik quanstrom
bpf-style filters address a situation where jitter is not
important, one potentally large packet is returned per
system call and there may be many filters sitting on the
same interfaces.
with devtrace, the situation is the opposite. jitter is
important (to eliminate), 541 small fixed-sized events can
be returned per system call and there is one trace device
per system.
tracepoints can already be inserted on a single function
entry or exit and be filtered by pid.
the common problem between bpf and devtrace is
buffer space. i haven't run into this problem yet, but
lowering KZERO by 256mb would be enough for an
additiona 7.4 million (32-bit) buffered tracepoints.
- erik
ron minnich
> with devtrace, the situation is the opposite. jitter is
> important (to eliminate), 541 small fixed-sized events can
> be returned per system call and there is one trace device
> per system.
Note that we can filter on up to 16 PIDs already. Also, you can grow
the buffer space to 256 KB -- but yes, we could grow it further.
I'd be glad to hear we don't need more complex kernel support for
devtrace. User mode is my favorite place to filter. But, that said,
putting in a simple min/max test for the params would be simple.
Anyway, we'll work on seeing what we need.
ron
erik quanstrom
> the buffer space to 256 KB -- but yes, we could grow it further.
i upped that pidwatch limit to 64 and added a binary search on the
pid array, so 64 is really an arbitrary limit. there would be
essentially no cost to upping the limit.
> I'd be glad to hear we don't need more complex kernel support for
> devtrace. User mode is my favorite place to filter. But, that said,
> putting in a simple min/max test for the params would be simple.
agreed.
- erik
Nathaniel W Filardo
> yet too dangerous due to its possible unbounded runtime
I keep hearing this brought up, but (while I am not an expert) AFAICT, the
runtime for each D hook should be strictly bounded by the number of
instructions lobbed in, since D does not (without root override, perhaps?)
support backwards jumps. Am I mistaken in my understanding of DTrace?
--nwf;
ron minnich
You are right. I don't think runtime is unbounded. At the same time,
I'm still trying to locate example scripts to get an idea of how
complex it is. I'm talking to people at sun to get a handle on the
question.
ron
Roman Shaposhnik
No, you are not.
Thanks,
Roman.
Roman Shaposhnik
What exactly do you want to know? I worked with DTrace quite extensively.
Thanks,
Roman.
erik quanstrom
> > runtime for each D hook should be strictly bounded by the number of
> > instructions lobbed in, since D does not (without root override, perhaps?)
> > support backwards jumps. Am I mistaken in my understanding of DTrace?
>
> You are right. I don't think runtime is unbounded. At the same time,
> I'm still trying to locate example scripts to get an idea of how
> complex it is. I'm talking to people at sun to get a handle on the
> question.
before we go all crazy, has anyone else tried ron's
great tracing?
- erik
Devon H. O'Dell
>> > I keep hearing this brought up, but (while I am not an expert) AFAICT, the
>> > runtime for each D hook should be strictly bounded by the number of
>> > instructions lobbed in, since D does not (without root override, perhaps?)
>> > support backwards jumps. Am I mistaken in my understanding of DTrace?
>>
>> You are right. I don't think runtime is unbounded. At the same time,
>> I'm still trying to locate example scripts to get an idea of how
>> complex it is. I'm talking to people at sun to get a handle on the
>> question.
D doesn't do loops or conditionals. You can't branch. But you can get
around that with some tricks, like aggregations and predicates. For
what you'd typically use it for, it should be ok. That said, my
knowledge of DTrace internals greatly surpasses my knowledge of how to
actually use it, so I'll let Roman help out on that front :)
> before we go all crazy, has anyone else tried ron's
> great tracing?
It's good, I read the paper and played around with it while I was
doing some 9vx work. But it's not really the same. Devtrace implements
a subset of DTrace's functionality.
DTrace does more than syscalls, and a lot of the talk on this thread
seems to be focused around that, so please let me clarify a bit. This
is partially due to compiling binaries with CTF (Compact C Type Format
-- when I asked the guys about this, they laughed and said it was so
compact that they cut off a C from the acronym). CTF stores type /
name / size information about symbols in a program, but is not as
heavy as DWARF: no line information is stored, no file references,
etc. The data is stored in an extended ELF header, compressed with
zlib.
DTrace by itself is pretty lame. It needs providers to be interesting.
By itself, it's a very limited interpreter that supports BEGIN and
END, and a couple other tiny things (ERROR, maybe?). Devtrace would be
analogous to the syscall provider for DTrace, from my understanding.
Because of CTF, DTrace also has providers to do function boundary
tracing (function entries / exits) on any executable (including a live
kernel) that has been compiled with CTF symbols. The fasttrap (or PID)
provider allows for instruction level tracing by inserting a
breakpoint at a given location, catching it, executing what was
previously at the breakpoint, executing the relevant D code (which may
in turn be traced by DTrace up to a finite amount of recursion), until
it comes back up the stack. The USDT providers allow userland
applications -- languages such as Perl, PHP, or Python for instance --
to register DTrace hooks. Once a userland app has done this, you can
follow e.g. a Python function call from the script, through Python,
through libc, through a syscall, all the way to a device driver... and
back up again.
It's also fairly easy to do this. I was at a party with the Sun guys
at OSCon in 2005. Wez Furlong wrote a DTrace provider for PHP in about
30 minutes with Bryan Cantrill, who then rather excitedly ran to me
explaining how I needed to come look (this is a gross understatement
for brevity). It was pretty cool, and the code needed for the DTrace
hooking was smaller than the PHP module skeleton code.
While extending it may be done relatively easy, it is quite big. When
I was working on porting it to FreeBSD (before jb@ essentially usurped
the project, anyway), I spent 3 weeks getting CTF and the base DTrace
provider ported over. Granted, I was much less experienced than now,
but it is a significant amount of work to reproduce all the
functionality. I'd estimate a finished version to be about the size of
the Plan 9 kernel.
--me
erik quanstrom
> By itself, it's a very limited interpreter that supports BEGIN and
> END, and a couple other tiny things (ERROR, maybe?). Devtrace would be
> analogous to the syscall provider for DTrace, from my understanding.
i may be misunderstanding you, but devtrace can trace any
call site or return in the kernel. not just system calls. and
not just entire functions; one could enable tracing on just
one return, for example.
you just give it a range of addresses and it loops through finding
the magic left by the linker. enabling a trace atomicly modifies
each site to turn on tracing.
> I'd estimate a finished version to be about the size of
> the Plan 9 kernel.
minooka; wc -l /sys/src/9/*/*trace*.[chs] /sys/src/libc/386/trace.s
118 /sys/src/9/pc/archtrace.c
18 /sys/src/9/pc/archtrace.h
578 /sys/src/9/port/devtrace.c
33 /sys/src/libc/386/trace.s
747 total
this stuff's ready to take off. (har har har.)
- erik
Devon H. O'Dell
>> DTrace by itself is pretty lame. It needs providers to be interesting.
>> By itself, it's a very limited interpreter that supports BEGIN and
>> END, and a couple other tiny things (ERROR, maybe?). Devtrace would be
>> analogous to the syscall provider for DTrace, from my understanding.
>
> i may be misunderstanding you, but devtrace can trace any
> call site or return in the kernel. not just system calls. and
> not just entire functions; one could enable tracing on just
> one return, for example.
>
> you just give it a range of addresses and it loops through finding
> the magic left by the linker. enabling a trace atomicly modifies
> each site to turn on tracing.
Ok, I didn't realize the scope.
>> I'd estimate a finished version to be about the size of
>> the Plan 9 kernel.
>
> minooka; wc -l /sys/src/9/*/*trace*.[chs] /sys/src/libc/386/trace.s
> 118 /sys/src/9/pc/archtrace.c
> 18 /sys/src/9/pc/archtrace.h
> 578 /sys/src/9/port/devtrace.c
> 33 /sys/src/libc/386/trace.s
> 747 total
>
> this stuff's ready to take off. (har har har.)
Sure, but it's still not the entire functionality :)
--dho
> - erik
>
>
Roman Shaposhnik
> 2009/11/9 erik quanstrom <quan...@coraid.com>:
> That said, my knowledge of DTrace internals greatly surpasses my knowledge of how to
> actually use it, so I'll let Roman help out on that front :)
Which means -- we can nicely compliment each other ;-)
>> before we go all crazy, has anyone else tried ron's
>> great tracing?
>
> It's good, I read the paper and played around with it while I was
> doing some 9vx work. But it's not really the same. Devtrace implements
> a subset of DTrace's functionality.
Well said. I played with it in a very limited context and for what it
was built -- I felt
it was a nice tool. As a matter of fact, at the same time I suggested
to Ron that
a more DTrace'ish extensions of his work could be interesting. More on
that later.
> DTrace does more than syscalls, and a lot of the talk on this thread
> seems to be focused around that, so please let me clarify a bit.
To me, the greatest value of DTrace comes from the fact that it is a
*unified* tracing framework that lets you trace events throughout multiple
layers of software. You can start tracing at the level of hardware interrupt,
pick the event up in Apache, continue into Tomcat and all the way into
middleware.
Yes, I know, on a cleanly designed system -- none of this nonsense would
be needed anyway, but when it is "turtles all the way down" DTrace can let
you see how stuff percolates quite neatly. All the way up to...
> It's also fairly easy to do this. I was at a party with the Sun guys
> at OSCon in 2005. Wez Furlong wrote a DTrace provider for PHP in about
> 30 minutes with Bryan Cantrill, who then rather excitedly ran to me
> explaining how I needed to come look (this is a gross understatement
> for brevity). It was pretty cool, and the code needed for the DTrace
> hooking was smaller than the PHP module skeleton code.
...your own application. Being able to expose the states of your app in
an efficient and unified manner is quite helpful.
> While extending it may be done relatively easy, it is quite big. When
> I was working on porting it to FreeBSD (before jb@ essentially usurped
> the project, anyway), I spent 3 weeks getting CTF and the base DTrace
> provider ported over. Granted, I was much less experienced than now,
> but it is a significant amount of work to reproduce all the
> functionality. I'd estimate a finished version to be about the size of
> the Plan 9 kernel.
Right. I guess the trick is to see how close are we to a point of diminishing
returns. Ron's stuff covers kernel pretty neatly and for everything else -- its
easy enough to spill the guts via 9P.
Thanks,
Roman.
dav...@mac.com
> What exactly do you want to know? I worked with DTrace quite
> extensively.
What is the upper bound on the runtime of a single D bytecode sequence?
Or to put it another way, what's the longest time delay that DTrace&co
can cause in your kernel?
D
Roman Shaposhnik
Would this answer your question:
http://blogs.sun.com/jonh/entry/the_dtrace_deadman_mechanism
Or are you literally trying to figure out the upper bound on the # of
virtual instructions
in a single probe?
Thanks,
Roman.
dav...@mac.com
> http://blogs.sun.com/jonh/entry/the_dtrace_deadman_mechanism
Well, it answers the question "What is the DTrace so-called deadman
mechanism?" I think.
That's a sort of part of a possible solution, which is OK.
To be pedantic, it's not a true deadman mechanism,
which operates at the point of failure (incapacity of the driver), not
when it's all over (death of the passengers).
What's described there is more like a pair of buffers at the end of
the crowded terminus station.
> Or are you literally trying to figure out the upper bound on the # of
> virtual instructions
> in a single probe?
Well, more importantly, the consequent real-time involved.
IOW: assuming you have some common sense (and a degree of chutzpah,
given the complexity of the solaris kernel),
and thus you have a reasonable idea about how long a time "too long"
is to be holding up your kernel,
how do you know whether a given D bytecode blob is gonna execute for
"too long" or not?
D