DTrace San Francisco un-conference
G Bulmer
I thought you might be interested in my notes abut the San Francisco
DTrace conference, so here they are.
The DTrace conference was both good and disappointing.
It was good because I learned some really interesting stuff which is
immediately relevant, and suggests some 'stretch goals'.
It was disappointing because I didn't hear anything about the stuff I
wanted (though it may not exist:-).
Specifically, they didn't talk about any new mechanisms for writing
providers, which I had hoped for, but others suggested some sneaky
approaches which I'll mention.
I did learn that there is already a facility, translators, which may
help handle (non-iterative) Erlang data in a tidy way, so it was
useful to go just for that.
Presentations/demo's:
DT_Logger collects, over a few days, system behaviour in such a way
that you can generate documentation about what the system is doing. It
is used to trace an application so that someone can see what it's
major paths are. Very handy for 'code archaeology'.
More interesting to me was the yet-to-be-launched tools from Team
DTrace which provide very deep monitoring/observation tools and a web-
based GUI. It looked extremely useful for observing production
systems, and drilling into behaviour, without knowing much about
DTrace. It is a built as a server with the GUI delivered through a web
browser. It demoed well.
Most fun/interesting were the Java DTrace guys. They are reworking the
Java DTrace provider to make it very low overhead.
They are also providing the means to write the equivalent of DTrace
USDT (static) probes, but for developers to use in their own Java
code. Developers can insert static Java DTrace probes into their Java,
just like USDT probes, and they will incur virtually no overhead until
they are enabled.
The approach is rather clever; they are dynamically generating machine-
code for DTrace probes, and wiring them into JITed code. Very cool.
This will be Open Sourced soon, so we can see how it all works.
Though he didn't present, I also spoke to the guy who is doing a
similar USDT-probe implementation for Ruby. Currently, he fires off
gcc, and dynamically links to the newly-generated DLL. Very neat, and
relatively simple to do if you always have a C compiler and dynamic
code loader.
Finally, in the Dynamic language camp is the guy who is putting DTrace
support into Adobe AIR. Sadly it is only a technology demonstrator,
but it looked very useful, with probes for all of the usual subjects
including garbage collection events, etc. It looked very good in his
demo.
Jarrod Jenson had some great 'war stories' about fixing Java program
performance. Things like notifyall are a disaster when a program has a
large number of threads because, potentially every thread
simultaneously gets released to run, and wants a CPU, even though only
one thread can (usually) win. One of his points is that there are some
very dangerous mechanisms in Java, but they are rarely, if ever,
talked about.
The guys who implemented the PostgerSQL provider, and used them, were
there. That was pretty interesting as they can correlate actual disk
reads with table access (which should be cached) so that they can see
which tables are 'hot' and try to do something about it. Jarrod Jenson
talked about simple techniques like spotting a tcp/ip request entering
MySQL, and seeing how much disk access results, which can indicate
cache misses.
A guy from VMWare was there talking about his VProbe technology, a Sun
guy working on Xen/Zen, and a Sun guy talking about Solris Zones
+Brands. This later technology is worth finding a bit more about
simply because it largely does a good job of running Linux
applications but with DTrace. The VMWare VProbe technology looked
superb; potentially a DTrace-like system for any OS running within
VMWare!
Brandon Gregg talked about his IP, UDP, TCP/IP providers, which look
amazingly useful to the Erlang-Dtrace world; we should be able to
DTrace socket-traffic to, from and between Erlang VM's. Maybe some
thought-out, well-placed static probes in the Erlang-VM would give us
great observation points which would also make correlation easier.
Andrew Garden talked about his attempts to build DTrace probes
directly into FPGAs as a tool to debug the FPGA IP! Very interesting.
Apple, FreeBSD and QNX talked about porting DTrace, which was
interesting, though much of the interest was about the licensing of
FreeBSD which constrained what they cold put into the kernel, and
which made DTrace very attractive.
Brian Cantrill demoed a (quick hack) of a distributed DTrace. Rather
than do something very complete, he did something relatively easy,
which would use the underlying DTrace infrastructure to do the 'heavy
lifting'. What he implemented was only distributed aggregation, on the
basis that it is very useful, and answers many of the needs without
getting into very hard problems. Nice thinking. He could run
dtrace(1M) on several boxes, with one the master, and the others the
slaves, and run a D script which did some aggregation, say count the
number of system calls, then, when the script terminates, the
distributed (per machine) aggregations are further aggregated,
providing the machine wide results. Amazingly he is not interested in
taking it further as he has other priorities, but he said if anyone is
interested, get in touch, and take it forward.
It occurred to me that we could use Erlang to integrate distributed
DTraces if we had an Erlang DTrace consumer, but we need to get more
stuff working first :-)
My talk and demo seemed to go well. I think this was partly because
most people knew little about Erlang, but they had heard about it, so
I satisfied a need. It got several people following up about Erlang
and what it could be used for; I think there will be a small spurt of
puchases for Joe's Erlang book. My talk also encourged the Java, AIR
and Ruby DTrace provider guys to spend time explaining stuff to me,
which was very good fun and very interesting. I may circle back later
in the year to think about 'proper' USDT-like probes in Erlang.
So, very good, exciting, yet fell short of what I had hoped for; how
hard am I to please?-)
GB