State of the 'Bag
Mark Mentovai
==========
I wanted to describe some of what we plan to provide in Airbag, to let
everyone know where we're going until more formal documentation is
written. What we're doing can be broken up into three major groups:
- Client-side libraries that generate dumps and assist sending them to
a crash server. This code will live in src/client/<platform>. As I've
mentioned before, there are two client-side libraries:
- The crash handler, which is linked into the application that you want
to detect and report crashes in. The handler responds to a crash by
writing a dump. This library will also have an interface to generate
a snapshot dump in an application upon direct call. Developers can
use this to generate dumps when certain non-crash unexpected conditions
occur. We'll obviously need divergent implementations for each
supported OS and CPU.
- The "sender," (for lack of a better term), which picks up dumps and
sends them to a crash reporting server somewhere. This part can be
linked into your main application, or can be some simple external
application (which is good if the main application is complex). Some
of this code may be shareable between platforms, but there's still
a need for some platform-specific code here.
- A server-side processor library that reads dumps and turns them into
useful debugging information. This component includes, among other
things, a dump reader, a stackwalker, a symbol translator, and a simple
symbol supplier (which is what we're calling the symbol server
interface). This code lives in src/processor. Most of the processor
code is generic and platform-agnostic, but there are obviously cases
where we need CPU- or OS-specific implementations of certain things.
For example, we've got an abstract base Stackwalker, and a concrete
StackwalkerX86 that understands how frames are laid out on an x86.
- Build tools, which, where necessary, interpret native debugging
information and turn it into a format that the processor can understand.
These tools are justified by Airbag's goal of having a cross-platform
(or at least Linuxy/Unixy) server. We can't read native Microsoft
debugging information without running on Windows: pdb is not officially
documented and its structure changes between releases. The sanctioned
way to interface with pdb files is to use the DIA APIs, so we provide
a tool that runs on Windows and can be integrated with build systems
or run "out of process" to produce Airbag-friendly symbol files. This
code lives in src/tools/<platform>. Obviously, this stuff isn't
shareable between platforms.
PLATFORMS
=========
By the way, in case it wasn't clear, Airbag's goal is to be
open-source and cross-platform. We're interpreting cross-platform to
mean that it handles crashes from the platforms we consider important.
At least for now, we care about 32-bit applications on:
- Mac OS X/ppc
- Mac OS X/x86
- Windows/x86
- Linux/x86
The processor component is somewhat less "cross-platform," it does
have a few gcc-isms and Unix-isms and even CPU-isms. Right now, it
runs on Linux and the Mac, both x86 and ppc. Getting it to run on
Windows isn't a high priority at the moment, but it might happen. The
important part is that the processor be able to interpret dumps from
any supported client platform.
PROCESSOR DESIGN
================
Here are some more details on the processor, because that's where
development's been primarily focused. A lot of what I describe has
been written already, some is in the process of being written, and
some is still in the planning stage. Some of this might even wind up
changing. We're flexible.
Minidumps
---------
We plan to use minidump files to represent dumps on all platforms.
Minidump's a Microsoft format, but it's documented and quite
reasonable, and after working with it for a while, I'm confident that
we can adapt it to suit our needs even on non-MS operating systems and
non-x86 CPUs. (As some of you, especially those who know me as "the
Mac guy" might expect, I had to get over the Microsoft hurdle here
myself, but I've done it, and I urge would-be format detractors to do
the same.) The server-side processor library contains a programmatic
interface to harness the contents of a minidump file, handling all of
the structure-finding and byte-swapping. This is a public API, but
most users will not need to deal much with it directly.
Stackwalking
------------
The stackwalker API is of more interest to library users, although
this too will probably be one step removed from the API that most
users interact with. The stackwalker's primary task is to generate a
call backtrace for a thread, given the thread's memory and CPU
register state. Optionally, a list of code modules (from the
minidump) can be passed to the stackwalker, so that it will be able to
determine which code module contains the instruction for a specific
stack frame. If a module list is present, the stackwalker can also
make use of optional debugging symbols, which enables it to resolve
which function, source file, and line corresponds to a specific stack
frame. This debugging information can also be used to aid the
stackwalker in locating stack frame boundaries, which allows for the
production of reliable stack backtraces even when the relevant code
doesn't store frame pointers.
Abstractions
------------
All of the arguments that the stackwalker accepts describe generic
interfaces, so it's possible to find alternate uses for the
stackwalker. This has made it possible to write a cute little test
implementation that obtains thread memory and CPU context from the
current process, instead of a minidump file, and walks the current
thread's stack. An abstraction with a more likely use is the symbol
supplier interface. libairbag itself will initially only contain a
simple local filesystem-based symbol supplier, but users can take this
as a starting point and develop alternate systems, such as
network-accessible symbol servers, or combinations of systems. One
possibility is a symbol supplier that knows how to talk to Microsoft
symbol servers. We could do some interesting things there in terms of
interoperating with existing symbol servers, including MS' own servers
that offer symbolic information for many libraries that ship with the
system.
MOVING FORWARD
==============
So, where do you, the community member, come in?
We're mostly providing the core library stuff - the raw bytes and even
rawer bits. Pretty soon, within a couple of weeks, well have airbag
in a state where others can begin to build real tools and real systems
around it - first, only supporting win32/x86, but soon, other
platforms as well. Go on and tie it into databases, and start
designing your crash-handling UI. Start thinking about contributing
some of your own work so that others may benefit, and brainstorm new
cool ways to take advantage of libairbag. I've heard a few really
good suggestions over the past few weeks, let's get those out in the
open.
I mentioned in airbag-dev that we're in the midst of ironing out all
of the licensing-and-contribution details, but the licensing's nearly
resolved, and you don't need to contribute to the project yet to start
using the library. Of course, we'll (hopefully soon) get to the point
where we can take contributions.
Oh, one more thing you can do: make sure your own applications are
really crashy, so we can start getting some good test data!
Mark
Ludovic Hirlimann
> - The "sender," (for lack of a better term), which picks up dumps and
How about drum or Tam-Tam
[snip]