CrashGenerationServer on linux

[willw]

Hello,

I'd like to get oop crash generation working on Linux. I understand
that I should use CrashGenerationServer::CreateReportChannel to get
the fd's and then the server should start the application I want to
monitor, passing it the fd.

I have a few questions:
- Is it expected that CrashGenerationServer is not in any library and
you have to compile the file within your server application?
- Does the client have to be a child process of the server? If so what
permissions etc are needed to make it work
- Is this working for anyone outside google? Comments on the web
suggest that it is heavily tailored to what chrome needs and may not
yet be general purpose.

Thanks
Will

[Ted Mielczarek]

On Wed, Mar 16, 2011 at 11:57 AM, willw <will...@googlemail.com> wrote:
> Hello,
>
> I'd like to get oop crash generation working on Linux. I understand
> that I should use CrashGenerationServer::CreateReportChannel to get
> the fd's and then the server should start the application I want to
> monitor, passing it the fd.

This is essentially correct. You create the file descriptors using
that API, then there's a constructor for ExceptionHandler that accepts
a fd to communicate to the server.

> I have a few questions:
> - Is it expected that CrashGenerationServer is not in any library and
> you have to compile the file within your server application?

I hadn't noticed that. We could fix it to build as part of
libbreakpad_client.a, that wouldn't be hard.

> - Does the client have to be a child process of the server? If so what
> permissions etc are needed to make it work

Not really, but for practical reasons, yes. The only requirement is
that you pass the open file descriptor to the other process so it can
report back on it. There may be other ways of accomplishing this, but
I haven't looked into it. I have been investigating changing the way
Firefox's main process exception handling works, so that it would
spawn a helper process when crashes happen and then write the dump
from the other process, but I had to work around a few issues. You can
see my sample application here (working on Linux and OS X at the
moment):
http://hg.mozilla.org/users/tmielczarek_mozilla.com/breakpad-oop-test

That code relies on a couple of Breakpad patches that I haven't
bothered landing yet:
http://hg.mozilla.org/users/tmielczarek_mozilla.com/breakpad-mq/file/dfd8d050f0a3/no-close-on-exec
http://hg.mozilla.org/users/tmielczarek_mozilla.com/breakpad-mq/file/dfd8d050f0a3/crash-generation-server-port

> - Is this working for anyone outside google? Comments on the web
> suggest that it is heavily tailored to what chrome needs and may not
> yet be general purpose.

I think that Firefox may be the only consumer, actually. We're using
it for crash reporting for plugin processes (and content processes in
Firefox mobile). It's working fine for that purpose, but we do build
everything with our own build system, and we have lots of support
code. The code in the Breakpad repository was refactored from some
code in the Chromium repository, but I don't know that Chromium was
ever changed to use these classes.

-Ted

[willw]

Thanks for the help Ted.

On Mar 16, 4:16 pm, Ted Mielczarek <t...@mielczarek.org> wrote:

> On Wed, Mar 16, 2011 at 11:57 AM, willw <willwm...@googlemail.com> wrote:
> > Hello,
>
> > I'd like to get oop crash generation working on Linux. I understand
> > that I should use CrashGenerationServer::CreateReportChannel to get
> > the fd's and then the server should start the application I want to
> > monitor, passing it the fd.
>
> This is essentially correct. You create the file descriptors using
> that API, then there's a constructor for ExceptionHandler that accepts
> a fd to communicate to the server.
>
> > I have a few questions:
> > - Is it expected that CrashGenerationServer is not in any library and
> > you have to compile the file within your server application?
>
> I hadn't noticed that. We could fix it to build as part of
> libbreakpad_client.a, that wouldn't be hard.

If I find the time I'll submit a patch.

>
> > - Does the client have to be a child process of the server? If so what
> > permissions etc are needed to make it work
>
> Not really, but for practical reasons, yes. The only requirement is
> that you pass the open file descriptor to the other process so it can
> report back on it. There may be other ways of accomplishing this, but
> I haven't looked into it. I have been investigating changing the way
> Firefox's main process exception handling works, so that it would
> spawn a helper process when crashes happen and then write the dump
> from the other process, but I had to work around a few issues. You can
> see my sample application here (working on Linux and OS X at the
> moment):http://hg.mozilla.org/users/tmielczarek_mozilla.com/breakpad-oop-test

Ok, your example code is great. Your creating the socketpair in the
client(crasher) and then starting the server(handler) when it crashes
whereas I was thinking about always starting the server and then it
starting the client. Hadn't though of doing it that way round as you
can't do it on windows like that (unless I'm missing a trick). I guess
if I'm happy for it not to be symmetrical on both platforms then your
way round is much neater.

Thanks for the pointers
Will

[Ted Mielczarek]

On Wed, Mar 16, 2011 at 12:41 PM, willw <will...@googlemail.com> wrote:
> Ok, your example code is great. Your creating the socketpair in the
> client(crasher) and then starting the server(handler) when it crashes
> whereas I was thinking about always starting the server and then it
> starting the client. Hadn't though of doing it that way round as you
> can't do it on windows like that (unless I'm missing a trick). I guess
> if I'm happy for it not to be symmetrical on both platforms then your
> way round is much neater.

I plan on making this sample work on Windows when I have the time,
FWIW, even if it requires patching Breakpad (as I did for Linux and
Mac). That being said, I have no idea how much work it will be to
accomplish that at the moment.

If you can start the server first, then the code is much simpler. The
code was originally written to support Chrome, which is handling
crashes in a "main browser" process from its "renderer" processes. My
sample code flips the paradigm around, which makes it a bit more
involved.

-Ted

[Daniel Jones]

Hey,

I know that this discussion has been quiet for a while, but I thought I should just mention that I have had an issue with implementing OOP for Linux and the code in no-close-on-exec was part of the fix that I had found.

If I don't comment out those lines, my application just hangs when there is a error.

The other part of the problem was to do with the ExceptionHandler constructor.  It states in the header that you should pass the server_fd as the last parameter, but I have found that it will only work if I pass the client_fd to it (using CreateReportChannel).  I should probably mention that I and creating the server from the client process at the start program so I don't know if that could be the reason that the server and client are reversed.

Daniel

[Mark Final]

Hi,

I, too, found that I needed the no-close-on-exec patch to make OOP work on Linux - or the process with the CrashGenerationServer never received any messages. In addition to this, I added this utility function to my own code, to validate any file descriptors passed across processes:

  bool isValidFd(const int fd)
  {
    if (0 == fd) {
      // http://en.wikipedia.org/wiki/File_descriptor
      // we won't be passing stdin as a file descriptor
      return false;
    }
    const bool valid = (fcntl(fd, F_GETFL) != -1) || (errno != EBADF);
    return valid;
  }

With regard to your question on the ExceptionHandler, my answer may not be valid for all as I've been informed I am using Breakpad in a different way to others (ExceptionHandler in the parent process (my app) and the CrashGenerationServer as a child process) - there have been a few posts about this recently on this forum. That sounds like your description though.
However, the code in my app looks like this:

int StartCrashReportingProcess(const CrashHandler::Parameters &params)
{
  int client_fd = -1;
  int server_fd = -1;
  if (!google_breakpad::CrashGenerationServer::CreateReportChannel(&server_fd, &client_fd)) {
    return -1;
  }
  <snip>
  fork child
  <snip>
  return client_fd;
}

<snip>

CrashHandler::CrashHandler(const Parameters &params)
{
  int client_fd = -1;
  if (CrashHandler::eOutOfProcess == params._mode) {
    client_fd = StartCrashReportingProcess(params);
  }

  // create the exception handler
  const std::string dump_path = params._minidumpDir;
  google_breakpad::ExceptionHandler::FilterCallback filter = BreakpadFilter;
  google_breakpad::ExceptionHandler::MinidumpCallback callback = BreakpadMinidumpCallback;
  void *callback_context = 0;
  const bool installMinidumpHandler = true;

  google_breakpad::MinidumpDescriptor descriptor(dump_path);
  google_breakpad::ExceptionHandler *breakpadExceptionHandler =
    new google_breakpad::ExceptionHandler(descriptor,
                                          filter,
                                          callback,
                                          callback_context,
                                          installMinidumpHandler,
                                          client_fd);
  <snip>

So, I too, pass the client file descriptor to the ExceptionHandler.
It could be reversed to the documentation because of the way it's being used here.

HTH,

Mark

[Daniel Jones]

Thanks for that Mark,

It's nice to have a bit of a sanity check with the code I had implemented :)

Daniel