Using crashpad in place of Breakpad

[wst...@gmail.com]

My understanding is that Crashpad is intended as a drop-in replacelment for Breakpad. I've been using Breakpad for years. I have been unable use the BREAKPAD_IN_PROCESS trick for 10.10, when I intentially crahs my application I just see a hang and no crash reporting dialog. I'm excited to hear the mac version of Crashpad is finally "complete" and am trying to use it. I followed the instructions and installed depot_tools, gyp, fetched Crashpad and compiled it without trouble. My questions:

-With Breakpad after compiling I had:

Breakpad.framework

dump_syms

After compiling Crashpad I don't see any tool for dumping symbols or a framework. How does one dump symbols with Crashpad? Is there a framework planned or should I copy something like 

Release/crashpad_handler 

into my applications Resources folder within my .app bundle?

-With Breakpad I started up the crash handler like so:

BreakpadRef breakpad = NULL;

void StartupCrashHandler(){

  if(breakpad != NULL)
    return;

  NSAutoreleasePool *pool  = [[NSAutoreleasePool alloc] init];
  NSDictionary      *plist = [[NSBundle mainBundle] infoDictionary];

  if(plist != NULL)
    breakpad = BreakpadCreate(plist);

  [pool release];
}

How does one do this with Crashpad? I can't find any documentation? 

I apologize if I'm asking these questions on the wrong list. If there is crashpad-users type list where I should post such questions please let me know.

[Mark Mentovai]

Will-

There’s no crashpad-users list yet, and crashpad-dev is appropriate for your question.

There’s also no formal integration documentation yet, but I’m working on this along with a sample project that showcases Crashpad integration.

We do have a full set of generated interface documentation linked from the Crashpad project site. Crashpad::CrashpadHandler is the primary interface for a client to enable reporting; Crashpad::CrashpadInfo allows you to control some behaviors; and the Settings hanging off of Crashpad::CrashReportDatabase are where you control uploads being enabled or disabled. In the source code, all of the things of interest to a client embedding Crashpad are in the client directory. Things are well-documented enough that you should be able to find your way easily.

The gist of integration is that your project will need to link against the libcrashpad_client library and package the crashpad_handler executable I recommend placing the executable in your bundle’s Helpers directory. The library has library dependencies that you will also need to link against, such as libcrashpad_util and base. You will also need to set your #include path correctly to be able to use the library: at a minimum, you’ll need to specify the crashpad project root directory as well as its crashpad/third_party/mini_chromium/mini_chromium subdirectory.

Until the integration sample is checked in, the best example of integration is Chromium, where Crashpad was integrated at d413b2dcb54d. Everything that talks to libcrashpad_client is isolated to the components/crash/app/crashpad_mac.mm file, so you can see how things are brought together in there.

At the moment, Crashpad only contains a usable client component for Mac OS X. If you’re integrating with a Breakpad-type crash reporting system based on the Breakpad processor and tools, you’ll still need to use Breakpad’s dump_syms build-time tool to produce symbol files that your processor will understand. In time, Crashpad will have its own processor component, but for the time being, its client component is intended to be a replacement for Breakpad’s equivalent in an existing Breakpad-type system. With the new Crashpad client, you’ll be able to replace Breakpad.framework, crash_inspector, crash_report_sender.app, and any of the other Breakpad client-side libraries and executables. crashpad_handler includes the exception handler and minidump writer (the equivalent of Breakpad’s crash_inspector) as well as the uploader (the equivalent of Breakpad’s crash_report_sender.app).

Good luck, and let me (and the list) know if you have any other questions. I’ll announce when the integration documentation and sample project are available.

Mark

--
You received this message because you are subscribed to the Google Groups "Crashpad-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crashpad-dev...@chromium.org.
To post to this group, send email to crashp...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/6d85f8b4-dd00-4d78-9c8c-7ded12f5dca8%40chromium.org.

[wst...@gmail.com]

Thanks Mark. That was all very helpful. I've begun the process of transitioning my application to Crashpad. I've updated my include path and verified I can allocate a  crashpad::CrashpadClient object. Unfortunately I'm running into trouble linking with libcrashpad_client.a. It turned out that .a file depended on a few others, so I'm now linking against the following:

libcrashpad_client.a
libcrashpad_util.a
libbase.a

So far so good, except now I'm running into what I suspect is the last linker error:

Undefined symbols for architecture x86_64:

"_audit_token_to_pid", referenced from:

crashpad::AuditPIDFromMachMessageTrailer(mach_msg_trailer_t const*) in libcrashpad_util.a(crashpad_util.mach_message.o)

Here are relevant portions of my attempt to link:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -lc++ -Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk -stdlib=libc++ -mmacosx-version-min=10.7 -Wl

In other words I'm using the 10.10 SDK but setting the minimum OS X version I'll support as 10.7.

Some Googling for audit_token_to_pid turned up:

https://github.com/nneonneo/osx-10.9-opensource/blob/master/configd-596.12/configd.tproj/session.c

This page suggests audit_token_to_pid is only available in 10.8+ and that audit_token_to_au32 should be used for 10.7 and prior. This all made me realize I have not specified the minimum OS X version I need to support when compiling Crashpad. Note my development environment is 10.10.2 using XCode 6.1.1. I'm a bit confused since when I checked out the latest Chromium nightly that is including Crashpad indicates via it's Info.plist file that the minimum OS X version support is 10.6. Thoughts?

Another question I had is the various out directories. What are the Release_x64 and Debug_x64 directories for? I compiled all four and using lipo determined that all four were 64 bit.

-Will

[Jiang Jiang]

What matters here is how you build crashpad, not how you link it. If
you take a look at:

https://code.google.com/p/chromium/codesearch#chromium/src/third_party/crashpad/crashpad/util/mach/mach_message.cc&sq=package:chromium&type=cs&l=234

When building crashpad it needs the correct Mac deployment target
(macosx-version-min). For instance you can use:

build/gyp_crashpad.py -Dmac_deployment_target=10.7

Then you will get the correct libcrashpad_util.a binary:

$ nm out/Debug/libcrashpad_util.a | grep audit_token
U _audit_token_to_au32

- Jiang

[Mark Mentovai]

wst...@gmail.com wrote:

Undefined symbols for architecture x86_64:

"_audit_token_to_pid", referenced from:

crashpad::AuditPIDFromMachMessageTrailer(mach_msg_trailer_t const*) in libcrashpad_util.a(crashpad_util.mach_message.o)

Jiang’s answer is correct: you need to tell your deployment target to the Crashpad build with the mac_deployment_target GYP variable.

You’ll also need to link against /usr/lib/libbsm.dylib, which provides the audit_token_to_*() functions.

Another question I had is the various out directories. What are the Release_x64 and Debug_x64 directories for? I compiled all four and using lipo determined that all four were 64 bit.

They’re unused. The Windows ninja build requires them for some reason. In our environment, they’ll behave exactly the same as Release and Debug, so if you configure for a 32-bit x86 build, the _x64 directories would also wind up with 32-bit output.

You should ignore the _x64 directories. Since the Mac builds don’t require them, I’ll see if I can keep them from appearing at all on non-Windows.

Mark

[wst...@gmail.com]

Thanks again for the replies. I feel like I"m inching closer to getting this all working. I've hit another snag and I was hoping one of you might see where I'm going wrong. I'm actually trying to call

crashpad_client.StartHandler

now but I'm getting the following linker error:

Undefined symbols for architecture x86_64:

"crashpad::CrashpadClient::StartHandler(base::FilePath const&, base::FilePath const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)", referenced from:

StartupCrashHandler() in crashHandler.o

ld: symbol(s) not found for architecture x86_64

This is very strange. If I comment out that call and still declare the crashpad_client variable I'll get a linker error only if I omit linking to the .a files. For reference this is the relevant portion of my linking line:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ 

-headerpad_max_install_names

-lc++ -Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk

-mmacosx-version-min=10.7

-Wl,-rpath

... lots of my .o files ...

-L/path/to/crashpad/crashpad/out/Release/ -lcrashpad_client

-lcrashpad_compat -lcrashpad_minidump -lcrashpad_snapshot

-lcrashpad_tool_support -lcrashpad_util -lgmock -lgmock_main -lbase

-lbsm

You'll notice I'm specifying the various crashpad .a files *after* I specify my .o files in order to pull in required symbols after they are used by my crashHandler.o. You'll also notice I tried pulling in every .a file I could imagine without any benifit. My impression is that I should only require

-lcrashpad_client

-lcrashpad_util

-lbase

I specifically listed those in that order so that symbols from base used by client would be pulled when linking in base.

Any ideas? For reference I upgrade my Qt project to add C++11 support like so:

    QMAKE_CXXFLAGS += -std=c++11

    QMAKE_CXXFLAGS += -stdlib=libc++

    QMAKE_OBJECTIVE_CFLAGS += -std=c++11

    QMAKE_OBJECTIVE_CFLAGS += -stdlib=libc++

    QMAKE_LFLAGS += -lc++

 

[Mark Mentovai]

wst...@gmail.com wrote:

Thanks again for the replies. I feel like I"m inching closer to getting this all working. I've hit another snag and I was hoping one of you might see where I'm going wrong. I'm actually trying to call

crashpad_client.StartHandler

now but I'm getting the following linker error:

Undefined symbols for architecture x86_64:

"crashpad::CrashpadClient::StartHandler(base::FilePath const&, base::FilePath const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)", referenced from:

StartupCrashHandler() in crashHandler.o

ld: symbol(s) not found for architecture x86_64

This is very strange. If I comment out that call and still declare the crashpad_client variable I'll get a linker error only if I omit linking to the .a files. For reference this is the relevant portion of my linking line:

Does the symbol actually show up in the libcrashpad_client.a that you’re linking against? For example:

mark@mela zsh% nm -mp -s __TEXT __text out/Release/libcrashpad_client.a | grep -F StartHandler

warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm: no name list

0000000000000100 (__TEXT,__text) private external __ZN8crashpad14CrashpadClient12StartHandlerERKN4base8FilePathES4_RKNSt3__112basic_stringIcNS5_11char_traitsIcEENS5_9allocatorIcEEEERKNS5_3mapISB_SB_NS5_4lessISB_EENS9_INS5_4pairISC_SB_EEEEEERKNS5_6vectorISB_NS9_ISB_EEEE

(The “no name list” warning is harmless, it’s referring to the emptiness of client/simple_string_dictionary.cc.)

That compares to an .o file that references crashpad::CrashpadClient::StartHandler():

mark@mela zsh% nm -mp out/Release/obj/tools/mac/run_with_crashpad.run_with_crashpad.o | grep -F StartHandler

                 (undefined) external __ZN8crashpad14CrashpadClient12StartHandlerERKN4base8FilePathES4_RKNSt3__112basic_stringIcNS5_11char_traitsIcEENS5_9allocatorIcEEEERKNS5_3mapISB_SB_NS5_4lessISB_EENS9_INS5_4pairISC_SB_EEEEEERKNS5_6vectorISB_NS9_ISB_EEEE

The defined symbol and undefined reference both use exactly the same name.

Be sure you’re looking at the same libcrashpad_client that the linker is using when you poke at it with nm.

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ 

-headerpad_max_install_names

-lc++ -Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk

-mmacosx-version-min=10.7

-Wl,-rpath

... lots of my .o files ...

-L/path/to/crashpad/crashpad/out/Release/ -lcrashpad_client

-lcrashpad_compat -lcrashpad_minidump -lcrashpad_snapshot

-lcrashpad_tool_support -lcrashpad_util -lgmock -lgmock_main -lbase

-lbsm

You'll notice I'm specifying the various crashpad .a files *after* I specify my .o files in order to pull in required symbols after they are used by my crashHandler.o. You'll also notice I tried pulling in every .a file I could imagine without any benifit. My impression is that I should only require

-lcrashpad_client

-lcrashpad_util

-lbase

I think that’s correct too. You may also need -lcrashpad_compat.

You shouldn’t need the minidump, snapshot, or tool_support libraries. You definitely don’t need the gmock library. You positively don’t need the gmock_main library, and it’s actually a little dangerous to specify it because it defines a main() function.

I specifically listed those in that order so that symbols from base used by client would be pulled when linking in base. 

The Apple linker doesn’t actually require that libraries be listed in any particular order relative to one another.

Any ideas? For reference I upgrade my Qt project to add C++11 support like so:

    QMAKE_CXXFLAGS += -std=c++11

    QMAKE_CXXFLAGS += -stdlib=libc++

    QMAKE_OBJECTIVE_CFLAGS += -std=c++11

    QMAKE_OBJECTIVE_CFLAGS += -stdlib=libc++

    QMAKE_LFLAGS += -lc++

-lc++ is dangerous to specify. The C++ library should be chosen for you by the compiler driver when it runs the linker. If you added this because the linker reported undefined symbols that should have been provided by the C++ library, you should go back and make sure that you’re using the C++ compiler driver (c++ or clang++) to run the linker, and not the C compiler driver (cc or clang).

With -stdlib=libc++, clang++ will add -lc++ and anything else necessary for you. You shouldn’t normally need to specify -stdlib=libc++, because the compiler driver normally does a good job of choosing an appropriate standard library implementation based on your chosen deployment target. For a deployment target of 10.9 or later, -stdlib=libc++ is the default; earlier, -stdlib=libstdc++ is.

I think that you may still have a C++ standard library, SDK, or deployment target disagreement between your code and Crashpad. Things will only work correctly if you make sure that you build Crashpad with the same settings that you’re using for your own build. For example, if I do a Crashpad build with -Dmac_sdk=10.6 -Dmac_deployment_target=10.6 (as we do in Chrome), I wind up using libstdc++ instead of libc++ because the deployment target is under 10.9. If I repeat the nm experiment from above, I get:

mark@mela zsh% nm -mp -s __TEXT __text out106/out/Release/libcrashpad_client.a | grep -F StartHandler

warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm: no name list

0000000000000100 (__TEXT,__text) private external __ZN8crashpad14CrashpadClient12StartHandlerERKN4base8FilePathES4_RKSsRKSt3mapISsSsSt4lessISsESaISt4pairIS5_SsEEERKSt6vectorISsSaISsEE

which is subtly different than what happens when using libc++.

Crashpad will build with the default C++ library chosen by the compiler driver. If you’ve build Crashpad with -Dmac_deployment_target=10.7 as Jiang suggested, it’ll be using libstdc++ and not libc++. You’ll need to make both sides agree.

[Will Stokes]

OK that was all *extremely* helpful. I compiled Crashpad for 10.7+ which resulted in it using the traditional libstdc++. I verified the symbol in question was in both the .a and .o files. It is the latter variant you posted. When I fixed my project to use libstd++ instead of the newer libc++ and removed the -lc++ linker flag things finally started to work. I'm successfully calling  StartHandler and if I give it a valid path to the helper instead of seeing a goofy crash (and my application still launching) I now see a slightly different message on the console complaining that I'm not providing a database (I passed an empty string for that param). What should I be passing in for this database parameter? I never passed anything like that to Breakpad in the past. For reference these are the settings Breakpad was getting in the past from my Info.plist file:

        <key>BreakpadReportInterval</key>

        <string>0</string>

        <key>BreakpadVendor</key>

        <string>GSL Biotech</string>

        <key>BreakpadRequestComments</key>

        <string>YES</string>

        <key>BreakpadRequestEmail</key>

        <string>YES</string>

        <key>BreakpadConfirmTimeout</key>

        <string>NO</string>

        <key>BreakpadMinidumpLocation</key>

        <string>~/Library/Caches/SnapGene/Crash Reports/</string>

        <key>BreakpadURL</key>

        <string>http://www.snapgene.com/reportCrash.php</string>

        <key>BreakpadPrivacyPolicyURL</key>

        <string>http://www.snapgene.com/privacy.html</string>

I suppose many of these could go into "annotations". In the past I did not set BreapadProduct or BreakpadVersion since if these were not set Breakpad would fetch the values using CFBundleName and CFBundleVersion. It appears the new Crashpad code is more platform agnostic in that it never deals with an Info.plist file. It's up to me to pull those values out and push them into an annotations object. I suppose there is no reason any of those settings technically need to be in an Info.plist file now, they could all be hard coded if I really wanted to in my own code.

Anyhow, I fetched CFBundleName, CFBundleVersion, BreakpadURL, and BreakpadMinidumpLocation from my Info.plist and tried starting it up as follows:

  QString product  = convertNSStringToQString( [plist valueForKey:@"CFBundleName"] );

  QString version  = convertNSStringToQString( [plist valueForKey:@"CFBundleVersion"] );

  QString url      = convertNSStringToQString( [plist valueForKey:@"BreakpadURL"] );

  QString database = convertNSStringToQString( [plist valueForKey:@"BreakpadMinidumpLocation"] );

  std::map<std::string, std::string> annotations;
  annotations["prod"] = product.toStdString();
  annotations["ver"]  = version.toStdString();
  annotations["plat"] = std::string("OS X");

    if (crashpad_client.StartHandler(base::FilePath(handler.toStdString()),
                                     base::FilePath(database.toStdString()),
                                     url.toStdString(),
                                     annotations,
                                     std::vector<std::string>()))
    {
          crashpad_client.UseHandler();
    }

I'm not sure if I need to include the "plat" annotation, or if I should include any other annotations that I am not currently. RIght now I'm just studying that Chromium code you linked to. When I threw in some print statements I can see StartHandler is returning true and I end up calling UseHandler. In standard output I'm seeing the following printed out:

product= "SnapGene"
version= "2.8.0a"
url= "http://www.snapgene.com/reportCrash.php"
database= "~/Library/Caches/SnapGene/Crash Reports/"

[55643:588685:20150318,220221.487709:ERROR crash_report_database_mac.mm:86] mkdir: No such file or directory (2)

My impression from reading the header files is that if no database exists it will be created. How you tell the database to uploader dumps it beyond me. The ERROR being printed out above worries me. I went out my way to actually create that directory and the error went away. Perhaps Breakpad was more flexible and could create the database directory if it was not present. Anyhow, I can easily do that and now do.

 When I went ahead and crashed my application it just up and quit. I saw no crash report dialog, neither one from Apple or the old one provided by Breakpad's framework. I suspect with Crashpad I'll have to roll my own reporting dialog somehow like I did on Windows. Is that correct? If you could point me in the direction to how I begin to tell crashpad what dialog to pop up that would be great.

For reference when I triggered the crash this is what I see dumped to the console:


55885:609153:20150318,222526.406966:WARNING mach_o_image_segment_reader.cc:118] section.segname incorrect in segment __TEXT, section __LD,__compact_unwind 2/5, load command 0x19 0/6, module cl_kernels, address 0x109c95000

[55885:609153:20150318,222526.407577:WARNING mach_o_image_segment_reader.cc:118] section.segname incorrect in segment __TEXT, section __LD,__compact_unwind 3/6, load command 0x19 0/6, module cl_kernels, address 0x109df2000

[55885:609153:20150318,222526.408499:WARNING mach_o_image_segment_reader.cc:118] section.segname incorrect in segment __TEXT, section __LD,__compact_unwind 2/5, load command 0x19 0/6, module cl_kernels, address 0x115cac000

[55885:609153:20150318,222526.408553:WARNING mach_o_image_segment_reader.cc:118] section.segname incorrect in segment __TEXT, section __LD,__compact_unwind 3/6, load command 0x19 0/6, module cl_kernels, address 0x115cb0000

Presumably once I hook up some other aspect of Crashpad such messages will no longer be dumped to the console although I'm not sure.

Another question I have now that I have code up and running is that in the past I made use of BreakpadAddUploadParameter to add a slew of additional params to be sent along side crash reports. For example beta testers of our software are required to give us their email and I can automatically attach it to crash reports in case they don't bother to enter it in the crash reporting dialog Breakpad would provide in the past. Since I don't necessarily have access to that information when I startup Crashpad (as early as possible during application startup), and honestly this seems less like a process-level annotation and more like a module-level annotation, I'm curious how I set module-level annotations that are referred to in your recent commit to handler/mac/crashpad_handler.ad

https://codereview.chromium.org/1009743004/diff/1/handler/mac/crashpad_handler.ad

Thanks again for all of your help. Once I get all this working I would be more than happy to provide a summary of what I did to get things all working.

Will Stokes

Chief Software Architect

[Mark Mentovai]

Will Stokes wrote:

OK that was all *extremely* helpful. I compiled Crashpad for 10.7+ which resulted in it using the traditional libstdc++. I verified the symbol in question was in both the .a and .o files. It is the latter variant you posted. When I fixed my project to use libstd++ instead of the newer libc++ and removed the -lc++ linker flag things finally started to work. I'm successfully calling  StartHandler and if I give it a valid path to the helper instead of seeing a goofy crash (and my application still launching) I now see a slightly different message on the console complaining that I'm not providing a database (I passed an empty string for that param). What should I be passing in for this database parameter?

Here’s the documentation for CrashpadClient::StartHandler and crashpad_handler. For the database parameter, you need to pass the path to where you want the crash report database to be stored. The handler parameter is required, and crashpad_handler requires that the database be set too. And without a URL, you obviously won’t get reports uploaded.

I never passed anything like that to Breakpad in the past. For reference these are the settings Breakpad was getting in the past from my Info.plist file:

        <key>BreakpadReportInterval</key>

        <string>0</string>

        <key>BreakpadVendor</key>

        <string>GSL Biotech</string>

        <key>BreakpadRequestComments</key>

        <string>YES</string>

        <key>BreakpadRequestEmail</key>

        <string>YES</string>

        <key>BreakpadConfirmTimeout</key>

        <string>NO</string>

        <key>BreakpadMinidumpLocation</key>

        <string>~/Library/Caches/SnapGene/Crash Reports/</string>

        <key>BreakpadURL</key>

        <string>http://www.snapgene.com/reportCrash.php</string>

        <key>BreakpadPrivacyPolicyURL</key>

        <string>http://www.snapgene.com/privacy.html</string>

BreakpadMinidumpLocation is like the database parameter you give to CrashpadClient::StartHandler. However, it’s got a ~ in it, and Crashpad won’t expand that for you, you’ll need to do it yourself.

I suppose many of these could go into "annotations". In the past I did not set BreapadProduct or BreakpadVersion since if these were not set Breakpad would fetch the values using CFBundleName and CFBundleVersion. It appears the new Crashpad code is more platform agnostic in that it never deals with an Info.plist file. It's up to me to pull those values out and push them into an annotations object. I suppose there is no reason any of those settings technically need to be in an Info.plist file now, they could all be hard coded if I really wanted to in my own code.

That’s right.

I’m weighing adding an Objective-C interface to make it easier for Mac projects to integrate Crashpad. That interface might have the option to read these things out of the main bundle’s Info.plist.

Anyhow, I fetched CFBundleName, CFBundleVersion, BreakpadURL, and BreakpadMinidumpLocation from my Info.plist and tried starting it up as follows:

  QString product  = convertNSStringToQString( [plist valueForKey:@"CFBundleName"] );

  QString version  = convertNSStringToQString( [plist valueForKey:@"CFBundleVersion"] );

  QString url      = convertNSStringToQString( [plist valueForKey:@"BreakpadURL"] );

  QString database = convertNSStringToQString( [plist valueForKey:@"BreakpadMinidumpLocation"] );

  std::map<std::string, std::string> annotations;
  annotations["prod"] = product.toStdString();
  annotations["ver"]  = version.toStdString();
  annotations["plat"] = std::string("OS X");

    if (crashpad_client.StartHandler(base::FilePath(handler.toStdString()),
                                     base::FilePath(database.toStdString()),
                                     url.toStdString(),
                                     annotations,
                                     std::vector<std::string>()))
    {
          crashpad_client.UseHandler();
    }

I’m not sure if you have another reason to use QString, but the easiest way to get a C++ std::string from an NSString* is by using -[NSString fileSystemRepresentation] if the string is a path, and -[NSString UTF8String] otherwise. Since you’ve got access to the base library here, you can also #include "base/mac/foundation_util.h" and use ObjCCast<NSString> to ensure that you’re really getting strings out of the plist, where -[NSDictionary valueForKey:] just returns id. So you could do:

  std::string product([base::mac::ObjCCast<NSString>(

      [plist valueForKey:@"CFBundleName"]) UTF8String]);

  base::FilePath database_path([base::mac::ObjCCast<NSString>(

      [plist valueForKey:@"BreakpadMinidumpLocation"]) fileSystemRepresentation]);

I'm not sure if I need to include the "plat" annotation, or if I should include any other annotations that I am not currently.

That’s up to your collection server. Yours would be totally different than mine, and it’s what dictates what’s required. Crashpad itself imposes no requirement, and is perfectly happy without any annotations at all.

My server doesn’t require “plat” either, and it’s redundant because we encode the platform in the product name and it’s also carried elsewhere in a dedicated field in the minidump. Chromium’s only including it because it’s been there all along, and I don’t know if anyone’s depending on it.

RIght now I'm just studying that Chromium code you linked to. When I threw in some print statements I can see StartHandler is returning true and I end up calling UseHandler. In standard output I'm seeing the following printed out:

product= "SnapGene"
version= "2.8.0a"
url= "http://www.snapgene.com/reportCrash.php"
database= "~/Library/Caches/SnapGene/Crash Reports/"

[55643:588685:20150318,220221.487709:ERROR crash_report_database_mac.mm:86] mkdir: No such file or directory (2)

That’s because of the ~ in the database path. Crashpad uses the path you give it as-is, without any transformations, so the ~ is taken as a literal ~ and not as “home directory.” If you’ve got a ~-path that you want expanded, you can get Foundation to do it for you with -[NSString stringByExpandingTildeInPath], like

  base::FilePath database_path([[base::mac::ObjCCast<NSString>(

      [plist valueForKey:@"BreakpadMinidumpLocation"])

          stringByExpandingTildeInPath] fileSystemRepresentation]);

My impression from reading the header files is that if no database exists it will be created. How you tell the database to uploader dumps it beyond me. The ERROR being printed out above worries me. I went out my way to actually create that directory and the error went away. Perhaps Breakpad was more flexible and could create the database directory if it was not present. Anyhow, I can easily do that and now do.

The database will be created as long as its parent directory already exists.

In order to upload dumps, crashpad_handler needs to have a good URL, and the database’s settings need to have uploads enabled. We recommend https for all URLs, although http will work. You can enable uploads with crashpad::Settings::SetUploadsEnabled(). Get the settings object from a database with crashpad::CrashReportDatabase::GetSettings().

  scoped_ptr<crashpad::CrashReportDatabase> database =

      crashpad::CrashReportDatabase::Initialize(database_path);

  database->GetSettings()->SetUploadsEnabled(true);

 

 When I went ahead and crashed my application it just up and quit. I saw no crash report dialog, neither one from Apple or the old one provided by Breakpad's framework. I suspect with Crashpad I'll have to roll my own reporting dialog somehow like I did on Windows. Is that correct? If you could point me in the direction to how I begin to tell crashpad what dialog to pop up that would be great.

By default, after Crashpad handles a crash, it will forward it to Apple’s ReportCrash and you will see its UI. You can suppress the forwarding if you don’t want it by calling crashpad::CrashpadInfo::set_system_crash_reporter_forwarding().

You should see a crash report show up in the Crashpad database and in the user’s ~/Library/Logs/DiagnosticReports (if forwarding is enabled). If you don’t see a report in either place, look in the root /Library/Logs/DiagnosticReports. If crashpad_handler isn’t running or can’t even make basic sense of the crash, the failure can result in the system delivering it to the root ReportCrash.

Crashpad doesn’t have any UI of its own, it operates silently in the background. If you want to add UI, you can look at modifying crashpad_handler. handler/mac/crash_report_exception_handler.cc has crashpad::CrashReportExceptionHandler::CatchMachException() which writes the crash dump, gives it to the upload thread, and then forwards the exception.

For reference when I triggered the crash this is what I see dumped to the console:


55885:609153:20150318,222526.406966:WARNING mach_o_image_segment_reader.cc:118] section.segname incorrect in segment __TEXT, section __LD,__compact_unwind 2/5, load command 0x19 0/6, module cl_kernels, address 0x109c95000

[55885:609153:20150318,222526.407577:WARNING mach_o_image_segment_reader.cc:118] section.segname incorrect in segment __TEXT, section __LD,__compact_unwind 3/6, load command 0x19 0/6, module cl_kernels, address 0x109df2000

[55885:609153:20150318,222526.408499:WARNING mach_o_image_segment_reader.cc:118] section.segname incorrect in segment __TEXT, section __LD,__compact_unwind 2/5, load command 0x19 0/6, module cl_kernels, address 0x115cac000

[55885:609153:20150318,222526.408553:WARNING mach_o_image_segment_reader.cc:118] section.segname incorrect in segment __TEXT, section __LD,__compact_unwind 3/6, load command 0x19 0/6, module cl_kernels, address 0x115cb0000

Presumably once I hook up some other aspect of Crashpad such messages will no longer be dumped to the console although I'm not sure.

cl_kernels is a weird module. It’s not produced by ld. Those are valid warnings about its structure in that the strings that show up as __TEXT and __LD are supposed to match. This shouldn’t impact Crashpad’s ability to handle crashes, it just won’t be aware of these modules at all.

Seeing these is a sign that you’re headed in the right direction. These are produced when crashpad_handler creates a snapshot.

crashpad_handler writes its log messages to stderr. When users launch normally by double-clicking the app, stderr doesn’t go anywhere visible (since 10.8).

Another question I have now that I have code up and running is that in the past I made use of BreakpadAddUploadParameter to add a slew of additional params to be sent along side crash reports. For example beta testers of our software are required to give us their email and I can automatically attach it to crash reports in case they don't bother to enter it in the crash reporting dialog Breakpad would provide in the past. Since I don't necessarily have access to that information when I startup Crashpad (as early as possible during application startup), and honestly this seems less like a process-level annotation and more like a module-level annotation, I'm curious how I set module-level annotations that are referred to in your recent commit to handler/mac/crashpad_handler.ad

https://codereview.chromium.org/1009743004/diff/1/handler/mac/crashpad_handler.ad

You set module-level annotations with crashpad::CrashpadInfo::set_simple_annotations. You can do something like this:

  // Use new, and don’t delete it while crashpad_info points to it.

  crashpad::SimpleStringDictionary* annotations = new crashpad::SimpleStringDictionary();

  annotations->SetKeyValue("email", "wi...@snapgene.com");

  crashpad::CrashpadInfo* crashpad_info = crashpad::GetCrashpadInfo();

  crashpad_info->set_simple_annotations(annotations);

  // If you hold on to annotations, you can also make changes after calling set_simple_annotations().

  annotations->SetKeyValue("used_new_ui", "yes");

crashpad::TSimpleStringDictionary<>::SetKeyValue() has more information on things you can do with a SimpleStringDictionary.

Thanks again for all of your help. Once I get all this working I would be more than happy to provide a summary of what I did to get things all working.

It sounds like you’re getting close!

[Will Stokes]

OK, now I'm definitely getting close. I followed your suggestions and have successfully started sending crash reports to my server and generated emails with summaries of the crashes. I've successfully attached other information (e.g. beta tester email addresses). That said there are a few issues that remain:

-I'm seeing the following warnings at compile time. Perhaps these can be fixed in the future.

crashpad/third_party/mini_chromium/mini_chromium/base/memory/scoped_ptr.h:102: warning: unused parameter 'other' [-Wunused-parameter]
  template <typename U> DefaultDeleter(const DefaultDeleter<U>& other) {

crashpad/third_party/mini_chromium/mini_chromium/base/logging.h:39: warning: unused parameter 'file' [-Wunused-parameter]
static inline int GetVlogLevel(const char* file) {
                                           ^
-I'm now using CRASHPAD_SIMULATE_CRASH in my application to test the handler without actually crashing. I actually have a cascading crash menu for testing triggering and catching various ways in which to crash the application. I noticed CRASHPAD_SIMULATE_CRASH is defined in client/simulate_crash_mac.h. Is it correct to assume this macro will be provided on Windows in the future as well? I'm aware that the Windows port is ongoing. I've updated and seen the Release_64 and Debug_64 directories are now gone which is great.

-Is it kosher to delete the database directory on a regular bases? I'm concerned that the completed set of crash reports will balloon over time. I suspect I don't technically need my database settings to survive sessions. At the moment jst before starting up the crash handler I'm deleting the contents of the new, pending and completed directories. Perhaps through the API there could/should be a way to clear the database?

-If I create a crashpad::SimpleStringDictionary* annotations and then call

crashpad_info->set_simple_annotations 

will crashpad take care of deallocating that for me? It's probably not a big deal to leak that memory when the application either

crashes or quits normally since the OS should in theory reclaim the memory anyhow but I figured I would ask.

-How do I shut down the handler when my application is quitting, or is that even necessary any more? In the past I called 

BreakpadRelease(breakpad);

-Currently I'm calling crashpad_info->set_simple_annotations *after* I call StartHandler and UseHandler. This seems to work fine. Is that when you would suggest calling that function? I do add annotations later using the technique you suggested.

-I've successfully triggered a crash and my simple annotations came through just fine. Unfortuantely the call stack isn't super human readable, e.g.:

Thread 0 (crashed)
 0  SnapGene!__ZN12CrashHandler12triggerCrashENS_10CRASH_TYPEE + 0x2f
    rax = 0x0000000000000000   rdx = 0x0000000000000000
    rcx = 0x0000000000000002   rbx = 0x00007fff5eefa288
    rsi = 0x0000000000000000   rdi = 0x00007fff5eefa288
    rbp = 0x00007fff5eefa350   rsp = 0x00007fff5eefa250
     r8 = 0x0000000000000000    r9 = 0x00006080000eb580
    r10 = 0x0000000000028248   r11 = 0x0000000102bd6440
    r12 = 0x0000000102ea6710   r13 = 0x00006080002db6d0
    r14 = 0x0000610000014c90   r15 = 0x00007fff5eefa368
    rip = 0x0000000101741f6f
    Found by: given as instruction pointer in context

 1  SnapGene!__ZN9DebugMenu16crashApplicationEv + 0x4f
    rbx = 0x0000608000201d20   rbp = 0x00007fff5eefa390
    rsp = 0x00007fff5eefa360   r12 = 0x0000000102ea6710
    r13 = 0x00006080002db6d0   r14 = 0x0000610000014c90
    r15 = 0x00007fff5eefa368   rip = 0x0000000100fc52cf
    Found by: call frame info

 2  SnapGene!__ZN9DebugMenu18qt_static_metacallEP7QObjectN11QMetaObject4CallEiPPv + 0x44e
    rbx = 0x000060800009be90   rbp = 0x00007fff5eefa3e0
    rsp = 0x00007fff5eefa3a0   r12 = 0x0000000102ea6710
    r13 = 0x00006080002db6d0   r14 = 0x00007fff7b97a070
    r15 = 0x000060800009be90   rip = 0x000000010174e2be
    Found by: call frame info

Which correlates with:

CrashHandler::triggerCrash(CRASH_TYPE type)
DebugMenu::crashApplication()
DebugMenu::qt_static_metacall(QObject....

In the past the :: and stuff was visible when viewing call stacks provided by Breakpad. I'm still dumping symbols with the old Breakpad dumper. Any idea why the call stack looks so ugly?

-I still need to throw together a UI to give the user a chance to write a description of what they were doing, provide their email address in case I don't have it, and some day it would be great if they could attach a file or files to the crash report (e.g. if one of our file importers is choking on a non-native file format we support opening). The Breakpad framework provided that out of the box. On Windows I accomplished this by specifying a minidump callback function when allocating the ExceptionHandler. Are you suggesting that adopters of Crashpad are suggested to compile their own version of crashpad_handler to provide this UI? I wonder if this could be simplified, or at least if an example of that could be included in the future example program.

[Robert Sesek]

Answers to some, but not all, of your questions.

On Thu, Mar 19, 2015 at 5:03 PM, Will Stokes <wi...@snapgene.com> wrote:

-I'm now using CRASHPAD_SIMULATE_CRASH in my application to test the handler without actually crashing. I actually have a cascading crash menu for testing triggering and catching various ways in which to crash the application. I noticed CRASHPAD_SIMULATE_CRASH is defined in client/simulate_crash_mac.h. Is it correct to assume this macro will be provided on Windows in the future as well? I'm aware that the Windows port is ongoing. I've updated and seen the Release_64 and Debug_64 directories are now gone which is great.

Yes, a similar macro will be provided for Windows in the future, but the port is not at that stage yet.

 

-Is it kosher to delete the database directory on a regular bases? I'm concerned that the completed set of crash reports will balloon over time. I suspect I don't technically need my database settings to survive sessions. At the moment jst before starting up the crash handler I'm deleting the contents of the new, pending and completed directories. Perhaps through the API there could/should be a way to clear the database?

It is fine to delete the database, but it will reset the client ID, which is what will uniquely identify crash reports for a given installation. We plan to add automatic, configurable pruning of old reports in the future. I would recommend not doing anything in your application to flush this directory, but rather wait for Crashpad to implement its pruning/clean up system. This is what we're doing in Chromium.

 

-If I create a crashpad::SimpleStringDictionary* annotations and then call

crashpad_info->set_simple_annotations 

will crashpad take care of deallocating that for me? It's probably not a big deal to leak that memory when the application either

crashes or quits normally since the OS should in theory reclaim the memory anyhow but I figured I would ask.

No, it won't clean it up. See https://codereview.chromium.org/1016613003/. Chromium leaks its module-level annotations, which is fine.

 

-How do I shut down the handler when my application is quitting, or is that even necessary any more? In the past I called 

BreakpadRelease(breakpad);

The handler exits after all the processes it is monitoring quit (assuming it has no other work to do, like uploading crash reports).

 

-Currently I'm calling crashpad_info->set_simple_annotations *after* I call StartHandler and UseHandler. This seems to work fine. Is that when you would suggest calling that function? I do add annotations later using the technique you suggested.

Yes, that is fine. Just be cognizant that any crash that occurs between UseHandler and set_simple_annotations will not have those annotations.

rsesek / @chromium.org

[Mark Mentovai]

Will Stokes wrote:

OK, now I'm definitely getting close. I followed your suggestions and have successfully started sending crash reports to my server and generated emails with summaries of the crashes. I've successfully attached other information (e.g. beta tester email addresses). That said there are a few issues that remain:

Glad to hear it!

(I’ll skip over things that Robert’s already answered.)

-I'm seeing the following warnings at compile time. Perhaps these can be fixed in the future.

crashpad/third_party/mini_chromium/mini_chromium/base/memory/scoped_ptr.h:102: warning: unused parameter 'other' [-Wunused-parameter]
  template <typename U> DefaultDeleter(const DefaultDeleter<U>& other) {

crashpad/third_party/mini_chromium/mini_chromium/base/logging.h:39: warning: unused parameter 'file' [-Wunused-parameter]
static inline int GetVlogLevel(const char* file) {

We don’t consider -Wunused-parameter a valuable warning and build our own code with it disabled, by using -Wno-unused-parameter. However, if this is the only instance of that warning that you’re seeing as a result of #including our files, I’d be willing to fix it for you.

-If I create a crashpad::SimpleStringDictionary* annotations and then call

crashpad_info->set_simple_annotations 

will crashpad take care of deallocating that for me? It's probably not a big deal to leak that memory when the application either

crashes or quits normally since the OS should in theory reclaim the memory anyhow but I figured I would ask.

The simple annotations need to be accessible as long as they’re pointed to by CrashpadInfo. It is correct to not delete it. You wouldn’t want to delete it anyway, because then it would be garbage if a crash happened after it was deleted. Sure, you could set_simple_annotations(nullptr), but then you’ve lost any annotations if the crash happens after it was cleared out.

All of the memory that belongs to a process gets cleaned up when the process exits. This is practice, not theory. :)

-How do I shut down the handler when my application is quitting, or is that even necessary any more? In the past I called 

BreakpadRelease(breakpad);

There is intentionally no API to do this. As with the annotations, you want the handler around to catch crashes that could happen even as the application is exiting.

The handler will exit on its own once it doesn’t have any processes to monitor for crashes.

-Currently I'm calling crashpad_info->set_simple_annotations *after* I call StartHandler and UseHandler. This seems to work fine. Is that when you would suggest calling that function? I do add annotations later using the technique you suggested.

I prefer to UseHandler() as early as possible, and then set_simple_annotations() after. If something’s wrong with set_simple_annotations() or anything that you need to do to get the annotation values, I’d prefer to get a crash report about it, even with an incomplete set of annotations. The best strategy is to get the handler set up as early as possible, ideally in main() or even before main().

Those are mangled (“decorated”) symbol names. You can convert them to human-readable names with c++filt. You can pipe the minidump_stackwalk output through c++filt, or pipe the dump_syms output through c++filt.

That said, I don’t know why this would have changed for you, if you were using the same dump_syms and same minidump_stackwalk as before. Crashpad has no influence over whether your processing machinery shows mangled or unmangled names. In a typical Breakpad pipeline, dump_syms calls abi::__cxa_demangle() and outputs demangled names all on its own.

-I still need to throw together a UI to give the user a chance to write a description of what they were doing, provide their email address in case I don't have it, and some day it would be great if they could attach a file or files to the crash report (e.g. if one of our file importers is choking on a non-native file format we support opening). The Breakpad framework provided that out of the box. On Windows I accomplished this by specifying a minidump callback function when allocating the ExceptionHandler. Are you suggesting that adopters of Crashpad are suggested to compile their own version of crashpad_handler to provide this UI? I wonder if this could be simplified, or at least if an example of that could be included in the future example program.

We don’t provide UI in Crashpad as-is because we prefer reporting to be silent and automatic when enabled. I realize that not all developers will be motivated by the same constraints that informed our decision.

That the Mac Breakpad framework even offered UI at all was kind of a goof. It was never in the design for that software.

With Crashpad’s design, once a program has crashed, there’s no longer any chance to run any code in it. The only thing that you can do is run code in the handler. This is really much more reliable, because there aren’t any guarantees about the state of a crashed process, and so there’s no assurance that it’s healthy enough to do complex tasks like run UI.

That leaves the handler as the only place to do UI. If you need this, I do suggest a custom crashpad_handler. Right now, the easiest way to do this would be to make local modifications to crashpad/handler/mac in the areas I’ve identified, but this will become easier with time: a lot of the logic in crashpad_handler will be moving out of the handler and into library functions. At that point, it’ll be much easier to roll your own handler. Until then, the least invasive way about it would be to take control in crashpad_handler right before it tries to write the minidump, and show some UI that might result in more annotations being added or might result in not writing or uploading the report at all.

…

You might also be happy to know that I took a look at the errors you mentioned seeing for cl_kernels, and have a fix for Crashpad out for review to suppress those. Thanks for all of the early feedback, it’s really helping us appreciate how this all looks from an outsider’s perspective, and what we need to focus on in our documentation and samples.

Mark

[Will Stokes]

-I'm seeing the following warnings at compile time. Perhaps these can be fixed in the future.

crashpad/third_party/mini_chromium/mini_chromium/base/memory/scoped_ptr.h:102: warning: unused parameter 'other' [-Wunused-parameter]
  template <typename U> DefaultDeleter(const DefaultDeleter<U>& other) {

crashpad/third_party/mini_chromium/mini_chromium/base/logging.h:39: warning: unused parameter 'file' [-Wunused-parameter]
static inline int GetVlogLevel(const char* file) {

We don’t consider -Wunused-parameter a valuable warning and build our own code with it disabled, by using -Wno-unused-parameter. However, if this is the only instance of that warning that you’re seeing as a result of #including our files, I’d be willing to fix it for you.

Those are the only two warnings I'm getting. If you could fix that that would be great. I personally like that warning since it alerts me to places in my code where I no longer need to be passing in such information and could avoid stack overhead. It happens as a project evolves over time.

The simple annotations need to be accessible as long as they’re pointed to by CrashpadInfo. It is correct to not delete it. You wouldn’t want to delete it anyway, because then it would be garbage if a crash happened after it was deleted. Sure, you could set_simple_annotations(nullptr), but then you’ve lost any annotations if the crash happens after it was cleared out.

All of the memory that belongs to a process gets cleaned up when the process exits. This is practice, not theory. :)

OK great, that all sounds perfectly logical.

 

-How do I shut down the handler when my application is quitting, or is that even necessary any more? In the past I called 

BreakpadRelease(breakpad);

There is intentionally no API to do this. As with the annotations, you want the handler around to catch crashes that could happen even as the application is exiting.

The handler will exit on its own once it doesn’t have any processes to monitor for crashes.

Perfect

 

-Currently I'm calling crashpad_info->set_simple_annotations *after* I call StartHandler and UseHandler. This seems to work fine. Is that when you would suggest calling that function? I do add annotations later using the technique you suggested.

I prefer to UseHandler() as early as possible, and then set_simple_annotations() after. If something’s wrong with set_simple_annotations() or anything that you need to do to get the annotation values, I’d prefer to get a crash report about it, even with an incomplete set of annotations. The best strategy is to get the handler set up as early as possible, ideally in main() or even before main().

I like that approach and it makes a lot of sense. It's possible I could trigger a crash while setting up the simple annotations and I would prefer to not miss such a crash. I'll take your approach. I'm doing all this fairly early on in main().

 

Those are mangled (“decorated”) symbol names. You can convert them to human-readable names with c++filt. You can pipe the minidump_stackwalk output through c++filt, or pipe the dump_syms output through c++filt.

That said, I don’t know why this would have changed for you, if you were using the same dump_syms and same minidump_stackwalk as before. Crashpad has no influence over whether your processing machinery shows mangled or unmangled names. In a typical Breakpad pipeline, dump_syms calls abi::__cxa_demangle() and outputs demangled names all on its own.

Interesting. I recently upgraded my build environment from 10.6 to 10.10 and stopped supporting 10.6. In order to support debuggin with the modern XCode and Instruments I switched from passing the compiler -gstabs to -g. I wonder if this may have caused the problem. All the same c++filt appears to do a great job. The one gotcha is that I used to get nice filenames and line numbers at the end and now I'm missing those. All I see now at he end is the funny offset or soemthing values, e.g. + 0x10 which mean nothing to me. E.g. in the past I might see:

45  SnapGene!BaseApplication::notify(QObject*, QEvent*) [baseApplication.cpp : 201 + 0x16]

where as now I"m seeing stuff like this:

 2  SnapGene!__ZN9DebugMenu18qt_static_metacallEP7QObjectN11QMetaObject4CallEiPPv + 0x44e

Which when I give the content beginning with the second _ all the way to the space to c++lint gives me back:

c++filt -n _ZN9DebugMenu18qt_static_metacallEP7QObjectN11QMetaObject4CallEiPPv

->

DebugMenu::qt_static_metacall(QObject*, QMetaObject::Call, int, void**)

So I have to be smart enough to pop off the "SnapGene!" as well as the first of the _'s, and any content after the first space, before passing it off to c++lint. Why I'm losing my filenames and line numbers of beyond me. I just verified I'm seeing a similar issue for my existing back builds that are still using Breakpad but were built with 10.10 so the issue is not Crashpad. Do you recommend any changes to my compiler flags in order to preserve the filenames and line numbers? I suspect by switching from STABS to DWARF for release builds I may have lost encoding the filename / line number information entirely. 

-I still need to throw together a UI to give the user a chance to write a description of what they were doing, provide their email address in case I don't have it, and some day it would be great if they could attach a file or files to the crash report (e.g. if one of our file importers is choking on a non-native file format we support opening). The Breakpad framework provided that out of the box. On Windows I accomplished this by specifying a minidump callback function when allocating the ExceptionHandler. Are you suggesting that adopters of Crashpad are suggested to compile their own version of crashpad_handler to provide this UI? I wonder if this could be simplified, or at least if an example of that could be included in the future example program.

We don’t provide UI in Crashpad as-is because we prefer reporting to be silent and automatic when enabled. I realize that not all developers will be motivated by the same constraints that informed our decision.

That the Mac Breakpad framework even offered UI at all was kind of a goof. It was never in the design for that software.

The Mac framework always felt kind of weird. I once I got over the hurdle of getting breakpad working on Windows I actually appreciated the flexibility I had there.

 

With Crashpad’s design, once a program has crashed, there’s no longer any chance to run any code in it. The only thing that you can do is run code in the handler. This is really much more reliable, because there aren’t any guarantees about the state of a crashed process, and so there’s no assurance that it’s healthy enough to do complex tasks like run UI.

Agreed 

That leaves the handler as the only place to do UI. If you need this, I do suggest a custom crashpad_handler. Right now, the easiest way to do this would be to make local modifications to crashpad/handler/mac in the areas I’ve identified, but this will become easier with time: a lot of the logic in crashpad_handler will be moving out of the handler and into library functions. At that point, it’ll be much easier to roll your own handler. Until then, the least invasive way about it would be to take control in crashpad_handler right before it tries to write the minidump, and show some UI that might result in more annotations being added or might result in not writing or uploading the report at all.

Interesting. Moving a much logical out of the handler into library code sounds great. Once that is done I wonder if the handler itself will be so trivial that it could be easily reproduced in a new program and then that program would just link to the .a files. That said, I'm not entirely sure how this will all work. I don't consider myself a Cocoa expert at all. I'm used to developing within the confines of Qt so that as much of our application is cross platform by default. With Breakpad you could set a callback function that would be run when the application crashed. This worked really well for me. I wrote a small Qt app that showed a dialog that apologized for the crash and let you enter a comment, your email, etc. I would run this reporter application from my call back function and pass it the filename of the mini dump that should be submitted. I wonder if a similar setup could be supported in crashpad on both Mac and Windows. If so then my custom GUI for reporting a crash, which based on Qt is already cross platform, could be used for both Mac and Windows and my crash handling experience would be identical on both OS's at long last. ;-)

 

You might also be happy to know that I took a look at the errors you mentioned seeing for cl_kernels, and have a fix for Crashpad out for review to suppress those.

Awesome, thanks for doing that. In my experience the cleaner and quieter you can keep the output and compiler output the more obvious real issues are when they arise.

 

Thanks for all of the early feedback, it’s really helping us appreciate how this all looks from an outsider’s perspective, and what we need to focus on in our documentation and samples.

Not a problem. My hope is to adopt Crashpad on Windows as well once it is ready so I would be happy to provide feedback there as well when the time is right. Thank you all for being so responsive and helpful as I embark on this migration.

-Will 

[Mark Mentovai]

Will Stokes wrote:

Those are the only two warnings I'm getting. If you could fix that that would be great. I personally like that warning since it alerts me to places in my code where I no longer need to be passing in such information and could avoid stack overhead. It happens as a project evolves over time.

I like the warning in theory too, but in our case, we’ve found that it contributes much more noise than utility. We often have abstract interfaces where some, but not all implementations make use of all parameters, or we’ll have parameters that are used or not based on build-time configuration differences, such as debug/release or OS.

I fixed this in 8b4932e56034 (also in mini_chromium).

Interesting. I recently upgraded my build environment from 10.6 to 10.10 and stopped supporting 10.6. In order to support debuggin with the modern XCode and Instruments I switched from passing the compiler -gstabs to -g. I wonder if this may have caused the problem. All the same c++filt appears to do a great job. The one gotcha is that I used to get nice filenames and line numbers at the end and now I'm missing those. All I see now at he end is the funny offset or soemthing values, e.g. + 0x10 which mean nothing to me. E.g. in the past I might see:

45  SnapGene!BaseApplication::notify(QObject*, QEvent*) [baseApplication.cpp : 201 + 0x16]

where as now I"m seeing stuff like this:

2 SnapGene!__ZN9DebugMenu18qt_static_metacallEP7QObjectN11QMetaObject4CallEiPPv + 0x44e

 

Which when I give the content beginning with the second _ all the way to the space to c++lint gives me back:

c++filt -n _ZN9DebugMenu18qt_static_metacallEP7QObjectN11QMetaObject4CallEiPPv

->

DebugMenu::qt_static_metacall(QObject*, QMetaObject::Call, int, void**)

So I have to be smart enough to pop off the "SnapGene!" as well as the first of the _'s, and any content after the first space, before passing it off to c++lint. Why I'm losing my filenames and line numbers of beyond me. I just verified I'm seeing a similar issue for my existing back builds that are still using Breakpad but were built with 10.10 so the issue is not Crashpad. Do you recommend any changes to my compiler flags in order to preserve the filenames and line numbers? I suspect by switching from STABS to DWARF for release builds I may have lost encoding the filename / line number information entirely.

This is definitely related to your switch from stabs to DWARF.

Not seeing source code file and line information means that dump_syms is only seeing the symbol table, and not any of the DWARF debugging information. You can verify this by looking at dump_syms’ output. If it’s processed any debugging information, you’ll see lines that begin with FUNC corresponding to each function that it found debugging information for. Each of these will be followed by lines that begin with numbers, which contain an encoded form of the source code file and line information.

With Apple’s toolchain, stabs debugging information was stored in the symbol table (that’s the “stab” in stabs) but DWARF debugging information is never stored in ld output. It lives in .o files, and is linked into a separate .dSYM companion by a special linker, dsymutil.

In order to create a full Breakpad dump_syms symbol file from something with DWARF debugging information on Mac OS X, you need to tell dump_syms about both the final linked image and its .dSYM. You can do “dump_syms -g /path/to/my_app.dSYM /path/to/my_app.app/Contents/MacOS/my_app”.

If you don’t have a .dSYM, run “dsymutil /path/to/my_app.app” first.

That should fix your source code file and line number problem. It may also fix the problem that you’re using c++filt to work around.

 

Interesting. Moving a much logical out of the handler into library code sounds great. Once that is done I wonder if the handler itself will be so trivial that it could be easily reproduced in a new program and then that program would just link to the .a files. That said, I'm not entirely sure how this will all work. I don't consider myself a Cocoa expert at all. I'm used to developing within the confines of Qt so that as much of our application is cross platform by default. With Breakpad you could set a callback function that would be run when the application crashed. This worked really well for me. I wrote a small Qt app that showed a dialog that apologized for the crash and let you enter a comment, your email, etc. I would run this reporter application from my call back function and pass it the filename of the mini dump that should be submitted. I wonder if a similar setup could be supported in crashpad on both Mac and Windows. If so then my custom GUI for reporting a crash, which based on Qt is already cross platform, could be used for both Mac and Windows and my crash handling experience would be identical on both OS's at long last. ;-)

In the future, we might be able to support something where the handler invokes another executable and reads instructions (like “don’t report this crash”) or additional annotations from its stdout. You’d be able to do UI from there, and you could do it using the platform’s native toolkit, Qt, or anything else.

We can’t accommodate that right now because I’m worried about blocking. It’s very important to us that crashpad_handler deal with a crash as quickly as possible, so that it can return to its exception-handling loop and deal with any other crashes. In our situation, crashpad_handler monitors many processes simultaneously, and if several of them crash close in time to each other, we don’t want crashpad_handler becoming a bottleneck.

We have a similar blocking problem with forwarding exceptions to the system’s crash reporter, and although it’s not a major problem given how we’re using Crashpad (we suppress exception forwarding for all but the main process), it’s a small wart on the handler that I’d like to remove. It’s possible that when we fix that, it would pave the way for having crashpad_handler launching an external program to do UI.

Not a problem. My hope is to adopt Crashpad on Windows as well once it is ready so I would be happy to provide feedback there as well when the time is right. Thank you all for being so responsive and helpful as I embark on this migration.

You’re welcome, we’re happy to see adoption.

Mark

[Mark Mentovai]

Will Stokes wrote:

Thank you, this fixed the problem. dsymutil took a long time and produced a > 600 MB directory, but in the end it all worked out and the breakpad symbol file is only about 22 MB. That said, I noticed a TON of warnings being dumped out while running dump_syms now, e.g.:

Unfortunately, it’s normal for dsymutil to take a very long time. For Chrome, it takes almost 10 minutes to produce a 4GB .dSYM.

Someday, someone will produce a faster open-source alternative, unless Apple makes the improvements first.

/path/to/SnapGene.dSYM/Contents/Resources/DWARF/SnapGene: warning: function at offset 0xba61ca0 has no name

/path/to/SnapGene.dSYM/Contents/Resources/DWARF/SnapGene: in compilation unit '../src/_viewers/sequenceViewer/sequenceView/sequenceViewLine/sequenceViewElementSet.cpp' (offset 0xba913b7):

As far as I can tell things all work out just fine. I suspect this is because I'm linking against Qt w/o debugging symbols. I'm fine with that. If that is indeed the issue, is there any way to quiet these warnings somehow?

This is also normal. These warnings aren’t about any code built without debugging information, because in that case, there won’t be any debugging information to complain about. These warnings are more common under compiler optimization, but you wouldn’t want to turn that off just because of this.

There aren’t really any plans to silence them in dump_syms, but the good news is that the plan for Crashpad’s eventual processing pipeline is to not require the use of dump_syms or a dump_syms-like tool on most platforms.

That's all very interesting. I look forward to watching your progress on things. I'll look into how to implement my UI based on your prior message as well. At the very least I now have a working headless crash reporter on Yosemite. :-)

Congratulations! I’m glad it’s working for you.

Mark

[Will Stokes]

There aren’t really any plans to silence them in dump_syms, but the good news is that the plan for Crashpad’s eventual processing pipeline is to not require the use of dump_syms or a dump_syms-like tool on most platforms.

Whoa, but dump_syms allows you to reduce a giant .dSYM and non-stripped binary into a much smaller stripped binary and tiny breakpad symbol file in comparison. What is the alternative you guys are envisioning?

 

[Mark Mentovai]

Will Stokes wrote:

There aren’t really any plans to silence them in dump_syms, but the good news is that the plan for Crashpad’s eventual processing pipeline is to not require the use of dump_syms or a dump_syms-like tool on most platforms.

Whoa, but dump_syms allows you to reduce a giant .dSYM and non-stripped binary into a much smaller stripped binary and tiny breakpad symbol file in comparison. What is the alternative you guys are envisioning?

There’s a lot of good information in the debugging information that isn’t carried in the dumped symbol format, like memory locations of variables and function parameters. Breakpad can’t get at these at all, and doesn’t even try. We’d like Crashpad’s processor to be able to make sense of these, so we’d like to to consume raw DWARF directly.

It’ll still be possible to use a reduced form of DWARF. When you build with -gline-tables-only, you get less debugging information. Even if you don’t build with this option, it should be possible to slim down the DWARF to just preserve this information. I’d like for the Crashpad processor to not require that anything be done to the DWARF to use it, but if you want to reduce or transform it, I don’t think it’s going to mind.

When I ship a build, though, I do generally want to retain full debugging information. As far as the executable goes, you can always ship a stripped executable, even if you save an unstripped copy for yourself for some reason.

The Crashpad processor will most likely support multiple formats for debugging information, and it’s possible that the dump_syms format or something like it will be one of them. The format for dump_syms isn’t ideal for processing crash reports because it requires the processor to read an entire text file to be able to make any sense of it, but we may wind up supporting it for compatibility with all of the existing dumped symbol files already running around out there.

[Will Stokes]

Very interesting. I think I'll stick with -g even for release builds and look forward to the enhanced CrashPad improvements. :-)

Will Stokes

Chief Software Architect

--
You received this message because you are subscribed to a topic in the Google Groups "Crashpad-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/crashpad-dev/NaB7PrfW04g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to crashpad-dev...@chromium.org.

To post to this group, send email to crashp...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/CAHk1GDwz5txTxhmyw5FaxKm87JE67YWD5TKJ3X_TMFoUGwnONg%40mail.gmail.com.

[Jiakuan Wang]

This thread is extremely helpful for me while I'm trying to integrate Crashpad into a Qt application (Qt 5.9.2).

I've been able to build Crashpad on macOS 10.12.6 successfully with Mac deployment target 10.7

And then I added dependencies into my Qt project file:

```
LIBS += -L/usr/lib/ -lbsm

macx:CONFIG(release, debug|release): LIBS += -L$$(HOME)/workrepo/crashpad/lib/Release/ -lbase
else:macx:CONFIG(debug, debug|release): LIBS += -L$$(HOME)/workrepo/crashpad/lib/Debug/ -lbase

macx:CONFIG(release, debug|release): LIBS += -L$$(HOME)/workrepo/crashpad/lib/Release/ -lcrashpad_client
else:macx:CONFIG(debug, debug|release): LIBS += -L$$(HOME)/workrepo/crashpad/lib/Debug/ -lcrashpad_client

macx:CONFIG(release, debug|release): LIBS += -L$$(HOME)/workrepo/crashpad/lib/Release/ -lcrashpad_compat
else:macx:CONFIG(debug, debug|release): LIBS += -L$$(HOME)/workrepo/crashpad/lib/Debug/ -lcrashpad_compat

macx:CONFIG(release, debug|release): LIBS += -L$$(HOME)/workrepo/crashpad/lib/Release/ -lcrashpad_util
else:macx:CONFIG(debug, debug|release): LIBS += -L$$(HOME)/workrepo/crashpad/lib/Debug/ -lcrashpad_util

INCLUDEPATH += $$(HOME)/workrepo/crashpad/include
DEPENDPATH += $$(HOME)/workrepo/crashpad/include

macx:CONFIG(release, debug|release): PRE_TARGETDEPS += $$(HOME)/workrepo/crashpad/lib/Release/libbase.a
else:macx:CONFIG(debug, debug|release): PRE_TARGETDEPS += $$(HOME)/workrepo/crashpad/lib/Debug/libbase.a

macx:CONFIG(release, debug|release): PRE_TARGETDEPS += $$(HOME)/workrepo/crashpad/lib/Release/libcrashpad_client.a
else:macx:CONFIG(debug, debug|release): PRE_TARGETDEPS += $$(HOME)/workrepo/crashpad/lib/Debug/libcrashpad_client.a

macx:CONFIG(release, debug|release): PRE_TARGETDEPS += $$(HOME)/workrepo/crashpad/lib/Release/libcrashpad_compat.a
else:macx:CONFIG(debug, debug|release): PRE_TARGETDEPS += $$(HOME)/workrepo/crashpad/lib/Debug/libcrashpad_compat.a

macx:CONFIG(release, debug|release): PRE_TARGETDEPS += $$(HOME)/workrepo/crashpad/lib/Release/libcrashpad_util.a
else:macx:CONFIG(debug, debug|release): PRE_TARGETDEPS += $$(HOME)/workrepo/crashpad/lib/Debug/libcrashpad_util.a

```

When compiling my Qt project, I got a strange error:

```
Undefined symbols for architecture x86_64:

"_SecACLGetTypeID", referenced from:
OpaqueSecTrustRef* base::mac::CFCast<OpaqueSecTrustRef*>(void const* const&) in libbase.a(base.foundation_util.o)
"_SecTrustedApplicationGetTypeID", referenced from:
OpaqueSecTrustedApplicationRef* base::mac::CFCast<OpaqueSecTrustedApplicationRef*>(void const* const&) in libbase.a(base.foundation_util.o)

ld: symbol(s) not found for architecture x86_64

clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [crashpad-qt-test.app/Contents/MacOS/crashpad-qt-test] Error 1
```

This looks to me that I missed some dynamic libraries in Qt app, but after a long time of research via Google I'm still not sure which library is missing. This error is from Crashpad libbase.a

Anyone has any ideas about this error?

[Jiakuan Wang]

I'm able to compile my Qt project successfully now, after I added these libraries:

LIBS += -framework AppKit
LIBS += -framework Security