PSA: Breakpad is now compiled into Chromium by default on Linux

[Mark Seaborn]

Breakpad (the crash reporter) is now compiled into Chromium by default on Linux, though it's not enabled at run time by default, and this doesn't require compiling with "-g".

The motivation is to allow Chrome's Breakpad support to be tested on the buildbots (see http://crbug.com/105778).

This brings the Linux build into line with the Mac and Windows builds, where Breakpad support has been compiled in by default for a while.

You can still omit Breakpad support by passing linux_breakpad=0 to Gyp.  I expect distros will want to do that.  One caveat here is that there's an unfortunately large number of "#ifdef USE_LINUX_BREAKPAD"s in the code, so the linux_breakpad=0 case might bitrot.

Cheers,
Mark

[Daniel Erat]

[cc chromium-os-dev in case other developers have run into this]

I think that this might break the "daisy" (ARM-based) platform on Chrome OS.  I haven't dug into it much, but in recent builds I get many errors about redefined functions, starting with the following:

In file included from breakpad/src/client/linux/log/log.cc:35:0:

breakpad/src/third_party/lss/linux_syscall_support.h: In function 'ssize_t sys_recvmsg(int, kernel_msghdr*, int)':

breakpad/src/third_party/lss/linux_syscall_support.h:3251:2: error: redefinition of 'ssize_t sys_recvmsg(int, kernel_msghdr*, int)'

In file included from breakpad/src/client/linux/log/log.cc:35:0:

breakpad/src/third_party/lss/linux_syscall_support.h:3238:2: error: 'ssize_t sys_recvmsg(int, kernel_msghdr*, int)' previously defined here

(Let me know if you'd like the full 2.3 MB build log.)  I'm able to build Chrome successfully when I set EXTRA_BUILD_ARGS="linux_breakpad=0" before running Chrome OS's build_packages script.

[Denis Glotov]

And what about ChromeOS? Do you still need -g there?

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

--
Thank you,
Denis

[Mark Seaborn]

On 23 January 2013 09:56, Daniel Erat <de...@chromium.org> wrote:

[cc chromium-os-dev in case other developers have run into this]

On Wed, Jan 16, 2013 at 2:51 PM, Mark Seaborn <msea...@chromium.org> wrote:

Breakpad (the crash reporter) is now compiled into Chromium by default on Linux, though it's not enabled at run time by default, and this doesn't require compiling with "-g".

The motivation is to allow Chrome's Breakpad support to be tested on the buildbots (see http://crbug.com/105778).

This brings the Linux build into line with the Mac and Windows builds, where Breakpad support has been compiled in by default for a while.

You can still omit Breakpad support by passing linux_breakpad=0 to Gyp.  I expect distros will want to do that.  One caveat here is that there's an unfortunately large number of "#ifdef USE_LINUX_BREAKPAD"s in the code, so the linux_breakpad=0 case might bitrot.

I think that this might break the "daisy" (ARM-based) platform on Chrome OS.  I haven't dug into it much, but in recent builds I get many errors about redefined functions, starting with the following:

Is that failing on a buildbot/trybot?  All my recent cros_daisy trybot runs are fine, and so is the ChromiumOS daisy builder on build.chromium.org.

Cheers,
Mark

[Daniel Erat]

Not that I'm aware of; I've only noticed it in my local builds.  Full command line within the chroot:

  USE="-build_tests gobi -chrome_internal -chrome_debug chrome_media" \

    FEATURES="-usersandbox" \

    EXTRA_BUILD_ARGS="linux_breakpad=0" \

    CHROME_ORIGIN=LOCAL_SOURCE KEEP_CHROME_DEBUG_SYMBOLS=1 \

    ./build_packages --board=daisy

[Mark Seaborn]

On 23 January 2013 10:06, Denis Glotov <glo...@chromium.org> wrote:

And what about ChromeOS? Do you still need -g there?

I'm not sure what you're asking.  "-g" is only needed if you want to be able to decode Breakpad's crash dumps to get backtraces with symbols and line numbers.  Passing linux_dump_symbols=1 to Gyp enables "-g", and official Chrome builds enable linux_dump_symbols=1.  AFAICT, this isn't different for ChromeOS/ChromiumOS.

Mark
 

[John Abd-El-Malek]

To bring this thread back since I just broke this yesterday. In my case, one engineer used to happen to build without USE_LINUX_BREAKPAD and he fixed it.

Why do we need USE_LINUX_BREAKPAD anymore? Distros don't have to enable this at runtime. If we don't have any builders for this, then it will bitrot and it's not clear to me why we should maintain building chrome without breakpad.

--

[David Turner]

On Thu, Jul 11, 2013 at 1:35 PM, John Abd-El-Malek <j...@chromium.org> wrote:

To bring this thread back since I just broke this yesterday. In my case, one engineer used to happen to build without USE_LINUX_BREAKPAD and he fixed it.

Why do we need USE_LINUX_BREAKPAD anymore? Distros don't have to enable this at runtime. If we don't have any builders for this, then it will bitrot and it's not clear to me why we should maintain building chrome without breakpad.

We build without Breakpad on Android for non-official builds. This is useful during development because the system generates stack traces in the log and tombstone files that are very useful to analyze failures on the spot (i.e. tombstones get more info than Breakpad minidumps, and there are more tools to process them).

On Wed, Jan 23, 2013 at 11:56 AM, Mark Seaborn <msea...@chromium.org> wrote:

On 23 January 2013 10:06, Denis Glotov <glo...@chromium.org> wrote:

And what about ChromeOS? Do you still need -g there?

I'm not sure what you're asking.  "-g" is only needed if you want to be able to decode Breakpad's crash dumps to get backtraces with symbols and line numbers.  Passing linux_dump_symbols=1 to Gyp enables "-g", and official Chrome builds enable linux_dump_symbols=1.  AFAICT, this isn't different for ChromeOS/ChromiumOS.

Mark
 

On Thu, Jan 17, 2013 at 2:51 AM, Mark Seaborn <msea...@chromium.org> wrote:

Breakpad (the crash reporter) is now compiled into Chromium by default on Linux, though it's not enabled at run time by default, and this doesn't require compiling with "-g".

The motivation is to allow Chrome's Breakpad support to be tested on the buildbots (see http://crbug.com/105778).

This brings the Linux build into line with the Mac and Windows builds, where Breakpad support has been compiled in by default for a while.

You can still omit Breakpad support by passing linux_breakpad=0 to Gyp.  I expect distros will want to do that.  One caveat here is that there's an unfortunately large number of "#ifdef USE_LINUX_BREAKPAD"s in the code, so the linux_breakpad=0 case might bitrot.

Cheers,
Mark

 

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

--

[Mark Seaborn]

On 11 July 2013 14:57, David Turner <di...@chromium.org> wrote:

On Thu, Jul 11, 2013 at 1:35 PM, John Abd-El-Malek <j...@chromium.org> wrote:

To bring this thread back since I just broke this yesterday. In my case, one engineer used to happen to build without USE_LINUX_BREAKPAD and he fixed it.

Why do we need USE_LINUX_BREAKPAD anymore? Distros don't have to enable this at runtime. If we don't have any builders for this, then it will bitrot and it's not clear to me why we should maintain building chrome without breakpad.

Yes, I think it would make sense to remove the USE_LINUX_BREAKPAD #define.

For some reason, there are a lot of #ifdefs for Breakpad on Linux, but not many for Windows and Mac.

 

We build without Breakpad on Android for non-official builds. This is useful during development because the system generates stack traces in the log and tombstone files that are very useful to analyze failures on the spot (i.e. tombstones get more info than Breakpad minidumps, and there are more tools to process them).

That's just about whether Breakpad is enabled at run time, though, right?  As my first post in this thread said, Breakpad is disabled at run time in Chromium, so it shouldn't interfere with other crash handlers.

Cheers,
Mark

[Yaron Friedman]

Right. We actually did change it in response to this thread so breakpad is always compiled on Android, it's just disabled for debug builds.

 

Cheers,
Mark

[Chris Hopman]

On Thu, Jul 11, 2013 at 2:57 PM, David Turner <di...@chromium.org> wrote:

On Thu, Jul 11, 2013 at 1:35 PM, John Abd-El-Malek <j...@chromium.org> wrote:

To bring this thread back since I just broke this yesterday. In my case, one engineer used to happen to build without USE_LINUX_BREAKPAD and he fixed it.

Why do we need USE_LINUX_BREAKPAD anymore? Distros don't have to enable this at runtime. If we don't have any builders for this, then it will bitrot and it's not clear to me why we should maintain building chrome without breakpad.

We build without Breakpad on Android for non-official builds. This is useful during development because the system generates stack traces in the log and tombstone files that are very useful to analyze failures on the spot (i.e. tombstones get more info than Breakpad minidumps, and there are more tools to process them).

I fixed this long ago so that breakpad re-installs the previous signal handler after doing its stuff. That is, we now get both a minidump and a tombstone.

[David Turner]

On Fri, Jul 12, 2013 at 10:34 AM, Chris Hopman <cjho...@chromium.org> wrote:

On Thu, Jul 11, 2013 at 2:57 PM, David Turner <di...@chromium.org> wrote:

On Thu, Jul 11, 2013 at 1:35 PM, John Abd-El-Malek <j...@chromium.org> wrote:

To bring this thread back since I just broke this yesterday. In my case, one engineer used to happen to build without USE_LINUX_BREAKPAD and he fixed it.

Why do we need USE_LINUX_BREAKPAD anymore? Distros don't have to enable this at runtime. If we don't have any builders for this, then it will bitrot and it's not clear to me why we should maintain building chrome without breakpad.

We build without Breakpad on Android for non-official builds. This is useful during development because the system generates stack traces in the log and tombstone files that are very useful to analyze failures on the spot (i.e. tombstones get more info than Breakpad minidumps, and there are more tools to process them).

I fixed this long ago so that breakpad re-installs the previous signal handler after doing its stuff. That is, we now get both a minidump and a tombstone.

 

Excellent, thanks for the clarifications guys :)

[Matthew Dempsky]

On Thu, Jul 11, 2013 at 1:35 PM, John Abd-El-Malek <j...@chromium.org> wrote:

Why do we need USE_LINUX_BREAKPAD anymore?

I know OpenBSD's chromium port configures with -Dlinux_breakpad=0, but I don't know much about breakpad beyond that.

[John Abd-El-Malek]

Here's a change to remove USE_LINUX_BREAKPAD: https://codereview.chromium.org/18770006/

If anyone has any strong objections to this, please speak up now.

[Lei Zhang]

Just flipping the flag was a nice way to have a transition period
instead of suddenly removing the option when we made Breakpad compiled
in by default on Linux. If the non-default option doesn't build and
nobody has complained, then it's clearly not being used. In which
case, I fully support removing it, since it's one less build
configuration to maintain.

On Thu, Jul 11, 2013 at 1:35 PM, John Abd-El-Malek <j...@chromium.org> wrote:

> --

[Petar Jovanovic]

I know I am one day late, but I did not see this discussion and the change yesterday.

Ergo, this change has just broken the MIPS port.

http://www.rt-rk.com/mips-buildbot/builders/build_and_test_chromium_trunk_for_MIPS/builds/46

It breaks the build, since Breakpad does not have support for MIPS (yet).

Now, the good part is that patches for breakpad are ready and can be upstreamed shortly.

However, MIPS is broken as we speak and whoever tries to build it will see it broken, so I wish you had delayed this change until breakpad patches for MIPS are reviewed and submitted.

Knowing the change is already in, any advice how to handle this now until the breakpad patches get committed?

Petar

[John Abd-El-Malek]

On Tue, Jul 16, 2013 at 4:30 PM, Petar Jovanovic <pet...@mips.com> wrote:

I know I am one day late, but I did not see this discussion and the change yesterday.

Ergo, this change has just broken the MIPS port.

http://www.rt-rk.com/mips-buildbot/builders/build_and_test_chromium_trunk_for_MIPS/builds/46

It breaks the build, since Breakpad does not have support for MIPS (yet).

Now, the good part is that patches for breakpad are ready and can be upstreamed shortly.

However, MIPS is broken as we speak and whoever tries to build it will see it broken, so I wish you had delayed this change until breakpad patches for MIPS are reviewed and submitted.

Knowing the change is already in, any advice how to handle this now until the breakpad patches get committed?

Do you have a way to apply local patches to your build after syncing? I'm not familiar with the MIPS build.

Chromium devs are only responsible for maintaining builds on our buildbot, so I would try to apply your breakpad patches locally until they're upstreamed.

[Petar Jovanovic]

The problem is not me building Chromium, I can revert the change that creates a problem or apply the breakpad patches. But other people pull from the trunk, so it will be broken for them.

---

From: jabde...@google.com [jabde...@google.com] on behalf of John Abd-El-Malek [j...@chromium.org]
Sent: Wednesday, July 17, 2013 1:46 AM
To: Petar Jovanovic
Cc: chromium-dev; Mark Seaborn; glo...@chromium.org
Subject: Re: [chromium-dev] PSA: Breakpad is now compiled into Chromium by default on Linux

[John Abd-El-Malek]

If you have a mailing list for people who build this config, you can alert them not to sync past the revision that broke this or give them a patch to fix it. It's up to you.

Stepping back a bit, it is not feasible to have a config that should stay buildable and runnable even though it's not on the chromium waterfall. You will get pushback to add it there, as we already have too many configs, and can't support new ones for external parties. Perhaps you should have your own tracking repository that you sync chromium to, and have your own patches applied there. That way you can rev it only when you have a known good build.

[Petar Jovanovic]

I am aware that as long as no chromium buildbots test MIPS port regularly, it is doomed to break now and then. We now have a year-long history of catching up with changes on the trunk, you can see it at:

https://code.google.com/p/chromium/issues/detail?id=130022

so I am not opening that discussion here. I replied to this post and the particular change as I was wondering if we can have a workaround patch for MIPS until breakpad patches land, or the port has to remain broken up to that point.

Petar

[Mark Seaborn]

On 16 July 2013 16:30, Petar Jovanovic <pet...@mips.com> wrote:

I know I am one day late, but I did not see this discussion and the change yesterday.

Ergo, this change has just broken the MIPS port.

http://www.rt-rk.com/mips-buildbot/builders/build_and_test_chromium_trunk_for_MIPS/builds/46

It breaks the build, since Breakpad does not have support for MIPS (yet).

Now, the good part is that patches for breakpad are ready and can be upstreamed shortly.

However, MIPS is broken as we speak and whoever tries to build it will see it broken, so I wish you had delayed this change until breakpad patches for MIPS are reviewed and submitted.

Knowing the change is already in, any advice how to handle this now until the breakpad patches get committed?

Have a look at how the Mac OS X build allows Breakpad support to be omitted.  It has:

  chrome/app/breakpad_mac.mm
  chrome/app/breakpad_mac_stubs.mm

I suspect you can add a similarly trivial breakpad_linux_stubs.cc which defines no-op versions of the functions declared in breakpad_linux.h.  This would be a cleaner way of making Breakpad support optional than the USE_LINUX_BREAKPAD ifdefs that used to be spinkled across the codebase (which were removed in https://src.chromium.org/viewvc/chrome?view=rev&revision=211755).

Cheers,

Mark

[John Abd-El-Malek]

I hope you don't mean checking this in to the chromium repo? We should avoid stub files that are used in this manner; this has come up before especially in the context of porting, ios builds etc.

 

Cheers,

Mark

[Mark Seaborn]

Well, it's not my decision because I'm (thankfully!) not an OWNER of chrome/app/breakpad_linux.cc.  There is precedent for having a breakpad_linux_stubs.cc since there's already a breakpad_mac_stubs.mm, so it would at least be consistent.  On the other hand, I expect breakpad_mac_stubs.mm is not tested by any bot, so I take your point.

Your suggestion of having a Git branch of Chromium for MIPS seems sensible, since it's not really realistic to expect tip-of-tree Chromium to always build for MIPS when this configuration is not covered by trybots.  (One could say the same for ARM Chromium. ;-) )  So adding a breakpad_linux_stubs.cc on a MIPS branch would be a simple way of fixing the build with a low risk of merge conflicts.

Cheers,

Mark

[John Abd-El-Malek]

On Tue, Jul 16, 2013 at 6:01 PM, Mark Seaborn <msea...@chromium.org> wrote:

On 16 July 2013 17:46, John Abd-El-Malek <j...@chromium.org> wrote:

On Tue, Jul 16, 2013 at 5:34 PM, Mark Seaborn <msea...@chromium.org> wrote:

On 16 July 2013 16:30, Petar Jovanovic <pet...@mips.com> wrote:

I know I am one day late, but I did not see this discussion and the change yesterday.

Ergo, this change has just broken the MIPS port.

http://www.rt-rk.com/mips-buildbot/builders/build_and_test_chromium_trunk_for_MIPS/builds/46

It breaks the build, since Breakpad does not have support for MIPS (yet).

Now, the good part is that patches for breakpad are ready and can be upstreamed shortly.

However, MIPS is broken as we speak and whoever tries to build it will see it broken, so I wish you had delayed this change until breakpad patches for MIPS are reviewed and submitted.

Knowing the change is already in, any advice how to handle this now until the breakpad patches get committed?

Have a look at how the Mac OS X build allows Breakpad support to be omitted.  It has:

  chrome/app/breakpad_mac.mm
  chrome/app/breakpad_mac_stubs.mm

I suspect you can add a similarly trivial breakpad_linux_stubs.cc which defines no-op versions of the functions declared in breakpad_linux.h.  This would be a cleaner way of making Breakpad support optional than the USE_LINUX_BREAKPAD ifdefs that used to be spinkled across the codebase (which were removed in https://src.chromium.org/viewvc/chrome?view=rev&revision=211755).

I hope you don't mean checking this in to the chromium repo? We should avoid stub files that are used in this manner; this has come up before especially in the context of porting, ios builds etc.

Well, it's not my decision because I'm (thankfully!) not an OWNER of chrome/app/breakpad_linux.cc.  There is precedent for having a breakpad_linux_stubs.cc since there's already a breakpad_mac_stubs.mm, so it would at least be consistent.

We should remove those stubs if possible, instead of using it as a precedent for adding more :) I'll follow up with Mac folks to see why/if we need them.

[Petar Jovanovic]

If somebody has bandwidth to review MIPS breakpad patches, they have been
uploaded at:

https://breakpad.appspot.com/614002/

Thanks.

Regards,
Petar