Linux + Official vs Release + Symbols

[dan...@chromium.org]

Hello,

In compiler.gni we have the following:

 } else if ((!is_nacl && !is_linux) || is_debug || is_official_build ||
             is_chromecast) {
    # Linux is slowed by having symbols as part of the target binary, whereas
    # Mac and Windows have them separate, so in Release Linux, default them off,
    # but keep them on for Official builds and Chromecast builds.
    symbol_level = 2
This comment says that on linux the binary runs slower with symbols. There are 2 strange things going on here though:

1. Official build gets symbol_level = 2 on linux. This means the official build performance is negatively affected. This was changed to be the case in https://codereview.chromium.org/1373323002.

2. Release build doesn't get symbol_level = 2 on linux to be faster. This creates a situation where official build has worse performance than non-official, which AFAIK is the opposite to the intent of official.

So I am wondering, if it's important to have the symbols in official, why do we bother to remove them on linux release non-official?

Thanks,

Dana

[Torne (Richard Coles)]

Also, the actual performance impact might now be small/zero if debug fission is being used, which appears to be the default in some configurations?

On Mon, 13 Feb 2017 at 17:17 Torne (Richard Coles) <to...@chromium.org> wrote:

Official builds get their symbols stripped before being sent to users, so it has no impact on performance after that - but we need the symbols at build time to generate the breakpad db for the binary to decode crashes. For non-official builds we don't care about generating breakpad symbols because we don't ship them to users, so it's not necessary to build them in the first place.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Torne (Richard Coles)]

Official builds get their symbols stripped before being sent to users, so it has no impact on performance after that - but we need the symbols at build time to generate the breakpad db for the binary to decode crashes. For non-official builds we don't care about generating breakpad symbols because we don't ship them to users, so it's not necessary to build them in the first place.

On Mon, 13 Feb 2017 at 17:07 <dan...@chromium.org> wrote:

--

[Michael Moss]

I'd guess the performance issue is not so much about having symbols enabled at all, but trying to run multi-GB binaries with the symbols in them. Official builds have the symbols separated from the binary (eu-strip -f) and archived, which keeps them available for debugging, but without bloating what is shipped. One possibility to avoid confusion would be to make that eu-strip step part of the normal build, rather than part of packaging, as it is now.

--

[dan...@chromium.org]

Perf bots run an official build to match what we ship to users. In this case are they profiling with symbols that would make different performance characteristics? Or do perf bots run these symbol stripping steps?

[Nico Weber]

I think this is about build speed, not execution speed.

[Michael Moss]

That doesn't seem to match what's implied by the comment, unless building mac/win symbols is somehow free.

[Nico Weber]

It roughly is, since mac/win use something similar to fission by default and have for a long time*

(*: on windows, this isn't true with goma or clang)

[Michael Moss]

OK, I didn't realize the impact was so substantial, so yeah, I guess it could imply to build time (although it would be nice if the comment was explicit). Any idea why we wouldn't use fission for the Linux official builds as well?

[Weiliang Chen]

I thought the comment was about execution speed?

Trying to profile perf test on local Linux box, w/o symbol it runs ~25 us, and turning on symbol_level=2 make same times ~145us.

[Primiano Tucci]

What specifically takes 25 vs 145 us?
The .text section is not affected by symbol_level.  Afaik the only thing that could be slower is early startup time due to the dynamic loader having to parse a bigger elf header (+ the kernel readahead filling the I/O scheduler with  read jobs on useless parts of the binary). But that shouldn't affect perf measurements as it happens before we reach the first line of code in Chrome, hence have any opportunity to bring up tracing or any perf measuring harness.

[Dirk Pranke]

Like some of the others, I'm not sure if we're talking about compile/build speed or runtime speed.

@danakj - can you clarify?

-- Dirk

[dan...@chromium.org]

On Mon, Feb 13, 2017 at 7:05 PM, Dirk Pranke <dpr...@chromium.org> wrote:

Like some of the others, I'm not sure if we're talking about compile/build speed or runtime speed.

@danakj - can you clarify?

I traced that comment back to its original form and its ambiguous but I have a hard time reading it as being about compile time tbh. My original concern/confusion was that it seemed to be claiming runtime perf but then official bots would use the slower one, and we'd be using that on perf bots, making our linux perf results maybe incorrect.

Wei's got some perf test results that show symbols make a difference, and has been trying with eu-strip as well. Wei can you report what commands you're running and what the time that is being measured represents so we can get a better idea?

[Nico Weber]

I did that yesterday too :-) https://chromium.googlesource.com/chromium/src/+/29682b848742042388ebd29c9df775f074c1414c%5E%21/#F1 added it to gn to make things consistent with gyp, and in gyp I kind of got lost. I'm pretty sure (63%?) we didn't do symbols on linux release because of build time though.

[Dirk Pranke]

I believe that we think that building with full symbols is both significantly slower and takes up more disk space and as a result most people don't actually want full symbols when doing release builds.

We can certainly change that if people would prefer.

The release bots don't build with full symbols, and aren't going to -- they only need minimal symbols so that they can provide stack traces -- but those are configured explicitly.

We can also strip the binaries if that's useful, too.

-- Dirk

[Primiano Tucci]

> We can also strip the binaries if that's useful, too.

If you do that, in the case of a crash (or just a CHECK) base::debug::StackTrace will start printing addresses and lose actionability. This will make diagnosing a unittest/browsertest crash on the CQ quite hard. 

I would be really curious to understand if there is a perf impact due to symbol_level. It should be quite easy to get some numbers using perf trybots.

> The release bots don't build with full symbols, and aren't going to 

I assume you are talking only about the main waterfall and not the official release bots?

Those will still need symbols as they are used to generate the breakpad syms for crash reporting.

> I believe that we think that building with full symbols is both significantly slower and takes up more disk space and as a result most people don't actually want full symbols when doing release builds.

When you say "full" you mean symbol_level=2 or symbol_level>0 ? 

one thing to keep in mind is that symbol_level=0 means no human-readable stacktrace on (D)CHECKs, which might be something that developers still care about on local builds. symbol_level=1 is enough for that, and maybe is a reasonable tradeoff in terms of build time?

> We can also strip the binaries if that's useful, too.

My $0.02 is that what would make sense for developers is:

is_debug=true -> symbol_level=2 by default. Rationale: if I build for debug I expect to be able to use a debugger. that means symbol_level=2)

is_debug=false -> symbol_level=1 by default. Rationale: if I build for release I still expect crashes and CHECKs to spit out a meaningful stack trace.

(but maybe I  am just being confused by the various comments in this thread and just stating the obvious?)

[Dirk Pranke]

On Tue, Feb 14, 2017 at 9:54 AM, Primiano Tucci <prim...@chromium.org> wrote:

> We can also strip the binaries if that's useful, too.

If you do that, in the case of a crash (or just a CHECK) base::debug::StackTrace will start printing addresses and lose actionability. This will make diagnosing a unittest/browsertest crash on the CQ quite hard. 

To be clear, I wouldn't suggest we strip regular release binaries on the non-perf builders. 

 

I would be really curious to understand if there is a perf impact due to symbol_level. It should be quite easy to get some numbers using perf trybots.

Right, I was trying to say that *if* there was a perf impact on the perf  tests due to symbols, we could consider stripping.

> The release bots don't build with full symbols, and aren't going to 

I assume you are talking only about the main waterfall and not the official release bots?

Those will still need symbols as they are used to generate the breakpad syms for crash reporting.

Right :).

 

> I believe that we think that building with full symbols is both significantly slower and takes up more disk space and as a result most people don't actually want full symbols when doing release builds.

When you say "full" you mean symbol_level=2 or symbol_level>0 ? 

one thing to keep in mind is that symbol_level=0 means no human-readable stacktrace on (D)CHECKs, which might be something that developers still care about on local builds. symbol_level=1 is enough for that, and maybe is a reasonable tradeoff in terms of build time?

> We can also strip the binaries if that's useful, too.

My $0.02 is that what would make sense for developers is:

is_debug=true -> symbol_level=2 by default. Rationale: if I build for debug I expect to be able to use a debugger. that means symbol_level=2)

is_debug=false -> symbol_level=1 by default. Rationale: if I build for release I still expect crashes and CHECKs to spit out a meaningful stack trace.

(but maybe I  am just being confused by the various comments in this thread and just stating the obvious?)

full == symbol_level=2

symbol_level=1 is called "minimal" in the MB configs the builders use and is what they use virtually across the board.

I would probably be fine with us defaulting a release build (is_debug=false is_official_build=false) to symbol_level=1 on linux. Android is already defaulting to 1 in most cases, and Mac and Windows default to 2 because I've heard repeatedly from mac and win devs that they want full symbols.

It's hard to please everybody :).

[Weiliang Chen]

To clarify, I was trying to profile a test I added to cc_perftests (not checked in, just on my local machine) and came across the symbol_level setting in gn args. I was under the impression that the comment in compiler.gni is talking about execution speed being slow with symbol_level = 2 on Linux, and was surprised to see that when is_official_build, we turn on symbol_level = 2 even on Linux.

I tried running my addition to cc_perftests (similar to ones in cc_serialization_perftest but more complex cases) with enable_profiling = true, and is_debug = false, and did not set symbol_level, one of the test is taking ~25us per iteration.

Running same test with enable_profiling = true, is_debug = false, and symbol_level = 2, now takes ~145us per iteration.

I also tried eu-strip -f the symbol_level = 2 build, which result in a smaller binary, but same ~145us per iteration. Also tried strip on same build, similar binary size and execution speed with eu-strip -f binary.

I wasn't paying attention to build speed so I could not comment on that. 

Was the execution speed slowing down when build w/ symbols because cc_perftests have different setupt than just building Chromium?

I am mostly just surprised that is_official_build on Linux would have slower execution speed than Release build.

[Dmitry Skiba]

Hmm, but enable_profiling=true turns on frame pointers (on Linux / Android at least), which means we're doing more work in the function prologue / have one register less.

See https://cs.chromium.org/chromium/src/build/config/compiler/BUILD.gn?l=1387

You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

[Weiliang Chen]

As I was running more experiment w/ different configuration I realized my slow build has enable_full_stack_frames_for_profiling = true. Execution speed for symbol_level = 2 and enable_profiling does not slow down.

Sorry for the noise. I am just gonna go to the corner and hide in my shame cube now.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[dan...@chromium.org]

It'd be nice to make the config.gni comment a bit more clear that this is about build speed then. I will send a patch.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.