Compile time increase over time

[Daniel Bratell]

We all have the feeling (and numbers) that tells us that compile times are increasing. In an earlier thread someone asked for harder data than feelings but it's hard to find because over time circumstances and hardware change.

The best I could find is this plot of one machine building a few hundred Opera Desktop Win32 builds from 2013 until now. Compile time on the y axis, calendar date on the x axis.

If this is representative of the typical experience it's an increase of +320%. I would be very interested to know if someone else have data that support or contradict the image above.

If there is a huge compile time increase, what are the causes?

I don't know, but I am pretty sure reasons include more code, more features and C++11 (C++11 headers are slower to compile). Still, I doubt it's possible to pinpoint one thing, and even less possible to revert something back to 2013.

/Daniel

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[Nico Weber]

Nice data!

45 min and 64 min kind of look like jumps, while the 15->30 buildup looks more gradual. Especially the 64 minutes jump like it might be bisectable.

When did you update toolchains / SDKs on the bot? `git log build/vs_toolchain.py` suggests that our bots got MSVC2015 around Mar 2016, and we switched to 2015 update 3 on Sep 7 2016. If you did the same updates slightly later, that lines up pretty well with the recent increases.

[Nico Weber]

An easy way to test this: Do you have similar graphs for non-Windows bots?

[Yoav Weiss]

Can we similarly track the increase in files built on the common targets (e.g. "chrome", "content_shell" and "blink_tests").

My anecdotal impression is that the number of files built has gotten significantly larger (from ~15K to ~30K) over the last few years.  

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Nico Weber]

For number of build edges: Note that in the gyp build, a good chunk of the codebase was built a second time with the nacl codechain by some python script, which was a single build rule (that took 20 minutes to run or so). With gn, nacl can instead use a gn toolchain, so every .cc file being built with the nacl toolchain is now a separate build edge that ninja knows about, which means the number of build steps with gn is much higher due to nacl (without there being more work). If you put disable_nacl=true in your args.gn, the number of build edges should be much lower (and your build a good chunk faster).

(I sometimes wonder if we should disable_nacl=false in non-official builds, given that the nacl code isn't changing too much these days.)

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Scott Hess]

Someone recently suggested disable_nacl=false as being super awesome, so on my next build of the browser I jumped on it, and ... I didn't really notice a difference.  So maybe the slow pace of change means that having that change wouldn't even matter.

-scott

[I may have been holding it wrong.]

[Primiano Tucci]

On Fri, Mar 24, 2017 at 6:21 PM Scott Hess <sh...@chromium.org> wrote:

Someone recently suggested disable_nacl=false as being super awesome,

I think that was me

 

so on my next build of the browser I jumped on it, and ... I didn't really notice a difference.  So maybe the slow pace of change means that having that change wouldn't even matter.

So, I checked from a recent Linux checkout for a clean build.

For the chrome target the difference is 27k vs 30k build steps.

For the "all" target 45k vs 51k

 

-scott

[I may have been holding it wrong.]

I think that the issue is that due to dependency, building the extra base target happens pretty early. So if you interrupt your build midway through, disable nacl, and re-build without touching anything in base/, you won't see any difference in the remaining build targets because those 3k object for nacl's base have already been built.

In other words, I think disabling nacl doesn't affect incremental builds, unless you make changes to //base, but will affect the time it takes to build:

-  from clean / after a landmine

- after a rebase, assuming that changes in //base happen quite frequently.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Colin Blundell]

I'm confused: what would be the possible benefit from "disable_nacl=false", assuming that you don't need nacl?

[Matthew Menke]

From Primiano's comment, it sounds like building after updating your checkout and clean builds will be faster, but incremental builds generally won't be (Unless you're touching the files used by both nacl and chrome, maybe?)

[Colin Blundell]

Heh, I was reading disable_nacl=false as disable_nacl=false. I assume everyone else read it as enable_nacl=false. :)

[Sergio Villar Senin]

O Mér, 22-03-2017 ás 16:49 +0100, Daniel Bratell escribiu:
> We all have the feeling (and numbers) that tells us that compile
> times are increasing. In an earlier thread someone asked for harder
> data than feelings but it's hard to find because over time
> circumstances and hardware change.

I can confirm your figures on linux desktop (amd64). It was indeed
around 15 minutes back in 2013 using a distributed compilation with an
i7 laptop (2 cores) and a i7 desktop (8 cores). Build times have been
increasing over the time, to the point that buying new hardware is the
sole alternative. Last year the build time for blink_tests target (I am
not talking about chromium) from scratch was ~75 minutes with the same
hardware.

Regarding the causes, I have not examined them in detail, but the
number of targets have increased a lot for sure specially these last 2
years I'd say. Nowadays a clean build of blink_tests is ~21k targets.
It was ~12k a couple of years ago.

BR

[Daniel Bratell]

Really sorry I was silent in this thread. It fell between two filters and I just discovered the responses.

Switches between VS2013 and VS2015 was gradual as it hit different release branches. It seems like we did the switch on master March 23, 2016 which is below the "m" in "45 minutes" and is indeed a ~10% jump upwards.

The move to Update3 was July 25, 2016 and I can't connect the big jump there in time. The gap is 2-3 months wide below so I went and looked at other data points for other machines at the same time and it seems like a gradual increase each week. So maybe many small jumps but not one big jump.

About the same for other platforms, Linux and Mac builds are less clear because our build system there has many more random factors (distributed loads, caches) and they also have no single machine that has gone for a long time but I have attached the best I could find, an Opera Mac64 build machine, not the fastest but one with a bit of a history.

There seems to be a strong build time growth from mic/late October 2016 to late November/early December and it wasn't a single moment. Did a series of related changes land during that time or was it just many people adding much code?

@yoav asked about the number of targets. I can probably find something, but it's more work. It would specifically be a bit of work to see if there are more blink files (known to be heavier to build than average files).

GN was mentioned. I don't think it increased the build time but when was it, does it match any change?

nacl was mentioned, we don't use that so that is not a factor in these graphs.

/Daniel

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[PhistucK]

On Tue, Apr 18, 2017 at 7:20 PM, Daniel Bratell <bra...@opera.com> wrote:

There seems to be a strong build time growth from mic/late October 2016 to late November/early December and it wasn't a single moment. Did a series of related changes land during that time or was it just many people adding much code?

​Perhaps using more C++11 syntax increases compile time (like V8 is slower when ECMAScript 2016 syntax is used)?

I could imagine that using auto is more expensive ​than specifying the real type and would increase compile time, but I do not really know what I am talking about. :)

☆PhistucK

[Daniel Bratell]

C+11 features is indeed slower to compile than C++older, but I can't imagine that much of the source code changed that quickly, unless it was in base headers or WTF headers or templates or something else that is compiled into everything.

Or said in another way: I would be surprised if it's just general code changes, but I don't rule out anything.

[Wez]

Which target(s) are the original graph corresponding to?  Is that an "all" build, or just Chrome?

We've done a load of migration e.g. scoped_ptr<> -> std::unique_ptr<> which means pulling in STL headers rather than our own base/ headers, and migration toward std::move() that in some cases means having to replace forward declarations with concrete declarations, again via additional includes. Might that explain some of the creep?

[Daniel Bratell]

These are for the Opera desktop browser but we believe that the growth is happening inside chromium code (and non-Opera people seem to confirm that they see similar numbers).

I don't think anyone knows why. Is the time frame for the increase, mid October to late November 2016, consistent with the smart pointer changes?

/Daniel

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Wez]

On 18 April 2017 at 12:04, Daniel Bratell <bra...@opera.com> wrote:

These are for the Opera desktop browser but we believe that the growth is happening inside chromium code (and non-Opera people seem to confirm that they see similar numbers).

What I meant was how much of the Chromium code are you building?  If you're building both Opera binaries and all the unit-tests, browser-tests, etc, then addition of tests may be a factor, in which case that's expected.  If just the browser binary then we can rule out new test code as a factor.

I don't think anyone knows why. Is the time frame for the increase, mid October to late November 2016, consistent with the smart pointer changes?

Looking at your graph it seemed that there was a steady, but more rapid than was previously the case, increase from Jan 2016 to Jan 2017; unique_ptr<> migration was in-progress in early to mid 2016, IIRC, and since then there has been various other bits of migration work (e.g. greater use of std::move()[1], more use of Mojo rather than Chrome IPC[2]).

Wez

[1] May mean we need to include more headers in some cases?

[2] Mojo IPC requires a mojom->cc/h step, and the GN template has several targets, so each mojom may cost more at build-time than an IPC messages header.

 

/Daniel

On Tue, 18 Apr 2017 19:35:12 +0200, Wez <w...@chromium.org> wrote:

Which target(s) are the original graph corresponding to?  Is that an "all" build, or just Chrome?

We've done a load of migration e.g. scoped_ptr<> -> std::unique_ptr<> which means pulling in STL headers rather than our own base/ headers, and migration toward std::move() that in some cases means having to replace forward declarations with concrete declarations, again via additional includes. Might that explain some of the creep?

On 18 April 2017 at 09:38, Daniel Bratell <bra...@opera.com> wrote:

On Tue, 18 Apr 2017 18:30:10 +0200, PhistucK <phis...@gmail.com> wrote:

On Tue, Apr 18, 2017 at 7:20 PM, Daniel Bratell <bra...@opera.com> wrote:

There seems to be a strong build time growth from mic/late October 2016 to late November/early December and it wasn't a single moment. Did a series of related changes land during that time or was it just many people adding much code?

​Perhaps using more C++11 syntax increases compile time (like V8 is slower when ECMAScript 2016 syntax is used)?

I could imagine that using auto is more expensive ​than specifying the real type and would increase compile time, but I do not really know what I am talking about. :)

C+11 features is indeed slower to compile than C++older, but I can't imagine that much of the source code changed that quickly, unless it was in base headers or WTF headers or templates or something else that is compiled into everything.

Or said in another way: I would be surprised if it's just general code changes, but I don't rule out anything.

/Daniel

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Daniel Bratell]

On Tue, 18 Apr 2017 22:31:44 +0200, Wez <w...@chromium.org> wrote:

On 18 April 2017 at 12:04, Daniel Bratell <bra...@opera.com> wrote:

These are for the Opera desktop browser but we believe that the growth is happening inside chromium code (and non-Opera people seem to confirm that they see similar numbers).

What I meant was how much of the Chromium code are you building?  If you're building both Opera binaries and all the unit-tests, browser-tests, etc, then addition of tests may be a factor, in which case that's expected.  If just the browser binary then we can rule out new test code as a factor.

Ah, I see. It may have changed over the last 3-4 years.

Right now it's a subset of chromium plus opera code plus content_unittests. I don't think the testing code will contribute much to the build time. It would be a surprise at least.

I don't think anyone knows why. Is the time frame for the increase, mid October to late November 2016, consistent with the smart pointer changes?

Looking at your graph it seemed that there was a steady, but more rapid than was previously the case, increase from Jan 2016 to Jan 2017; unique_ptr<> migration was in-progress in early to mid 2016, IIRC, and since then there has been various other bits of migration work (e.g. greater use of std::move()[1], more use of Mojo rather than Chrome IPC[2]).

Thousand papercuts?

[Nico Weber]

I'd expect that to be fairly consistent over time. To me, the graph feels like there's some cause that someone determined enough could discover. 

[Daniel Bratell]

On Wed, 19 Apr 2017 17:11:25 +0200, Nico Weber <tha...@chromium.org> wrote:

I don't think anyone knows why. Is the time frame for the increase, mid October to late November 2016, consistent with the smart pointer changes?

Looking at your graph it seemed that there was a steady, but more rapid than was previously the case, increase from Jan 2016 to Jan 2017; unique_ptr<> migration was in-progress in early to mid 2016, IIRC, and since then there has been various other bits of migration work (e.g. greater use of std::move()[1], more use of Mojo rather than Chrome IPC[2]).

Thousand papercuts?

I'd expect that to be fairly consistent over time. To me, the graph feels like there's some cause that someone determined enough could discover. 

I'm pretty motivated so I have been trying to dig deeper, but not on Mac since I don't have a Mac. What I get is the more I dig, the more like a line it looks, with no jumps. I also looked at the number of targets and it increases at about the same rate but slightly slower. (See attached image). 

All in all it's ending up in +5% compile time and +3% targets in a 2 month span. The Mac change seemed to be bigger, more like +10%, so there might be something Mac specific.  

But one thing I noticed while zooming in as that you lose the big picture. You might land at something that has increased the compile time by a couple of percent, or maybe even 10%, but the total increase since 2013 has been 300-400% and I can't imagine there is anything local that can undo that.