We're removing support for the jumbo build

3335 views
Skip to first unread message

Dirk Pranke

unread,
Aug 15, 2019, 5:25:02 PM8/15/19
to chromium-dev
Hi all,

We have decided to remove support for the jumbo (aka unity) build configuration from the project.

We are very conscious of the fact that Chromium is very expensive and slow to compile, and that that represents a real barrier to contributing.

However, based on the experience we've gained with this configuration over the past many months it's been supported, we've reached the conclusion that the way jumbo is currently implemented, it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary. As a result, we feel that the cost to the project of maintaining jumbo is not worth the benefit it is providing.

We continue to invest in other approaches to reducing build time. We've released the Goma build accelerator code and hope to continue making it easier to use for others. We are also working to enable our Goma clusters for external contributors (anyone who has tryjob access) who don't have their own build acceleration tools. We're starting with Linux support now and will have more to share on that soon. If you are interested in getting access to this system, please fill out this form. We're also working on Clang and LLVM and have made significant improvements this year to cycle time as a result, and we expect to continue to do so for the foreseeable future.

We plan to remove support a week from now, on August 22nd.

Please let me know if you have questions about this.

-- Dirk

Takuto Ikuta

unread,
Aug 15, 2019, 5:32:45 PM8/15/19
to Dirk Pranke, chromium-dev
Hi Dirk,

Do you have tracking issue for this?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTAP4t-XpNe3O4GWbvNq5r%2BHdwObcUvMJ%2BvOQjdbDDYZBQ%40mail.gmail.com.

Lei Zhang

unread,
Aug 15, 2019, 5:51:54 PM8/15/19
to Dirk Pranke, chromium-dev
What's the plan for getting rid of jumbo_source_set and friends?

Dirk Pranke

unread,
Aug 15, 2019, 6:19:04 PM8/15/19
to Takuto Ikuta, chromium-dev

Dirk Pranke

unread,
Aug 15, 2019, 6:20:05 PM8/15/19
to Lei Zhang, chromium-dev
I haven't fully worked out the plan, but first we'll remove the bots, and then most likely go through and turn them all back into the normal targets.

We do not plan to try and explicitly undo any of the code changes that were made for jumbo compatibility, but once the bots are gone, individual devs will be free to go and remove them as they see fit.

-- Dirk 

Ryan Gonzalez

unread,
Aug 15, 2019, 10:37:30 PM8/15/19
to Dirk Pranke, Takuto Ikuta, chromi...@chromium.org
Question, would it be worthwhile to edit the link to the original post here into the issue, so it's easy to see the rationale from a glance?

Daniel Bratell

unread,
Aug 16, 2019, 4:18:32 AM8/16/19
to chromium-dev, dpr...@google.com
I think this is a mistake (and I've communicated as much), because the "interferes too much", seems to be anecdotal evidence from people having access to Google's distributed build system, and there is currently (and not a week from now) no kind of useful alternative for those that want to get involved in the project.

Large companies can work around the extreme compilation times in various way, but this is sending a strong signal of "go away" to everyone else.

I understand that this is Google's prerogrative as they are the ones contributing the vast mass of resources to the project but I still think it is a mistake.

/Daniel
--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTAP4t-XpNe3O4GWbvNq5r%2BHdwObcUvMJ%2BvOQjdbDDYZBQ%40mail.gmail.com.



--
/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

Avi Drissman

unread,
Aug 16, 2019, 12:24:06 PM8/16/19
to Dirk Pranke, chromium-dev
On Thu, Aug 15, 2019 at 5:24 PM 'Dirk Pranke' via Chromium-dev <chromi...@chromium.org> wrote:
we've reached the conclusion that the way jumbo is currently implemented

Is there an alternative implementation that would be better?
 
it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary

Can you give specific examples? I've seen the occasional request to rename conflicting anonymous namespace functions, but I've never found anyone who considered this to be interfering in an unreasonable way.
 
We plan to remove support a week from now, on August 22nd.

That's soon. Is there a pressing need to turn down the jumbo build before everyone is successfully onboarded to Goma?

Avi 

Dirk Pranke

unread,
Aug 16, 2019, 1:59:42 PM8/16/19
to Avi Drissman, chromium-dev
On Fri, Aug 16, 2019 at 9:22 AM Avi Drissman <a...@chromium.org> wrote:
On Thu, Aug 15, 2019 at 5:24 PM 'Dirk Pranke' via Chromium-dev <chromi...@chromium.org> wrote:
we've reached the conclusion that the way jumbo is currently implemented

Is there an alternative implementation that would be better?

I believe that bra...@opera.com and some others had some suggested changes to Clang that might've been able to make things work more transparently, but the Clang team was reluctant to adopt them because it wasn't standard C++ and wasn't sure how important or useful (or perhaps even effective) the changes would've been. Perhaps bratell@ can say more here.

I think there is a hope that in the long run C++ Modules will be the way to generally address the problem that jumbo addresses, but we have no plausible timeline for adopting modules in Chromium that I am aware of, and it's an unproven (and I believe still not-fully-standardized) technology, at least for a cross-platform project like Chromium.
 
 
it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary

Can you give specific examples? I've seen the occasional request to rename conflicting anonymous namespace functions, but I've never found anyone who considered this to be interfering in an unreasonable way.

I don't have specific examples on hand I can point at, but I suspect there are others on the list who could. I believe a common problem that people objected to was header pollution, e.g., a header that was safe when included from one C++ file would get effectively included into another file and cause problems. I think this was/is particularly a problem with third-party headers like zlib or the Windows APIs.

 
We plan to remove support a week from now, on August 22nd.

That's soon. Is there a pressing need to turn down the jumbo build before everyone is successfully onboarded to Goma?

The decision was that Jumbo is and has been an ongoing source of pain, and that we've waited as long as we were willing to wait.

-- Dirk

Mostyn Bramley-Moore

unread,
Aug 16, 2019, 4:17:21 PM8/16/19
to Dirk Pranke, Avi Drissman, chromium-dev
On Fri, Aug 16, 2019 at 7:59 PM 'Dirk Pranke' via Chromium-dev <chromi...@chromium.org> wrote:
On Fri, Aug 16, 2019 at 9:22 AM Avi Drissman <a...@chromium.org> wrote:
On Thu, Aug 15, 2019 at 5:24 PM 'Dirk Pranke' via Chromium-dev <chromi...@chromium.org> wrote:
we've reached the conclusion that the way jumbo is currently implemented

Is there an alternative implementation that would be better?

I believe that bra...@opera.com and some others had some suggested changes to Clang that might've been able to make things work more transparently, but the Clang team was reluctant to adopt them because it wasn't standard C++ and wasn't sure how important or useful (or perhaps even effective) the changes would've been. Perhaps bratell@ can say more here.

Jens Widell (jl@opera) did the heavy lifting on that work. This was the clang-dev thread:

I spoke about how well this worked for a sample Chromium CL in a lightning talk at BlinkOn9, slides here:

Nothing ever came of it, but my feeling was that the Clang team would like C++ modules to take over the world, which would hopefully make jumbo less relevant. But I haven't seen modules used in practice (not even sure if they are standardised yet?), and they are impossible to use in Chromium without buy in from Google.

Ideally I would like C++ to have a standardised file scope for anonymous namespaces, but that seems like an uphill battle that I wouldn't even know how to start.

Note: I don't object to jumbo support being dropped, since we (Vewd Software) have our own Goma implementation and don't get much if any benefit from jumbo anymore.  In theory I agree with Daniel Bratell's view that dropping jumbo support will discourage individual non-Google Chromium contributors, but in practice I think there are several other hurdles that are likely to keep the size of this group extremely small either way.

-Mostyn.

I think there is a hope that in the long run C++ Modules will be the way to generally address the problem that jumbo addresses, but we have no plausible timeline for adopting modules in Chromium that I am aware of, and it's an unproven (and I believe still not-fully-standardized) technology, at least for a cross-platform project like Chromium.
 
 
it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary

Can you give specific examples? I've seen the occasional request to rename conflicting anonymous namespace functions, but I've never found anyone who considered this to be interfering in an unreasonable way.

I don't have specific examples on hand I can point at, but I suspect there are others on the list who could. I believe a common problem that people objected to was header pollution, e.g., a header that was safe when included from one C++ file would get effectively included into another file and cause problems. I think this was/is particularly a problem with third-party headers like zlib or the Windows APIs.

 
We plan to remove support a week from now, on August 22nd.

That's soon. Is there a pressing need to turn down the jumbo build before everyone is successfully onboarded to Goma?

The decision was that Jumbo is and has been an ongoing source of pain, and that we've waited as long as we were willing to wait.

-- Dirk

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

Chris Hamilton

unread,
Aug 16, 2019, 5:01:34 PM8/16/19
to Dirk Pranke, Avi Drissman, chromium-dev
Is there a fast(ish) automated source-to-source translation (that can be built as a Clang plugin) that can address the shortcomings of jumbo-build? Basically, auto renaming of conflicting names? Have your cake (jumbo build) and eat it too (not have to worry about the manual conflict resolution and spooky action at a distance).

Cheers,

Chris

--

Daniel Bratell

unread,
Aug 16, 2019, 5:54:27 PM8/16/19
to Dirk Pranke, Chris Hamilton, Avi Drissman, chromium-dev
I don't think anyone has tried that particular approach, but code maintainers have opinions about names so I don't think that would be easy. That said, I've not found this to be a common issue, and if it's happened, the approach has typically been trivial.

(I've locally maintained an extreme jumbo configuration with as many files grouped together as possible to catch potential problems long before anyone sees them and it's only triggered a problem every now and then and they are typically easy even for someone unfamiliar with the code to resolve.)

/Daniel

Marshall Greenblatt

unread,
Aug 16, 2019, 6:09:07 PM8/16/19
to Dirk Pranke, Lei Zhang, chromium-dev
Hi Dirk,

To get a sense of how this change will impact build time and disk usage I performed a clean build comparison on my development machine (4 core i7 @ 2.8Ghz, fast SSD, 16GB RAM, Windows 10) using a recent version of CEF based on Chromium master. The results were as follows:

Debug + component + jumbo = 22310 steps, 1:38:46.18 (5926.18s total) build time, 33.8GB out directory size.
Debug + component (no jumbo) = 37812 steps, 4:44:31.78 (17071.78s total) build time, 53.3GB out directory size.

Removing jumbo support therefore represents a ~188% increase in clean build time and a ~58% increase in disk usage. Personally, I perform a clean build every few weeks as part of my normal development process (updating the Chromium version), and I don't always have a fast internet connection that would allow me to benefit from Goma. I'm not sure if Goma keeps a local disk cache, but if so I imagine that could become a further issue for people who have smaller SSD drives. Beyond the immediate impact on my own workflows I'm concerned that this change raises the bar to entry for less frequent CEF/Chromium developers who often have older hardware, and that community involvement in these projects will suffer as a result.

I understand that you've concluded that the cost of maintaining jumbo support is too high for Google's developers. It would help me to better understand your perspective if you or someone else could share the data that led to this conclusion. Is this cost inherent to the existence of the jumbo configuration, or more related to fixing breakages that occur from time to time? If the latter then perhaps we can make the configuration community-supported (e.g. those of us that care about it can keep it working) instead of deleting it all together?

Thanks,
Marshall


Christian Biesinger

unread,
Aug 16, 2019, 6:19:48 PM8/16/19
to Dirk Pranke, chromium-dev
On Thu, Aug 15, 2019 at 4:24 PM 'Dirk Pranke' via Chromium-dev
<chromi...@chromium.org> wrote:
> However, based on the experience we've gained with this configuration over the past many months it's been supported, we've reached the conclusion that the way jumbo is currently implemented, it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary. As a result, we feel that the cost to the project of maintaining jumbo is not worth the benefit it is providing.

I don't really understand this decision. Who made the decision? Based
on what data? What is "too much" interference?

Christian

Dirk Pranke

unread,
Aug 16, 2019, 6:46:18 PM8/16/19
to Christian Biesinger, chromium-dev
The decision was made in eng review (i.e., with the people listed in //ENG_REVIEW_OWNERS, along with a couple of other people involved in build-related things, like myself).

We don't have a comprehensive set of data that we can share, but one set of observations was that we'd hit multiple breakages per week due to jumbo-specific issues that devs would have to work around. The general consensus was that the workarounds almost never resulted in better code. 

Again, I can't point to specific CLs here, though perhaps others can. However, I also suspect that we can debate the merits of changes to specific CLs quite a bit and I'm not sure how productive that would be at this point.

-- Dirk 


Christian

Dirk Pranke

unread,
Aug 16, 2019, 6:53:01 PM8/16/19
to Marshall Greenblatt, Lei Zhang, chromium-dev
Hi Marshall,

Thanks for sharing the data. The impact does not surprise me. 

I don't think it's feasible to support jumbo as a community-supported thing on trunk, since the primary perceived problem is that supporting it causes us to write code we don't want to write; i.e., we wouldn't otherwise want to accept CLs making the changes that you'd need to make.

I suppose it's conceivable that people could somehow figure out how to support a fork or a branch that kept jumbo working, but the cost of that seems like it would also be higher than what would be readily justifiable.

-- Dirk

Daniel Bratell

unread,
Aug 17, 2019, 3:31:54 AM8/17/19
to Marshall Greenblatt, 'Dirk Pranke' via Chromium-dev, dpr...@google.com, Lei Zhang
"causes us to write code we don't want to write" is rather vague. The only thing I can think of is that people want to copy functions around and due to jumbo can't do it without renaming said functions and if that is it, I think jumbo is doing us a favor. Code should preferably be at one place, not many places.

/Daniel

Yoav Weiss

unread,
Aug 17, 2019, 4:25:45 AM8/17/19
to Dirk Pranke, Avi Drissman, chromium-dev
As I believe this is the first time this is being discussed in the wider forum of chromium-dev, 1 week notice before taking jumbo builds down seems awfully short. Do we have large amounts of data indicating that it is actively harming project members that are not using jumbo, that would justify such urgency?

If the answer to the above is no, could we make sure that the deprecation timelines are extended to a point in time where reasonable alternatives are available?


-- Dirk

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

Jan van Oort

unread,
Aug 17, 2019, 4:30:44 AM8/17/19
to Yoav Weiss, Dirk Pranke, Avi Drissman, chromium-dev
Agree with Yoav Weiss. Deprecation timelines in major open-source projects should give participants a reasonable delay to look around for options.‎ Often such delays have an order of magnitude of months; one week seems unreasonably short.

Jan van Oort
Head of Research
KIVU Technologies GmbH 
www.kivu.tech
From: Yoav Weiss
Sent: Samstag, 17. August 2019 10:25
To: Dirk Pranke
Reply To: yo...@yoav.ws
Cc: Avi Drissman; chromium-dev
Subject: Re: [chromium-dev] We're removing support for the jumbo build

Dirk Pranke

unread,
Aug 18, 2019, 3:34:04 PM8/18/19
to Ryan Gonzalez, Takuto Ikuta, chromium-dev
Good suggestion; done.

-- Dirk

Matt Giuca

unread,
Aug 18, 2019, 8:58:46 PM8/18/19
to Dirk Pranke, Ryan Gonzalez, Takuto Ikuta, chromium-dev, Daniel Bratell
I think this was the right decision but I agree it would have been good to justify it with data.

To be clear (since it hasn't been stated explicitly in this thread), the breakages that devs have to workaround are usually a name in an anonymous namespace in file A, that conflicts with the same name in an anonymous namespace in file B. This means you have to rename them to not have the same name, effectively defeating the entire point of anonymous namespaces.

To find some relevant data, I went looking through the commit log for commits with "jumbo" in the description. More often than not, it's bratell@ fixing things.

Here's an example of where it just makes the code (slightly) worse:

We had a couple of perfectly reasonable (in the local file context) names in an anonymous namespace, "kConfigBinaryPbFileName" and "LoadProtoFromDisk", which conflicted with something else with the same name in a different file. The fix was that these were renamed to "kSafetyTipsConfigBinaryPbFileName" and "LoadSafetyTipsProtoFromDisk". I think that qualifies as "code we don't want to write": we use namespaces in C++ so that we can have short names that give just enough detail for the context they're in, but don't need to write over-long names that are globally unique, which is how things are typically named in C code. It's "code smell" if you start to see variables and functions all including the name of the file in their own name.

Daniel Bratell wrote:
> The only thing I can think of is that people want to copy functions around and due to jumbo can't do it without renaming said functions and if that is it, I think jumbo is doing us a favor. Code should preferably be at one place, not many places.

True, but having two functions with the same name (in a project as large as Chromium or even a component thereof) does not necessarily mean they are duplicated. "LoadProtoFromDisk" is a good example of a name that could have different meanings in different contexts. The name is perfectly clear as it is, in the context of this file; disambiguating it globally is just adding noise.

Here's another example where you actually detected a duplicated function:

"DoesAndroidSupportMaskableIcons" was duplicated in an anonymous namespace in two different files, which is a potential for a bug later down the line. In this case, Jumbo is helping us detect dupes, but I'll point out that the fix for this was to just rename the two functions to "DoesAndroidSupportMaskableIconsForWebApk" and "DoesAndroidSupportMaskableIconsForHomescreen", which IMO makes the situation worse, because we still have duplicated code, but now we're implying to future editors that there's possibly a difference between "supporting maskable icons for WebAPK" and "supporting maskable icons for add-to-homescreen" (when in reality, these names were just chosen as a quick way to avoid a naming conflict to satisfy Jumbo builds).

I'm not trying to criticize you personally in relation to this particular CL, but just point out that overall, having a process that forces us to name variables to be globally unique may be forcing us to make suboptimal naming decisions that can actually affect the way future readers think about the code.

>>> import this
The Zen of Python, by Tim Peters
...
Namespaces are one honking great idea -- let's do more of those!


Daniel Bratell

unread,
Aug 19, 2019, 3:24:56 AM8/19/19
to Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
So using my jumbo commits is not the best example of how jumbo causes problems to the project since most of my commits are preventive, fixing issues that turn up in the most extreme jumbo configuration (jumbo chunk size = infinity) which nobody actually uses. If my commits is the majority, then there is no real problem.

What you want to find is a lot of CL:s with jumbo-only compile problems. The project has a couple of thousand CLs per week, if jumbo problems are plentiful or common, you'd expect dozens or hundreds of these and I remember seeing about three the last year. And I've been looking. You would also expect the non-CQ jumbo bots to constantly fail since they cover more than the CQ bots, and they fail occasionally but not all the time.

If you go through all failures of the CQ jumbo bot (which fails much less than other bots since it does much less) you mostly see it failing in tree wide breakages or when a CL contains code that doesn't compile in equivalant bots without jumbo, not because of jumbo.

My point is that even if what you say is true, some functions and constants end up getting names that wouldn't be your first choice, the data I have available indicates that it's overall a tiny problem. And there doesn't seem to be any public data that contradicts that. Had there been data that indicated a problem we could have acted on it (for instance if a certain compile target experienced this a lot, it could have had jumbo support removed). 

(I do agree that the function names you mention were not good and if you look through all of my commits I'm sure you can find more examples of functions names that could have been better, but where nobody could think of a better name at a time).

/Daniel

Gabriel Charette

unread,
Aug 19, 2019, 8:53:17 AM8/19/19
to Daniel Bratell, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
As a Googler with access to Goma I still make regular use of the jumbo build when compiling from my laptop. It's the only way to have a reasonable amount of targets to compile for small test suites (I pretty much never compile all of Chromium there).

From personal experience in the last year I don't think I've seen a single jumbo related conflict neither in a CL I've written nor reviewed.

The only naming conflict I've had to deal with was caused by regular C++ header pollution (a unittest.cc file in //base had a global named "data" which conflicted with base::data() when stl_util.h ended up in the transitive closure of that unittest.cc per an unrelated refactoring). Just like the occasional jumbo errors, it was easily fixed by renaming the global variable.

I agree with bratell@ that him being the main source of commits to address jumbo issues seems to indicate that there isn't much of a problem in practice for the average developer?

Alexei Svitkine

unread,
Aug 19, 2019, 9:08:00 AM8/19/19
to Gabriel Charette, Daniel Bratell, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
I've also found the jumbo build helpful. Sometimes I work on a plane where I don't have access to Goma and only have my MacBook pro laptop. It's helpful to have faster compile times in this situation.

Alex Clarke

unread,
Aug 19, 2019, 11:46:12 AM8/19/19
to Alexei Svitkine, Gabriel Charette, Daniel Bratell, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
I've also found jumbo builds useful on occasion and I can't remember the last time if at all jumbo builds caused me any problems.  This decision feels like a mistake.

Peter Kasting

unread,
Aug 19, 2019, 12:07:20 PM8/19/19
to Alex Clarke, Alexei Svitkine, Gabriel Charette, Daniel Bratell, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
+1 to what Gab and Alex said.

It feels like this may have been a well-intentioned decision made with misleading anecdata.  If that's possible, I'd love for the eng leads to reconsider.

PK

Ken Rockot

unread,
Aug 19, 2019, 12:08:36 PM8/19/19
to Alex Clarke, Alexei Svitkine, Gabriel Charette, Daniel Bratell, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
I've tripped over jumbo many times. I'm glad we're removing it, though I wish we could have supported it with some help from clang. The current approach in principle feels like a hack., and in practice it means I've had to deal with naming collisions among unrelated components much more often than I would like.

Mason Freed

unread,
Aug 19, 2019, 12:22:20 PM8/19/19
to roc...@google.com, Alex Clarke, Alexei Svitkine, Gabriel Charette, Daniel Bratell, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
This is likely a naive suggestion, but is there any way to just append (on the fly, during jumbo compilation, in python) a hash of the translation unit name to the name of all objects defined inside an anonymous namespace? That sounds like it would eliminate the primary complaint here of name collisions.

David Benjamin

unread,
Aug 19, 2019, 12:30:12 PM8/19/19
to mason...@chromium.org, Ken Rockot, Alex Clarke, Alexei Svitkine, Gabriel Charette, Daniel Bratell, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
That seems like the cleanest option to me, though a clang plugin may be better for an accurate parse. That way jumbo doesn't break developer expectations for C++. We can probably make the transformation even simpler with inline namespaces:

1. Transform namespace { ... } to namespace { inline namespace <hash-of-filename> { ... } }
2. Transform static foo ... to inline namespace <hash-of-filename> { static foo ... }

That way we don't need to go find the places a name is referenced or risk oddities around argument-dependent lookup.

Erik Staab

unread,
Aug 19, 2019, 12:43:54 PM8/19/19
to Peter Kasting, Alex Clarke, Alexei Svitkine, Gabriel Charette, Daniel Bratell, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
On Mon, Aug 19, 2019 at 12:07 PM 'Peter Kasting' via Chromium-dev <chromi...@chromium.org> wrote:
+1 to what Gab and Alex said.

It feels like this may have been a well-intentioned decision made with misleading anecdata.  If that's possible, I'd love for the eng leads to reconsider.

I think it would be hard to get good data and quantify the impact but I made an attempt when we started discussing costs of jumbo support. I looked at the number of non-bot references to jumbo in CL descriptions and comments over a six month period and found that there were 254 CLs with either. Many of the CL comments were people uploading new patchsets when they realized there was a symbol collision. I'm not sure how much time is lost when people need to do this (maybe a CQ cycle? so 50 minutes at the 50th %ile?) but maybe others can estimate.

Here is a spreadsheet with the data if you want to look through yourself:

Also, we're still open to helping people get set up with the Goma server we open-sourced at the last BlinkOn. infr...@chromium.org is a good list to reach the team or #goma on slack sometimes.

Erik
 

Samuel Attard

unread,
Aug 19, 2019, 12:47:35 PM8/19/19
to Chromium-dev, rym...@gmail.com, tik...@chromium.org
Just to add to what other folks have said in this thread the 1 week timeline is incredibly short notice.  We have been experimenting with the open sourced version of Goma for months and have constantly run into performance issues that we've been working through with the Goma / RBE folks but in it's current state it is not ready to "replace" the performance benefits that we got through the jumbo build.  (For googlers you might be able to see this thread --> https://issuetracker.google.com/issues/136104063)

I'm currently spinning up some CI runs to measure the exact build time cost of disabling jumbo (and that build time cost will directly translate into monetary cost for anyone running their own CI infrastructure).  I hope to share some specific numbers next week but initial estimates from when I was experimenting with goma are around 2-3x slower, similar to Marshall's numbers above.

So far in this thread I see a lot of "jumbo is bad, we're removing a bad thing" but as others have pointed out a lot of us are missing the context / metrics for why jumbo is bad especially when we fundamentally rely on it to get Chromium build times down from "insane" to "kind of reasonable".  This decision from Google's side makes sense because y'all have a massive set of goma infrastructure, but the rest of do not and I think this impact was not taken into consideration or dramatically underestimated.
To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

Scott Violet

unread,
Aug 19, 2019, 12:56:43 PM8/19/19
to Ken Rockot, Alex Clarke, Alexei Svitkine, Gabriel Charette, Daniel Bratell, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
I have also tripped over the jumbo build a number of times. Beyond the simple name changes, the biggest source of pain I've encountered is in template specialization, and in particular the ClassProperty related ones. See  https://chromium-review.googlesource.com/c/chromium/src/+/786190/ for more information. If you get the macro order wrong, the compiler errors are rather inscrutable.

  -Scott

Dirk Pranke

unread,
Aug 19, 2019, 1:43:56 PM8/19/19
to Jan van Oort, Yoav Weiss, Avi Drissman, chromium-dev
As far as the one-week notice being too short goes, I think that's fair criticism and I apologize.

I set a short deadline because I didn't think having a timeline for investigating alternatives would be all that useful, since there's not an obvious alternative to what jumbo does, particularly if you can't use a compile farm like Goma (e.g., because you're working offline or on a slow or metered network). 

However, I suppose it could be useful to keep jumbo around for a time while people helped contribute to Goma to make it be more useful in other environments (either to set up their own goma clusters or to make our Goma/RBE-based service easier to use), so perhaps I was mistaken in this thinking, too.

-- Dirk



As others have commented, something like Goma acceleration can be a good alternative in some cases, but if you can use Goma you'd likely be investigating it already and jumbo actually makes goma slower. If you can't use Goma (e.g., because you need to work offline or have a network connection that is too slow, metered, or whatever), I don't know of another alternative to jumbo.

-- Dirk


Morten Stenshorne

unread,
Aug 19, 2019, 3:02:26 PM8/19/19
to 'Dirk Pranke' via Chromium-dev, Jan van Oort, dpr...@google.com, Yoav Weiss, Avi Drissman
"'Dirk Pranke' via Chromium-dev" <chromi...@chromium.org> writes:

> As others have commented, something like Goma acceleration can be a
> good alternative in some cases, but if you can use Goma you'd likely
> be investigating it already and jumbo actually makes goma slower.

That depends. It's true for my Linux workstation, but on my MacBook pro
it takes:

just goma: about 25 mins
just jumbo: about an hour
goma + jumbo: less than 15 mins

I disable goma when I'm on a slow/flaky/metered network [1] or no
network at all [2].

[1] Cafés, hotels, airports, cabins, remote villages
[2] Airplanes

I also sometimes run with just jumbo + ccache because the huge number of
translation units (probably the reason?) seems to affect the performance
of various tools (linkers and such, I guess?), so that jumbo + ccache is
way faster when rebuilding a few translation units and everything hits
the cache (like 6 seconds instead of 30-60 seconds).

I'd very much like to see the need for jumbo eliminated before it's
removed.

--
Morten Stenshorne, Software developer,
Blink/Layout, Google, Oslo, Norway

Dirk Pranke

unread,
Aug 19, 2019, 3:29:23 PM8/19/19
to Morten Stenshorne, 'Dirk Pranke' via Chromium-dev, Jan van Oort, Yoav Weiss, Avi Drissman
That's interesting. I don't recall us seeing results before that showed jumbo + goma being faster than jumbo alone, when we'd done testing (and I don't think we've seen that on the CQ bots, either, but we could re-check).

I believe that jumbo does make linking faster due to the fewer TU's and less need for code stripping, as you say.

-- Dirk

Yoav Weiss

unread,
Aug 19, 2019, 3:43:01 PM8/19/19
to Dirk Pranke, Jan van Oort, Avi Drissman, chromium-dev
On Mon, Aug 19, 2019 at 6:42 PM Dirk Pranke <dpr...@google.com> wrote:
As far as the one-week notice being too short goes, I think that's fair criticism and I apologize.

I set a short deadline because I didn't think having a timeline for investigating alternatives would be all that useful, since there's not an obvious alternative to what jumbo does, particularly if you can't use a compile farm like Goma (e.g., because you're working offline or on a slow or metered network). 

However, I suppose it could be useful to keep jumbo around for a time while people helped contribute to Goma to make it be more useful in other environments (either to set up their own goma clusters or to make our Goma/RBE-based service easier to use), so perhaps I was mistaken in this thinking, too.

Thanks for listening. Highly appreciated! :)

I think there's another angle to consider when talking about Goma vs. jumbo - sustainability.
As jumbo builds require ~3x less CPU power, applying (something like) them to everyone could have a significant impact on reducing our GCP bill (disclaimer: I don't know if the Chrome team is actually charged and if that argument actually makes sense for us. It definitely makes sense for other orgs).
It would also mean that the project's CO2 footprint would be significantly smaller, at least when it comes to computing infrastructure.

Maybe those latter arguments could be a motivating factor to work on a jumbo equivalent that doesn't suffer from its downsides:
* Doesn't result in anonymous namespace collisions that require manual intervention.
* Doesn't result in significantly slower single .cc file change build times (my anecdotal experience, on which I have no data). There are probably tradeoffs here we'd need to figure out.
* Doesn't result in slower Goma builds in some OSes. 

Dirk Pranke

unread,
Aug 19, 2019, 3:55:52 PM8/19/19
to Yoav Weiss, Jan van Oort, Avi Drissman, chromium-dev
On Mon, Aug 19, 2019 at 12:41 PM Yoav Weiss <yo...@yoav.ws> wrote:


On Mon, Aug 19, 2019 at 6:42 PM Dirk Pranke <dpr...@google.com> wrote:
As far as the one-week notice being too short goes, I think that's fair criticism and I apologize.

I set a short deadline because I didn't think having a timeline for investigating alternatives would be all that useful, since there's not an obvious alternative to what jumbo does, particularly if you can't use a compile farm like Goma (e.g., because you're working offline or on a slow or metered network). 

However, I suppose it could be useful to keep jumbo around for a time while people helped contribute to Goma to make it be more useful in other environments (either to set up their own goma clusters or to make our Goma/RBE-based service easier to use), so perhaps I was mistaken in this thinking, too.

Thanks for listening. Highly appreciated! :)

I think there's another angle to consider when talking about Goma vs. jumbo - sustainability.
As jumbo builds require ~3x less CPU power, applying (something like) them to everyone could have a significant impact on reducing our GCP bill (disclaimer: I don't know if the Chrome team is actually charged and if that argument actually makes sense for us. It definitely makes sense for other orgs).
It would also mean that the project's CO2 footprint would be significantly smaller, at least when it comes to computing infrastructure.

Indeed, this was one of the primary reasons I was initially interested in jumbo. 

However, jumbo appears to reduce the amount we're able to reuse cache hits in goma, so it seems like we don't see as much of a benefit as I had initially hoped for. It's hard to get an apples-to-apples comparison, though, since it's hard to get wide-scale experiments with equivalent traffic. It's also harder to get comparable numbers out due to the ongoing migration to the RBE-based Goma instance, where we have much less insight into the costs compared to when we were running our own servers.

-- Dirk

Daniel Bratell

unread,
Aug 20, 2019, 4:32:49 AM8/20/19
to mason...@chromium.org, David Benjamin, Ken Rockot, Alex Clarke, Alexei Svitkine, Gabriel Charette, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
We have suggested such schemes (see Mostyn's post earlier in the thread) but they have not materialized, mainly because they would have required changes in clang. Small changes, but non-standard C++ changes, and they were not accepted.

/Daniel

Jan van Oort

unread,
Aug 20, 2019, 4:38:06 AM8/20/19
to Dirk Pranke, Yoav Weiss, Avi Drissman, chromium-dev
Kudos for listening :-) 

Daniel Bratell

unread,
Aug 20, 2019, 4:48:45 AM8/20/19
to Ken Rockot, Scott Violet, Alex Clarke, Alexei Svitkine, Gabriel Charette, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
On Mon, 19 Aug 2019 18:54:16 +0200, Scott Violet <s...@chromium.org> wrote:

I have also tripped over the jumbo build a number of times. Beyond the simple name changes, the biggest source of pain I've encountered is in template specialization, and in particular the ClassProperty related ones. See  https://chromium-review.googlesource.com/c/chromium/src/+/786190/ for more information. If you get the macro order wrong, the compiler errors are rather inscrutable.

Indeed. Took me a while to even figure out what was going on.

This is illegal in C++:

template <typename T> int calc(T x);
int do_calc(int x) {return calc(x);}
template <> int calc(int x) { return x / 2; }

../testtemplate.cc:3:27: error: specialization of ‘int calc(T) [with T = int]’ after instantiation
template <> int calc(int x) { return x / 2; }

What it's basically saying is that by providing the specialization after the function was used, it might have ended up using the wrong template instantiation. Assume all three of those lines were provided in different files, then the compiler might not know about the specialization and everything compiles fine. With jumbo, the compiler suddenly knew about the specialization and we got those errors, and they are not well understood.

Sometimes this error message will prevent bugs, but if there is no default implementation of the template, then the actual "instantiation" won't happen until linking time where the linker will probably choose the right implementation.

Should we be happy jumbo uncovered this or be annoyed by it? I feel like the compiler could have skipped the error in some circumstances so it's both helpful and annoying depending on circumstance. And when the actual lines are hidden by macros (as in ClassProperty), that compiler error can really seem hard to understand.

/Daniel

Daniel Bratell

unread,
Aug 20, 2019, 5:23:03 AM8/20/19
to Peter Kasting, Erik Staab, Alex Clarke, Alexei Svitkine, Gabriel Charette, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
On Mon, 19 Aug 2019 18:42:23 +0200, Erik Staab <est...@chromium.org> wrote:

On Mon, Aug 19, 2019 at 12:07 PM 'Peter Kasting' via Chromium-dev <chromi...@chromium.org> wrote:
+1 to what Gab and Alex said.

It feels like this may have been a well-intentioned decision made with misleading anecdata.  If that's possible, I'd love for the eng leads to reconsider.

I think it would be hard to get good data and quantify the impact but I made an attempt when we started discussing costs of jumbo support. I looked at the number of non-bot references to jumbo in CL descriptions and comments over a six month period and found that there were 254 CLs with either. Many of the CL comments were people uploading new patchsets when they realized there was a symbol collision. I'm not sure how much time is lost when people need to do this (maybe a CQ cycle? so 50 minutes at the 50th %ile?) but maybe others can estimate.

Here is a spreadsheet with the data if you want to look through yourself:

254 patch uploads over 6 months that mention jumbo. I recognize a number of them being things I've been working on where I've mentioned jumbo in a comment but which hasn't been a jumbo breakage so I will have to see if I can get some feeling for how many of these are real, though it seems the error logs might have been removed/archived[1]. One factor that I've seen in the past that makes it hard to judge the problem is that sometimes the jumbo builder compiles first so it's mentioned in the patch set with the fix. And of course there is no guarantee someone mentions "jumbo" when fixing a jumbo problem so the error margin goes both ways.

Do you have some context of how many patch uploads in total there were over this time? 

/Daniel

Daniel Bratell

unread,
Aug 20, 2019, 5:48:33 AM8/20/19
to Peter Kasting, Erik Staab, Alex Clarke, Alexei Svitkine, Gabriel Charette, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
I went through a bit of the list. Lines n*15 + 2 to be specific, and only in the lists that were expanded (not sure why there were 250 entries in the link list but only 145 in the expanded one).

Of those:

3 were about variable shadowing where I mentioned in the commit message that I hoped it might prevent jumbo problems.

2 were anonymous namespace jumbo problem that resulted in renames. (Potentially worse names)

1 was a  anonymous namespace jumbo problem that resulted in more optimized code.

1 was an anonymous namespace jumbo problem because the wrong name was used.

1 was ambiguity between shadowed namespaces (::switches, ::foo::switches) which happens in all kinds of builds but more in jumbo builds and is a construct that is not allowed in the code style guide.

1 was me doing preventive code cleanup to avoid future problems.

1 was a revert of a jumbo fix because of concurrent code changes landing, breaking the build.

----

So 2 out of 10, or 20% of the samples, were of the kind that we don't like, where the result might be worse code (longer, worse, uglier, different symbol names).

There is a large error margin on this, but the sampling has confirmed my belief that the perceived problem ("jumbo makes us write worse code") is much smaller than the people making this decision thought.

I get it that jumbo is not a zero cost feature, but it's also not a zero benefit feature. How much time did it save during these six months? How much more work might have been done because of it? We have no data on that either, but it ought to be much more than what it cost.

/Daniel

David Benjamin

unread,
Aug 20, 2019, 9:49:57 AM8/20/19
to Daniel Bratell, mason...@chromium.org, Ken Rockot, Alex Clarke, Alexei Svitkine, Gabriel Charette, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
I think you misunderstand me. The suggestion is to build a tool external to clang which performs this transformation as part of the jumbo build process. I.e. transform the source files and then pass the transformed files into the compiler. That way there isn't any non-standard C++ involved, either on the compiler's side or the programmer's side. The hacks are left to the jumbo machinery itself.

David Benjamin

unread,
Aug 20, 2019, 9:55:12 AM8/20/19
to Daniel Bratell, mason...@chromium.org, Ken Rockot, Alex Clarke, Alexei Svitkine, Gabriel Charette, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
Oh, but the particular transformation I suggested wouldn't actually work because the call sites are ambiguous. Still, Mason's version should be possible.

Daniel Bratell

unread,
Aug 20, 2019, 10:06:47 AM8/20/19
to David Benjamin, mason...@chromium.org, Ken Rockot, Alex Clarke, Alexei Svitkine, Gabriel Charette, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
Ah, not sure anyone has tried that one (transforming all "global" symbol names in the anonymous namespace before sending the text to the compiler). Maybe it could work, though there would be a lot of edge cases. If someone wants to try it, please add me to the discussion because I know a bit about where the rough edges are.

/Daniel

Chris Hamilton

unread,
Aug 20, 2019, 10:08:10 AM8/20/19
to David Benjamin, Daniel Bratell, mason...@chromium.org, Ken Rockot, Alex Clarke, Alexei Svitkine, Gabriel Charette, Dirk Pranke, Matt Giuca, Ryan Gonzalez, Takuto Ikuta, chromium-dev
This is pretty much what I was suggesting earlier (a source-to-source Clang plugin that runs as part of the jumbo build process). You don't need to actually upstream any compiler support that way.

On Tue, Aug 20, 2019 at 9:48 AM David Benjamin <davi...@chromium.org> wrote:

Nicolas Ouellet-Payeur

unread,
Aug 20, 2019, 10:08:41 AM8/20/19
to Chromium-dev, bra...@opera.com, mason...@chromium.org, roc...@google.com, alexc...@google.com, asvi...@chromium.org, g...@chromium.org, dpr...@google.com, mgi...@chromium.org, rym...@gmail.com, tik...@chromium.org
We'd need to be careful with how we mangle the symbol names to dedupe them.

If we change the symbols naively (e.g. with a regexp from a python script),
then we could miss some call-sites and fail to compile valid code.

If we change them by parsing the source code (e.g. with a clang plugin),
then we could lose the benefits of jumbo wrt build time.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.
--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.
--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.
--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.



--
/* Opera Software, Linköping, Sweden: CEST (UTC+2) */
--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.
--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.
--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.
--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.
--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.
--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.