We're removing support for the jumbo build

[Dirk Pranke]

Hi all,

We have decided to remove support for the jumbo (aka unity) build configuration from the project.

We are very conscious of the fact that Chromium is very expensive and slow to compile, and that that represents a real barrier to contributing.

However, based on the experience we've gained with this configuration over the past many months it's been supported, we've reached the conclusion that the way jumbo is currently implemented, it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary. As a result, we feel that the cost to the project of maintaining jumbo is not worth the benefit it is providing.

We continue to invest in other approaches to reducing build time. We've released the Goma build accelerator code and hope to continue making it easier to use for others. We are also working to enable our Goma clusters for external contributors (anyone who has tryjob access) who don't have their own build acceleration tools. We're starting with Linux support now and will have more to share on that soon. If you are interested in getting access to this system, please fill out this form. We're also working on Clang and LLVM and have made significant improvements this year to cycle time as a result, and we expect to continue to do so for the foreseeable future.

We plan to remove support a week from now, on August 22nd.

Please let me know if you have questions about this.

-- Dirk

[Takuto Ikuta]

Hi Dirk,

Do you have tracking issue for this?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTAP4t-XpNe3O4GWbvNq5r%2BHdwObcUvMJ%2BvOQjdbDDYZBQ%40mail.gmail.com.

[Lei Zhang]

What's the plan for getting rid of jumbo_source_set and friends?

[Dirk Pranke]

I do now :). https://bugs.chromium.org/p/chromium/issues/detail?id=994387

-- Dirk

[Dirk Pranke]

I haven't fully worked out the plan, but first we'll remove the bots, and then most likely go through and turn them all back into the normal targets.

We do not plan to try and explicitly undo any of the code changes that were made for jumbo compatibility, but once the bots are gone, individual devs will be free to go and remove them as they see fit.

-- Dirk 

[Ryan Gonzalez]

Question, would it be worthwhile to edit the link to the original post here into the issue, so it's easy to see the rationale from a glance?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTBrDJq6EVRND5E%3D1wuvEPjXKyOv9qOPZ8%3DMP2%2BJOvYAFQ%40mail.gmail.com.

[Daniel Bratell]

I think this is a mistake (and I've communicated as much), because the "interferes too much", seems to be anecdotal evidence from people having access to Google's distributed build system, and there is currently (and not a week from now) no kind of useful alternative for those that want to get involved in the project.

Large companies can work around the extreme compilation times in various way, but this is sending a strong signal of "go away" to everyone else.

I understand that this is Google's prerogrative as they are the ones contributing the vast mass of resources to the project but I still think it is a mistake.

/Daniel

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTAP4t-XpNe3O4GWbvNq5r%2BHdwObcUvMJ%2BvOQjdbDDYZBQ%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

[Avi Drissman]

On Thu, Aug 15, 2019 at 5:24 PM 'Dirk Pranke' via Chromium-dev <chromi...@chromium.org> wrote:

we've reached the conclusion that the way jumbo is currently implemented

Is there an alternative implementation that would be better?

 

it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary

Can you give specific examples? I've seen the occasional request to rename conflicting anonymous namespace functions, but I've never found anyone who considered this to be interfering in an unreasonable way.

 

We plan to remove support a week from now, on August 22nd.

That's soon. Is there a pressing need to turn down the jumbo build before everyone is successfully onboarded to Goma?

Avi 

[Dirk Pranke]

On Fri, Aug 16, 2019 at 9:22 AM Avi Drissman <a...@chromium.org> wrote:

On Thu, Aug 15, 2019 at 5:24 PM 'Dirk Pranke' via Chromium-dev <chromi...@chromium.org> wrote:

we've reached the conclusion that the way jumbo is currently implemented

Is there an alternative implementation that would be better?

I believe that bra...@opera.com and some others had some suggested changes to Clang that might've been able to make things work more transparently, but the Clang team was reluctant to adopt them because it wasn't standard C++ and wasn't sure how important or useful (or perhaps even effective) the changes would've been. Perhaps bratell@ can say more here.

I think there is a hope that in the long run C++ Modules will be the way to generally address the problem that jumbo addresses, but we have no plausible timeline for adopting modules in Chromium that I am aware of, and it's an unproven (and I believe still not-fully-standardized) technology, at least for a cross-platform project like Chromium.

 

 

it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary

Can you give specific examples? I've seen the occasional request to rename conflicting anonymous namespace functions, but I've never found anyone who considered this to be interfering in an unreasonable way.

I don't have specific examples on hand I can point at, but I suspect there are others on the list who could. I believe a common problem that people objected to was header pollution, e.g., a header that was safe when included from one C++ file would get effectively included into another file and cause problems. I think this was/is particularly a problem with third-party headers like zlib or the Windows APIs.

 

We plan to remove support a week from now, on August 22nd.

That's soon. Is there a pressing need to turn down the jumbo build before everyone is successfully onboarded to Goma?

The decision was that Jumbo is and has been an ongoing source of pain, and that we've waited as long as we were willing to wait.

-- Dirk

[Mostyn Bramley-Moore]

On Fri, Aug 16, 2019 at 7:59 PM 'Dirk Pranke' via Chromium-dev <chromi...@chromium.org> wrote:

On Fri, Aug 16, 2019 at 9:22 AM Avi Drissman <a...@chromium.org> wrote:

On Thu, Aug 15, 2019 at 5:24 PM 'Dirk Pranke' via Chromium-dev <chromi...@chromium.org> wrote:

we've reached the conclusion that the way jumbo is currently implemented

Is there an alternative implementation that would be better?

I believe that bra...@opera.com and some others had some suggested changes to Clang that might've been able to make things work more transparently, but the Clang team was reluctant to adopt them because it wasn't standard C++ and wasn't sure how important or useful (or perhaps even effective) the changes would've been. Perhaps bratell@ can say more here.

Jens Widell (jl@opera) did the heavy lifting on that work. This was the clang-dev thread:

http://lists.llvm.org/pipermail/cfe-dev/2018-April/thread.html#57579

I spoke about how well this worked for a sample Chromium CL in a lightning talk at BlinkOn9, slides here:

https://docs.google.com/presentation/d/1tOV4dZ5yCo7tJJKu7fN2oS3jYVsshOGTD81UxqD6sbU/edit#slide=id.p

Nothing ever came of it, but my feeling was that the Clang team would like C++ modules to take over the world, which would hopefully make jumbo less relevant. But I haven't seen modules used in practice (not even sure if they are standardised yet?), and they are impossible to use in Chromium without buy in from Google.

Ideally I would like C++ to have a standardised file scope for anonymous namespaces, but that seems like an uphill battle that I wouldn't even know how to start.

Note: I don't object to jumbo support being dropped, since we (Vewd Software) have our own Goma implementation and don't get much if any benefit from jumbo anymore.  In theory I agree with Daniel Bratell's view that dropping jumbo support will discourage individual non-Google Chromium contributors, but in practice I think there are several other hurdles that are likely to keep the size of this group extremely small either way.

-Mostyn.

I think there is a hope that in the long run C++ Modules will be the way to generally address the problem that jumbo addresses, but we have no plausible timeline for adopting modules in Chromium that I am aware of, and it's an unproven (and I believe still not-fully-standardized) technology, at least for a cross-platform project like Chromium.

 

 

it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary

Can you give specific examples? I've seen the occasional request to rename conflicting anonymous namespace functions, but I've never found anyone who considered this to be interfering in an unreasonable way.

I don't have specific examples on hand I can point at, but I suspect there are others on the list who could. I believe a common problem that people objected to was header pollution, e.g., a header that was safe when included from one C++ file would get effectively included into another file and cause problems. I think this was/is particularly a problem with third-party headers like zlib or the Windows APIs.

 

We plan to remove support a week from now, on August 22nd.

That's soon. Is there a pressing need to turn down the jumbo build before everyone is successfully onboarded to Goma?

The decision was that Jumbo is and has been an ongoing source of pain, and that we've waited as long as we were willing to wait.

-- Dirk

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTCMjv6emtxSXt7kFrT5Mjga0QwymD6Md176_45R_59msg%40mail.gmail.com.

--

Mostyn Bramley-Moore

Vewd Software

mos...@vewd.com

[Chris Hamilton]

Is there a fast(ish) automated source-to-source translation (that can be built as a Clang plugin) that can address the shortcomings of jumbo-build? Basically, auto renaming of conflicting names? Have your cake (jumbo build) and eat it too (not have to worry about the manual conflict resolution and spooky action at a distance).

Cheers,

Chris

--

[Daniel Bratell]

I don't think anyone has tried that particular approach, but code maintainers have opinions about names so I don't think that would be easy. That said, I've not found this to be a common issue, and if it's happened, the approach has typically been trivial.

(I've locally maintained an extreme jumbo configuration with as many files grouped together as possible to catch potential problems long before anyone sees them and it's only triggered a problem every now and then and they are typically easy even for someone unfamiliar with the code to resolve.)

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAA2pCLd1gvt8wBFYNSD49gkqB87FQYeKkNoNKbxWo3FeXJk6FA%40mail.gmail.com.

[Marshall Greenblatt]

Hi Dirk,

To get a sense of how this change will impact build time and disk usage I performed a clean build comparison on my development machine (4 core i7 @ 2.8Ghz, fast SSD, 16GB RAM, Windows 10) using a recent version of CEF based on Chromium master. The results were as follows:

Debug + component + jumbo = 22310 steps, 1:38:46.18 (5926.18s total) build time, 33.8GB out directory size.

Debug + component (no jumbo) = 37812 steps, 4:44:31.78 (17071.78s total) build time, 53.3GB out directory size.

Removing jumbo support therefore represents a ~188% increase in clean build time and a ~58% increase in disk usage. Personally, I perform a clean build every few weeks as part of my normal development process (updating the Chromium version), and I don't always have a fast internet connection that would allow me to benefit from Goma. I'm not sure if Goma keeps a local disk cache, but if so I imagine that could become a further issue for people who have smaller SSD drives. Beyond the immediate impact on my own workflows I'm concerned that this change raises the bar to entry for less frequent CEF/Chromium developers who often have older hardware, and that community involvement in these projects will suffer as a result.

I understand that you've concluded that the cost of maintaining jumbo support is too high for Google's developers. It would help me to better understand your perspective if you or someone else could share the data that led to this conclusion. Is this cost inherent to the existence of the jumbo configuration, or more related to fixing breakages that occur from time to time? If the latter then perhaps we can make the configuration community-supported (e.g. those of us that care about it can keep it working) instead of deleting it all together?

Thanks,

Marshall

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTA2tPLoDUSrq3x5E%3DabPCHpqGEQ%2BReAGj2-4LVwZcY%2B0g%40mail.gmail.com.

[Christian Biesinger]

On Thu, Aug 15, 2019 at 4:24 PM 'Dirk Pranke' via Chromium-dev
<chromi...@chromium.org> wrote:
> However, based on the experience we've gained with this configuration over the past many months it's been supported, we've reached the conclusion that the way jumbo is currently implemented, it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary. As a result, we feel that the cost to the project of maintaining jumbo is not worth the benefit it is providing.

I don't really understand this decision. Who made the decision? Based
on what data? What is "too much" interference?

Christian

[Dirk Pranke]

The decision was made in eng review (i.e., with the people listed in //ENG_REVIEW_OWNERS, along with a couple of other people involved in build-related things, like myself).

We don't have a comprehensive set of data that we can share, but one set of observations was that we'd hit multiple breakages per week due to jumbo-specific issues that devs would have to work around. The general consensus was that the workarounds almost never resulted in better code. 

Again, I can't point to specific CLs here, though perhaps others can. However, I also suspect that we can debate the merits of changes to specific CLs quite a bit and I'm not sure how productive that would be at this point.

-- Dirk 

Christian

[Dirk Pranke]

Hi Marshall,

Thanks for sharing the data. The impact does not surprise me. 

I don't think it's feasible to support jumbo as a community-supported thing on trunk, since the primary perceived problem is that supporting it causes us to write code we don't want to write; i.e., we wouldn't otherwise want to accept CLs making the changes that you'd need to make.

I suppose it's conceivable that people could somehow figure out how to support a fork or a branch that kept jumbo working, but the cost of that seems like it would also be higher than what would be readily justifiable.

-- Dirk

[Daniel Bratell]

"causes us to write code we don't want to write" is rather vague. The only thing I can think of is that people want to copy functions around and due to jumbo can't do it without renaming said functions and if that is it, I think jumbo is doing us a favor. Code should preferably be at one place, not many places.

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTCUt1W9p7YLX%2B_%3DSrP%2BEfWgbsZdA1Sf3Jxc8XjtRPBa1g%40mail.gmail.com.

[Yoav Weiss]

As I believe this is the first time this is being discussed in the wider forum of chromium-dev, 1 week notice before taking jumbo builds down seems awfully short. Do we have large amounts of data indicating that it is actively harming project members that are not using jumbo, that would justify such urgency?

If the answer to the above is no, could we make sure that the deprecation timelines are extended to a point in time where reasonable alternatives are available?

-- Dirk

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTCMjv6emtxSXt7kFrT5Mjga0QwymD6Md176_45R_59msg%40mail.gmail.com.

[Jan van Oort]

Agree with Yoav Weiss. Deprecation timelines in major open-source projects should give participants a reasonable delay to look around for options.‎ Often such delays have an order of magnitude of months; one week seems unreasonably short.

Jan van Oort
Head of Research
KIVU Technologies GmbH 
www.kivu.tech

From: Yoav Weiss Sent: Samstag, 17. August 2019 10:25 To: Dirk Pranke Reply To: yo...@yoav.ws Cc: Avi Drissman; chromium-dev Subject: Re: [chromium-dev] We're removing support for the jumbo build

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CACj%3DBEjR7tTBsSR84iA6B0uV%3DK09tygLnHm%3DoC%3DhgG0Zh_6qtQ%40mail.gmail.com.

[Dirk Pranke]

Good suggestion; done.

-- Dirk

[Matt Giuca]

I think this was the right decision but I agree it would have been good to justify it with data.

To be clear (since it hasn't been stated explicitly in this thread), the breakages that devs have to workaround are usually a name in an anonymous namespace in file A, that conflicts with the same name in an anonymous namespace in file B. This means you have to rename them to not have the same name, effectively defeating the entire point of anonymous namespaces.

To find some relevant data, I went looking through the commit log for commits with "jumbo" in the description. More often than not, it's bratell@ fixing things.

Here's an example of where it just makes the code (slightly) worse:

https://chromium-review.googlesource.com/c/chromium/src/+/1717300

We had a couple of perfectly reasonable (in the local file context) names in an anonymous namespace, "kConfigBinaryPbFileName" and "LoadProtoFromDisk", which conflicted with something else with the same name in a different file. The fix was that these were renamed to "kSafetyTipsConfigBinaryPbFileName" and "LoadSafetyTipsProtoFromDisk". I think that qualifies as "code we don't want to write": we use namespaces in C++ so that we can have short names that give just enough detail for the context they're in, but don't need to write over-long names that are globally unique, which is how things are typically named in C code. It's "code smell" if you start to see variables and functions all including the name of the file in their own name.

Daniel Bratell wrote:

> The only thing I can think of is that people want to copy functions around and due to jumbo can't do it without renaming said functions and if that is it, I think jumbo is doing us a favor. Code should preferably be at one place, not many places.

True, but having two functions with the same name (in a project as large as Chromium or even a component thereof) does not necessarily mean they are duplicated. "LoadProtoFromDisk" is a good example of a name that could have different meanings in different contexts. The name is perfectly clear as it is, in the context of this file; disambiguating it globally is just adding noise.

Here's another example where you actually detected a duplicated function:

https://chromium-review.googlesource.com/c/chromium/src/+/1720772

"DoesAndroidSupportMaskableIcons" was duplicated in an anonymous namespace in two different files, which is a potential for a bug later down the line. In this case, Jumbo is helping us detect dupes, but I'll point out that the fix for this was to just rename the two functions to "DoesAndroidSupportMaskableIconsForWebApk" and "DoesAndroidSupportMaskableIconsForHomescreen", which IMO makes the situation worse, because we still have duplicated code, but now we're implying to future editors that there's possibly a difference between "supporting maskable icons for WebAPK" and "supporting maskable icons for add-to-homescreen" (when in reality, these names were just chosen as a quick way to avoid a naming conflict to satisfy Jumbo builds).

I'm not trying to criticize you personally in relation to this particular CL, but just point out that overall, having a process that forces us to name variables to be globally unique may be forcing us to make suboptimal naming decisions that can actually affect the way future readers think about the code.

>>> import this
The Zen of Python, by Tim Peters
...
Namespaces are one honking great idea -- let's do more of those!

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTAf9BR7MEsqG8X3h%2B9sTv1fCfOFTQA7bW%3DaBR%2Bi1s-rNw%40mail.gmail.com.

[Daniel Bratell]

So using my jumbo commits is not the best example of how jumbo causes problems to the project since most of my commits are preventive, fixing issues that turn up in the most extreme jumbo configuration (jumbo chunk size = infinity) which nobody actually uses. If my commits is the majority, then there is no real problem.

What you want to find is a lot of CL:s with jumbo-only compile problems. The project has a couple of thousand CLs per week, if jumbo problems are plentiful or common, you'd expect dozens or hundreds of these and I remember seeing about three the last year. And I've been looking. You would also expect the non-CQ jumbo bots to constantly fail since they cover more than the CQ bots, and they fail occasionally but not all the time.

If you go through all failures of the CQ jumbo bot (which fails much less than other bots since it does much less) you mostly see it failing in tree wide breakages or when a CL contains code that doesn't compile in equivalant bots without jumbo, not because of jumbo.

My point is that even if what you say is true, some functions and constants end up getting names that wouldn't be your first choice, the data I have available indicates that it's overall a tiny problem. And there doesn't seem to be any public data that contradicts that. Had there been data that indicated a problem we could have acted on it (for instance if a certain compile target experienced this a lot, it could have had jumbo support removed). 

(I do agree that the function names you mention were not good and if you look through all of my commits I'm sure you can find more examples of functions names that could have been better, but where nobody could think of a better name at a time).

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAHqYdcb8o6Q%3DcbsabX18NgFhc2DFqAHUZ-Odfj8t1%2BSNfzQrZw%40mail.gmail.com.

[Gabriel Charette]

As a Googler with access to Goma I still make regular use of the jumbo build when compiling from my laptop. It's the only way to have a reasonable amount of targets to compile for small test suites (I pretty much never compile all of Chromium there).

From personal experience in the last year I don't think I've seen a single jumbo related conflict neither in a CL I've written nor reviewed.

The only naming conflict I've had to deal with was caused by regular C++ header pollution (a unittest.cc file in //base had a global named "data" which conflicted with base::data() when stl_util.h ended up in the transitive closure of that unittest.cc per an unrelated refactoring). Just like the occasional jumbo errors, it was easily fixed by renaming the global variable.

I agree with bratell@ that him being the main source of commits to address jumbo issues seems to indicate that there isn't much of a problem in practice for the average developer?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.z6rdh4mkrbppqq%40cicero2.linkoping.osa.

[Alexei Svitkine]

I've also found the jumbo build helpful. Sometimes I work on a plane where I don't have access to Goma and only have my MacBook pro laptop. It's helpful to have faster compile times in this situation.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAJTZ7LK0uoJ6Zg9XWvCXqgjKmAS1YQ1zjktjFcGG0Yc%2B4rpAVA%40mail.gmail.com.

[Alex Clarke]

I've also found jumbo builds useful on occasion and I can't remember the last time if at all jumbo builds caused me any problems.  This decision feels like a mistake.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAKFU2SB0DSU%3DxUdDVK8PBdc_-nHXDpzGfTUgJQ-6nQM41gHptw%40mail.gmail.com.

[Peter Kasting]

+1 to what Gab and Alex said.

It feels like this may have been a well-intentioned decision made with misleading anecdata.  If that's possible, I'd love for the eng leads to reconsider.

PK

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAPG_qM6H-Zg3EU0pL4vFbmSe4k4iTjXfc7rXte%3DrhZhbW%2BTGyg%40mail.gmail.com.

[Ken Rockot]

I've tripped over jumbo many times. I'm glad we're removing it, though I wish we could have supported it with some help from clang. The current approach in principle feels like a hack., and in practice it means I've had to deal with naming collisions among unrelated components much more often than I would like.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAPG_qM6H-Zg3EU0pL4vFbmSe4k4iTjXfc7rXte%3DrhZhbW%2BTGyg%40mail.gmail.com.

[Mason Freed]

This is likely a naive suggestion, but is there any way to just append (on the fly, during jumbo compilation, in python) a hash of the translation unit name to the name of all objects defined inside an anonymous namespace? That sounds like it would eliminate the primary complaint here of name collisions.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CA%2BapAgGEav0jLFvvSBNwMs7T4zWxfNLai%2B-QZFExP0m-UQRV6w%40mail.gmail.com.

[David Benjamin]

That seems like the cleanest option to me, though a clang plugin may be better for an accurate parse. That way jumbo doesn't break developer expectations for C++. We can probably make the transformation even simpler with inline namespaces:

1. Transform namespace { ... } to namespace { inline namespace <hash-of-filename> { ... } }

2. Transform static foo ... to inline namespace <hash-of-filename> { static foo ... }

That way we don't need to go find the places a name is referenced or risk oddities around argument-dependent lookup.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAM%3DNeDgZGaSqFc_XnXxEKWYQQaMCXpJrJqPBoP7XtmCBU1ik_g%40mail.gmail.com.

[Erik Staab]

On Mon, Aug 19, 2019 at 12:07 PM 'Peter Kasting' via Chromium-dev <chromi...@chromium.org> wrote:

+1 to what Gab and Alex said.

It feels like this may have been a well-intentioned decision made with misleading anecdata.  If that's possible, I'd love for the eng leads to reconsider.

I think it would be hard to get good data and quantify the impact but I made an attempt when we started discussing costs of jumbo support. I looked at the number of non-bot references to jumbo in CL descriptions and comments over a six month period and found that there were 254 CLs with either. Many of the CL comments were people uploading new patchsets when they realized there was a symbol collision. I'm not sure how much time is lost when people need to do this (maybe a CQ cycle? so 50 minutes at the 50th %ile?) but maybe others can estimate.

Here is a spreadsheet with the data if you want to look through yourself:

https://docs.google.com/spreadsheets/d/1JEHEOjyq91PjgCMy2mmr-bFm8b5f3nj8oz5mnXm0mjo/edit#gid=1556975522

Also, we're still open to helping people get set up with the Goma server we open-sourced at the last BlinkOn. infr...@chromium.org is a good list to reach the team or #goma on slack sometimes.

Erik

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAHOzFAwcYjPOX%2BN%2BgfSBJLQDhSw1J3NddOmVg-cVCfgzF%2BDKw%40mail.gmail.com.

[Samuel Attard]

Just to add to what other folks have said in this thread the 1 week timeline is incredibly short notice.  We have been experimenting with the open sourced version of Goma for months and have constantly run into performance issues that we've been working through with the Goma / RBE folks but in it's current state it is not ready to "replace" the performance benefits that we got through the jumbo build.  (For googlers you might be able to see this thread --> https://issuetracker.google.com/issues/136104063)

I'm currently spinning up some CI runs to measure the exact build time cost of disabling jumbo (and that build time cost will directly translate into monetary cost for anyone running their own CI infrastructure).  I hope to share some specific numbers next week but initial estimates from when I was experimenting with goma are around 2-3x slower, similar to Marshall's numbers above.

So far in this thread I see a lot of "jumbo is bad, we're removing a bad thing" but as others have pointed out a lot of us are missing the context / metrics for why jumbo is bad especially when we fundamentally rely on it to get Chromium build times down from "insane" to "kind of reasonable".  This decision from Google's side makes sense because y'all have a massive set of goma infrastructure, but the rest of do not and I think this impact was not taken into consideration or dramatically underestimated.

To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTAP4t-XpNe3O4GWbvNq5r%2BHdwObcUvMJ%2BvOQjdbDDYZBQ%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

[Scott Violet]

I have also tripped over the jumbo build a number of times. Beyond the simple name changes, the biggest source of pain I've encountered is in template specialization, and in particular the ClassProperty related ones. See  https://chromium-review.googlesource.com/c/chromium/src/+/786190/ for more information. If you get the macro order wrong, the compiler errors are rather inscrutable.

  -Scott

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CA%2BapAgGEav0jLFvvSBNwMs7T4zWxfNLai%2B-QZFExP0m-UQRV6w%40mail.gmail.com.

[Dirk Pranke]

As far as the one-week notice being too short goes, I think that's fair criticism and I apologize.

I set a short deadline because I didn't think having a timeline for investigating alternatives would be all that useful, since there's not an obvious alternative to what jumbo does, particularly if you can't use a compile farm like Goma (e.g., because you're working offline or on a slow or metered network). 

However, I suppose it could be useful to keep jumbo around for a time while people helped contribute to Goma to make it be more useful in other environments (either to set up their own goma clusters or to make our Goma/RBE-based service easier to use), so perhaps I was mistaken in this thinking, too.

-- Dirk

As others have commented, something like Goma acceleration can be a good alternative in some cases, but if you can use Goma you'd likely be investigating it already and jumbo actually makes goma slower. If you can't use Goma (e.g., because you need to work offline or have a network connection that is too slow, metered, or whatever), I don't know of another alternative to jumbo.

-- Dirk

[Morten Stenshorne]

"'Dirk Pranke' via Chromium-dev" <chromi...@chromium.org> writes:

> As others have commented, something like Goma acceleration can be a
> good alternative in some cases, but if you can use Goma you'd likely
> be investigating it already and jumbo actually makes goma slower.

That depends. It's true for my Linux workstation, but on my MacBook pro
it takes:

just goma: about 25 mins
just jumbo: about an hour
goma + jumbo: less than 15 mins

I disable goma when I'm on a slow/flaky/metered network [1] or no
network at all [2].

[1] Cafés, hotels, airports, cabins, remote villages
[2] Airplanes

I also sometimes run with just jumbo + ccache because the huge number of
translation units (probably the reason?) seems to affect the performance
of various tools (linkers and such, I guess?), so that jumbo + ccache is
way faster when rebuilding a few translation units and everything hits
the cache (like 6 seconds instead of 30-60 seconds).

I'd very much like to see the need for jumbo eliminated before it's
removed.

--
Morten Stenshorne, Software developer,
Blink/Layout, Google, Oslo, Norway

[Dirk Pranke]

That's interesting. I don't recall us seeing results before that showed jumbo + goma being faster than jumbo alone, when we'd done testing (and I don't think we've seen that on the CQ bots, either, but we could re-check).

I believe that jumbo does make linking faster due to the fewer TU's and less need for code stripping, as you say.

-- Dirk

[Yoav Weiss]

On Mon, Aug 19, 2019 at 6:42 PM Dirk Pranke <dpr...@google.com> wrote:

As far as the one-week notice being too short goes, I think that's fair criticism and I apologize.

I set a short deadline because I didn't think having a timeline for investigating alternatives would be all that useful, since there's not an obvious alternative to what jumbo does, particularly if you can't use a compile farm like Goma (e.g., because you're working offline or on a slow or metered network). 

However, I suppose it could be useful to keep jumbo around for a time while people helped contribute to Goma to make it be more useful in other environments (either to set up their own goma clusters or to make our Goma/RBE-based service easier to use), so perhaps I was mistaken in this thinking, too.

Thanks for listening. Highly appreciated! :)

I think there's another angle to consider when talking about Goma vs. jumbo - sustainability.

As jumbo builds require ~3x less CPU power, applying (something like) them to everyone could have a significant impact on reducing our GCP bill (disclaimer: I don't know if the Chrome team is actually charged and if that argument actually makes sense for us. It definitely makes sense for other orgs).

It would also mean that the project's CO2 footprint would be significantly smaller, at least when it comes to computing infrastructure.

Maybe those latter arguments could be a motivating factor to work on a jumbo equivalent that doesn't suffer from its downsides:

* Doesn't result in anonymous namespace collisions that require manual intervention.

* Doesn't result in significantly slower single .cc file change build times (my anecdotal experience, on which I have no data). There are probably tradeoffs here we'd need to figure out.

* Doesn't result in slower Goma builds in some OSes. 

[Dirk Pranke]

On Mon, Aug 19, 2019 at 12:41 PM Yoav Weiss <yo...@yoav.ws> wrote:

On Mon, Aug 19, 2019 at 6:42 PM Dirk Pranke <dpr...@google.com> wrote:

As far as the one-week notice being too short goes, I think that's fair criticism and I apologize.

I set a short deadline because I didn't think having a timeline for investigating alternatives would be all that useful, since there's not an obvious alternative to what jumbo does, particularly if you can't use a compile farm like Goma (e.g., because you're working offline or on a slow or metered network). 

However, I suppose it could be useful to keep jumbo around for a time while people helped contribute to Goma to make it be more useful in other environments (either to set up their own goma clusters or to make our Goma/RBE-based service easier to use), so perhaps I was mistaken in this thinking, too.

Thanks for listening. Highly appreciated! :)

I think there's another angle to consider when talking about Goma vs. jumbo - sustainability.

As jumbo builds require ~3x less CPU power, applying (something like) them to everyone could have a significant impact on reducing our GCP bill (disclaimer: I don't know if the Chrome team is actually charged and if that argument actually makes sense for us. It definitely makes sense for other orgs).

It would also mean that the project's CO2 footprint would be significantly smaller, at least when it comes to computing infrastructure.

Indeed, this was one of the primary reasons I was initially interested in jumbo. 

However, jumbo appears to reduce the amount we're able to reuse cache hits in goma, so it seems like we don't see as much of a benefit as I had initially hoped for. It's hard to get an apples-to-apples comparison, though, since it's hard to get wide-scale experiments with equivalent traffic. It's also harder to get comparable numbers out due to the ongoing migration to the RBE-based Goma instance, where we have much less insight into the costs compared to when we were running our own servers.

-- Dirk

[Daniel Bratell]

We have suggested such schemes (see Mostyn's post earlier in the thread) but they have not materialized, mainly because they would have required changes in clang. Small changes, but non-standard C++ changes, and they were not accepted.

/Daniel

[Jan van Oort]

Kudos for listening :-) 

[Daniel Bratell]

On Mon, 19 Aug 2019 18:54:16 +0200, Scott Violet <s...@chromium.org> wrote:

I have also tripped over the jumbo build a number of times. Beyond the simple name changes, the biggest source of pain I've encountered is in template specialization, and in particular the ClassProperty related ones. See  https://chromium-review.googlesource.com/c/chromium/src/+/786190/ for more information. If you get the macro order wrong, the compiler errors are rather inscrutable.

Indeed. Took me a while to even figure out what was going on.

This is illegal in C++:

template <typename T> int calc(T x);

int do_calc(int x) {return calc(x);}

template <> int calc(int x) { return x / 2; }

../testtemplate.cc:3:27: error: specialization of ‘int calc(T) [with T = int]’ after instantiation
template <> int calc(int x) { return x / 2; }

What it's basically saying is that by providing the specialization after the function was used, it might have ended up using the wrong template instantiation. Assume all three of those lines were provided in different files, then the compiler might not know about the specialization and everything compiles fine. With jumbo, the compiler suddenly knew about the specialization and we got those errors, and they are not well understood.

Sometimes this error message will prevent bugs, but if there is no default implementation of the template, then the actual "instantiation" won't happen until linking time where the linker will probably choose the right implementation.

Should we be happy jumbo uncovered this or be annoyed by it? I feel like the compiler could have skipped the error in some circumstances so it's both helpful and annoying depending on circumstance. And when the actual lines are hidden by macros (as in ClassProperty), that compiler error can really seem hard to understand.

/Daniel

[Daniel Bratell]

On Mon, 19 Aug 2019 18:42:23 +0200, Erik Staab <est...@chromium.org> wrote:

On Mon, Aug 19, 2019 at 12:07 PM 'Peter Kasting' via Chromium-dev <chromi...@chromium.org> wrote:

+1 to what Gab and Alex said.

It feels like this may have been a well-intentioned decision made with misleading anecdata.  If that's possible, I'd love for the eng leads to reconsider.

I think it would be hard to get good data and quantify the impact but I made an attempt when we started discussing costs of jumbo support. I looked at the number of non-bot references to jumbo in CL descriptions and comments over a six month period and found that there were 254 CLs with either. Many of the CL comments were people uploading new patchsets when they realized there was a symbol collision. I'm not sure how much time is lost when people need to do this (maybe a CQ cycle? so 50 minutes at the 50th %ile?) but maybe others can estimate.

Here is a spreadsheet with the data if you want to look through yourself:

https://docs.google.com/spreadsheets/d/1JEHEOjyq91PjgCMy2mmr-bFm8b5f3nj8oz5mnXm0mjo/edit#gid=1556975522

254 patch uploads over 6 months that mention jumbo. I recognize a number of them being things I've been working on where I've mentioned jumbo in a comment but which hasn't been a jumbo breakage so I will have to see if I can get some feeling for how many of these are real, though it seems the error logs might have been removed/archived[1]. One factor that I've seen in the past that makes it hard to judge the problem is that sometimes the jumbo builder compiles first so it's mentioned in the patch set with the fix. And of course there is no guarantee someone mentions "jumbo" when fixing a jumbo problem so the error margin goes both ways.

Do you have some context of how many patch uploads in total there were over this time? 

/Daniel

[1] : https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8931982143184577936/+/steps/compile/0/stdout doesn't load for me.

[Daniel Bratell]

I went through a bit of the list. Lines n*15 + 2 to be specific, and only in the lists that were expanded (not sure why there were 250 entries in the link list but only 145 in the expanded one).

Of those:

3 were about variable shadowing where I mentioned in the commit message that I hoped it might prevent jumbo problems.

2 were anonymous namespace jumbo problem that resulted in renames. (Potentially worse names)

1 was a  anonymous namespace jumbo problem that resulted in more optimized code.

1 was an anonymous namespace jumbo problem because the wrong name was used.

1 was ambiguity between shadowed namespaces (::switches, ::foo::switches) which happens in all kinds of builds but more in jumbo builds and is a construct that is not allowed in the code style guide.

1 was me doing preventive code cleanup to avoid future problems.

1 was a revert of a jumbo fix because of concurrent code changes landing, breaking the build.

----

So 2 out of 10, or 20% of the samples, were of the kind that we don't like, where the result might be worse code (longer, worse, uglier, different symbol names).

There is a large error margin on this, but the sampling has confirmed my belief that the perceived problem ("jumbo makes us write worse code") is much smaller than the people making this decision thought.

I get it that jumbo is not a zero cost feature, but it's also not a zero benefit feature. How much time did it save during these six months? How much more work might have been done because of it? We have no data on that either, but it ought to be much more than what it cost.

/Daniel

[David Benjamin]

I think you misunderstand me. The suggestion is to build a tool external to clang which performs this transformation as part of the jumbo build process. I.e. transform the source files and then pass the transformed files into the compiler. That way there isn't any non-standard C++ involved, either on the compiler's side or the programmer's side. The hacks are left to the jumbo machinery itself.

[David Benjamin]

Oh, but the particular transformation I suggested wouldn't actually work because the call sites are ambiguous. Still, Mason's version should be possible.

[Daniel Bratell]

Ah, not sure anyone has tried that one (transforming all "global" symbol names in the anonymous namespace before sending the text to the compiler). Maybe it could work, though there would be a lot of edge cases. If someone wants to try it, please add me to the discussion because I know a bit about where the rough edges are.

/Daniel

[Chris Hamilton]

This is pretty much what I was suggesting earlier (a source-to-source Clang plugin that runs as part of the jumbo build process). You don't need to actually upstream any compiler support that way.

On Tue, Aug 20, 2019 at 9:48 AM David Benjamin <davi...@chromium.org> wrote:

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAF8qwaDHMNSDE0jcaCdT8oawZ1LK4rRrrKewTZph-CT5CbLzYQ%40mail.gmail.com.

[Nicolas Ouellet-Payeur]

We'd need to be careful with how we mangle the symbol names to dedupe them.

If we change the symbols naively (e.g. with a regexp from a python script),
then we could miss some call-sites and fail to compile valid code.

If we change them by parsing the source code (e.g. with a clang plugin),
then we could lose the benefits of jumbo wrt build time.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTAP4t-XpNe3O4GWbvNq5r%2BHdwObcUvMJ%2BvOQjdbDDYZBQ%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTBrDJq6EVRND5E%3D1wuvEPjXKyOv9qOPZ8%3DMP2%2BJOvYAFQ%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTAf9BR7MEsqG8X3h%2B9sTv1fCfOFTQA7bW%3DaBR%2Bi1s-rNw%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAHqYdcb8o6Q%3DcbsabX18NgFhc2DFqAHUZ-Odfj8t1%2BSNfzQrZw%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.z6rdh4mkrbppqq%40cicero2.linkoping.osa.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAJTZ7LK0uoJ6Zg9XWvCXqgjKmAS1YQ1zjktjFcGG0Yc%2B4rpAVA%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAKFU2SB0DSU%3DxUdDVK8PBdc_-nHXDpzGfTUgJQ-6nQM41gHptw%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAPG_qM6H-Zg3EU0pL4vFbmSe4k4iTjXfc7rXte%3DrhZhbW%2BTGyg%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CA%2BapAgGEav0jLFvvSBNwMs7T4zWxfNLai%2B-QZFExP0m-UQRV6w%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAM%3DNeDgZGaSqFc_XnXxEKWYQQaMCXpJrJqPBoP7XtmCBU1ik_g%40mail.gmail.com.

[Darin Fisher]

Hi folks,

Just to jump in on this thread and provide my perspective...

First of all, I definitely recognize the benefits jumbo has brought to the project and the community. I'm aware of how large our project is and how costly it is to do a fresh build (without Goma). I'm really excited that we are taking steps to make Goma more available, both with it being open source and with an instance that Google is making available to the community. Also the investments we've made in speeding up Clang really help too. There is more we can be doing.

When it came to the jumbo build specifically, I definitely tripped over a couple issues relatively quickly in the past month or so, and they surprised me. One was the anonymous namespace issue already mentioned in this thread. The other was macro pollution. In both cases, the failure mode was mysterious and perplexing. The errors I was seeing on the jumbo try bot made no sense considering the .cc file I was working with. So, I had to learn about jumbo build, and I probably spent more time than I'm willing to admit debugging these issues.

In the case of the anonymous namespace issue, I really didn't know what was going on. I "fixed" the issue by making the function declared in the anonymous namespace a static class method instead. I didn't think of trying to just rename it. It didn't occur to me that name collision was the problem. I just went the static class method route and moved on. Maybe I should have spent more time investigating. But it left me with a bad taste as I knew I was making code just a little bit less optimal.

In the case of the macro pollution, I asked around, and I was told that I could exclude certain .cc files from the jumbo build. I tried that, but the jumbo try bot kept failing. The error I was seeing had moved to a different .cc file. Looking at the error more closely, the issue was a macro defined via zlib.h conflicting with a header file from mojo/. I tried excluding more .cc files. Still no good. Finally, I set my CL aside, and opened up a fresh branch to try to make zlib not export this particular macro. CL here: https://chromium-review.googlesource.com/c/chromium/src/+/1709871

Now, arguably the second CL is just a good thing for the project irrespective of jumbo. I felt good about that change for that reason. But, again, I spent a lot of time across these two issues, and as an engineer feeling like I already have to know so many things in order to be productive on this project, this was just one more thing that I had to learn. Not great. How many other people are struggling similarly?

I asked others about the jumbo build. Is it really worth this kind of cost? How much of this kind of cost are people seeing? Erik Staab as he posted did a basic grep across the comments on gerrit and found about 2 per work day happening in the past 6 months. That seems like a good amount of accumulated cost, and it is hard to see how that can be a good thing for the project.

As Dirk mentions below, the eng review group got together with Dirk and a few others to discuss. Honestly, the only real question was when, not if, we should disable support for jumbo. Again, thanks to Goma availability and the speed-ups on Clang, the situation is not as bad as it once was. That gave us confidence that we could move forward with this change. Now, when it comes to timeframe, I agree we should be flexible in supporting the community through this change. Making an instance of Goma available to the community is a new thing, and we should give people time to adjust. I suggest that we extend the deprecation timeline by one month from Dirk’s original post.

Thanks,

-Darin

On Thu, Aug 15, 2019 at 2:24 PM 'Dirk Pranke' via Chromium-dev <chromi...@chromium.org> wrote:

Hi all,

We have decided to remove support for the jumbo (aka unity) build configuration from the project.

We are very conscious of the fact that Chromium is very expensive and slow to compile, and that that represents a real barrier to contributing.

However, based on the experience we've gained with this configuration over the past many months it's been supported, we've reached the conclusion that the way jumbo is currently implemented, it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary. As a result, we feel that the cost to the project of maintaining jumbo is not worth the benefit it is providing.

We continue to invest in other approaches to reducing build time. We've released the Goma build accelerator code and hope to continue making it easier to use for others. We are also working to enable our Goma clusters for external contributors (anyone who has tryjob access) who don't have their own build acceleration tools. We're starting with Linux support now and will have more to share on that soon. If you are interested in getting access to this system, please fill out this form. We're also working on Clang and LLVM and have made significant improvements this year to cycle time as a result, and we expect to continue to do so for the foreseeable future.

We plan to remove support a week from now, on August 22nd.

Please let me know if you have questions about this.

-- Dirk

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Darin Fisher]

I think this is an interesting idea too, but please note there are other issues as well. See what I just posted. I ran into problems with macro pollution, and this wouldn't help with that.

Thanks,

-Darin

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.z6tqr0uyrbppqq%40cicero2.linkoping.osa.

[Mostyn Bramley-Moore]

IIRC the "JumboSupport" Clang patch + plugin did not require an additional parsing step or intermediate source file generation, and it also helped with macro pollution.

Is there anyone interested in updating the Clang patch, and picking up the upstreaming effort?  Would Google consider including the "plugin" as part of the standard Chromium clang tarballs if we could eliminate or at least minimise the required Clang patch?

-Mostyn.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAP0-QpttiRPnLSu2BM8BE51b8LhrBR%2BqWqHZxi_a7Rewc38FmA%40mail.gmail.com.

--

Mostyn Bramley-Moore

Vewd Software

mos...@vewd.com

[Daniel Cheng]

I think a source-to-source translation would be interesting, but unless such a tool could work without parsing source files at all, the slowdown would likely be non-trivial. And anything that doesn't actually parse the source is prone to running into edge cases that it mishandles.

Disclaimer: I was one of the ones who suggested implementing a clang pragma for supporting jumbo builds, so I'm not entirely neutral here :)

Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CACfMsGHfkYJqP%2BFX%3DB7yRiVQZ1o5eOM2AQgH%3DWat13_LsVU%2B-A%40mail.gmail.com.

[Daniel Bratell]

I'm sorry that zlib macro caused so much problem. I've had an encounter with zlib.h in the past and know that it's not playing nicely in a modern C++ world.

But I can also not avoid thinking that if jumbo problems had been more common, you wouldn't have had to chase around for someone that could fill in the dots of what jumbo is and what it implies.

Adding more time seems reasonable since everyone (including I) still seem very unsure about the readiness of alternatives. Support for other build platforms than Linux has not been mentioned, as an example. Is the suggestion to cross compile everything on Linux, using a Linux goma?

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAP0-QpujCWNCY%2BUK7GNei08FRaE07TYo3STa705uwzotkzegEQ%40mail.gmail.com.

[pdknsk]

I strongly ask you to reconsider. Without it, building Chromium is very slow. And every new release is slower still. Most people cannot compile on $10000 28-core Xeon CPUs.

May I suggest as an intermediate step to make jumbo_file_merge_limit fixed to remove one possible point of failure. Even a fixed setting of 8 decreases compilation time significantly.

[Richard Townsend]

So just to add my experience: I recently got a new workstation with 2x Xeon Gold 6148's (80 threads.) Building chrome.exe (version 78) for Windows on Arm takes about 64 minutes on that system, but building with Jumbo takes about 22 minutes. We pay the cross-compilation tax in that configuration (several intense parts of the code base are compiled twice to create the snapshot), but switching on Jumbo still makes a massive difference in how fast I can bisect the tree to find issues. Google's current Goma offerings don't work for us: we have restrictions on sending code out before it's been reviewed, we can't use it in our CI systems, and the various third-party Goma server backend options don't seem to support Windows. Is there any possibility of keeping the build working for now whilst we find another solution?

Best

Richard

[David Barr]

I second keeping the jumbo build configuration alive until an alternative is developed. The benefits to total build resource consumption and early feedback to developers have outweighed the maintenance burden for my team at least. We adopted jumbo builds in combination with a distributed build cache after taking the time to optimize the interaction between them[1].

I acknowledge that the burden of resolving jumbo issues is carried by the wider chromium community and that the benefits have not yet been realized for most bots.
From this thread it appears that some Googlers have taken advantage of jumbo builds but general awareness of them is low.

[1] : https://chromium-review.googlesource.com/1102218

PK

To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTAP4t-XpNe3O4GWbvNq5r%2BHdwObcUvMJ%2BvOQjdbDDYZBQ%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTBrDJq6EVRND5E%3D1wuvEPjXKyOv9qOPZ8%3DMP2%2BJOvYAFQ%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTAf9BR7MEsqG8X3h%2B9sTv1fCfOFTQA7bW%3DaBR%2Bi1s-rNw%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAHqYdcb8o6Q%3DcbsabX18NgFhc2DFqAHUZ-Odfj8t1%2BSNfzQrZw%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.z6rdh4mkrbppqq%40cicero2.linkoping.osa.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAJTZ7LK0uoJ6Zg9XWvCXqgjKmAS1YQ1zjktjFcGG0Yc%2B4rpAVA%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAKFU2SB0DSU%3DxUdDVK8PBdc_-nHXDpzGfTUgJQ-6nQM41gHptw%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAPG_qM6H-Zg3EU0pL4vFbmSe4k4iTjXfc7rXte%3DrhZhbW%2BTGyg%40mail.gmail.com.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAHOzFAwcYjPOX%2BN%2BgfSBJLQDhSw1J3NddOmVg-cVCfgzF%2BDKw%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

[dan...@chromium.org]

Just want to say that I have bumped into the jumbo build a few times - mostly in renaming constants to not be clashing. It has been a minor inconvenience and only minimally affected (approaching 0 impact on) my productivity nonetheless. It seems like the zlib macro is the big problem we can point to right now, as well as problems being hard to diagnose when they happen due to the infinite jumbo build. While another solution is implemented can we prefix the zlib macro so that collisions are less likely? How can we document things to make understanding a jumbo failure easier? A google search goes to jumbo docs which don't explain possible failure conditions outside of a passing mention. Could someone motivated to keep the build going add more details there?

I was wondering also.. when we grepped for mentions of jumbo in gerrit, did we look at the content at all? Maybe "fix jumbo bot" was actually just the first bot that failed compile. And "thank goodness for jumbo being so fast" would also be included in that grep, for example.

Thanks,

Dana

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/1003670c-8fa4-4113-8168-f999176e4faa%40chromium.org.

[Dirk Pranke]

Hi all,

I appreciate the further feedback on this, and I really don't want to appear dismissive of the concerns, but the thirty days that Darin asked for have elapsed and nothing has been said or done that materially changed things, or raised significant new information. 

As a result, we're not going to revisit this decision at this time, and I'll be sending out CLs to start turning off the builds and removing jumbo support in the next day or two.

 -- Dirk

[Jakob Kummerow]

FWIW, this makes me sad. I've done my share of contributions to making the Jumbo build work and keeping it working, and I still believe that it saves more engineering productivity than it costs. (Anecdata: support for enabling Jumbo has spread as far as the qtwebengine library, saving everyone who compiles that a few hours of CPU time.)

Would it be an option to turn the Jumbo bots into FYI bots (and taking them off the CQ), instead of turning them off? Then folks who are happy with Goma wouldn't have to know/worry about Jumbo, and folks who feel that Jumbo saves them N hours per week could split the effort required to keep it working? If it then turns out that the bot is red most of the time, we can still remove it in a couple of months.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTAhdzxPqJsASMBp5hXHp74jb%2BFR52Si%2B_MEKY5vxV3c%3Dw%40mail.gmail.com.

[Dirk Pranke]

We talked about jumbo as a community-supported option up-thread a bit, but, to repeat and go into a bit more detail ...

The primary objection to jumbo (as I understand it) is not the annoyance of supporting another build config, but rather that the changes that you have to make are not changes that you would normally make and that they make the code worse. Supporting jumbo as an FYI config won't help this, because you'd still have to make the changes.

In addition, my general sense is that asking people to change their code to address a failure due to their CL after the CL lands is generally annoying, which is part of why we want configurations to be in the CQ (to catch them earlier). That said, I could see how moving to FYI would at least help in that the dev trying to land the CL wouldn't have to figure out what's wrong, they'd just have to review the fix.

Also, IIRC, we did discuss an FYI-only solution as part of the eng-review discussion, so, again, I don't think this is a new idea that might change the decision.

-- Dirk

[dan...@chromium.org]

On Wed, Sep 25, 2019 at 2:39 PM Dirk Pranke <dpr...@google.com> wrote:

We talked about jumbo as a community-supported option up-thread a bit, but, to repeat and go into a bit more detail ...

The primary objection to jumbo (as I understand it) is not the annoyance of supporting another build config, but rather that the changes that you have to make are not changes that you would normally make and that they make the code worse. Supporting jumbo as an FYI config won't help this, because you'd still have to make the changes.

I took a look through the supporting evidence of gerrit logs in https://docs.google.com/spreadsheets/d/1JEHEOjyq91PjgCMy2mmr-bFm8b5f3nj8oz5mnXm0mjo/edit#gid=1556975522

From looking through the first 20% of the CLs there. A large number (the majority?) of these are general improvements, far from making the code worse:

1. A bunch of CLs for getting rid of shadowed variables in general in blink. Shadowed variables are evil.
2. Getting rid of banned 'using namespace' uses. These are not allowed by our style guide.
3. Combining multiple identical methods. This reduced code duplication.

4. Improving grepping by naming things different. These didn't seem to be the majority of the changes, and I'm sure some of these were neutral (or worse..?) But we have a lot of things with the same names that make it hard to tell what is being called where when looking at grep results. Deep compiler analysis is needed (code search) in order to understand where code is going sometimes, and more unique names are a big welcome change from me.

I did see less than a handful of reverts due to jumbo breaking but it was very very few.

I guess I am wondering, what kind of analysis was done on these comments? Were they broken up into types/impact? Would you be willing to share more to help the community understand, given the large impact this seems like it will have on their productivity and ability to contribute? The majority of the CLs also are one-off "get jumbo working" type of CLs, and I saw many written and reviewed by folks at opera incurring zero cost on Google engineers really.

[Dirk Pranke]

estaab@ did the analysis, but I don't know how much more he did beyond reading the CL descriptions; he would have to say.

-- Dirk

[Peter Kasting]

On Wed, Sep 25, 2019 at 12:29 PM <dan...@chromium.org> wrote:

I guess I am wondering, what kind of analysis was done on these comments? Were they broken up into types/impact? Would you be willing to share more to help the community understand, given the large impact this seems like it will have on their productivity and ability to contribute? The majority of the CLs also are one-off "get jumbo working" type of CLs, and I saw many written and reviewed by folks at opera incurring zero cost on Google engineers really.

+1.

I think the core concerns eng leadership has are the right ones to have.  And I can understand why their initial glance at supporting data would have led them to conclude removing jumbo was correct.

There have been multiple replies on this thread from people who took a deeper look at the information available and concluded that, in their opinion:

• Net code quality was improved by jumbo rather than harmed
• Actual impact on most engineers was very low (note that anecdotal "I've tripped over this" is not the most useful metric due to salience)
• Jumbo is useful even to Googlers for use cases goma does not address

Finally, I note that reaction on this thread has been atypically one-sided for a chromium-dev thread, and support for maintaining jumbo has come not only from external contributors but from core Google contributors like gab, danakj, and me.

I am still willing to believe that this is the right call, but I think the above qualifies as either "new information" or at least something that would behoove the leaders arguing for jumbo removal to address directly, in the same way Darin already replied graciously to clarify the initial motivation.

PK

[Erik Staab]

On Wed, Sep 25, 2019 at 12:50 PM Dirk Pranke <dpr...@google.com> wrote:

estaab@ did the analysis, but I don't know how much more he did beyond reading the CL descriptions; he would have to say.

I only looked through the CL comment and description data enough to make sure my search was accurate (e.g. not robot comments about the jumbo builder running). I wanted to get a sense for how often people had to think about jumbo and do work to support it and didn't look at the positive side effects of it.

Erik

[Darin Fisher]

Hey folks,

I just want to clarify some things about this decision.

First off, on the technical side, my observation is that the jumbo build has created a bit of a landmine for people. Concatenating C++ files together can break basic assumptions programmers make about interactions across source files.

1- When you write code in an anonymous namespace, you don't expect it to have any interactions with code in other source files.

2- When you use preprocessor directives, you expect those to operate across a single source file, and you don't expect the macros you define to impact other source files unless you are putting them in a header file.

I trust you all will agree that these are pretty basic assumptions, and with jumbo build it is pretty clear that it is violating these basic assumptions. With #1, among other things it can end up causing the compiler to fail to resolve what symbol you are calling just because you happened to use the same name for an anonymous function as someone else. With #2, it can lead to errors like the one I described with zlib trampling over method names in completely unrelated code.

As a developer on this project, when you are faced with a compiler error, you hope to have a fighting chance of making sense of that compiler error by inspecting the code you are writing and the code around that code. However, with jumbo it is code from other C++ files that can interact poorly with the code you have written that then leads to very mysterious compiler errors. Did you know that there is another function in an anonymous namespace in some other random source file that happens to use the same descriptive name as you have chosen? Think of how hard it can be to debug template expansion issues. Now compound that with not knowing the set of inputs to the compiler!

From my perspective, it is imperative for this project that we fight to reduce landmines. I'm very committed to reducing the complexity associated with trying to work on Chromium. I see jumbo working counter to this goal, and that's why I have pushed for its deprecation.

Now, this was of course not a trivial decision. Jumbo helps people who do not have access to compile-in-the-cloud options to build Chromium in a more reasonable timeframe. I totally get the value of that, and as someone who really cares about developer productivity, I think it is important that we not be trite here. Fortunately, Goma is now available both as open source so others can stand up their own servers (and my understanding is that some people certainly have) but Google has also made available Goma servers for folks in the community to utilize. Repeating what Dirk wrote in his original post, if you are interested in getting access to this system, please fill out this form. By the way, we are working on making sure that this service is more discoverable.

Goma doesn't have to be the only answer of course. We should also be working to make Chromium less costly to build and to continue our efforts to make our tools faster and more efficient.

Thanks,

-Darin

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAN60yaJkRfAJ_fvzBkA7%2BbqruhmMB7c%3Dux-6FMxysBTx4ADQ_w%40mail.gmail.com.

[Nicholas Bishop]

I took a look at the Goma form linked above and was surprised by this item:

"I will not use the hosted Goma service for continuous integration or automated builds."

This seems to support the idea that Goma isn't quite ready for prime time yet?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAP0-QpvJjR9z%3DBmgBH9LzQG7s955PzW%2BfyC-BxkMm4xJgZHzTQ%40mail.gmail.com.

[Peter Kasting]

On Wed, Sep 25, 2019 at 2:26 PM Darin Fisher <da...@chromium.org> wrote:

From my perspective, it is imperative for this project that we fight to reduce landmines. I'm very committed to reducing the complexity associated with trying to work on Chromium. I see jumbo working counter to this goal, and that's why I have pushed for its deprecation.

Does that argument still apply if jumbo is an off-by-default config on an FYI bot?  At that point the primary impacts are

(a) Maintenance burden on the people who use it (which we have to see if it will be borne)

(b) Codebase impact from jumbo-supporting changes

The original argument was that making jumbo FYI was a non-starter because (b) was negative.  The argument on this thread was that (b) is actually a net positive.

Fortunately, Goma is now available both as open source so others can stand up their own servers (and my understanding is that some people certainly have) but Google has also made available Goma servers for folks in the community to utilize.

AIUI, these only apply to some platforms.  Notably, not Windows, by far the largest desktop browser market, for which there's no definitive timeframe yet.  It feels like there's great but as-yet-unrealized promise for goma as an externally-usable build accelerator.

Goma doesn't have to be the only answer of course. We should also be working to make Chromium less costly to build and to continue our efforts to make our tools faster and more efficient.

Sure, but I'm not aware of any efforts or even ideas on this front that have the sort of magnitude of jumbo.  The closest I can think of is some of the apparently-volunteer-led effort to reduce unnecessary #includes in core files, and the wins from that are far smaller.

PK

[Dirk Pranke]

That's referring to automated builds from a system other than the one we're hosting. I.e.,. we're not granting you access to goma for your own CI system, only for using as part of ours.

-- Dirk

[Matt Giuca]

On Thu, 26 Sep 2019 at 08:42, Peter Kasting <pkas...@google.com> wrote:

On Wed, Sep 25, 2019 at 2:26 PM Darin Fisher <da...@chromium.org> wrote:

From my perspective, it is imperative for this project that we fight to reduce landmines. I'm very committed to reducing the complexity associated with trying to work on Chromium. I see jumbo working counter to this goal, and that's why I have pushed for its deprecation.

Does that argument still apply if jumbo is an off-by-default config on an FYI bot?  At that point the primary impacts are

(a) Maintenance burden on the people who use it (which we have to see if it will be borne)

(b) Codebase impact from jumbo-supporting changes

The original argument was that making jumbo FYI was a non-starter because (b) was negative.  The argument on this thread was that (b) is actually a net positive.

On (b), I wrote an email to this thread dated August 19 where I looked at a handful of CLs doing renames to support Jumbo, showing I think clear examples of taking names that are perfectly clear in context being renamed to overly long names that are redundant given the context (e.g. "LoadProtoFromDisk" in a file called safety_tips_component_installer.cc renamed to "LoadSafetyTipsProtoFromDisk"). In the worst case, the renames are done by someone who doesn't have an understanding of the code they're changing (they are just trying to quickly fix a broken Jumbo build), and the new names are misleading, as in my second example renaming "DoesAndroidSupportMaskableIcons" to "DoesAndroidSupportMaskableIconsForWebApk".

Obviously these are some cherry-picked examples, and we could have a table side-by-side of renames that improved the clarity of the code versus renames that made it worse. I just want to point out that having globally unique names doesn't always make the code clearer. Maybe it is still a net positive.

Since Jumbo builds effectively nullify the utility of anonymous namespaces, I think we should decide to have one or the other. It doesn't make sense allowing anonymous namespaces in the codebase if every time we commit code where two objects in different files have the same name, someone comes along with a CL to rename them apart. If we do decide that it's a net positive to force all names to be globally unique, we should just outlaw anonymous namespaces; then the name collision errors come from the C++ language and not from an FYI bot later on. The corollary is that if we think anonymous namespaces are a valuable enough language feature to allow in our codebase, then we shouldn't accept CLs for the sole purpose of renaming everything in them to be globally unique.

[Colin Blundell]

To provide another datapoint on "the community's opinions on the matter":

I strongly +1 Darin's expressed comments here. I expressed similar discomfort with the jumbo build on a chromium-dev@ thread some time ago, and that discomfort has never left. My position is the same now as it was then: I very much appreciate the problem that those whom the jumbo build helps are facing, but I don't think that mitigating that problem should come at the cost of introducing unexpected C++ language complexity for developers who have to work with this already amazingly complex and baroque codebase. I believe that introducing the violation of the basic assumptions that Darin describes below has a long-term cost for all of Chromium in violating "principle of least surprise", even if the enforcement of jumbo is only done via an FYI bot. It's still the case that jumbo is a concept that developers would encounter from time to time in various ways, need to internalize, and take in as one more element of Chromium-specific unexpected complexity. We already have too many of these as it is!

Best,

Colin

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAP0-QpvJjR9z%3DBmgBH9LzQG7s955PzW%2BfyC-BxkMm4xJgZHzTQ%40mail.gmail.com.

[Darin Fisher]

On Wed, Sep 25, 2019 at 5:31 PM Matt Giuca <mgi...@chromium.org> wrote:

On Thu, 26 Sep 2019 at 08:42, Peter Kasting <pkas...@google.com> wrote:

On Wed, Sep 25, 2019 at 2:26 PM Darin Fisher <da...@chromium.org> wrote:

From my perspective, it is imperative for this project that we fight to reduce landmines. I'm very committed to reducing the complexity associated with trying to work on Chromium. I see jumbo working counter to this goal, and that's why I have pushed for its deprecation.

Does that argument still apply if jumbo is an off-by-default config on an FYI bot?  At that point the primary impacts are

(a) Maintenance burden on the people who use it (which we have to see if it will be borne)

(b) Codebase impact from jumbo-supporting changes

The original argument was that making jumbo FYI was a non-starter because (b) was negative.  The argument on this thread was that (b) is actually a net positive.

On (b), I wrote an email to this thread dated August 19 where I looked at a handful of CLs doing renames to support Jumbo, showing I think clear examples of taking names that are perfectly clear in context being renamed to overly long names that are redundant given the context (e.g. "LoadProtoFromDisk" in a file called safety_tips_component_installer.cc renamed to "LoadSafetyTipsProtoFromDisk"). In the worst case, the renames are done by someone who doesn't have an understanding of the code they're changing (they are just trying to quickly fix a broken Jumbo build), and the new names are misleading, as in my second example renaming "DoesAndroidSupportMaskableIcons" to "DoesAndroidSupportMaskableIconsForWebApk".

Obviously these are some cherry-picked examples, and we could have a table side-by-side of renames that improved the clarity of the code versus renames that made it worse. I just want to point out that having globally unique names doesn't always make the code clearer. Maybe it is still a net positive.

Since Jumbo builds effectively nullify the utility of anonymous namespaces, I think we should decide to have one or the other. It doesn't make sense allowing anonymous namespaces in the codebase if every time we commit code where two objects in different files have the same name, someone comes along with a CL to rename them apart. If we do decide that it's a net positive to force all names to be globally unique, we should just outlaw anonymous namespaces; then the name collision errors come from the C++ language and not from an FYI bot later on. The corollary is that if we think anonymous namespaces are a valuable enough language feature to allow in our codebase, then we shouldn't accept CLs for the sole purpose of renaming everything in them to be globally unique.

Well said! Of course, anonymous namespaces are useful for more than just organization of code. Static function calls can also be better optimized by our tools, so this really is a false choice. We simply aren't going to do away with anonymous namespaces.

Thanks,

-Darin

[Daniel Bratell]

I looked through the list of jumbo related commits when this thread was initially posted but I will repeat what I said back then since it's been a month:

The list *greatly* overestimates the amount of maintenance jumbo needs. Only 10-20% of the commits listed were doing immediate fixes needed for jumbo. The rest were commits such as work on variable shadowing that mentioned jumbo as a potential benefactor of fewer shadowed variables or commits done to support non-default jumbo configurations (specifically an extreme test I had where I compiled as much code as possible together).

If jumbo maintenance is a problem, it would be good to first try to reduce the maintenance cost but no such attempt was done. At least none that involved me. I only learned of this decision and urgent concern that jumbo was too expensive after the decision was made which too me made it look like a decision made in haste. That the supporting data also did not support the decision only strengthens that belief.

I'm not claiming that jumbo is a zero cost feature. Absolutely not. I've done dozens of commits the last couple of years to fix jumbo breakage on untested platforms/configurations, but that is very little compared to the time it has saved, even for me as the main maintainer.

And goma is better, but it doesn't seem to be ready yet. It has to be ready and available to be a replacement. As far as I know, the open source server is very very hard to get running, and neither Mac nor Windows support is there yet.

I had hoped that things would have changed since during the delay but I've seen no signs of that at all and therefore I think it makes sense to push it further until the replacement is available.

/Daniel

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAHOzFCZisamAPLHJ_Lz9GW_n8DO266kw5n7WqAhL2hoeLNzcQ%40mail.gmail.com.

[Marshall Greenblatt]

Hi Darin,

Goma is currently not supported on Windows for external contributors. What build configuration would you recommend for those contributors now that Jumbo support is being removed?

Regards,

Marshall

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAP0-QptkJVGggJjc5NSjQRYrhVEH1RWQs8eB4qPioWazWVCkZA%40mail.gmail.com.

[Samuel Attard]

Same question for macOS as well. Neither are supported.

[Darin Fisher]

That's right. Unfortunately, support is limited to Linux at this time. I realize that is an inconvenience for folks building on other platforms.

Regards,

-Darin

[Daniel Bratell]

I think that is a bit of an understatement.

This doubles, triples or quadruples the compile time for people working on Mac and Windows, and then it was insanely long from the beginning. It's especially frustrating since Mac and Windows are the platforms that suffer from the weakest generic build acceleration tools.

It seems to me that things are done in the wrong order. First the replacement, then the removal of the old system, that would be the recommended order.

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAP0-Qpt8Ho5aBiXYbhNPt2N-d-CwucuJbDq0Mk_ZqDjdOLX6%2BQ%40mail.gmail.com.

[Menard, Alexis]

Hi,

And Windows is an important platform for Chrome and various companies here. We have very few alternatives :

- Throwing several thousands dollars to get an average build time with a beefy computer

- Throwing even more expensive dollars into systems like incredibuild.

For macOS it’s even worst since buying powerful hardware is prohibitive. However we mitigated this with icecream and cross compilation on Linux nodes. It’s not optimal and icecream has a lot of limitations but it does the job.

I think ideally the community would love to see a roadmap or plan for GOMA not just for public GOMA but similarly for companies that would be interested to deploy their own internal GOMA (therefore we would appreciate if Google shares the pieces to get an head start to support sharing the build on Windows or MacOS).

Thanks.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/2c910beb-aee1-22ba-add2-b29890a2c4d4%40gmail.com.

[Darin Fisher]

On Wed, Oct 9, 2019 at 8:57 AM Menard, Alexis <alexis...@intel.com> wrote:

Hi,

And Windows is an important platform for Chrome and various companies here. We have very few alternatives :

- Throwing several thousands dollars to get an average build time with a beefy computer

- Throwing even more expensive dollars into systems like incredibuild.

For macOS it’s even worst since buying powerful hardware is prohibitive. However we mitigated this with icecream and cross compilation on Linux nodes. It’s not optimal and icecream has a lot of limitations but it does the job.

Hey Alexis,

Yeah, so true. Windows and macOS are both super important platforms, and building Chrome for those does take a lot of CPU time.

 

I think ideally the community would love to see a roadmap or plan for GOMA not just for public GOMA but similarly for companies that would be interested to deploy their own internal GOMA (therefore we would appreciate if Google shares the pieces to get an head start to support sharing the build on Windows or MacOS).

My understanding is that other companies have taken GOMA source code and implemented support for other platforms. I would love to see others sharing with the community as well.

I also see opportunities for us as a community to invest further in changes that make the edit-debug cycle more efficient. We've invested a bunch in LLVM performance for this reason. Maybe we should be looking into making more and smaller components so that the component build can be faster to work with. I realize a full clean build is expensive no matter what :-/

Regards,

-Darin

[Marcin Wiącek]

Dear Dirk,

When I started using Jumbo, compiling cost on laptop decreased from 4h to about 1h.

Currently I see 81_000 files without Jumbo.

This is more than sad, that instead of cleaning project rules (I mean - providing such rules, which could decrease amount of conflicts you were mentioning) and instead of making Jumbo more widely used in more configs it was just removed.

I understand, that complexity of project grows, but if (I will repeat: if) Google needs help from community with this, I'm sure, that asking for this was more than enough and you could do it.

You should think also about environmental effect of your decision - time, energy usage, etc.

This decision unfortunately means, that worse is replacing better and I'm thinking, if you just lack people in team or you don't have resources or what is real reason of this decision (key word - real).

With kind regards,

Marcin Wiącek

[Matt Giuca]

On Sun, 17 Nov 2019 at 12:41, Marcin Wiącek <mar...@mwiacek.com> wrote:

Dear Dirk,

When I started using Jumbo, compiling cost on laptop decreased from 4h to about 1h.

Currently I see 81_000 files without Jumbo.

This is more than sad, that instead of cleaning project rules (I mean - providing such rules, which could decrease amount of conflicts you were mentioning) and instead of making Jumbo more widely used in more configs it was just removed.

I understand, that complexity of project grows, but if (I will repeat: if) Google needs help from community with this, I'm sure, that asking for this was more than enough and you could do it.

I just want to highlight (based on the long conversation above), this is not a question of insufficient resources to enforce the rules Jumbo requires (e.g., names in anonymous namespaces and preprocessor directives cannot conflict even across files), it's that we don't want our codebase to be constrained by those rules.

 

You should think also about environmental effect of your decision - time, energy usage, etc.

This decision unfortunately means, that worse is replacing better and I'm thinking, if you just lack people in team or you don't have resources or what is real reason of this decision (key word - real).

Have a look at the email from Darin in this thread dated 26 September (beginning "I just want to clarify some things about this decision"). That explains a bunch of the problems that arise from the Jumbo build. (The short version is, C++ isn't designed for this; Jumbo violates the rules of C++ causing compiler errors and in some cases behavioural differences which can catch engineers off guard if they are using a regular C++ compiler toolchain.)

With kind regards,

Marcin Wiącek

On Thursday, August 15, 2019 at 11:25:02 PM UTC+2, Dirk Pranke wrote:

Hi all,

We have decided to remove support for the jumbo (aka unity) build configuration from the project.

We are very conscious of the fact that Chromium is very expensive and slow to compile, and that that represents a real barrier to contributing.

However, based on the experience we've gained with this configuration over the past many months it's been supported, we've reached the conclusion that the way jumbo is currently implemented, it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary. As a result, we feel that the cost to the project of maintaining jumbo is not worth the benefit it is providing.

We continue to invest in other approaches to reducing build time. We've released the Goma build accelerator code and hope to continue making it easier to use for others. We are also working to enable our Goma clusters for external contributors (anyone who has tryjob access) who don't have their own build acceleration tools. We're starting with Linux support now and will have more to share on that soon. If you are interested in getting access to this system, please fill out this form. We're also working on Clang and LLVM and have made significant improvements this year to cycle time as a result, and we expect to continue to do so for the foreseeable future.

We plan to remove support a week from now, on August 22nd.

Please let me know if you have questions about this.

-- Dirk

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/56591d89-3b1f-4c26-8cb5-aef193178cab%40chromium.org.

[Marcin Wiącek]

Hi Matt,

I just want to highlight (based on the long conversation above), this is not a question of insufficient resources to enforce the rules Jumbo requires (e.g., names in anonymous namespaces and preprocessor directives cannot conflict even across files), it's that we don't want our codebase to be constrained by those rules.

 

Have a look at the email from Darin in this thread dated 26 September (beginning "I just want to clarify some things about this decision"). That explains a bunch of the problems that arise from the Jumbo build. (The short version is, C++ isn't designed for this; Jumbo violates the rules of C++ causing compiler errors and in some cases behavioural differences which can catch engineers off guard if they are using a regular C++ compiler toolchain.)

 

C++ is "as is" & Chrome is "as is".

and now - how many times per day code is compiled in Google and how many times is compiled by external people?

I guess, thousands or million times. This is energy & this is time.

Disabling option, which decreased compiling from 4h to 1h is moving project into stone age and from such lead market projects it's expected more that saying "go into cloud" (again: CO2 footprint !!!!!!!!!) or "we're thinking about other options".

If Jumbo didn't work, what could work ? For example: is it possible to make things modular in other way?

[Matt Giuca]

I understand the nature of the problem. I think we agree it's a problem, but the discussion above reflects the issues with Jumbo as a solution (that it violates the rules of C++ as a language).

There are engineering trade-offs here: Jumbo solves one problem and introduces another. The decision was made to drop support for Jumbo because the new problems that it created were unacceptable. I'm not the decision maker. I'm not here to argue for or against Jumbo. I'm just reiterating what has already been stated above, to clarify why the decision was made.

I would personally like to see Clank support this sort of thing directly. At the end of the day, this is a problem with the C++ language / compiler infrastructure, not with the Chrome build infrastructure. Chrome just happens to be one very large C++ project. A compiler should be able to do what Jumbo does (compile 1000s of C++ files into a single object file), while respecting the semantics of the language, as opposed to literally concatenating the files together before feeding them into the compiler, which means we're asking the compiler to compile a different C++ program than the one our engineers are writing (with who-knows-what subtle differences).

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/215622cb-b257-4fa7-ae4e-dc6947db43d3%40chromium.org.

[Marcin Wiącek]

Some things didn't happen in one moment & right now many people are left with problem (once again: cloud is not solution).

Decision was made, because somebody didn't maybe start reorganizing project structure on time or something similar (in other words: technically it was too complicated or nobody cared). It didn't happen and it shows, that this project doesn't care about some aspects.

Also sorry to say, but removing something without giving real alternative is failure.

PS. It would be fine, when we will see announcement - in 12 months we will do this, this, this (but concrete solutions please working even on laptop). I really ask Google people here for stopping with these secrets - this is not helpful.

To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

[Dirk Pranke]

Hi Marcin,

You made several of the same points on the bug and I just responded to them there. Matt has also responded to several of them here, so I won't repeat what Matt said, but I will repeat some of my answers on the bug for the benefit of those not following along on the bug ...

On Sat, Nov 16, 2019 at 5:42 PM Marcin Wiącek <mar...@mwiacek.com> wrote:

Dear Dirk,

When I started using Jumbo, compiling cost on laptop decreased from 4h to about 1h.

Currently I see 81_000 files without Jumbo.

This is more than sad, that instead of cleaning project rules (I mean - providing such rules, which could decrease amount of conflicts you were mentioning) and instead of making Jumbo more widely used in more configs it was just removed.

Making jumbo more widely used in more configs wouldn't have helped anything; if anything, it would've increased the cost of supporting jumbo slightly (e.g., we never turned it on for the CrOS configs, so we weren't having to work around CrOS-specific conflicts).

I understand, that complexity of project grows, but if (I will repeat: if) Google needs help from community with this, I'm sure, that asking for this was more than enough and you could do it.

You should think also about environmental effect of your decision - time, energy usage, etc.

As I mentioned previously on the thread, the potential for environmental savings from jumbo was one of the main things I was interested in when we started pursuing it. As best I could tell, though, it was a net loss. We can't tell who all is compiling Chromium worldwide, but it seems like a safe guess that most compilation is happening on Google machines (thanks to the enormous overall cost of our CI system) which uses Goma as well, and Goma+jumbo seems to be a net loss because using jumbo decreases the cache hit rate we get from Goma, resulting in more overall work.

Best,

-- Dirk

This decision unfortunately means, that worse is replacing better and I'm thinking, if you just lack people in team or you don't have resources or what is real reason of this decision (key word - real).

With kind regards,

Marcin Wiącek

On Thursday, August 15, 2019 at 11:25:02 PM UTC+2, Dirk Pranke wrote:

Hi all,

We have decided to remove support for the jumbo (aka unity) build configuration from the project.

We are very conscious of the fact that Chromium is very expensive and slow to compile, and that that represents a real barrier to contributing.

However, based on the experience we've gained with this configuration over the past many months it's been supported, we've reached the conclusion that the way jumbo is currently implemented, it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary. As a result, we feel that the cost to the project of maintaining jumbo is not worth the benefit it is providing.

We continue to invest in other approaches to reducing build time. We've released the Goma build accelerator code and hope to continue making it easier to use for others. We are also working to enable our Goma clusters for external contributors (anyone who has tryjob access) who don't have their own build acceleration tools. We're starting with Linux support now and will have more to share on that soon. If you are interested in getting access to this system, please fill out this form. We're also working on Clang and LLVM and have made significant improvements this year to cycle time as a result, and we expect to continue to do so for the foreseeable future.

We plan to remove support a week from now, on August 22nd.

Please let me know if you have questions about this.

-- Dirk

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/56591d89-3b1f-4c26-8cb5-aef193178cab%40chromium.org.

[Hans Wennborg]

Just wanted to reply to the "I would personally like to see Clank
support this sort of thing directly" part. (I assume you meant Clang
and not Clank :-)

We (the Chrome toolchain team) looked into it, but the combination of
re-using the redundant parsing and semantic analysis across different
source files, while at the same time keeping them apart enough to
respect C++ semantics is hard, and something we're working on at the
moment.

But we are well aware the build times for Chromium are really bad, and
reducing them is a top priority for us.

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAHqYdcb44cSzVe%3DtBw0TTd099ZXYFODP7AZWGScM0rxgn2Zr3A%40mail.gmail.com.

[Daniel Bratell]

There was a small patch, I think two years ago now, which created a
mechanism for "resetting" the anonymous namespace. By using that between
each translation unit in a jumbo chunk, you avoided almost all
compatibility problems. The patch was small but I think it was
considered non-compatible with the path people wanted clang to take.
Mostyn will know more about it.

That would be a more maintainable alternative to building full support
for parallel compilation of translation units into clang.

/Daniel

[Mostyn Bramley-Moore]

On Tue, Nov 26, 2019 at 10:34 PM Daniel Bratell <brat...@gmail.com> wrote:

There was a small patch, I think two years ago now, which created a
mechanism for "resetting" the anonymous namespace. By using that between
each translation unit in a jumbo chunk, you avoided almost all
compatibility problems. The patch was small but I think it was
considered non-compatible with the path people wanted clang to take.
Mostyn will know more about it.

That would be a more maintainable alternative to building full support
for parallel compilation of translation units into clang.

/Daniel

I linked to the clang-dev thread and my lightning talk slides earlier in this thread:

https://groups.google.com/a/chromium.org/d/msg/chromium-dev/DP9TQszzQLI/vuESMKTmDAAJ

The clang plugin + clang patch only needed two trivial updates to apply and build on the tip of master just now:

https://github.com/mostynb/llvm-project/commit/c4c041491edac606e07b5b5027d19ff61973e754

This was my test case, making //net build in jumbo mode:

Without the plugin: https://chromium-review.googlesource.com/c/chromium/src/+/966523

With the plugin: https://chromium-review.googlesource.com/c/chromium/src/+/962062

On 2019-11-26 12:30, Hans Wennborg wrote:
> Just wanted to reply to the "I would personally like to see Clank
> support this sort of thing directly" part. (I assume you meant Clang
> and not Clank :-)
>
> We (the Chrome toolchain team) looked into it, but the combination of
> re-using the redundant parsing and semantic analysis across different
> source files, while at the same time keeping them apart enough to
> respect C++ semantics is hard, and something we're working on at the
> moment.
>
> But we are well aware the build times for Chromium are really bad, and
> reducing them is a top priority for us.

Do you have a roadmap and/or some way for external contributors to help? 

-Mostyn.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/869f7944-fc38-be23-b1b8-da78a130a219%40gmail.com.

--

Mostyn Bramley-Moore

Vewd Software

mos...@vewd.com

[Matt Giuca]

Yes, I did mean Clang :)

On Wed, 27 Nov 2019 at 10:42, Mostyn Bramley-Moore <mos...@vewd.com> wrote:

On Tue, Nov 26, 2019 at 10:34 PM Daniel Bratell <brat...@gmail.com> wrote:

There was a small patch, I think two years ago now, which created a
mechanism for "resetting" the anonymous namespace. By using that between
each translation unit in a jumbo chunk, you avoided almost all
compatibility problems. The patch was small but I think it was
considered non-compatible with the path people wanted clang to take.
Mostyn will know more about it.

That would be a more maintainable alternative to building full support
for parallel compilation of translation units into clang.

/Daniel

I linked to the clang-dev thread and my lightning talk slides earlier in this thread:

https://groups.google.com/a/chromium.org/d/msg/chromium-dev/DP9TQszzQLI/vuESMKTmDAAJ

The clang plugin + clang patch only needed two trivial updates to apply and build on the tip of master just now:

https://github.com/mostynb/llvm-project/commit/c4c041491edac606e07b5b5027d19ff61973e754

That's pretty great!

This was my test case, making //net build in jumbo mode:

Without the plugin: https://chromium-review.googlesource.com/c/chromium/src/+/966523

With the plugin: https://chromium-review.googlesource.com/c/chromium/src/+/962062

I'm curious that changes are still required. Looks like your Clang patch means the compiler can deal with name conflicts in anonymous namespaces and macro definitions, but you still need to make some changes to Chrome to make Jumbo builds happy.

Even though you have to make much fewer changes, having to make any changes could still be a problem. The whole problem here is that engineers who aren't building with Jumbo may not realise they're breaking the Jumbo build, because Jumbo imposes requirements on Chrome outside of the C++ language. Ideally, it would be a transparent compiler optimization.

It looks like, for example in crypto_secret_boxer.cc you had to rename the static const kAEAD due to a name conflict? So perhaps Clang doesn't support static name conflicts across files, unlike anonymous namespaces. You also had to add some #includes and add a reinterpret_cast; I'm not sure why those would have been necessary.

Perhaps you can make a list of "ways in which Clang + your patch + Jumbo build violates C++ semantics" and work on burning those down to 0.

[Mostyn Bramley-Moore]

On Thu, Nov 28, 2019 at 12:23 AM Matt Giuca <mgi...@chromium.org> wrote:

Yes, I did mean Clang :)

On Wed, 27 Nov 2019 at 10:42, Mostyn Bramley-Moore <mos...@vewd.com> wrote:

On Tue, Nov 26, 2019 at 10:34 PM Daniel Bratell <brat...@gmail.com> wrote:

There was a small patch, I think two years ago now, which created a
mechanism for "resetting" the anonymous namespace. By using that between
each translation unit in a jumbo chunk, you avoided almost all
compatibility problems. The patch was small but I think it was
considered non-compatible with the path people wanted clang to take.
Mostyn will know more about it.

That would be a more maintainable alternative to building full support
for parallel compilation of translation units into clang.

/Daniel

I linked to the clang-dev thread and my lightning talk slides earlier in this thread:

https://groups.google.com/a/chromium.org/d/msg/chromium-dev/DP9TQszzQLI/vuESMKTmDAAJ

The clang plugin + clang patch only needed two trivial updates to apply and build on the tip of master just now:

https://github.com/mostynb/llvm-project/commit/c4c041491edac606e07b5b5027d19ff61973e754

That's pretty great!

This was my test case, making //net build in jumbo mode:

Without the plugin: https://chromium-review.googlesource.com/c/chromium/src/+/966523

With the plugin: https://chromium-review.googlesource.com/c/chromium/src/+/962062

I'm curious that changes are still required. Looks like your Clang patch means the compiler can deal with name conflicts in anonymous namespaces and macro definitions, but you still need to make some changes to Chrome to make Jumbo builds happy.

Even though you have to make much fewer changes, having to make any changes could still be a problem. The whole problem here is that engineers who aren't building with Jumbo may not realise they're breaking the Jumbo build, because Jumbo imposes requirements on Chrome outside of the C++ language. Ideally, it would be a transparent compiler optimization.

It looks like, for example in crypto_secret_boxer.cc you had to rename the static const kAEAD due to a name conflict? So perhaps Clang doesn't support static name conflicts across files, unlike anonymous namespaces. You also had to add some #includes and add a reinterpret_cast; I'm not sure why those would have been necessary.

I went back and rebuilt everything to refresh my memory. These were the things that needed fixing when using the JumboSupport plugin:

1) net/if.h linux/if.h conflict: these headers come from different places (glibc and the linux kernel) and can be incompatible unless included in a specific order. This was also the reason for the reinterpet_cast. Yuck. There is a simpler solution than in the original CL however, remove a single unneeded net/if.h include:

https://chromium-review.googlesource.com/c/chromium/src/+/1944847

2) Remove "using base::File;" directive and update all the places with the missing "base::" in net/disk_cache/simple/simple_synchronous_entry.cc. This is required because there's a disk_cache::File type in net/disk_cache/blockfile/file.h. Another solution might be to give this a better name, eg disk_cache::AsyncFile, based on the class comment.

3) Rename LoadState function in an anonymous namespace to ReadLoadState in net/http/transport_security_persister.cc. This is required because there's a net::LoadState enum in net/base/load_states.h.

4) Rename kNonceSize in net/quic/core/crypto/crypto_protocol.h to kQuicNonceSize. This is required because it is in the net namespace, and there are other kNonceSize consts in anonymous namespaces in this component which become ambiguous if this header is included. Another solution might be to introduce a net::quic namespace.

5) Rename the kAEAD function pointer in net/quic/core/crypto/crypto_secret_boxer.cc. This is required because there is a net::kAEAD constant in net/quic/core/crypto/crypto_protocol.h. It feels like a Google C++ style violation to use a "kFoo" name for something that isn't a const or constexpr variable anyway.

IMO these changes are probably a net win for readability and future refactoring, even without considering jumbo.

Perhaps you can make a list of "ways in which Clang + your patch + Jumbo build violates C++ semantics" and work on burning those down to 0.

Jumbo builds themselves don't change C++ semantics, they just change the size of the compilation unit and increase the chance of conflicts and ambiguity between code that is normally compiled separately. Jumbo+this plugin only change the semantics of two things, but in such a way that is consistent with non-jumbo builds (limit anonymous namespaces and macros defined in top-level source files to a pretend top-level-file scope).

There are lots of ways that code can "leak" from one file to another in "legitimate" ways, and real code depends on this behaviour. So I don't think jumbo builds can ever be 100% automatic, at least not without re-parsing header files for each "top-level" source file- which (without major changes to Clang) would be slow and defeat the entire purpose of jumbo builds.

A more realistic goal for jumbo might be to exist as a second-class build configuration, where patches are accepted as long as they don't make the existing code significantly worse (also for Chromium's Clang toolchain itself, since tooling depends on downloading Clang binaries with statically linked plugins).

-Mostyn.

[Matt Giuca]

Sure, they probably are, but that doesn't change the fact that they're necessary, which means you're creating a secondary language "C++ with Jumbo" which has different semantics to C++. That means when the rest of us write regular C++ code, we might break the "C++ with Jumbo" build which is compiling a different programming language to what we're writing.

Also, as I've argued way back earlier in the year (on this thread), I don't think changes like #4 are good. Having to change a variable called "kNonceSize" which is perfectly meaningful within the context of this one file, to "kQuicNonceSize" adds redundant information to the variable name, which doesn't benefit the readability of the code. You're just doing that because of a cross-file naming conflict.

Perhaps you can make a list of "ways in which Clang + your patch + Jumbo build violates C++ semantics" and work on burning those down to 0.

Jumbo builds themselves don't change C++ semantics, they just change the size of the compilation unit and increase the chance of conflicts and ambiguity between code that is normally compiled separately. Jumbo+this plugin only change the semantics of two things, but in such a way that is consistent with non-jumbo builds (limit anonymous namespaces and macros defined in top-level source files to a pretend top-level-file scope).

That is changing the semantics though. The C++ language specification defines what a compilation unit is, and specifies under what circumstances two variables with the same name are allowed, and under what circumstances they aren't. Jumbo is modifying the compiler toolchain to change the definition of a compilation unit, and thus decreasing the set of circumstances where name collisions are allowed. In effect, Jumbo creates a new programming language that is very very similar to C++, but not quite the same.

Your patch seems to narrow the gap, moving the language closer and closer to C++, but still not quite the same.

The fundamental problem is that we (the Chromium authors overall) want to write C++, not some very slightly different language. So ideally we would be able to modify Clang so that it compiles the project much faster, but with the exact same semantics as C++ as specified.

[Marcin Wiącek]

Sure, they probably are, but that doesn't change the fact that they're necessary, which means you're creating a secondary language "C++ with Jumbo" which has different semantics to C++. That means when the rest of us write regular C++ code, we might break the "C++ with Jumbo" build which is compiling a different programming language to what we're writing.

Also, as I've argued way back earlier in the year (on this thread), I don't think changes like #4 are good. Having to change a variable called "kNonceSize" which is perfectly meaningful within the context of this one file, to "kQuicNonceSize" adds redundant information to the variable name, which doesn't benefit the readability of the code. You're just doing that because of a cross-file naming conflict.

Perhaps you can make a list of "ways in which Clang + your patch + Jumbo build violates C++ semantics" and work on burning those down to 0.

Jumbo builds themselves don't change C++ semantics, they just change the size of the compilation unit and increase the chance of conflicts and ambiguity between code that is normally compiled separately. Jumbo+this plugin only change the semantics of two things, but in such a way that is consistent with non-jumbo builds (limit anonymous namespaces and macros defined in top-level source files to a pretend top-level-file scope).

That is changing the semantics though. The C++ language specification defines what a compilation unit is, and specifies under what circumstances two variables with the same name are allowed, and under what circumstances they aren't. Jumbo is modifying the compiler toolchain to change the definition of a compilation unit, and thus decreasing the set of circumstances where name collisions are allowed. In effect, Jumbo creates a new programming language that is very very similar to C++, but not quite the same.

Your patch seems to narrow the gap, moving the language closer and closer to C++, but still not quite the same.

The fundamental problem is that we (the Chromium authors overall) want to write C++, not some very slightly different language. So ideally we would be able to modify Clang so that it compiles the project much faster, but with the exact same semantics as C++ as specified.

In big projects there are always limitations (variable names, non-using some unsafe language abilities, having compatibility with some versions or compilers, etc.) and you can't use whole language.

Somebody didn't resolve problem on time, removed something working and people are left with non-economic solution - if this one works, please apply it and work on something better.

I don't know how long modifying Clang needs, but this seems to be ready. Solution is required today and please don't stop progress and nice tries of resolving this bug problem.

[dan...@chromium.org]

Do you think that if it was originally written as kQuicNonceSize in a CL that the reviewer would object and have it changed? That seems well below the bar of things to care about to me personally. There are a couple other kNonceSize written with additional context in their name which also seem.. fine? https://cs.chromium.org/search/?q=f:%5Esrc+k%5BA-Z%5D%2B%5Ba-z%5D*NonceSize&sq=package:chromium&type=cs

 

Perhaps you can make a list of "ways in which Clang + your patch + Jumbo build violates C++ semantics" and work on burning those down to 0.

Jumbo builds themselves don't change C++ semantics, they just change the size of the compilation unit and increase the chance of conflicts and ambiguity between code that is normally compiled separately. Jumbo+this plugin only change the semantics of two things, but in such a way that is consistent with non-jumbo builds (limit anonymous namespaces and macros defined in top-level source files to a pretend top-level-file scope).

That is changing the semantics though. The C++ language specification defines what a compilation unit is, and specifies under what circumstances two variables with the same name are allowed, and under what circumstances they aren't. Jumbo is modifying the compiler toolchain to change the definition of a compilation unit, and thus decreasing the set of circumstances where name collisions are allowed. In effect, Jumbo creates a new programming language that is very very similar to C++, but not quite the same.

Your patch seems to narrow the gap, moving the language closer and closer to C++, but still not quite the same.

The fundamental problem is that we (the Chromium authors overall) want to write C++, not some very slightly different language. So ideally we would be able to modify Clang so that it compiles the project much faster, but with the exact same semantics as C++ as specified.

I want to write a more narrow language than "C++". The style guide restricts what we can do, causing us to write a slightly different language, and I am glad. :) We also choose a single version of C++, and that version changes over time. When it does we all have to learn far more difficult rules and restrictions that those imposed by jumbo. As much as I want and think we should continue to use move semantics, I am confident that we have got that wrong in the code in far more cases and in more harmful ways than what jumbo would impose on us for example. At least it tells us when we do it wrong. :)

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAHqYdcb1x1gBs%3D%2Bisqz%3Dxx6rA8UqKZpye6F9WM-HnqMuYDoAdA%40mail.gmail.com.

[Igor Bukanov]

On Thursday, December 5, 2019 at 4:22:33 AM UTC+1, Matt Giuca wrote:

The fundamental problem is that we (the Chromium authors overall) want to write C++, not some very slightly different language. So ideally we would be able to modify Clang so that it compiles the project much faster, but with the exact same semantics as C++ as specified.

Chromium is not written in C++. At best one can say it is written in C++ with exceptions and RTTI. Or consider the ban on private inheritance. This makes few useful C++ code patterns not available in Chromium and one has to learn to live without them.    

I understand that Jumbo caused problems for Google. But do not claim that it was unacceptable to restrict C++ usage. The style guide places far bigger restrictions on C++ code for various reasons including performance and compile time than what Jambo ever did. If anything I suspect that the real problem with Jambo is that for the vast majority of code it just works. Thus restrictions that it placed on the code were never mentioned in the style guide or other documentations. That in turn made it harder to deal with errors caused by it as one never heard about the feature and its consequences for the code until one experienced the error.     

And please do not claim that Jambo changes the semantics of code. Jambo restricts C++ to its subset, but as long as one stays within that subset, the semantic of the code is the same with or without Jambo. 

[Marcin Wiącek]

Good described, unfortunately this looks currently like saying "How dare you" to world leaders - although there was nothing better created than Jumbo, Jumbo was removed. Point. Decision was done (even without analysis or plan B).

It looks, that arguments like "this is problem for dependent projects and a lot of more for resources usage" are not important anymore. SWE simply don't accept anything because it will mean that this decision was wrong. Shame especially because some of them (these which I known from patches work) are very smart. And shame because of this CO2, which we will get thx to it.

[Marcin Wiącek]

Post-Mortem:

I've just submitted P1, because Canary doesn't open ANY URL (https://bugs.chromium.org/p/chromium/issues/detail?id=1034233). This is second version in Google Play, which went with this issue.Yesterday I proved, that there is regression in Site Settings screen (https://bugs.chromium.org/p/chromium/issues/detail?id=966659#c9). Also my proposal for doing very helpful (small) change in history screen waits for some decisions from May (https://bugs.chromium.org/p/chromium/issues/detail?id=962309). There are also others (I have seen clear info, that devs don't want consistency in code, etc. etc.)

Chrome, where are you going? Do you really need fresh blood in team?

[John Abd-El-Malek]

There are incorrect assumptions in the last few emails; I encourage you to spend more time reading the earlier messages so we don't go in circles.

I'm replying mostly to point you at Chromium's Code of Conduct; emails like this and recent comments in bugs don't meet it.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/179f78b5-77d2-42c4-8af8-bf95be1068d7%40chromium.org.

[Marcin Wiącek]

I will answer in technical way - things are done correctly or not, there is no "maybe".

In this concrete case a lot is confirming, that Jumbo decision was made without considering, how much ecosystem will be affected... also somebody is proposing here some other solution and it looks, that it's already rejected.

I don't know if you know, that we have P0 over last hours (https://bugs.chromium.org/p/chromium/issues/detail?id=1033655), I've seen also P1, where Canary can't open any url (https://bugs.chromium.org/p/chromium/issues/detail?id=1034233).

Project definitely needs exploratory testing done by experienced QAs in addition to simple manual and automatic testing; not only me is writing comments, that bugs are not touched or addressed in effective way; etc.

I don't know how to address these things to responsible managers now, but some project decisions are simply not working (and they're affecting thousands of people), it's also obvious, that discussion about can be difficult and I strongly believe, that CoC was not broken in any way - it was not my intention.

Returning to initial sentence - things are done correctly or not, there is no "maybe"; in case of Jumbo it would be good to see at least some plan with schedule - for example in January 2020 we will try this and this, in February this and this, etc.

To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.

[PhistucK]

Not entirely sure why you are conflating runtime bugs with build concerns, but, fine - you should know that the canary is not going through any kind of manual testing (except maybe for a few days right after branching, you can generally tell if it may have undergone manual testing if the patch number in the version - major.minor.build.patch - 80.0.1111.0 - is not 0). This is by design. Living on the edge can be risky, this is why it is not intended for general browsing.

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/b2f84e3a-7af1-4b28-8174-cad3c7f2da2d%40chromium.org.

[Marcin Wiącek]

I'm conflating things, because it looks, that in both cases some decision chains can be potentially broken - both in Jumbo and 1033655 changes were probably done without deeper checking, how they will affect world (and 1033655 went to production and made some data looses); for 1034233 - yes, Canary is given without any warranty, but such serious problem was not notified in few Google Play releases. This is not correct as well.

One of my last comments used word "post-portmen" - I tried to summarize, that we have probably issue with project structure and rules. I hope, that this voice will be heard. Please note, that we have middle of Dec and we externally don't know if Chrome will have in any moment any replacement for Jumbo.

Lack of dialogue in project affecting so many people is very bad; if you know, how to address these "system" issues, please help - in my professional understanding even Canary needs some additional checks before going to GP.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/b2f84e3a-7af1-4b28-8174-cad3c7f2da2d%40chromium.org.

[PhistucK]

But that is kind of the whole point of the canary - you get a daily untested, experimental build. If you want the greatest and relatively latest with a bit of assurance, use Chrome Dev, released every week or two.

Chrome/Chromium is a huge project, in terms of code and contributors and like in any other huge projects, mistakes, even critical ones, are made. The project adds more and more automated tests that are supposed to catch this even before it reaches the canary, but not everything can be predicted.

The profile migration issue you mentioned has indeed, unfortunately, reached stable WebView. This is not the first time similar things have happened (but they do not happen a lot), but this is life, mistakes - sometimes crucial and critical (and life threatening/terminating in some cases) - happen.

But I really think we digress here. Maybe keep this thread Jumbo-specific because I doubt you can change the entire project with this thread (fewer people will pay attention). Overall, bugs like this do not occur that often/are not that widespread (but each one is very unfortunate).

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e02dc41a-1f64-48cb-b220-b11332ce3577%40chromium.org.

[Marcin Wiącek]

Well, you're silently confirm what I have just written. What I see here:

1. adding more automatic tests should resolve issues (which is obvious mistake)

2. some serious issues are going to production and this is normal (well, it sounds much worse than: there will be done step x, y, z to improve or eliminate this)

3. project is complex and nothing can be done about it (in real engineering you can always find the way)

In normal situation you never say 2 and 3 to people and these are exactly problems I talking about - some project decisions were made and thx to it some things are not changing for better... and they're going to worse. It looks, that current decision makers don't want changes. They probably don't want to see, that some things can be done better. Project needs fresh blood, which is blocked (-1 in the https://chromium-review.googlesource.com/c/chromium/src/+/1608741 is one of very good examples).

About Jumbo - it was canceled somewhere in August, now in Dec 17th we don't see even any estimation what will be done with it. Four months. Somebody proposed something few days ago & it was commented - we don't want it.

This is saying to people more than everything about the project condition and this is wrong, wrong and once again wrong. Point.

If you want to speak about Jumbo - where is the replacement plan? where is this cloud solution? where is standalone solution?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e02dc41a-1f64-48cb-b220-b11332ce3577%40chromium.org.