| Goma client is now open source | Shinya Kawanaka | 08/01/18 10:51 م | Hi,
Chrome's build time is getting longer and longer. According to https://groups.google.com/a/chromium.org/forum/#!msg/chromium-dev/ThDAjO7fTro/bPQzvbVLAgAJ, build time was 30 minutes in July 2014, but it is now more than 150 minutes (without jumbo) in 2017 on particular Mac. Google is internally using a distributed compiler service, Goma. With Goma, chrome can be built in a few minutes. However, it's available only for googlers, so non googlers need to wait for long time to build chrome, and this must be painful. To cope with this, Goma team is working so that all Chromium developers can utilize improved compile performance enabled by Goma. As a first step, Goma team has made the client code open source in https://chromium.googlesource.com/infra/goma/client/ Unfortunately, Goma server is not publicly available yet, but we're working on this. We hope we can make the server publicly available this year so that all Chromium developers can be more productive. Stay tuned. |
| Re: [chromium-dev] Goma client is now open source | PhistucK | 09/01/18 12:08 ص | Woo hoo! Next-next step - providing virtual machines a-la-Google-Cloud-Shell that are ready with the latest Chromium commit on which Chromium can be coded? :D ☆PhistucK -- |
| Re: [chromium-dev] Goma client is now open source | Daniel Bratell | 09/01/18 09:22 ص | Exciting! Tell us more! ---- As part of the jumbo project (unity build system for Chromium) I do some kind of tracking of compilation times to see in what direction things move so I can provide an update to the 150 min from a year ago. The compilation time kept increasing, but slower, during the first half of 2017 and reached 2 hours and 45 minutes in July (165 min) and there my history ends, though other measurements indicate that it would have kept increasing by about 1% per week during the second half of 2017. Attaching image I produced yesterday for other reasons. /Daniel -- -- /* Opera Software, Linköping, Sweden: CET (UTC+1) */ |
| Re: [chromium-dev] Goma client is now open source | Ryan Tseng | 09/01/18 11:02 ص | awesome news! -- |
| Re: [chromium-dev] Goma client is now open source | Alexis Menard | 09/01/18 11:21 ص | Awesome! Really appreciated. One suggestion to do a staged roll out is to open the farm to the Chromium committers first. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAMk4ikPS-kVyzLmxZJB_zWqzAJXwVwbvKWqhxjJ7ui-baArvog%40mail.gmail.com. |
| Re: Goma client is now open source | st...@chromium.org | 09/01/18 04:25 م | Awesome. Looking forward to it. |
| Re: [chromium-dev] Re: Goma client is now open source | Jan Keromnes | 10/01/18 01:08 ص | Excellent news, thanks a lot!
This already exists: https://janitor.technology (spawns dev containers that are pre-configured with the latest Chromium checkout in 1.2 seconds, with web-based Terminal + IDE + Remote Desktop, so you can code Chromium from any device e.g. a Chromebook or Android tablet.) François Beaufort already uses Janitor to contribute to Chromium from his Pixel, but build times are painfully slow even on our bare-metal servers, so I'm very much looking forward to providing secure GOMA access to Chromium committers who use Janitor. :) --To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e9788a92-8d9e-43bd-9537-54bd3b80d363%40chromium.org. |
| Re: [chromium-dev] Re: Goma client is now open source | PhistucK | 10/01/18 01:17 ص | Huh, interesting. Is it free? How do I get invited faster (I submitted my e-mail address there)? :P ☆PhistucK To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAA6PgK45hWtLWQd10ORRSf7_QeDkQymahdfB5w2-F1XZVjUfpA%40mail.gmail.com. |
| Re: [chromium-dev] Re: Goma client is now open source | Jan Keromnes | 10/01/18 01:21 ص | It's free thanks to sponsorships (www.irill.org and Mozilla to date). Thanks for your interest! I've just sent you an alpha invite. |
| Re: [chromium-dev] Re: Goma client is now open source | PhistucK | 10/01/18 01:24 ص | This is great! Thank you! I might have much less time for watching television from now on! :S :P ☆PhistucK |
| Re: [chromium-dev] Re: Goma client is now open source | Jan Keromnes | 10/01/18 01:30 ص | You're welcome! (And sorry about your TV time :P) Please let me know if anything looks confusing or broken. Also, maybe let's continue this discussion off-thread, in order not to derail/spam it too much. |
| Re: [chromium-dev] Goma client is now open source | Alexis Menard | 10/01/18 08:44 ص | Btw does Goma requires special ports to be opened? If so it would be beneficial to share them (could be off list) because some of us work in big corporations and we need to work with our IT departments to get these ports open. |
| Re: [chromium-dev] Goma client is now open source | Jan van Oort | 10/01/18 12:20 م | Brilliant news ! Of course, the real bomb would be for Goma server to be publicly available. Eagerly awaiting that one. Thumbs up ! -- |
| Re: [chromium-dev] Goma client is now open source | Victor Costan | 10/01/18 04:46 م | This is great news! Thank you very much for your work on open sourcing Goma, and congratulations on this milestone!
Victor -- |
| Re: [chromium-dev] Goma client is now open source | Kinuko Yasuda | 10/01/18 09:12 م | (Oops, sending from the right address...)
Just realized this thread, sounds very exciting, and congrats to the team on this important first milestone! \o/ On Wed, Jan 10, 2018 at 8:50 PM, Kinuko Yasuda <kin...@google.com> wrote:
|
| Re: [chromium-dev] Goma client is now open source | Robert Hogan | 15/01/18 02:49 م | +1 that this is going to be great. I'm really looking forward to seeing this happen.
--To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CABA057_WbBC4zsMv1-cZ97auGPObrCvufeyoc_X_ZdgnWVfKwA%40mail.gmail.com. |
| Re: Goma client is now open source | br...@amazon.com | 20/02/18 07:56 م | Shinya this is great news. A few questions: 1. Will the server source code itself be open sourced, or is the intention here simply to open up Google's own fleet of build machines to external users? 2. Does Goma assume a homogeneous fleet, at least with respect to each host having the same build toolchain? Or, is Goma more like the Icecream distributed compiler, where heterogeneous clients tar up their build environment and ship that off to the other machines to use when performing the compilation? As an aside, I have some anecdata collected a few days ago on Chromium build times. On a workstation-esque machine, the chrome target took ~177 minutes to compile. With jumbo builds enabled, this dropped to ~77 minutes. Needless to say, 5-minute builds are going deliver a step-function improvement for teams working on Chromium outside of Google; it's no surprise people are so excited. |
| Re: [chromium-dev] Re: Goma client is now open source | Shinya Kawanaka | 20/02/18 08:21 م | On Wed, Feb 21, 2018 at 12:30 PM, bryct via Chromium-dev <chromi...@chromium.org> wrote: We might be able to make server source code open, or might be able to make the server publicly available. I can say we need more work to make them happen...
Icecream seems somewhat similar to goma. In goma, a developer uploads a toolchain to the server side in advance, and client sends some information (e.g. digest) to specify which toolchain should be used.
|
| Re: Goma client is now open source | popov654 | 22/02/18 08:40 ص | And why did Chromium become so heavy that it takes so long to compile? Is it because of many new functions, or is something wrong with architecture of code? I know it always required very powerful hardware to compile... But the distributed network? :\ вторник, 9 января 2018 г., 9:51:06 UTC+3 пользователь Shinya Kawanaka написал: |
| Re: [chromium-dev] Re: Goma client is now open source | Torne (Richard Coles) | 22/02/18 08:58 ص | The distributed compiler service isn't required to build (a sufficiently powerful machine can build it in a reasonable time); it's only required to build *really* fast. Also, Chromium didn't really "become" so heavy - while compile times have gotten longer they were never very fast unless you had very powerful hardware, as chromium has always been very large - even with the build times from several years ago we were already using the distributed compiler service to get build times down to well under 10 minutes, which wasn't feasible on a single machine even when the code was smaller. --To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/562bff63-b495-4846-8cb8-bdb1849c0170%40chromium.org. |
| Re: [chromium-dev] Re: Goma client is now open source | popov654 | 22/02/18 09:38 ص | Thanks for the clarification. However, I still wonder, if every major browser is compiling in several hours (or these times can be somehow reduced). We know some examples: Maxthon, Opera Presto (now dead for 6 years), Internet Explorer (also dead). Are they also so heavy? Asking just of curiosity. четверг, 22 февраля 2018 г., 19:58:17 UTC+3 пользователь Torne (Richard Coles) написал:
|
| Re: [chromium-dev] Re: Goma client is now open source | Torne (Richard Coles) | 22/02/18 11:24 ص | Last I heard Chromium is slower to build than firefox, but not by a huge amount (same order of magnitude); I don't know about any other browsers as most of the rest aren't open source (or are based on chromium and so are just as big/slow to build as chromium is). Browsers are just extremely large and complex programs, and also care a lot about performance so may use aggressive build options that produce better binaries at the expense of slower build times. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/9662acc2-6199-474a-bf92-9f45ca703029%40chromium.org. |
| Re: [chromium-dev] Re: Goma client is now open source | Daniel Bratell | 23/02/18 02:58 ص | I happen to be in a situation that I can compare, having worked on Mozilla (long+ ago), Presto and Chromium. Of those Presto seems to be the odd one out with dramatically lower compilation times but it's also not instant, just 10x faster. As how it got here it's been a slippery slope of combining factors. Modern C++ is heavy to compile. Last I checked (~2 years ago) 30% of the compilation time of a file in Blink was template instantiations. Presto happens to not use templates much/at all. Chromium makes heavy use of templates. Features. The browser is replacing the operating system in many cases and growing an awesome number of features. Every new feature adds code. Lack of good ways to share code. In the jumbo project I keep stumbling upon code that is copied rather than reused, and where there is no acceptable place to put it. There is //base but not for code that is only ever getting to be used twice and sibling projects can't depend on each other just to share code. Recent problem: Extreme code splitting. Every unit, be it a gn target or a C++ file or a header, has some inherent overhead. Both in processing by tools, and by adding more and more abstraction layers to make the units fit each other nicely. This is one thing that worries me. Code is split out of too large targets to make it more maintainable, but is one too large target worse than 200 tiny modules where nobody has any overview and where every tiny module has added one, two or three more abstraction layers? The actual compilation time problem is also (sadly?) hidden or mostly mitigated by the awesome distributed compilation system that is goma. On some platforms (Linux, ~Mac) icecc can also assist really well, but only for people with access to large sets of computers. Anyone sitting at home, or in their student dorm, they are out of luck. There is a need of more diversity in the project to cover all users and potential users but while the people are friendly, they development environment is not. This problem might be known to most developers but "knowing" and "understanding" is not the same thing. So where to go from here? I don't think use of modern C++ is going away, and with the attitude of the language developers ("the compiler will fix it") when confronted with inefficiencies in the language, I think it might get worse. I would like there to be a minimum size of code modules, and if things are too small, merge them. Realistically I don't see this happening except when things have become silly (as in some single file blink modules that merged last year). I would like there to be a taboo against duplicate functionality and code. A taboo only broken when there are extremely good reasons. A taboo that covers the whole project, not just locally within a function or file, or directory. It requires adding trash heap for utility functions. Oh, it will look bad in a design document. It will make people scowl. I don't care. It will make the product better. Making a problem visible isn't making it worse. It is making it better. Ok, this mail ended up much longer than intended. If you get down here, well done! :) /Daniel To unsubscribe from this group and stop receiving emails from it, send an email to chromi...@chromium.org.To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEV-rjeFFOA1mSNgVAwFHxVHjPTuyOBGB-A5pzz%2BcTJ%2BgG7W%2Bw%40mail.gmail.com.
|
| Re: Goma client is now open source | pet...@vivaldi.com | 03/12/18 07:12 ص |
Sorry for the thread necro, but how's the progress on this part? - Petter |
| Re: [chromium-dev] Re: Goma client is now open source | Shinya Kawanaka | 03/12/18 06:35 م | I'm not sure I can say the details, but I can say it's moving forward. ----To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/c875d40c-b7f6-43aa-ba65-b0591fdd8f9e%40chromium.org. |
| Re: [chromium-dev] Re: Goma client is now open source | Ion Lupascu | 07/03/19 10:11 م | Hello, Seems the goma server repo is already created, bu empty: https://chromium.googlesource.com/infra/goma/server/ Is any plan when it will be ready for public? Thank you in advance, Ion On Tuesday, 4 December 2018 02:35:04 UTC, Shinya Kawanaka wrote:
|
| Re: Goma client is now open source | Ion Lupascu | 07/03/19 10:13 م | Hello,On Tuesday, 9 January 2018 06:51:06 UTC, Shinya Kawanaka wrote:
|
| Re: [chromium-dev] Re: Goma client is now open source | Dirk Pranke | 08/03/19 09:53 ص | Hi, We're actively working on it but we don't have a date to announce yet. Thanks for checking in :). -- Dirk --To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/7c85566b-b9ff-452b-b655-9fd0355abb5c%40chromium.org. |