Amazon EC2 with High End configuration or Goma Server/Re-client to build chromuim

[Adnan Rashid]

Hi folks,
I'm assessing the right set of infrastructure build chromium browser
As per the chromium document, to improve build time, specially for local development, we can setup a goma server/reclient infra with bazel build grid support. Seems like too much of effort to setup the whole infra structure and I haven't set it up yet to know the actual build time

Instead of that I created an EC2 instance of type c5.12xlarge (48 cores / 192Gb) machine with storage at 3000 IOPS is ~70-75 min. I am assuming, if I would have created an instance with 96 cores / 192Gb machine with storage at 64000 IOPS, the build time would be around 45-50 minutes

My question is
1. What can I expect to be the build time with GOMA Server/ Re-client setup considering that we can only parallelize to an extend and will reach a saturation (Not sure how many workee I can create in AWS for build grid) ?
2. I haven't found much discussion on this hence asking, why should one build the whole infra if something like an EC2 instance makes life simple ?

[K. Moon]

I'm not sure this is the best list for this question (as this list is specifically for Chromium project infrastructure, not general questions about setting up infrastructure for other projects that just happen to use Chromium), but I think the first thing you need to ask yourself is whether you need to build Chromium on more than one machine.

If you only build Chromium once, and then distribute the results, then a single big machine actually is a pretty good solution.

If you have multiple developers, each building their own copies of Chromium, then a solution like reclient lets all those developers share the same pool of build machines and reuse the cache of compiled artifacts. This not only is faster (if you have access to thousands of cores, rather than only hundreds), but substantially cheaper than giving every developer their own really powerful machine.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/7cddea36-b369-4ef2-a3db-2e16b22cdee4n%40chromium.org.

[Adnan Rashid]

Hi K. Moon

Thanks for taking time and responding to this query. We want to build a browser on top of chromium which will have additional security controls. And we want to build it for MAC,Windows and Linux. 

We are just starting and have a team of 8 developers now. So, does it make sense that for production release, we should have some powerful machine but for developer build/testing, setting up the infrastructure is more appropriate. 

Also, is there a bench mark or stats which is present that can tell, how cheap it can be to setup an infra compared to a single powerful machine

[K. Moon]

I'm pretty sure you're going to have to do your own benchmarking/cost comparisons. Even if someone else was willing to put in the work to report their own numbers, those numbers may have no relationship to your own, unique situation. For example, how much it costs to maintain separate infrastructure depends on your labor costs, cloud resource contracts, frequency and timing of builds, etc.

As one data point, Chromium bot builds are done with reclient (and previously were done with Goma). Developer builds usually also are done with Goma or reclient (if you're lucky enough to work for Google, or have access to similar infrastructure). It's almost always going to be a "better" solution for a team larger than 1.

One wrinkle to keep in mind is that linking release-quality Chromium builds is very expensive, and can't be massively parallelized. A machine with lots of memory, fast I/O, and a small number of powerful cores will help here. That's only needed for preparing the final release builds, however.

My ultimate recommendation would be to set up a build cluster, and don't worry too much about the relative cost. Your developers will be much happier using distributed builds.

[K. Moon]

Another thing to note is that reclient uses a standard protocol, so you can look for a hosted solution and compare pricing, rather than building your own.

You probably will have better luck asking about this on chromium-dev or embedder-dev, to draw from the experience of developers who aren't using Chromium project infrastructure.