Optimize parallel build, any recommendation on number of jobs?

[Hoa Phan]

Would the number of processor:

Runtime.getRuntime().availableProcessors()

be the optimized number of job for a bazel build.

[Austin Schuh]

If you don't mind completely slamming the box, add 2 to that number.  Otherwise there will be gaps when Bazel is figuring out what next, or your jobs aren't using 100% CPU.  I run 18 jobs on a 16 core box.

--experimental_multi_threaded_digest is a massive speedup as well when your core count gets anywhere high, as long as you are running on SSDs.  Ir parallelizes checksumming the results.

I've found --experimental_sandbox_base=/dev/shm/ to be helpful as well.  It creates the symlink tree in /dev/shm instead of on your disk, which is significantly faster at the cost of more RAM usage.

Austin

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/dfb89e52-770e-4416-bccd-a2b5f3461523%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Philipp Wollermann]

Great tips, Austin. I agree with them.

Also always add --sandbox_tmpfs_path=/tmp to improve performance, hermeticity and prevent JVMs from crashing with SIGBUS.

(I still hope that I can make this the default some day...)

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CABsbf%3DGGLstudcLQnnY5GbOychyFMpChaYsHLz%3Da2OYitMsJtg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Philipp Wollermann
Software Engineer
phi...@google.com

Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

[Hoa Phan]

., I'll try each on several run and see how they compared.

[Hoa Phan]

Hi mind if I ask if you know which code process the --jobs option.

It worked perfectly fine for me on my machine and I can see the number of jobs printed like

X actions, (X-...) running

where X is the job count I passed in. If I leave this option, "auto" seems to put in my processor count, which is also reasonable.

However on some remote build machine for some reason the job is 50, so I want to take a look at the code that decides the count.

Would it be in Java like com.google.devtools.build.lib.buildtool.BuildRequest or some *.cc  (I still haven't got use to the code base yet, long time no use cpp :< ) ? 

Thanks,

Regards