Pool-based job scheduling merged to master

[iannucci]

Hello ninja-build,

As of 8fcc4caf8f98c532f0f5078b7a0593b0904871a0, ninja's manifest format has gained a new feature called 'pools' which allow you to control the maximum concurrency for build statements and rule invocations on a finer-grained level than just adjusting the number of jobs on the command line.

I believe an example is in order:

Example <<END

pool linkers

  depth = 4

rule link

  command = link ...

  pool = linkers

END

In the example above, no more than 4 jobs in the 'linkers' pool can run at a time, no matter how many jobs ninja is allowed to schedule. There's more detail in the README :).

Performance-wise, there should be zero impact on projects which do not use this feature. Builds which utilize the new feature should suffer no performance impact (other than the obvious effects of limiting the build's concurrency :).

Next steps:

Now that the basic pool scheduling interface is in place, I'm planning to build higher-level support for the feature into GYP (http://code.google.com/p/gyp/). I'll then instrument the chromium (http://www.chromium.org/Home) build process so that it uses pools to replace the current link-serializing-flock which exists (and which absolutely kills build performance in some circumstances).

Additionally, I would like to experiment with some of the following features:

  * allowing non-uniform weight edges (manually or perhaps automatically calculated weights)

  * different prioritization algorithms (currently pools are queues, and no re-ordering of jobs is allowed)

  * I'm especially interested in using build profiling and build graph data to try to learn about how to better schedule the build.

Let me know if you have any questions about, or issues with, the new feature!

Cheers,

Robbie

[Samuel Bronson]

On Friday, December 14, 2012 at 3:04:06 AM UTC-5 iannucci wrote:

Additionally, I would like to experiment with some of the following features:

  * allowing non-uniform weight edges (manually or perhaps automatically calculated weights)

I just noticed this is missing, though I could see you had contemplated it. Has anyone put any thought into what that should look like?

The obvious way would be to add another variable, but what if you wanted multiple pools for one edge? (Or are multiple pools obviously a bad idea idea anyway?)

Maybe an optional integer before the pool name in the "pool" variable? That would leave the "multiple pools for a single edge" idea on the table if somebody wanted to pursue that in another decade or so.

(I'm trying to build LLVM on my phone without running out of RAM, so I was thinking of increasing the weight on some of the instruction-delection code to avoid building that in parallel ...)

  * different prioritization algorithms (currently pools are queues, and no re-ordering of jobs is allowed)

I'm assuming this won't *exactly* work well with variable weights: say the head of the queue weighs too much to be scheduled yet, but the next item does not.

Let me know if you have any questions about, or issues with, the new feature!

... nearly 12 years later ...

- Sam