Revisiting persistent server mode

[Tyler Mandry]

Large projects using ninja have a problem: the minimum time required to do anything with ninja is bound by the time to parse build files, which can be in the 5-6 second range for several projects I know of.

There have been a couple attempts to address this with a persistent build server mode which forks into the background and holds the dependency tree in memory. As far as I'm aware there were no design concerns around the second of these, and I've now begun resurrecting that PR with the added ability to handle arbitrary build and tool requests from one server.

On a (minimal) Fuchsia build, this takes full noop build times from 6s to 1s. If you select a specific target (ergo don't have to stat every input file in the build), that goes down to 200ms. Tools that query the build graph have no latency.

Even if build file parsing was sped up by a factor of two, it would not come close to this. As many developers on large projects will tell you, waiting an extra 5 seconds for every build can be painful for staying in development flow and getting quick feedback.

It seems like a lot of projects using ninja stand to benefit from this, so I'm starting this thread to gauge interest and ideally get it on a future release milestone.

[Dan Willemsen]

Even if build file parsing was sped up by a factor of two, it would not come close to this.

Loading can be sped up for these projects by much more than a factor of two with Ryan's parallelism patches. We've been running these in Android's ninja fork for a few years now. I'd highly recommend using them for larger projects. On one of our smaller build graphs, my no-op ninja build time is ~2.5s. That's loading ~2.5GB of ninja files, 240MB of .ninja_log/deps files, and evaluating/checking ~160k edges (out of >1million).

- Dan

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/88c17bc4-1b84-4587-9ab9-ad9d5e6cb9e7n%40googlegroups.com.

[Alfred Zien]

What if ninja will dump built graph into some file that need not to be parsed (something like .ninja_deps file)? It seems like an obvious vector for optimization, how hard is it to implement?

-- Alfred

To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CALQgHdm_PF6K0rgLKbwdg6KVkzOAJMyfsUnroX6PmgqfRXyAuA%40mail.gmail.com.