Distributed ninja build

[Richard Geary]

Has anyone tried to distribute the build commands issued from Ninja? eg. using parallel ssh

[Evan Martin]

As far as I know the state of the art is distcc:
http://code.google.com/p/distcc/
You will need to adjust the "-j" flag to parallelize your build to
more tasks than the local thread count.

[Scott Franklin]

I actually wrote a distributed build system for a cloud computing course which happens to read .ninja files. They lend themselves well to pretty cool things like supporting heterogeneous slaves (each builder registers a list of rule statements it knows how to execute: eg cc_32 vs cc_64); plus you can do awesome stuff like cache the output of build statements so that anyone can always noop build a clean trunk.

One of these weekends I'll get around to making the code fit for public consumption and uploading it to github, but there would be a lot of work to do to make it usable in a non-personal setting, important stuff like not hashing every file on a noop build and fixing the security vulnerabilities that allow anyone to run arbitrary shell commands on your build slaves.

So, long answer to your question, yes, I have done it by rewriting Ninja from scratch and use it semiregularly for LaTeX documents.

--

Scott Franklin

[Evan Martin]

On Tue, Jul 17, 2012 at 9:38 PM, Scott Franklin <sd...@cornell.edu> wrote:
> I actually wrote a distributed build system for a cloud computing course
> which happens to read .ninja files. They lend themselves well to pretty cool
> things like supporting heterogeneous slaves (each builder registers a list
> of rule statements it knows how to execute: eg cc_32 vs cc_64); plus you can
> do awesome stuff like cache the output of build statements so that anyone
> can always noop build a clean trunk.
>
> One of these weekends I'll get around to making the code fit for public
> consumption and uploading it to github, but there would be a lot of work to
> do to make it usable in a non-personal setting, important stuff like not
> hashing every file on a noop build and fixing the security vulnerabilities
> that allow anyone to run arbitrary shell commands on your build slaves.

That sounds awesome! Even if you never end up releasing the code I'd
love to read more details about what worked and what didn't.

One of the features I had always imagined implementing in Ninja sounds
like your build output caching: even in a non-distributed build, if
you aggressively ran Ninja in the background and cached the build
output, when the user directly instructed Ninja to run it could just
print the cached output. Another project for another day...

[Konstantin Tokarev]

18.07.2012, 20:51, "Evan Martin" <mar...@danga.com>:

> On Tue, Jul 17, 2012 at 9:38 PM, Scott Franklin <sd...@cornell.edu> wrote:
>

>> О©╫I actually wrote a distributed build system for a cloud computing course
>> О©╫which happens to read .ninja files. They lend themselves well to pretty cool
>> О©╫things like supporting heterogeneous slaves (each builder registers a list
>> О©╫of rule statements it knows how to execute: eg cc_32 vs cc_64); plus you can
>> О©╫do awesome stuff like cache the output of build statements so that anyone
>> О©╫can always noop build a clean trunk.
>>
>> О©╫One of these weekends I'll get around to making the code fit for public
>> О©╫consumption and uploading it to github, but there would be a lot of work to
>> О©╫do to make it usable in a non-personal setting, important stuff like not
>> О©╫hashing every file on a noop build and fixing the security vulnerabilities
>> О©╫that allow anyone to run arbitrary shell commands on your build slaves.
>
> That sounds awesome! О©╫Even if you never end up releasing the code I'd

> love to read more details about what worked and what didn't.
>
> One of the features I had always imagined implementing in Ninja sounds
> like your build output caching: even in a non-distributed build, if
> you aggressively ran Ninja in the background and cached the build
> output, when the user directly instructed Ninja to run it could just

> print the cached output. О©╫Another project for another day...

ccache?

--
Regards,
Konstantin

[Evan Martin]

On Wed, Jul 18, 2012 at 10:10 AM, Konstantin Tokarev <ann...@yandex.ru> wrote:
>> That sounds awesome! Even if you never end up releasing the code I'd

>> love to read more details about what worked and what didn't.
>>
>> One of the features I had always imagined implementing in Ninja sounds
>> like your build output caching: even in a non-distributed build, if
>> you aggressively ran Ninja in the background and cached the build
>> output, when the user directly instructed Ninja to run it could just

>> print the cached output. Another project for another day...
>
> ccache?

I believe ccache only caches the result on success. What I had
imagined is Ninja could cache the stdout/stderr of the process even on
failure, so if for example your code left out a semicolon Ninja would
have precached the error message such that when you next ran the build
it would appear as if the compiler had discovered this instantly.

[Gavin Peters (蓋文彼德斯)]

I've thought about using inotify to watch all deps, so that the solution is always instantly available. But your idea is even better. Perhaps in combination.

- Gavin 

[Konstantin Tokarev]

18.07.2012, 22:49, "Gavin Peters (蓋文彼德斯)" <ga...@ytz.ca>:

Inotify is Linux-only.

--
Regards,
Konstantin

[Richard Geary]

This is another really interesting idea. Though if your build process isn't distributed and it eats cpu & memory resources, you may need to throttle it.

On Wednesday, 18 July 2012 13:35:05 UTC-4, Evan Martin wrote:

[Richard Geary]

Windows has a similar file system change watcher
  http://msdn.microsoft.com/en-us/library/windows/desktop/aa365261%28v=vs.85%29.aspx

On Wednesday, 18 July 2012 15:13:44 UTC-4, Konstantin Tokarev wrote:

18.07.2012, 22:49, "Gavin Peters (蓋文彼德斯)" <>:
> On Wed, Jul 18, 2012 at 1:35 PM, Evan Martin <> wrote:

[Konstantin Tokarev]

19.07.2012, 21:42, "Richard Geary" <richar...@gmail.com>:

> Windows has a similar file system change watcher

> О©╫ http://msdn.microsoft.com/en-us/library/windows/desktop/aa365261%28v=vs.85%29.aspx

So you will need to implement (or find existing) cross-platform library for file system watching.
Qt has ready to use solution [1], but it maybe too heavy-weight dependency for Ninja.

[1] http://qt-project.org/doc/qt-4.8/qfilesystemwatcher.html

--
Regards,
Konstantin

[Scott Graham]

On Thu, Jul 19, 2012 at 10:42 AM, Richard Geary <richar...@gmail.com> wrote:

Windows has a similar file system change watcher
  http://msdn.microsoft.com/en-us/library/windows/desktop/aa365261%28v=vs.85%29.aspx

I went pretty far down that path on Windows. There's more info in the archives, but file system change watcher isn't reliable, and I found the changejournal code to be too much complexity for the benefit it offered. Perhaps someone else could write a cleaner/simpler version though. The code is in one of my non-master github forks if anyone wants to investigate further.

[Richard Geary]

Could we use the file system change watcher just as a trigger to rebuild, rather than an authoritative source of what's changed? Then reliability matters less.