Ninja output buffering question

[Kent Williams]

Hey, first time poster.

I began to use the Ninja generator in CMake recently, precisely because of the output buffering -- parallel build output is much more useful when several processes' output isn't garbled up together.

I did find one way in which this is sub-optimal -- when Ninja calls Ninja.

This happens in CMake if you use the ExternalProject facility -- in the CMake-generated Ninja project, Ninja invokes Ninja to build the external project.

The output buffering method of Ninja means that you see no output from this nested Ninja run until it's complete. We regularly build packages that take a half hour or 45 minutes, which is a long time to wait without seeing any warnings or error messages.

There isn't a perfect solution to this, and fixing this would involve some careful coding.

But what would be great is if Ninja could cooperate with nested Ninja runs, such that the output would be buffered on a per-command basis by the calling Ninja.

In other words, when Ninja calls Ninja, instead of buffering it's entire output before displaying it, it could display the inferior Ninja's output a command at a time.

I can live with how Ninja works now, but this would be useful, and it's doable -- for instance Ninja could pass a hidden flag to subsidiary Ninja runs and the subsidiary Ninja could wrap each command's output with sentinel text.

For example if Ninja runs Ninja, it adds '--inferior-ninja' to the flags.

If Ninja is involked with --inferior-ninja, when it does command output buffering it would output it like this:

### NINJA <PID> ###

(output of command

### NINJA <PID> END ##

[Nicolas Desprès]

There is the console pool for that. CMake should put build edge generated by external project command in this pool.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Nicolas Desprès

[Reid Kleckner]

Here's a crazy idea: instead of re-invoking ninja on the external project, why not subninja the build.ninja file of the external project?  Then you would get the full set of actions in one build.  The only problem I see with this approach is that the external project might wish to re-generate its build.ninja file with CMake as the first action.  I don't think ninja supports more than one of these regeneration actions.

[Peter Collingbourne]

On Thu, Jul 24, 2014 at 09:43:44AM -0700, 'Reid Kleckner' via ninja-build wrote:
> Here's a crazy idea: instead of re-invoking ninja on the external project,
> why not subninja the build.ninja file of the external project? Then you
> would get the full set of actions in one build. The only problem I see
> with this approach is that the external project might wish to re-generate
> its build.ninja file with CMake as the first action. I don't think ninja
> supports more than one of these regeneration actions.

I don't think this will work if the external project uses relative paths, as
paths will be interpreted relative to the root ninja file.

Thanks,
--
Peter

[Nico Weber]

Rule names are currently global. I don't know how cmake assigns rule names, just subninja'ing build.ninja might lead to rule name conflicts. (Rule names probably shouldn't be global.)

[Matthew Woehlke]

On 2014-07-24 11:33, Kent Williams wrote:
> I began to use the Ninja generator in CMake recently, precisely because of
> the output buffering -- parallel build output is much more useful when
> several processes' output isn't garbled up together.
>
> I did find one way in which this is sub-optimal -- when Ninja calls Ninja.
>
> This happens in CMake if you use the ExternalProject facility -- in the
> CMake-generated Ninja project, Ninja invokes Ninja to build the external
> project.
>
> The output buffering method of Ninja means that you see no output from this
> nested Ninja run until it's complete. We regularly build packages that take
> a half hour or 45 minutes, which is a long time to wait without seeing any
> warnings or error messages.

There's another problem here; a ninja job (AFAIK) is considered no
different from any other job, which means it is subject to being run in
parallel with other jobs. This can lead to running up to N*N process
concurrently, instead of only N as intended. This is probably better
addressed at the CMake end.

I've also suggested (more than once) that external projects should use
Ninja's console pool, which would eliminate both problems (only one
sub-ninja run at once, immediate output from that job). Unfortunately I
do not believe anyone is able to work on this currently.

Also, there is a problem with immediately dumping command output from
sub-ninja; if I have three sub-ninja's running at once compiling
different projects, interleaving the output from these can still be
undesirable.

That said...

> But what would be great is if Ninja could cooperate with nested Ninja runs,
> such that the output would be buffered on a per-command basis by the
> calling Ninja.

I think it might still be useful if ninja had something like a
'porcelain' mode, where it outputs additional control characters to
indicate e.g. progress indicators, start/end of command output, etc. in
a manner that allows for easy machine parsing of this information.

p.s. A really easy way to implement this, without ninja needing to know
if it's calling itself recursively, would be for ninja to write to the
environment of all jobs some variables that ninja itself would then
consider to e.g. limit jobs¹ or change the default output format.

(¹ I'd really like such an environment variable anyway. Besides fixing
the above mentioned exponential job count issue, it would allow me to
limit the total jobs for a sub-ninja, which otherwise I don't think is
possible? In particular, one of the projects I work on - which is built
via an external project - needs a lot of resources to link, and brings
my system to a crawl with the default job count.)

--
Matthew

[Richard Geary]

The N*M jobs issue is a latent problem, and the obvious workaround is a slower build. If you run the first ninja with -j1 and the others with -jN then you're underusing the machine, waiting whilst each child ninja completes.

Another use case where "external" job scheduling control would be useful is for distcc jobs.  I have a ninja script generator which can use distcc to compile the targets, so I can run ninja with -j50 on a 4 core box & a compute grid. However, if there's a problem with the compute grid or the job is not distributable (eg. linking or running unit tests), then distcc reverts to the local machine. Running 50 linkers & stress tests on a 4 core 8GB box is a sure way to invoke the linux OOM killer, which is an arbitrary death gamble for the box.

I've written a resource blocker using named semaphores to wrap each distcc job. Not sure if there's a better solution, but it feels like it should be ninja's responsibility to manage its spawned jobs within the resource limits of the machine. Perhaps ninja could SIGTSTP some jobs if it detects low memory or excessive loadavg.

--
Matthew

[Evan Martin]

Make uses a shared pipe as a cross-process semaphore:

http://mad-scientist.net/make/jobserver.html

We sometimes have talked about making Ninja obey the same semantics.  That would allow Ninja within Make within Ninja etc to all pull from the same job pool.  It'd require generalizing some of the ppoll/select logic [1] to also poll the semaphore fd.  It's not as ugly as described in the above page because we don't rely on SIGCHLD (which is less correct but in practice hasn't yet mattered).

[1] https://github.com/martine/ninja/blob/b0ada0efd9ee1d80d38c781fa15d73bece4a3d69/src/subprocess-posix.cc#L203

[Matthew Woehlke]

On 2014-07-25 02:50, Evan Martin wrote:
> Make uses a shared pipe as a cross-process semaphore:
> http://mad-scientist.net/make/jobserver.html
>
> We sometimes have talked about making Ninja obey the same semantics. That
> would allow Ninja within Make within Ninja etc to all pull from the same
> job pool. It'd require generalizing some of the ppoll/select logic [1] to
> also poll the semaphore fd. It's not as ugly as described in the above
> page because we don't rely on SIGCHLD (which is less correct but in
> practice hasn't yet mattered).

Honestly, I'm not sure that I've had problems with N² jobs with
sub-ninja. Where I *do* have problems is with certain projects that
aren't okay with even N jobs, that are normally run as sub-ninja's. If
such mechanism would allow me to run a "root" ninja with -jM and have
the sub-ninja(s) respect M, that would be hugely useful.

(p.s. Thanks for this awesome tool!)

--
Matthew

[Richard Geary]

Out of interest, what causes you to need to use M jobs (where M<N) ? Is it memory constraints or something else?  If you control the subninja files, you could add a global pool for edges in that file and set it to depth M.

For us, we have several compile jobs which can take ~5Gb of RAM, but we don't know in advance which compiles are memory-hungry. So the ability to dynamically adjust the ninja -j number via IPC would be useful. Ideally we would run a separate daemon to monitor memory usage and reduce ninja's -j if it's low on memory. Or if we know that a different compile on the same box needs to take priority.

[Matthew Woehlke]

On 2014-07-25 10:55, Richard Geary wrote:
> Out of interest, what causes you to need to use M jobs (where M<N) ? Is it
> memory constraints or something else?

This is a CMake project (specifically, http://slicer.org/), so I'm not
directly editing the sub-ninja file. I forget offhand if CMake has been
taught yet to understands link pools, or if Slicer requires a
sufficiently new CMake to use the feature. But most especially I really
don't want to try to dig that deeply into the build system to enable
them :-).

I guess it's mainly memory constraints; various links consume a LOT of
resources, to the point that my machine becomes unusable if I let the
default number of jobs (10) run. (And yes, some of the links consume a
GiB or several of RAM, and anyway take a *very* long time to run, i.e.
minutes.)

> For us, we have several compile jobs which can take ~5Gb of RAM, but we
> don't know in advance which compiles are memory-hungry. So the ability to
> dynamically adjust the ninja -j number via IPC would be useful. Ideally we
> would run a separate daemon to monitor memory usage and reduce ninja's -j
> if it's low on memory. Or if we know that a different compile on the same
> box needs to take priority.

Yes, some form of monitoring would be ideal :-). For my case, it's easy
to just run the top build with a smaller -jN, but unfortunately that
doesn't propagate, so I instead have to run the top build (usually -j1
since it's a meta-build), then kill it when it gets to slicer proper,
and manually invoke ninja in the subdirectory. (It's further
inconvenient because I have shell functions and IDEs set up to make
running the top level build very convenient, but can't as readily do
that with the sub-build.)

--
Matthew

[Kent Williams]

In the particular case of Slicer + CMake, and the many projects who have copied the Slicer/CMake 'SuperBuild' pattern:

Ninja should probably have SOME communication with sub-ninjas -- through the process environment properly, which is the most portable -- because yeah if you have something like slicer starting up 8 parallel ExternalProject builds on an 8-core machine, they shouldn't each spawn 8 jobs, to where 64 processes are slamming CPU and Memory.

CMake's Ninja generator could be made smarter as well, to use the -l flag to base # of processes on the load average.

Another CMake consideration is that CMake doesn't call Ninja directly on external projects, it calls CMake --build <ExternalProject-build-directory>

[Nico Weber]

On Fri, Jul 25, 2014 at 8:39 AM, Kent Williams <nkwmail...@gmail.com> wrote:

In the particular case of Slicer + CMake, and the many projects who have copied the Slicer/CMake 'SuperBuild' pattern:

Ninja should probably have SOME communication with sub-ninjas -- through the process environment properly, which is the most portable -- because yeah if you have something like slicer starting up 8 parallel ExternalProject builds on an 8-core machine, they shouldn't each spawn 8 jobs, to where 64 processes are slamming CPU and Memory.

Isn't it possible to set up your build in a way that there's only a single ninja invocation? That should be more efficient, solves all sub-ninja process issues (there are no sub-ninja processes), etc.

 

--

[Matthew Woehlke]

On 2014-07-25 12:32, Nico Weber wrote:

> On Fri, Jul 25, 2014 at 8:39 AM, Kent Williams wrote:
>> In the particular case of Slicer + CMake, and the many projects who have
>> copied the Slicer/CMake 'SuperBuild' pattern:
>>
>> Ninja should probably have SOME communication with sub-ninjas -- through
>> the process environment properly, which is the most portable -- because
>> yeah if you have something like slicer starting up 8 parallel
>> ExternalProject builds on an 8-core machine, they shouldn't each spawn 8
>> jobs, to where 64 processes are slamming CPU and Memory.
>
> Isn't it possible to set up your build in a way that there's only a single
> ninja invocation? That should be more efficient, solves all sub-ninja
> process issues (there are no sub-ninja processes), etc.

Given that we're talking about CMake, here, CMake would need to have
knowledge of when it is running a custom command that is to build
another CMake project, then special-case this for when both generators
are Ninja (*NOT* guaranteed!). There's also the rule names issue you
mentioned previously.

This also does nothing for the potential case of ninja -> make -> ninja.
While I agree it would be best (to include the rules directly and let
the one ninja know about the entire task set), I think there are still
going to be cases where this can't work, and thus it would still be
helpful to have better handling of ninja invoking (as a job process,
directly *or indirectly*) ninja.

(Also, setting an environment variable, especially if we start with the
really stupid fix of setting said variable to our -jN so that it can
propagate, is really simple. In the longer run, setting an environment
variable to indicate how to make use of a job server is probably better.)

--
Matthew

[Neil Mitchell]

I suspect that recursive Ninja, or Ninja and Make combined, is
probably a bad idea - mostly for the reasons set out in the classic
"Recursive Make Considered Harmful"
http://aegis.sourceforge.net/auug97.pdf

Thanks, Neil

[Nico Weber]

On Fri, Jul 25, 2014 at 10:03 AM, Matthew Woehlke <matthew...@kitware.com> wrote:

On 2014-07-25 12:32, Nico Weber wrote:

> On Fri, Jul 25, 2014 at 8:39 AM, Kent Williams wrote:
>> In the particular case of Slicer + CMake, and the many projects who have
>> copied the Slicer/CMake 'SuperBuild' pattern:
>>
>> Ninja should probably have SOME communication with sub-ninjas -- through
>> the process environment properly, which is the most portable -- because
>> yeah if you have something like slicer starting up 8 parallel
>> ExternalProject builds on an 8-core machine, they shouldn't each spawn 8
>> jobs, to where 64 processes are slamming CPU and Memory.
>
> Isn't it possible to set up your build in a way that there's only a single
> ninja invocation? That should be more efficient, solves all sub-ninja
> process issues (there are no sub-ninja processes), etc.

Given that we're talking about CMake, here, CMake would need to have
knowledge of when it is running a custom command that is to build
another CMake project,

I don't know CMake well, but if there was some CMake build construct that said "build_subproject()", then it seems that the generator should have enough information to build the subproject with the same generator that's currently selected, no?

 

then special-case this for when both generators
are Ninja (*NOT* guaranteed!). There's also the rule names issue you
mentioned previously.

This also does nothing for the potential case of ninja -> make -> ninja.

One of ninja's core features is performance, which you probably sacrifice by doing these nested build system invocations.

(If Evan wants to support make's job queue chances are it's a good idea, but I'd try to get my build into a state where this wouldn't be needed instead.)

 

While I agree it would be best (to include the rules directly and let
the one ninja know about the entire task set), I think there are still
going to be cases where this can't work, and thus it would still be
helpful to have better handling of ninja invoking (as a job process,
directly *or indirectly*) ninja.

(Also, setting an environment variable, especially if we start with the
really stupid fix of setting said variable to our -jN so that it can
propagate, is really simple. In the longer run, setting an environment
variable to indicate how to make use of a job server is probably better.)

--
Matthew

[Maxim Kalaev]

One of ninja's core features is performance, which you probably sacrifice by doing these nested build system invocations.

(If Evan wants to support make's job queue chances are it's a good idea, but I'd try to get my build into a state where this wouldn't be needed instead.)

 

We have 3d party libraries (and linux kernel in some cases) built as part of the top-level project. Top-level project is built using ninja, 3d part libraries typically use standard GNU approach autoconf+make, and the build rule for these is simply invoking 'make -j something'. It's not practical converting these to be built using ninja directly.
I think having ninja supporting make's job queue would be beneficial for such projects. I was thinking to implement that it in the past myself ('redo' has a clean implementation I planned to refer to) but that hasn't proven to be critical and was neglected.

[Matthew Woehlke]

On 2014-07-26 16:20, Nico Weber wrote:

> On Fri, Jul 25, 2014 at 10:03 AM, Matthew Woehlke wrote:
>> On 2014-07-25 12:32, Nico Weber wrote:
>>> Isn't it possible to set up your build in a way that there's
>>> only a single ninja invocation? That should be more efficient,
>>> solves all sub-ninja process issues (there are no sub-ninja
>>> processes), etc.
>>
>> Given that we're talking about CMake, here, CMake would need to have
>> knowledge of when it is running a custom command that is to build
>> another CMake project,
>
> I don't know CMake well, but if there was some CMake build construct that
> said "build_subproject()", then it seems that the generator should have
> enough information to build the subproject with the same generator that's
> currently selected, no?

That's the problem; there is no such thing. In CMake, this boils down to
add_custom_command (or maybe add_custom_target; I didn't check), which
is a generic mechanism that can be used to run *any* executable (even
the regular compiler if you were feeling especially masochistic). CMake
would either need to be newly taught such a command, or else would need
to be taught how to "recognize" when a custom command/target is in fact
calling 'cmake --build' in order to treat it differently. (The former is
probably preferred.)

Also, as was elsewhere stated, for this to work correctly requires that
ninja be able to build such a project where the .ninja files change in
the middle of the build. (Actually it's worse than that; they may not
even exist at the start of the build!) It seems to me this could be very
hard to implement.

Just to be clear, such a build consists of:

configure ProjectA (i.e. creates .ninja files for same)
build ProjectA
configure ProjectB
build ProjectB
[...]

There can be additional per-project steps, but the important part is
that the .ninja files for ProjectB *do not exist* at the start of the
root build. Worse, *generating* them may well (in fact, often *does*)
depend on the *build* of ProjectB having completed first.

IOW, ninja would need to be able to reload the build graph multiple
times in the middle of a build. I'd also be worried about getting
dependencies right in order to do that reload at a consistent spot.
(There's a good chance this would fall to CMake rather than Ninja, but
it's still likely to be complicated. Worse, now you're asking CMake to
carry a whole lot of code that is specific to only one of its
generators, which may not go over well.)

>> This also does nothing for the potential case of ninja -> make -> ninja.
>
> One of ninja's core features is performance, which you probably sacrifice
> by doing these nested build system invocations.

Possibly... Certainly you would have much better scope for interleaving
rules/targets across projects if ninja knew about the entire build. (As
an example, let's say that ProjectA has LibraryA1 and LibraryA2,
ProjectB has LibraryB which depends on LibraryA1, and ProjectC has
LibraryC which depends on LibraryA2 and LibraryB. Right now, for build
correctness it is necessary for ProjectC to depend on ProjectB which in
turn depends on ProjectA. If Ninja knew about everything, LibraryA2 and
LibraryB could be built in parallel.)

I'm not saying anywhere this wouldn't be better :-). Just that it
requires changes to CMake as well as Ninja.

That said, it's still possible that some superbuild contains a
sub-project that *must* be built with Make. (Or worse, some
project-specific tool, e.g. Qt, Boost, etc.) On it's own this isn't
horrible, but things could get really odd if such a project were to turn
around and used ninja again in its bowels.

--
Matthew

[Bill Hoffman]

On 7/26/2014 4:20 PM, Nico Weber wrote:
> I don't know CMake well, but if there was some CMake build construct
> that said "build_subproject()", then it seems that the generator should
> have enough information to build the subproject with the same generator
> that's currently selected, no?

In the case of CMake's external project feature, this is not really
possible. It is basically a complete encapsulation of the configure and
build of a project. It is done in a totally different process and could
be CMake, or autoconf or something else.

Seems like this approach would be the best:

-l flag to base # of processes on the load average.

With gmake, we just call make with $(MAKE) and that links up the jobserver.


Is there a reason the ninja -l would not work for this?

-Bill