New feature: Log per target output to disk?

[Daniel Weber]

Hi there,

first of all thanks for creating an incredibly fast build tool :)

I am currently looking into replacing our make-based system with one based on ninja.

There is one feature in our current system that could be implemented in ninja, I just don't know if it contradicts your goals.

We have the ability to create a log file per target. This is for example used to store compiler output and create compiler 

warning reports after a build. The benefit of having individual log files is that you can create a full report even after 

incremental builds.

What do you think about adding a new "rule variable", e.g. log:

  rule cc

     log = ${out}.log

     command = ...

Ninja has the complete command output in a buffer anyway. All it needs to do is dump it to the specified file after 

the command is done (in addition to stdout). 

I know this can easily be done using a wrapper script, which was a reason to reject ninja feature requests. But I'd like 

to work without wrapper scripts whenever possible to avoid the extra overhead of spawning additional processes (our 

builds are running under Windows) and have a minimum of dependencies in the build system.

From my point of view it sounds like little effort for ninja, should have no performance impact as long as it is not used.

But to those build using it it will make things a lot easier and potentially even faster (compared to doing the same with 

wrapper scripts).

Please let me know what you think. I'd like to know what the chances are of such a feature before I start thinking about

how to implement it ;-).

Thanks,

Daniel

[Evan Martin]

It'd be worth trying out the wrapper script approach before you discard it.
Windows process overhead is greater than other platforms, but
StackOverflow indicates it's something like ~20ms. That's pretty
cheap, especially because this isn't in the critical path.

(One reason the idea of Ninja writing files while compiling makes me
nervous is that it puts disk IO in the main Ninja process, which is
currently non-blocking [except for creating output directories]. It's
possible that disk caching means this isn't a problem, but if it was a
problem the way you'd imagine working around it is to put the log
writing into a separate thread or process, which is exactly what the
wrapper-script idea does.)

> --
> You received this message because you are subscribed to the Google Groups
> "ninja-build" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ninja-build...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Scott Graham]

On Windows, the Chromium build uses a wrapper around every command too, for what that's worth

For the .cc compilation it needs to set the parent environment to choose the toolchain type. That one is a .exe wrapper, but all other tools use a python wrapper (generally to filter out spurious output lines, or set up the environment in other ways).

> email to ninja-build+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build+unsubscribe@googlegroups.com.

[Evan Martin]

Just for fun last night I looked into making a log-writing wrapper and was quickly thwarted by the problem that a process might write to stdout or stderr.  O Windows wizard Scott, how would you structure such a program on Windows?

[Scott Graham]

I would refer to the ninja subprocess code, in all likelihood. :) I don't think there's any simpler way to read both if that's the part you're asking about.

One complexity that occurred to me is that the wrapper for the compiler would have to let through the /showIncludes output so that the deps=msvc code can receive that.

[Fabio Porcedda]

Hi all,

could please someone please point me to some Linux wrapper that is able ta save the output of gcc compiler?

Thanks 

Fabio

[Evan Martin]

#!/bin/bash

echo "$@" >>logfile

exec "$@"

--

[Evan Martin]

I guess you could pipe the latter to " | tee logfile" as well.

[Daniel Weber]

Hi again,

you responded quickly back then, sorry for not replying to your reply at all...

Now, almost 4 years later, we are actually working on switching our internal make based build system to ninja.

While working at this we encountered the same challenge again. My colleagues implemented a small patch for 

ninja and created a PR: https://github.com/ninja-build/ninja/pull/1797

Would be great if you could have a look and provide feedback, I still think this would be a valuable addition with 

no overhead if unused.

Regarding your concerns about I/O in the main process: We're already writing all the output to the "console", 

doesn't this have the same or even more potential for blocking / slowing things down?

BR,

Daniel