New feature: Log per target output to disk?

145 views
Skip to first unread message

Daniel Weber

unread,
Oct 10, 2016, 10:08:31 AM10/10/16
to ninja-build
Hi there,

first of all thanks for creating an incredibly fast build tool :)
I am currently looking into replacing our make-based system with one based on ninja.

There is one feature in our current system that could be implemented in ninja, I just don't know if it contradicts your goals.
We have the ability to create a log file per target. This is for example used to store compiler output and create compiler 
warning reports after a build. The benefit of having individual log files is that you can create a full report even after 
incremental builds.

What do you think about adding a new "rule variable", e.g. log:

  rule cc
     log = ${out}.log
     command = ...

Ninja has the complete command output in a buffer anyway. All it needs to do is dump it to the specified file after 
the command is done (in addition to stdout). 

I know this can easily be done using a wrapper script, which was a reason to reject ninja feature requests. But I'd like 
to work without wrapper scripts whenever possible to avoid the extra overhead of spawning additional processes (our 
builds are running under Windows) and have a minimum of dependencies in the build system.

From my point of view it sounds like little effort for ninja, should have no performance impact as long as it is not used.
But to those build using it it will make things a lot easier and potentially even faster (compared to doing the same with 
wrapper scripts).

Please let me know what you think. I'd like to know what the chances are of such a feature before I start thinking about
how to implement it ;-).

Thanks,
Daniel

Evan Martin

unread,
Oct 10, 2016, 10:49:48 AM10/10/16
to Daniel Weber, ninja-build
It'd be worth trying out the wrapper script approach before you discard it.
Windows process overhead is greater than other platforms, but
StackOverflow indicates it's something like ~20ms. That's pretty
cheap, especially because this isn't in the critical path.

(One reason the idea of Ninja writing files while compiling makes me
nervous is that it puts disk IO in the main Ninja process, which is
currently non-blocking [except for creating output directories]. It's
possible that disk caching means this isn't a problem, but if it was a
problem the way you'd imagine working around it is to put the log
writing into a separate thread or process, which is exactly what the
wrapper-script idea does.)
> --
> You received this message because you are subscribed to the Google Groups
> "ninja-build" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ninja-build...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Scott Graham

unread,
Oct 11, 2016, 11:14:07 AM10/11/16
to Evan Martin, Daniel Weber, ninja-build
On Windows, the Chromium build uses a wrapper around every command too, for what that's worth

For the .cc compilation it needs to set the parent environment to choose the toolchain type. That one is a .exe wrapper, but all other tools use a python wrapper (generally to filter out spurious output lines, or set up the environment in other ways).


> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build+unsubscribe@googlegroups.com.

Evan Martin

unread,
Oct 11, 2016, 11:25:01 AM10/11/16
to Scott Graham, ninja-build, Daniel Weber

Just for fun last night I looked into making a log-writing wrapper and was quickly thwarted by the problem that a process might write to stdout or stderr.  O Windows wizard Scott, how would you structure such a program on Windows?

Scott Graham

unread,
Oct 11, 2016, 12:02:35 PM10/11/16
to Evan Martin, ninja-build, Daniel Weber
I would refer to the ninja subprocess code, in all likelihood. :) I don't think there's any simpler way to read both if that's the part you're asking about.

One complexity that occurred to me is that the wrapper for the compiler would have to let through the /showIncludes output so that the deps=msvc code can receive that.

Fabio Porcedda

unread,
Mar 28, 2017, 11:01:18 AM3/28/17
to ninja-build, sco...@chromium.org, daniel.w...@gmail.com
Hi all,

could please someone please point me to some Linux wrapper that is able ta save the output of gcc compiler?

Thanks 
Fabio

Evan Martin

unread,
Mar 28, 2017, 5:22:20 PM3/28/17
to Fabio Porcedda, ninja-build, Scott Graham, Daniel Weber
#!/bin/bash

echo "$@" >>logfile
exec "$@"

--

Evan Martin

unread,
Mar 28, 2017, 5:22:41 PM3/28/17
to Fabio Porcedda, ninja-build, Scott Graham, Daniel Weber
I guess you could pipe the latter to " | tee logfile" as well.

Daniel Weber

unread,
Jun 8, 2020, 6:00:25 AM6/8/20
to ninja-build
Hi again,

you responded quickly back then, sorry for not replying to your reply at all...
Now, almost 4 years later, we are actually working on switching our internal make based build system to ninja.
While working at this we encountered the same challenge again. My colleagues implemented a small patch for 

Would be great if you could have a look and provide feedback, I still think this would be a valuable addition with 
no overhead if unused.

Regarding your concerns about I/O in the main process: We're already writing all the output to the "console", 
doesn't this have the same or even more potential for blocking / slowing things down?

BR,
Daniel
Reply all
Reply to author
Forward
0 new messages