Add new feature for ninja concerning build times

[Giora Guttsait]

Hello Evan,
I hope it's okay to address you about ninja by email and not only by github.
I have this thing I have to do with ninja - 
I need to add a flag to our build command to output build-time per subninja, so that we could analayze that output and figure out why some subninjas take long to build.
As i've noted in my github issue, I have a possible way of partially doing this.
I'm working on a bash script to wrap the command in the build.ninja file, so that it will time each rule when I run the build. The build.ninja files are generated each build, so I can modify the command if I activate the flag or not. Currently I have issues with timing the command in bash, so I'll probably redo the script to python.
But regarding the second problem, I can't link a rule to a subninja.
A way that I could have done that is, theoratically, if Ninja exported an env variable, like it does with $out, just that it would be $subninja. That way I could easily filter each rule to a specific subninja.
Would it be possible to add that feature? I didn't come across anything related to timing subninjas that's already built in ninja, so I assume that would have to be added.
Sincerely, giora guttsait.
.....
For example, the output that I seek to achieve is either a csv file or anything that I can analayze.
If I had the $subninja variable, I would basically log each subninja in a seperate file, for example:
subninja_1.csv:
cc,0.342
sh,0.090
copy, 0.030
py-test-and-zip,1.050
total,1.512
and I would have several of those files, and eventually I'll output a final file, which would basically be the total of each subninja
subninja_1, 1.512
subninja_2, 0.452
subninja_3, 1.145
Thanks again.

This is the email I send to one of the Github ninja owners (I suppose) and he recommended showing this here.
My team manager handed me an assignment to get the build time of each target (subninja, if you may call it).
My way of doing it was dynamically, if needed, wrap the normal command of each rule with a timing script that would do that.
Problem is, we have a several hundred targets (we used to build with Scons, so we still have SconsXML and we use them to generate our ninja.build files, so by Scons vocabulary we have like 7K targets, each of is either a static library, a program, a test program, a script and whatsoever).

So there are a couple of suggestions:

The most simple one would be exporting a $subninja variable, like you do with $out and $target (these are the ones I know about), and that would help sorting each rule output to a subninja, but there is still a script needed, which is an obstacle on its own.
The less simple suggestion would be a feature for you to do, which would be a build time log output.
Each target/subninja would be timed, and a csv or whatever log would be outted for analysis. I have some examples on the email above me.

I would be glad to hear from you about my suggestions as soon as possible :)

Sincerely,
Giora Guttsait

[Nico Weber]

It sounds like you want figure out where time is spent in your build. There are a few third-party libraries for this. https://chromium.googlesource.com/infra/infra/+/master/appengine/chromium_build_stats/ is what generates the trace files used in Chromium's infrastructure (example: http://chromium-build-stats.appspot.com/ninja_log/2015/06/28/build51-m1/ninja_log.build51-m1.chrome-bot.20150628-134112.944.gz/trace.html – navigate with wasd). https://github.com/nico/ninjatracing is a simpler version of the same thing. Maybe that could be useful for you?

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Evan Martin]

> The most simple one would be exporting a $subninja variable, like you do
> with $out and $target (these are the ones I know about), and that would help
> sorting each rule output to a subninja, but there is still a script needed,
> which is an obstacle on its own.

You can trivially implement this one yourself:
When generating each subninja file, write out a variable at the top like
my_subninja = this_ninja_file.ninja
then within the rules, you can use $my_subninja within the command.

> The less simple suggestion would be a feature for you to do, which would be
> a build time log output.
> Each target/subninja would be timed, and a csv or whatever log would be
> outted for analysis. I have some examples on the email above me.

In your build directory, Ninja writes a file named .ninja_log with
build timings.
Nico linked to some tools that already analyze this format below.

[Giora Guttsait]

about the subninja variable, so you're saying that i write "var=val"  for each ninja.build file of each subninja it would export this variable?

בתאריך יום שבת, 26 בדצמבר 2015 בשעה 21:22:16 UTC+2, מאת Evan Martin:

[Evan Martin]

I think this will work, not sure! When ninja looks up the value of
"my_var" for the command line, it will find the definition in the
local file.

build.ninja:
rule foo
command = foo.py --subninja=$my_var

otherfile.ninja:
my_var = otherfile.ninja
build bar: foo xyz

[Evan Martin]

See the variable lookup order at the bottom of this section:
https://ninja-build.org/manual.html#ref_scope
This idea relies on step #4.