Android now builds with ninja
5,568 views
Skip to first unread message
Evan Martin
Nov 4, 2015, 4:07:24 PM11/4/15
to Colin Cross, ninja-build
This is awesome news!
For some context for the greater outside-of-Google Ninja list, as far
as I know the Android adoption of Ninja was separate from any of the
existing Ninja developers (it was at least a surprise to me when I
heard about it). I also don't know a lot about how all the pieces fit
together.
From my reading, it appears there are two projects around Android and Ninja.
1) kati[1] translates Makefiles to Ninja files.
2) soong[2] appears to use blueprint[3] to generate Ninja files
directly from a higher-level Go description.
It appears that this uses Kati. Do you have any comments on the
status of Soong?
The other thing I'm interested in is performance (of course!). The
announcement says: "Incremental builds with a warm disk cache will
start within 5 seconds.". That seems longer than I'd like -- if you
get a chance, could you run a no-op build with the "-d stats" flag
passed to Ninja and paste your results here? We have collected some
timing numbers in the past on this wiki page that could help put them
in context:
https://github.com/martine/ninja/wiki/Timing-Numbers
[1]: https://github.com/google/kati/blob/master/README.md
[2]: https://android.googlesource.com/platform/build/soong/+/master-soong/doc.go
[3]: https://github.com/google/blueprint
> --
> You received this message because you are subscribed to the Google Groups
> "ninja-build" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ninja-build...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
For some context for the greater outside-of-Google Ninja list, as far
as I know the Android adoption of Ninja was separate from any of the
existing Ninja developers (it was at least a surprise to me when I
heard about it). I also don't know a lot about how all the pieces fit
together.
From my reading, it appears there are two projects around Android and Ninja.
1) kati[1] translates Makefiles to Ninja files.
2) soong[2] appears to use blueprint[3] to generate Ninja files
directly from a higher-level Go description.
It appears that this uses Kati. Do you have any comments on the
status of Soong?
The other thing I'm interested in is performance (of course!). The
announcement says: "Incremental builds with a warm disk cache will
start within 5 seconds.". That seems longer than I'd like -- if you
get a chance, could you run a no-op build with the "-d stats" flag
passed to Ninja and paste your results here? We have collected some
timing numbers in the past on this wiki page that could help put them
in context:
https://github.com/martine/ninja/wiki/Timing-Numbers
[1]: https://github.com/google/kati/blob/master/README.md
[2]: https://android.googlesource.com/platform/build/soong/+/master-soong/doc.go
[3]: https://github.com/google/blueprint
> You received this message because you are subscribed to the Google Groups
> "ninja-build" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ninja-build...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Colin Cross
Nov 4, 2015, 4:31:06 PM11/4/15
to Evan Martin, ninja-build
On Wed, Nov 4, 2015 at 1:07 PM, Evan Martin <mar...@danga.com> wrote:
> This is awesome news!
>
> For some context for the greater outside-of-Google Ninja list, as far
> as I know the Android adoption of Ninja was separate from any of the
> existing Ninja developers (it was at least a surprise to me when I
> heard about it). I also don't know a lot about how all the pieces fit
> together.
>
> From my reading, it appears there are two projects around Android and Ninja.
> 1) kati[1] translates Makefiles to Ninja files.
> 2) soong[2] appears to use blueprint[3] to generate Ninja files
> directly from a higher-level Go description.
>
> It appears that this uses Kati. Do you have any comments on the
> status of Soong?
Soong is the longer term plan, and involves replacing all of the
> This is awesome news!
>
> For some context for the greater outside-of-Google Ninja list, as far
> as I know the Android adoption of Ninja was separate from any of the
> existing Ninja developers (it was at least a surprise to me when I
> heard about it). I also don't know a lot about how all the pieces fit
> together.
>
> From my reading, it appears there are two projects around Android and Ninja.
> 1) kati[1] translates Makefiles to Ninja files.
> 2) soong[2] appears to use blueprint[3] to generate Ninja files
> directly from a higher-level Go description.
>
> It appears that this uses Kati. Do you have any comments on the
> status of Soong?
Android.mk files throughout the tree with structured data files, and
describing the build logic in Go. In order to do a phased transition
from Makefiles to Soong, Kati will translate Makefiles to a Ninja
file, which will be combined with a Ninja file from Soong, and then
all built together with Ninja. Eventually when there are no more
Makefiles, Kati will be removed.
> The other thing I'm interested in is performance (of course!). The
> announcement says: "Incremental builds with a warm disk cache will
> start within 5 seconds.". That seems longer than I'd like -- if you
> get a chance, could you run a no-op build with the "-d stats" flag
> passed to Ninja and paste your results here? We have collected some
> timing numbers in the past on this wiki page that could help put them
> in context:
> https://github.com/martine/ninja/wiki/Timing-Numbers
entirely in ninja. kati needs to do a number of checks before
deciding the build.ninja file is up to date - it checks all referenced
environment variables, modification times on all referenced makefiles,
and reruns all $(wildcard) and $(shell) functions to make sure nothing
has changed.
On my build machine, a no-op build actually takes 7 seconds, around 4
of which are in ninja (1.6.0). Stats:
metric count avg (us) total (ms)
.ninja parse 1 2597182.0 2597.2
canonicalize str 1123566 0.1 147.5
canonicalize path 1123566 0.1 95.0
lookup node 1289308 0.2 276.2
.ninja_log load 1 253438.0 253.4
.ninja_deps load 1 371684.0 371.7
node stat 138522 1.5 206.8
path->node hash load 0.83 (324568 entries / 393241 buckets)
This is with a 328MB build.ninja file with 533k lines, 116k build
statements, 103k rule statements.
Evan Martin
Nov 4, 2015, 5:31:08 PM11/4/15
to Colin Cross, ninja-build
On Wed, Nov 4, 2015 at 1:31 PM, Colin Cross <ccr...@google.com> wrote:
> We run kati every time before running ninja, so the 5 seconds is not
> entirely in ninja. kati needs to do a number of checks before
> deciding the build.ninja file is up to date - it checks all referenced
> environment variables, modification times on all referenced makefiles,
> and reruns all $(wildcard) and $(shell) functions to make sure nothing
> has changed.
>
> On my build machine, a no-op build actually takes 7 seconds, around 4
> of which are in ninja (1.6.0). Stats:
> metric count avg (us) total (ms)
> .ninja parse 1 2597182.0 2597.2
> canonicalize str 1123566 0.1 147.5
> canonicalize path 1123566 0.1 95.0
> lookup node 1289308 0.2 276.2
> .ninja_log load 1 253438.0 253.4
> .ninja_deps load 1 371684.0 371.7
> node stat 138522 1.5 206.8
>
> path->node hash load 0.83 (324568 entries / 393241 buckets)
>
> This is with a 328MB build.ninja file with 533k lines, 116k build
> statements, 103k rule statements.
That input build file is enormous! Parsing it is half the load time.
> We run kati every time before running ninja, so the 5 seconds is not
> entirely in ninja. kati needs to do a number of checks before
> deciding the build.ninja file is up to date - it checks all referenced
> environment variables, modification times on all referenced makefiles,
> and reruns all $(wildcard) and $(shell) functions to make sure nothing
> has changed.
>
> On my build machine, a no-op build actually takes 7 seconds, around 4
> of which are in ninja (1.6.0). Stats:
> metric count avg (us) total (ms)
> .ninja parse 1 2597182.0 2597.2
> canonicalize str 1123566 0.1 147.5
> canonicalize path 1123566 0.1 95.0
> lookup node 1289308 0.2 276.2
> .ninja_log load 1 253438.0 253.4
> .ninja_deps load 1 371684.0 371.7
> node stat 138522 1.5 206.8
>
> path->node hash load 0.83 (324568 entries / 393241 buckets)
>
> This is with a 328MB build.ninja file with 533k lines, 116k build
> statements, 103k rule statements.
I am a little surprised it can be parsed in only 2.5s, in fact. I
guess Nico's work on optimizing the parser paid off. Is it a single
file? In the past I've played with loading multiple files in
parallel...
It seems likely that the input file size and the number of rules
(almost as many as there are files) is the fault of Kati, which I
imagine works at too low of a level to unify rules together. Soong
will certainly do much better on this.
Colin Cross
Nov 4, 2015, 5:38:31 PM11/4/15
to Evan Martin, ninja-build
write out multiple independent files if ninja could parse them in
parallel.
> It seems likely that the input file size and the number of rules
> (almost as many as there are files) is the fault of Kati, which I
> imagine works at too low of a level to unify rules together. Soong
> will certainly do much better on this.
graph to a separate rule + build, soong will have much more structured
output.
jk.m...@ignitarium.com
Nov 29, 2017, 1:14:02 PM11/29/17
to ninja-build
Hi ,
I am building android oreo for the msm8996 chipset and i am getting the error as :
ninja: error: out/soong/build.ninja:493: empty path
10:47:36 ninja failed with: exit status 1
build/core/main.mk:21: recipe for target 'run_soong_ui' failed
make: *** [run_soong_ui] Error 1
no files and generated in the out/soong/.intermediates/ , the libcflags check fails of generating some *.a files in the .intermediates/
is this becuase of some parsing issue , where should be the debugging point for this , i wld like to get clarification on how these files in out/soong/.intermediates/ re generated
Reply all
Reply to author
Forward
0 new messages