discovery: ninja already supports implicit outputs! (at least partially)

[Avinash Baliga]

Hey Evan + Everyone,

In this past week, I discovered that ninja can support implicit outputs, if you carefully produce a certain dependency graph structure.  It hinges on one quirk of the dirty-checking logic, which is arguably a bug -- but I would like to petition for that to be officially declared a feature.

By "implicit output", I mean the nastiest case -- file names that are not knowable until all input files of a build task exist.

For example, java insists on a separate .class file to be created for every inner class declared in a source file.  If any java source code is generated, there is no general solution for predicting the class file names while generating build.ninja.

With a trivial java compilation build rule, the inner class files are completely unknown to ninja. This means if you delete them from disk, ninja will continue to think the build is up-to-date, but some downstream task will fail due to the missing file.  With java it's more likely that the missing file won't even be detected until runtime, since typically the class files are immediately zipped up into a JAR file, which in simple terms is a java loadable library or executable.

How do we fix this?

Have your generator emit two build tasks for every logical command:

1. A 'clean-build' task of the form
   build foo.list A.class B.class : java_compile <A.java B.java>
2. An 'incremental' task of the form
   build foo.fanin : java_compile_fanin foo.list

Where the above tasks' rules' commands abstractly take the following forms:

def invoke_java_compiler():

    call the actual compiler

    scan output directory for inner class files

    write .list file containing names of all inner class files

1. java_compile:
       invoke_java_compiler()
2. java_compile_fanin:
       if any filename in <.list file> does not exist:
           invoke_java_compiler()
       create_dep_file that states .fanin depends on all filenames listed in .list file
       touch .fanin  # contents of this file don't matter, it's only for sequencing in the overall scheme of things
   

To avoid retyping all the details of why this works, please see my full write-up that explains each scenario step-by-step here.  I also used a zip file as an example in there, so you can see the pattern unfold:

https://github.com/fifoforlifo/pynja/blob/master/doc/ImplicitInputsOutputs.md

Which special ninja feature does this rely on?

When ninja performs its dirty checks, apparently if any file listed in a task's deps file is missing, then the task is re-run -- even if the explicitly listed inputs are unchanged.

This could arguably be considered a bug on principle, since a task should depend  on its inputs, not create them.  And with real implicit dependency scenarios, if one implicit input was deleted, another input should also have changed to no longer refer to it, or else the build would be broken.  Other build systems may flag this is a build error "XYZ file does not exist for task T".

However, by keeping this behavior in ninja, we get the very useful result I described above.

So ... can we keep this behavior, and make it official?

So far I can see no downsides to keeping things this way.  It also (happily) would not regress any existing software.

If you made it this far, thanks for reading!

- Avinash

PS: Ninja is great -- it's given me hope that build software will actually get better over time :)

[Maxim Kalaev]

When ninja performs its dirty checks, apparently if any file listed in a task's deps file is missing, then the task is re-run -- even if the explicitly listed inputs are unchanged.

It's not a bug. It is specified in the documentation that ninja automatically adds an implicit phony rule for each dependency mentioned in the .dep files to let build proceed when a header file is removed. :)