How to force `ninja` to ignore missing intermediary files?

[Ciprian Dorin Craciun]

Hello all!

I've stumbled upon an issue which I can't figure out how to
**correctly** solve...

Say I have a build graph like {input -> intermediary -> output}. Now
say that the intermediary is almost useless and to make matters worse
it eats a lot of space, thus I would like to get rid of it after the
build is complete.

Now the simplest solution would be to have a single `build output :
rule input` which creates the intermediary and then after generating
the output just deletes it. (I.e. `command = transform-1 <input
>intermediary && tranform-2 <intermediary >output && rm
intermediary`.)


However this doesn't always work when either:
* the intermediary has its own build statement (for whatever reason,
but also see below;)
* you would want to force the intermediary in being built (i.e. ask
explicitly for it;)
* the intermediary might be used by multiple rules as in `input ->
intermediary -> {output-1, output-2}`, but the two outputs are not
part of the same build;


I have tried to use the "order-only" dependency types, and it doesn't
quite work. If after the build I manually remove the intermediary
files, when invoking the build again, `ninja` creates the intermediary
files, although thereafter it doesn't build the actual outputs
anymore...


Any idea on how to solve this?

Thanks,
Ciprian.

[Neil Mitchell]

Hi Ciprian,

My first thought was maybe you didn't have any default statements, and
thus it was building the intermediate because it wanted it as an
end-product - but checking, that isn't the case. I compared Ninja to
my reimplementation of Ninja (where I reimplemented Ninja from
experiments and the manual, see http://shakebuild.com/ninja) and found
my implementation follows the semantics you expected. My small test
case was:

rule run
command = touch $out
build intermediate.txt: run
build out1.txt: run || intermediate.txt
default out1.txt


One way around the problem is to have the rule for the intermediate
file spit out intermediate.big and intermediate.stamp - where the big
file is the real one, and the stamp file is zero bytes. You can then
depend on the stamp file in all the rules, but use the big file in the
rules. Since Ninja doesn't know about the big file, deleting it will
not make any difference.

Thanks, Neil

> --
> You received this message because you are subscribed to the Google Groups "ninja-build" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ciprian Dorin Craciun]

On Sun, Feb 28, 2016 at 11:51 AM, Neil Mitchell <ndmit...@gmail.com> wrote:
> My first thought was maybe you didn't have any default statements, and
> thus it was building the intermediate because it wanted it as an
> end-product - but checking, that isn't the case.

Indeed I forgot to mention that the behaviour I was experienced was
even in the case I called explicitly `ninja output` (thus not the
default rule which builds everything.)

> I compared Ninja to
> my reimplementation of Ninja (where I reimplemented Ninja from
> experiments and the manual, see http://shakebuild.com/ninja) and found
> my implementation follows the semantics you expected. My small test
> case was:
>
> rule run
> command = touch $out
> build intermediate.txt: run
> build out1.txt: run || intermediate.txt
> default out1.txt

I have made a simple test with `make` (the GNU version) for a similar scenario:

~~~~
input :
touch input

intermediary : input
touch intermediary

output : input | intermediary
touch output
~~~~


Then tested it with:

~~~~
make -f ./makefile output

rm ./intermediary

make -f ./makefile output
~~~~


"Unfortunately" even `make` has the same odd behaviour, i.e. it
insists in re-creating the intermediary file although it is not needed
for rebuilding anything being asked for.

Thus this might be one explanation why `ninja` behaves the way it
does, namely to be compliant with `make` (and thus easier to be used
as a drop-in replacement in various build generator tools).

> One way around the problem is to have the rule for the intermediate
> file spit out intermediate.big and intermediate.stamp - where the big
> file is the real one, and the stamp file is zero bytes. You can then
> depend on the stamp file in all the rules, but use the big file in the
> rules. Since Ninja doesn't know about the big file, deleting it will
> not make any difference.

Indeed this would be a solution I haven't though about. Thanks for
pointing it out.

However, although it solves my particular problem, I find this
solution problematic. First of all it could lead to buggy builds
because now Ninja knows nothing about the "actual" file, and imagine
that the developer manually (who uses a build generator thus not being
aware of these "hacks") deletes the "big" file to force Ninja to
re-build its dependencies, which unfortunately doesn't happen anymore.


The second reason would be for build generators. Now I failed to
mention that the Ninja script I use is not written by hand, but
instead generated with my own custom-built (for that purpose) Python
script. This script of mine already doesn't use a lot of Ninja
features (like the `$in` and `$out` variables), but instead relies on
build-scoped variables like `input__whatever_a`, `output__whatever_b`,
etc., thus it wouldn't be hard for me to make it use the "build-stamp"
files you've suggested. But in doing so (and because I want to
minimize the changes to my own Python script), I'll resort to create a
"build-stamp" file for every other output-file, thus doubling the
number of files in my temporary folder, and possibly introducing other
bugs in the process.

Thus I see this "hack" as problematic when applied "on scale".


Perhaps the best solution for this issue to introduce a third category
of dependencies, something as "optional", which behave just like
"order-only" but don't require rebuilding.

Ciprian.

[Claus Klein]

your makefile is wrong!

make handles intermediate files very well:

Claus-MBP:Workspace clausklein$ make clean

rm -f input output

Claus-MBP:Workspace clausklein$ make

touch input

touch intermediary

touch output

rm intermediary

Claus-MBP:Workspace clausklein$ make

make: Nothing to be done for `all'.

Claus-MBP:Workspace clausklein$ 

[Ciprian Dorin Craciun]

On Sun, Feb 28, 2016 at 2:12 PM, Claus Klein <claus...@arcormail.de> wrote:
> your makefile is wrong!

It could well be that my makefile is wrong. However the behaviour I
presented above is still the one I encounter using the makefile I
provided.

Could you perhaps point out where I made a mistake? Or even better
could you paste the makefile you've used in your tests?


Moreover I guess you are using an OSX system, thus perhaps your
"flavour" of `make` differs than the Linux one, which as said is GNU
Make:
~~~~
GNU Make 4.0
Built for x86_64-unknown-linux-gnu
~~~~

This might be a different reason for the difference in the observed behaviour.

Ciprian.

[Neil Mitchell]

Claus attached his Makefile, but since that doesn't show on my mobile
device, I've repasted it here:

.PHONY: all clean
all: output

clean:
$(RM) input output

input:
touch input

.INTERMEDIATE: intermediary
intermediary: input
touch intermediary

output: intermediary
touch output

The key point was he used .INTERMEDIATE to mark the file as an
intermediate using Make syntax, rather than |, which is "order only"
in Makefile, but not quite what you want.

Thanks, Neil

[Ciprian Dorin Craciun]

On Sun, Feb 28, 2016 at 5:33 PM, Neil Mitchell <ndmit...@gmail.com> wrote:
> Claus attached his Makefile, but since that doesn't show on my mobile
> device, I've repasted it here:

Sorry. Didn't see the attachment.


> .INTERMEDIATE: intermediary

>
> The key point was he used .INTERMEDIATE to mark the file as an
> intermediate using Make syntax, rather than |, which is "order only"
> in Makefile, but not quite what you want.

Exactly. I know about the `.INTERMEDIATE` feature of `make`, however
as `ninja` doesn't have any such equivalent feature I didn't include
it in my test. (Basically I just wanted to see if `make` in a similar
situation would behave the same or not. I didn't want to get into a
`make` topic on this list.)


In any case, I find both `ninja` and `make` behaviour odd when it
comes to "order-only" dependencies. (Order-only implies no functional
dependency only "ordering" if they both happen to be built in the same
"session".)


In fact the dependency type I've described is more than "order-only".
Order-only would imply that building `output` doesn't in any way
functionally depend on the contents of `intermediary`, but only that
they should be ordered as such.

However the type of dependency I'm trying to describe is more similar
to a "normal" prerequisite relation, but say of a "weaker" type,
namely: if `output` doesn't exist, then build `intermediary` as we
need it in `output` build; however if `output` already exists, but
`intermediary` is missing then just consider it up-to-date.


A good use-case for such a feature would be say "caches", where for
various reasons (mainly space), a cache could be cleaned, but without
triggering a re-build.

Ciprian.

[Nico Weber]

Since you want to hide your output from ninja in a way, I'd suggest you make the command that writes the large file also write a stamp file (just `touch out.stamp`) and tell ninja that the dependencies are {input -> intermediary -> output}. Then if your input changes, your rule will rerun, but if you delete the large file then nothing should be rebuilt. Could that work?

Nico

[Ciprian Dorin Craciun]

On Sun, Feb 28, 2016 at 6:23 PM, Nico Weber <tha...@chromium.org> wrote:
> Since you want to hide your output from ninja in a way, I'd suggest you make
> the command that writes the large file also write a stamp file (just `touch
> out.stamp`) and tell ninja that the dependencies are {input -> intermediary
> -> output}. Then if your input changes, your rule will rerun, but if you
> delete the large file then nothing should be rebuilt. Could that work?

Neil proposed the same solution, and technically it could work.
(However see my comments in my second email about the implications of
using such a solution.)


Technically I think a minor "tweak" of the proposed solution could be
applied safely in similar situations. (As said see my second email
about what I mean by "safely".)

One could create a rule like `build intermediary intermediary.marker :
some-rule some-input`. Then wherever in the build graph one needs to
use the intermediary, but could allow for it to miss, one should use
`build output : intermediary.marker` (but inside the command use the
`intermediary`). This way one "fools" `ninja` in ignoring the
intermediary file, but still allow for its creation if explicitly
asked for. (Moreover it solves the issue when some rules might
"strongly" depend on intermediary, meanwhile others don't.)

(As a disclaimer I haven't tested the above, but I assume it works.)


Anyway, given that `ninja` tries to be a generic build system,
targeted by many generators, I think such features might be useful,
without needing to resort to this kind of "hacks".

Ciprian.

[Nico Weber]

Why do you think this is a hack? If you don't want your file to be part of your build graph, then make it not part of your build graph :-) Your email says that one problem of this approach is that deleting the large file now no longer causes a rebuild, but isn't that what you wanted to achieve in the first place? For the other problem, if you're worried about too many stamp files, your generator could have some attribute "ignores_outputs" and only use the stamp technique for these. I think the Chromium and Android builds use a stamp file for every custom build action and that seems to work well though, so maybe having many stamp files isn't much of an issue.