Semantics of phony

[Neil Mitchell]

Hi,

Given:

build foo: phony foo.txt
build bar.txt: copy foo

(I've truncated out the relevant bit in this email, but a full example
is at https://gist.github.com/ndmitchell/aaef551b662fb715a258)

If I build bar.txt that builds foo.txt and then bar.txt. If I touch
foo.txt and then build bar.txt nothing rebuilds. That surprised me.

In contrast, if I switch phony for a phony2 rule defined as:

rule phony2
command = cmd /c "echo nothing to do"

Then touching foo.txt does cause bar.txt to rebuild. The manual states
"Semantically, the phony rule is equivalent to a plain rule where the
command does nothing" - which doesn't seem to be the case.

My best guess from trying a range of examples is that phony rules are
always treated as order-only (|| syntax) rules, so:

build bar.txt: copy foo

Is roughly equivalent to:

build bar.txt: copy || foo

But that's only a guess. Is this a bug? Are the semantics of phony a
bit different than what I expect?

Thanks, Neil

[Neil Mitchell]

Hi,

Any thoughts on this issue? I'm still of the opinion this is a bug in
Ninja, but without any clear guidance as to what the semantics of
Ninja are, it's hard to say for sure.

Thanks, Neil

[Richard Geary]

phony is infamous for being poorly defined.  It's been mentioned here a few times.  I agree this is a bug, and should be fixed, as I can't see an advantage of the current implementation.

The reason why this happens is because phony is implemented in an odd way. Last time I looked, Ninja internally uses timestamp 0 as an indicator that a file does does not exist, but also reuses the same timestamp field to track the latest transitive dependency timestamp. Thus the updated foo.txt timestamp has no effect.

I reported this bug 2 1/2 years ago, and submitted a pull request with unit tests, but it's still unfixed https://github.com/martine/ninja/pull/634. Ninja core needs to be refactored to avoid reusing the timestamp field for multiple purposes, but no updates to core ninja have been made in years.

From my perspective, whilst nico does an excellent job with many issues, it feels that the only ninja core issues that get addressed are those affecting Chromium.  The majority of my pull requests and bug reports are still open, after 3 years.  IMHO, this project needs to clean up the pull request & bug list.  The project might benefit from adding a second maintainer who isn't on the Google Chrome or Chromium team.

Richard

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Fredrik Medley]

I've also been encountering problems with phony rules and would like to get the phony rule clarified.

My specific setup includes phony targets for tests. Consider the following example:

build foo.lib: ...

build foo_test1.exe: ...

build foo_test2.exe: ...

build foo_test1.output: runtest foo_test1.exe

build foo_test2.output: runtest foo_test2.exe

build foo_runtests: phony foo_test1.output foo_test2.output

build bar.lib: ...

build bar_runtests: phony

The pattern is that for every library x, have a phony rule x_runtests that will run all the tests, but sometimes there are no tests. This pattern does not work as bar_runtests will always be dirty as it has no inputs. What is the design decision for "If a phony build statement is written without any dependencies, the target will be considered out of date if it does not exist." (cited from the manual)?

Is the solution to change to the following? Why does a build rule without any dependencies be considered up to date but a phony rule always out of date?

rule touch

  command = touch $out

build bar_runtests: phony .dummyalwaysexistingoutput

build .dummyalwaysexistingoutput: touch

In https://github.com/martine/ninja/pull/634, a few situations are defined:

1. output edge does not exist, inputs are not real
2. output edge does not exist, no inputs
3. output edge does not exist, inputs are real, newest mtime is M
4. output edge is real, inputs are not real
5. output edge is real, no inputs
6. output edge is real, inputs are real, newest mtime is M

 

Expected results :
1. Edge is marked as clean, mtime is newest mtime of dependents.
    Touching inputs will cause dependents to rebuild.
2. Edge is marked as dirty, causing dependent edges to always rebuild
3. Edge is marked as clean, mtime is newest mtime of dependents.
    Touching inputs will cause dependents to rebuild.
4. Edge is marked as clean, mtime is newest mtime of dependents.
    Touching inputs will cause dependents to rebuild.
5. Edge is marked as dirty, causing dependent edges to always rebuild
6. Edge is marked as clean, mtime is newest mtime of dependents.
    Touching inputs will cause dependents to rebuild.

In my configuration I would rather see 2 and 5 to never rebuild, but I understand that such change would definitely break a lot of projects out there. There is, as I said a workaround.

What is the reasoning behind "phony can also be used to create dummy targets for files which may not exist at build time."? I don't understand the use. Does this correspond to the points 4, 5 and 6?

Best regards,

Fredrik

[Nico Weber]

On Fri, May 22, 2015 at 7:09 AM, Richard Geary <richar...@gmail.com> wrote:

phony is infamous for being poorly defined.  It's been mentioned here a few times.  I agree this is a bug, and should be fixed, as I can't see an advantage of the current implementation.

The reason why this happens is because phony is implemented in an odd way. Last time I looked, Ninja internally uses timestamp 0 as an indicator that a file does does not exist, but also reuses the same timestamp field to track the latest transitive dependency timestamp. Thus the updated foo.txt timestamp has no effect.

I reported this bug 2 1/2 years ago, and submitted a pull request with unit tests, but it's still unfixed https://github.com/martine/ninja/pull/634. Ninja core needs to be refactored to avoid reusing the timestamp field for multiple purposes, but no updates to core ninja have been made in years.

From my perspective, whilst nico does an excellent job with many issues, it feels that the only ninja core issues that get addressed are those affecting Chromium.  The majority of my pull requests and bug reports are still open, after 3 years.  IMHO, this project needs to clean up the pull request & bug list.

(Just to address this specific thing, I did make a scrub of the bug and pull request queue two months ago and brought it down to a single page. In general, I feel that ninja is working pretty well as-is, and I'm conservative in merging pull requests that change tricky internals if only a single person needs them. Pull requests that have lots of "that would be useful to me too" comments are more likely to be merged, see e.g. the SIGTERM patch.)

(About this thread: I don't know much of the history of why phony behaves the way it does, and I'm hoping someone who does know will chime in.)

[Evan Martin]

On Fri, May 22, 2015 at 7:09 AM, Richard Geary <richar...@gmail.com> wrote:

> phony is infamous for being poorly defined. It's been mentioned here a few
> times. I agree this is a bug, and should be fixed, as I can't see an
> advantage of the current implementation.

The relevant history of phony is simply that I didn't think it through
very well.

In Make the ".PHONY" attribute is on the file (Node in Ninja terms),
while in Ninja I made it a special edge type for reasons I don't even
remember. Looking at the commit history it looks like I first
implemented by hand (just a "rule phony" block) and then discovered
the various reasons you need to special-case it.

(It might be the case that a special edge type is accidentally the
right design. I am just saying it's not the product of any careful
thought.)

[Neil Mitchell]

In my mind, there are two semantics of phony that make sense, both of
which I believe to be equivalent. Then there is the semantics that
Ninja actually uses, which (to me) seems wrong.

SEMANTICS 1

A phony rule is a normal rule, but where the file it is read/writing
from is uniquely generated, and stored in such a way that it is
cleared before each run. Specifically:

build foo: phony ...

Is equivalent to:

rule phony_rule:
command = touch $out

build temporary_and_unique_storage_location/foo: phony_rule ...

And in addition, you rm temporary_and_unique_storage_location before
running Ninja.

This is the semantics the manual strongly hints at.

SEMANTICS 2

A phony rule is just an alias, so anywhere you have foo, you expand it out.

build foo: phony bar baz

Is equivalent to doing a substitution of foo for "bar baz" everywhere
it is used in Ninja.

This has the complication that if you have:

build foo: phony bar || baz

Then you need to substitute "bar || baz", and thus you need rules that say:

a (b || c) d ==> a b d || c
a || (b || c) ==> a || b c

This is the semantics that Shake follows when implementing Ninja, and
I believe is equivalent to the first semantics.

Shake currently treats order-only dependencies on a phony as normal
dependencies on a phony, since I wanted to pin down what Ninja did
first, which is when I discovered the surprising behaviour.

SEMANTICS 3

What Ninja currently does. As far as I can tell, this seems to be
approximately treating all phony dependencies as order-only. But that
might just be in my set of test cases, or it might be an emergent
property of the implementation.

Thanks, Neil

[jvp...@g.rit.edu]

On Monday, June 1, 2015 at 1:19:16 PM UTC-5, Evan Martin wrote:

In Make the ".PHONY" attribute is on the file (Node in Ninja terms),
while in Ninja I made it a special edge type for reasons I don't even
remember.  Looking at the commit history it looks like I first
implemented by hand (just a "rule phony" block) and then discovered
the various reasons you need to special-case it.

I think the current semantics get some things right, but the main thing I think it gets wrong is that "phony" isn't a rule, it's a class of rules. In Make, the canonical examples of PHONY are 1) aliases to other targets (e.g. `make all`), and 2) commands that don't directly produce files relevant to the rest of the build system (e.g. `make check` or `make install`). Ninja's phony rule covers the alias case, and should arguably have been named "alias". To cover the second case, I think Ninja would need to add a rule-level variable to indicate that the "output" for that rule isn't a real file. Then, the existing "phony" rule could just be implemented as:

rule phony
  phony = 1

Likewise, one could define a "command" rule for running arbitrary commands:

rule command
  command = $cmd
  phony = 1
build test: command
  cmd = run-tests

Obviously, you could also create more-specific phony rules instead of just a generic "command" rule, but the idea is about the same.

- Jim