Ninja 1.10.0 on Windows keeps rebuilding due to a header wrongly considered as an output

[Stefano Bonicatti]

Hello everyone, I've recently hit a strange issue that I'm having difficulty to debug.
I'm on Windows, using CMake 3.17.1 to generate the build system files for Ninja 1.10.0.

After a full successful build, if I rerun Ninja, it wants to rebuild part of the project.
I've used ninja -d explain to understand what was happening and almost at the start I noticed:
ninja explain: output <path to a header file> of phony edge with no inputs doesn't exist

Also later that same header file is marked as dirty and is the cause of all the rebuilds.
Now that header file is not generated, it's part of the source code that's being built.
I've checked in the CMake code first to find how that file could've been marked as an output but found nothing.

I also figured that if I did something in CMake that would've marked it as an output, it would've been marked as an output in the ninja files.
I couldn't find any trace of it though; I've also saved the output of ninja -t targets all to then process it to be able to pass each target to ninja -t query <target name>,

but even there I couldn't find it.

At this point I have no idea on how to debug this issue (would've been nice to be able to point ninja to the output file and be able to know which target is supposed to output it),

and I cannot share much information about this project because it's private.

Do you have any pointer on how I could go further debugging this issue?
Thanks!

[Scott Graham]

(I find it's almost always the *very* first explain edge that's the problem, not one at almost the start.)

Dumb question though: does <path to a header file> exist, relativized to the build directory?

I'd probably try to create a standalone project you can share, otherwise it's going to be hard for anyone to help, and/or you might figure out what's going on by doing so.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/067d3190-aa7b-471d-88d5-a81c661ccc27o%40googlegroups.com.

[Michael Jones]

The most effective thing you can possibly do to get this fixed is

1) Create a minimal reproduction. This means a ninjafile and associated project files, that have as much stuff from it as possible removed that still produces the behavior that you think is a bug. If you can produce the bug without a ninjafile generator involved (e.g. no CMakeLists.txt needed), that's ideal.

2) Open a ticket on github with your minimal reproduction

[Stefano Bonicatti]

Hello Scott, thanks for the attention!

I skipped the few first lines mentally because they refers to an all target of a couple of subfolders where in CMake there are targets which do not build anything, but those folders are unrelated.

Your question is not dumb at all! I realized later that I should've clarified and, yes, the path shown is indeed correct and the header is present, relative to the root of the build folder.

The problem with creating a standalone project is that I have no idea whatsoever why that header is considered an output, so I wouldn't really know what to do to try to reproduce that issue.
I should also add that this wasn't always happening, so indeed some change in the CMake code made it happen, I can definitely bisect it, but it's something that would take

possibly several hours since it's a big project.

What I mainly wanted to know is if there was a way (different from what I tried already) to understand what is that phony target/edge that considers that header to be its output.

[Scott Graham]

I'm not sure what might be happening, sorry.

I'd try looking through the .ninja file to see where there's a "phony your-header", or otherwise bisecting over CMake seems like it might be the best option since it sounds like it should be conclusive in a fixed amount of time.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/ee977598-d2a2-4981-b84c-a7d0c73a07c4o%40googlegroups.com.

[Stefano Bonicatti]

The tricky part is that I cannot find a reference to that header anywhere; I've already checked the ninja files, unfortunately.

So it seems I'm left with bisecting then, thanks for the help!

To unsubscribe from this group and stop receiving emails from it, send an email to ninja...@googlegroups.com.

[Maksim Gorev]

Hello,

I'm not sure if this is the root cause to the problem here as subject mentions Windows, but I had the same issue on Linux and the reasons for these header files to be not found by ninja are case-sensitive folder or file names. Somewhere in the source code or CMakeLists.txt there are file/folder name references with some letter in wrong case.

Also for Windows there is separate topic on case-sensitivity: https://github.com/ninja-build/ninja/issues/1882 . This fix seems to be a part of ninja 1.11.0. But for me it didn't work.

[Ingo Müller]

I have a similar situation, where I get the following output from `ninja -d explain` (using ninja v1.11.1 on linux):

ninja explain: output /usr/include/tbb/tbb.h of phony edge with no inputs doesn't exist

ninja explain: /usr/include/tbb/tbb.h is dirty

Similar to Stefano, I don't see any occurrence of tbb.h anywhere in my project (LLVM) or the ninja.build file. Also, if I look at the depfile (produced with `ninja -d keepdefile`), that file is not among the dependencies. However, if I grep for `tbb.h` in all of the header files, I do get the following output:

/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/x86_64-linux-gnu/c++/12/bits/c++config.h:829:#  define _GLIBCXX_USE_TBB_PAR_BACKEND __has_include(<tbb/tbb.h>

If I remove that line the message above about a phony edge goes away. This suggest that something is parsing my header files wrongly and detects and edge where there shouldn't be one...

[Evan Martin]

Ninja only parses .ninja files and depfiles, so the only way it can know that path exists is through those.

To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/25722a9b-20d5-426c-b939-edc0a385ea6dn%40googlegroups.com.

[Ben Boeckel]

On Fri, May 12, 2023 at 01:13:04 -0700, 'Ingo Müller' via ninja-build wrote:
> Similar to Stefano, I don't see any occurrence of tbb.h anywhere in my
> project (LLVM) or the ninja.build file. Also, if I look at the depfile
> (produced with `ninja -d keepdefile`), that file is not among the
> dependencies. However, if I grep for `tbb.h` in all of the header files, I
> do get the following output:
>
> /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/x86_64-linux-gnu/c++/12/bits/c++config.h:829:#
> define _GLIBCXX_USE_TBB_PAR_BACKEND __has_include(<tbb/tbb.h>

This is a stdlib header, so it's no wonder that it doesn't appear in
your projects.

> If I remove that line the message above about a phony edge goes away. This
> suggest that something is parsing my header files wrongly and detects and
> edge where there shouldn't be one...

So it seems that GCC is reporting `__has_include` queries in `-MF`
outputs. I'm not 100% sure that this is correct (as it doesn't care if
the file *changes*, but *exists* and, as you found out, existence
queries are not well-defined/handled in such situations). This is
probably worth an issue to GCC (and I'd also coordinate Clang with its
behavior in this area; MSVC should also be checked for its behavior
here just for completeness).

Note that not reporting this would mean that caches and such are
"desynced" with the system state of TBB if built when it is not present
and then rebuilt afterwards (i.e., the cache won't notice the
installation and will gladly provide the no-TBB state even though TBB is
now present). The reverse is probably fine in practice as any truth from
that query is likely to follow immediately with an `#include` of the
queried header.

--Ben

[Ingo Müller]

Thanks for your reply! I checked again, and indeed, the dependency to tbb.h does show up in the dep file. I guess that this makes my observation compatible with your explanation of how ninja works.

The question remains why it ended up there. I double-checked for the original file and a few others that the only mentioning of the tbb.h header in any of the files listed in the dep file is the one that I quoted, which is, in fact, not a dependency.

BTW, as a short-term solution, I can "solve" my problem by simply creating an empty file at the place where the dependency is expected.

[David Turner]

In the case of Ingo, this is very likely that the depfile generated for your output include /usr/include/tbb/tbb.h due to the use of `__has_include(<tbb/tbb.h>)`. I.e. the compiler itself thinks that's an included header, even if this is not the case.

To verify that, try looking at the depfile generated for the output command, e.g.:

```
rm build_dir/.ninja_deps
ninja -C build_dir  -d keepdepfile path/to/output
cat build_dir/path/to/output.d
```

Though the exact path to the dependency file will depend on the exact content of your Ninja build plan.
If tbb/tbb.h is there, it is because the compiler thinks it is included by your sources, and there is nothing Ninja can do about it.

[Ingo Müller]

I checked the commands above and, indeed, the TBB header shows up in the dep file. I guess this means that the compiler I use detects this dependency...

FYI, I have created an issue for LLVM here: https://github.com/llvm/llvm-project/issues/62710.

You received this message because you are subscribed to a topic in the Google Groups "ninja-build" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ninja-build/O1UDnbMDbVs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ninja-build...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/7410ed39-6de3-4659-b6f2-372a45981eean%40googlegroups.com.

[Evan Martin]

This is possibly getting a bit too off-topic, but I recently tried to think through what a build system's behaviors ought to be around missing files and wrote it down here, in the context of a non-Ninja project:

https://github.com/evmar/n2/blob/main/doc/design_notes.md#missing-files

The tbb header you're encountering here is called a 'discovered' input in that text.  Note that Ninja already has special behaviors around 'discovered' inputs (as described in the third bullet in that list), so possibly this case is an argument for making that special behavior even more special.

Unfortunately, at that time I had convinced myself that it's right to always rebuild in this case.  I will try to keep this example in mind if I ever revisit this again...

To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CACYQdL9iJKXs_F8GU-6igKYa6OEBypyvb5xdV77N0%3DcTOQnAUw%40mail.gmail.com.