Top-level binding for working directory (aka reconcile depfile paths)

[Brad King]

Hi Folks,

The build manifest (build.ninja) is encouraged to use relative paths,
especially for files that appear beneath the top of the build tree.
Therefore running `ninja` in `<dir>` or `ninja -C <dir>` can only work
with a given build manifest if `<dir>` matches what the build manifest
expects. Using `ninja -f /some/other/build.ninja` does not work unless
the current working directory happens to match what that manifest expects.

I propose that we add a new top-level binding that generators can use
to specify the expected working directory. This can also be interpreted
as the starting point for relative paths in the build manifest. For
example:

```
ninja_workdir = /absolute/path/to/expected/working/directory
```

Ninja could chdir to this location after reading the build manifest(s),
thus ensuring that all the relative paths work as intended.

---------------------------------------------------------------------------

Furthermore, the above feature will help fix a problem with depfile
loading. The ninja manual currently has a warning for depfile content:

```
File paths are compared as is, which means that an absolute path and a
relative path, pointing to the same file, are considered different by Ninja.
```

See issue 1251 [1] for some problems this causes in practice. We can
fix it by using `ninja_workdir`. If a path in the depfile does not
exist in the build manifest then we can check for alternatives matches:

* If the depfile path is absolute and starts with the workdir then we
can strip off the workdir prefix and then check if the relative
path exists instead.

* If the depfile path is relative we can prepend the workdir and then
check if the absolute path exists instead (after CanonicalizePath).

Note that we cannot use `getcwd` for this because it would not
preserve symbolic links that the generator may have used in absolute
paths written to the build manifest. The `ninja_workdir` binding
allows the generator to tell ninja exactly what logical path it used.

I have changes locally using this approach to resolve issue 1251, but
I'd like to get some feedback here before polishing it up for a PR.

Thanks,
-Brad


[1] https://github.com/ninja-build/ninja/issues/1251

[jha...@gmail.com]

What if the generator would always resolve symbolic paths?

[Brad King]

On 4/29/19 11:13 AM, jha...@gmail.com wrote:
> What if the generator would always resolve symbolic paths?

It can't because there are environments in which such paths need
to be preserved. CMake preserves symlinks in paths specified as
the source and build trees and avoids generating relative paths
that have enough `../` components to cross outside of either one.

For example, I've seen network environments where `/home/myuser`
is a symlink to a machine-specific path that changes regularly.
All absolute paths referring to one's home directory must not
resolve that symlink. I've seem similar cases for scratch
directories in HPC environments.

At no point between the generator and expansion of `$in` (and
friends) can symlinks in the path be resolved or paths in the
compiled objects such as debug symbols or `__FILE__` expansion
may be wrong.

-Brad

[Jan Niklas Hasse]

On Mon, 29 Apr 2019, at 17:34, Brad King wrote:
> It can't because there are environments in which such paths need
> to be preserved. CMake preserves symlinks in paths specified as
> the source and build trees and avoids generating relative paths
> that have enough `../` components to cross outside of either one.

How does CMake do that? Doesn't it use getcwd, too?

[Brad King]

On 4/29/19 1:26 PM, Jan Niklas Hasse wrote:
> How does CMake do that? Doesn't it use getcwd, too?

When an absolute path is given (without any `..`) it is
respected. When the current working directory is used as a
default build tree (very common) we check for the case that
`realpath($PWD) == getcwd()` and if so use `$PWD` instead.

Note that issue 1251 can occur even without symlinks, and
in such cases `getcwd()` is sufficient. Therefore that could
be fixed even without `ninja_workdir` by using `getcwd()` to
implement the additional depfile path lookups I described in
the first post of this thread. `ninja_workdir` can then be
considered separately to handle the symlink cases.

-Brad

[Sebastian Kienzl]

Hello,

when doing out-of-source builds, why doesn't CMake use absolute path names for (generated) files/include directories that appear beneath the top of the build dir, like it does for paths in the source dir? What is the rationale behind Ninja's "preference for relative paths in the build manifest"?

If CMake used absolute -- not canonicalized absolute -- path names, the paths in the dependencies output by the compiler would also be absolute and there wouldn't be a mismatch with the names in the Ninja build manifest.

Regards,

Seb.

[Chris Hill]

Hello all,

I'm new to using CMake, CTest and Ninja and found this old discussion [1] of an issue matching my own, which brought me here. I thought I'd share my use-case in the hope it will help validate some of the proposed solutions.

I'm attempting to use Ninja instead of Unix Makefiles with CTest and coverage. As you said in the old discussion, Brad, configure/build/testing completely out-of-source-tree to get absolute paths works with Ninja - almost. It resolves all source/header files included through find_package (properly exported with target_include_directories) but does not resolve sources obtained using FetchContent. In this case, it's gtest header files which cannot be found in "path_to_out_of_source_dir/build/debug/Testing/CoverageInfo/_deps/gtest-src/googletest/include/gtest/". With Unix Makefiles as the generator this is not an issue.

[1] http://cmake.3232098.n2.nabble.com/CMake-Coverage-broken-with-Ninja-td7596219.html

[Claus Klein]

Why not always use realpath() values?

Claus-MBP:test-ninja clausklein$ ninja -t deps

foo.o: #deps 2, deps mtime 1583908096000000000 (VALID)

    /var/folders/wb/ckvxxgls5db7qyhqq4y5_l1c0000gq/T/foo.c

    foo.h

There seems more wrong with ninja dependency tracking, try this:

TMPDIR=$PWD/ sh ./ninja-bug.bash.txt

TMPDIR=/tmp/ sh ./ninja-bug.bash.txt

TMPDIR=      sh ./ninja-bug.bash.txt

[claus.k...@googlemail.com]

Claus-MBP:ninja-build clausklein$ realpath .

/Users/clausklein/Downloads/test-ninja

Claus-MBP:ninja-build clausklein$ TMPDIR=/tmp/ sh ./ninja-bug.bash.txt

initial build:

[1/2] cp foo.h.in foo.h

[2/2] cc -I/Users/clausklein/Downloads/ninja-build/ -c /tmp/foo.c -o /Users/clausklein/Downloads/ninja-build/foo.o -MD -MT /Users/clausklein/Downloads/ninja-build/foo.o -MF /Users/clausklein/Downloads/ninja-build/foo.o.d

--------------------------------------------------

sleep to ensure timestamp change

--------------------------------------------------

modify input

--------------------------------------------------

should fully rebuild:

ninja explain: output foo.h older than most recent input foo.h.in (1583910568000000000 vs 1583910570000000000)

[1/1] cp foo.h.in foo.h

--------------------------------------------------

should build nothing:

ninja explain: output /Users/clausklein/Downloads/ninja-build/foo.o older than most recent input /Users/clausklein/Downloads/ninja-build/foo.h (1583910569000000000 vs 1583910570000000000)

ninja explain: /Users/clausklein/Downloads/ninja-build/foo.o is dirty

[1/1] cc -I/Users/clausklein/Downloads/ninja-build/ -c /tmp/foo.c -o /Users/clausklein/Downloads/ninja-build/foo.o -MD -MT /Users/clausklein/Downloads/ninja-build/foo.o -MF /Users/clausklein/Downloads/ninja-build/foo.o.d

--------------------------------------------------

should build nothing again:

ninja: no work to do.

Claus-MBP:ninja-build clausklein$ ninja -t deps

/Users/clausklein/Downloads/ninja-build/foo.o: #deps 2, deps mtime 1583910570000000000 (VALID)

    /tmp/foo.c

    /Users/clausklein/Downloads/ninja-build/foo.h

Claus-MBP:ninja-build clausklein$ cc -I/Users/clausklein/Downloads/ninja-build/ -c /tmp/foo.c -o /Users/clausklein/Downloads/ninja-build/foo.o -MD -MT /Users/clausklein/Downloads/ninja-build/foo.o -MF /Users/clausklein/Downloads/ninja-build/foo.o.d

Claus-MBP:ninja-build clausklein$ cat *.d

/Users/clausklein/Downloads/ninja-build/foo.o: /tmp/foo.c \

  /Users/clausklein/Downloads/ninja-build/foo.h

Claus-MBP:ninja-build clausklein$ realpath .

/Users/clausklein/Downloads/test-ninja

Claus-MBP:ninja-build clausklein$ 

„Es ist nicht genug, zu wissen, man muß auch anwenden; 
     es ist nicht genug, zu wollen, man muß auch tun.“ 
Johann Wolfgang von Goethe (Werk: Wilhelm Meisters Wanderjahre)

[Brad King]

On 3/11/20 2:48 AM, Claus Klein wrote:
> Why not always use *realpath()* values?

See my earlier response in this thread to the same question:

https://groups.google.com/d/msg/ninja-build/yJvs7u0n2iA/71NTf6RABgAJ

-Brad