Confused about depfiles

[Kaspar Emanuel]

Hi all,

thank you very much for your work on ninja!

I am using an NodeJS module (ninja-build-gen) to write my configure for a browser extension written in Coffeescript. I am using a command line tool called browserify which concatenates js/coffeescript into single files using a module system. Browserify outputs the dependencies of an input file if you give it the --list flag. I have managed to write a rule that generates the depfile and one that depends on it.

rule copy
  command = cp $in $out
rule browserify_deps
  command = echo -n '$target: ' > $out && browserify --debug --extension=".coffee" --transform coffeeify $in --list | tr '\n' ' ' >> $out
rule browserify
  command = browserify --debug --extension=".coffee" --transform coffeeify $in -o $out
  depfile = $out.d
  deps = gcc
build build/chrome/js/main.js: browserify build/.temp-chrome/main.coffee || build/chrome/js/main.js.d
build build/chrome/js/main.js.d: browserify_deps build/.temp-chrome/main.coffee | build/.temp-chrome/main.coffee build/.temp-chrome/fillout.coffee
  target = build/chrome/js/main.js
build build/.temp-chrome/main.coffee: copy src/chrome/coffee/main.coffee
build build/.temp-chrome/fillout.coffee: copy src/common/coffee/fillout.coffee

But this doesn’t behave as expected. Firstly this seems to re-run build build/chrome/js/main.js.d on the second invocation of ninja when no files have changed. Secondly when I touch src/common/fillout.coffee I need to run ninja twice to get build/chrome/js/main.js to rebuild. What am I doing wrong here?

Full source of what I am working on.

Cheers,

Kaspar

​

[Mike Seplowitz]

Kaspar, I think the way that this is intended to work is with a single build/rule set to generate the output file and the depfile at once.

From the manual:

To bring this information into Ninja requires cooperation. On the Ninja side, the depfile attribute on the build must point to a path where this data is written. (Ninja only supports the limited subset of the Makefile syntax emitted by compilers.) Then the command must know to write dependencies into the depfile path.

The example gives a way to invoke gcc to do compilation while generating the dependency file as a side effect -- it would generate both the compiled object file (often $out.o) and the depfile (often $out.d).

In your case, you may need to invoke browserify twice in a single command to produce both files if there isn't a way to get it to do both types of processing in one pass.

[Dirk Pranke]

On Wed, Aug 5, 2015 at 10:59 AM, Mike Seplowitz <msepl...@bloomberg.net> wrote:

Kaspar, I think the way that this is intended to work is with a single build/rule set to generate the output file and the depfile at once.

Exactly.

-- Dirk

 

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kaspar Emanuel]

Hi,

thanks for your answers. Seems like this is a feature tacked on as a workaround for C/C++ compilers and not really targeted at general usage. It would be great for instance if it could just accept a list of files in the depfile, one per line so I don't have to hack a simple output into a makefile format. 

Anyway, I merged my rules into one and came across a different problem:
```

rule copy
  command = cp $in $out
rule browserify

  command = echo -n '$out: ' > $out.d && browserify --extension=".coffee" --transform coffeeify $in --list | tr '\n' ' ' >> $out.d && browserify --extension=".coffee" --transform coffeeify $in -o $out && cat $out.d

  depfile = $out.d
  deps = gcc

build build/chrome/js/main.js: browserify build/.temp-chrome/main.coffee || build/.temp-chrome/browser.coffee build/.temp-chrome/bom_manager.coffee build/.temp-chrome/popup.coffee
build build/.temp-chrome/browser.coffee: copy src/chrome/coffee/browser.coffee
build build/.temp-chrome/bom_manager.coffee: copy src/common/coffee/bom_manager.coffee
build build/.temp-chrome/popup.coffee: copy src/common/coffee/popup.coffee
```

Since Browserify doesn't support adding an include path in a nice way I first copy all the source files over to `build/.temp-chrome`. As you can see I added the copying as an order-only dependency.

The generated `build/chrome/js/main.js.d` (inspected by using cat in the command itself) looks something like this:
```makefile
build/chrome/js/main.js: build/.temp-chrome/main.coffee build/.temp-chrome/bom_manager.coffee
```

The problem is when I `touch src/common/coffee/bom_manager.coffee` I need two invocations of ninja to get main.js to be rebuilt. The first one copies the file over and the second one rebuilds main.js. If I change the copied files to be implicit dependencies then main.js is rebuilt when files it doesn't need are changed (e.g. `popup.coffee`) and I thought that whatever was in the depfile would be an implicit dependency. Is there a good way to do what I want?

Cheers,

Kaspar

[Evan Martin]

On Sun, Aug 9, 2015 at 5:07 PM, Kaspar Emanuel <kaspar...@gmail.com> wrote:
> thanks for your answers. Seems like this is a feature tacked on as a
> workaround for C/C++ compilers and not really targeted at general usage. It
> would be great for instance if it could just accept a list of files in the
> depfile, one per line so I don't have to hack a simple output into a
> makefile format.

Another way of looking at it is that we have to support some format,
and rather than inventing one we reused the format that is already the
standard. I imagine the browserify maintainers might be interested in
making it output this format as well, for the same reason. Another
idea is to wrap up your browserify invocation that munges the .d files
into a helper program and use that as your build command (this is
equivalent, it just makes your build file cleaner).

I'm not certain I followed you, but I think the right way to express
this is with order-only dependencies. (That is, the build edge for
main.js should have an order-only dep on the copy outputs like you
have.)

Can you put a .ninja file and a trace of the commands (run with ninja
-v) into a bug report? I'm curious why the first time you ask ninja
to build main.js, main.js isn't built without any error message.

[Evan Martin]

Made this file:

rule copy
command = cp $in $out

build x/main.cof: copy main.cof
build x/a.cof: copy a.cof
build x/b.cof: copy b.cof

rule webpack
command = echo "main: x/a.cof" > $depfile; touch $out
depfile = $out.d

build main: webpack x/main.cof || x/a.cof x/b.cof


Set things up:
touch main.cof a.cof b.cof

Ran some commands, it works as I'd expect:

alpaca:~/t$ ninja -d explain -v main
ninja explain: depfile 'main.d' is missing
ninja explain: output x/main.cof doesn't exist
ninja explain: x/main.cof is dirty
ninja explain: output x/a.cof doesn't exist
ninja explain: output x/b.cof doesn't exist
[1/4] cp main.cof x/main.cof
[2/4] cp a.cof x/a.cof
[3/4] cp b.cof x/b.cof
[4/4] echo "main: x/a.cof" > main.d; touch main
alpaca:~/t$ ninja -d explain -v main
ninja: no work to do.
alpaca:~/t$ touch a.cof
alpaca:~/t$ ninja -d explain -v main
ninja explain: output x/a.cof older than most recent input a.cof
(1439157739 vs 1439157746)
ninja explain: x/a.cof is dirty
[1/2] cp a.cof x/a.cof
[2/2] echo "main: x/a.cof" > main.d; touch main
alpaca:~/t$ ninja -d explain -v main
ninja: no work to do.
alpaca:~/t$ touch b.cof
alpaca:~/t$ ninja -d explain -v main
ninja explain: output x/b.cof older than most recent input b.cof
(1439157739 vs 1439157753)
[1/1] cp b.cof x/b.cof
alpaca:~/t$ ninja -d explain -v main
ninja: no work to do.
alpaca:~/t$

[Evan Martin]

PS you can make an extra step like

build copies: phony .temp-chrome/foo.coffee .temp-chrome/bar.coffee

and that should allow your other rules to just have an order-only
dependency on "copies", like
build main: .temp-chrome/main.coffee || copies

That just makes the file easier to read, no other real reason to do it.

[Kaspar Emanuel]

Thank you very much for taking a look and trying to get to the root of the issue. I will try and re-create a minimal example. In the meantime you could play with the full source here if you can be bothered (all you would need is to do is install npm and then npm install && export PATH=$PATH:$(pwd)/node_modules/.bin to be able to run the build).

Here is a paste of the build.coffee and the output from invoking ninja. At the end you can see Ninja has to be invoked twice after the touch.

Cheers,

Kaspar

​

[Evan Martin]

Your .d file looks like this:
build/chrome/js/main.js:
/home/kaspar/projects/1clickBOM/code/build/.temp-chrome/messenger.coffee
[...]

Ninja doesn't know that /home/kaspar/projects/... paths are the same
files as relative paths like build/.temp-chrome/bom_manager.coffee.
So the first time you build, it rebuilds the build/... path and thinks
it is done, and then the second time you build, it notices that the
/home/kaspar/... file has changed.

Ninja's philosophy is to not do any work that you could have done
yourself, such as making your paths match up. I know it is
inconvenient, I am sorry! Determining whether two paths are
equivalent in the limit involves expanding symlinks etc. and Ninja
doesn't want to do any work to remain fast.

[Kaspar Emanuel]

Ah, gotcha, thanks, that works now.

FWIW I was expecting it to expand relative paths but would have been surprised if it followed symlinks. Whatever you need to get that speed I guess. It also makes sense since its just listing Make targets, not files, really.
 
Personally I am not that interested in the speed of Ninja since my project is a very small codebase. I was interested in the minimalist nature so I was surprised by some of the GCC/MSVC things tacked on. Makes me feel my use case is a second class citizen. I guess it does have a focus though, which is compiling Chromium and C++ codebases of similar size. Not meant as criticism, just some observations from a newcomer.

Ninja is definitely a great tool and IMO the only attempt I have come across that actually manages to improve on Make without losing the essential (I have already looked at Cake, Gulp, Grunt, Jake and Brunch for this project, coming back to Make each time). I will likely switch my project over to using it completely in the next few weeks. Thank you very much for your help.

[Kaspar Emanuel]

On 10 August 2015 at 08:43, Evan Martin <mar...@danga.com> wrote:

Ninja's philosophy is to not do any work that you could have done
yourself, such as making your paths match up. 

Though it does make the required folders for you (which is really great, because that is a PITA with Make).

Thinking about the speed some more I guess I do benefit from it quite a lot now since I made a ‘watch’ target like this:

rule loop
  command = while true; do ninja -v | grep --invert-match 'ninja: no work to do' && echo '[DONE]'; sleep 0.5s; done;
  description = calling 'ninja -v' in a loop, press Ctrl-C to stop
  pool = console
build watch: loop

Really love that, no need for weird file watchers in the compilers/concatenators and it will copy the static files as well.

​

[Evan Martin]

On Mon, Aug 10, 2015 at 3:59 PM, Kaspar Emanuel <kaspar...@gmail.com> wrote:
> Personally I am not that interested in the speed of Ninja since my project
> is a very small codebase. I was interested in the minimalist nature so I was
> surprised by some of the GCC/MSVC things tacked on. Makes me feel my use
> case is a second class citizen. I guess it does have a focus though, which
> is compiling Chromium and C++ codebases of similar size. Not meant as
> criticism, just some observations from a newcomer.

In this section of the manual I tried to deter people from using Ninja
for small projects for more or less the reasons you mention:
https://martine.github.io/ninja/manual.html#_using_ninja_for_your_project

[Evan Martin]

I'm a little surprised that works! Running Ninja from within Ninja
might have the processes stomp on the same files in bad ways.

An approach I use is to have the file system notify you when the files change.
I put something like this in a shell script:

while inotifywait -r -q -e src; do
make
done

[Fredrik Medley]

As I tried to explain in https://groups.google.com/d/topic/ninja-build/tIcKNLWWKzo/discussion, it is not obvious from the documentation that you need to make all paths relative. I'm not sure if something will break when my third-party libraries in Windows are located on an other drive and therefore require absolute paths. Should it be mentioned that the deps-files need to specify relative paths as well?

Another problematic case is that "ninja -f /home/.../foo.ninja /home/.../bar" won't find any build rule for bar, as it is specified relative in foo.ninja, and won't regenerate foo.ninja, because it is also specified with relative path. Should I create phony rules between relative and absolute paths to avoid such problems? Also casing on Windows will cause similar problems.

By the way, how expensive is it to resolve symlinks?

Best regards,

Fredrik