Adding lint-check step to a build

[Michael Jones]

Hi,

I would like to run a lint-checker step on some source files as part of the build process. I'm struggling to figure out how this best fits into Ninja as I feel that a lint-checker, for example pyflakes, runs on a source file but doesn't produce a file as output. But I would like a situation where it is only run if the source file has changed and the source is considered the output of the task so that other tasks can chain off that output and work with the original source file.

Is this possible? I can imagine running the lint-checker and then copying the input file to some output location and having the downstream jobs work with the copy but I can't figure out how to get the original file considered as the output. I'm happy to be told if it is complete nonsensical.

I've have tried:

rule lint-check
    command = cp $in $tmp && $pyflakes $in

 

build file.py file.tmp: lint-check file.py
    tmp = file.tmp

default file.py

Hoping that ninja would forgive the fact that file.py is dependent on file.py as file.tmp is there as well. It understandably does not forgive me this. 

Otherwise I feel that a way forward might be to be able to say "is this file newer than the last time I ran this command?" but I'm not sure that is currently expressible and might be flawed for reasons I don't know yet.

Any help would be much appreciated,

Michael 

[Scott Graham]

I might be misunderstanding, but do you really need the linting to be completed before everything else?

Couldn't you just have the rule be "lint ... && touch linting-..-done" and then have default depend on linting-...-done?

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Evan Martin]

I think something like this should work:

rule lint-check

  command = $pyflakes $in && touch $out

build file.py.checked: lint-check file.py

default file.py.checked

Your idea, "is this file newer than the last time I ran this command?" is interesting.  It mostly makes sense but you need some way to tell Ninja that it ought to consider running the command (which my above "default" does, by using an explicit path to represent whether the command was run).

This is actually very related to what I've been thinking about for Ninja v2.  Ninja already actually tracks when a command was run.  If there was some way to express a no-output rule (like lint-check) and still pull it in as a dependency (which currently only works as files) you could express your goal in a more ideal way, without the extra "checked" file.  That still leaves the question of why Ninja should know to run the command, if you didn't ask for it, though..

(Sorry for the confusing paragraph, just thinking aloud.) 

On Thu, Jul 18, 2013 at 3:19 AM, Michael Jones <m.pric...@gmail.com> wrote:

--

[Michael Jones]

Thank you for the responses.

My desire to have the linting completed before other tasks is so that I can chain "install" tasks which won't be triggered if there are issues with the source code.

Thinking on it further I feel I need to extend my ninja generation code a bit so it can track the original file and the temporary files to end up with something like:

rule lint-check

    command = $pyflakes $in && touch $out

 

rule install

    command = cp $in $out

 

build /tmp/location/file.py.checked: lint-check file.py

build /installed/file.py: install file.py | /tmp/location/file.py.checked

 

default /installed/file.py

So then my "install" task still receives my local file.py as the input but knows that the lint task has to run first. I just got stuck thinking in terms of explicit inputs and outputs.

I can see that the "file that only depends on itself and when the command was last run" might be a tricky thing to fit into the system.

More generally, I'd like to express how awesome I think the ninja concept is. I have suffered in the past from build systems where huge amounts of logic were implemented in Makefiles in make's nearly abstraction (and indentation) free syntax purely because you always need the timestamp-based dependency execution. Ninja gives me hope. I like hope.

Michael

[Nico Weber]

On Fri, Jul 19, 2013 at 5:26 PM, Michael Jones <m.pric...@gmail.com> wrote:

Thank you for the responses.

My desire to have the linting completed before other tasks is so that I can chain "install" tasks which won't be triggered if there are issues with the source code.

Thinking on it further I feel I need to extend my ninja generation code a bit so it can track the original file and the temporary files to end up with something like:

rule lint-check

    command = $pyflakes $in && touch $out

 

rule install

    command = cp $in $out

 

build /tmp/location/file.py.checked: lint-check file.py

build /installed/file.py: install file.py | /tmp/location/file.py.checked

 

default /installed/file.py

So then my "install" task still receives my local file.py as the input but knows that the lint task has to run first. I just got stuck thinking in terms of explicit inputs and outputs.

I think you can simplify this if you combine it with Evan's suggestion like so:

rule install

    command = $pyflakes $in && cp $in $out

 

build /installed/file.py: install file.py

 

default /installed/file.py

[Michael Jones]

(Just accidentally sent this to only Nico. Including ninja-build for completeness.)

Hi, thanks for the thought. 

I agree entirely but I'm attempting to set up a system where the users can kind of declare a mini pipeline. I'm hoping to get to the point where they can declare a set of source files as a kind of "source" node in the pipeline and then at the end they'll be able to declare a "python module" node which will take the inputs from and source node and install them as a python module somewhere. In between, the users can declare additional lint check nodes, or stat-counting nodes, or unit tests, or anything else they might want to add. 

This coarse node pipeline then gets converted to a ninja file and I think it is easier to chain whole build statements together than to process the nodes and merge the final commands. Though I might end up with a lot of temporary files like file.py.checked, which I why I was hoping there might be a temporary-file-less way of achieving the same goal. 

It has been a fun investigation so far.

[Evan Martin]

Two related thoughts:

- If your intermediate steps are relatively cheap, you can merge them into a single command:

  lint-check1 $in && link-check2 $in && touch $out

If either fails, both will be repeated, but depending on the circumstances that is fine.

- The nodes in your conceptual pipeline don't need to map directly to ninja build statements.  You can merge two (like in the above) as well as have them write out files on the side that aren't mentioned in your source language.

GYP, the system Ninja was built alongside of, has a higher-level notion of a "library" target that may include running code generators or compiling source; each library expands to multiple Ninja build rules, while dependencies in between libraries will occasionally elide intermediate stamp (".checked") files when GYP figures out that they're unnecessary.