Performing actions not generating files

[simon.g...@arcor.de]

Hi all,

while fiddling around with various build systems I found myself in the following test driven development scenario: test executables are first generated and then executed. Generation and execution should be performed if and only if necessary.

The good thing is that I managed to implement a simple shake build system modeling this behaviour. My solution relies on shake's digest feature. In order to avoid running a test whose executable was regenerated but did not change, I ended up with separate rules for test generation and execution. Since rules seem to be defined by output rather than action and its dependencies, however, I felt compelled to generating files that I actually don't need. Is there a way around it?

For comparison, I have attached a boiled down example using tup. It might be incorporated into the Build System Shootout.

Thank you very much for your efforts, kind regards,
Simon

[Neil Mitchell]

Hi Simon,

Can you send over your Shake version? I have some ideas about how you might have done it, but it's probably easier to discuss the necessary tweaks you could make from your existing code.

Thanks, Neil

[simon.g...@arcor.de]

Hi Neil,

thank you for having a look at my case. You can find attached a rough shake-equivalent of the tup example.

The build system itself looks as follows.

--
import Development.Shake

main :: IO ()
main = shakeArgs shakeOptions{shakeChange = ChangeDigest} $ do

want ["output.sh"]

-- not really...
want ["output.run"]

"output.sh" *> \out -> do
need ["input.sh"]
cmd "./process.sh" "input.sh" [out]

"output.run" *> \out -> do
need ["output.sh"]
() <- cmd "./output.sh"
cmd "touch" [out]
--

Best regards,
Simon

[Neil Mitchell]

Hi Simon,

As it stands, Shake doesn't have anything that is only run if the
dependencies change, but doesn't produce output. There are a few types
of rule that don't produce output, but they always rerun regardless of
if the dependencies have changed or not (alwaysRerun, phony, action).
That is somewhat intentional, as I find that if you can't prod a rule
to force it to rerun then developing the rule becomes harder.

In the tup example, if nothing had changed (other than perhaps your
$PATH variable), how would you force the rerunning of the test? In the
Shake example you could rm output.run. I've done much the same setup
as you have, and have found the ability to force rerunning very
useful. In particular, I usually make output.run contain the stdout of
the process, namely:

"output.run" *> \out -> do
need ["output.sh"]

Stdout stdout <- cmd "./output.sh"
writeFile' out stdout

I've found that works nicely. I'm also not afraid to use "stamp" files
which are empty files for helping the build system track stuff - it
still gives the correct results, but I agree its a bit ugly
conceptually.

I think it would be feasible to add an noOutput rule to Shake, and
wonder if I should do so?

I also think this would be a good inclusion in the build-shootout.
Please feel free to raise a ticket, or submit a pull request.

Thanks, Neil

[simon.g...@arcor.de]

Hi Neil,

thank you for explaining and sharing your thoughts.

I think that recording stdout in a "stamp" file is a good solution, particularly in the test driven development setting that gave rise to my question, originally. Probably, writing those files would have naturally emerged in the course of further development.

Regarding the Build System Shootout: should I prepare the version discussed here or do you prefer a variant which models unit-testing but depends on gcc?

I was attracted by build systems just recently, so I am afraid that I do not have much to say. I'm certainly not an expert in this area.

If I understood the philosophy of tup rerunning a rule should never be required. There is no "clean" target and it is considered a bug if incremental results differ from those of a clean full rebuild. As far as I know, rules can only be indirectly triggered through their dependencies. Of course, one is always free to manually invoke the commands of a particular rule.

In a basic "input, process, output" model, producing output is the only purpose of processing input. In this view, there is no sense in targets (phony or not) which do not produce output of some kind. Following the tradition of make, perhaps, many build tools narrow (tracked) output to generating files. Thinking about it, this restriction does not limit expressiveness. As you have shown by example, it is always possible to emulate recording arbitrary output as generating files. That said, increasing system complexity in the form of "noOutput" rules might not be worth it. I think that shake is in more than good shape, here. Capturing stdout, stderr and exit code is not only supported but even described in the manual. Thumbs up! ;)

Kind regards, thank you again,
Simon

[Neil Mitchell]

Hi Simon,

> Regarding the Build System Shootout: should I prepare the version discussed here or do you
> prefer a variant which models unit-testing but depends on gcc?

For the Shootout I usually prefer versions that test one particular
build aspect in isolation. In this case it would probably be best just
to model running a command that produces no output but should only be
run if its dependencies change.

> If I understood the philosophy of tup rerunning a rule should never be required. There is no "clean" target and it is
> considered a bug if incremental results differ from those of a clean full rebuild. As far as I know, rules can only
> be indirectly triggered through their dependencies. Of course, one is always free to manually invoke the commands
> of a particular rule.

Yep, that's very much the tup philosophy. Shake disagrees with that
for a few reasons:

1) Sometimes the result of a command depends on things that are not
tracked, e.g. the presence/absence of directories, the $PATH variable
etc. In Tup these things are untracked, and can cause problems. In
Shake these things could be tracked, but realistically I don't think
anyone ever will.

2) Sometimes actions are non-deterministic. Imagine a QuickCheck test
that passes 99% of runs. Occasionally you want to just retry the build
action, and pretend the last time round didn't happen.

3) Sometimes you are not interested in the build-products, but
observations that happen while producing them - e.g. peak process RAM,
time until the first message on stdout, number of files read. It's
useful to rerun a specific part to collect that information with
system monitors.

4) Shake doesn't track the build system rules itself, so sometimes you
want to force a rebuild because the rules have changed. Tup does track
the rules, so this argument doesn't apply to Tup.

Thanks, Neil