needHasChanged returns zero files, even if last build was killed and didn't finish

[rob...@gmail.com]

I use a build tool which translates a set of source files to a new set of different source files (j2objc - Java to Objective-C), and it works most efficiently when it is given all changed files as input to one run. It is able to take a set of changed Java files and figure out the total set of output files which would be affected by them, and regenerate just them.

In Shake I create a rule for a dummy.stamp file which I write to after the translation has run.
In this action I found "needHasChanged <all java files>" to be very useful, in that it returns the list of only those files which have changed, which I can then pass directly to j2objc. But, I found that if I kill the build before it has finished, then "needHasChanged" will return zero files next time! It seems like it records the changed files in the database before knowing that the action has succeeded.

Is there a way to make this work reliably, even in the face of Ctrl-C?

- Robin

[Neil Mitchell]

Hi Robin,

Thanks for the email. You're right, needHasChanged has a terrible
issue with not being persistent - something that I never considered
before.

The current implementation approximately sets a flag to True when a
file rebuilds, but that flag is serialised as False, so it can tell
the difference between something created in this run vs something
created in a previous run. As you have spotted, Ctrl-C breaks that,
but so would lots of other things like doing a build with the input to
need, then doing another build. All in all, it's a hack that isn't
robust to any of the normal invariants you expect.

Fortunately, I can think of a better implementation. Every Shake rule
records the timestamp when it last ran and the timestamp when it last
changed. You just need to compare the timestamp against those for the
files which will easily tell you if the file has changed since this
rule was run previously. I'll have a hack and see how easy that is to
do...

In the meantime I'd consider needHasChanged haunted at best, and
malicious at worst. Sorry!

Thanks, Neil

[Neil Mitchell]

Hi Robin,

I've pushed a fix to the repo, are you able to try it out? I intend to
make a release including it tomorrow. It uses the technique described
in the previous email.

Thanks, Neil

[Neil Mitchell]

I've released shake-0.17.2 which fixes this issue.

Thanks, Neil

[Robin Eidissen]

Hi!

Sorry, I wasn't able to test it until now, but it works perfectly. Thanks!

- Robin