Oracles and Caches

[Joachim Breitner]

Hi Neil,

I sometimes see shake recompiling everything when I believe it should
not, as the rebuilt targets are insulated from the change file by
oracles, whose result I believe to not change.

I am worried that I am using oracles and caches wrong. So this is what
I do right now: The relevant code is in
https://anonscm.debian.org/cgit/pkg-haskell/pkg-haskell-tools.git/tree/src/make-all.hs?h=6d921e6916643b1a60370f2b2f19fb032157ca61#n273
where I calculate a text file with all data (and I am happy for this to
be recalculated often).

I use a Cache (debBuiltByMap) to read this into memory (using a cache
to avoid reading the file for each query to the oracle), and I use an
oracle (getDebBuiltBy, wrapped in debBuiltBy) to actually query the
file.

The problematic rule is then
https://anonscm.debian.org/cgit/pkg-haskell/pkg-haskell-tools.git/tree/src/make-all.hs?h=6d921e6916643b1a60370f2b2f19fb032157ca61#n350
where I access this data using only the oracles.

Still, I observed this:
# enumerating sources...
# building built-by cache...
# enumerating all binaries...
# dpkg-scanpackages (for lab/haskell-mtl_2.1.3.1-3_amd64.build)
# sbuild (for lab/haskell-mtl_2.1.3.1-3_amd64.build)
^C
where a change to the data (“building built-by cache”) seems to have
triggered a rebuild of the build file

Am I doing something obviously wrong here?

Greetings,
Joachim


--
Joachim “nomeata” Breitner
ma...@joachim-breitner.de • http://www.joachim-breitner.de/
Jabber: nom...@joachim-breitner.de • GPG-Key: 0xF0FBF51F
Debian Developer: nom...@debian.org

[Neil Mitchell]

Hi Joachim,

The best way to figure out what is going on is usually to run with
shakeVerbosity=Diagnostic (-vvv on the command line), which prints out
everything it does and why. In particular, you can see what rebuilds,
what rebuilds and doesn't change etc. The output is very long and
designed so I can entirely debug peoples programs, but it does explain
everything.

The other option is to run profiling, for instance
--profile=profile.json (get some JSON and munge through it yourself)
or profile.html (get some HTML and use the user interface, which might
be a little complex to figure out). Alternatively, give me the output
from the profile and the name of the target you are interested in, and
I can tell you.

I have often thought that the cache should have an "answer" mode,
where you can ask "why did this rebuild" and get an answer. Maybe it
should also be a command line flag --explain=target says why the
target last rebuilt.

The other thing to do is augment writeFileChanged to print when it
decides to write out (I'd start by copying the Shake implementation
into your repo).

A quick look at the oracle and cache pieces reveals nothing untoward.

Thanks, Neil

[Joachim Breitner]

Hi,

Am Dienstag, den 18.08.2015, 17:55 +0100 schrieb Neil Mitchell:
> The best way to figure out what is going on is usually to run with
> shakeVerbosity=Diagnostic (-vvv on the command line), which prints
> out
> everything it does and why. In particular, you can see what rebuilds,
> what rebuilds and doesn't change etc. The output is very long and
> designed so I can entirely debug peoples programs, but it does
> explain everything.

I found that, but had a hard time making sense of it. I did not try it
in this particular instance yet.

> The other option is to run profiling, for instance
> --profile=profile.json (get some JSON and munge through it yourself)
> or profile.html (get some HTML and use the user interface, which might
> be a little complex to figure out). Alternatively, give me the output
> from the profile and the name of the target you are interested in, and
> I can tell you.

Ok, I’ll try that next.

> I have often thought that the cache should have an "answer" mode,
> where you can ask "why did this rebuild" and get an answer. Maybe it
> should also be a command line flag --explain=target says why the
> target last rebuilt.

Oh, that would be great!

> The other thing to do is augment writeFileChanged to print when it
> decides to write out (I'd start by copying the Shake implementation
> into your repo).

Unless I am mistaken, this should not make a difference, as I have set
shakeChange = ChangeModtimeAndDigestInput
so I hope that writeFileChanged and writeFile make no semantic
difference.

> A quick look at the oracle and cache pieces reveals nothing untoward.

Thanks. I’ll debug it further with these options.


Greetings,
Joachim

--