Local cache of packages?

[Magnus Therning]

I'm trying to find ways to make our CI builds a little faster. I've
managed to sneak in some Haskell code and use `stack` to build it. The
builds happen inside clean docker images (based on Debian Jessie). I've
pre-installed `stack` itself of course, and also run a `stack setup` to
ensure the compiler is pre-installed. So far so good.

The other day our internet connection was flaky for a few hours, which
reminded me that our builds rely on stackage.org and hackage.org being
reachable while building. It would be nice to remove that requirement. I
guess some sort of caching would be the way to go, but I'm not sure how
to go about it.

Are there nice documents relating to this that google-fu was too weak to
find?

What are the sophisticated Haskell shops using?

/M

--
Magnus Therning OpenPGP: 0x927912051716CE39
email: mag...@therning.org jabber: mag...@therning.org
twitter: magthe http://therning.org/magnus

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter

[Magnus Therning]

(I probably posted the first message to the wrong group.)

Magnus Therning <mag...@therning.org> writes:

> I'm trying to find ways to make our CI builds a little faster. I've
> managed to sneak in some Haskell code and use `stack` to build it. The
> builds happen inside clean docker images (based on Debian Jessie). I've
> pre-installed `stack` itself of course, and also run a `stack setup` to
> ensure the compiler is pre-installed. So far so good.
>
> The other day our internet connection was flaky for a few hours, which
> reminded me that our builds rely on stackage.org and hackage.org being
> reachable while building. It would be nice to remove that requirement. I
> guess some sort of caching would be the way to go, but I'm not sure how
> to go about it.
>
> Are there nice documents relating to this that google-fu was too weak to
> find?
>
> What are the sophisticated Haskell shops using?

After posing the same question for Clojure [1] I thought I'd attempt the
same solution for stack too. That is, I'll tell the CI system to cache
my `.stack-work` folder between builds. This doesn't seem to work tough.
On every build stack still produces output indicating that it's
downloading and building all the dependencies before building my code:

~~~
Linking /root/.stack/setup-exe-cache/x86_64-linux/tmp-setup-Simple-Cabal-1.22.5.0-ghc-7.10.3 ...
extra-1.4.9: download
ansi-terminal-0.6.2.3: download
posix-realtime-0.0.0.4: download
network-2.6.2.1: download
posix-realtime-0.0.0.4: configure
posix-realtime-0.0.0.4: build
ansi-terminal-0.6.2.3: configure
ansi-terminal-0.6.2.3: build
extra-1.4.9: configure
extra-1.4.9: build
network-2.6.2.1: configure
ansi-terminal-0.6.2.3: copy/register
posix-realtime-0.0.0.4: copy/register
ansi-wl-pprint-0.6.7.3: download
... more deps
void-0.7.1: download
void-0.7.1: configure
void-0.7.1: build
void-0.7.1: copy/register
extensible-effects-1.11.0.4: download
extensible-effects-1.11.0.4: configure
extensible-effects-1.11.0.4: build
extensible-effects-1.11.0.4: copy/register
... configure & building my code
~~~

Any pointers would be most welcome.

[1]: http://stackoverflow.com/questions/37945677/getting-leiningen-to-cache-packages

--
Magnus Therning OpenPGP: 0x927912051716CE39
email: mag...@therning.org jabber: mag...@therning.org
twitter: magthe http://therning.org/magnus

There's a big difference between making something easy to use and
making it productive.
— Adam Bosworth

[Magnus Therning]

Erik Hesselink <hess...@gmail.com> writes:

> On 23 June 2016 at 13:27, Magnus Therning <mag...@therning.org> wrote:
>>
>> (I probably posted the first message to the wrong group.)
>>
>> Magnus Therning <mag...@therning.org> writes:
>>
>>> I'm trying to find ways to make our CI builds a little faster. I've
>>> managed to sneak in some Haskell code and use `stack` to build it. The
>>> builds happen inside clean docker images (based on Debian Jessie). I've
>>> pre-installed `stack` itself of course, and also run a `stack setup` to
>>> ensure the compiler is pre-installed. So far so good.
>>>
>>> The other day our internet connection was flaky for a few hours, which
>>> reminded me that our builds rely on stackage.org and hackage.org being
>>> reachable while building. It would be nice to remove that requirement. I
>>> guess some sort of caching would be the way to go, but I'm not sure how
>>> to go about it.
>>>
>>> Are there nice documents relating to this that google-fu was too weak to
>>> find?
>>>
>>> What are the sophisticated Haskell shops using?
>>
>> After posing the same question for Clojure [1] I thought I'd attempt the
>> same solution for stack too. That is, I'll tell the CI system to cache
>> my `.stack-work` folder between builds. This doesn't seem to work tough.
>> On every build stack still produces output indicating that it's
>> downloading and building all the dependencies before building my code:
>

> I think you have to cache $HOME/.stack for this to work. The
> .stack-work directory will only contain packages from your project,
> whereas $HOME/.stack will contain the packages from the snapshot
> you're using.

Ah, when reading up on this I found this bit in the user docs:

Anything which is not installed from a snapshot ends up in the local
database.

I didn't expect stack to be that clever. Since I'm largely depending on
snapshots it means my pre-fetching of ghc and then only caching the
local database isn't enough. I also need to move the stack root into my
build dir, since the CI system (gitlab CI) only can cache things below
the build dir.

A bit of testing confirms that this works nicely. However, it blows up
the cache quite remarkably -- from ~50M to 1.6G!

I posted two follow-up questions at [1].

[1]: http://stackoverflow.com/questions/38054594/caching-stack-databases-between-builds-in-gitlab-ci

/M

--
Magnus Therning OpenPGP: 0x927912051716CE39
email: mag...@therning.org jabber: mag...@therning.org
twitter: magthe http://therning.org/magnus

The brain is a wonderful organ. It starts working the moment you get
up and does not stop until you get into the office
— Robert Frost