Re: Local cache of packages?

[Magnus Therning]

(I probably posted the first message to the wrong group.)

Magnus Therning <mag...@therning.org> writes:

> I'm trying to find ways to make our CI builds a little faster. I've
> managed to sneak in some Haskell code and use `stack` to build it. The
> builds happen inside clean docker images (based on Debian Jessie). I've
> pre-installed `stack` itself of course, and also run a `stack setup` to
> ensure the compiler is pre-installed. So far so good.
>
> The other day our internet connection was flaky for a few hours, which
> reminded me that our builds rely on stackage.org and hackage.org being
> reachable while building. It would be nice to remove that requirement. I
> guess some sort of caching would be the way to go, but I'm not sure how
> to go about it.
>
> Are there nice documents relating to this that google-fu was too weak to
> find?
>
> What are the sophisticated Haskell shops using?

After posing the same question for Clojure [1] I thought I'd attempt the
same solution for stack too. That is, I'll tell the CI system to cache
my `.stack-work` folder between builds. This doesn't seem to work tough.
On every build stack still produces output indicating that it's
downloading and building all the dependencies before building my code:

~~~
Linking /root/.stack/setup-exe-cache/x86_64-linux/tmp-setup-Simple-Cabal-1.22.5.0-ghc-7.10.3 ...
extra-1.4.9: download
ansi-terminal-0.6.2.3: download
posix-realtime-0.0.0.4: download
network-2.6.2.1: download
posix-realtime-0.0.0.4: configure
posix-realtime-0.0.0.4: build
ansi-terminal-0.6.2.3: configure
ansi-terminal-0.6.2.3: build
extra-1.4.9: configure
extra-1.4.9: build
network-2.6.2.1: configure
ansi-terminal-0.6.2.3: copy/register
posix-realtime-0.0.0.4: copy/register
ansi-wl-pprint-0.6.7.3: download
... more deps
void-0.7.1: download
void-0.7.1: configure
void-0.7.1: build
void-0.7.1: copy/register
extensible-effects-1.11.0.4: download
extensible-effects-1.11.0.4: configure
extensible-effects-1.11.0.4: build
extensible-effects-1.11.0.4: copy/register
... configure & building my code
~~~

Any pointers would be most welcome.

[1]: http://stackoverflow.com/questions/37945677/getting-leiningen-to-cache-packages

--
Magnus Therning OpenPGP: 0x927912051716CE39
email: mag...@therning.org jabber: mag...@therning.org
twitter: magthe http://therning.org/magnus

There's a big difference between making something easy to use and
making it productive.
— Adam Bosworth

[Erik Hesselink]

On 23 June 2016 at 13:27, Magnus Therning <mag...@therning.org> wrote:
>
> (I probably posted the first message to the wrong group.)
>
> Magnus Therning <mag...@therning.org> writes:
>
>> I'm trying to find ways to make our CI builds a little faster. I've
>> managed to sneak in some Haskell code and use `stack` to build it. The
>> builds happen inside clean docker images (based on Debian Jessie). I've
>> pre-installed `stack` itself of course, and also run a `stack setup` to
>> ensure the compiler is pre-installed. So far so good.
>>
>> The other day our internet connection was flaky for a few hours, which
>> reminded me that our builds rely on stackage.org and hackage.org being
>> reachable while building. It would be nice to remove that requirement. I
>> guess some sort of caching would be the way to go, but I'm not sure how
>> to go about it.
>>
>> Are there nice documents relating to this that google-fu was too weak to
>> find?
>>
>> What are the sophisticated Haskell shops using?
>
> After posing the same question for Clojure [1] I thought I'd attempt the
> same solution for stack too. That is, I'll tell the CI system to cache
> my `.stack-work` folder between builds. This doesn't seem to work tough.
> On every build stack still produces output indicating that it's
> downloading and building all the dependencies before building my code:

I think you have to cache $HOME/.stack for this to work. The
.stack-work directory will only contain packages from your project,
whereas $HOME/.stack will contain the packages from the snapshot
you're using.

Erik

[Magnus Therning]

Ah, when reading up on this I found this bit in the user docs:

Anything which is not installed from a snapshot ends up in the local
database.

I didn't expect stack to be that clever. Since I'm largely depending on
snapshots it means my pre-fetching of ghc and then only caching the
local database isn't enough. I also need to move the stack root into my
build dir, since the CI system (gitlab CI) only can cache things below
the build dir.

A bit of testing confirms that this works nicely. However, it blows up
the cache quite remarkably -- from ~50M to 1.6G!

I posted two follow-up questions at [1].

[1]: http://stackoverflow.com/questions/38054594/caching-stack-databases-between-builds-in-gitlab-ci

/M

--
Magnus Therning OpenPGP: 0x927912051716CE39
email: mag...@therning.org jabber: mag...@therning.org
twitter: magthe http://therning.org/magnus

The brain is a wonderful organ. It starts working the moment you get
up and does not stop until you get into the office
— Robert Frost