OCaml and shake

[moo...@gmail.com]

Hello,

During my attempt to convert existing project to Shake I've encountered few
problems and the lack of Haskell knowledge doesn't allow me to go further.

a. Rebuilds when command line changes. There are few questions with the
same motif on Stackoverflow, but no real solution (that I can easily grok).
Allusion to defining new rule instance was made, but I'm at a loss as to how,
perhaps someone has an example?
[FWIW lack of this feature is what makes certain speed measurements made in
some Shake papers suspect. IOW comparing Shake to Ninja given that one does
something while the other doesn't, seems a bit unfair]

b. Getting the list of all the dependencies that lead to the target.
a needs [b,c], b needs [d,e,f], getting dep-sorted list ([a,b,c,d,e,f])
would have made life so much easier. For OCaml at least where modules
supplied to the linking stage must be in order.

c. Probably the easiest one:
Making the rule depend on the output of the external (shell) command (something
like git-describe(1) for instance)

P.S. I'm sure that this attempt of mine, while successful in the sense that it
works at all, probably is not the best utilization of Shake, so if someone
would take a look that would be highly appreciated:
http://repo.or.cz/w/llpp.git/shortlog/refs/heads/yabs http://repo.or.cz/w/llpp.git/blob_plain/7047bb9f320531a9e33e95f8b20aaca73663ffa9:/Build.hs

Thanks for Shake, integrating OCaml (with all it's quirks) has never been easier
(GNU Make, OMake, Ninja have all been tried and failed in one way or another)

[Neil Mitchell]

Hi

> a. Rebuilds when command line changes. There are few questions with the
> same motif on Stackoverflow, but no real solution (that I can easily grok).
> Allusion to defining new rule instance was made, but I'm at a loss as to how,
> perhaps someone has an example?

The sad answer, is that there is no good answer in general, but often
for specific instances there are workable solutions that in practice
do everything you need. If you change a lot of stuff, just change
shakeVersion and everything will rebuild. If you have particular rules
that are likely to have a lot of code changes which effect the result
(e.g. a generator in Haskell) then a need on the source file of part
of the build system works. If you are storing metadata about the build
separate to the rules (either in a separate file or in the Haskell
itself), then you can track the rules.

Looking at your particular instance, I would suggest moving cclib and
cmos up to the top of the file, and then all the regularly changing
metadata is captured by the variables at the top. Once you've done
that, you can push it into a config file (which is probably more
appropriate since it has hardcoded paths in it), then use
https://hackage.haskell.org/package/shake-0.15.5/docs/Development-Shake-Config.html
to dependency track the values in the config file individually. Then
you can increment shakeVersion if the rules fundamentally change, but
most of the time you'll just be changing your metadata, and won't need
any changes.

You could also track the output of cm' directly, probably using an
Oracle, but that's a little more complex (still quite feasible
though), and moving the variables out of the Haskell file is probably
valuable anyway.

> [FWIW lack of this feature is what makes certain speed measurements made in
> some Shake papers suspect. IOW comparing Shake to Ninja given that one does
> something while the other doesn't, seems a bit unfair]

http://neilmitchell.blogspot.co.uk/2014/05/build-system-performance-shake-vs-ninja.html
(which is probably the most complete description of the performance
results) includes "Shake does not implement the Ninja feature of
rebuilding when the command line changes - adding that feature would
be unlikely to have any impact on the full build but may slightly slow
down the Shake zero build.". I am reasonably confident the feature
could be implemented using the existing features of Shake with minimal
overhead. For the full build, minimal overhead has almost no impact on
the results. For the zero build it may be measurable.

> b. Getting the list of all the dependencies that lead to the target.
> a needs [b,c], b needs [d,e,f], getting dep-sorted list ([a,b,c,d,e,f])
> would have made life so much easier. For OCaml at least where modules
> supplied to the linking stage must be in order.

The trick you need is
http://shakebuild.com/manual#generated-transitive-imports. Basically
for each file you generate a .dep file being the things that directly
imports. You then generate a .deps file being the transitive closure,
but you can also do that in dependency order. Then when linking you
just read the .deps file and that's the information. You probably
wouldn't actually want the automatic need-dependencies, since then
you'd end up with things like config keys as dependencies.

> c. Probably the easiest one:
> Making the rule depend on the output of the external (shell) command (something
> like git-describe(1) for instance)

This could mean one of two things:

1) You want a rule to capture the output of git-describe, in which
case Stdout contents <- cmd "git describe" should work. I suspect you
don't mean that, as you've already used that formulation in the code.

2) You want to rebuild a certain file when git-describe changes, in
which case alwaysRerun
(https://hackage.haskell.org/package/shake-0.15.5/docs/Development-Shake.html#v:alwaysRerun)
or oracles (https://hackage.haskell.org/package/shake-0.15.5/docs/Development-Shake.html#g:11)
will both work. The alwaysRerun formulation tends to be a bit simpler.

> P.S. I'm sure that this attempt of mine, while successful in the sense that it
> works at all, probably is not the best utilization of Shake, so if someone
> would take a look that would be highly appreciated:
> http://repo.or.cz/w/llpp.git/shortlog/refs/heads/yabs http://repo.or.cz/w/llpp.git/blob_plain/7047bb9f320531a9e33e95f8b20aaca73663ffa9:/Build.hs

Seems entirely reasonable to me.

> Thanks for Shake, integrating OCaml (with all it's quirks) has never been easier
> (GNU Make, OMake, Ninja have all been tried and failed in one way or another)

Great!

Thanks, neil

[moo...@gmail.com]

Neil Mitchell <ndmit...@gmail.com> writes:

> Hi
>
>> a. Rebuilds when command line changes. There are few questions with the
>> same motif on Stackoverflow, but no real solution (that I can easily grok).
>> Allusion to defining new rule instance was made, but I'm at a loss as to how,
>> perhaps someone has an example?
>
> The sad answer, is that there is no good answer in general, but often
> for specific instances there are workable solutions that in practice
> do everything you need. If you change a lot of stuff, just change
> shakeVersion and everything will rebuild. If you have particular rules
> that are likely to have a lot of code changes which effect the result
> (e.g. a generator in Haskell) then a need on the source file of part
> of the build system works. If you are storing metadata about the build
> separate to the rules (either in a separate file or in the Haskell
> itself), then you can track the rules.
>

It feels like that if a rule is semi-pure (side effects are limited to
`cmd _') it should be doable (just produce the concatenation of all
strings passed to the cmds and compare that across runs), but at the
same time it also feels like that's probably an oversimplification on my
part.

> Looking at your particular instance, I would suggest moving cclib and
> cmos up to the top of the file, and then all the regularly changing
> metadata is captured by the variables at the top. Once you've done
> that, you can push it into a config file (which is probably more
> appropriate since it has hardcoded paths in it), then use
> https://hackage.haskell.org/package/shake-0.15.5/docs/Development-Shake-Config.html
> to dependency track the values in the config file individually. Then
> you can increment shakeVersion if the rules fundamentally change, but
> most of the time you'll just be changing your metadata, and won't need
> any changes.

Thanks.

>
> You could also track the output of cm' directly, probably using an
> Oracle, but that's a little more complex (still quite feasible
> though), and moving the variables out of the Haskell file is probably
> valuable anyway.
>

Yes it is.

>> [FWIW lack of this feature is what makes certain speed measurements made in
>> some Shake papers suspect. IOW comparing Shake to Ninja given that one does
>> something while the other doesn't, seems a bit unfair]
>
> http://neilmitchell.blogspot.co.uk/2014/05/build-system-performance-shake-vs-ninja.html
> (which is probably the most complete description of the performance
> results) includes "Shake does not implement the Ninja feature of
> rebuilding when the command line changes - adding that feature would
> be unlikely to have any impact on the full build but may slightly slow
> down the Shake zero build.". I am reasonably confident the feature
> could be implemented using the existing features of Shake with minimal
> overhead. For the full build, minimal overhead has almost no impact on
> the results. For the zero build it may be measurable.

Thing is - one has to store/load those commands somewhere and those tend
to be on the lengthy side of things (especially for C and multiple
different cflags (pkg-config or whatnot produced)), this can impact the
performance quite a bit (looking at generated .cmd files after building
a linux kernel can give one a taste)

>
>> b. Getting the list of all the dependencies that lead to the target.
>> a needs [b,c], b needs [d,e,f], getting dep-sorted list ([a,b,c,d,e,f])
>> would have made life so much easier. For OCaml at least where modules
>> supplied to the linking stage must be in order.
>
> The trick you need is
> http://shakebuild.com/manual#generated-transitive-imports. Basically
> for each file you generate a .dep file being the things that directly
> imports. You then generate a .deps file being the transitive closure,
> but you can also do that in dependency order. Then when linking you
> just read the .deps file and that's the information. You probably
> wouldn't actually want the automatic need-dependencies, since then
> you'd end up with things like config keys as dependencies.
>

Let me see if I understand this correctly (I certainly don't in the case
of the final sentence, so please elaborate), sorry haven't read the
cited link yet:

a. inside the compilation rules the stuff that is fed to the need
function is also recorded in the file
b. inside the linking rule the generated file(s) are read and that is
used to construct the command line passed to the linker

It looks like an ad-hoc version of:
a. inside the compilation rules the stuff is stored in IORefs of
sorts
b. linker rule uses
c. upon completion shake stores those in it's database

Or maybe I'm confused and this is not at all what you are hinting at.

>> c. Probably the easiest one:
>> Making the rule depend on the output of the external (shell)
>> command (something
>> like git-describe(1) for instance)
>
> This could mean one of two things:
>
> 1) You want a rule to capture the output of git-describe, in which
> case Stdout contents <- cmd "git describe" should work. I suspect you
> don't mean that, as you've already used that formulation in the code.
>

Correct, this is not what I mean.

> 2) You want to rebuild a certain file when git-describe changes, in
> which case alwaysRerun
> (https://hackage.haskell.org/package/shake-0.15.5/docs/Development-Shake.html#v:alwaysRerun)
> or oracles
> (https://hackage.haskell.org/package/shake-0.15.5/docs/Development-Shake.html#g:11)
> will both work. The alwaysRerun formulation tends to be a bit simpler.
>

I've tried, not very hard I suppose, to find some examples of oracle
usage to better familiarize myself the concept - but failed, perhaps
you've got some other links? (can't say that hackage was very
illuminating)

[..snip..]

--
mailto:moo...@gmail.com

[Neil Mitchell]

Apologies for brevity - I'm on a mobile phone only til Tuesday. 

On Saturday, 29 August 2015, <moo...@gmail.com> wrote:

Neil Mitchell <ndmit...@gmail.com> writes:

> Hi
>
>> a. Rebuilds when command line changes. There are few questions with the
>>    same motif on Stackoverflow, but no real solution (that I can easily grok).
>>    Allusion to defining new rule instance was made, but I'm at a loss as to how,
>>    perhaps someone has an example?
>
> The sad answer, is that there is no good answer in general, but often
> for specific instances there are workable solutions that in practice
> do everything you need. If you change a lot of stuff, just change
> shakeVersion and everything will rebuild. If you have particular rules
> that are likely to have a lot of code changes which effect the result
> (e.g. a generator in Haskell) then a need on the source file of part
> of the build system works. If you are storing metadata about the build
> separate to the rules (either in a separate file or in the Haskell
> itself), then you can track the rules.
>

It feels like that if a rule is semi-pure (side effects are limited to
`cmd _') it should be doable (just produce the concatenation of all
strings passed to the cmds and compare that across runs), but at the
same time it also feels like that's probably an oversimplification on my
part.

No, that is exactly right. You could even sugar up something that did it automatically with a restrictive type signature - eg gave back a list of files. I suspect you are throwing away many good bits though - pushing build logic exclusively into command lines (as Ninja and Make do) results in lots of sed where trivial Haskell is clearer and more portable. 

Shake already stores a log entry for each file. The trick is you only need to store the hash of the command, as you only care about equality. That means for full builds you end up storing perhaps 10 additional bytes where you already store 100 and the cost is dominated by running the rule anyway, so it will be insignificant. For the zero builds you now have to compute and hash the command line, which is cheap, but not entirely free. 

Not really - it's proper dependencies so far more robust than an IORef and fully tracked. If the link doesn't make it clear let me know and I'll describe it more fully a when I have a keyboard and a compiler so can provide real examples. 

 

>> c. Probably the easiest one:
>>    Making the rule depend on the output of the external (shell)
>> command (something
>>    like git-describe(1) for instance)
>
> This could mean one of two things:
>
> 1) You want a rule to capture the output of git-describe, in which
> case Stdout contents <- cmd "git describe" should work. I suspect you
> don't mean that, as you've already used that formulation in the code.
>

Correct, this is not what I mean.

> 2) You want to rebuild a certain file when git-describe changes, in
> which case alwaysRerun
> (https://hackage.haskell.org/package/shake-0.15.5/docs/Development-Shake.html#v:alwaysRerun)
> or oracles
> (https://hackage.haskell.org/package/shake-0.15.5/docs/Development-Shake.html#g:11)
> will both work. The alwaysRerun formulation tends to be a bit simpler.
>

I've tried, not very hard I suppose, to find some examples of oracle
usage to better familiarize myself the concept - but failed, perhaps
you've got some other links? (can't say that hackage was very
illuminating)

The example under addOracle is pretty much textbook how I'd expect a git describe, just swap out ghc --version.

Thanks, Neil 

[moo...@gmail.com]

Neil Mitchell <ndmit...@gmail.com> writes:

> Apologies for brevity - I'm on a mobile phone only til Tuesday.
>
> On Saturday, 29 August 2015, <moo...@gmail.com> wrote:
>

>> Neil Mitchell <ndmit...@gmail.com <javascript:;>> writes:
>>
>> > Hi
>> >

[..snip..]

>>
>> I've tried, not very hard I suppose, to find some examples of oracle
>> usage to better familiarize myself the concept - but failed, perhaps
>> you've got some other links? (can't say that hackage was very
>> illuminating)
>
>
> The example under addOracle is pretty much textbook how I'd expect a git
> describe, just swap out ghc --version.
>

After (not so) quick fight with oracles and ghc itself (hard to estimate
who put up a better fight) I believe it is now working with
oracles. Relevant rule depends on `git describe` and ocaml productions
depend on the name of the command that executes ocaml compiler and it's
arguments. It feels like I'm almost there only few tiny details to iron
out + Shake.Config utilization (famous last words I suppose).

World would have been a better place if every machine had a working GHC
installation and one wouldn't have to come up with workarounds so that
mere mortals would be able to build his stuff.

If anyone's interested and perhaps for future reference - here are the
gory difftails:
http://repo.or.cz/w/llpp.git/commitdiff/39e10d247ba8b580ab605c8633e119645a53c05b

--
mailto:moo...@gmail.com

[moo...@gmail.com]

Neil Mitchell <ndmit...@gmail.com> writes:

Two more things came up:

a. Capturing and postprocessing stderr on `cmd' failure, f.e.

(Stderr emsg, Exit code) <- cmd Shell "echo moo 1>&2; false"

If `code == ExitFailure _' I need to be able to run some
transformation function on emsg, output its result to stderr and
convince shake that the rule has failed, my Haskell is not up to
snuff and the error messages GHC produces make me want to cringe.

I can always cop out and use a script (that's basically what Ninja
based build system was doing), but it should be possible to do this
in Shake without resorting to hacks.

b. After adding cmdline oracles I couldn't help but notice how shakedb
have grown up substantionally, light analysis of the file shows that
it is choke full of repeated command lines, in OCaml the problem
could have been mitigated by "interning" the strings before
marshalling, perhaps something similar is possible here too (I
suspect that this could also help with C dependencies, even in
modestly sized projects there will be quite a lot of duplicated
strings)

[..snip..]

--
mailto:moo...@gmail.com

[Neil Mitchell]

> a. Capturing and postprocessing stderr on `cmd' failure, f.e.
>
> (Stderr emsg, Exit code) <- cmd Shell "echo moo 1>&2; false"
>
> If `code == ExitFailure _' I need to be able to run some
> transformation function on emsg, output its result to stderr and
> convince shake that the rule has failed, my Haskell is not up to
> snuff and the error messages GHC produces make me want to cringe.

It should be as simple as:

fail $ toUpper emsg

Equally, using error instead of fail would work too. If you have a
concrete example, along with the error, let us know. Some of GHC's
errors aren't ideal.

> I can always cop out and use a script (that's basically what Ninja
> based build system was doing), but it should be possible to do this
> in Shake without resorting to hacks.

Indeed, avoiding a script should make the system much more pleasant
and maintainable.

> b. After adding cmdline oracles I couldn't help but notice how shakedb
> have grown up substantionally, light analysis of the file shows that
> it is choke full of repeated command lines, in OCaml the problem
> could have been mitigated by "interning" the strings before
> marshalling, perhaps something similar is possible here too (I
> suspect that this could also help with C dependencies, even in
> modestly sized projects there will be quite a lot of duplicated
> strings)

How big has the database become? You may find that the repeated
command lines are actually different versions of the same string,
since oracles change in each iteration, so they will be rewritten on
each build (even if nothing actually gets built). Every so often, the
build system will compact out old versions, and then I'd expect each
command line to be different. We have a lot of that at work, but it's
never been a problem.

Note that the Shake database does have some limited form of
application-specific interning - there are lots of pointers that share
things where they can. But it's not as complete as every string.

Thanks, Neil

[Ryan Gonzalez]

Ok: this is slightly off-topic and useless...

...but...


...does anyone else here feel the irony of using a Haskell tool to build OCaml code?

--
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.

[Neil Mitchell]

Remember the first Haskell compiler was written in ML (Lazy ML specifically)...

But people from ML may want to try https://github.com/janestreet/jenga, which started out Heavily Shake inspired (although I don't know it's current state)

[rudi.g...@gmail.com]

Jenga isn’t cross platform unfortunately. It definitely doesn’t work on Windows and last time I’ve tried it didn’t even work on OSX.

[moo...@gmail.com]

Ryan Gonzalez <rym...@gmail.com> writes:

> Ok: this is slightly off-topic and useless...
>
> ...but...
>
>
> ...does anyone else here feel the irony of using a Haskell tool to
> build OCaml code?
>

I do feel sad, but take some consolation in the fact that it's
Shake/Haskell that builds the stuff not Ninja/C++. FWIW - orinally my
own toy OCaml based build system was used, but it was never released and
will stay that way.

[..snip..]

--
mailto:moo...@gmail.com

[moo...@gmail.com]

Neil Mitchell <ndmit...@gmail.com> writes:

>> a. Capturing and postprocessing stderr on `cmd' failure, f.e.
>>
>> (Stderr emsg, Exit code) <- cmd Shell "echo moo 1>&2; false"
>>
>> If `code == ExitFailure _' I need to be able to run some
>> transformation function on emsg, output its result to stderr and
>> convince shake that the rule has failed, my Haskell is not up to
>> snuff and the error messages GHC produces make me want to cringe.
>
> It should be as simple as:
>
> fail $ toUpper emsg
>
> Equally, using error instead of fail would work too. If you have a
> concrete example, along with the error, let us know. Some of GHC's
> errors aren't ideal.
>

After some wrestling I managed to pull it off, but the result isn't
pretty, mostly due to the lack of regex substitution in base and/or my
petty haskell skills.

>> I can always cop out and use a script (that's basically what Ninja
>> based build system was doing), but it should be possible to do this
>> in Shake without resorting to hacks.
>
> Indeed, avoiding a script should make the system much more pleasant
> and maintainable.
>
>> b. After adding cmdline oracles I couldn't help but notice how shakedb
>> have grown up substantionally, light analysis of the file shows that
>> it is choke full of repeated command lines, in OCaml the problem
>> could have been mitigated by "interning" the strings before
>> marshalling, perhaps something similar is possible here too (I
>> suspect that this could also help with C dependencies, even in
>> modestly sized projects there will be quite a lot of duplicated
>> strings)
>
> How big has the database become? You may find that the repeated
> command lines are actually different versions of the same string,
> since oracles change in each iteration, so they will be rewritten on
> each build (even if nothing actually gets built). Every so often, the
> build system will compact out old versions, and then I'd expect each
> command line to be different. We have a lot of that at work, but it's
> never been a problem.
>

llpp% rm -fr build/.shake.database
llpp% du -b build/.shake.database
du: cannot access 'build/.shake.database': No such file or directory
llpp% runghc Build.hs --digest -q
llpp% du -b build/.shake.database
12533 build/.shake.database
llpp% runghc Build.hs --digest -q
llpp% du -b build/.shake.database
13610 build/.shake.database
llpp% runghc Build.hs --digest -q
llpp% du -b build/.shake.database
14687 build/.shake.database

It looks as if you are indeed correct, thanks.

> Note that the Shake database does have some limited form of
> application-specific interning - there are lots of pointers that share
> things where they can. But it's not as complete as every string.
>
> Thanks, Neil

--
mailto:moo...@gmail.com

[moo...@gmail.com]

Neil Mitchell <ndmit...@gmail.com> writes:

> Remember the first Haskell compiler was written in ML (Lazy ML
> specifically)...
>
> But people from ML may want to try https://github.com/janestreet/jenga,
> which started out Heavily Shake inspired (although I don't know it's
> current state)
>

Shake - GHC
Jenga - Jane Street Core as a dependency

I'm not quite sure which one is scarier.

[..snip..]

--
mailto:moo...@gmail.com

[moo...@gmail.com]

Neil Mitchell <ndmit...@gmail.com> writes:

> Hi
>

[..snip..]

>
>> b. Getting the list of all the dependencies that lead to the target.
>> a needs [b,c], b needs [d,e,f], getting dep-sorted list ([a,b,c,d,e,f])
>> would have made life so much easier. For OCaml at least where modules
>> supplied to the linking stage must be in order.
>
> The trick you need is
> http://shakebuild.com/manual#generated-transitive-imports. Basically
> for each file you generate a .dep file being the things that directly
> imports. You then generate a .deps file being the transitive closure,
> but you can also do that in dependency order. Then when linking you
> just read the .deps file and that's the information. You probably
> wouldn't actually want the automatic need-dependencies, since then
> you'd end up with things like config keys as dependencies.

I went with a not so clean MVar approach and oracles [1]

Seems to work, once again the feeling of a kid playing with pretty but
deadly toys that are beyond his comprehension comes to mind.

[1] http://repo.or.cz/w/llpp.git/blob/f125af24bdc7aeaeabe0beeb3060f478e32badb6:/Build.hs
http://repo.or.cz/w/llpp.git/commitdiff/f125af24bdc7aeaeabe0beeb3060f478e32badb6?hp=3f9ee8222a8d88352ed12811f729fb93409936ad


[..snip..]

--
mailto:moo...@gmail.com

[moo...@gmail.com]

On Tuesday, September 1, 2015 at 12:47:27 AM UTC+3, Ryan Gonzalez wrote:
> Ok: this is slightly off-topic and useless...
>
>
>
> ...but...
>
>
>
>
>
> ...does anyone else here feel the irony of using a Haskell tool to build OCaml code?

After getting fed up with my Linux distribution(the way it packages ghc/cabal/etc) decision was made that enough is enough it's time to migrate to another build system. And there it is in all it's glory: https://github.com/moosotc/llpp/blob/master/build.sh

This is not meant as a critique of Shake (I'm forever indebted to it for showing me that dependencies can(and should) be tracked properly), nor ghc/cabal (I'd venture a guess that OPAM/Cargo/Language-packager-du-jour all share the same set of problems), it's just that not being invested in haskell/ghc makes using Shake problematic to say the least (while I can probably understand what GeneralizedNewtypeDeriving does (despite Lennarts less than flattering description of it), the TypeFamilies is definitely outside of my reach.

Shake did one thing that aforementioned build.sh didn't do - parallel builds, then again I could never muster the courage to make Shake work on macOS... So there...

Lastly a bite - Don't you find it ironic that haskell is replaced by posix shell command language to build ocaml code and in the process ~a gigabyte of disk space is saved?

[Ryan Gonzalez]

FYI you can statically compile Shake build systems, so you could
technically use Docker or IsolateKit (personal plug there) to build it and
then just run the prebuilt binary. That way, you're also only messing with
one distro's packaging.

BTW what distro do you use?

--
Ryan (ライアン)
Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else
https://refi64.com/

[moo...@gmail.com]

Ryan Gonzalez <rym...@gmail.com> writes:

> FYI you can statically compile Shake build systems, so you could
> technically use Docker or IsolateKit (personal plug there) to build it
> and then just run the prebuilt binary. That way, you're also only
> messing with one distro's packaging.
>

That's being invested in my book.

> BTW what distro do you use?

Arch.

[..snip..]

--
mailto:moo...@gmail.com

[moo...@gmail.com]

Ryan Gonzalez <rym...@gmail.com> writes:

> FYI you can statically compile Shake build systems, so you could
> technically use Docker or IsolateKit (personal plug there) to build it
> and then just run the prebuilt binary. That way, you're also only
> messing with one distro's packaging.
>
> BTW what distro do you use?
>
> On April 12, 2018 8:23:10 AM moo...@gmail.com wrote:

[..snip..]

And just to put things into perspective:
It takes ~45 seconds ot build OCaml on my machine (i5 with 8 logical/4
physical cores).
It takes 4.5 seconds to build "the product".
It takes few minutes to fetch/build mupdf.

Raltively it takes forever to build shake (I also have to enable swap
otherwise ghc runs out of memory).

- free -h
total used free shared buff/cache available
Mem: 3.7Gi 1.0Gi 937Mi 326Mi 1.8Gi 2.3Gi
Swap: 0B 0B 0B

--
mailto:moo...@gmail.com

[moo...@gmail.com]

moo...@gmail.com writes:

> Ryan Gonzalez <rym...@gmail.com> writes:
>
>> FYI you can statically compile Shake build systems, so you could
>> technically use Docker or IsolateKit (personal plug there) to build it
>> and then just run the prebuilt binary. That way, you're also only
>> messing with one distro's packaging.
>>
>> BTW what distro do you use?
>>
>> On April 12, 2018 8:23:10 AM moo...@gmail.com wrote:
>
> [..snip..]
>
> And just to put things into perspective:
> It takes ~45 seconds ot build OCaml on my machine (i5 with 8 logical/4

Make that 4 2

[Neil Mitchell]

Shame! Never-the-less, its good that it got you to where you needed to
be in the end. Sounds like making GHC faster/leaner would be effort
well spent (a theme that crops up regularly)

Thanks, Neil

[moo...@gmail.com]

Neil Mitchell <ndmit...@gmail.com> writes:

> Shame! Never-the-less, its good that it got you to where you needed to
> be in the end. Sounds like making GHC faster/leaner would be effort
> well spent (a theme that crops up regularly)

Yes, and thank you one more time for that, knowing what is that you need
is a giant step towards making it there... Vague ideas of things that
one wants, do not help (at least they do not help me)

And i'd say that situation with GHC/Cabal, OCaml/OPAM and distros is
atrocious... I can't even build OPAM from source here, all those
external SAT solvers and things of that nature makes my head spin...

It's much easier to just get the source tarball with ocaml and build
things from scratch than to rely on Debian/Arch or other (benevolent yet
strangely malicious) thirdparty to do the things for you..

I mean c'mon:

- tar xf ocaml-4.06.1.tar.xz
~/x/inc
- cd ocaml-4.06.1
~/x/inc/ocaml-4.06.1
- ./configure -prefix /tmp
Configuring OCaml version 4.06.1
/home/malc/x/inc/ocaml-4.06.1/config/auto-aux
[..snip for brevity..]
unix str dynlink bigarray raw_spacetime_lib systhreads threads graph
Configuration for the "graph" library:
options for compiling ....
options for linking ...... -lX11

** OCaml configuration completed successfully **

~/x/inc/ocaml-4.06.1
- \time -p make wolrd.opt -j3 -s
make: *** No rule to make target 'wolrd.opt'. Stop.
Command exited with non-zero status 2
real 0.01
user 0.01
sys 0.00
~/x/inc/ocaml-4.06.1
- \time -p make world -j3 -s
/home/malc/x/inc/ocaml-4.06.1/stdlib
/home/malc/x/inc/ocaml-4.06.1/boot
/home/malc/x/inc/ocaml-4.06.1/stdlib
220 states, 5466 transitions, table size 23184 bytes
2548 additional bytes used for bindings
14 states, 678 transitions, table size 2796 bytes
2232 additional bytes used for bindings
98 states, 1199 transitions, table size 5384 bytes
1802 additional bytes used for bindings
5 shift/reduce conflicts.
5 shift/reduce conflicts.
251 states, 2458 transitions, table size 11338 bytes
41 states, 1026 transitions, table size 4350 bytes
1285 additional bytes used for bindings
50 states, 614 transitions, table size 2756 bytes
20 states, 264 transitions, table size 1176 bytes
111 states, 2871 transitions, table size 12150 bytes
Warning: Element Clflags.include_dirs not found
Warning: Element Config.load_path not found
Warning: Element Clflags.open_modules not found
Warning: Element Clflags.for_package not found
Warning: Element Clflags.debug not found
real 52.49
user 121.47
sys 8.74

[..snip..]

I have no proof but i think GHC build times are a bit larger... Heck if
my memory serves me it takes more time to build Shake (and by more i
mean 'a lot more') than to build OCaml.

--
mailto:moo...@gmail.com