ghc using 6+ GB of memory for module with Persistent models and Yesod handlers

[Tero Laitinen]

I used https://github.com/tlaitinen/yesod-dsl to generate some boilerplate code for Persistent models and associated Yesod handlers.

Using ghc 7.4.1 to compile the scaffolded site ( https://www.dropbox.com/s/6gnpdp910s4dkfv/scaffolded-0.0.0.tar.gz ) with optimizations takes about 7 minutes on a i7 920 with 12 GB of memory. The maximum memory usage during compilation exceeds 6 GB.

The obfuscated version of the Haskell module likely to choke ghc is at http://pastebin.com/yFxbDduU

I'm aware that the generated code is very bloated, but I was not expecting such terrible compile behavior.

I'm thinking of splitting the generated code to multiple Haskell modules as a remedy. Are there other ways to improve compile memory usage and time?

Tero

[Felipe Almeida Lessa]

(Disclaimer: I've never used yesod-dsl myself.)

IME, splitting into different modules is the only option. Even so, it
will reduce your compilation times but only by so much. Memory usage
is likely to be greatly improved.

I've had to split my Yesod app many, many times until I got acceptable
compile times. I've even started to use mutually recursive modules in
order to split things like the Foundation module into 6 pieces.

It seems to me that the main problem is that GHC is not able to
iterate its optimizer on the parts of the module that need it, and
instead always loops through the whole thing.

Cheers,

> --
> You received this message because you are subscribed to the Google Groups
> "Yesod Web Framework" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to yesodweb+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Felipe.

[Tero Laitinen]

Thanks for the reply. I'll modify yesod-dsl to output multiple modules and then see what ghc thinks of it. A slight inconvenience is that the .cabal-file needs to be updated, but this can of course be automated. 

Splitting Persistent models definition can be tricky, though, due to dependency cycles, e.g. entity Foo references Bar which references Baz which references Foo, etc.

Tero

[Michael Snoyman]

What version of the text library are you using? Some older versions had some very bad inlining for string constants.

Tero

[Tero Laitinen]

I think the version is text-0.11.1.13-9b63b6813ed4eef16b7793151cdbba4d

[Michael Snoyman]

Try upgrading to at least 0.11.2.0, that might fix the problem.

[Tero Laitinen]

Is there another way to upgrade 'text' than to install it first and then to manually compile all the packages that depend on it? 

If not, I'll wait until text >= 0.11.2 comes automatically with yesod-platform dependencies. 

[Tero Laitinen]

trying with 

cabal install yesod-platform --constraint=text==0.11.3.1 --force-reinstalls

[Erik de Castro Lopo]

Tero Laitinen wrote:

> Is there another way to upgrade 'text' than to install it first and then to
> manually compile all the packages that depend on it?

You could use cabal-dev to install local versions of the latest everything
just for testing.

Cheers,
Erik
--
----------------------------------------------------------------------
Erik de Castro Lopo
http://www.mega-nerd.com/

[Tero Laitinen]

perjantai, 12. heinäkuuta 2013 12.27.04 UTC+3 Erik de Castro Lopo kirjoitti:

Tero Laitinen wrote:

> Is there another way to upgrade 'text' than to install it first and then to
> manually compile all the packages that depend on it?

You could use cabal-dev to install local versions of the latest everything
just for testing.

 

Thanks for the tip. For the moment, I don't have need for other Haskell environments, so I guess starting with a clean ~/.ghc and ~/.cabal allow to experiment with newer packages.

[Tero Laitinen]

This was indeed the culprit. Now it uses "only" 2.4 GB of memory instead of 7.7GB and the compile time was halved. 

perjantai, 12. heinäkuuta 2013 10.17.38 UTC+3 Michael Snoyman kirjoitti: