Template Haskell in Persistent 2?

[Rehno Lindeque]

Having just ported my schema to persistent, I'm finding that compiling the Model.hs file takes about 40 seconds with such a large number of tables, which I guess must come from using template haskell? I'm going to work around by just generating the non-TH file anyway. Are there any plans to move away from TH in Persistent 2? Perhaps -XDeriveGeneric?

[Greg Weber]

What did you port your schema from?

On Fri, Apr 11, 2014 at 11:14 AM, Rehno Lindeque <rehno.l...@gmail.com> wrote:

Having just ported my schema to persistent, I'm finding that compiling the Model.hs file takes about 40 seconds with such a large number of tables, which I guess must come from using template haskell? I'm going to work around by just generating the non-TH file anyway. Are there any plans to move away from TH in Persistent 2? Perhaps -XDeriveGeneric?

--
You received this message because you are subscribed to the Google Groups "Yesod Web Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rehno Lindeque]

Oh just a regular mysql database, nothing special really. It seems to scale pretty directly with the number of tables: I have 51 entities in there at the moment with around 10 to 20 fields per entity and json instances derived on all of them.

So they're really pretty simple, just something like this:

share [mkPersist sqlOnlySettings] [persistUpperCase| 

 

MyTable json sql=myTable

             -- ....

 

|] 

[Greg Weber]

Is the speed of this compilation the most disruptive part of your workflow? It sounds like you just made a ton of changes to the schema from creating a new one, but once you have a schema in place changes are usually a lot more infrequent, and the TH should only need to get generated once, or perhaps the module dependencies are causing it to get re-compiled when it doesn't need to?

[Rehno Lindeque]

I've yet to see how disruptive it might be to be honest, I'm still refining my process. At the moment I'm doing most of my development with `ghci` open because `:reload` is just a really fast way to typecheck (+ other cool benefits). Some times `:reload` doesn't work though and then I need to `:load` whatever module I happen to be working with. Unfortunately Model.hs is imported almost everywhere that we have business logic so that makes `:load` quite slow... (although, in the mean time, I'm sure I can find a work-around to speed things up).

[Felipe Lessa]

Can't you compile the Model? That way ghci would use the compiled
version, and you'd see the slowdown just when you changed it.

> from it, send an email to yesodweb+u...@__googlegroups.com.
>
> For more options, visit
> https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.

>
>
> --
> You received this message because you are subscribed to the
> Google Groups "Yesod Web Framework" group.
> To unsubscribe from this group and stop receiving emails from

> it, send an email to yesodweb+u...@googlegroups.com <javascript:>.

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Yesod Web Framework" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to yesodweb+u...@googlegroups.com

> <mailto:yesodweb+u...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.

--

Felipe.

[Rehno Lindeque]

I'm still experimenting - thank you for the suggestion... I will let the list know once I've found a solution. I'm afraid that I think the problem might be more involved/severe than I'd thought actually... it looks a bit like compiling Model.hs actually breaks our deploy. I'm only guessing at this point but I think that perhaps ghc running out of memory on the build server? (I'm using https://github.com/ddollar/heroku-anvil to build at the moment). Compilation just stops at Model.hs and then it starts linking immediately:

[16 of 32] Compiling Model            ( Model.hs, dist/build/Model.o )

Loading package ghc-prim ... linking ... done.

Loading package integer-gmp ... linking ... done.

Loading package base ... linking ... done.

Loading package array-0.4.0.1 ... linking ... done.

Loading package stm-2.4.3 ... linking ... done.

Loading package async-2.0.1.5 ... linking ... done.

Loading package deepseq-1.3.0.1 ... linking ... done.

Loading package bytestring-0.10.0.2 ... linking ... done.

Loading package base64-bytestring-1.0.0.1 ... linking ... done.

Loading package byteable-0.1.1 ... linking ... done.

Loading package containers-0.5.0.0 ... linking ... done.

Loading package filepath-1.3.0.1 ... linking ... done.

Loading package old-locale-1.0.0.5 ... linking ... done.

Loading package time-1.4.0.1 ... linking ... done.

Loading package unix-2.6.0.1 ... linking ... done.

Loading package directory-1.2.0.1 ... linking ... done.

Loading package base-unicode-symbols-0.2.2.4 ... linking ... done.

Loading package transformers-0.3.0.0 ... linking ... done.

Loading package transformers-base-0.4.1 ... linking ... done.

Loading package monad-control-0.3.2.3 ... linking ... done.

Loading package lifted-base-0.2.2.1 ... linking ... done.

Loading package mmorph-1.0.2 ... linking ... done.

Loading package mtl-2.1.3.1 ... linking ... done.

Loading package resourcet-0.4.10.2 ... linking ... done.

Loading package text-1.0.0.1 ... linking ... done.

Loading package text-stream-decode-0.1.0.5 ... linking ... done.

Loading package hashable-1.2.1.0 ... linking ... done.

Loading package nats-0.1.2 ... linking ... done.

Loading package unordered-containers-0.2.3.3 ... linking ... done.

Loading package semigroups-0.13.0.1 ... linking ... done.

Loading package void-0.6.1 ... linking ... done.

Loading package conduit-1.0.17.1 ... linking ... done.

Loading package conduit-extra-1.0.0.1 ... linking ... done.

Loading package cryptohash-0.11.4 ... linking ... done.

Loading package cryptohash-conduit-0.1.1 ... linking ... done.

Loading package data-default-class-0.0.1 ... linking ... done.

Loading package data-default-instances-base-0.0.1 ... linking ... done.

Loading package data-default-instances-containers-0.0.1 ... linking ... done.

Loading package dlist-0.7.0.1 ... linking ... done.

Loading package data-default-instances-dlist-0.0.1 ... linking ... done.

Loading package data-default-instances-old-locale-0.0.1 ... linking ... done.

Loading package data-default-0.5.3 ... linking ... done.

Loading package pretty-1.1.1.0 ... linking ... done.

Loading package template-haskell ... linking ... done.

Loading package file-embed-0.0.6 ... linking ... done.

Loading package blaze-builder-0.3.3.2 ... linking ... done.

Loading package utf8-string-0.3.7 ... linking ... done.

Loading package language-javascript-0.5.12 ... linking ... done.

Loading package hjsmin-0.1.4.6 ... linking ... done.

Loading package case-insensitive-1.2.0.0 ... linking ... done.

Loading package http-types-0.8.4 ... linking ... done.

Loading package mime-types-0.1.0.4 ... linking ... done.

Loading package old-time-1.1.0.1 ... linking ... done.

Loading package process-1.1.0.2 ... linking ... done.

Loading package parsec-3.1.5 ... linking ... done.

Loading package system-filepath-0.4.10 ... linking ... done.

Loading package system-fileio-0.3.12 ... linking ... done.

Loading package shakespeare-1.2.1.1 ... linking ... done.

Loading package shakespeare-css-1.0.7.4 ... linking ... done.

Loading package unix-compat-0.4.1.1 ... linking ... done.

Loading package network-2.4.2.2 ... linking ... done.

Loading package vault-0.3.0.3 ... linking ... done.

Loading package wai-2.0.0 ... linking ... done.

Loading package blaze-markup-0.6.0.0 ... linking ... done.

Loading package blaze-html-0.7.0.1 ... linking ... done.

Loading package scientific-0.2.0.2 ... linking ... done.

Loading package attoparsec-0.11.2.1 ... linking ... done.

Loading package http-date-0.0.4 ... linking ... done.

Loading package ansi-terminal-0.6.1.1 ... linking ... done.

Loading package ansi-wl-pprint-0.6.7.1 ... linking ... done.

Loading package optparse-applicative-0.8.0.1 ... linking ... done.

Loading package blaze-builder-conduit-1.0.0 ... linking ... done.

Loading package fast-logger-2.1.5 ... linking ... done.

Loading package stringsearch-0.3.6.5 ... linking ... done.

Loading package byteorder-1.0.4 ... linking ... done.

Loading package unix-time-0.2.2 ... linking ... done.

Loading package wai-logger-2.1.1 ... linking ... done.

Loading package word8-0.0.4 ... linking ... done.

Loading package zlib-0.5.4.1 ... linking ... done.

Loading package zlib-bindings-0.1.1.5 ... linking ... done.

Loading package zlib-conduit-1.0.0 ... linking ... done.

Loading package wai-extra-2.0.3.3 ... linking ... done.

Loading package network-conduit-1.0.4 ... linking ... done.

Loading package simple-sendfile-0.2.13 ... linking ... done.

Loading package warp-2.0.3.4 ... linking ... done.

Loading package wai-app-static-2.0.1 ... linking ... done.

Loading package syb-0.4.1 ... linking ... done.

Loading package primitive-0.5.2.1 ... linking ... done.

Loading package vector-0.10.9.1 ... linking ... done.

Loading package aeson-0.7.0.2 ... linking ... done.

Loading package attoparsec-conduit-1.0.1.2 ... linking ... done.

Loading package cereal-0.4.0.1 ... linking ... done.

Loading package securemem-0.1.3 ... linking ... done.

Loading package crypto-cipher-types-0.0.9 ... linking ... done.

Loading package cipher-aes-0.2.7 ... linking ... done.

Loading package crypto-random-0.0.7 ... linking ... done.

Loading package random-1.0.1.1 ... linking ... done.

Loading package cprng-aes-0.5.2 ... linking ... done.

Loading package entropy-0.2.2.4 ... linking ... done.

Loading package tagged-0.7.1 ... linking ... done.

Loading package crypto-api-0.13 ... linking ... done.

Loading package skein-1.0.9 ... linking ... done.

Loading package clientsession-0.9.0.3 ... linking ... done.

Loading package cookie-0.4.0.1 ... linking ... done.

Loading package exceptions-0.5 ... linking ... done.

Loading package failure-0.2.0.2 ... linking ... done.

Loading package hamlet-1.1.9.2 ... linking ... done.

Loading package monad-loops-0.4.2 ... linking ... done.

Loading package stm-chans-3.0.0.2 ... linking ... done.

Loading package monad-logger-0.3.6 ... linking ... done.

Loading package path-pieces-0.1.3.1 ... linking ... done.

Loading package safe-0.3.4 ... linking ... done.

Loading package shakespeare-i18n-1.0.0.5 ... linking ... done.

Loading package shakespeare-js-1.2.0.4 ... linking ... done.

Loading package yesod-routes-1.2.0.6 ... linking ... done.

Loading package yesod-core-1.2.12 ... linking ... done.

Loading package yesod-static-1.2.2.4 ... linking ... done.

Loading package yaml-0.8.8.1 ... linking ... done.

Loading package binary-0.5.1.1 ... linking ... done.

Loading package SHA-1.6.4 ... linking ... done.

Loading package publicsuffixlist-0.1 ... linking ... done.

Loading package streaming-commons-0.1.1 ... linking ... done.

Loading package http-client-0.3.0.2 ... linking ... done.

Loading package http-client-conduit-0.2.0.1 ... linking ... done.

Loading package socks-0.5.4 ... linking ... done.

Loading package asn1-types-0.2.3 ... linking ... done.

Loading package asn1-encoding-0.8.1.3 ... linking ... done.

Loading package cipher-rc4-0.1.4 ... linking ... done.

Loading package crypto-numbers-0.2.3 ... linking ... done.

Loading package crypto-pubkey-types-0.4.2.2 ... linking ... done.

Loading package crypto-pubkey-0.2.4 ... linking ... done.

Loading package asn1-parse-0.8.1 ... linking ... done.

Loading package pem-0.2.1 ... linking ... done.

Loading package x509-1.4.11 ... linking ... done.

Loading package x509-store-1.4.4 ... linking ... done.

Loading package x509-validation-1.5.0 ... linking ... done.

Loading package tls-1.2.6 ... linking ... done.

Loading package x509-system-1.4.2 ... linking ... done.

Loading package connection-0.2.0 ... linking ... done.

Loading package http-client-tls-0.2.1.1 ... linking ... done.

Loading package http-conduit-2.0.0.10 ... linking ... done.

Loading package xml-types-0.3.4 ... linking ... done.

Loading package xml-conduit-1.2.0.1 ... linking ... done.

Loading package tagstream-conduit-0.5.5.1 ... linking ... done.

Loading package authenticate-1.3.2.8 ... linking ... done.

Loading package email-validate-2.0.1 ... linking ... done.

Loading package mime-mail-0.4.5.1 ... linking ... done.

Loading package resource-pool-0.2.1.1 ... linking ... done.

Loading package silently-1.2.4.1 ... linking ... done.

Loading package persistent-1.3.0.6 ... linking ... done.

Loading package persistent-template-1.3.1.3 ... linking ... done.

Loading package pureMD5-2.1.2.1 ... linking ... done.

Loading package pwstore-fast-2.4.1 ... linking ... done.

Loading package css-text-0.1.2.1 ... linking ... done.

Loading package tagsoup-0.13.1 ... linking ... done.

Loading package xss-sanitize-0.3.5.2 ... linking ... done.

Loading package yesod-persistent-1.2.2.3 ... linking ... done.

Loading package yesod-form-1.3.8.2 ... linking ... done.

Loading package yesod-auth-1.2.7.1 ... linking ... done.

Loading package yesod-1.2.5.2 ... linking ... done.

Loading package shakespeare-text-1.0.2 ... linking ... done.

Loading package querystring-pickle-0.1.9 ... linking ... done.

Loading package mysql-0.1.1.6 ... linking ... done.

Loading package base16-bytestring-0.1.1.6 ... linking ... done.

Loading package blaze-textual-0.2.0.9 ... linking ... done.

Loading package pcre-light-0.4 ... linking ... done.

Loading package mysql-simple-0.2.2.4 ... linking ... done.

Loading package persistent-mysql-1.3.0.2 ... linking ... done.

Loading package transformers-compat-0.1.1.1 ... linking ... done.

Loading package contravariant-0.4.4 ... linking ... done.

Loading package distributive-0.4.3.1 ... linking ... done.

Loading package comonad-4.0.1 ... linking ... done.

Loading package semigroupoids-4.0.1 ... linking ... done.

Loading package bifunctors-4.1.1 ... linking ... done.

Loading package parallel-3.2.0.4 ... linking ... done.

Loading package profunctors-4.0.3 ... linking ... done.

Loading package reflection-1.4 ... linking ... done.

Loading package split-0.2.2 ... linking ... done.

Loading package lens-4.0.7 ... linking ... done.

Loading package esqueleto-1.3.9 ... linking ... done.

Loading package iso3166-country-codes-0.20130302.4 ... linking ... done.

Loading package currency-0.2.0.0 ... linking ... done.

Loading package animalcase-0.1.0.1 ... linking ... done.

 !    Heroku client internal error.

 !    Search for help at: https://help.heroku.com

 !    Or report a bug at: https://github.com/heroku/heroku/issues/new

The "! Heroku client internal error" happens only after waiting for several minutes. Commenting out all but a few of the entities in the Model actually appears solves the problem, I'm just trying to see if I can narrow it down to any specific one right now... Thank you for all the help so far. For reference, I'm on GHC 7.6.3.

[Greg Weber]

It is likely that there is not a specific model that is the problem but that you are instead just out of memory. Can you verify that is the case by running top or free or a monitoring tool? If we get some more specifics we can open a bug report. Memory for compilation is a general issue. We have resolved issues about the model compilation being too slow in the past, but I don't recall reports about it using up too much memory. If you can increase the memory of your build environment that will be your quickest way forward.

To unsubscribe from this group and stop receiving emails from it, send an email to yesodweb+u...@googlegroups.com.

[Michael Snoyman]

This has come up in the past, and at that point, the TH was a red herring (though exactly my first guess as well). It turned out that it was rewrite rules from text firing. The code generate by TH in Persistent uses a lot of string literals. This specific bug was fixed in the text library. Perhaps it's resurfaced, or something else is happening.

I actually have an item on my todo list for this next week to research why a model in our code base takes so long to compile when optimizations are turned on, and I suspect it's related. Can you try compiling with -O0 (possibly just in that module via {-# OPTIONS_GHC -O0 #-}) and see what happens?

On Fri, Apr 11, 2014 at 9:14 PM, Rehno Lindeque <rehno.l...@gmail.com> wrote:

Having just ported my schema to persistent, I'm finding that compiling the Model.hs file takes about 40 seconds with such a large number of tables, which I guess must come from using template haskell? I'm going to work around by just generating the non-TH file anyway. Are there any plans to move away from TH in Persistent 2? Perhaps -XDeriveGeneric?

[Rehno Lindeque]

It is likely that there is not a specific model that is the problem but that you are instead just out of memory. Can you verify that is the case by running top or free or a monitoring tool? If we get some more specifics we can open a bug report. Memory for compilation is a general issue. We have resolved issues about the model compilation being too slow in the past, but I don't recall reports about it using up too much memory. If you can increase the memory of your build environment that will be your quickest way forward.

Fortunately I'm actually getting the same locally now. Memory usage doesn't seem that bad actually - it goes to about 65MB. But compilation takes about 3m20s and then just stops.

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                                                                           

20084 rehno     20   0 2763m 2.4g  15m R  77.1 64.1   2:35.94 ghc  

I have: 

* persistent 1.3.0.6

* text 1.1.0.1

This has come up in the past, and at that point, the TH was a red herring (though exactly my first guess as well). It turned out that it was rewrite rules from text firing. The code generate by TH in Persistent uses a lot of string literals. This specific bug was fixed in the text library. Perhaps it's resurfaced, or something else is happening.

So, -O0 does seem to work...

$ time ghc -O0 -XFlexibleContexts -XTemplateHaskell -XQuasiQuotes -XOverloadedStrings -XTypeFamilies -XDeriveDataTypeable -XGADTs -package-db ******/.cabal-sandbox/x86_64-linux-ghc-7.6.3-packages.conf.d Model.hs

[3 of 3] Compiling Model            ( Model.hs, Model.o )

Loading package ghc-prim ... linking ... done.

Loading package integer-gmp ... linking ... done.

Loading package base ... linking ... done.

Loading package transformers-0.3.0.0 ... linking ... done.

Loading package mtl-2.1.2 ... linking ... done.

Loading package array-0.4.0.1 ... linking ... done.

Loading package stm-2.4.2 ... linking ... done.

Loading package primitive-0.5.0.1 ... linking ... done.

Loading package filepath-1.3.0.1 ... linking ... done.

Loading package deepseq-1.3.0.1 ... linking ... done.

Loading package old-locale-1.0.0.5 ... linking ... done.

Loading package time-1.4.0.1 ... linking ... done.

Loading package bytestring-0.10.0.2 ... linking ... done.

Loading package unix-2.6.0.1 ... linking ... done.

Loading package directory-1.2.0.1 ... linking ... done.

Loading package containers-0.5.0.0 ... linking ... done.

Loading package text-1.0.0.1 ... linking ... done.

Loading package blaze-builder-0.3.3.2 ... linking ... done.

Loading package blaze-markup-0.6.0.0 ... linking ... done.

Loading package blaze-html-0.7.0.1 ... linking ... done.

Loading package fast-logger-2.1.5 ... linking ... done.

Loading package hashable-1.2.1.0 ... linking ... done.

Loading package scientific-0.2.0.2 ... linking ... done.

Loading package attoparsec-0.11.2.1 ... linking ... done.

Loading package dlist-0.7.0.1 ... linking ... done.

Loading package syb-0.4.0 ... linking ... done.

Loading package pretty-1.1.1.0 ... linking ... done.

Loading package template-haskell ... linking ... done.

Loading package unordered-containers-0.2.3.3 ... linking ... done.

Loading package vector-0.10.9.1 ... linking ... done.

Loading package aeson-0.7.0.2 ... linking ... done.

Loading package base-unicode-symbols-0.2.2.4 ... linking ... done.

Loading package transformers-base-0.4.1 ... linking ... done.

Loading package monad-control-0.3.2.3 ... linking ... done.

Loading package lifted-base-0.2.2.1 ... linking ... done.

Loading package mmorph-1.0.2 ... linking ... done.

Loading package resourcet-0.4.10.2 ... linking ... done.

Loading package text-stream-decode-0.1.0.5 ... linking ... done.

Loading package nats-0.1.2 ... linking ... done.

Loading package semigroups-0.13.0.1 ... linking ... done.

Loading package void-0.6.1 ... linking ... done.

Loading package conduit-1.0.17.1 ... linking ... done.

Loading package exceptions-0.5 ... linking ... done.

Loading package monad-loops-0.4.2 ... linking ... done.

Loading package stm-chans-3.0.0.2 ... linking ... done.

Loading package monad-logger-0.3.5.1 ... linking ... done.

Loading package base64-bytestring-1.0.0.1 ... linking ... done.

Loading package path-pieces-0.1.3.1 ... linking ... done.

Loading package resource-pool-0.2.1.1 ... linking ... done.

Loading package silently-1.2.4.1 ... linking ... done.

Loading package persistent-1.3.0.6 ... linking ... done.

Loading package persistent-template-1.3.1.3 ... linking ... done.

Loading package zlib-0.5.4.1 ... linking ... done.

Loading package random-1.0.1.1 ... linking ... done.

Loading package process-1.1.0.2 ... linking ... done.

Loading package old-time-1.1.0.1 ... linking ... done.

Loading package byteorder-1.0.4 ... linking ... done.

Loading package case-insensitive-1.2.0.0 ... linking ... done.

Loading package http-types-0.8.4 ... linking ... done.

Loading package parsec-3.1.5 ... linking ... done.

Loading package network-2.4.2.2 ... linking ... done.

Loading package unix-time-0.2.2 ... linking ... done.

Loading package vault-0.3.0.3 ... linking ... done.

Loading package wai-2.0.0 ... linking ... done.

Loading package wai-logger-2.1.1 ... linking ... done.

Loading package system-filepath-0.4.10 ... linking ... done.

Loading package system-fileio-0.3.12 ... linking ... done.

Loading package shakespeare-1.2.1.1 ... linking ... done.

Loading package shakespeare-js-1.2.0.4 ... linking ... done.

Loading package data-default-class-0.0.1 ... linking ... done.

Loading package data-default-instances-base-0.0.1 ... linking ... done.

Loading package data-default-instances-containers-0.0.1 ... linking ... done.

Loading package data-default-instances-dlist-0.0.1 ... linking ... done.

Loading package data-default-instances-old-locale-0.0.1 ... linking ... done.

Loading package data-default-0.5.3 ... linking ... done.

Loading package cookie-0.4.0.1 ... linking ... done.

Loading package blaze-builder-conduit-1.0.0 ... linking ... done.

Loading package simple-sendfile-0.2.13 ... linking ... done.

Loading package network-conduit-1.0.4 ... linking ... done.

Loading package http-date-0.0.4 ... linking ... done.

Loading package attoparsec-conduit-1.0.1.2 ... linking ... done.

Loading package byteable-0.1.1 ... linking ... done.

Loading package securemem-0.1.3 ... linking ... done.

Loading package crypto-cipher-types-0.0.9 ... linking ... done.

Loading package cipher-aes-0.2.7 ... linking ... done.

Loading package crypto-random-0.0.7 ... linking ... done.

Loading package cprng-aes-0.5.2 ... linking ... done.

Loading package cereal-0.4.0.1 ... linking ... done.

Loading package entropy-0.2.2.4 ... linking ... done.

Loading package tagged-0.7.1 ... linking ... done.

Loading package crypto-api-0.13 ... linking ... done.

Loading package skein-1.0.9 ... linking ... done.

Loading package clientsession-0.9.0.3 ... linking ... done.

Loading package conduit-extra-1.0.0.1 ... linking ... done.

Loading package failure-0.2.0.2 ... linking ... done.

Loading package hamlet-1.1.9.2 ... linking ... done.

Loading package safe-0.3.4 ... linking ... done.

Loading package shakespeare-css-1.0.7.4 ... linking ... done.

Loading package shakespeare-i18n-1.0.0.5 ... linking ... done.

Loading package unix-compat-0.4.1.1 ... linking ... done.

Loading package ansi-terminal-0.6.1.1 ... linking ... done.

Loading package stringsearch-0.3.6.5 ... linking ... done.

Loading package word8-0.0.4 ... linking ... done.

Loading package zlib-bindings-0.1.1.5 ... linking ... done.

Loading package zlib-conduit-1.0.0 ... linking ... done.

Loading package wai-extra-2.0.3.3 ... linking ... done.

Loading package warp-2.0.3.4 ... linking ... done.

Loading package yesod-routes-1.2.0.6 ... linking ... done.

Loading package yesod-core-1.2.11.1 ... linking ... done.

Loading package esqueleto-1.3.9 ... linking ... done.

real 0m16.722s

user 0m16.093s

sys 0m0.575s

whereas with -O1 does seem to finish, but only after a minute or two

$ time ghc -O2 -XFlexibleContexts -XTemplateHaskell -XQuasiQuotes -XOverloadedStrings -XTypeFamilies -XDeriveDataTypeable -XGADTs -package-db **********/.cabal-sandbox/x86_64-linux-ghc-7.6.3-packages.conf.d Model.hs 

[3 of 3] Compiling Model            ( Model.hs, Model.o )

Loading package ghc-prim ... linking ... done.

Loading package integer-gmp ... linking ... done.

Loading package base ... linking ... done.

Loading package transformers-0.3.0.0 ... linking ... done.

Loading package mtl-2.1.2 ... linking ... done.

Loading package array-0.4.0.1 ... linking ... done.

Loading package stm-2.4.2 ... linking ... done.

Loading package primitive-0.5.0.1 ... linking ... done.

Loading package filepath-1.3.0.1 ... linking ... done.

Loading package deepseq-1.3.0.1 ... linking ... done.

Loading package old-locale-1.0.0.5 ... linking ... done.

Loading package time-1.4.0.1 ... linking ... done.

Loading package bytestring-0.10.0.2 ... linking ... done.

Loading package unix-2.6.0.1 ... linking ... done.

Loading package directory-1.2.0.1 ... linking ... done.

Loading package containers-0.5.0.0 ... linking ... done.

Loading package text-1.0.0.1 ... linking ... done.

Loading package blaze-builder-0.3.3.2 ... linking ... done.

Loading package blaze-markup-0.6.0.0 ... linking ... done.

Loading package blaze-html-0.7.0.1 ... linking ... done.

Loading package fast-logger-2.1.5 ... linking ... done.

Loading package hashable-1.2.1.0 ... linking ... done.

Loading package scientific-0.2.0.2 ... linking ... done.

Loading package attoparsec-0.11.2.1 ... linking ... done.

Loading package dlist-0.7.0.1 ... linking ... done.

Loading package syb-0.4.0 ... linking ... done.

Loading package pretty-1.1.1.0 ... linking ... done.

Loading package template-haskell ... linking ... done.

Loading package unordered-containers-0.2.3.3 ... linking ... done.

Loading package vector-0.10.9.1 ... linking ... done.

Loading package aeson-0.7.0.2 ... linking ... done.

Loading package base-unicode-symbols-0.2.2.4 ... linking ... done.

Loading package transformers-base-0.4.1 ... linking ... done.

Loading package monad-control-0.3.2.3 ... linking ... done.

Loading package lifted-base-0.2.2.1 ... linking ... done.

Loading package mmorph-1.0.2 ... linking ... done.

Loading package resourcet-0.4.10.2 ... linking ... done.

Loading package text-stream-decode-0.1.0.5 ... linking ... done.

Loading package nats-0.1.2 ... linking ... done.

Loading package semigroups-0.13.0.1 ... linking ... done.

Loading package void-0.6.1 ... linking ... done.

Loading package conduit-1.0.17.1 ... linking ... done.

Loading package exceptions-0.5 ... linking ... done.

Loading package monad-loops-0.4.2 ... linking ... done.

Loading package stm-chans-3.0.0.2 ... linking ... done.

Loading package monad-logger-0.3.5.1 ... linking ... done.

Loading package base64-bytestring-1.0.0.1 ... linking ... done.

Loading package path-pieces-0.1.3.1 ... linking ... done.

Loading package resource-pool-0.2.1.1 ... linking ... done.

Loading package silently-1.2.4.1 ... linking ... done.

Loading package persistent-1.3.0.6 ... linking ... done.

Loading package persistent-template-1.3.1.3 ... linking ... done.

Loading package zlib-0.5.4.1 ... linking ... done.

Loading package random-1.0.1.1 ... linking ... done.

Loading package process-1.1.0.2 ... linking ... done.

Loading package old-time-1.1.0.1 ... linking ... done.

Loading package byteorder-1.0.4 ... linking ... done.

Loading package case-insensitive-1.2.0.0 ... linking ... done.

Loading package http-types-0.8.4 ... linking ... done.

Loading package parsec-3.1.5 ... linking ... done.

Loading package network-2.4.2.2 ... linking ... done.

Loading package unix-time-0.2.2 ... linking ... done.

Loading package vault-0.3.0.3 ... linking ... done.

Loading package wai-2.0.0 ... linking ... done.

Loading package wai-logger-2.1.1 ... linking ... done.

Loading package system-filepath-0.4.10 ... linking ... done.

Loading package system-fileio-0.3.12 ... linking ... done.

Loading package shakespeare-1.2.1.1 ... linking ... done.

Loading package shakespeare-js-1.2.0.4 ... linking ... done.

Loading package data-default-class-0.0.1 ... linking ... done.

Loading package data-default-instances-base-0.0.1 ... linking ... done.

Loading package data-default-instances-containers-0.0.1 ... linking ... done.

Loading package data-default-instances-dlist-0.0.1 ... linking ... done.

Loading package data-default-instances-old-locale-0.0.1 ... linking ... done.

Loading package data-default-0.5.3 ... linking ... done.

Loading package cookie-0.4.0.1 ... linking ... done.

Loading package blaze-builder-conduit-1.0.0 ... linking ... done.

Loading package simple-sendfile-0.2.13 ... linking ... done.

Loading package network-conduit-1.0.4 ... linking ... done.

Loading package http-date-0.0.4 ... linking ... done.

Loading package attoparsec-conduit-1.0.1.2 ... linking ... done.

Loading package byteable-0.1.1 ... linking ... done.

Loading package securemem-0.1.3 ... linking ... done.

Loading package crypto-cipher-types-0.0.9 ... linking ... done.

Loading package cipher-aes-0.2.7 ... linking ... done.

Loading package crypto-random-0.0.7 ... linking ... done.

Loading package cprng-aes-0.5.2 ... linking ... done.

Loading package cereal-0.4.0.1 ... linking ... done.

Loading package entropy-0.2.2.4 ... linking ... done.

Loading package tagged-0.7.1 ... linking ... done.

Loading package crypto-api-0.13 ... linking ... done.

Loading package skein-1.0.9 ... linking ... done.

Loading package clientsession-0.9.0.3 ... linking ... done.

Loading package conduit-extra-1.0.0.1 ... linking ... done.

Loading package failure-0.2.0.2 ... linking ... done.

Loading package hamlet-1.1.9.2 ... linking ... done.

Loading package safe-0.3.4 ... linking ... done.

Loading package shakespeare-css-1.0.7.4 ... linking ... done.

Loading package shakespeare-i18n-1.0.0.5 ... linking ... done.

Loading package unix-compat-0.4.1.1 ... linking ... done.

Loading package ansi-terminal-0.6.1.1 ... linking ... done.

Loading package stringsearch-0.3.6.5 ... linking ... done.

Loading package word8-0.0.4 ... linking ... done.

Loading package zlib-bindings-0.1.1.5 ... linking ... done.

Loading package zlib-conduit-1.0.0 ... linking ... done.

Loading package wai-extra-2.0.3.3 ... linking ... done.

Loading package warp-2.0.3.4 ... linking ... done.

Loading package yesod-routes-1.2.0.6 ... linking ... done.

Loading package yesod-core-1.2.11.1 ... linking ... done.

Loading package esqueleto-1.3.9 ... linking ... done.

real 1m37.716s

user 1m36.243s

sys 0m1.235s

With -O2, ghc gets killed after about 2 minutes

$ time ghc -O2 -XFlexibleContexts -XTemplateHaskell -XQuasiQuotes -XOverloadedStrings -XTypeFamilies -XDeriveDataTypeable -XGADTs -package-db ************************/.cabal-sandbox/x86_64-linux-ghc-7.6.3-packages.conf.d Model.hs

[3 of 3] Compiling Model            ( Model.hs, Model.o )

Loading package ghc-prim ... linking ... done.

Loading package integer-gmp ... linking ... done.

Loading package base ... linking ... done.

Loading package transformers-0.3.0.0 ... linking ... done.

Loading package mtl-2.1.2 ... linking ... done.

Loading package array-0.4.0.1 ... linking ... done.

Loading package stm-2.4.2 ... linking ... done.

Loading package primitive-0.5.0.1 ... linking ... done.

Loading package filepath-1.3.0.1 ... linking ... done.

Loading package deepseq-1.3.0.1 ... linking ... done.

Loading package old-locale-1.0.0.5 ... linking ... done.

Loading package time-1.4.0.1 ... linking ... done.

Loading package bytestring-0.10.0.2 ... linking ... done.

Loading package unix-2.6.0.1 ... linking ... done.

Loading package directory-1.2.0.1 ... linking ... done.

Loading package containers-0.5.0.0 ... linking ... done.

Loading package text-1.0.0.1 ... linking ... done.

Loading package blaze-builder-0.3.3.2 ... linking ... done.

Loading package blaze-markup-0.6.0.0 ... linking ... done.

Loading package blaze-html-0.7.0.1 ... linking ... done.

Loading package fast-logger-2.1.5 ... linking ... done.

Loading package hashable-1.2.1.0 ... linking ... done.

Loading package scientific-0.2.0.2 ... linking ... done.

Loading package attoparsec-0.11.2.1 ... linking ... done.

Loading package dlist-0.7.0.1 ... linking ... done.

Loading package syb-0.4.0 ... linking ... done.

Loading package pretty-1.1.1.0 ... linking ... done.

Loading package template-haskell ... linking ... done.

Loading package unordered-containers-0.2.3.3 ... linking ... done.

Loading package vector-0.10.9.1 ... linking ... done.

Loading package aeson-0.7.0.2 ... linking ... done.

Loading package base-unicode-symbols-0.2.2.4 ... linking ... done.

Loading package transformers-base-0.4.1 ... linking ... done.

Loading package monad-control-0.3.2.3 ... linking ... done.

Loading package lifted-base-0.2.2.1 ... linking ... done.

Loading package mmorph-1.0.2 ... linking ... done.

Loading package resourcet-0.4.10.2 ... linking ... done.

Loading package text-stream-decode-0.1.0.5 ... linking ... done.

Loading package nats-0.1.2 ... linking ... done.

Loading package semigroups-0.13.0.1 ... linking ... done.

Loading package void-0.6.1 ... linking ... done.

Loading package conduit-1.0.17.1 ... linking ... done.

Loading package exceptions-0.5 ... linking ... done.

Loading package monad-loops-0.4.2 ... linking ... done.

Loading package stm-chans-3.0.0.2 ... linking ... done.

Loading package monad-logger-0.3.5.1 ... linking ... done.

Loading package base64-bytestring-1.0.0.1 ... linking ... done.

Loading package path-pieces-0.1.3.1 ... linking ... done.

Loading package resource-pool-0.2.1.1 ... linking ... done.

Loading package silently-1.2.4.1 ... linking ... done.

Loading package persistent-1.3.0.6 ... linking ... done.

Loading package persistent-template-1.3.1.3 ... linking ... done.

Loading package zlib-0.5.4.1 ... linking ... done.

Loading package random-1.0.1.1 ... linking ... done.

Loading package process-1.1.0.2 ... linking ... done.

Loading package old-time-1.1.0.1 ... linking ... done.

Loading package byteorder-1.0.4 ... linking ... done.

Loading package case-insensitive-1.2.0.0 ... linking ... done.

Loading package http-types-0.8.4 ... linking ... done.

Loading package parsec-3.1.5 ... linking ... done.

Loading package network-2.4.2.2 ... linking ... done.

Loading package unix-time-0.2.2 ... linking ... done.

Loading package vault-0.3.0.3 ... linking ... done.

Loading package wai-2.0.0 ... linking ... done.

Loading package wai-logger-2.1.1 ... linking ... done.

Loading package system-filepath-0.4.10 ... linking ... done.

Loading package system-fileio-0.3.12 ... linking ... done.

Loading package shakespeare-1.2.1.1 ... linking ... done.

Loading package shakespeare-js-1.2.0.4 ... linking ... done.

Loading package data-default-class-0.0.1 ... linking ... done.

Loading package data-default-instances-base-0.0.1 ... linking ... done.

Loading package data-default-instances-containers-0.0.1 ... linking ... done.

Loading package data-default-instances-dlist-0.0.1 ... linking ... done.

Loading package data-default-instances-old-locale-0.0.1 ... linking ... done.

Loading package data-default-0.5.3 ... linking ... done.

Loading package cookie-0.4.0.1 ... linking ... done.

Loading package blaze-builder-conduit-1.0.0 ... linking ... done.

Loading package simple-sendfile-0.2.13 ... linking ... done.

Loading package network-conduit-1.0.4 ... linking ... done.

Loading package http-date-0.0.4 ... linking ... done.

Loading package attoparsec-conduit-1.0.1.2 ... linking ... done.

Loading package byteable-0.1.1 ... linking ... done.

Loading package securemem-0.1.3 ... linking ... done.

Loading package crypto-cipher-types-0.0.9 ... linking ... done.

Loading package cipher-aes-0.2.7 ... linking ... done.

Loading package crypto-random-0.0.7 ... linking ... done.

Loading package cprng-aes-0.5.2 ... linking ... done.

Loading package cereal-0.4.0.1 ... linking ... done.

Loading package entropy-0.2.2.4 ... linking ... done.

Loading package tagged-0.7.1 ... linking ... done.

Loading package crypto-api-0.13 ... linking ... done.

Loading package skein-1.0.9 ... linking ... done.

Loading package clientsession-0.9.0.3 ... linking ... done.

Loading package conduit-extra-1.0.0.1 ... linking ... done.

Loading package failure-0.2.0.2 ... linking ... done.

Loading package hamlet-1.1.9.2 ... linking ... done.

Loading package safe-0.3.4 ... linking ... done.

Loading package shakespeare-css-1.0.7.4 ... linking ... done.

Loading package shakespeare-i18n-1.0.0.5 ... linking ... done.

Loading package unix-compat-0.4.1.1 ... linking ... done.

Loading package ansi-terminal-0.6.1.1 ... linking ... done.

Loading package stringsearch-0.3.6.5 ... linking ... done.

Loading package word8-0.0.4 ... linking ... done.

Loading package zlib-bindings-0.1.1.5 ... linking ... done.

Loading package zlib-conduit-1.0.0 ... linking ... done.

Loading package wai-extra-2.0.3.3 ... linking ... done.

Loading package warp-2.0.3.4 ... linking ... done.

Loading package yesod-routes-1.2.0.6 ... linking ... done.

Loading package yesod-core-1.2.11.1 ... linking ... done.

Loading package esqueleto-1.3.9 ... linking ... done.

Killed

real 2m7.241s

user 1m51.542s

sys 0m3.348s

We're working on a commercial product, so private source code - but I can share my Model.hs with either of you or jump on skype/hangout if it would help!

[Felipe Lessa]

Are you compiling with -O2? You should be using just -O, which is
cabal's default.

--
Felipe.

[Rehno Lindeque]

Are you compiling with -O2?  You should be using just -O, which is 
cabal's default. 

 

Most of the time I just do a standard cabal build, nothing special. I think that it only occasionally crashes with -O, sometimes it succeeds.

Can you try compiling with -O0 (possibly just in that module via {-# OPTIONS_GHC -O0 #-} and see what happens?

I think that {-# OPTIONS_GHC -O0 #-} does help. I read my numbers incorrectly before, even with -O0 ghc does chew through quite a lot of memory (over 1GB) - surely there must be a space leak somewhere?

PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND 

28073 rehno     20   0 1364m 1.1g 3496 R  95.7 29.7   1:32.37 ghc

[Michael Snoyman]

On Sun, Apr 13, 2014 at 7:55 PM, Rehno Lindeque <rehno.l...@gmail.com> wrote:

Are you compiling with -O2?  You should be using just -O, which is 
cabal's default. 

 

Most of the time I just do a standard cabal build, nothing special. I think that it only occasionally crashes with -O, sometimes it succeeds.

Can you try compiling with -O0 (possibly just in that module via {-# OPTIONS_GHC -O0 #-} and see what happens?

I think that {-# OPTIONS_GHC -O0 #-} does help. I read my numbers incorrectly before, even with -O0 ghc does chew through quite a lot of memory (over 1GB) - surely there must be a space leak somewhere?

PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND 

28073 rehno     20   0 1364m 1.1g 3496 R  95.7 29.7   1:32.37 ghc

I'm fairly certain the space leak isn't in our TH code, as if my testing is accurate, the TH code completes successfully before the huge memory usage kicks in. But I still have more testing to get through to be certain.

Michael 

[Rehno Lindeque]

I'm fairly certain the space leak isn't in our TH code, as if my testing is accurate, the TH code completes successfully before the huge memory usage kicks in. But I still have more testing to get through to be certain.

Michael 

Thanks a lot, I appreciate it. Let me know if I can do anything to help...

[Michael Snoyman]

FWIW, I have a module right now that has under 1000 lines of code, uses no CPP, TemplateHaskell, or DeriveGeneric, and with -O2 turned on, takes 47 seconds to compile. I'm whittling down that module one bit at a time to see what might be causing it to be that slow. No idea if it's connected to your problem though...

Michael

[Michael Snoyman]

OK, in my case, it seems that the `deriving` clauses we had on our data types were the culprit; removing them brought compile time down from 1:38 to 0:02. (Obviously, I can't do that in reality, but it was a good sanity check.) Do you by any chance have a large number of deriving clauses in your code?

Michael 

[Rehno Lindeque]

I responded privately in order to share the code with you.

[Michael Snoyman]

Unfortunately, I can't find any particular cause here. It still looks like it's *not* the TH code to me, but instead just death by a thousand papercuts. I think the best option for now is to add {-# OPTIONS_GHC -O1 #-} or something similar to your file. I'm similarly stumped on a way to improve compile times in the code I was looking at.

--

[Rehno Lindeque]

Oh thanks Michael, sorry I hadn't realized that you'd replied here - for some reason I missed it. For now I've just commented out most of my entities since I'm not directly working with all of them. I'll also try and keep an eye out for anything I can see. My machine is doesn't have a lot of memory so quite frequently it's something to do with that. I might also try to split up the model into multiple files when I start getting to that point again. Thanks for taking a look.

[jsch...@gmail.com]

Wow, thank you for this thread!!! Damn, I use ghci but with :l DevelMain instead of :r DevelMain for months!

:l always recompiles the model which is very slow and also recompiles many other modules that seem unrelated to my changes. And I still thought ghci + :l is a wonderful improvement compared to GHC. But ghci + :r is amazing!

Thank you, Michael for yesod and persist!