Heroku & Boot Times

[Scott Parker]

Anyone else running a production website with Clojure in Heroku and
struggling with boot time problems? After digging through our logs
from the past month, I've noticed it's not uncommon to have a dyno
crashed for awhile because of boot time problems. It seems especially
likely when dynos are cycling once/day - I'm guessing because of
additional delays in picking up latest snapshot dependencies.

Has anyone else run into this problem and has a bright idea? I have
already verified we're using lein compile :all at deploy (by virtue of
being on lein2), running in the production profile, and I am working
on removing those snapshot dependencies. I've checked the app for
obvious bottlenecks like web/DB/IO requests during initialization with
no luck. We don't have a ton of code or dependencies right now, so I'm
a bit skeptical that we can remain on Heroku as we grow. As a last
resort, I suppose we could try a proxy bound to a Unix socket as in
https://github.com/dblock/heroku-forward but I'd rather avoid that if
possible.

If not advice specifically in the context of pleasing Heroku, advice
on troubleshooting slow app init times generally would also be
welcome. I've done some minimal code benchmarking in Clojure
previously, but never specifically towards resolving time-to-init.

Thanks,
-SP

[Jeroen van Dijk]

Hi Scott,

We had some issues as well. SNAPSHOTS are likely to be an issue because they are re-checked at least once a day. So if your app needs a restart this will be re-checked and might slow down boot time. We also had problems due to this in combination with failing maven mirrors. It is probably best to just not use SNAPSHOT versions in production.

Another thing that can have impact on boot time is the reading of static files on booting. We solved this by using '(def resource (delay (expensive-task ))) and @resource to post-pone this execution for after booting.

We also use this feature https://devcenter.heroku.com/articles/labs-preboot/ to minimize possible downtime between deploys (not sure if this works for restarts as well)

On top of that, we always have a minimum of two dynos for redundancy if one is down.

HTH,

Jeroen

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

[Scott Parker]

Jeroen - thanks for the advice. Yeah, SNAPSHOTS in prod is a poor practice anyway, this gives me a good incentive to find them and kill them.

Static files on boot... dang. When I was first investigating our slow boot time I swear I checked Enlive, but another quick glance at the source indicates it's almost certainly a contributor. Expletive! I will start capturing some metrics on how much time we're spending loading HTML templates at init. We're not loading a godawful number of separate HTML files, so I'd like to avoid a bunch of delays throughout my views if possible. 

Thanks a bunch Jeroen.

-SP

[Weber, Martin S]

Obviously it helps to make sure the dependencies you are using are named with the exact snapshot version.

The biggest time-saver for me though is convincing lein to not do the dependency dance all the time. I'm surprised though to see that you are dependency checking at all though. Shouldn't you just create a jar and then call directly into that?

Fun observation about reducing lein startup times though:

So I add my deps to the project.clj; use a repo path that is inside the project (so it's easier to DCVS it, key :local-repo ); lein deps; run my test cases (lein deps sadly does not pull all potentially used dependencies, gotta exercise it some); then add ":offline? true" to the project.clj.

Difference on my old laptop is lein repl: ~20 secs to start with :offline? True; without it, it takes 45-50 seconds instead. That's a rough x 2 on a  dual core ht 3 Ghz w/ 3G RAM (it's still taking about factor 200 too long, but hey! Halved it already! On my work laptop though, the difference is 2 or 4 seconds. Not that big of a deal).

Regards,

-Martin

[Sean Corfield]

On Wed, Jan 23, 2013 at 9:42 AM, Scott Parker <scott.p...@gmail.com> wrote:
> Static files on boot... dang. When I was first investigating our slow boot
> time I swear I checked Enlive, but another quick glance at the source
> indicates it's almost certainly a contributor. Expletive! I will start
> capturing some metrics on how much time we're spending loading HTML
> templates at init. We're not loading a godawful number of separate HTML
> files, so I'd like to avoid a bunch of delays throughout my views if
> possible.

FWIW, what I do with Enlive is to cache templates in an atom (as a map
from names to html-resource call results) so we only load each
template once - unless we force the app to reload.
--
Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
World Singles, LLC. -- http://worldsingles.com/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

[Phil Hagelberg]

Scott Parker writes:

> Jeroen - thanks for the advice. Yeah, SNAPSHOTS in prod is a poor practice
> anyway, this gives me a good incentive to find them and kill them.

Yeah, doing any dependency resolution at process launch time is brutal
in production; apart from boot time issues it introduces variability and
counteracts the benefits of the explicit slug compilation process. You
can usually avoid this by just freezing your snapshots, but sometimes
problems can still arise transitively, so I highly recommend putting
`:offline true` into your production profile. This will prevent them
from being checked during boot even if you still have snapshots in
project.clj. I'll try to make sure this is clearer in devcenter docs.

Happy to help debug further if there are boot issues beyond simply
dependency resolution at launch; just let me know.

-Phil