Overly aggressive constant unloading of Rails 3.0

[Mislav Marohnić]

In Rails 3.0 (both RC and edge), any constant that's defined while loading a file in development mode will get unloaded on each request, regardless of whether it comes from an autoload path in the application or from external code such as a gem.

Example file (tested on a fresh edge app):

  # kittens.rb

  require 'nibbler/json'  # "nibbler" is a gem specified in Gemfile

  module Kittens

    NibblerJSON

  end

This file can be either in "app/models/" or in "lib/". Regardless of whether it was autoloaded or required, Rails will unload both Kittens and NibblerJSON modules regardless of the fact the latter comes from an external source (the "nibbler" gem). Additionally, if "kittens.rb" was required from "lib/" directory, Rails will never reload it, which might be a bug by itself.

There's been a confusing amount of changes and reverts to autoloading/eager loading nature of the "lib" directory in Rails 3.0. I don't know where we stand anymore, so I'm asking here in hope to get some insight.

[José Valim]

The fact lib is no longer autoloaded is not the issue here, it just
exposed the real issue since you need to require files manually.

In fact, you could reproduce this bug by even requiring open-uri:

require "open-uri"

class User < ActiveRecord::Base
OpenURI
end

Nonetheless, I believe Yehuda fixed it in recent commits. :)

About lib being autoloaded or not, it won't be autoloaded because
autoloading is not thread safe. The only way to make autoloading lib
threadsafe is if we eagerload *everything* inside lib when the server
starts, but this can easily lead to confusion and pain since people
are used to put generators and code specific to environments inside
lib, causing production to load development or test only specific
code.

That said, in Rails 3 you will need to require code in lib manually.
Gems in Rails 2.3 were also allowed to autoload all the code in lib
and we completely removed that in 3.0. We pretend to remove
autoloading code from lib in plugins in 3.1 as well. We will only
autoload code in app/.

On Aug 17, 8:13 am, Mislav Marohnić <mislav.maroh...@gmail.com> wrote:
> In Rails 3.0 (both RC and edge), any constant that's defined while loading a
> file in development mode will get unloaded on each request, regardless of
> whether it comes from an autoload path in the application or from external
> code such as a gem.
>
> Example file (tested on a fresh edge app):
>
>   # kittens.rb
>   require 'nibbler/json'  # "nibbler" is a gem specified in Gemfile
>
>   module Kittens
>     NibblerJSON
>   end
>
> This file can be either in "app/models/" or in "lib/". Regardless of whether

> it was autoloaded or required, *Rails will unload both Kittens and
> NibblerJSON modules* regardless of the fact the latter comes from an

> external source (the "nibbler" gem). Additionally, if "kittens.rb" was
> required from "lib/" directory, Rails will never reload it, which might be a
> bug by itself.
>
> There's been a confusing amount of changes and

> reverts<http://github.com/rails/rails/commits/7e2399a42feb47407ad0/railties/l...>to

[Mateo Murphy]

On 20-Aug-10, at 2:25 PM, José Valim wrote:

> About lib being autoloaded or not, it won't be autoloaded because
> autoloading is not thread safe. The only way to make autoloading lib
> threadsafe is if we eagerload *everything* inside lib when the server
> starts, but this can easily lead to confusion and pain since people
> are used to put generators and code specific to environments inside
> lib, causing production to load development or test only specific
> code.
>
> That said, in Rails 3 you will need to require code in lib manually.
> Gems in Rails 2.3 were also allowed to autoload all the code in lib
> and we completely removed that in 3.0. We pretend to remove
> autoloading code from lib in plugins in 3.1 as well. We will only
> autoload code in app/.

Out of curiosity, why is loading the contents of app threadsafe but
not lib? Is it simply a question of the kinds of stuff people tend to
put in lib?

[Christian Seiler]

it's not for thread-safety reasons:
https://rails.lighthouseapp.com/projects/8994/tickets/5218-rails-3-rc-does-not-autoload-from-lib

and my two cents:
I think there really should be a standard folder in any Rails app
(convention over configuration anyone?) which is autoloaded (and
reloaded in dev mode) where I can put shared or abstracted code.
Maybe app/lib (if lib causes problems)?

[Evgeniy Dolzhenko]

> I think there really should be a standard folder in any Rails app
> (convention over configuration anyone?) which is autoloaded (and
> reloaded in dev mode) where I can put shared or abstracted code.
> Maybe app/lib (if lib causes problems)?

Isn't it like it should be working already?

http://github.com/rails/rails/commit/781d0a9baef77a1d749bc8d7ea1b8e550a13c34a#L2R85

[Mislav Marohnić]

Jose: I can't see a fix in 3-0-stable. I'll test on master. Is there a lighthouse ticket for this? Is it definitely fixed?

On Sun, Aug 22, 2010 at 17:36, Mateo Murphy <mateo....@gmail.com> wrote:

Out of curiosity, why is loading the contents of app threadsafe but not lib? Is it simply a question of the kinds of stuff people tend to put in lib?

Mateo: If "lib" is in autoload paths, the complete contents of the lib directory would have to be eager-loaded in production mode in order to achieve thread-safety. Rails core team seem to have decided to not eager-load stuff inside lib, so the only option is to disable autoload in lib.

The math is simple: if you want autoload in development, you have to go with eager-load in production. If you don't want eager-load, you can't use autoload. There is no alternative.

[Xavier Noria]

On Sun, Aug 22, 2010 at 5:36 PM, Mateo Murphy <mateo....@gmail.com> wrote:

> Out of curiosity, why is loading the contents of app threadsafe but not lib?
> Is it simply a question of the kinds of stuff people tend to put in lib?

It is assumed that it is fine to preload everything in app/models etc
because it is happening in 2.3 in production mode. It is even done in
a multiprocess setup so that eg passenger + ree can perform better due
to COW.

But as José mentioned you throw generators in lib and other assorted
stuff Rails just can't blindly execute. It has been long pondered, but
it seems it is the only sensible solution albeit it becomes an
important item in your migration checklist.

BTW, config.autoload_paths means just that, it does not trigger eager
loading as of this writing. So a temporary (?!) easy way to mark that
item done in a multiprocess setup is to add lib to
config.autoload_paths.