Rake tasks in production, application models, and Rails 2.3.8-3.0

[Mislav Marohnić]

Consider this simple rake task:

  task :heavy_process => :environment do

    MyModel.process

  end

The problem I'm seeing here is that this will work in development mode, in which autoloading kicks in when the "MyModel" constant is referenced for the first time, but will fail in production mode because of the behavior described below. This issue has been bugging me for a while. I can't recall it being a problem with Rails 2.3.5. Not sure if this is expected behavior.

If someone is writing a rake task that needs the application to be loaded, he or she will make it depend on the ":environment" task which will boot the app in the respective environment. It will also set the `$rails_rake_task` global to true. This has been around for a while.

However, since the 2.3.8 and 3.0 releases, this global prevents preloading of application classes in production (or any environment in which "cache_classes" setting is enabled). This is probably so the app boots faster; users are expected to require what they need in the rake task. Here is this logic in Rails 3:

      initializer :eager_load! do

        if config.cache_classes && !$rails_rake_task

          ActiveSupport.run_load_hooks(:before_eager_load, self)

          eager_load!

        end

      end

Going back to our initial rake task, our initial reaction is that we should put `require my_model` before trying to reference MyModel. However, if MyModel is linked to other models via associations, we also need to require all of the associated models in case the `MyModel.process` method uses these associations. In the end, wherever we put the require (or `require_dependency`) statements, it just feels wrong and clumsy.

We've also been seeing migrations fail in production mode because they reference a model to do some processing. Again, the biggest problem here is that all this works like a breeze in development, making us developers feel relaxed and secure, but then surprises us when we push to production.

It occurred to me that an easy solution might be to turn off "cache_classes" setting in production if $rails_rake_task is true. Comments on that?

Thanks

[Michael Koziarski]

> . In the end, wherever we put the require (or
> `require_dependency`) statements, it just feels wrong and clumsy.
> We've also been seeing migrations fail in production mode because they
> reference a model to do some processing. Again, the biggest problem here is
> that all this works like a breeze in development, making us developers feel
> relaxed and secure, but then surprises us when we push to production.
> It occurred to me that an easy solution might be to turn off "cache_classes"
> setting in production if $rails_rake_task is true. Comments on that?

eager loading is a relatively new piece of functionality, and
disabling it for rake tasks should *ideally* simply fall back on the
const_missing hook. I'm a little confused why it's not doing that.
Do you have config.threadsafe! set or something?

Either way, unless you have config.threadsafe! set, this is *clearly*
a bug. And if you do, we should probably force the eager loading to
take place irrespective of whether it's a rake task.

> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Ruby on Rails: Core" group.
> To post to this group, send email to rubyonra...@googlegroups.com.
> To unsubscribe from this group, send email to
> rubyonrails-co...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/rubyonrails-core?hl=en.
>

--
Cheers

Koz

[Mislav Marohnić]

On Mon, Jul 12, 2010 at 02:15, Michael Koziarski <mic...@koziarski.com> wrote:

Either way, unless you have config.threadsafe! set, this is *clearly*
a bug.  And if you do, we should probably force the eager loading to
take place irrespective of whether it's a rake task.

You're right, we had threadsafe on. I still think it's a bug, because rake tasks are not threaded web server runtime and therefore shouldn't guard against race conditions and such.

For now I'll just turn off threadsafe (I guess it's not needed for Passenger?). Thanks Koz

[Yehuda Katz]

Passenger should be able to make use of threadsafe mode. Either way, the problem here seems to be that Rake tasks have a kludge in this one area of the system, but not in others. We either need to decide that Rake tasks are never considered threadsafe (even if the threadsafe! flag is on), or eager load for Rake tasks.

I'm surprised nobody else hit this before.

Yehuda Katz
Architect | Engine Yard
(ph) 718.877.1325

--

[Rodrigo Rosenfeld Rosas]

Actually, this is happening in my project since the beggining. I just didn't have enough time to debug this behavior. I required the necessary files from inside the task to solve my needs...

Rodrigo.

[Michael Koziarski]

> You're right, we had threadsafe on. I still think it's a bug, because rake
> tasks are not threaded web server runtime and therefore shouldn't guard
> against race conditions and such.
> For now I'll just turn off threadsafe (I guess it's not needed for
> Passenger?). Thanks Koz

config.threadsafe! just does:

# Enable threaded mode. Allows concurrent requests to controller actions and
# multiple database connections. Also disables automatic dependency loading
# after boot, and disables reloading code on every request, as these are
# fundamentally incompatible with thread safety.
def threadsafe!
self.preload_frameworks = true
self.cache_classes = true
self.dependency_loading = false
self.action_controller.allow_concurrency = true
self
end

It's the dependency_loading = false that's messing you up. The
initializer should probably be checking that, not cache_classes.

[Mislav Marohnić]

On Wed, Jul 14, 2010 at 05:48, Michael Koziarski <mic...@koziarski.com> wrote:

     self.dependency_loading = false

How about changing that line to:

  self.dependency_loading = defined?($rails_rake_task) && !!$rails_rake_task

Good idea?

[Michael Koziarski]

> How about changing that line to:
>   self.dependency_loading = defined?($rails_rake_task) && !!$rails_rake_task
> Good idea?

That has a few potential side effects, but how does this look for the
2-3-stable change:

http://gist.github.com/476526

--
Cheers

Koz

[Mislav Marohnić]

On Thu, Jul 15, 2010 at 07:03, Michael Koziarski <mic...@koziarski.com> wrote:

That has a few potential side effects, but how does this look for the
2-3-stable change: http://gist.github.com/476526

Looks good. Why just 2.3-stable? 

[Michael Koziarski]

> Looks good. Why just 2.3-stable?

2.3 needs a 'smallest possible bugfix' change, master can 'do the
right thing' which probably *isn't* adding another conditional.

--
Cheers

Koz