#8087 Vendor Everything

[Josh Peek]

Just trying to raise some awareness and discussion on a patch I
recently posted.

The idea of freezing gems in your rails app has been around for a long
time. However there is no suggested convention to follow. Chris
Wanstrath has suggested the directory "vendor/gems" to be use in his
post "Vendor Everything" (http://errtheblog.com/post/2120).

Once you set your path to this directory, everything works great.
There is no code difference between loading gems from your local
machine with rubygems or from the "vendor/gems" directory.

If everyone is down with this convention, we should add a few things
in rails to encourage it.

1. Make the "vendor/gems" directory when you run the rails command.
2. Add "vendor/gems" to the load path
3. Load tasks from "vendor/gems" (I don't really use this but Nic
Williams finds it useful)
4. Add some rake tasks to help you freeze gems.

Are there any other techniques out there that work well for you? Or
does this sound like a delicious patch?

[Josh Peek]

Link to the patch: http://dev.rubyonrails.org/ticket/8087

[Parker Thompson]

I loved Chris' post, I love this patch. +1. Here's to never
deploying breaking production you forgot to install a gem or having
to ssh to a bunch of machines to 'sudo gem install' (decidedly not
DRY) ever again!

pt.

[Ben Munat]

What about the native code issue? Are you going to create a
sub-directory per platform?

Also, the idea of specifying dependencies (ala Maven) in yaml sounds
pretty cool. Were you thinking about doing something like that?

Ben

[Marcel Molina Jr.]

On Thu, Apr 19, 2007 at 07:39:03PM -1000, Ben Munat wrote:
> Also, the idea of specifying dependencies (ala Maven) in yaml sounds
> pretty cool. Were you thinking about doing something like that?

Gems already have a mechanism for specifying dependencies.

Several weeks ago when I checked in the changes to the plugin loading code
Chad and I worked on functionality similar to the proposed patch. We took it
a step further with a slightly different approach whereby gem files placed
in a certain location would automaticaly be unpacked and loaded like regular
plugins on startup. The code is complete (as we intended it), tested and
documented, but despite Chad's repeated insistance, I haven't checked it in
yet...I've attached a diff for those who are interested.

marcel
--
Marcel Molina Jr. <mar...@vernix.org>

[Ben Munat]

I meant the app's dependencies... which gems an app needs to have
installed. Seems like there would need to be some outside help (new code
and a config file) to do that.

b

> ------------------------------------------------------------------------
>
> Index: test/plugin_locator_test.rb
> ===================================================================
> --- test/plugin_locator_test.rb (revision 6292)
> +++ test/plugin_locator_test.rb (working copy)
> @@ -1,4 +1,6 @@
> require File.dirname(__FILE__) + '/plugin_test_helper'
> +require 'fileutils'
> +require 'rubygems/builder'
>
> class TestPluginFileSystemLocator < Test::Unit::TestCase
> def setup
> @@ -37,5 +39,180 @@
> private
> def new_locator(initializer = @initializer)
> Rails::Plugin::FileSystemLocator.new(initializer)
> - end
> + end
> +
> +end
> +
> +class TestPluginGemLocator < Test::Unit::TestCase
> + def setup
> + configuration = Rails::Configuration.new
> + configuration.gem_home = gem_home_root_path
> + @initializer = Rails::Initializer.new(configuration)
> + @locator = new_gem_locator
> + end
> +
> + def teardown
> + purge_gems
> + end
> +
> + def test_gem_paths_are_set_to_the_appropriate_gem_home_for_the_app
> + application_gem_home_registered_with_rubygems = lambda do
> + !Gem.path.grep(/#{@initializer.configuration.gem_home}/).empty?
> + end
> + @locator.send(:setup_gem_environment)
> + assert application_gem_home_registered_with_rubygems.call
> + end
> +
> + def test_gems_are_not_installed_unecessarily_if_they_are_already_installed
> + gem_home_mtime = lambda { File.mtime(@initializer.configuration.gem_home) }
> + using_scenario 'gem_with_dep_that_has_its_own_deps' do
> + gem_home_mtime_before_installing = gem_home_mtime.call
> + @locator.send(:prepare_gems)
> + after_installing = gem_home_mtime.call
> + assert_not_equal gem_home_mtime_before_installing, after_installing
> + new_gem_locator.send(:prepare_gems)
> + assert_equal after_installing, gem_home_mtime.call
> + end
> + end
> +
> + def test_that_all_gems_in_the_bundled_gem_directory_are_installed_prior_to_loading
> + bundled_gems_are_installed = lambda do
> + !@locator.installer.installed_gems.empty?
> + end
> +
> + using_scenario 'gem_with_no_deps' do
> + assert !bundled_gems_are_installed.call
> + @locator.send(:prepare_gems)
> + assert bundled_gems_are_installed.call
> + end
> + end
> +
> + %w[gem_with_one_level_of_deps gem_with_dep_that_has_its_own_deps].each do |scenario|
> + define_method("test_unpacking_order_for_the_#{scenario}_scenario") do
> + using_scenario scenario do
> + @locator.send(:setup_gem_environment)
> + assert_equal expected_gem_loading_order, @locator.installer.send(:gems_to_install).map(&:full_name)
> + end
> + end
> + end
> +
> + %w[gem_with_no_deps gem_with_one_level_of_deps gem_with_dep_that_has_its_own_deps].each do |scenario|
> + define_method("test_the_appropriate_gem_plugins_are_located_for_the_#{scenario}_scenario") do
> + using_scenario scenario do
> + assert_equal expected_gem_loading_order, @locator.plugin_names
> + end
> + end
> + end
> +
> + def test_declaring_an_explicit_plugin_load_order_that_contradicts_the_gem_dependency_loading_order_raises_a_load_error
> + using_scenario 'gem_with_dep_that_has_its_own_deps' do
> + only_load_the_following_plugins! expected_gem_loading_order.reverse
> + assert_raises(LoadError) do
> + @initializer.load_plugins
> + end
> + end
> + end
> +
> + def test_declaring_an_explicit_plugin_load_order_that_jives_with_the_gem_depency_loading_order_works
> + using_scenario 'gem_with_dep_that_has_its_own_deps' do
> + only_load_the_following_plugins! expected_gem_loading_order
> + assert_nothing_raised do
> + @initializer.load_plugins
> + end
> + end
> + end
> +
> + def test_trying_to_load_a_gem_with_a_missing_depedency_raises_an_install_error
> + using_scenario 'gem_with_a_missing_dep' do
> + assert_raises(Gem::InstallError) do
> + @locator.send(:prepare_gems)
> + end
> + end
> + end
> +
> + def test_trying_load_a_gem_whose_dependency_constraint_can_not_be_satisfied_raises
> + using_scenario 'gem_with_version_constraint_on_dep_that_can_not_be_satisfied' do
> + assert_raises(Gem::InstallError) do
> + @locator.send(:prepare_gems)
> + end
> + end
> + end
> +
> + def test_unneeded_gems_are_uninstalled
> + flunk 'Need a test for this'
> + end
> +
> + private
> + def new_gem_locator(initializer = @initializer)
> + Rails::Plugin::GemLocator.new(initializer)
> + end
> +
> + def using_scenario(scenario)
> + set_bundled_gem_path! scenario
> + generate_gems
> + yield
> + ensure
> + purge_gems
> + end
> +
> + def set_bundled_gem_path!(bundled_gem_path)
> + @initializer.configuration.bundled_gem_path = File.join(plugin_gem_fixture_root_path, bundled_gem_path)
> + end
> +
> + def expected_gem_loading_order
> + @expected_gem_loading_order ||= YAML.load_file(File.join(bundled_gem_path, 'dependency_loading_order.yml'))
> + end
> +
> + def bundled_gem_path
> + @initializer.configuration.bundled_gem_path
> + end
> +
> + def generate_gems
> + Dir["#{bundled_gem_path}/**/*.gemspec"].each do |spec_file|
> + # Initialize outside the block so it is in scope
> + gem_path, specification = nil
> + execute_from_within(File.dirname(spec_file)) do
> + specification = Gem::Specification.load(File.basename(spec_file))
> + # Usually the Gem::Builder prints out its progress. We want to silent that.
> + Gem::DefaultUserInteraction.use_ui(Gem::SilentUI.new) do
> + Gem::Builder.new(specification).build
> + end
> + end
> + FileUtils.mv File.join(File.dirname(spec_file), "#{specification.full_name}.gem"), bundled_gem_path
> + end
> + end
> +
> + # This is a hack. Gem::Specification.load eval's the passed in spec file. Usually,
> + # when this is done with the gem command, it is executed from the same directory as
> + # where the gemspec is located. If this is not the case though, the eval is done relative to
> + # the calling code's directory. To work around this, while loading the gem spec, we move to
> + # the gemspec's directory.
> + def execute_from_within(path)
> + originating_path = Dir.pwd
> + Dir.chdir path
> + yield
> + ensure
> + Dir.chdir originating_path
> + end
> +
> + def purge_gems
> + FileUtils.rm_rf(generated_gems)
> + FileUtils.rm_rf(installed_gems)
> + end
> +
> + def generated_gems
> + Dir["#{bundled_gem_path}/*.gem"]
> + end
> +
> + def installed_gems
> + Dir["#{gem_home_root_path}/*"]
> + end
> +
> + def gem_home_root_path
> + File.join(fixture_path, 'tmp', 'gem_home')
> + end
> +
> + def plugin_gem_fixture_root_path
> + File.join(fixture_path, 'gems')
> + end
> end
> \ No newline at end of file
> Index: test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/dependency_one/dependency_one.gemspec
> ===================================================================
> --- test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/dependency_one/dependency_one.gemspec (revision 0)
> +++ test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/dependency_one/dependency_one.gemspec (revision 0)
> @@ -0,0 +1,6 @@
> +Gem::Specification.new do |s|
> + s.name = 'dependency_one'
> + s.version = '1.0.0'
> + s.summary = 'This gem is a terminal dependency on top_level_gem but its version is too low'
> + s.files = Dir['*.rb'] + Dir['lib/*']
> +end
> Index: test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/dependency_one/init.rb
> ===================================================================
> --- test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/dependency_one/init.rb (revision 0)
> +++ test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/dependency_one/init.rb (revision 0)
> @@ -0,0 +1 @@
> +# This gem does nothing for now
> Index: test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/top_level_gem_with_unsatisfiable_dependency/init.rb
> ===================================================================
> --- test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/top_level_gem_with_unsatisfiable_dependency/init.rb (revision 0)
> +++ test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/top_level_gem_with_unsatisfiable_dependency/init.rb (revision 0)
> @@ -0,0 +1 @@
> +# This gem does nothing for now
> Index: test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/top_level_gem_with_unsatisfiable_dependency/top_level_gem_with_unsatisfiable_dependency.gemspec
> ===================================================================
> --- test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/top_level_gem_with_unsatisfiable_dependency/top_level_gem_with_unsatisfiable_dependency.gemspec (revision 0)
> +++ test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/top_level_gem_with_unsatisfiable_dependency/top_level_gem_with_unsatisfiable_dependency.gemspec (revision 0)
> @@ -0,0 +1,7 @@
> +Gem::Specification.new do |s|
> + s.name = 'top_level_gem_with_unsatisfiable_dependency'
> + s.version = '1.0.0'
> + s.summary = 'This gem has a dependency on dependency_one'
> + s.files = Dir['*.rb'] + Dir['lib/*']
> + s.add_dependency 'dependency_one', '> 3.0.0'
> +end
> Index: test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/dependency_loading_order.yml
> ===================================================================
> --- test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/dependency_loading_order.yml (revision 0)
> +++ test/fixtures/gems/gem_with_version_constraint_on_dep_that_can_not_be_satisfied/dependency_loading_order.yml (revision 0)
> @@ -0,0 +1,2 @@
> +- dependency_one-1.0.0
> +- top_level_gem_with_unsatisfiable_dependency-1.0.0
> \ No newline at end of file
> Index: test/fixtures/gems/gem_with_dep_that_has_its_own_deps/dependency_loading_order.yml
> ===================================================================
> --- test/fixtures/gems/gem_with_dep_that_has_its_own_deps/dependency_loading_order.yml (revision 0)
> +++ test/fixtures/gems/gem_with_dep_that_has_its_own_deps/dependency_loading_order.yml (revision 0)
> @@ -0,0 +1,3 @@
> +- second_generation_dep-1.0.0
> +- first_generation_dep-1.0.0
> +- top_level_gem_with_nested_deps-1.0.0
> \ No newline at end of file
> Index: test/fixtures/gems/gem_with_dep_that_has_its_own_deps/top_level_gem_with_nested_deps/init.rb
> ===================================================================
> --- test/fixtures/gems/gem_with_dep_that_has_its_own_deps/top_level_gem_with_nested_deps/init.rb (revision 0)
> +++ test/fixtures/gems/gem_with_dep_that_has_its_own_deps/top_level_gem_with_nested_deps/init.rb (revision 0)
> @@ -0,0 +1 @@
> +# This gem does nothing for now
> Index: test/fixtures/gems/gem_with_dep_that_has_its_own_deps/top_level_gem_with_nested_deps/top_level_gem_with_nested_deps.gemspec
> ===================================================================
> --- test/fixtures/gems/gem_with_dep_that_has_its_own_deps/top_level_gem_with_nested_deps/top_level_gem_with_nested_deps.gemspec (revision 0)
> +++ test/fixtures/gems/gem_with_dep_that_has_its_own_deps/top_level_gem_with_nested_deps/top_level_gem_with_nested_deps.gemspec (revision 0)
> @@ -0,0 +1,7 @@
> +Gem::Specification.new do |s|
> + s.name = 'top_level_gem_with_nested_deps'
> + s.version = '1.0.0'
> + s.summary = 'This gem has a depency on first_generation_dep which has its own dependency'
> + s.files = Dir['*.rb'] + Dir['lib/*']
> + s.add_dependency 'first_generation_dep'
> +end
> Index: test/fixtures/gems/gem_with_dep_that_has_its_own_deps/first_generation_dep/init.rb
> ===================================================================
> --- test/fixtures/gems/gem_with_dep_that_has_its_own_deps/first_generation_dep/init.rb (revision 0)
> +++ test/fixtures/gems/gem_with_dep_that_has_its_own_deps/first_generation_dep/init.rb (revision 0)
> @@ -0,0 +1 @@
> +# This gem does nothing for now
> Index: test/fixtures/gems/gem_with_dep_that_has_its_own_deps/first_generation_dep/first_generation_dep.gemspec
> ===================================================================
> --- test/fixtures/gems/gem_with_dep_that_has_its_own_deps/first_generation_dep/first_generation_dep.gemspec (revision 0)
> +++ test/fixtures/gems/gem_with_dep_that_has_its_own_deps/first_generation_dep/first_generation_dep.gemspec (revision 0)
> @@ -0,0 +1,7 @@
> +Gem::Specification.new do |s|
> + s.name = 'first_generation_dep'
> + s.version = '1.0.0'
> + s.summary = 'This gem is an intermediary dependency'
> + s.files = Dir['*.rb'] + Dir['lib/*']
> + s.add_dependency 'second_generation_dep'
> +end
> Index: test/fixtures/gems/gem_with_dep_that_has_its_own_deps/second_generation_dep/second_generation_dep.gemspec
> ===================================================================
> --- test/fixtures/gems/gem_with_dep_that_has_its_own_deps/second_generation_dep/second_generation_dep.gemspec (revision 0)
> +++ test/fixtures/gems/gem_with_dep_that_has_its_own_deps/second_generation_dep/second_generation_dep.gemspec (revision 0)
> @@ -0,0 +1,6 @@
> +Gem::Specification.new do |s|
> + s.name = 'second_generation_dep'
> + s.version = '1.0.0'
> + s.summary = 'This is the terminal dependency in multiple levels of dependencies'
> + s.files = Dir['*.rb'] + Dir['lib/*']
> +end
> Index: test/fixtures/gems/gem_with_dep_that_has_its_own_deps/second_generation_dep/init.rb
> ===================================================================
> --- test/fixtures/gems/gem_with_dep_that_has_its_own_deps/second_generation_dep/init.rb (revision 0)
> +++ test/fixtures/gems/gem_with_dep_that_has_its_own_deps/second_generation_dep/init.rb (revision 0)
> @@ -0,0 +1 @@
> +# This gem does nothing for now
> Index: test/fixtures/gems/gem_with_a_missing_dep/dependency_loading_order.yml
> ===================================================================
> --- test/fixtures/gems/gem_with_a_missing_dep/dependency_loading_order.yml (revision 0)
> +++ test/fixtures/gems/gem_with_a_missing_dep/dependency_loading_order.yml (revision 0)
> @@ -0,0 +1,2 @@
> +- dependency_one-1.0.0
> +- top_level_gem_with_missing_dep-1.0.0
> \ No newline at end of file
> Index: test/fixtures/gems/gem_with_a_missing_dep/top_level_gem_with_missing_dep/top_level_gem_with_missing_dep.gemspec
> ===================================================================
> --- test/fixtures/gems/gem_with_a_missing_dep/top_level_gem_with_missing_dep/top_level_gem_with_missing_dep.gemspec (revision 0)
> +++ test/fixtures/gems/gem_with_a_missing_dep/top_level_gem_with_missing_dep/top_level_gem_with_missing_dep.gemspec (revision 0)
> @@ -0,0 +1,7 @@
> +Gem::Specification.new do |s|
> + s.name = 'top_level_gem_with_missing_dep'
> + s.version = '1.0.0'
> + s.summary = 'This gem has a dependency on dependency_one but dependency_one is missing'
> + s.files = Dir['*.rb'] + Dir['lib/*']
> + s.add_dependency 'dependency_one'
> +end
> Index: test/fixtures/gems/gem_with_a_missing_dep/top_level_gem_with_missing_dep/init.rb
> ===================================================================
> --- test/fixtures/gems/gem_with_a_missing_dep/top_level_gem_with_missing_dep/init.rb (revision 0)
> +++ test/fixtures/gems/gem_with_a_missing_dep/top_level_gem_with_missing_dep/init.rb (revision 0)
> @@ -0,0 +1 @@
> +# This gem does nothing for now
> Index: test/fixtures/gems/gem_with_no_deps/top_level_gem_with_no_deps/init.rb
> ===================================================================
> --- test/fixtures/gems/gem_with_no_deps/top_level_gem_with_no_deps/init.rb (revision 0)
> +++ test/fixtures/gems/gem_with_no_deps/top_level_gem_with_no_deps/init.rb (revision 0)
> @@ -0,0 +1 @@
> +# This gem does nothing for now
> Index: test/fixtures/gems/gem_with_no_deps/top_level_gem_with_no_deps/top_level_gem_with_no_deps.gemspec
> ===================================================================
> --- test/fixtures/gems/gem_with_no_deps/top_level_gem_with_no_deps/top_level_gem_with_no_deps.gemspec (revision 0)
> +++ test/fixtures/gems/gem_with_no_deps/top_level_gem_with_no_deps/top_level_gem_with_no_deps.gemspec (revision 0)
> @@ -0,0 +1,6 @@
> +Gem::Specification.new do |s|
> + s.name = 'top_level_gem_with_no_deps'
> + s.version = '1.0.0'
> + s.summary = 'This gem has no depencies'
> + s.files = Dir['*.rb'] + Dir['lib/*']
> +end
> Index: test/fixtures/gems/gem_with_no_deps/dependency_loading_order.yml
> ===================================================================
> --- test/fixtures/gems/gem_with_no_deps/dependency_loading_order.yml (revision 0)
> +++ test/fixtures/gems/gem_with_no_deps/dependency_loading_order.yml (revision 0)
> @@ -0,0 +1 @@
> +- top_level_gem_with_no_deps-1.0.0
> \ No newline at end of file
> Index: test/fixtures/gems/gem_with_one_level_of_deps/dependency_one/dependency_one.gemspec
> ===================================================================
> --- test/fixtures/gems/gem_with_one_level_of_deps/dependency_one/dependency_one.gemspec (revision 0)
> +++ test/fixtures/gems/gem_with_one_level_of_deps/dependency_one/dependency_one.gemspec (revision 0)
> @@ -0,0 +1,6 @@
> +Gem::Specification.new do |s|
> + s.name = 'dependency_one'
> + s.version = '1.0.0'
> + s.summary = 'This gem is a terminal dependency on top_level_gem'
> + s.files = Dir['*.rb'] + Dir['lib/*']
> +end
> Index: test/fixtures/gems/gem_with_one_level_of_deps/dependency_one/init.rb
> ===================================================================
> --- test/fixtures/gems/gem_with_one_level_of_deps/dependency_one/init.rb (revision 0)
> +++ test/fixtures/gems/gem_with_one_level_of_deps/dependency_one/init.rb (revision 0)
> @@ -0,0 +1 @@
> +# This gem does nothing for now
> Index: test/fixtures/gems/gem_with_one_level_of_deps/top_level_gem/top_level_gem.gemspec
> ===================================================================
> --- test/fixtures/gems/gem_with_one_level_of_deps/top_level_gem/top_level_gem.gemspec (revision 0)
> +++ test/fixtures/gems/gem_with_one_level_of_deps/top_level_gem/top_level_gem.gemspec (revision 0)
> @@ -0,0 +1,7 @@
> +Gem::Specification.new do |s|
> + s.name = 'top_level_gem'
> + s.version = '1.0.0'
> + s.summary = 'This gem has a dependency on dependency_one'
> + s.files = Dir['*.rb'] + Dir['lib/*']
> + s.add_dependency 'dependency_one'
> +end
> Index: test/fixtures/gems/gem_with_one_level_of_deps/top_level_gem/init.rb
> ===================================================================
> --- test/fixtures/gems/gem_with_one_level_of_deps/top_level_gem/init.rb (revision 0)
> +++ test/fixtures/gems/gem_with_one_level_of_deps/top_level_gem/init.rb (revision 0)
> @@ -0,0 +1 @@
> +# This gem does nothing for now
> Index: test/fixtures/gems/gem_with_one_level_of_deps/dependency_loading_order.yml
> ===================================================================
> --- test/fixtures/gems/gem_with_one_level_of_deps/dependency_loading_order.yml (revision 0)
> +++ test/fixtures/gems/gem_with_one_level_of_deps/dependency_loading_order.yml (revision 0)
> @@ -0,0 +1,2 @@
> +- dependency_one-1.0.0
> +- top_level_gem-1.0.0
> \ No newline at end of file
> Index: test/plugin_test_helper.rb
> ===================================================================
> --- test/plugin_test_helper.rb (revision 6292)
> +++ test/plugin_test_helper.rb (working copy)
> @@ -9,9 +9,13 @@
> RAILS_ROOT = '.' unless defined?(RAILS_ROOT)
> class Test::Unit::TestCase
> def plugin_fixture_root_path
> - File.join(File.dirname(__FILE__), 'fixtures', 'plugins')
> + File.join(fixture_path, 'plugins')
> end
>
> + def fixture_path
> + File.join(File.dirname(__FILE__), 'fixtures')
> + end
> +
> def only_load_the_following_plugins!(plugins)
> @initializer.configuration.plugins = plugins
> end
> Index: lib/rails_generator/generators/applications/app/app_generator.rb
> ===================================================================
> --- lib/rails_generator/generators/applications/app/app_generator.rb (revision 6292)
> +++ lib/rails_generator/generators/applications/app/app_generator.rb (working copy)
> @@ -154,10 +154,12 @@
> test/unit
> vendor
> vendor/plugins
> + vendor/gems
> tmp/sessions
> tmp/sockets
> tmp/cache
> tmp/pids
> + tmp/gem_home
> )
>
> MYSQL_SOCKET_LOCATIONS = [
> Index: lib/initializer.rb
> ===================================================================
> --- lib/initializer.rb (revision 6293)
> +++ lib/initializer.rb (working copy)
> @@ -1,5 +1,6 @@
> require 'logger'
> require 'set'
> +require 'fileutils'
> require File.join(File.dirname(__FILE__), 'railties_path')
> require File.join(File.dirname(__FILE__), 'rails/version')
> require File.join(File.dirname(__FILE__), 'rails/plugin/locator')
> @@ -346,7 +347,16 @@
> unless configuration.plugins.nil?
> unless loaded_plugins == configuration.plugins
> missing_plugins = configuration.plugins - loaded_plugins
> - raise LoadError, "Could not locate the following plugins: #{missing_plugins.to_sentence}"
> + message = if missing_plugins.any?
> + "Could not locate the following plugins: #{missing_plugins.to_sentence}"
> + else
> + "It seems as though you have specified an explicit plugin loading order " +
> + "that contradicts the dependency loading order of a gem plugin. " +
> + "Try comparing your explicit loading order with the dependency loading " +
> + "order of the gem plugins you have installed in #{configuration.gem_home}. " +
> + "You probably should just remove the explicit plugin loading. "
> + end
> + raise LoadError, message
> end
> end
> end
> @@ -441,7 +451,15 @@
> # The path to the root of the plugins directory. By default, it is in
> # <tt>vendor/plugins</tt>.
> attr_accessor :plugin_paths
> +
> + # The path to the root of the bundled gems directory. By default, it is in
> + # <tt>vendor/gems</tt>.
> + attr_accessor :bundled_gem_path
>
> + # The path into which Rails will on-the-fly install gems from its
> + # bundled_gem_path on startup. By default, it is in <tt>tmp/gem_home</tt>
> + attr_accessor :gem_home
> +
> # The classes that handle finding the desired plugins that you'd like to load for
> # your application. By default it is the Rails::Plugin::FileSystemLocator which finds
> # plugins to load in <tt>vendor/plugins</tt>. You can hook into gem location by subclassing
> @@ -468,6 +486,8 @@
> self.whiny_nils = default_whiny_nils
> self.plugins = default_plugins
> self.plugin_paths = default_plugin_paths
> + self.bundled_gem_path = default_bundled_gem_path
> + self.gem_home = default_gem_home
> self.plugin_locators = default_plugin_locators
> self.plugin_loader = default_plugin_loader
> self.database_configuration_file = default_database_configuration_file
> @@ -625,8 +645,16 @@
> ["#{root_path}/vendor/plugins"]
> end
>
> + def default_bundled_gem_path
> + "#{root_path}/vendor/gems/"
> + end
> +
> + def default_gem_home
> + "#{root_path}/tmp/gem_home"
> + end
> +
> def default_plugin_locators
> - [Plugin::FileSystemLocator]
> + [Plugin::GemLocator, Plugin::FileSystemLocator]
> end
>
> def default_plugin_loader
> Index: lib/rails/plugin/locator.rb
> ===================================================================
> --- lib/rails/plugin/locator.rb (revision 6293)
> +++ lib/rails/plugin/locator.rb (working copy)
> @@ -27,33 +27,151 @@
> end
>
> class FileSystemLocator < Locator
> - private
> - def located_plugins
> - initializer.configuration.plugin_paths.flatten.inject([]) do |plugins, path|
> + private
> + def located_plugins
> + initializer.configuration.plugin_paths.flatten.inject([]) do |plugins, path|
> + plugins.concat locate_plugins_under(path)
> + plugins
> + end.flatten
> + end
> +
> + # This starts at the base path looking for directories that pass the plugin_path? test of the Plugin::Loader.
> + # Since plugins can be nested arbitrarily deep within an unspecified number of intermediary directories,
> + # this method runs recursively until it finds a plugin directory.
> + #
> + # e.g.
> + #
> + # locate_plugins_under('vendor/plugins/acts/acts_as_chunky_bacon')
> + # => 'acts_as_chunky_bacon'
> + def locate_plugins_under(base_path)
> + Dir.glob(File.join(base_path, '*')).inject([]) do |plugins, path|
> + plugin_loader = initializer.configuration.plugin_loader.new(initializer, path)
> + if plugin_loader.loadable?
> + plugins << plugin_loader
> + elsif File.directory?(path)
> plugins.concat locate_plugins_under(path)
> - plugins
> - end.flatten
> + end
> + plugins
> end
> + end
> + end
> +
> + class GemLocator < Locator
> + require 'rubygems/dependency_list'
> + require 'rubygems/format'
> + require 'rubygems/installer'
> +
> + attr_reader :installer
> +
> + def initialize(*args)
> + super
> + @installer = Installer.new(self)
> + end
> +
> + def plugins
> + # Unlike the parent class, we don't want to sort the loaders,
> + # as RubyGems takes care of sorting them in depency order already.
> + located_plugins.select(&:enabled?)
> + end
> +
> + def spec_to_path_mapping
> + @spec_to_path_mapping ||= Dir[File.join(initializer.configuration.bundled_gem_path, "*.gem")].inject({}) do |mapping, gem_path|
> + mapping[specification_for(gem_path)] = gem_path
> + mapping
> + end
> + end
> +
> + def specification_for(gem_path)
> + Gem::Format.from_file_by_path(gem_path).spec
> + end
> +
> + def bundled_gems
> + spec_to_path_mapping.values
> + end
>
> - # This starts at the base path looking for directories that pass the plugin_path? test of the Plugin::Loader.
> - # Since plugins can be nested arbitrarily deep within an unspecified number of intermediary directories,
> - # this method runs recursively until it finds a plugin directory.
> - #
> - # e.g.
> - #
> - # locate_plugins_under('vendor/plugins/acts/acts_as_chunky_bacon')
> - # => 'acts_as_chunky_bacon'
> - def locate_plugins_under(base_path)
> - Dir.glob(File.join(base_path, '*')).inject([]) do |plugins, path|
> - plugin_loader = initializer.configuration.plugin_loader.new(initializer, path)
> - if plugin_loader.plugin_path? && plugin_loader.enabled?
> - plugins << plugin_loader
> - elsif File.directory?(path)
> - plugins.concat locate_plugins_under(path)
> + def path_for(spec)
> + spec_to_path_mapping[spec]
> + end
> +
> + def plugin_path_for(spec)
> + File.join(initializer.configuration.gem_home, 'gems', spec.full_name)
> + end
> +
> + private
> + def located_plugins
> + prepare_gems
> +
> + installer.installed_gems.inject([]) do |plugins, gem_plugin_directory|
> + plugin_loader = initializer.configuration.plugin_loader.new(initializer, gem_plugin_directory)
> + plugins << plugin_loader if plugin_loader.loadable?
> + plugins
> + end
> + end
> +
> + def prepare_gems
> + setup_gem_environment
> + install_gems
> + end
> +
> + # Sets a local GEM_HOME for this Rails application. Installs all gems from vendor/gems
> + # (or configured bundled_gem_path) into this
> + # directory and makes them available to be loaded as plugins.
> + def setup_gem_environment
> + Gem.manage_gems
> + # N.B. the gem_home must be an absolute path or else the Gem::Installer will fail
> + Gem.use_paths(File.expand_path(initializer.configuration.gem_home), [Gem.dir])
> + end
> +
> + def install_gems
> + installer.install
> + end
> +
> + class Installer
> + attr_reader :locator, :installed_gems, :source_index
> +
> + def initialize(locator)
> + @locator = locator
> + @source_index = Gem::SourceIndex.from_gems_in(File.join(locator.initializer.configuration.gem_home, 'specifications'))
> + @installed_gems = []
> + end
> +
> + def install
> + gems_to_install.each do |spec|
> + gem_path = locator.path_for(spec)
> + Gem::Installer.new(gem_path).install unless installed?(spec)
> + @installed_gems << locator.plugin_path_for(spec)
> + end
> + uninstall_unused_gems!
> + end
> +
> + private
> + def gems_to_install
> + @gems_to_install ||= locator.bundled_gems.inject(Gem::DependencyList.new) do |dependency_list, gem_path|
> + dependency_list.add locator.specification_for(gem_path)
> + dependency_list
> + end.dependency_order.reverse
> + end
> +
> + def installed?(spec)
> + !source_index.find_name(spec.name, spec.version).empty?
> + end
> +
> + def uninstall_unused_gems!
> + unused_gems.each do |spec|
> + Gem::DefaultUserInteraction.use_ui(Gem::SilentUI.new) do
> + Gem::Uninstaller.new(spec.name, :version => "= #{spec.version}").uninstall
> + end
> end
> - plugins
> end
> - end
> +
> + def unused_gems
> + previously_installed_gems - gems_to_install
> + end
> +
> + def previously_installed_gems
> + source_index.latest_specs.values
> + end
> + end
> end
> end
> end
> \ No newline at end of file
> Index: lib/rails/plugin/loader.rb
> ===================================================================
> --- lib/rails/plugin/loader.rb (revision 6293)
> +++ lib/rails/plugin/loader.rb (working copy)
> @@ -28,6 +28,10 @@
> def loaded?
> initializer.loaded_plugins.include?(name)
> end
> +
> + def loadable?
> + plugin_path? && enabled?
> + end
>
> def plugin_path?
> File.directory?(directory) && (has_lib_directory? || has_init_file?)

[Chad Fowler]

On 4/20/07, Ben Munat <bmu...@gmail.com> wrote:
>
> I meant the app's dependencies... which gems an app needs to have
> installed. Seems like there would need to be some outside help (new code
> and a config file) to do that.

The code we're working on will work for both plugins (as gems) and
arbitrary gems.

Chad

[Ben Munat]

Sweet!

b

[Josh Peek]

Awesome! When will something like this make it into core?

[epfre...@gmail.com]

Our main reservation vis-a-vis the vendor everything approach is that
it makes for painful distribution when you have multiple teams and
multiple applications that share the same 'operating
environment' (i.e. colocated with a shared GEM_HOME).

If you have a major security hole in shared dependency Foo that you've
backported to all of the major releases, you would need to hunt
through every app and replace the gem in question with the right gem
from based on release. Instead, we just build a gem for each backport,
install them on the gem server, and the 'shared environment' need only
update its repository and restart the apps. No need for a code deploy
on the applications themselves, which would cause the need to create a
maintenance branch on what might have been a short-lived release for
the sole purpose of updating a dependency micro.

>From a development perspective, repositories have worked out great for
us as well. When I fix a major bug in a shared dependency, my
coworkers get the latest and greatest when they 'svn up; plugem
up' (which leverages gem update) in the morning--no wack SVN externals
(we started this journey in svn:external hell) or emails to links with
the latest gem.

Platform-specific gems are also a dilemma in the vendor everything
approach (rmagick, hpricot, etc.)--compiling things at application
start seems somewhat unusual. We have both OSX developers and Windows
developers, so it's nice to just add hpricot to the manifest and let
the repository expose the different platform variations (as opposed to
bundling them all in vendor and adding even more logic to the plugin
stuff).

Gem repositories just do such a good job at acting as a repository for
gems :-)

I know this isn't the world that everyone is in (or wants to be in for
that matter), but I don't think it makes it much more difficult on the
KISS crowd (while it opens up a lot of doors for us fools that like to
overcomplicate things.)

The catch is the need for the application manifest file to declare the
dependencies, but thats brings a lot of secondary value to the table
as well. You can use it to generate spec files to gemify apps (as we
do), guarantee the presence of required third-party gems at startup
time, etc. It's not so bad.

More background on why we'd like to lobby for this approach in the
body and comments of :

http://revolutiononrails.blogspot.com/2007/05/release-plugems-runtime.html
http://revolutiononrails.blogspot.com/2007/05/plugems-more-than-dependency-management.html
http://interblah.net/2007/5/2/plugems-enter-the-rails-addon-fray/comment/297#comment-297

Thoughts?

Eddie
rails-trunk AT revolution.com

[Edward Frederick]

Update:

Val just released the packaging/update component... that + runtime
should be all folks need to try plugems for themselves.

http://revolutiononrails.blogspot.com/2007/05/release-plugems-packaging.html

[Shane Vitarana]

I patched edge with vendor_everything.diff but am getting this error:

uninitialized constant Gem::UnpackCommand
/usr/local/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:2028:in `const_missing'
/Users/shane/rails/contrib/vendor/rails/railties/lib/tasks/gems.rake:15

Did something happen to Gem::UnpackCommand?

Shane

--
http://shanesbrain.net

[Kyle Maxwell]

> What about the native code issue? Are you going to create a
> sub-directory per platform?

It's easy to compile C extensions per-platform. i.e.:

http://svn.kylemaxwell.com/rails_plugins/vendor_everything_extensions/tasks/vendor_everything_extensions_tasks.rake

This works in *nix-land, with a special case needed for Windows (as always).

Kyle