plugin dependencies

7 views
Skip to first unread message

Andrew Kaspick

unread,
Sep 19, 2007, 5:12:22 AM9/19/07
to Ruby on Rails: Core
Hello all,

The topic of plugin dependencies has come up before and it doesn't
seem to have been addressed by core or core doesn't seem to think it's
an issue. I've looked at the current edge code and don't see anything
new, so if I've missed something *please* let me know.

The following article makes mention of a require_plugin functionality...
http://www.pluginaweek.org/2006/11/05/plugin-dependencies-revisited/

and the referenced article seems to make the point that such a feature
may not be required...
http://weblog.techno-weenie.net/2006/10/31/plugin-dependencies

The reason I bring this up is that with many current core features
being moved to plugins, existing plugins that have been written to
improve functionality in some of those core areas may no longer work
since they will now be dependent on these "core plugins" being
installed... and even if they are installed, there's no guarantee that
the load order will be correct.

For example, I have written a plugin to improve the functionality of
the in_place_editor code. This code works just fine with the current
public version of rails. Now in edge this core code has been moved to
a plugin and my plugin will no longer work as expected since the core
plugin is now "required" by my plugin. This is fine (although a more
informational warning would be useful for the user) as the user simply
needs to install the "core plugin" as well. This is where problems
start to occur. Since rails loads plugins in alphabetical order by
default, if my plugin comes before the core plugin in name, my plugin
code is overridden.

With many rails features being deprecated and being moved to plugins,
this will cause many dependencies on any existing plugins. This can
be resolved by specifying the load order with the config.plugins
option, but a user shouldn't have to jump through those hoops to get
things working... at least not before a fair amount of possible
debugging since there may be no indication of the underlying problem
of an existing plugin not working (a user may believe it's their own
code causing problems). Another possible solution is making sure ones
own plugin is alpabetically higher than the one it depends on to force
the proper load order, but that would be rather nasty hack.

So I'd like to know what people think about having a "require_plugin"
type functionality in core? Is there a valid workaround that exists
already? If there isn't, do people even consider this a problem?

Thanks,
Andrew

Pratik

unread,
Sep 19, 2007, 5:28:47 AM9/19/07
to rubyonra...@googlegroups.com
You can actually control the order of plugin loading with
config.plugins array, but I'd guess that'd be PITA

I believe require_plugin ( I'd prefer just "plugin" ) would be a good
to have feature.


--
Cheers!
- Pratik
http://m.onkey.org

Frederick Cheung

unread,
Sep 19, 2007, 5:32:30 AM9/19/07
to rubyonra...@googlegroups.com

On 19 Sep 2007, at 10:28, Pratik wrote:

>
> You can actually control the order of plugin loading with
> config.plugins array, but I'd guess that'd be PITA
>

Eg if plugin_a, previously without dependencies starts depending on
plugin_b (where say plugin_b is some bit of core that has recently
been pushed to a plugin). You have to go and edit config.plugins of
all the apps using plugin_a.

> I believe require_plugin ( I'd prefer just "plugin" ) would be a good
> to have feature.

I'd love that too. We've been using plugins to share code between our
apps and this would really make life easier

Fred
>

> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google
> Groups "Ruby on Rails: Core" group.
> To post to this group, send email to rubyonra...@googlegroups.com
> To unsubscribe from this group, send email to rubyonrails-core-
> unsub...@googlegroups.com
> For more options, visit this group at http://groups.google.com/
> group/rubyonrails-core?hl=en
> -~----------~----~----~----~------~----~------~--~---
>

Andrew Kaspick

unread,
Sep 19, 2007, 5:51:30 AM9/19/07
to rubyonra...@googlegroups.com
I mentioned config.plugins in my OP...

"This can be resolved by specifying the load order with the
config.plugins option, but a user shouldn't have to jump through those
hoops to get things working..."

and yes, PITA is right.

Michael Koziarski

unread,
Sep 19, 2007, 8:03:09 AM9/19/07
to rubyonra...@googlegroups.com
> "This can be resolved by specifying the load order with the
> config.plugins option, but a user shouldn't have to jump through those
> hoops to get things working..."

We're still pretty skeptical about plugin dependencies on the whole.
Plugins monkeypatching plugins seems like a bit of a recipe for
disaster. However we could do something easy like:

config.plugins = [:gems, :some_kind_of_thing, :all]

So you specify the key things at the beginning of the array, then just
use :all to mean 'then everything else in alphabetical order'.

Seems like that'll solve most of the more-common problems where a
specific plugin needs to be early on the load path.


--
Cheers

Koz

Frederick Cheung

unread,
Sep 19, 2007, 8:18:38 AM9/19/07
to rubyonra...@googlegroups.com

On 19 Sep 2007, at 13:03, Michael Koziarski wrote:

>
>> "This can be resolved by specifying the load order with the
>> config.plugins option, but a user shouldn't have to jump through
>> those
>> hoops to get things working..."
>
> We're still pretty skeptical about plugin dependencies on the whole.
> Plugins monkeypatching plugins seems like a bit of a recipe for
> disaster. However we could do something easy like:
>

It seems to me that it's perfectly possible to depend on the presence
of stuff without necessarily patching it (and to be fair a lot of the
time you don't need access to your dependencies at plugin load times
in which case it all just goes away (which has been the case when
we've been using plugins internally to share code between apps) but I
wouldn't want to rely on it always working out that easily).

Fred

Michael Koziarski

unread,
Sep 19, 2007, 8:47:58 AM9/19/07
to rubyonra...@googlegroups.com
> It seems to me that it's perfectly possible to depend on the presence
> of stuff without necessarily patching it (and to be fair a lot of the
> time you don't need access to your dependencies at plugin load times
> in which case it all just goes away (which has been the case when
> we've been using plugins internally to share code between apps) but I
> wouldn't want to rely on it always working out that easily).

If the plugins aren't dependent on each other at load time, the we
don't need something like require_plugin.

--
Cheers

Koz

Frederick Cheung

unread,
Sep 19, 2007, 8:55:04 AM9/19/07
to rubyonra...@googlegroups.com

Sure, and what we've done so far internally is twist things so that
this isn't the case and we don't have to rely on load order. It would
be nice not to have to perform those contorsions.

Fred

Michael Koziarski

unread,
Sep 19, 2007, 9:02:09 AM9/19/07
to rubyonra...@googlegroups.com
> Sure, and what we've done so far internally is twist things so that
> this isn't the case and we don't have to rely on load order. It would
> be nice not to have to perform those contorsions.

So for those situations you could use config.plugins =
[:your_first_thing, :all] ? Or am I missing something.


--
Cheers

Koz

James Adam

unread,
Sep 19, 2007, 9:23:03 AM9/19/07
to rubyonra...@googlegroups.com
On 9/19/07, Michael Koziarski <mic...@koziarski.com> wrote:
> If the plugins aren't dependent on each other at load time, the we
> don't need something like require_plugin.

A very minor change that would help mitigate this type of issue is
separating the addition of plugins to the load path from the
evaulation of init.rb.

If every plugin's lib was in the load path *before* any plugin could
start playing around or monkeypatching, many situations where you
might want to explicitly ensure a plugin was loaded can be made to go
away.

--
* J *
~

Frederick Cheung

unread,
Sep 19, 2007, 9:22:57 AM9/19/07
to rubyonra...@googlegroups.com

Yes you could, but that puts the onus on the containing application.
I've got say 3 plugins that my 5 apps share. Then I add plugin X and
then change plugin A to use something funky from plugin X. I don't
what to encode this new information in 5 different environment.rb
files. It would be nice for that knowledge to stay in plugin A. It's
okish for internal stuff when the author of the plugin and the
consumer of the plugin are the same person/team, but it's a bit nasty
if you're releasing a plugin to a wider audience.

Andrew White

unread,
Sep 19, 2007, 10:04:30 AM9/19/07
to rubyonra...@googlegroups.com

On 19 Sep 2007, at 14:02, Michael Koziarski wrote:

Perhaps some mechanism of changing the order config.plugins from
within a plugin would be simpler, e.g:

Three plugins A, B and C need to be loaded in the order C, A, B as B
depends on A & C and A depends on C.

In a dependencies.rb within the plugin's top level directory:

Plugin A: depends_on :plugin_c
Plugin B: depends_on :plugin_a, :plugin_c
Plugin C: <empty>

The plugin loader would start off with the alphabetical list and then
remove and re-insert each plugin depending on it's dependencies. e.g:

Initial: [:plugin_a, :plugin_b, :plugin_c]
A/deps : [:plugin_b, :plugin_c, :plugin_a] # A re-inserted
after C
B/deps : [:plugin_c, :plugin_a, :plugin_b] # B re-inserted
after A
C/deps : [:plugin_c, :plugin_a, :plugin_b] # No changes

I'm sure there's some situation which wouldn't work but at least it
help reduce the need to manually specify config.plugins.


Andrew White

Pratik

unread,
Sep 19, 2007, 10:14:34 AM9/19/07
to rubyonra...@googlegroups.com
> Initial: [:plugin_a, :plugin_b, :plugin_c]
> A/deps : [:plugin_b, :plugin_c, :plugin_a] # A re-inserted
> after C
> B/deps : [:plugin_c, :plugin_a, :plugin_b] # B re-inserted
> after A
> C/deps : [:plugin_c, :plugin_a, :plugin_b] # No changes
>

After looking at that, I don't think it's a good idea to support it at
all. Mainly because plugins have no cocept of versions like gems. And
because of distributed nature of plugins, it can break things very
easily. If plugins really have this level of dependancy, they're
better off duplication the functionality I guess.

Aaron Pfeifer

unread,
Sep 19, 2007, 10:38:54 AM9/19/07
to Ruby on Rails: Core
I thought I'd offer my 2 cents on the subject. plugin_dependencies
has evolved since that article was written. I never use
require_plugin anymore and instead rely on simply calling require,
which allows the dependency to be loaded either as a plugin or as a
gem. As is, my plugin development practices allow me to release
plugins as both a Rails plugin and a Ruby gem.

Say we have 2 plugins, encrypted_strings and encrypted_attributes
(which depends on encrypted_strings). In this case, we do the
following:

encrypted_attributes/init.rb:
require 'encrypted_attributes'

encrypted_attributes/lib/encrypted_attributes.rb:
require 'encrypted_strings'
...do stuff...

With plugin_dependencies loaded, this will look for encrypted_strings
as either a vendor/plugin within the application or as a gem. This
lets me be flexible in case I want the same plugin/gem/whatever used
across multiple applications in a single environment. The basic idea
is that we're not specifically tying a library as a plugin using
require_plugin, but instead treating it the same as you do with Rails
either being loaded locally in vendor/ or as a gem.

With regard to partial loading, both myself and James Adam use the
following technique:

Rails::Initializer.run do |config|
config.plugins = %w(
plugins_plus
something_else
*
)
end

This is currently available as a plugin (partial_plugin_list). Not
sure if it's the best syntax, but it's there if you need it.

I really do think that dependencies between plugins/gems are
important. If you browse through all of the PluginAWeek plugins,
you'll notice several dependencies (has_emails depending on
validates_as_email_address, user_authentication depending on
encrypted_attributes). In some cases, the dependent library *must* be
loaded before continuing and in other cases it doesn't. However, my
whole argument behind adding "require" for each dependency has been
that this information belongs in the plugin and not in a
configuration. This way, I don't have to duplicate that configuration
in every application I use those plugins.

Hope this helps :)

James Adam

unread,
Sep 19, 2007, 11:19:45 AM9/19/07
to rubyonra...@googlegroups.com
On 9/19/07, Aaron Pfeifer <aaron....@gmail.com> wrote:
> Say we have 2 plugins, encrypted_strings and encrypted_attributes
> (which depends on encrypted_strings). In this case, we do the
> following:
>
> encrypted_attributes/init.rb:
> require 'encrypted_attributes'
>
> encrypted_attributes/lib/encrypted_attributes.rb:
> require 'encrypted_strings'
> ...do stuff...
>
> With plugin_dependencies loaded, this will look for encrypted_strings
> as either a vendor/plugin within the application or as a gem. This
> lets me be flexible in case I want the same plugin/gem/whatever used
> across multiple applications in a single environment.

Ignoring anything about requiring specific versions, couldn't this be
achieved by having all of the plugins added to the load path before
any init.rb files are evaluated (my suggestion above)? That way the
normal Ruby "require" would seem to provide everything that you
describe here.

Jay Levitt

unread,
Sep 19, 2007, 11:28:49 AM9/19/07
to rubyonra...@googlegroups.com
On Wed, 19 Sep 2007 14:03:09 +0200, Michael Koziarski wrote:

> We're still pretty skeptical about plugin dependencies on the whole.
> Plugins monkeypatching plugins seems like a bit of a recipe for
> disaster. However we could do something easy like:
>
> config.plugins = [:gems, :some_kind_of_thing, :all]
>
> So you specify the key things at the beginning of the array, then just
> use :all to mean 'then everything else in alphabetical order'.

That still means that plugins aren't really plug-in anymore. If I use a
plugin that needs acts_as_list at initialization, and that plugin happens
to be named "aardvark" instead of "zebra", I'll have to go manually add
acts_as_list to the beginning of config.plugins? Why not just manually
copy the plugin code into lib/ as well?

I understand the core team's traditional reluctance to add plugin
dependencies, for both complexity and YAGNI reasons. I remember a ticket a
while back where somebody said "But if I have 70 plugins, it becomes a
pain", and DHH rightly responded "You don't need 70 plugins".

But with the move towards pushing more and more core functionality into
plugins, it really should be revisited - if only to be re-argued in the
present tense. I'm working on a relatively small app on Edge that already
uses a dozen plugins, and we've just gotten started. I don't think that's
going to be uncommon.

--
Jay Levitt |
Boston, MA | My character doesn't like it when they
Faster: jay at jay dot fm | cry or shout or hit.
http://www.jay.fm | - Kristoffer

Jay Levitt

unread,
Sep 19, 2007, 11:29:56 AM9/19/07
to rubyonra...@googlegroups.com
On Wed, 19 Sep 2007 15:14:34 +0100, Pratik wrote:

> After looking at that, I don't think it's a good idea to support it at
> all. Mainly because plugins have no cocept of versions like gems. And
> because of distributed nature of plugins, it can break things very
> easily. If plugins really have this level of dependancy, they're
> better off duplication the functionality I guess.

Do you really want to go back to the days where we all had to roll our own
linked lists?

Aaron Pfeifer

unread,
Sep 19, 2007, 11:42:07 AM9/19/07
to Ruby on Rails: Core
Indeed, I actually had proposed this way back when:
http://dev.rubyonrails.org/ticket/6418#comment:17. I'm not sure why I
came to the conclusion that this was already being done on edge. I
think this did result in some other issues that needed to be
considered, such as:
1. When a plugin is loaded, it's lib path always comes before other
plugins, therefore if I trying to require a file called "node", it'll
look in my plugin's lib path before looking elsewhere. This would not
be the case with the paths added before initialization.
2. Similarly, the order of the paths would not reflect the order in
which the plugins were loaded, resulting in potential issues with the
priority in which files are loaded
3. You would still need support for loading gems as plugins because
otherwise they would not go through the same motions as normal plugins
(such as evaluating the init.rb).
4. You'd need support for loading tasks from gems.
5. If you call require 'encrypted_strings', and encrypted_strings/lib/
encrypted_strings.rb gets loaded, then the plugin is still going to
get loaded again later on if it's in vendor/plugins. This again means
that the list of plugins loaded is out of order and would break
engines/plugins+.

That's all I can think of right now. I certainly agree that this
would be simpler, although I'm not sure if it brings up additional
issues that need to be addressed. Any technique that allows us to
describe dependencies in the plugins and not in the application's
configuration I support.

On Sep 19, 11:19 am, "James Adam" <james.a...@gmail.com> wrote:

Pratik

unread,
Sep 19, 2007, 11:46:50 AM9/19/07
to rubyonra...@googlegroups.com
So let me get it straight.

I release Plugin A. Joe release Plugin B which uses "your"
functionality to depend on my Plugin A. One fine morning I get
enormous motiviation to improve performance of my Plugin. So I take a
day off at work and rewrite the entire damn thing and release it. Poor
Joe has no clue what I did. So every person installing his Plugin B,
will have broken functionality because noone has a goddman idea about
what I did with Plugin A last night.

Welcome to Stone age.

If I were you, I'd refrain from personal attacks.

>
> Do you really want to go back to the days where we all had to roll our own
> linked lists?
>
> --
> Jay Levitt |
> Boston, MA | My character doesn't like it when they
> Faster: jay at jay dot fm | cry or shout or hit.
> http://www.jay.fm | - Kristoffer
>
>
> >
>

Aaron Pfeifer

unread,
Sep 19, 2007, 11:52:43 AM9/19/07
to Ruby on Rails: Core
That "70 plugins" somebody was me and I guess I was an outlier back
then :) Probably still am ;)

Michael Koziarski

unread,
Sep 19, 2007, 12:02:07 PM9/19/07
to rubyonra...@googlegroups.com
> Ignoring anything about requiring specific versions, couldn't this be
> achieved by having all of the plugins added to the load path before
> any init.rb files are evaluated (my suggestion above)? That way the
> normal Ruby "require" would seem to provide everything that you
> describe here.

This sounds like it's probably harmless and could solve a bunch of the
different issues people have been mentioning. If you wanted to take a
look at this, I'd be happy to apply it.

We're definitely not going to go down the route of a massive
dependency system of our own. Rubygems does this kind of thing
already, and perhaps we just need to go down the plugins-as-gems
approach.

Either way, I'd be open to applying a patch to add :all to
config.plugins or the load order changing, but that's about it for
2.0.

--
Cheers

Koz

Aaron Pfeifer

unread,
Sep 19, 2007, 12:08:04 PM9/19/07
to Ruby on Rails: Core
I like the plugins-as-gems approach. I never personally saw a reason
for any difference between plugins and gems, which is why I write my
plugins to be supported as both. Is the only thing holding back a
plugins-as-gems approach RubyGems's inability to support multiple
search paths? (at least, that's what I think I had read a while back)

On Sep 19, 12:02 pm, "Michael Koziarski" <mich...@koziarski.com>
wrote:

Andrew Kaspick

unread,
Sep 19, 2007, 12:55:55 PM9/19/07
to rubyonra...@googlegroups.com
Adding something like :all is not an ideal fix as this doesn't allow
any type of error reporting for the user that dependent plugins are
missing or possibly being loaded incorrectly. I mentioned something
similiar in my OP.

Back to why I brought this all up in the first place...

Somebody was using my plugin that I wrote to improve the
in_place_editor code. The user was using edge rails. The
in_place_editor code was recently moved to a plugin, so now my plugin
stopped working and the user contacts me that it doesn't work in edge
(which is to expected with edge code), but the user was not aware that
the original rails code had been moved to an external plugin and
neither was I. It turns out that simply installing the new "core
plugin" fixes the problem (since the alphabetical ordering was ok in
this case), but the user had no indication that this was the problem
and neither did I initially. I'm sure this will become more of a
common case in the future (especially with the release of rails 2).

If there was a method to report back some sort of debug information to
the user informing them they need to install the additional plugin
(and possibly modifying the config.plugins) property that would save
many headaches I'm sure along with saving time to support users on how
to fix the issue that some feedback from the plugin could easily do.
I'm not saying that a full error reporting system be implemented, that
could easily be accomplished with a simple print statement, but the
functionality to find out if there's a dependency problem would be
required.

Using :all would be better than what's currently there which is
basically and all or nothing approach to loading plugins, but I think
it's far from ideal.

What do you mean by "load order changing"? I've missed the details of
this somewhere.

On 9/19/07, Michael Koziarski <mic...@koziarski.com> wrote:
>

Jay Levitt

unread,
Sep 19, 2007, 1:22:37 PM9/19/07
to rubyonra...@googlegroups.com
On Wed, 19 Sep 2007 16:46:50 +0100, Pratik wrote:

> I release Plugin A. Joe release Plugin B which uses "your"
> functionality to depend on my Plugin A. One fine morning I get
> enormous motiviation to improve performance of my Plugin. So I take a
> day off at work and rewrite the entire damn thing and release it. Poor
> Joe has no clue what I did. So every person installing his Plugin B,
> will have broken functionality because noone has a goddman idea about
> what I did with Plugin A last night.

I see what you're saying, and I don't disagree. There's always going to be
a tension between reuse and incompatibility; it's been true with UNIX
shared libraries, it's true for Windows "DLL hell", I imagine it's true for
managed .NET and Java applications, and it will be true in Ruby and Rails.
It's why we freeze gems and Rails into our apps, and it's why 3rdRail comes
with not only its own copy of Eclipse but its own copy of Java.

And that's the problem with the nothing-shared rule - just how much of your
dependencies do you include? What stable, sane environment can you count
on? Should plugins include Ruby? After all, 1.8.5 to 1.8.6 broke quite a
few things.

But packaging everything can destabilize you too, because you won't pick up
bug fixes to the core libraries. On Windows, nearly everything includes
its own "gdiplus" DLL. Well, when Microsoft found a security hole in
gdiplus, the world went crazy trying to release new versions of everything
with an updated gdiplus. I'm pretty sure there's something somewhere on my
hard drive that's vulnerable.

These aren't new problems, and we'll do well to learn from past mistakes
and experiments as we carve out solutions.

> If I were you, I'd refrain from personal attacks.

I didn't intend my post to be a personal attack, and I'm sorry that it made
you feel attacked. I didn't make any statements about your person; it's
your position that I questioned.

Perhaps it was a bit hyperbolic to suggest we all invent our linked-list
libraries, but in fact 15 years ago that's exactly where we were. It took
ages (and IIRC several core language improvements!) for the C++ STL to
become adopted and truly standardized, because reuse, versioning and
efficency (developer AND system) are huge hairy issues. I'm sure the same
happened before that with stdlib, and after that with Java, and it's
happening here now.

Java was born in an age where complex programs solved complex problems, and
the idea of perfect portability and modular object reuse was going to save
us all. It didn't.

Rails comes at the problem a different way, and perhaps as a result, Rails
developers have a strong cultural bias against reuse. Maybe it's because
it's often as easy to write something from scratch as it is to reuse it.
Maybe it's because we're learning and reinventing so quickly that any
rewrite is going to be a huge improvement. Maybe it's a rebellion against
the idea of "one true way" to do something. Maybe it's because we haven't
started building complex Rails apps yet; maybe it's because we *shouldn't*
build any. Or maybe it's because we just haven't hit the pain point, but
we're about to.

What I find really fascinating is that Perl, which comes from a very
similar TIMTOWTDI perspective, has begat CPAN, which (from what I can tell)
very often has One Good Library for a given function. I would love to
understand more about how that evolved, and see what we can borrow.

Sorry for the novel. Executive summary: These aren't new problems. Let's
learn from at old solutions.

Pratik

unread,
Sep 19, 2007, 3:40:33 PM9/19/07
to rubyonra...@googlegroups.com
> I didn't intend my post to be a personal attack, and I'm sorry that it made
> you feel attacked. I didn't make any statements about your person; it's
> your position that I questioned.
>

Point taken. No worries :) Thanks for clearing the air.

>
> Sorry for the novel. Executive summary: These aren't new problems. Let's
> learn from at old solutions.

Exactly. That's why plugins as gems approach is better, as you can
have versioning of your plugins and any other plugin can easily say "I
need version X of plugingem Foo" even though the latest version of Foo
is X+2.

Andrew Kaspick

unread,
Sep 19, 2007, 3:46:47 PM9/19/07
to rubyonra...@googlegroups.com
Except there is no real concept of versions with plugins. Starting
with simple dependency requirements of "plugin X requires plugin Y"
would make me happy for now.

On 9/19/07, Pratik <prati...@gmail.com> wrote:
>

Matt Palmer

unread,
Sep 19, 2007, 7:50:31 PM9/19/07
to rubyonra...@googlegroups.com
On Wed, Sep 19, 2007 at 02:46:47PM -0500, Andrew Kaspick wrote:
> Except there is no real concept of versions with plugins. Starting
> with simple dependency requirements of "plugin X requires plugin Y"
> would make me happy for now.

Emphasis on "for now", I hope. When your dependencies can change, it's not
long before you start needing to encode temporal information in your
dependency tree to keep everything working.

- Matt

--
Imagine an orkplace where you were the only non executive: Make them all
CEO. Give them all at least one Masters degree and/or a PhD, and the ego
trip that comes with that. Now double it. That's education.
-- GB, in the Monastery

Eric Anderson

unread,
Sep 19, 2007, 8:28:51 PM9/19/07
to rubyonra...@googlegroups.com
Michael Koziarski wrote:
> We're definitely not going to go down the route of a massive
> dependency system of our own. Rubygems does this kind of thing
> already, and perhaps we just need to go down the plugins-as-gems
> approach.

I just want to add my +1 to this comment. It seems like plugins are
trying to solve all the same problems RubyGems is trying to solve. But
plugins are just doing it badly while RubyGems is doing better (not
perfect but much better).

In my mind plugins should be deprecated and eventually removed. Having a
"gem" route and a "plugin" route is just too confusing. To me if I want
to enhance Rails to have the functionality of acts_as_paranoid then I
should just:

1. gem install acts_as_paranoid
2. Add "require acts_as_paranoid" in my environment.rb. If I care about
binding it to a specific version of acts_as_paranoid I would instead put
"require_gem 'acts_as_versioned', '0.3'"

Then I'm done. This would remove a lot of code from Rails core making it
simpler but it would give us the following benefits we don't currently have:

1. Distribution - yes I know "./script/plugin install blah" but it is
crappy. Requires SVN to be laid out in a non-standard way, has a flaky
way of finding the package so you pretty much always have to get the SVN
URL and has problems with some repositories (SSL, etc). Gem is quick,
simple and works.

2. Versioning - This ensures stability. The plugin route has nothing for
this other than bringing it into your own repository and then having to
use third party tools like piston to allow updates when desired.

3. Dependencies - Specifying a load order is just archane. Yes it works
but as I user of acts_as_paraniod I shouldn't have to care if it has a
dependency. When I install acts_as_paranoid it should tell rubygems its
dependencies and all I have to do is say install with all dependencies.

4. Simplicity - No more spending hours tracking to load order dependency
problems. Reduction of Rails core code.

Eric

Pascal Belloncle

unread,
Sep 20, 2007, 1:26:33 AM9/20/07
to rubyonra...@googlegroups.com
I like the approach, a lot. But (isn't there always a but...), isn't
this going to be a problem for people using shared hosting, or people
that just can't install gems? Unless there is an easy way to locally
install gems...

Cheers,
Pascal
--
http://blog.nanorails.com

Gabe da Silveira

unread,
Sep 20, 2007, 1:40:46 AM9/20/07
to rubyonra...@googlegroups.com
On 9/19/07, Pascal Belloncle <p...@nanorails.com> wrote:

I like the approach, a lot.  But (isn't there always a but...), isn't
this going to be a problem for people using shared hosting, or people
that just can't install gems?  Unless there is an easy way to locally
install gems...

On any decent shared host you can install your own gems.  If you can't then you will be in for a world of hurt anyway, and frankly, the world of shared Rails hosting is too low of a common denominator too target.  There are too many issues with memory and stability as it is, so I say we make Rails do the best thing it can assuming a VPS or dedicated box, and then let the shared hosts figure out how to make it work for $5/month.

Putting Rails' weight behind the gem system seems like a win-win to me.

Josh Susser

unread,
Sep 20, 2007, 1:57:16 AM9/20/07
to rubyonra...@googlegroups.com
You can also freeze gems into the vendor directory. Now what's the downside compared to plugins in vendor?

Pascal Belloncle

unread,
Sep 20, 2007, 2:36:01 AM9/20/07
to rubyonra...@googlegroups.com
Should have thought of that. I suppose that would be workable. I'm sure someone could even come up with a script that downloads a gem and installs it directly into vendor.

That takes care of my concern :)

Cheers,
Pascal.
--

DHH

unread,
Sep 20, 2007, 7:14:44 PM9/20/07
to Ruby on Rails: Core
> If every plugin's lib was in the load path *before* any plugin could
> start playing around or monkeypatching, many situations where you
> might want to explicitly ensure a plugin was loaded can be made to go
> away.

Can you look into writing a patch for that?

For the scenarios where you do need some load order, I've requested
http://dev.rubyonrails.org/ticket/9613 -- it takes just enough pain
out of config.plugins to make it bearable, but still leaves enough in
that people won't go crazy with dependency overload. Some one PDI this
likely-just-a-few-lines patch.

Frederick Cheung

unread,
Sep 20, 2007, 7:22:03 PM9/20/07
to rubyonra...@googlegroups.com
On 21 Sep 2007, at 00:14, DHH wrote:
>
> For the scenarios where you do need some load order, I've requested
> http://dev.rubyonrails.org/ticket/9613 -- it takes just enough pain
> out of config.plugins to make it bearable, but still leaves enough in
> that people won't go crazy with dependency overload. Some one PDI this
> likely-just-a-few-lines patch.
>
I've come up with an initial stab at 9613

Fred

Eric Anderson

unread,
Sep 20, 2007, 8:04:41 PM9/20/07
to rubyonra...@googlegroups.com
Pascal Belloncle wrote:
> I like the approach, a lot. But (isn't there always a but...), isn't
> this going to be a problem for people using shared hosting, or people
> that just can't install gems? Unless there is an easy way to locally
> install gems...

I'm on Site5 which is a cheap shared host. I can install gems perfectly
fine to a local directory. They have a few sitewide ones installed but I
have my own installed also. The only ones I might have problems with
(haven't tried) are ones that require native code.

Eric

Eric Anderson

unread,
Sep 20, 2007, 8:06:37 PM9/20/07
to rubyonra...@googlegroups.com
Josh Susser wrote:
> You can also freeze gems into the vendor directory. Now what's the
> downside compared to plugins in vendor?

It seems to me the versioning feature of gems solves that. Just require
the gem with a specific version requested. You are now frozen on that
version. Updates to the package don't affect you and you can move up to
new versions just by specifying what version you want.

Eric

Aaron Pfeifer

unread,
Sep 20, 2007, 10:56:51 PM9/20/07
to rubyonra...@googlegroups.com
Not to beat a dead horse, but I threw together a few ideas for working towards the plugins-as-gems approach.  Attached are two possible ways of going about it.  They are not, by any means, complete solutions (or completely tested solutions) but offer an insight at a possibility for plugins.  Both patches basically replace the current plugin loading mechanism with the gem system.  This way, we get support for versioning, the gem repository, dependencies, etc.

For both patches, existing plugins would:
1. No longer need an init.rb
2. Follow the standard gem structure (i.e. my_plugin/lib/my_plugin.rb)
3. Need to create a Rakefile which defines the gem specification (which some already do)

gem_always.diff - This patch assumes that everything (plugins and gems) would exist under vendor/ (a true vendor everything).

gem_optional.diff - This patch would require config.plugin_loader to be set to Rails::Plugin::GemLoader in order to use the gem system for loading plugins.  This patch assumes that plugins and gems (except rails) would exist under vendor/plugins.

Essentially, plugins would now simply refer to gems installed locally in the application.  This would also mean that you wouldn't need to do anything special for loading gems which are normally stored under vendor/ or vendor/gems.

Anyway, what are everyone's thoughts?

On 9/19/07, Michael Koziarski <mic...@koziarski.com> wrote:
gem_always.diff
gem_optional.diff

James Adam

unread,
Sep 21, 2007, 4:12:31 AM9/21/07
to rubyonra...@googlegroups.com
On 9/21/07, DHH <david.he...@gmail.com> wrote:
>
> > If every plugin's lib was in the load path *before* any plugin could
> > start playing around or monkeypatching, many situations where you
> > might want to explicitly ensure a plugin was loaded can be made to go
> > away.
>
> Can you look into writing a patch for that?

Absolutely.

> For the scenarios where you do need some load order, I've requested
> http://dev.rubyonrails.org/ticket/9613 -- it takes just enough pain
> out of config.plugins to make it bearable, but still leaves enough in
> that people won't go crazy with dependency overload. Some one PDI this
> likely-just-a-few-lines patch.

This is the mechanism I stuck with myself, despite requests to add
some kind of dependency management via the engines plugin. This is a
situation where the 80/20 (99/1?) rule can easily be applied.

James Adam

unread,
Oct 5, 2007, 11:57:38 AM10/5/07
to rubyonra...@googlegroups.com
On 9/21/07, James Adam <james...@gmail.com> wrote:
> On 9/21/07, DHH <david.he...@gmail.com> wrote:
> >
> > > If every plugin's lib was in the load path *before* any plugin could
> > > start playing around or monkeypatching, many situations where you
> > > might want to explicitly ensure a plugin was loaded can be made to go
> > > away.
> >
> > Can you look into writing a patch for that?
>
> Absolutely.
>

I've just created two (independent) patches to address this issue,
which you guys can now see on Trac. Both apply cleanly on edge as of
10 minutes ago, though not at the same time, of course.

The first - http://dev.rubyonrails.org/ticket/9793 - does the job with
the minimal amount of change to the existing code, and would be great
to apply, but still leaves me feeling like more could be done. Hence:

The second - http://dev.rubyonrails.org/ticket/9795 - which also does
the job, but also refactors the plugin locator/loader relationship to
be (hopefully) clearer, more testable and easier to develop in the
future. It's quite a bit bigger, but that is only because it is mostly
refactoring (which always inflates a diff) and testing, rather than
significant new behaviour.

While I'm aware that getting feedback on this larger patch is going to
be more effort on your part, once you've applied it to a clean edge
checkout, browsing around the code in railties/lib/rails/ should be
clear enough.

Feedback greatly appreciated!

Cheers,

Nathaniel Brown

unread,
Oct 6, 2007, 2:28:39 PM10/6/07
to rubyonra...@googlegroups.com
What about looking at a script/gem install which installs the gem into the load path of a vendor/gems/ directory? This could provide the dependency requirements by leveraging what's existing.

One thing that comes to mind which the config.plugins does that a gem won't neccessarly do, is the load order of them.

I am strongly for the idea of using Gems instead of plugins. This has always been a wish of mine.
--
Nathaniel Steven Henry Brown
604-724-6624

ara.t.howard

unread,
Oct 9, 2007, 1:23:00 AM10/9/07
to First
On Oct 6, 2007, at 12:28 PM, Nathaniel Brown wrote:

What about looking at a script/gem install which installs the gem into the load path of a vendor/gems/ directory? This could provide the dependency requirements by leveraging what's existing.

it's been written, submitted, and ignored for quite a while (like > 3 years).  here it is:

#!/usr/bin/env ruby

require 'rubygems'
Gem.manage_gems

required_version = Gem::Version::Requirement.new(">= 1.8.0")
unless  required_version.satisfied_by?(Gem::Version.new(RUBY_VERSION))
  puts "Expected Ruby Version #{required_version}, was #{RUBY_VERSION}"
  exit(1)
end

# We need to preserve the original ARGV to use for passing gem options
# to source gems.  If there is a -- in the line, strip all options after
# it...its for the source building process.
args = !ARGV.include?("--") ? ARGV.clone : ARGV[0...ARGV.index("--")]

#
# munge rbconfig from any command line or environment kv pair <<-------- this is the important bit!
#
kvs = []
re = %r/^ \s* ([^=\s]+) \s* = \s* ([^\s]+) \s* $/x
args.delete_if{|arg| (m = re.match(arg)) and (kvs << m.to_a.last(2))}
ENV::each{|k,v| (k =~ %r/^GEM_/) and (kvs << [k.delete('GEM_'), v])}
unless kvs.empty?
  require 'rbconfig'
  kvs.each{|k,v| Config::CONFIG[k] = v}
end

Gem::GemRunner.new.run(args)

this allows one to

  gem install somegem libdir=./vender/gems/

in other words fine grained control of every aspect of rbconfig.  we've been using it for years on our production systems to install into non-standard locations

supposedly it was getting committed but it always seems to get lost in the cracks.  in any case a patch to rubygems is, imho, a saner way to go about the process


One thing that comes to mind which the config.plugins does that a gem won't neccessarly do, is the load order of them.


but it doesn't need to?  gems already does that automatically, as does ruby.  if you require 'a' and it requires 'b' which requires 'c' and all of them are in the load path then, whammo, you have constructed a dag and you are in business - so long as the load path is set correctly before hand - aka GEM_HOME



I am strongly for the idea of using Gems instead of plugins. This has always been a wish of mine.

agreed.  we don't use any plugins in our code for precisely the reason that gems, though not perfect, is light years ahead of plugins for code management.  building on top of this functionality is clearly the path of least resistance.

on a side note, i'm very surprised that no one in this thread has mentioned tsort.rb - it's part of the stdlib and does dependancy (dag) tree construction, including detecting cycles, in a few methods calls.

i know eric is working very hard on gems now - maybe a few patches and his momentum would solve this in short order.

regards.

--
we can deny everything, except that we have the possibility of being better. simply reflect on that.
h.h. the 14th dalai lama



Michael Koziarski

unread,
Oct 9, 2007, 1:35:19 AM10/9/07
to rubyonra...@googlegroups.com
> it's been written, submitted, and ignored for quite a while (like > 3
> years). here it is:

http://dev.rubyonrails.org/attachment/ticket/8511/gems_loaded_from_vendor_like_plugins.2.diff

That's where the effort's at at present. Eric's indicated that the
current implementation relies on some rubygems internals and has
offered to update it. However his plate's a bit full at present
getting gems ready for 1.9.

If someone really wants to see this in, that's the place to start,
and chad fowler and eric may have some spare time to help out.


--
Cheers

Koz

ara.t.howard

unread,
Oct 9, 2007, 1:52:44 AM10/9/07
to rubyonra...@googlegroups.com

On Oct 8, 2007, at 11:35 PM, Michael Koziarski wrote:

>
>> it's been written, submitted, and ignored for quite a while (like > 3
>> years). here it is:
>
> http://dev.rubyonrails.org/attachment/ticket/8511/
> gems_loaded_from_vendor_like_plugins.2.diff
>
> That's where the effort's at at present. Eric's indicated that the
> current implementation relies on some rubygems internals and has
> offered to update it. However his plate's a bit full at present
> getting gems ready for 1.9.

this is just loading afaikt? has anyone addressed the install issue
- eg into vendor/gems - is the current thinking still to do this via
custom ./script/xxx ?

Michael Koziarski

unread,
Oct 9, 2007, 3:09:56 AM10/9/07
to rubyonra...@googlegroups.com
> this is just loading afaikt? has anyone addressed the install issue
> - eg into vendor/gems - is the current thinking still to do this via
> custom ./script/xxx ?
>
> regards.
>
> a @ http://codeforpeople.com/

I'm not sure what issues remain beyond the existing --install-dir
option to gem install?

gem install --install-dir=~/myapp/vendor tzinfo

?

--
Cheers

Koz

Eric Hodel

unread,
Oct 9, 2007, 4:07:49 AM10/9/07
to First, ara.t.howard
On Oct 8, 2007, at 22:23 , ara.t.howard wrote:
> On Oct 6, 2007, at 12:28 PM, Nathaniel Brown wrote:
>> What about looking at a script/gem install which installs the gem
>> into the load path of a vendor/gems/ directory? This could provide
>> the dependency requirements by leveraging what's existing.
>
> it's been written, submitted, and ignored for quite a while (like >
> 3 years). here it is:
>
> [...]

You shouldn't need this anymore. With trunk (beta this week) RubyGems:

$ rm -rf ~/mygems
$ GEM_PATH=~/mygems ruby -rubygems -e "require 'dike'"
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
`gem_original_require': no such file to load -- dike (LoadError)
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/
custom_require.rb:27:in `require'
from -e:1
$ gem install dike --install-dir ~/mygems
Installing gem pervasives-1.1.0
Downloading gem pervasives-1.1.0.gem
Installing gem attributes-3.7.0
Downloading gem attributes-3.7.0.gem
Installing gem orderedhash-0.0.2
Downloading gem orderedhash-0.0.2.gem
Installing gem dike-0.0.3
Downloading gem dike-0.0.3.gem
Successfully installed pervasives-1.1.0
Successfully installed attributes-3.7.0
Successfully installed orderedhash-0.0.2
Successfully installed dike-0.0.3
4 gems installed
$ GEM_PATH=~/mygems ruby -rubygems -e "require 'dike'"
$ GEM_PATH=~/mygems ruby -rubygems -e "require 'zentest'"

zentest was loaded from GEM_HOME. GEM_PATH stacks with it.

>> One thing that comes to mind which the config.plugins does that a
>> gem won't neccessarly do, is the load order of them.
>
> but it doesn't need to? gems already does that automatically, as
> does ruby. if you require 'a' and it requires 'b' which requires
> 'c' and all of them are in the load path then, whammo, you have
> constructed a dag and you are in business - so long as the load
> path is set correctly before hand - aka GEM_HOME

You probably don't want to mess with GEM_HOME. GEM_PATH is much
better since you can have your own set of gems and the system set, or
even multiple sets (system, user, rails) if you're crazy like that.

>> I am strongly for the idea of using Gems instead of plugins. This
>> has always been a wish of mine.
>
> agreed. we don't use any plugins in our code for precisely the
> reason that gems, though not perfect, is light years ahead of
> plugins for code management. building on top of this functionality
> is clearly the path of least resistance.
>
> on a side note, i'm very surprised that no one in this thread has
> mentioned tsort.rb - it's part of the stdlib and does dependancy
> (dag) tree construction, including detecting cycles, in a few
> methods calls.

See Gem::DependencyList#dependency_order. Its what
Gem::DependencyInstaller uses to figure out how to install things.
See above.

Of course, you should just wait and use Gem::DependencyInstaller.
I'm confident it and the rest of RubyGems does everything you could
possibly want.

It installs from cache:

$ rm -rf ~/mygems/gems/ ~/mygems/specifications/
$ ls ~/mygems/cache/
attributes-3.7.0.gem orderedhash-0.0.2.gem
dike-0.0.3.gem pervasives-1.1.0.gem
$ gem install dike --install-dir ~/mygems
Installing gem pervasives-1.1.0
Installing gem attributes-3.7.0
Installing gem orderedhash-0.0.2
Installing gem dike-0.0.3
Successfully installed pervasives-1.1.0
Successfully installed attributes-3.7.0
Successfully installed orderedhash-0.0.2
Successfully installed dike-0.0.3
4 gems installed
$ GEM_PATH=~/mygems/ ruby -rubygems -e "require 'dike'"

Note that the downloading was skipped.

And lets you fetch gems:

$ gem fetch rake
Downloaded rake-0.7.3.gem
$ mv rake-0.7.3.gem ~/mygems/cache/
$ gem install rake --install-dir ~/mygems
Installing gem rake-0.7.3
Successfully installed rake-0.7.3
1 gem installed
Installing ri documentation for rake-0.7.3...
Installing RDoc documentation for rake-0.7.3...

> i know eric is working very hard on gems now - maybe a few patches
> and his momentum would solve this in short order.

Actually, I'm trying not to work hard. I'm doing the minimum
possible cleanup necessary to get a RubyGems beta out the door this
week.

PS: Don't use Gem::GemRunner to invoke gem stuff. Go in through the
proper interfaces. There's too many moving parts in Gem::GemRunner,
so I can't promise its stability.

--
Poor workers blame their tools. Good workers build better tools. The
best workers get their tools to do the work for them. -- Syndicate Wars


Eric Hodel

unread,
Oct 9, 2007, 4:13:32 AM10/9/07
to Ruby on Rails: Core
On Oct 8, 10:35 pm, "Michael Koziarski" <mich...@koziarski.com> wrote:
> > it's been written, submitted, and ignored for quite a while (like > 3
> > years). here it is:
>
> http://dev.rubyonrails.org/attachment/ticket/8511/gems_loaded_from_ve...

>
> That's where the effort's at at present. Eric's indicated that the
> current implementation relies on some rubygems internals and has
> offered to update it. However his plate's a bit full at present
> getting gems ready for 1.9.

I don't recall offering to update it, and I won't have the chance to
do so until at least two weeks after RubyConf, probably longer. My
other projects have been neglected for too long with all my work on
RubyGems.

> If someone really wants to see this in, that's the place to start,
> and chad fowler and eric may have some spare time to help out.

I can answer questions, they're best directed to the RubyGems
developers list: rubygems-...@rubyforge.org

ara.t.howard

unread,
Oct 9, 2007, 9:54:05 AM10/9/07
to Eric Hodel, First

On Oct 9, 2007, at 2:07 AM, Eric Hodel wrote:

> You shouldn't need this anymore. With trunk (beta this week)
> RubyGems:
>

<snip crapload of excellent stuff>

eric - keep up the seriously great work. it's really appreciated.

cheers.

it is not enough to be compassionate. you must act.

James Adam

unread,
Oct 9, 2007, 7:01:30 PM10/9/07
to rubyonra...@googlegroups.com
While I appreciate that the "gems as plugins" debate is valuable, can
we can get a tiny bit more feedback on Tickets 9793 and 9795, which
hopefully provides a reasonable solution to the "plugin dependencies"
issue within the current plugins philosophy?

http://dev.rubyonrails.org/ticket/9793
http://dev.rubyonrails.org/ticket/9795

Cheers guys

- James

Michael Koziarski

unread,
Oct 12, 2007, 11:39:56 PM10/12/07
to rubyonra...@googlegroups.com
> http://dev.rubyonrails.org/ticket/9793
> http://dev.rubyonrails.org/ticket/9795


Of these two 9793 is definitely something I'm more comfortable with,
but I'm not sure the problems we're setting out to solve?

Can you give me an example of a plugin that is hard to implement
before this patch goes in?


--
Cheers

Koz

James Adam

unread,
Oct 14, 2007, 1:31:03 PM10/14/07
to rubyonra...@googlegroups.com
On 10/12/07, Michael Koziarski <mic...@koziarski.com> wrote:
> Of these two 9793 is definitely something I'm more comfortable with,
> but I'm not sure the problems we're setting out to solve?

The patches have one major aim, and that is to side-step the need for any
"plugin dependency" mechanism. So, for those developers who need to
work with classes defined in another plugin, it becomes less necessary
for them to have to carefully manage load order.

So, to directly answre your question: the *problem* is people asking
for a plugin dependency management mechanism, and the *solution* is to
change the way that plugins are managed in Rails to remove that need
for the large majority of developers.

The second patch also aims to solve this problem, but additionally
refactors the plugin loading mechanism into a clearer set of classes
and objects, to make it more testable, flexible and easier to enhance
in the future.

As I mentioned in the email and the ticket notes, because #9793 is
*far* more conservative, I would expect it to be the preference of
Rails Core. However, I can't help but feel that eventually (as more
things move out from Rails into plugins), plugin loading code that is
clear, more obvious and transparent will only make that easier. That
was the aim of #9795.

Again, I understand that it's the nature of the core committers to
apply only the smallest patch to get the desired behaviour, but I
really think that it's worth expending a bit more effort to get this
key aspect of Rails right sooner rather than later - be that via
#9795, or some other mechanism to make the plugin loading mechanism
clearer and more obvious. I'm happy to discuss this further via
IRC/Campfire/Email :)

Hope that explains it,

- james

Reply all
Reply to author
Forward
0 new messages