Plugins in modules with environments and the puppetmaster libdir

[Nigel Kersten]

Unless I'm missing something, we kind of have a fatal flaw with using
plugins in modules with environments when it comes to making those
same plugins available to the puppetmaster itself.

If you're deploying a custom type/provider with the plugins in modules
model, you still need to make these plugins discoverable in the
puppetmaster libdir.

ie, if I'm deploying:
<modulepath>/base/lib/puppet/type/foo.rb
<modulepath>/base/lib/puppet/provider/foo.rb

Then I need to ensure that these plugins are available at:
$libdir/puppet/{type,provider}/foo.rb
on the puppetmaster itself.

The problem with this is that it makes it difficult to release
different versions of the same plugin to different clients, as you
have to pick one version and one version only of the plugin to be in
the puppetmaster libdir, and this version may conflict with the
version you're supplying to clients in terms of valid syntax and
acceptable parameters.

How are other people dealing with this?

--
nigel

[Matthew Hyclak]

We created a module called plugins and put our code there. For example, we have:

modules/plugins/files/facter/hugepagesize.rb

That gets replicated across environments (and we can have different
hugepagesize.rb files per environment if we need to).

Matt

[Nigel Kersten]

So facter plugins are kind of different, as they're not actually
required to be in the puppetmaster libdir.

Say this was a type/provider, and you wanted to add a new parameter,
but only roll it out to your testing environments.

What do you do then?

If the version in the puppetmaster libdir doesn't accept that
parameter, the manifest compilation will fail for clients who *are*
getting a version of the plugin that supports that parameter.

>
> Matt
>
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>
>

--
nigel

[Thomas Bellman]

Nigel Kersten wrote:

> So facter plugins are kind of different, as they're not actually
> required to be in the puppetmaster libdir.
>
> Say this was a type/provider, and you wanted to add a new parameter,
> but only roll it out to your testing environments.

Functions also have this limitation, by the way.

> What do you do then?
>
> If the version in the puppetmaster libdir doesn't accept that
> parameter, the manifest compilation will fail for clients who *are*
> getting a version of the plugin that supports that parameter.

You lose...

Well, there are a couple of things you can do to work around this
limitation. For one thing, you could run a second puppetmaster
process, perhaps on another port, or listening on another IP address,
or perhaps even on another machine.

Another thing you can do, is to run puppet with local manifests,
instead of puppetd connecting to puppetmasterd, when developing
the new version of the plugin. That's what I do. It does get a
bit iffy if your manifests want to fetch files from the puppetmaster
(i.e. that aren't in the "modules" namespace) though, or otherwise
need to access files that are only available on the puppetmaster.

Good news, though, is that the upcoming "Rowlf" version of Puppet is
supposed to solve this for at least resource types. At least Luke has
posted patches to the devel list that I gather will do that. But it
will probably still be a problem for custom functions (I have somewhat
volunteered to take a look at it, but I'm unlikely to find the time
before Rowlf is supposed to be out).

/Bellman

[Nigel Kersten]

+ puppet-dev

Luke, how is this going to be solved in practice? The puppetmasterd
will automatically add modules/*/lib dirs to it's own libdir in the
context/environment of the current client?

I'm thinking that in the meantime I may need to encode plugin versions
in their names, so when a client's manifest contains a given plugin,
it's always going to refer to that version, and I script something
that collates all the lib/... directories into the puppetmasterd
libdir.

I feel dirty.

>
>
>        /Bellman

[Luke Kanies]

Hmm, not quite true yet.

At this point, rowlf has a bunch of refactoring around just the
language stuff, not the pure-ruby stuff.

We're on the path to converging the language resource types and the
pure ruby Puppet::Type classes, but we're still a ways away. Part of
that convergence will be fixing this problem, but I don't expect to
see a real long-term fix until at least the release after rowlf -
mostly just because we don't want to delay rowlf too much further.

> I'm thinking that in the meantime I may need to encode plugin versions
> in their names, so when a client's manifest contains a given plugin,
> it's always going to refer to that version, and I script something
> that collates all the lib/... directories into the puppetmasterd
> libdir.
>
> I feel dirty.

Yeah, that's ugly. I think it'd be way easier to run environment-
specific masters on a different port or server, wouldn't it?

BTW, if this is a big priority to someone, I think it's approachable
today - far more than six months ago, with recent refactoring - and
I'm happy to work with someone who's committed to getting it fixed.

And I know this has been coming up a lot recently, so it's getting
pushed up on my priority list.

--
Men never do evil so completely and cheerfully as when they do it from a
religious conviction. --Blaise Pascal
---------------------------------------------------------------------
Luke Kanies -|- http://reductivelabs.com -|- +1(615)594-8199

[Nigel Kersten]

Can you give us an overview of the way this is intended to work? 

 

I'm thinking that in the meantime I may need to encode plugin versions
in their names, so when a client's manifest contains a given plugin,
it's always going to refer to that version, and I script something
that collates all the lib/... directories into the puppetmasterd
libdir.

I feel dirty.

Yeah, that's ugly.  I think it'd be way easier to run environment-specific masters on a different port or server, wouldn't it?

Not for the number of environments we have... 

 

BTW, if this is a big priority to someone, I think it's approachable today - far more than six months ago, with recent refactoring - and I'm happy to work with someone who's committed to getting it fixed.

And I know this has been coming up a lot recently, so it's getting pushed up on my priority list.

What would you need from those of us who are interested in working on this with you Luke?

 

--
Men never do evil so completely and cheerfully as when they do it from a
religious conviction. --Blaise Pascal
---------------------------------------------------------------------
Luke Kanies  -|-   http://reductivelabs.com   -|-   +1(615)594-8199

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To post to this group, send email to puppe...@googlegroups.com.
To unsubscribe from this group, send email to puppet-dev+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.

--
nigel

[Luke Kanies]

Not sure exactly how to answer this question, since there's a lot there.

The convergence of the two classes (Puppet::Type and Puppet::Resource::Type) is complex to describe.  Basically, current resource types are subclasses of Puppet::Type, and resources are instances of those subclasses.  We'll be changing things so that resource types are instances of Puppet::Resource::Type, and resources are instances of Puppet::Resource.  In terms of observable differences, other than the constants involved there won't really be any - it'll just make contribution, maintenance, and custom development much easier.

However, it'll also enable us to do a lot of things we currently can't, like maintain non-global (i.e., environment-specific) resource types.  The refactoring I did to enable the pure-ruby DSL was the main step toward this:  When you use 'define' in the language, you're creating an environment-specific resource type, it's just not as functional as a builtin resource type.  I think the next major release will normalize the functionality between the two, and hopefully also make it possible to use existing providers with new-style resource types.  Then it's just a question of porting over the existing types to use the new class structure.

I'm thinking that in the meantime I may need to encode plugin versions
in their names, so when a client's manifest contains a given plugin,
it's always going to refer to that version, and I script something
that collates all the lib/... directories into the puppetmasterd
libdir.

I feel dirty.

Yeah, that's ugly.  I think it'd be way easier to run environment-specific masters on a different port or server, wouldn't it?

Not for the number of environments we have... 

Ah.

 

BTW, if this is a big priority to someone, I think it's approachable today - far more than six months ago, with recent refactoring - and I'm happy to work with someone who's committed to getting it fixed.

And I know this has been coming up a lot recently, so it's getting pushed up on my priority list.

What would you need from those of us who are interested in working on this with you Luke?

Code? :)

-- 

Finn's Law:
    Uncertainty is the final test of innovation.