Plugins and Environments in 0.25.x - is there any solution?

20 views
Skip to first unread message

Nigel Kersten

unread,
Aug 14, 2009, 7:39:07 PM8/14/09
to puppe...@googlegroups.com
To make it clearer, my primary concern is per-environment facts (and thus plugins)


So here's how things stand as far as I can see:

* factsyncing is deprecated for 0.25.x

* plugins in modules with environments won't be working for 0.25.x so we can't distribute facts that way

* We can no longer create a 'plugins' module per-environment and serve out of plugins/files/ as it always falls back to the autogenerated mount point

* If we try to override this by defining a path for the plugins mount, Puppet ignores your attempt to set it.


The only workaround I have right now to allow you to have separate facts per environment is to create another module, say 'custom' and store all your plugins there, so a fact would be at:

custom/files/facter/foo.rb

and then the clients have:

--pluginsync
--factpath $vardir/facter
--pluginsource puppet://$server/custom


This is pretty crappy. I missed watching the relevant bug, so didn't notice Luke bumping it to 0.26.0

http://projects.reductivelabs.com/issues/1175


Am I missing something?



--
Nigel Kersten
nig...@google.com
System Administrator
Google, Inc.

Luke Kanies

unread,
Aug 14, 2009, 8:10:07 PM8/14/09
to puppe...@googlegroups.com
On Aug 14, 2009, at 4:39 PM, Nigel Kersten wrote:

> To make it clearer, my primary concern is per-environment facts (and
> thus plugins)
>
>
> So here's how things stand as far as I can see:
>
> * factsyncing is deprecated for 0.25.x
>
> * plugins in modules with environments won't be working for 0.25.x
> so we can't distribute facts that way

Plugin distribution per-environment should work just fine in 0.25, and
as far as I've ever heard it does.

The only thing that doesn't work per-environment, AFAIK, is anything
that loads into memory in the puppetmaster, which means functions,
resource types, and providers.

>
> * We can no longer create a 'plugins' module per-environment and
> serve out of plugins/files/ as it always falls back to the
> autogenerated mount point
>
> * If we try to override this by defining a path for the plugins
> mount, Puppet ignores your attempt to set it.

These two seem to be a similar problem and definitely what I would
call a bug.

>
> The only workaround I have right now to allow you to have separate
> facts per environment is to create another module, say 'custom' and
> store all your plugins there, so a fact would be at:
>
> custom/files/facter/foo.rb
>
> and then the clients have:
>
> --pluginsync
> --factpath $vardir/facter
> --pluginsource puppet://$server/custom
>
>
> This is pretty crappy. I missed watching the relevant bug, so didn't
> notice Luke bumping it to 0.26.0
>
> http://projects.reductivelabs.com/issues/1175
>
>
> Am I missing something?

A bunch of bug reports?

--
I am not young enough to know everything. -- Oscar Wilde
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com

Nigel Kersten

unread,
Aug 14, 2009, 8:14:39 PM8/14/09
to puppe...@googlegroups.com
On Fri, Aug 14, 2009 at 5:10 PM, Luke Kanies <lu...@madstop.com> wrote:

On Aug 14, 2009, at 4:39 PM, Nigel Kersten wrote:

> To make it clearer, my primary concern is per-environment facts (and
> thus plugins)
>
>
> So here's how things stand as far as I can see:
>
> * factsyncing is deprecated for 0.25.x
>
> * plugins in modules with environments won't be working for 0.25.x
> so we can't distribute facts that way

Plugin distribution per-environment should work just fine in 0.25, and
as far as I've ever heard it does.

The only thing that doesn't work per-environment, AFAIK, is anything
that loads into memory in the puppetmaster, which means functions,
resource types, and providers.

Maybe I'm doing something wrong then.
Can you describe how you think this should work so I can put in appropriate bug reports?

How do we define the source for plugins on a per-environment basis if we're not doing it in modules?

 



>
> * We can no longer create a 'plugins' module per-environment and
> serve out of plugins/files/ as it always falls back to the
> autogenerated mount point
>
> * If we try to override this by defining a path for the plugins
> mount, Puppet ignores your attempt to set it.

These two seem to be a similar problem and definitely what I would
call a bug.

>
> The only workaround I have right now to allow you to have separate
> facts per environment is to create another module, say 'custom' and
> store all your plugins there, so a fact would be at:
>
> custom/files/facter/foo.rb
>
> and then the clients have:
>
> --pluginsync
> --factpath $vardir/facter
> --pluginsource puppet://$server/custom
>
>
> This is pretty crappy. I missed watching the relevant bug, so didn't
> notice Luke bumping it to 0.26.0
>
> http://projects.reductivelabs.com/issues/1175
>
>
> Am I missing something?

A bunch of bug reports?

--
I am not young enough to know everything. -- Oscar Wilde
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com



Luke Kanies

unread,
Aug 14, 2009, 8:31:56 PM8/14/09
to puppe...@googlegroups.com
On Aug 14, 2009, at 5:14 PM, Nigel Kersten wrote:

>
>
> On Fri, Aug 14, 2009 at 5:10 PM, Luke Kanies <lu...@madstop.com> wrote:
>
> On Aug 14, 2009, at 4:39 PM, Nigel Kersten wrote:
>
> > To make it clearer, my primary concern is per-environment facts (and
> > thus plugins)
> >
> >
> > So here's how things stand as far as I can see:
> >
> > * factsyncing is deprecated for 0.25.x
> >
> > * plugins in modules with environments won't be working for 0.25.x
> > so we can't distribute facts that way
>
> Plugin distribution per-environment should work just fine in 0.25, and
> as far as I've ever heard it does.
>
> The only thing that doesn't work per-environment, AFAIK, is anything
> that loads into memory in the puppetmaster, which means functions,
> resource types, and providers.
>
> Maybe I'm doing something wrong then.
> Can you describe how you think this should work so I can put in
> appropriate bug reports?
>
> How do we define the source for plugins on a per-environment basis
> if we're not doing it in modules?

As an aside, can you ask whoever's in charge of Gmail to fix inline
responses so that a response to a response actually shows the
indentation? Gmail + mail.app == total failure.

Anyway...

You *should* use modules, and they should work right now. Have you
tested per-environment plugin downloading and verified it doesn't
work? Because it should.
--
'Tis better to be silent and be thought a fool, than to speak and
remove all doubt. --Abraham Lincoln

Nigel Kersten

unread,
Aug 14, 2009, 9:38:53 PM8/14/09
to puppe...@googlegroups.com
On Fri, Aug 14, 2009 at 5:31 PM, Luke Kanies <lu...@madstop.com> wrote:
>
> On Aug 14, 2009, at 5:14 PM, Nigel Kersten wrote:
>
> >
> >
> > On Fri, Aug 14, 2009 at 5:10 PM, Luke Kanies <lu...@madstop.com> wrote:
> >
> > On Aug 14, 2009, at 4:39 PM, Nigel Kersten wrote:
> >
> > > To make it clearer, my primary concern is per-environment facts (and
> > > thus plugins)
> > >
> > >
> > > So here's how things stand as far as I can see:
> > >
> > > * factsyncing is deprecated for 0.25.x
> > >
> > > * plugins in modules with environments won't be working for 0.25.x
> > > so we can't distribute facts that way
> >
> > Plugin distribution per-environment should work just fine in 0.25, and
> > as far as I've ever heard it does.
> >
> > The only thing that doesn't work per-environment, AFAIK, is anything
> > that loads into memory in the puppetmaster, which means functions,
> > resource types, and providers.
> >
> > Maybe I'm doing something wrong then.
> > Can you describe how you think this should work so I can put in
> > appropriate bug reports?
> >
> > How do we define the source for plugins on a per-environment basis
> > if we're not doing it in modules?
>
> As an aside, can you ask whoever's in charge of Gmail to fix inline
> responses so that a response to a response actually shows the
> indentation?  Gmail + mail.app == total failure.

I'll see what I can do :) Composing in plain text tends to help a lot.

>
> Anyway...
>
> You *should* use modules, and they should work right now.  Have you
> tested per-environment plugin downloading and verified it doesn't
> work?  Because it should.

So I'm confused then. Isn't this related to functions in modules that
has been bumped to 0.26.0 ?

I have tested plugins in modules with 0.25.x and they don't work with
this puppet.conf:

[main]
  confdir=/etc/puppet
  logdir=/var/log/puppet
  vardir=/var/lib/puppet
  ssldir=$vardir/ssl

[puppetmasterd]
  ca=false
  #servertype=mongrel
  logdest=syslog
  logdir=/var/log/puppet
  vardir=/var/lib/puppet
  certname=puppetmaster.corp.google.com
  autosign=false
  ssl_client_header=HTTP_X_SSL_SUBJECT

[first]
  modulepath=/var/lib/puppet/environments/first/modules
  manifestdir=/var/lib/puppet/environments/first/manifests
  manifest=/var/lib/puppet/environments/first/manifests/site.pp

[second]
  modulepath=/var/lib/puppet/environments/second/modules
  manifestdir=/var/lib/puppet/environments/second/manifests
  manifest=/var/lib/puppet/environments/first/manifests/site.pp


changes to fileserver.conf don't make any difference as far as I can see.

and then setting up the module 'base' in
/var/lib/puppet/environments/first/modules as:

base/manifests/init.pp
base/plugins/facter/foo.rb


server-side:

notice: Starting Puppet server version 0.25.0
debug: No modules mount given; autocreating with default permissions
debug: No plugins mount given; autocreating with default permissions
debug: Creating interpreter
debug: Finishing transaction 69971990338980 with 0 changes
debug: Finishing transaction 69971990335220 with 0 changes


client-side:

# puppetd -t --environment first --pluginsync --factpath
/var/puppet/lib/facter --debug --trace
... snip ...
err: /File[/var/puppet/lib]: Failed to retrieve current state of
resource: Could not retrieve information from source(s)
puppet://myserver.mydomain/plugins
debug: Finishing transaction 14077460 with 0 changes
debug: Puppet::Network::Format[json]: false value when expecting true
debug: Format s not supported for Puppet::Resource::Catalog; has not
implemented method 'from_s'
info: Caching catalog for c216f41a-f902-4bfb-a222-850dd957bebb
debug: Loaded state in 0.01 seconds
info: Applying configuration version '1250300100'
debug: Finishing transaction 2658790 with 0 changes
debug: Storing state
debug: Stored state in 0.12 seconds
notice: Finished catalog run in 0.13 seconds


I'm more than happy to work on getting this fixed. It's just been
unclear to me what is the expected behaviour in 0.25.x in this area.

Luke Kanies

unread,
Aug 16, 2009, 8:19:40 PM8/16/09
to puppe...@googlegroups.com
Expected behaviour is this:

* plugins should correctly download from specific environments (and I
could swear I tested this and they did)
* plugins loaded into memory on the server (e.g., functions) do not
correctly work in environments

Is that clear enough? Are the other behaviours unspecified here?

If you can spend time on this, great, otherwise Markus or I can try to
track it down.
--
Hollywood is a place where they'll pay you a thousand dollars for a
kiss and fifty cents for your soul. -- Marilyn Monroe

Nigel Kersten

unread,
Aug 17, 2009, 11:12:04 AM8/17/09
to puppe...@googlegroups.com
On Sun, Aug 16, 2009 at 5:19 PM, Luke Kanies<lu...@madstop.com> wrote:
> On Aug 14, 2009, at 6:38 PM, Nigel Kersten wrote:
>> I'm more than happy to work on getting this fixed. It's just been
>> unclear to me what is the expected behaviour in 0.25.x in this area.
>
> Expected behaviour is this:
>
> * plugins should correctly download from specific environments (and I
> could swear I tested this and they did)

So to be absolutely clear, we should be able to have plugins in
modules in environments and have them work correctly?

I just don't see any other way we get to specify plugin sources on a
per-environment basis without using $environment in $pluginsource on
the clients.

> * plugins loaded into memory on the server (e.g., functions) do not
> correctly work in environments
>
> Is that clear enough?  Are the other behaviours unspecified here?
>
> If you can spend time on this, great, otherwise Markus or I can try to
> track it down.

looking today.

Luke Kanies

unread,
Aug 17, 2009, 11:31:11 AM8/17/09
to puppe...@googlegroups.com
On Aug 17, 2009, at 8:12 AM, Nigel Kersten wrote:

>
> On Sun, Aug 16, 2009 at 5:19 PM, Luke Kanies<lu...@madstop.com> wrote:
>> On Aug 14, 2009, at 6:38 PM, Nigel Kersten wrote:
>>> I'm more than happy to work on getting this fixed. It's just been
>>> unclear to me what is the expected behaviour in 0.25.x in this area.
>>
>> Expected behaviour is this:
>>
>> * plugins should correctly download from specific environments (and I
>> could swear I tested this and they did)
>
> So to be absolutely clear, we should be able to have plugins in
> modules in environments and have them work correctly?

If by "work" you mean "download to the client and work there", then yes.

>
> I just don't see any other way we get to specify plugin sources on a
> per-environment basis without using $environment in $pluginsource on
> the clients.

All fileserving, including plugin downloading, includes the
environment in the URI; something like:

/$environment/$datatype/$thin

It's all handled internally by the REST support, you should never have
to specify it.

>
>> * plugins loaded into memory on the server (e.g., functions) do not
>> correctly work in environments
>>
>> Is that clear enough? Are the other behaviours unspecified here?
>>
>> If you can spend time on this, great, otherwise Markus or I can try
>> to
>> track it down.
>
> looking today.

Awesome.

--
Everything that is really great and inspiring is created by the
individual who can labor in freedom. -- Albert Einstein

Reply all
Reply to author
Forward
0 new messages