Jira (PUP-8766) puppet device fails with undefined method `loaders' for nil:NilClass when env is not production

5 views
Skip to first unread message

Thomas Kishel (JIRA)

unread,
May 21, 2018, 2:10:04 PM5/21/18
to puppe...@googlegroups.com
Thomas Kishel created an issue
 
Puppet / Bug PUP-8766
puppet device fails with undefined method `loaders' for nil:NilClass when env is not production
Issue Type: Bug Bug
Affects Versions: PUP 5.5.1, PUP 5.3.1
Assignee: Unassigned
Created: 2018/05/21 11:09 AM
Priority: Normal Normal
Reporter: Thomas Kishel

The puppet device command fails with when environment is not 'production' and environment is specified locally in either puppet.conf or via the --environment option ...

[root@pe-201810-agent ~]# puppet device --verbose --environment=development
Error: Could not run: undefined method `loaders' for nil:NilClass

It succeeds if environment is server-specified ...

[root@pe-201810-agent ~]# puppet device --verbose #--environment=development
Info: starting applying configuration to cisco.example.com at file:///etc/puppetlabs/puppet/devices/cisco.example.com.yaml
Notice: Local environment: 'production' doesn't match server specified node environment 'development', switching agent to 'development'.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for cisco.example.com
Info: Applying configuration version '1526925344'
Notice: Applied catalog in 0.12 seconds

Note that even when the agent is not in production, only the production directory is created locally:

[root@pe-201810-agent ~]#  ls -al /etc/puppetlabs/code/environments/
total 0
drwxr-xr-x 3 root root 23 May 21 17:28 .
drwxr-xr-x 4 root root 39 May 21 17:12 ..
drwxr-x--- 5 root root 87 May 21 17:29 production

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Craig Gomes (JIRA)

unread,
May 21, 2018, 4:42:04 PM5/21/18
to puppe...@googlegroups.com

Thomas Kishel (JIRA)

unread,
May 21, 2018, 4:47:02 PM5/21/18
to puppe...@googlegroups.com

Thomas Kishel (JIRA)

unread,
May 21, 2018, 4:48:03 PM5/21/18
to puppe...@googlegroups.com

Rick Sherman (JIRA)

unread,
May 21, 2018, 4:52:02 PM5/21/18
to puppe...@googlegroups.com

Thomas Hallgren (JIRA)

unread,
May 21, 2018, 5:36:03 PM5/21/18
to puppe...@googlegroups.com

Please run the command with --trace and attach the stacktrace for the error.

Thomas Kishel (JIRA)

unread,
May 21, 2018, 6:31:03 PM5/21/18
to puppe...@googlegroups.com
Thomas Kishel commented on Bug PUP-8766

[root@pe-201810-agent ~]# puppet device --verbose --environment=development --trace
 
Error: Could not run: undefined method `loaders' for nil:NilClass
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loaders.rb:21:in `new'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/device.rb:230:in `main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:383:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:661:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:73:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'

Henrik Lindberg (JIRA)

unread,
May 22, 2018, 2:22:02 AM5/22/18
to puppe...@googlegroups.com

Martin Ewings (JIRA)

unread,
May 22, 2018, 4:07:02 AM5/22/18
to puppe...@googlegroups.com

Thomas Hallgren (JIRA)

unread,
May 22, 2018, 4:54:03 AM5/22/18
to puppe...@googlegroups.com
Thomas Hallgren commented on Bug PUP-8766
 
Re: puppet device fails with undefined method `loaders' for nil:NilClass when env is not production

The problem is that this line:

env = Puppet.lookup(:environments).get(Puppet[:environment])

results in env being nil. AFAICT, that can only happen when given the environment doesn't exist. Testing the command with --environment=<an environment that indeed exist> works OK. So I'm lowering the priority on this one.

Thomas Hallgren (JIRA)

unread,
May 22, 2018, 4:54:04 AM5/22/18
to puppe...@googlegroups.com

Martin Ewings (JIRA)

unread,
May 22, 2018, 6:47:03 AM5/22/18
to puppe...@googlegroups.com

Henrik Lindberg (JIRA)

unread,
May 22, 2018, 7:34:02 AM5/22/18
to puppe...@googlegroups.com
Henrik Lindberg commented on Bug PUP-8766
 
Re: puppet device fails with undefined method `loaders' for nil:NilClass when env is not production

Guessing that puppet device does not set up the environment the same way as the regular agent does. IIRC it creates something known as a "Remote Environment" (which is quite different from the compiling side's directory based environment).

Thomas Kishel (JIRA)

unread,
May 22, 2018, 11:23:04 AM5/22/18
to puppe...@googlegroups.com
Thomas Kishel commented on Bug PUP-8766

Running `puppet device` on a non-master proxy puppet agent is common, if not recommended. 

Simply adding an empty directory on the proxy agent avoids this error.

 

Some tests ...

 

With agent and device pinned to the development environment group in the Console ...

With environment=development in puppet.conf on agent ...

[root@pe-201810-agent ~]# puppet config print environment
development

[root@pe-201810-agent ~]# rm -rf /etc/puppetlabs/code

[root@pe-201810-agent ~]# puppet device --verbose
Error: Could not run: undefined method `loaders' for nil:NilClass

[root@pe-201810-agent ~]# puppet device --verbose --environment=development
Error: Could not run: undefined method `loaders' for nil:NilClass

[root@pe-201810-agent ~]# puppet device --verbose --environment=production
Info: starting applying configuration to cisco.example.com at file:///etc/puppetlabs/puppet/devices/cisco.example.com.yaml
Notice: Local environment: 'production' doesn't match server specified node environment 'development', switching agent to 'development'.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for cisco.example.com
Info: Applying configuration version '1527000970'
Notice: Applied catalog in 0.08 seconds

[root@pe-201810-agent ~]# mkdir -p /etc/puppetlabs/code/environments/development

[root@pe-201810-agent ~]# puppet device --verbose
Info: starting applying configuration to cisco.example.com at file:///etc/puppetlabs/puppet/devices/cisco.example.com.yaml
Info: Using configured environment 'development'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for cisco.example.com
Info: Applying configuration version '1527000983'
Notice: Applied catalog in 0.09 seconds

With environment=production in puppet.conf on agent ...

[root@pe-201810-agent ~]# puppet config print environment
production

[root@pe-201810-agent ~]# rm -rf /etc/puppetlabs/code

[root@pe-201810-agent ~]# puppet device --verbose 
Info: starting applying configuration to cisco.example.com at file:///etc/puppetlabs/puppet/devices/cisco.example.com.yaml
Notice: Local environment: 'production' doesn't match server specified node environment 'development', switching agent to 'development'.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for cisco.example.com
Info: Applying configuration version '1527001770'
Notice: Applied catalog in 0.09 seconds

[root@pe-201810-agent ~]# puppet device --verbose --environment=development
Error: Could not run: undefined method `loaders' for nil:NilClass

[root@pe-201810-agent ~]# puppet device --verbose --environment=production
Info: starting applying configuration to cisco.example.com at file:///etc/puppetlabs/puppet/devices/cisco.example.com.yaml
Notice: Local environment: 'production' doesn't match server specified node environment 'development', switching agent to 'development'.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for cisco.example.com
Info: Applying configuration version '1527001824'
Notice: Applied catalog in 0.10 seconds

[root@pe-201810-agent ~]# mkdir -p /etc/puppetlabs/code/environments/development

[root@pe-201810-agent ~]# puppet device --verbose --environment=development
Info: starting applying configuration to cisco.example.com at file:///etc/puppetlabs/puppet/devices/cisco.example.com.yaml
Info: Using configured environment 'development'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for cisco.example.com
Info: Applying configuration version '1527001901'
Notice: Applied catalog in 0.10 seconds

Adam Bottchen (JIRA)

unread,
May 22, 2018, 11:29:03 AM5/22/18
to puppe...@googlegroups.com

Thomas Kishel (JIRA)

unread,
May 22, 2018, 1:05:05 PM5/22/18
to puppe...@googlegroups.com
Thomas Kishel commented on Bug PUP-8766
 
Re: puppet device fails with undefined method `loaders' for nil:NilClass when env is not production

Charlie suggests something like what puppet agent uses:

env = Puppet::Node::Environment.remote(Puppet[:environment])

Or if the agent is compiling a catalog like puppet apply ...

env = if options[:apply].nil?
  Puppet::Node::Environment.remote(Puppet[:environment])
else
  Puppet.lookup(:environments).get(Puppet[:environment])
end

Thomas Mashos (JIRA)

unread,
Oct 19, 2018, 6:54:03 PM10/19/18
to puppe...@googlegroups.com
Thomas Mashos commented on Bug PUP-8766

I've got this same issue. I've found that I don't get that error if I set the environment in the ENC rather that in puppet.conf or in the command line.

David Mallon (JIRA)

unread,
Oct 24, 2018, 2:50:03 PM10/24/18
to puppe...@googlegroups.com

David Mallon (JIRA)

unread,
Oct 25, 2018, 5:47:03 AM10/25/18
to puppe...@googlegroups.com

David Schmitt (JIRA)

unread,
Oct 25, 2018, 6:16:04 AM10/25/18
to puppe...@googlegroups.com

David Schmitt (JIRA)

unread,
Oct 26, 2018, 5:00:04 AM10/26/18
to puppe...@googlegroups.com
David Schmitt updated an issue

I could reproduce this in a local & clean 5.3 environment. Exploring this functionality raised two concerning issues:

  • puppet device is loading plugin code from the proxy agent's cache.
  • puppet device is using the plugin code before it is downloading the plugins required to do so

On the one hand this makes it actually work, on the other hand it completely evades environment isolation and will cause confusion when used.

Charlie&Thomas' fix above avoids the issue of the loaders failure, but does not help with the loadpath issue.

Some background

loading from the agent's libdir

The device.rb#L253 changes the vardir setting, but this does not update the $LOAD_PATH. I've instrumented the code for some debug output:

david@zion:~/git/puppet$ bundle exec puppet device --verbose --trace
libdir: replacing '' with '/home/david/.puppetlabs/opt/puppet/cache/lib'
libdir: replacing '/home/david/.puppetlabs/opt/puppet/cache/lib' with '/home/david/.puppetlabs/opt/puppet/cache/lib'
vardir before configuring: /home/david/.puppetlabs/opt/puppet/cache
libdir before configuring: /home/david/.puppetlabs/opt/puppet/cache/lib
$LOAD_PATH.last before configuring: /home/david/.puppetlabs/opt/puppet/cache/lib
vardir after configuring: /home/david/.puppetlabs/opt/puppet/cache/devices/panos
libdir after configuring: /home/david/.puppetlabs/opt/puppet/cache/devices/panos/lib
$LOAD_PATH.last after configuring: /home/david/.puppetlabs/opt/puppet/cache/lib
Error: Could not run: cannot load such file -- puppet/util/network_device/panos/device

This shows how puppet propagates the new vardir value into the computed libdir, without calling the hook for the updated libdir.

We can workaround this by explicitly managing libdir in device.rb, but really this is a dangerous lack of consistency in the underlying settings framework. Josh Cooper, is there a chance of getting this fixed for all?

.h3 initialising device before pluginsync

In device.rb#L258 we set up the device instance to be available for facts and providers, but only the configurer in L303 would activate the pluginsync.

We can likely fix this by using the PluginHandler from the Configurer to fetch plugins when we need it. In this case, I'd also suggest reviving the -pluginsync option for the facts}, {{resource, and -apply cases, to allow folks to force a pluginsync when needed. Josh Cooper/Henrik Lindberg does this sound like a sensible approach to fixing this?

.h2 conclusion

I don't think the original --environment issue is worth fixing before we're loading the code from the right location, and are able to bootstrap a new device without relying on the proxy agent's pluginsync.

Change By: David Schmitt
Affects Version/s: PUP 6.y
Affects Version/s: PUP 5.5.z
Affects Version/s: PUP 5.3.z

Henrik Lindberg (JIRA)

unread,
Oct 26, 2018, 5:55:02 AM10/26/18
to puppe...@googlegroups.com
Henrik Lindberg commented on Bug PUP-8766
 
Re: puppet device fails with undefined method `loaders' for nil:NilClass when env is not production

Sorry, don't know PluginHandler and related logic well enough to be able to comment on suggested approach. Being able to ask-for/force a plugin sync as a separate thing seems to be of value though.

Josh Cooper (JIRA)

unread,
Oct 26, 2018, 1:45:05 PM10/26/18
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-8766

About the loaders failure. Puppet device should definitely be using Puppet::Node::Environment.remote when requesting a catalog from a remote puppetserver (like agent).

We can workaround this by explicitly managing libdir in device.rb, but really this is a dangerous lack of consistency in the underlying settings framework. Josh Cooper, is there a chance of getting this fixed for all?

Welcome to the insanity that is puppet settings. If you change the default value of vardir, then any derived settings like libdir will resolve based on the new value of vardir, however, hooks for the derived settings are never called, so you end up with the ruby $LOAD_PATH pointing to the default location:

irb(main):001:0> require 'puppet'
=> true
irb(main):002:0> Puppet.initialize_settings
=> [:debug, :info, :notice, :warning, :err, :alert, :emerg, :crit]
irb(main):003:0> Puppet[:vardir] = '/tmp/vardir'
=> "/tmp/vardir"
irb(main):004:0> Puppet[:libdir]
=> "/tmp/vardir/lib"
irb(main):005:0> $LOAD_PATH.grep /cache/
=> ["/Users/josh/.puppetlabs/opt/puppet/cache/lib"]

Since device is somewhat special (requesting catalogs on behalf of other nodes), I would just have it set :libdir explicitly. Also note that puppet will ensure the $LOAD_PATH is updated to include all of the lib directories from modules in the current environment. If the device application needs to be able to switch environments based on the ENC like the agent does, then we should exclude the device application from there, otherwise device will be able to see modules that it shouldn't.

We can likely fix this by using the PluginHandler from the Configurer to fetch plugins when we need it.

You should be able to use Puppet::Configurer::PluginHandler to perform a pluginsync. The tricky part is making sure you're sending the right set of facts and that you're resolving the environment correctly (whether it set in the ENC, and whether the device node is allowed to override the ENC's value). If the ENC is authoritative, then its possible you'll receive a catalog compiled in an environment that's different than the environment the device application requested. It's also possible for a fact to cause the environment to change. This is why the Configurer has logic to ask for the {{Node}}s environment, switch environments when requesting the catalog, then checking to see that the catalog's environment is the same as what it requested. If not, it switches again, does another pluginsync, requests another catalog, etc. If it doesn't converge after a few times, it will raise an exception.

If the device application is always authoritative, then you can avoid all of that.

I would expect puppet device to always perform a pluginsync when applying a device catalog so that the plugins and catalog are consistent. Otherwise you can get into a state where the catalog is trying to manage a new property that the old plugin type/provider doesn't know about, and you'll run into difficult to reproduce bugs. So I don't think reviving the --pluginsync option is a good idea.

Rick Sherman (JIRA)

unread,
Oct 31, 2018, 12:38:05 PM10/31/18
to puppe...@googlegroups.com
Rick Sherman commented on Bug PUP-8766

Depending on the depth of this work, it may be a good time to look at PUP-8732 (limiting pluginsync to only remote providers w/ puppet device)

David Schmitt (JIRA)

unread,
Nov 19, 2018, 4:52:04 AM11/19/18
to puppe...@googlegroups.com

Branan Riley (JIRA)

unread,
Jan 8, 2019, 2:36:07 PM1/8/19
to puppe...@googlegroups.com

Jean Bond (JIRA)

unread,
Jan 8, 2019, 6:51:04 PM1/8/19
to puppe...@googlegroups.com
Jean Bond commented on Bug PUP-8766
 
Re: puppet device fails with undefined method `loaders' for nil:NilClass when env is not production

David Schmitt or Josh Cooper, the fix version on this is 5.5.9, but I see that affected versions include 6.y. Is this going into 6.2 and/or 6.0.5.?

Josh Cooper (JIRA)

unread,
Jan 8, 2019, 7:00:06 PM1/8/19
to puppe...@googlegroups.com

Jean Bond (JIRA)

unread,
Jan 8, 2019, 7:01:10 PM1/8/19
to puppe...@googlegroups.com
Jean Bond updated an issue
 
Change By: Jean Bond
Release Notes Summary: * `puppet device` now uses its own device-specific cache for pluginsynced code (facts, types and providers). Previously it would have used the cache of any agent running on the same system.

* `puppet device` now supports a `--libdir` option for overriding any pluginsynced code with a local directory for testing, and other extra-ordinary usecases.


Release note draft, jb: - Prior to this release, the `puppet device` command failed if the environment specified in `puppet.conf` or with the `--environment` option was not 'production'. This issue is fixed. Now `puppet device` uses its own device-specific cache for pluginsynced code (facts, types, and providers). Additionally, `puppet device` now supports a `--libdir` option for overriding any pluginsynced code with a local directory for testing. [PUP-8766](https://tickets.puppetlabs.com/browse/PUP-8766)

Jean Bond (JIRA)

unread,
Jan 8, 2019, 8:13:05 PM1/8/19
to puppe...@googlegroups.com

Jean Bond (JIRA)

unread,
Jan 8, 2019, 8:13:05 PM1/8/19
to puppe...@googlegroups.com

Jean Bond (JIRA)

unread,
Jan 8, 2019, 8:24:06 PM1/8/19
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Feb 27, 2019, 1:10:04 AM2/27/19
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Oct 23, 2020, 8:01:03 PM10/23/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Affects Version/s: PUP 5.5.z
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages