Custom Providers and Third Party Gems

[Chris Pitman]

Hey everyone,

I'm hoping someone can enlighten me here: Why is it so hard/complicated to use third party gems when developing a custom provider? It seems to me that pulling in gems should be priority #1, since it allows providers to leverage a lot of development already done in the ruby community.

However, right now it is complicated:

1. I have to get my users downstream to install gem dependencies before using any resources related to the provider. There is no way for me to annotate the module itself for these dependencies, and no support for automatically pulling them down.

2. "Features" offer very limited capability (as far as I understand), and are poorly documented. The only documentation is on how to link providers and types using features, not on how to require ruby libraries. There is no way for anything other than a type to specify requiring a feature. For example, what if a provider requires a gem?

3. "Features" only protect part of the lifecycle. Even without a feature present, puppet still attempts to resolve auto-require relationships. This makes sense based on how things work, but what am I supposed to do if I need a third party gem there?

4. Some gem dependencies need to be present on both the puppet master (in the master's environment) and on the node. There is no way for me to transparently take care of this for users.

So, am I just missing something? Is there some secret hook in the puppet source that makes dealing with gems better? Should I just start distributing this provider as a package, and ignore "puppet module", r10k, etc? Or are there any changes coming down the pipeline that will make this work better?

Thanks,
Chris

[Nan Liu]

Maybe post to puppet-dev to get the developer's feedback?

On Thu, Oct 2, 2014 at 10:54 AM, Chris Pitman <cpit...@gmail.com> wrote:

Hey everyone,

I'm hoping someone can enlighten me here: Why is it so hard/complicated to use third party gems when developing a custom provider? It seems to me that pulling in gems should be priority #1, since it allows providers to leverage a lot of development already done in the ruby community.

However, right now it is complicated:

1. I have to get my users downstream to install gem dependencies before using any resources related to the provider. There is no way for me to annotate the module itself for these dependencies, and no support for automatically pulling them down.

 

You can install gem dependency on the client as part of the puppet run. This was added around 2.7: http://projects.puppetlabs.com/issues/14822 

But there are some issues that require clearing the ruby gem path cache:

https://tickets.puppetlabs.com/browse/PUP-1879

I don't always have the latest puppet version, so this was a work around (as always should be accompanied with horrible disclaimer):

# Try one last time since PUP-1879 isn't always available:

unless defined? ::Faraday

  Gem.clear_paths unless defined? ::Bundler

  require 'faraday_middleware'

end

@connection = ::Faraday.new(url)

I was hoping in the type to autorequire class where the packages were installed, but was only able to get normal resources to work:

autorequire(:package) do
  'faraday_middleware'

end

2. "Features" offer very limited capability (as far as I understand), and are poorly documented. The only documentation is on how to link providers and types using features, not on how to require ruby libraries. There is no way for anything other than a type to specify requiring a feature. For example, what if a provider requires a gem?

The method missing code for Puppet.features.<feature_name>? is not just for checking gem availability, it also attempts to load the gem:

require 'some_feature' if Puppet.features.some_feature?

PUP-3032 adds a new setting. For the agent it should be always_cache_features=false, and =true for performance reasons on the server: https://tickets.puppetlabs.com/browse/PUP-3032.

3. "Features" only protect part of the lifecycle. Even without a feature present, puppet still attempts to resolve auto-require relationships. This makes sense based on how things work, but what am I supposed to do if I need a third party gem there?

4. Some gem dependencies need to be present on both the puppet master (in the master's environment) and on the node. There is no way for me to transparently take care of this for users.

Yeah, this one is annoying, and the best solution I can come up with at the moment is a master manifest, or a master gemfile in the site module. Also there's no way to have multiple gem versions on the master. The new puppet-server project is suppose to offer more isolation, but I'm not exactly sure how it will be able to do this.

So, am I just missing something? Is there some secret hook in the puppet source that makes dealing with gems better? Should I just start distributing this provider as a package, and ignore "puppet module", r10k, etc? Or are there any changes coming down the pipeline that will make this work better?

Yes, I agree we should have better tools, puppet module and r10k seems like good candidates. We need to agree how to specify master specific gems in the module metadata, and hopefully not something that reinvents the wheel.

HTH,

Nan 

[jcbollinger]

On Thursday, October 2, 2014 12:54:06 PM UTC-5, Chris Pitman wrote:

Hey everyone,

I'm hoping someone can enlighten me here: Why is it so hard/complicated to use third party gems when developing a custom provider? It seems to me that pulling in gems should be priority #1, since it allows providers to leverage a lot of development already done in the ruby community.

I cannot speak to the Puppet implementation issues involved here, but speaking from under my sysadmin hat, I will say that relying on any gems is the last thing I want any module I use to do.  My systems already have package management under control, thank you, and I do not want gems to complicate that.  I'll grant an exception to gems encapsulated in and managed via native packages, but what's the point of that, really?

 

However, right now it is complicated:

1. I have to get my users downstream to install gem dependencies before using any resources related to the provider. There is no way for me to annotate the module itself for these dependencies, and no support for automatically pulling them down.

If a certain module depended on gems, and if, despite my better judgement, I wanted to use it anyway, then the next last thing I would want it to do is automatically install gems on my master as part of its own installation.

 

2. "Features" offer very limited capability (as far as I understand), and are poorly documented. The only documentation is on how to link providers and types using features, not on how to require ruby libraries. There is no way for anything other than a type to specify requiring a feature. For example, what if a provider requires a gem?

3. "Features" only protect part of the lifecycle. Even without a feature present, puppet still attempts to resolve auto-require relationships. This makes sense based on how things work, but what am I supposed to do if I need a third party gem there?

I agree that the Features feature is not suitable for your purpose.  Features are not designed or intended for what you want to do.

 

4. Some gem dependencies need to be present on both the puppet master (in the master's environment) and on the node. There is no way for me to transparently take care of this for users.

You can address the node installation issue via your module implementation.  Yes, that makes the module more complicated.

 

So, am I just missing something? Is there some secret hook in the puppet source that makes dealing with gems better? Should I just start distributing this provider as a package, and ignore "puppet module", r10k, etc? Or are there any changes coming down the pipeline that will make this work better?

I'm sure I'm in the minority, but I would indeed prefer to receive your module on my master via a native package (i.e. an RPM).

Supposing that it were instead delivered via the module tool, I agree that it would be very good if the module metadata could document dependencies on external software such as gems and native applications, and it would be outstanding if the module tool could check those dependencies and maybe even offer to install any missing ones.


John

[Nan Liu]

On Fri, Oct 3, 2014 at 7:23 AM, jcbollinger <John.Bo...@stjude.org> wrote:

On Thursday, October 2, 2014 12:54:06 PM UTC-5, Chris Pitman wrote:

Hey everyone,

I'm hoping someone can enlighten me here: Why is it so hard/complicated to use third party gems when developing a custom provider? It seems to me that pulling in gems should be priority #1, since it allows providers to leverage a lot of development already done in the ruby community.

I cannot speak to the Puppet implementation issues involved here, but speaking from under my sysadmin hat, I will say that relying on any gems is the last thing I want any module I use to do.  My systems already have package management under control, thank you, and I do not want gems to complicate that.  I'll grant an exception to gems encapsulated in and managed via native packages, but what's the point of that, really?

Puppet modules like f5, vcenter (the list goes on), need additional libraries like savon, nokogiri, rbvmomi, etc. You might not use those modules, but for me it's helpful if module installation/puppet agent run can bring in these dependencies when appropriate. As much as I like packages, gems are really a practical way to provide these dependencies for the multitude of use case from bundler, system ruby, puppet enterprise ruby, and even Windows (yes I know). In lots of cases a native package is desirable, and I've seen the horrors of trying to build a gem, but the ecosystem is simply too complicated to dismiss gems and mandate native packages as the one true solution.

However, right now it is complicated:

1. I have to get my users downstream to install gem dependencies before using any resources related to the provider. There is no way for me to annotate the module itself for these dependencies, and no support for automatically pulling them down.

If a certain module depended on gems, and if, despite my better judgement, I wanted to use it anyway, then the next last thing I would want it to do is automatically install gems on my master as part of its own installation.

I don't think the tool should make assumptions and automatically install gems dependencies, but it should be an option. At least something like 'puppet module list_gem_dep' would be nice to have.

 

2. "Features" offer very limited capability (as far as I understand), and are poorly documented. The only documentation is on how to link providers and types using features, not on how to require ruby libraries. There is no way for anything other than a type to specify requiring a feature. For example, what if a provider requires a gem?

3. "Features" only protect part of the lifecycle. Even without a feature present, puppet still attempts to resolve auto-require relationships. This makes sense based on how things work, but what am I supposed to do if I need a third party gem there?

I agree that the Features feature is not suitable for your purpose.  Features are not designed or intended for what you want to do.

 

4. Some gem dependencies need to be present on both the puppet master (in the master's environment) and on the node. There is no way for me to transparently take care of this for users.

You can address the node installation issue via your module implementation.  Yes, that makes the module more complicated.

 

So, am I just missing something? Is there some secret hook in the puppet source that makes dealing with gems better? Should I just start distributing this provider as a package, and ignore "puppet module", r10k, etc? Or are there any changes coming down the pipeline that will make this work better?

I'm sure I'm in the minority, but I would indeed prefer to receive your module on my master via a native package (i.e. an RPM).

Supposing that it were instead delivered via the module tool, I agree that it would be very good if the module metadata could document dependencies on external software such as gems and native applications, and it would be outstanding if the module tool could check those dependencies and maybe even offer to install any missing ones.

Absolutely, you can repackage modules as packages for distribution on your platform. But it's not practical to ask module authors to package modules and it's gem dependencies for distribution for every platform (maybe an opportunity to have a toolchain around it). For your stated use case, you aren't a consumer of 'puppet module install' or r10k.

Respectfully,

Nan

[Corey Osman]

Just to add.

If you can remove any gem dependency by reimplementing that functionality inside your own provider or even create some helper code I would try that.  There is a section in Nan's book that details how to distribute external helper code inside the module for native types/providers.   Its some special folder called PUPPET_X or something (can't remember).  If your using open source puppet you could even augment the puppet gemspec to include new runtime dependencies.  However, you would also need to host this custom gem and ensure everyone downloads your custom gem instead of the puppet gem from puppet labs/rubygems.org.

I would also consider using omnibus and creating your own distributable packages that contain everything you need to run your puppet environment.  This way as long as your system has been bootstrapped with your "management" package you can ensure everything is there to run your custom native type/providers.  This would also make it easy to version your cocktail of management gems and packages.

Corey