Custom providers/types & gem dependencies

[Mohit Chawla]

Hello,

I am using a custom provider that uses an external gem, for eg., json.
I am seeing this behavior:

1) In the first puppet run - errors would be thrown like :
"err: Could not load downloaded file
/var/lib/puppet/lib/puppet/provider/custom_provider/json.rb: no such
file to load -- json"
*However*, the catalog compiles and works eventually. Perhaps because
I do have a puppet module which installs json lib before the module
which uses the custom provider is
executed.

2) If however, there was some other error in the first run, say a
syntax error somewhere, and then you try the puppet run, it simply
doesn't retrieve the catalog with an error like:
"Could not autoload sensu_client_config: Could not autoload
/var/lib/puppet/lib/puppet/provider/custom_provider/json.rb: no such
file to load -- json"

If this is not a bug (not sure how much this is related to
http://projects.puppetlabs.com/issues/6907 ), then what is the
suggested solution for handling gem dependencies in providers/types ?

[Jeff McCune]

On Tue, Nov 20, 2012 at 10:24 AM, Mohit Chawla
<mohit.cha...@gmail.com> wrote:
> Hello,
>
> I am using a custom provider that uses an external gem, for eg., json.
> I am seeing this behavior:
>
> 1) In the first puppet run - errors would be thrown like :
> "err: Could not load downloaded file
> /var/lib/puppet/lib/puppet/provider/custom_provider/json.rb: no such
> file to load -- json"
> *However*, the catalog compiles and works eventually. Perhaps because
> I do have a puppet module which installs json lib before the module
> which uses the custom provider is
> executed.

What version of Puppet are you running?

Yes, there is a dependency issue here. Providers are loaded right
after they are synchronized using pluginsync, which happens before the
catalog is applied. If the catalog is responsible for configuring the
json gem, then the provider will not have this dependency available
the first time it is loaded. Custom providers should definitely be
able to reuse code from Rubygem libraries.

> 2) If however, there was some other error in the first run, say a
> syntax error somewhere, and then you try the puppet run, it simply
> doesn't retrieve the catalog with an error like:
> "Could not autoload sensu_client_config: Could not autoload
> /var/lib/puppet/lib/puppet/provider/custom_provider/json.rb: no such
> file to load -- json"
>
> If this is not a bug (not sure how much this is related to
> http://projects.puppetlabs.com/issues/6907 ), then what is the
> suggested solution for handling gem dependencies in providers/types ?

That's the issue. I have no idea why that's marked as closed, it
definitely hasn't been resolved in 3.0 for the scenario of providers
relying on gem libraries. Perhaps we closed that issue because it was
more focused on system commands being available instead of gem
dependencies being available. Could you please create a new ticket
[1] and mark it as related to both #6907 and #7316 and then ping me
with the new ticket number? I'll bring this ticket to the attention
of the rest of the team. The platform team is currently working on
code loading issues for Puppet 3.1 and this issue definitely falls
under this umbrella, so I'd like to collect as much impact data and
related information about what you're trying accomplish as possible.
A new ticket to collect this information is the best tool we have at
this time.

[1] http://projects.puppetlabs.com/projects/puppet/issues/new

Hope this helps,
-Jeff

[Mohit Chawla]

Hello Jeff,

Thanks for the reply.

> A new ticket to collect this information is the best tool we have at
> this time.

This is on 2.7.19. Added ticket http://projects.puppetlabs.com/issues/17747.

[Andy Parker]

I believe this is related to 2.7 eagerly loading the providers during pluginsync. In 3.0 the providers are loaded only after the pluginsync is all done and I think they are delayed until they are needed, but I'm not certain about that.

 

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Jeff McCune]

On Tue, Nov 20, 2012 at 12:51 PM, Andy Parker <an...@puppetlabs.com> wrote:
>> This is on 2.7.19. Added ticket
>> http://projects.puppetlabs.com/issues/17747.

Sweet, thank you for taking the time to do this.

> I believe this is related to 2.7 eagerly loading the providers during
> pluginsync. In 3.0 the providers are loaded only after the pluginsync is all
> done and I think they are delayed until they are needed, but I'm not certain
> about that.

That's true, but I don't think the improved behavior in Puppet 3.0 is
sufficient to resolve this specific issue. This specific issue is a
chicken and egg problem where by a provider needs a gem, but the
catalog run itself is the thing that provides the gem dependency.
That is to say, even in Puppet 3.0 where we delay loading all of the
providers until after pluginsync finishes, the catalog run hasn't yet
installed the gem when the provider is loaded.

The reason I think this is basically a very specific incarnation of
#6907 is because that ticket is pretty specific from a product
functionality perspective, "You should not have to run puppet twice to
use a provider."

-Jeff