Custom plugin help ... 'singleton' behavior?

[Matt Wise]

Hey everyone, I could use some help with a custom plugin we've written. We have a plugin for Puppet that makes SOAP calls out to a remote API service (http://secretserveronline.com), returning passwords, SSL keys, etc for us. The original code was written using pure HTTP GET calls, but I re-wrote the plugin to use the SOAP API because the remote service is slowly removing their GET-based API calls.

Because the Ruby SOAP library is so slow at interpreting WSDL files, I wrote the Puppet Function so that it checks if an existing 'Thycotic' object has been instantiated before creating a new one. If an existing one is there, it just uses it. This prevents creating new 'Thycotic' objects for every single call to the function, and speeds the puppet compilation up quite a bit when you call the plugin many times. (We have some manifests that use this plugin 30+ times for different keys, passwords, etc).

The problem I'm running into is this:

Fri Aug 23 01:28:09 +0000 2013 Puppet (err): Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `has_key?' for nil:NilClass at /mnt/puppet/staging/puppet-base/modules/specific/auth/manifests/eng_users.pp:27 on node ...

We're seeing this very sparingly ... maybe 1/100 puppet manifest compilations. However, its still pretty odd. I am absolutely NOT a Ruby developer by trade, so alot of what I did was guess work unfortunately. I could really use some eyes on this code to see what I'm doing wrong, and suggest alternatives.

The entire codebase is here: https://github.com/Nextdoor/puppet_thycotic

The function file with the error is: https://github.com/Nextdoor/puppet_thycotic/blob/master/lib/puppet/parser/functions/getsecret.rb, and its obviously either failing on line 150 or 151 (the only calls to has_key?):

    # Create our Thycotic object if it doesn't already exist
    # Look for our config file in a few locations (in order):
    $thycotic ||= init(config)
    # Now request our secret
    secret = $thycotic.getSecret(secret_id)
    # Walk through the returned elements of the hash, and look for the one we want.
    if secret.has_key?(secret_name)
      if secret.has_key?(secret_name) == nil
        raise Puppet::ParseError, "Secret returned by Thycotic.getSecret(#{secretid}) was 'nil'. This is bad, erroring out."
      else
        return secret[secret_name].to_s
      end
    end

Any thoughts? 

[Matt Wise]

Just doing some more testing, it seems that the issue happens whether I have "$thycotic ||= init()" or "$thycotic = init()". I have to assume theres some kind of failure in the init() process -- or even worse, some case where the variable scope is wrong and dangerous?

[John Bollinger]

On Thursday, August 22, 2013 8:43:10 PM UTC-5, Matt Wise wrote:

Hey everyone, I could use some help with a custom plugin we've written. We have a plugin for Puppet that makes SOAP calls out to a remote API service (http://secretserveronline.com), returning passwords, SSL keys, etc for us. The original code was written using pure HTTP GET calls, but I re-wrote the plugin to use the SOAP API because the remote service is slowly removing their GET-based API calls.

Because the Ruby SOAP library is so slow at interpreting WSDL files, I wrote the Puppet Function so that it checks if an existing 'Thycotic' object has been instantiated before creating a new one. If an existing one is there, it just uses it. This prevents creating new 'Thycotic' objects for every single call to the function, and speeds the puppet compilation up quite a bit when you call the plugin many times. (We have some manifests that use this plugin 30+ times for different keys, passwords, etc).

The problem I'm running into is this:

Fri Aug 23 01:28:09 +0000 2013 Puppet (err): Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `has_key?' for nil:NilClass at /mnt/puppet/staging/puppet-base/modules/specific/auth/manifests/eng_users.pp:27 on node ...

That error message seems to be saying that you are trying to invoke a method named 'has_key?' on a nil object.  In the code you referenced, you are invoking a method by that name on the object referenced by variable 'secret' -- not on the one referenced by 'thycotic'.  You get 'secret' from 'thycotic' via

 

    # Now request our secret
    secret = $thycotic.getSecret(secret_id)

It seems reasonable to guess that in some cases the provided 'secret_id' does not correspond to any available secret, and in such cases 'getSecret()' returns nil.  You must catch that case and do something appropriate with it if you want to avoid the error you are getting.


John

[Matt Wise]

Thanks for the feedback ... after a bit of testing, I found you were indeed correct. A few bugs in the code were resulting in a Nil return value. After some tuning, and consulting with the only other semi-Ruby programmer here we've fixed the issue and cleaned things up a bit.

https://github.com/Nextdoor/puppet_thycotic/commit/710978b4b25d539854c233d28efae66f9d370199 < not really related, but cleanup

https://github.com/Nextdoor/puppet_thycotic/commit/df5ac6d624c56c3199b56f4d45a325838bec1dfa < first quick fix

https://github.com/Nextdoor/puppet_thycotic/commit/e1ea5ec3447ccda83186d2538bb9e81c14d9be7a < more complete real fix that cleans up the code

https://github.com/Nextdoor/puppet_thycotic

The code is working great now..

Matt Wise

Sr. Systems Architect

Nextdoor.com

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-dev/eMXCTlpuNkY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-dev+...@googlegroups.com.
To post to this group, send email to puppe...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev.
For more options, visit https://groups.google.com/groups/opt_out.