puppetca unable to sign new certs - Invalid argument error

[Jesse Reynolds]

Hello

I have a puppetmasterd installation running on a Mac OS X 10.6.3
Server with puppet installed via macports.

Earlier today it was happily signing requests, before I upgraded
puppet from 0.24.8 to 0.25.4. Now I get "Invalid argument":

bash-3.2# puppetca --sign bouti.carbonplanet.com
bouti.carbonplanet.com
err: Could not call sign: Invalid argument

The only mention I can find on the internets of this error is an IRC
chat on 25 May from bdd:

http://pelin.lovedthanlost.net/puppet/%23puppet-2010-05-25.log.html

<bdd> interesting. after an upgrade from 0.25.4 to 0.25.5, puppetca
fails to sign new requests with "err: Could not call sign: Invalid
argument"
<jamesturnbull> bdd: clean upgrade? no old code floating around?
<bdd> jamesturnbull: it wasn't a clean upgrade. that's solved. thanks.

I used mac ports "port upgrade facter" then "port upgrade puppet", is
this not good enough?

I've also tried to do a revoke, which seems to work but shows a similar error:

bash-3.2# puppetca --list --all
+ 243.carbonplanet.com
(snip)

bash-3.2# puppetca --revoke 243.carbonplanet.com
243.carbonplanet.com
notice: Revoked certificate with serial 14
err: Could not call revoke: Invalid argument

bash-3.2# puppetca --list --all
- 243.carbonplanet.com (certificate revoked)
(snip)

version:

bash-3.2# puppetca --version
0.25.4

which:

bash-3.2# which puppetca
/opt/local/sbin/puppetca

debug:

bash-3.2# puppetca --sign bouti.carbonplanet.com --debug
debug: Failed to load library 'selinux' for feature 'selinux'
debug: Failed to load library 'shadow' for feature 'libshadow'
debug: Puppet::Type::User::ProviderUser_role_add: file rolemod does not exist
debug: Puppet::Type::User::ProviderPw: file pw does not exist
debug: Failed to load library 'ldap' for feature 'ldap'
debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
debug: Puppet::Type::User::ProviderUseradd: file userdel does not exist
debug: Puppet::Type::User::ProviderDirectoryservice: Executing
'/usr/bin/dscl -plist . -list /Users'
debug: Puppet::Type::User::ProviderDirectoryservice: Executing
'/usr/bin/dscl -plist . -read /Users/puppet'
debug: /File[/etc/puppet/ssl/ca/requests]: Autorequiring
File[/etc/puppet/ssl/ca]
debug: /File[/etc/puppet/ssl/ca/signed]: Autorequiring File[/etc/puppet/ssl/ca]
debug: /File[/var/puppet/lib]: Autorequiring File[/var/puppet]
debug: /File[/etc/puppet/ssl/private]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/var/puppet/log]: Autorequiring File[/var/puppet]
debug: /File[/etc/puppet/ssl/ca]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/etc/puppet/ssl/ca/ca_crt.pem]: Autorequiring
File[/etc/puppet/ssl/ca]
debug: /File[/etc/puppet/ssl/ca/private]: Autorequiring File[/etc/puppet/ssl/ca]
debug: /File[/etc/puppet/ssl]: Autorequiring File[/etc/puppet]
debug: /File[/etc/puppet/ssl/certificate_requests]: Autorequiring
File[/etc/puppet/ssl]
debug: /File[/var/puppet/facts]: Autorequiring File[/var/puppet]
debug: /File[/etc/puppet/ssl/private_keys]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/etc/puppet/ssl/certs]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/etc/puppet/ssl/public_keys]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/etc/puppet/ssl/certs/ca.pem]: Autorequiring
File[/etc/puppet/ssl/certs]
debug: /File[/etc/puppet/ssl/private_keys/sylvester.adelaide.carbonplanet.com.pem]:
Autorequiring File[/etc/puppet/ssl/private_keys]
debug: /File[/etc/puppet/ssl/ca/inventory.txt]: Autorequiring
File[/etc/puppet/ssl/ca]
debug: /File[/var/puppet/state]: Autorequiring File[/var/puppet]
debug: /File[/etc/puppet/ssl/ca/ca_crl.pem]: Autorequiring
File[/etc/puppet/ssl/ca]
debug: /File[/var/puppet/run]: Autorequiring File[/var/puppet]
debug: /File[/etc/puppet/ssl/ca/private/ca.pass]: Autorequiring
File[/etc/puppet/ssl/ca/private]
debug: /File[/etc/puppet/ssl/ca/serial]: Autorequiring File[/etc/puppet/ssl/ca]
debug: /File[/etc/puppet/ssl/ca/ca_key.pem]: Autorequiring
File[/etc/puppet/ssl/ca]
debug: /File[/etc/puppet/ssl/ca/ca_pub.pem]: Autorequiring
File[/etc/puppet/ssl/ca]
debug: Finishing transaction 2168470120 with 0 changes
bouti.carbonplanet.com
err: Could not call sign: Invalid argument

Any ideas anyone?

Thank you
Jesse

--

Jesse Reynolds
Carbon Planet Limited - http://www.carbonplanet.com/
Virtual Artists Pty Ltd - http://www.va.com.au/

[James Turnbull]

Jesse Reynolds wrote:
> Hello
>
> I have a puppetmasterd installation running on a Mac OS X 10.6.3
> Server with puppet installed via macports.
>
> Earlier today it was happily signing requests, before I upgraded
> puppet from 0.24.8 to 0.25.4. Now I get "Invalid argument":
>
> bash-3.2# puppetca --sign bouti.carbonplanet.com
> bouti.carbonplanet.com
> err: Could not call sign: Invalid argument
>
> The only mention I can find on the internets of this error is an IRC
> chat on 25 May from bdd:

Looks like you've got some old code floating around. I'd remove all of
Puppet and then re-install.

Regards

James Turnbull

[Jesse Reynolds]

On 16 June 2010 00:06, James Turnbull <ja...@puppetlabs.com> wrote:
> Looks like you've got some old code floating around.  I'd remove all of
> Puppet and then re-install.

OK, I'll have a big hunt.

I've tried uninstalling puppet with mac ports and re-installing, doesn't help.

I've done a find over the whole filesystem for 'puppet' and found
nothing installed after doing the 'mac port uninstall puppet' except
the config files, ssl stuff etc (ie /etc/puppet and /var/puppet
stuff).

I'm pretty sure it's properly uninstalled and installed afresh.

I suppose though the mac port could have included some old code by
mistake. Hmmmm. Do you know which old code I should be looking for?

Thanks

Jesse

[Nigel Kersten]

Small world Jesse :)

Bloody Australians are everywhere these days...

There's really not much to the Portfile, it just runs the install.rb
script. The handling of old versions should all be done higher up in
the framework.

I wonder if the upgrade from 0.24.x to 0.25.x wasn't handled properly though...

what does 'type --all puppetca' show?

>
> Thanks
>
> Jesse
>
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>
>

--
nigel

[Jesse Reynolds]

On 16 June 2010 00:50, Nigel Kersten <nig...@google.com> wrote:
> On Tue, Jun 15, 2010 at 7:57 AM, Jesse Reynolds
> <jessedr...@gmail.com> wrote:
>> On 16 June 2010 00:06, James Turnbull <ja...@puppetlabs.com> wrote:
>>> Looks like you've got some old code floating around.  I'd remove all of
>>> Puppet and then re-install.
>>
>> OK, I'll have a big hunt.
>>
>> I've tried uninstalling puppet with mac ports and re-installing, doesn't help.
>>
>> I've done a find over the whole filesystem for 'puppet' and found
>> nothing installed after doing the 'mac port uninstall puppet' except
>> the config files, ssl stuff etc (ie /etc/puppet and /var/puppet
>> stuff).
>>
>> I'm pretty sure it's properly uninstalled and installed afresh.
>>
>> I suppose though the mac port could have included some old code by
>> mistake. Hmmmm. Do you know which old code I should be looking for?
>
> Small world Jesse :)

Aye!

> Bloody Australians are everywhere these days...

Excellent, and I get double points because I'm a Kiwi as well as an Aussie :-)

> There's really not much to the Portfile, it just runs the install.rb
> script. The handling of old versions should all be done higher up in
> the framework.
>
> I wonder if the upgrade from 0.24.x to 0.25.x wasn't handled properly though...
>
> what does 'type --all puppetca' show?

bash-3.2# type --all puppetca
puppetca is /opt/local/sbin/puppetca

Cheers
Jesse

[Eric Sorenson]

I have seen this too; I suspect (but have not been able to reduce a simple test case to confirm) that the ruby-openssl bindings in snow leopard are returning EINVAL (thus the "Invalid argument" string) when called from puppet. But it seems the transaction actually succeeds despite the error. When setting up new puppetd on 10.6.x I see this error at each stage of the certificate generation process: key generation, csr generation, cert submission, but re-running after the error bulls it through. This matches what you show with the revocation, where you got an error message but the cert actually was revoked. Very odd and I would love a way to isolate this outside of puppet and report it to the relevant people as it seems to affect all flavours of 10.6 release thus far.

-=Eric

[Jesse Reynolds]

Hi Eric

I would have thought I was using the ruby and OpenSSL that mac ports
had compiled for me, not the os's ruby OpenSSL bindings...? Or have I
misunderstood you?

Jesse Reynolds

[Nigel Kersten]

On Tue, Jun 15, 2010 at 4:13 PM, Eric Sorenson <eric.s...@me.com> wrote:

I have seen this too; I suspect (but have not been able to reduce a simple test case to confirm) that the ruby-openssl bindings in snow leopard are returning EINVAL (thus the "Invalid argument" string) when called from puppet. But it seems the transaction actually succeeds despite the error. When setting up new puppetd on 10.6.x I see this error at each stage of the certificate generation process: key generation, csr generation, cert submission, but re-running after the error bulls it through. This matches what you show with the revocation, where you got an error message but the cert actually was revoked.  Very odd and I would love a way to isolate this outside of puppet and report it to the relevant people as it seems to affect all flavours of 10.6 release thus far.

oh, now this sounds familiar... 

I think I ran into a similar issue on Snow Leopard, and it was reasonably obvious working out what went wrong by running ruby in debug mode like

/usr/bin/ruby --debug /path/to/puppetfoo whatever you're doing

as that way you were avoiding puppet trapping exceptions and re-raising them incorrectly.

I probably won't have time to look at it soon if we bug report it, as I'm heading back to Australia (hopefully will drop by and visit Jesse :) ) at the end of the week for 5 weeks vacation.

--
nigel

[Jesse Reynolds]

On 16 June 2010 10:13, Nigel Kersten <nig...@google.com> wrote:
>
>
> On Tue, Jun 15, 2010 at 4:13 PM, Eric Sorenson <eric.s...@me.com> wrote:
>>
>> I have seen this too; I suspect (but have not been able to reduce a simple
>> test case to confirm) that the ruby-openssl bindings in snow leopard are
>> returning EINVAL (thus the "Invalid argument" string) when called from
>> puppet. But it seems the transaction actually succeeds despite the error.
>> When setting up new puppetd on 10.6.x I see this error at each stage of the
>> certificate generation process: key generation, csr generation, cert
>> submission, but re-running after the error bulls it through. This matches
>> what you show with the revocation, where you got an error message but the
>> cert actually was revoked.  Very odd and I would love a way to isolate this
>> outside of puppet and report it to the relevant people as it seems to affect
>> all flavours of 10.6 release thus far.
>
> oh, now this sounds familiar...
> I think I ran into a similar issue on Snow Leopard, and it was reasonably
> obvious working out what went wrong by running ruby in debug mode like
> /usr/bin/ruby --debug /path/to/puppetfoo whatever you're doing
> as that way you were avoiding puppet trapping exceptions and re-raising them
> incorrectly.

OK, I've run it in debug mode, with /opt/local/bin/ruby though as I
haven't installed puppet in the system's ruby. Full output is in this
pastie: http://pastie.org/1006381 ... It has lots of Exception lines,
which I don't know what to make of! Many of the latter ones feature
'invalid value for Integer: "puppet"' which seems a bit suspect, eg:

Exception `ArgumentError' at
/opt/local/lib/ruby/site_ruby/1.8/puppet/type/file/owner.rb:56 -
invalid value for Integer: "puppet"
Exception `ArgumentError' at
/opt/local/lib/ruby/site_ruby/1.8/puppet/util/posix.rb:117 - invalid
value for Integer: "puppet"
...
Exception `ArgumentError' at
/opt/local/lib/ruby/site_ruby/1.8/puppet/util/posix.rb:94 - invalid
value for Integer: "puppet"

Some other exceptions:

Exception `LoadError' at
/opt/local/lib/ruby/vendor_ruby/1.8/rubygems.rb:1113 - no such file to
load -- rubygems/defaults/operating_system
Exception `NoMethodError' at /opt/local/lib/ruby/1.8/rational.rb:78 -
undefined method `gcd' for Rational(1, 2):Rational
Exception `LoadError' at
/opt/local/lib/ruby/vendor_ruby/1.8/rubygems/config_file.rb:34 - no
such file to load -- Win32API

Exception `LoadError' at
/opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:31 - no
such file to load -- active_record

Exception `LoadError' at
/opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:38 - no
such file to load -- Win32API

Exception `LoadError' at
/opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:31 - no
such file to load -- selinux

Exception `LoadError' at
/opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:31 - no
such file to load -- shadow
Exception `LoadError' at
/opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:38 - no
such file to load -- shadow

Exception `ArgumentError' at /opt/local/lib/ruby/1.8/open-uri.rb:32 -
illegal access mode 80

Exception `LoadError' at
/opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:31 - no
such file to load -- puppet/provider/confine/operatingsystem
Exception `LoadError' at
/opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:38 - no
such file to load -- puppet/provider/confine/operatingsystem
Exception `LoadError' at
/opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:31 - no
such file to load -- ldap
Exception `LoadError' at
/opt/local/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:38 - no
such file to load -- ldap

I take it most of these are fine and being handled by puppet or lower
down subsystesms.

> I probably won't have time to look at it soon if we bug report it, as I'm
> heading back to Australia (hopefully will drop by and visit Jesse :) ) at
> the end of the week for 5 weeks vacation.

Hope to catch up with you in person Nigel :-)

Cheers
Jesse