Could not find certificate Error: header too long

[Luca Gioppo]

I'm having a strange issue:


A new machine created this morning showed thi error:
ON PUPPET AGENT

[root@wso2greg ~]# puppet agent --environment=production --verbose --no-daemonize --debug
Debug: Puppet::Type::User::ProviderUser_role_add: file rolemod does not exist
Debug: Puppet::Type::User::ProviderPw: file pw does not exist
Debug: Failed to load library 'ldap' for feature 'ldap'
Debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
Debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/dsimport does not exist
Debug: /User[puppet]: Provider useradd does not support features libuser; not managing attribute forcelocal
Debug: Using settings: adding file resource 'localcacert': 'File[/var/lib/puppet/ssl/certs/ca.pem]{:links=>:follow, :ensure=>:fi              le, :backup=>false, :owner=>"puppet", :mode=>"644", :loglevel=>:debug, :path=>"/var/lib/puppet/ssl/certs/ca.pem"}'
Debug: Using settings: adding file resource 'vardir': 'File[/var/lib/puppet]{:links=>:follow, :ensure=>:directory, :backup=>fals              e, :loglevel=>:debug, :path=>"/var/lib/puppet"}'
Debug: Using settings: adding file resource 'rundir': 'File[/var/run/puppet]{:links=>:follow, :ensure=>:directory, :backup=>fals              e, :mode=>"755", :loglevel=>:debug, :path=>"/var/run/puppet"}'
Debug: Using settings: adding file resource 'certdir': 'File[/var/lib/puppet/ssl/certs]{:links=>:follow, :ensure=>:directory, :b              ackup=>false, :owner=>"puppet", :loglevel=>:debug, :path=>"/var/lib/puppet/ssl/certs"}'
Debug: Using settings: adding file resource 'requestdir': 'File[/var/lib/puppet/ssl/certificate_requests]{:links=>:follow, :ensu              re=>:directory, :backup=>false, :owner=>"puppet", :loglevel=>:debug, :path=>"/var/lib/puppet/ssl/certificate_requests"}'
Debug: Using settings: adding file resource 'plugindest': 'File[/var/lib/puppet/lib]{:links=>:follow, :ensure=>:directory, :back              up=>false, :loglevel=>:debug, :path=>"/var/lib/puppet/lib"}'
Debug: Using settings: adding file resource 'clientyamldir': 'File[/var/lib/puppet/client_yaml]{:links=>:follow, :ensure=>:direc              tory, :backup=>false, :mode=>"750", :loglevel=>:debug, :path=>"/var/lib/puppet/client_yaml"}'
Debug: Puppet::Type::Group::ProviderPw: file pw does not exist
Debug: Failed to load library 'ldap' for feature 'ldap'
Debug: Puppet::Type::Group::ProviderLdap: feature ldap is missing
Debug: Puppet::Type::Group::ProviderDirectoryservice: file /usr/bin/dscl does not exist
Debug: /Group[puppet]: Provider groupadd does not support features libuser; not managing attribute forcelocal
Debug: Using settings: adding file resource 'logdir': 'File[/var/log/puppet]{:links=>:follow, :ensure=>:directory, :group=>"pupp              et", :backup=>false, :owner=>"puppet", :mode=>"750", :loglevel=>:debug, :path=>"/var/log/puppet"}'
Debug: Using settings: adding file resource 'hostprivkey': 'File[/var/lib/puppet/ssl/private_keys/wso2greg.test.italy.cloudlabcs              i.local.pem]{:links=>:follow, :ensure=>:file, :backup=>false, :owner=>"puppet", :mode=>"600", :loglevel=>:debug, :path=>"/var/li              b/puppet/ssl/private_keys/wso2greg.test.italy.cloudlabcsi.local.pem"}'
Debug: Using settings: adding file resource 'clientbucketdir': 'File[/var/lib/puppet/clientbucket]{:links=>:follow, :ensure=>:di              rectory, :backup=>false, :mode=>"750", :loglevel=>:debug, :path=>"/var/lib/puppet/clientbucket"}'
Debug: Using settings: adding file resource 'ssldir': 'File[/var/lib/puppet/ssl]{:links=>:follow, :ensure=>:directory, :backup=>              false, :owner=>"puppet", :mode=>"771", :loglevel=>:debug, :path=>"/var/lib/puppet/ssl"}'
Debug: Using settings: adding file resource 'graphdir': 'File[/var/lib/puppet/state/graphs]{:links=>:follow, :ensure=>:directory              , :backup=>false, :loglevel=>:debug, :path=>"/var/lib/puppet/state/graphs"}'
Debug: Using settings: adding file resource 'privatekeydir': 'File[/var/lib/puppet/ssl/private_keys]{:links=>:follow, :ensure=>:              directory, :backup=>false, :owner=>"puppet", :mode=>"750", :loglevel=>:debug, :path=>"/var/lib/puppet/ssl/private_keys"}'
Debug: Using settings: adding file resource 'client_datadir': 'File[/var/lib/puppet/client_data]{:links=>:follow, :ensure=>:dire              ctory, :backup=>false, :mode=>"750", :loglevel=>:debug, :path=>"/var/lib/puppet/client_data"}'
Debug: Using settings: adding file resource 'confdir': 'File[/etc/puppet]{:links=>:follow, :ensure=>:directory, :backup=>false,               :loglevel=>:debug, :path=>"/etc/puppet"}'
Debug: Using settings: adding file resource 'hostpubkey': 'File[/var/lib/puppet/ssl/public_keys/wso2greg.test.italy.cloudlabcsi.              local.pem]{:links=>:follow, :ensure=>:file, :backup=>false, :owner=>"puppet", :mode=>"644", :loglevel=>:debug, :path=>"/var/lib/              puppet/ssl/public_keys/wso2greg.test.italy.cloudlabcsi.local.pem"}'
Debug: Using settings: adding file resource 'statedir': 'File[/var/lib/puppet/state]{:links=>:follow, :ensure=>:directory, :back              up=>false, :mode=>"1755", :loglevel=>:debug, :path=>"/var/lib/puppet/state"}'
Debug: Using settings: adding file resource 'publickeydir': 'File[/var/lib/puppet/ssl/public_keys]{:links=>:follow, :ensure=>:di              rectory, :backup=>false, :owner=>"puppet", :loglevel=>:debug, :path=>"/var/lib/puppet/ssl/public_keys"}'
Debug: Using settings: adding file resource 'privatedir': 'File[/var/lib/puppet/ssl/private]{:links=>:follow, :ensure=>:director              y, :backup=>false, :owner=>"puppet", :mode=>"750", :loglevel=>:debug, :path=>"/var/lib/puppet/ssl/private"}'
Debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/state/graphs]: Autorequiring File[/var/lib/puppet/state]
Debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/clientbucket]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring File[/var/lib/puppet/ssl/certs]
Debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl/public_keys/wso2greg.test.italy.cloudlabcsi.local.pem]: Autorequiring File[/var/lib/puppet/ssl/              public_keys]
Debug: /File[/var/lib/puppet/client_yaml]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl/private_keys/wso2greg.test.italy.cloudlabcsi.local.pem]: Autorequiring File[/var/lib/puppet/ssl              /private_keys]
Debug: /File[/var/lib/puppet/ssl/private]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/client_data]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl]
Debug: Finishing transaction 70041809621420
Debug: Using settings: adding file resource 'localcacert': 'File[/var/lib/puppet/ssl/certs/ca.pem]{:links=>:follow, :ensure=>:fi              le, :backup=>false, :owner=>"puppet", :mode=>"644", :loglevel=>:debug, :path=>"/var/lib/puppet/ssl/certs/ca.pem"}'
Debug: Using settings: adding file resource 'vardir': 'File[/var/lib/puppet]{:links=>:follow, :ensure=>:directory, :backup=>fals              e, :loglevel=>:debug, :path=>"/var/lib/puppet"}'
Debug: Using settings: adding file resource 'rundir': 'File[/var/run/puppet]{:links=>:follow, :ensure=>:directory, :backup=>fals              e, :mode=>"755", :loglevel=>:debug, :path=>"/var/run/puppet"}'
Debug: Using settings: adding file resource 'certdir': 'File[/var/lib/puppet/ssl/certs]{:links=>:follow, :ensure=>:directory, :b              ackup=>false, :owner=>"puppet", :loglevel=>:debug, :path=>"/var/lib/puppet/ssl/certs"}'
Debug: Using settings: adding file resource 'requestdir': 'File[/var/lib/puppet/ssl/certificate_requests]{:links=>:follow, :ensu              re=>:directory, :backup=>false, :owner=>"puppet", :loglevel=>:debug, :path=>"/var/lib/puppet/ssl/certificate_requests"}'
Debug: Using settings: adding file resource 'plugindest': 'File[/var/lib/puppet/lib]{:links=>:follow, :ensure=>:directory, :back              up=>false, :loglevel=>:debug, :path=>"/var/lib/puppet/lib"}'
Debug: Using settings: adding file resource 'logdir': 'File[/var/log/puppet]{:links=>:follow, :ensure=>:directory, :group=>"pupp              et", :backup=>false, :owner=>"puppet", :mode=>"750", :loglevel=>:debug, :path=>"/var/log/puppet"}'
Debug: Using settings: adding file resource 'hostprivkey': 'File[/var/lib/puppet/ssl/private_keys/wso2greg.test.italy.cloudlabcs              i.local.pem]{:links=>:follow, :ensure=>:file, :backup=>false, :owner=>"puppet", :mode=>"600", :loglevel=>:debug, :path=>"/var/li              b/puppet/ssl/private_keys/wso2greg.test.italy.cloudlabcsi.local.pem"}'
Debug: Using settings: adding file resource 'ssldir': 'File[/var/lib/puppet/ssl]{:links=>:follow, :ensure=>:directory, :backup=>              false, :owner=>"puppet", :mode=>"771", :loglevel=>:debug, :path=>"/var/lib/puppet/ssl"}'
Debug: Using settings: adding file resource 'privatekeydir': 'File[/var/lib/puppet/ssl/private_keys]{:links=>:follow, :ensure=>:              directory, :backup=>false, :owner=>"puppet", :mode=>"750", :loglevel=>:debug, :path=>"/var/lib/puppet/ssl/private_keys"}'
Debug: Using settings: adding file resource 'confdir': 'File[/etc/puppet]{:links=>:follow, :ensure=>:directory, :backup=>false,               :loglevel=>:debug, :path=>"/etc/puppet"}'
Debug: Using settings: adding file resource 'hostpubkey': 'File[/var/lib/puppet/ssl/public_keys/wso2greg.test.italy.cloudlabcsi.              local.pem]{:links=>:follow, :ensure=>:file, :backup=>false, :owner=>"puppet", :mode=>"644", :loglevel=>:debug, :path=>"/var/lib/              puppet/ssl/public_keys/wso2greg.test.italy.cloudlabcsi.local.pem"}'
Debug: Using settings: adding file resource 'statedir': 'File[/var/lib/puppet/state]{:links=>:follow, :ensure=>:directory, :back              up=>false, :mode=>"1755", :loglevel=>:debug, :path=>"/var/lib/puppet/state"}'
Debug: Using settings: adding file resource 'publickeydir': 'File[/var/lib/puppet/ssl/public_keys]{:links=>:follow, :ensure=>:di              rectory, :backup=>false, :owner=>"puppet", :loglevel=>:debug, :path=>"/var/lib/puppet/ssl/public_keys"}'
Debug: Using settings: adding file resource 'privatedir': 'File[/var/lib/puppet/ssl/private]{:links=>:follow, :ensure=>:director              y, :backup=>false, :owner=>"puppet", :mode=>"750", :loglevel=>:debug, :path=>"/var/lib/puppet/ssl/private"}'
Debug: /File[/var/lib/puppet/ssl/public_keys/wso2greg.test.italy.cloudlabcsi.local.pem]: Autorequiring File[/var/lib/puppet/ssl/              public_keys]
Debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl/private_keys/wso2greg.test.italy.cloudlabcsi.local.pem]: Autorequiring File[/var/lib/puppet/ssl              /private_keys]
Debug: /File[/var/lib/puppet/ssl/private]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring File[/var/lib/puppet/ssl]
Debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring File[/var/lib/puppet/ssl/certs]
Debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
Debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl]
Debug: Finishing transaction 70041808592580
Debug: Using cached certificate for ca
Error: Could not request certificate: Error 400 on SERVER: header too long


ON PUPPET MASTER
Notice: Starting Puppet master version 3.2.3-rc1
Info: access[^/catalog/([^/]+)$]: allowing 'method' find
Info: access[^/catalog/([^/]+)$]: allowing $1 access
Info: access[^/node/([^/]+)$]: allowing 'method' find
Info: access[^/node/([^/]+)$]: allowing $1 access
Info: access[/certificate_revocation_list/ca]: allowing 'method' find
Info: access[/certificate_revocation_list/ca]: allowing * access
Info: access[^/report/([^/]+)$]: allowing 'method' save
Info: access[^/report/([^/]+)$]: allowing $1 access
Info: access[/file]: allowing * access
Info: access[/certificate/ca]: adding authentication any
Info: access[/certificate/ca]: allowing 'method' find
Info: access[/certificate/ca]: allowing * access
Info: access[/certificate/]: adding authentication any
Info: access[/certificate/]: allowing 'method' find
Info: access[/certificate/]: allowing * access
Info: access[/certificate_request]: adding authentication any
Info: access[/certificate_request]: allowing 'method' find
Info: access[/certificate_request]: allowing 'method' save
Info: access[/certificate_request]: allowing * access
Info: access[/]: adding authentication any
Info: access[/facts]: adding authentication any
Info: access[/facts]: allowing 'method' find
Info: access[/facts]: allowing 'method' search
Info: access[/facts]: allowing * access
Info: Inserting default '/status' (auth true) ACL
Info: Could not find certificate for 'wso2greg.test.italy.cloudlabcsi.local'
Error: header too long


At first I started looking around and the puppet master had the disk full because the problem on the puppet-dashboard that could not save messages that were too long (see issue http://projects.puppetlabs.com/issues/18757) after fixing the problem altering dashboard columns and  removing the HUGE delayed_jobs file the server got its disk back, but still no prize !!!

Any idea on what is stuck?
Maybe the agent managed to send some thing in the first go and did not got back the right answer since the disk full on the master and something broke?

Thanks
Luca

[Luca Gioppo]

SOLVED
on puppet master there was a 0 size certificate request that caused all the problem even with puppet cert list I got no good output.
Removed the unwanted file and all works again.

Could it be a not so clear error message? May be telling something more could help solve the problem.
Luca

[T.J. Yang]

Thanks for the 0 byte file hint, I deleted the files with 0 byte size to resolve my same issue.

[root@lpuppet puppet]# find . -type f -size 0c

./ssl/ca/requests/ilmcentos65t2.test.com.pem

./ssl/ca/requests/lapppwmbkr02.test.com.pem

./ssl/ca/requests/lapppwm05.test.com.pem

[root@lpuppet puppet]# cd ssl/ca/requests/

[root@lpuppet requests]# ls -lrt

total 4

-rw-r----- 1 pe-puppet pe-puppet    0 Jun 18 12:45 lapppwm05.test.com.pem

-rw-r----- 1 pe-puppet pe-puppet    0 Jun 30 14:53 lapppwmbkr02.test.com.pem

-rw-r----- 1 pe-puppet pe-puppet 1586 Aug 15 15:36 ilcljgin01.pem

-rw-r----- 1 pe-puppet pe-puppet    0 Aug 16 16:24 ilmcentos65t2.test.com.pem

[root@lpuppet requests]# pwd

/etc/puppetlabs/puppet/ssl/ca/requests

[root@lpuppet requests]#