Jira (PUP-10228) Puppet 6 daemons do not release ssl_lockfile

24 views
Skip to first unread message

Charlie Sharpsteen (JIRA)

unread,
Jan 8, 2020, 5:33:05 PM1/8/20
to puppe...@googlegroups.com
Charlie Sharpsteen created an issue
 
Puppet / Bug PUP-10228
Puppet 6 daemons do not release ssl_lockfile
Issue Type: Bug Bug
Affects Versions: PUP 6.11.1
Assignee: Unassigned
Created: 2020/01/08 2:32 PM
Priority: Normal Normal
Reporter: Charlie Sharpsteen

The puppet service holds ssl_lockfile after submiting a Certificate
Signing Request. This blocks other processes from running puppet agent -t
or puppet ssl download_cert after the request has been signed by the
Puppet CA.

    1. Reproduction Case
  • Install Puppet Server 6.7 on CentOS 7 and initialize it:

yum install -y http://yum.puppetlabs.com/puppet6-release-el-7.noarch.rpm
 
yum install -y puppetserver
 
 
 
source /etc/profile.d/puppet-agent.sh
 
puppet config set server $(hostname -f)
 
puppetserver ca setup
 
systemctl start puppetserver

  • Install Puppet Agent 6.11.1 on another CentOS 7 node, configure it to talk
    with the server, and start the daemon:

yum install -y http://yum.puppetlabs.com/puppet6-release-el-7.noarch.rpm
 
yum install -y puppet-agent
 
 
 
source /etc/profile.d/puppet-agent.sh
 
puppet config set server <hostname of server from step 1>
 
systemctl start puppet

  • Sign the certificate on the server:

puppetserver ca sign --certname=<hostname of agent from step 2>

  • Run puppet agent -t or puppet ssl download_cert on the agent.
      1. Outcome

Both commands fail with "Could not run: Another puppet instance is already running; exiting"

# puppet agent -t --trace
 
Error: Could not run: Another puppet instance is already running; exiting
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/ssl/state_machine.rb:421:in `with_lock'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/ssl/state_machine.rb:392:in `run_machine'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/ssl/state_machine.rb:370:in `ensure_client_certificate'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:505:in `wait_for_certificates'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:368:in `run_command'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:687:in `exit_on_fail'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
 
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
 
 
 
# puppet ssl download_cert --trace
 
Error: Could not run: Another puppet instance is already running; exiting
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/ssl/state_machine.rb:421:in `with_lock'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/ssl/state_machine.rb:392:in `run_machine'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/ssl/state_machine.rb:361:in `ensure_ca_certificates'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/ssl.rb:130:in `main'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:390:in `run_command'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:687:in `exit_on_fail'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
 
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'

This is because the ssl_lockfile exists and is held by the daemon:

# ls $(puppet config print ssl_lockfile)
 
/etc/puppetlabs/puppet/ssl/ssl.lock
 
 
 
# echo $(cat $(puppet config print ssl_lockfile))
 
10460
 
 
 
# systemctl show -p MainPID puppet
 
MainPID=10460

      1. Expected Outcome

The daemon releases the ssl_lockfile when it is not actively updating certificate state.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Charlie Sharpsteen (JIRA)

unread,
Jan 8, 2020, 5:34:03 PM1/8/20
to puppe...@googlegroups.com
Charlie Sharpsteen updated an issue
Change By: Charlie Sharpsteen
The {{puppet}} service holds {{ssl_lockfile}} after submiting a Certificate
Signing Request. This blocks other processes from running {{puppet agent -t}}
or {{puppet ssl download_cert}} after the request has been signed by the
Puppet CA.
## h2. Reproduction Case

  - Install Puppet Server 6.7 on CentOS 7 and initialize it:

{code:bash}

yum install -y http://yum.puppetlabs.com/puppet6-release-el-7.noarch.rpm
yum install -y puppetserver

source /etc/profile.d/puppet-agent.sh
puppet config set server $(hostname -f)
puppetserver ca setup
systemctl start puppetserver
{code}

  - Install Puppet Agent 6.11.1 on another CentOS 7 node, configure it to talk

    with the server, and start the daemon:

{code:bash}

yum install -y http://yum.puppetlabs.com/puppet6-release-el-7.noarch.rpm
yum install -y puppet-agent

source /etc/profile.d/puppet-agent.sh
puppet config set server <hostname of server from step 1>
systemctl start puppet
{code}


  - Sign the certificate on the server:

{code:bash}

puppetserver ca sign --certname=<hostname of agent from step 2>
{code}

  - Run {{puppet agent -t}} or {{puppet ssl download_cert}} on the agent.

 ### h3. Outcome


Both commands fail with "Could not run: Another puppet instance is already running; exiting"

{noformat}
{noformat}


This is because the {{ssl_lockfile}} exists and is held by the daemon:

{noformat}

# ls $(puppet config print ssl_lockfile)
/etc/puppetlabs/puppet/ssl/ssl.lock

# echo $(cat $(puppet config print ssl_lockfile))
10460

# systemctl show -p MainPID puppet
MainPID=10460
{noformat}
### h3. Expected Outcome


The daemon releases the {{ssl_lockfile}} when it is not actively updating certificate state.

Josh Cooper (JIRA)

unread,
Jan 9, 2020, 12:11:03 PM1/9/20
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-10228
 
Re: Puppet 6 daemons do not release ssl_lockfile

Yeah, this was intentionally done this way as originally the state machine didn't start over from the very beginning, instead it just tried to generate the CSR and submit it each time. Later I realized that the ca cert/crl may have been wiped, so we now start over from the beginning https://github.com/puppetlabs/puppet/blob/605187329a42e200d011cbccfd9e79eb4de02145/lib/puppet/ssl/state_machine.rb#L283-L286 in which case it would be safe to release the lock before sleeping and try to reacquire after.

Josh Cooper (JIRA)

unread,
Feb 3, 2020, 1:19:03 PM2/3/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Change By: Josh Cooper
Sprint: Coremunity Hopper

Josh Cooper (JIRA)

unread,
Feb 3, 2020, 1:20:03 PM2/3/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.13.0

Josh Cooper (JIRA)

unread,
Feb 3, 2020, 1:21:03 PM2/3/20
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-10228
 
Re: Puppet 6 daemons do not release ssl_lockfile

The ssl lockfile was added in 6.5.0, so this doesn't need to go into 6.4.x, which will be EOL in this month.

Josh Cooper (JIRA)

unread,
Feb 7, 2020, 8:20:04 PM2/7/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.13.0
Fix Version/s: PUP 6.14.0

Josh Cooper (Jira)

unread,
Mar 2, 2020, 1:19:03 PM3/2/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.14.0
Fix Version/s: PUP 6.15.0
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Josh Cooper (Jira)

unread,
Apr 2, 2020, 7:55:03 PM4/2/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Sprint: Coremunity Hopper Platform Core KANBAN

Josh Cooper (Jira)

unread,
Apr 2, 2020, 7:55:04 PM4/2/20
to puppe...@googlegroups.com
Josh Cooper assigned an issue to Josh Cooper
Change By: Josh Cooper
Assignee: Josh Cooper

Josh Cooper (Jira)

unread,
Apr 3, 2020, 5:45:04 PM4/3/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Release Notes: Bug Fix
Release Notes Summary: Previously, daemonized puppet agents kept the ssl lockfile locked while waiting for the puppetserver to issue their certificate. That prevented foreground puppet commands from working. Now the agent will release the ssl lockfile while it sleeps and reacquire the lock when it wakes up.

Melissa Stone (Jira)

unread,
Apr 16, 2020, 2:10:03 PM4/16/20
to puppe...@googlegroups.com

Melissa Stone (Jira)

unread,
Apr 17, 2020, 12:33:05 PM4/17/20
to puppe...@googlegroups.com

This has passed ci with puppet-agent 6.14.0.285.ge81148b6

Claire Cadman (Jira)

unread,
Apr 27, 2020, 8:34:03 AM4/27/20
to puppe...@googlegroups.com
Claire Cadman updated an issue
 
Change By: Claire Cadman
Labels: doc_reviewed
Reply all
Reply to author
Forward
0 new messages