Jira (PUP-5609) Puppet apply doesn't protect shared files with a lock

[Erik Dalén (JIRA)]

Erik Dalén created an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock Issue Type: Bug Assignee: Unassigned Created: 2015/12/14 12:16 PM Priority: Normal Reporter: Erik Dalén Puppet apply should use a filesystem lock just like agent to protect shared state files from simultaneous writes (it actually writes to the same state files as agent even). Add Comment

This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc)

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Henrik Lindberg Scrum Team: Client Platform

Add Comment

[Michael Smith (JIRA)]

Michael Smith updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Michael Smith Sprint: Client 2016-02-10

Add Comment

[Michael Smith (JIRA)]

Michael Smith commented on PUP-5609

Re: Puppet apply doesn't protect shared files with a lock Hit a snag in acceptance tests. The tests/resource/service/puppet_mcollective_service_management.rb test bumps into an issue with the lockfile. Failed Tests Cases: Errored Tests Cases: Test Case tests/resource/service/puppet_mcollective_service_management.rb reported: #<Beaker::Host::CommandFailure: Host 'i51djgc66a34vp6.delivery.puppetlabs.net' exited with 1 running: puppet apply --detailed-exitcodes --verbose Last 10 lines of output were: Error: Could not run: undefined local variable or method `client_class' for #<Puppet::Application::Apply:0x00000003014ca0>> client_class isn't defined, so that needs to be fixed. But it shouldn't have hit that exception in the first place.

Add Comment

[Michael Smith (JIRA)]

Michael Smith assigned an issue to Michael Smith

Puppet / PUP-5609

Puppet apply doesn't protect shared files with a lock

Change By: Michael Smith Assignee: Michael Smith

Add Comment

[Michael Smith (JIRA)]

Michael Smith assigned an issue to Unassigned

[Michael Smith (JIRA)]

Michael Smith updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Michael Smith Sprint: Client  2016-02-10  Triage

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Josh Cooper Sprint: Client  Community  Triage

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-5609

Re: Puppet apply doesn't protect shared files with a lock I think we're running into bugs in the file locking code itself (as opposed to the PR that Erik submitted). In particular PUP-2515 and https://projects.puppetlabs.com/issues/21705#note-2.

Add Comment

[Eric Sorenson (JIRA)]

Eric Sorenson updated an issue

Puppet / PUP-5609

Puppet apply doesn't protect shared files with a lock

Change By: Eric Sorenson Sprint: Client Community Triage

Add Comment

This message was sent by Atlassian JIRA (v6.4.13#64028-sha1:b7939e9)

[Eric Sorenson (JIRA)]

Eric Sorenson updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Eric Sorenson Sprint: i18n

Add Comment

[David Kramer (JIRA)]

David Kramer updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: David Kramer Sprint: i18n Client Triage

Add Comment

[Kenn Hussey (JIRA)]

Kenn Hussey updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Kenn Hussey Sprint: Client Triage

Add Comment

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)

[Philipp Gassmann (JIRA)]

Philipp Gassmann commented on PUP-5609

Re: Puppet apply doesn't protect shared files with a lock the apply process should check and set the same lock files as the agent. With --statedir /tmp/foo it should be possible to run it anyways. We now use the following workaround with hardcoded paths: mkdir -p $environment # Puppet 4 requires an environment directory # check puppet agent disable lock file [ -f /opt/puppetlabs/puppet/cache/state/agent_disabled.lock ] && ( echo 'Puppet agent disabled - skipping' >&2; cat /opt/puppetlabs/puppet/cache/state/agent_disabled.lock >&2; echo >&2; exit 1 ) # check and set lockfile for puppet apply ( flock -n 9 || ( echo 'Puppet agent lock file exist - skipping' >&2; exit 1 ) puppet apply \ --test \ --show_diff \ --write-catalog-summary \ --modulepath=modules:site \ --hiera_config=hiera.yaml \ --environmentpath=. \ --environment $environment \ manifests/site.pp "$@" ) 9>/opt/puppetlabs/puppet/cache/state/agent_catalog_run.lock

Add Comment

[Michael Smith (JIRA)]

Michael Smith updated an issue

Puppet / PUP-5609

Puppet apply doesn't protect shared files with a lock

Change By: Michael Smith Sprint: Puppet Triage

Add Comment

[Erik Dalén (JIRA)]

Erik Dalén commented on PUP-5609

Re: Puppet apply doesn't protect shared files with a lock I rebased my PR for this against the master branch: https://github.com/puppetlabs/puppet/pull/4584 So, there is a fix for this waiting for merge

Add Comment

[Michael Smith (JIRA)]

Michael Smith assigned an issue to Michael Smith

Puppet / PUP-5609

Puppet apply doesn't protect shared files with a lock

Change By: Michael Smith Assignee: Michael Smith

Add Comment

[Michael Smith (JIRA)]

Michael Smith updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Michael Smith Sprint: Puppet Triage

Add Comment

[Michael Smith (JIRA)]

Michael Smith updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Michael Smith Fix Version/s: PUP 5.0.0

Add Comment

[Michael Smith (JIRA)]

Michael Smith updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Michael Smith Labels: maintenance

Add Comment

[Michael Smith (JIRA)]

Michael Smith updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Michael Smith Release Notes Summary: Puppet apply now creates a lock file, as puppet agent does. This prevents simultaneously writing shared state files, as could happen if a puppet agent and puppet apply run occurred simultaneously. Release Notes: Bug Fix

Add Comment

[Michael Smith (JIRA)]

Michael Smith commented on PUP-5609

Re: Puppet apply doesn't protect shared files with a lock Thanks Erik Dalén! I'm going to assume we cleaned up some tests that were misbehaving and that's allowed the patch to go through.

Add Comment

[Thomas Mueller (JIRA)]

Thomas Mueller commented on PUP-5609

Re: Puppet apply doesn't protect shared files with a lock

I'm using puppet apply in parallel to manage different nfs homedirectory roots with many users. Sounds like now agent_disabled_lockfile and agent_catalog_run_lockfile need to be set to achieve a parallel run. Might be good to have release notes about this.

Add Comment

[Michael Smith (JIRA)]

Michael Smith updated an issue

Puppet / PUP-5609

Puppet apply doesn't protect shared files with a lock

Change By: Michael Smith Release Notes Summary:

Puppet apply now creates a lock file, as puppet agent does. This prevents simultaneously writing shared state files, as could happen if a puppet agent and puppet apply run occurred simultaneously.

If you have a need to run multiple apply runs simultaneously, you can continue to do so by setting by configuring each run with a distinct {{statedir}}.

Add Comment

[Michael Smith (JIRA)]

Michael Smith commented on PUP-5609

Re: Puppet apply doesn't protect shared files with a lock The recommended approach would actually be to use a distinct statedir for each run. I've added that to the release notes.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-5609

Re: Puppet apply doesn't protect shared files with a lock

I reopened this, because it's been causing problems in CI. Our tests often use puppet apply to manage the puppet service. If the agent is daemonized and is applying a catalog in the background, then calling puppet apply to stop the service will fail with: [root@o3nvm9xl80j4opo ~]# puppet apply -e "service { 'puppet': ensure => stopped }" Notice: Run of Puppet already in progress; skipping (/opt/puppetlabs/puppet/cache/state/agent_catalog_run.lock exists) We could change the tests to specify a unique statedir, most likely via the apply_manifest_on beaker helper, but that is going to take some work. Reopening this until we have a chance to do that (as it affects opensource, PE, and modules pipelines).

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-5609

Puppet apply doesn't protect shared files with a lock

Change By: Geoff Nichols Sprint: Agent Grooming

Add Comment

[Kenn Hussey (JIRA)]

Kenn Hussey updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Kenn Hussey Team: Agent

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Geoff Nichols Sprint: Agent Grooming  On-Deck

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Geoff Nichols Fix Version/s: PUP 5.0.0 Fix Version/s: PUP 5.y

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Geoff Nichols Sprint: Agent Grooming On-Deck

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Henrik Lindberg Labels: maintenance triaged

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Josh Cooper Sub-team: Coremunity

Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Josh Cooper Fix Version/s: PUP 5.y

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Josh Cooper Team: Coremunity Night's Watch

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Bogdan Irimie (Jira)]

Bogdan Irimie updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Bogdan Irimie Sprint:

Add Comment

[Bogdan Irimie (Jira)]

Bogdan Irimie updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Bogdan Irimie Sprint: ready for triage

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Josh Cooper Epic Link: PUP-7532

Add Comment

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)

[Ciprian Badescu (Jira)]

Ciprian Badescu updated an issue

Puppet / PUP-5609 Puppet apply doesn't protect shared files with a lock

Change By: Ciprian Badescu Sprint: ready for triage

Add Comment