Jira (PUP-10285) Puppet upgrade should prevent a Puppet run from occurring

[Lucas Young (JIRA)]

Lucas Young created an issue

Puppet / PUP-10285 Puppet upgrade should prevent a Puppet run from occurring Issue Type: Improvement Affects Versions: PUP 5.5.16, PUP 5.y Assignee: Unassigned Attachments: Screen Shot 2020-02-07 at 2.29.57 PM.png Created: 2020/02/07 12:33 PM Priority: Normal Reporter: Lucas Young Description: Currently when a user runs Puppet while an agent upgrade is in progress on a Windows system, it will allow Puppet to run, but fail somewhere in the process as a result of the installation. This process appears to generate various error messages because it will fail at different times in the Puppet run process in an inconsistent manner.   Steps to reproduce: Perform an agent upgrade on a Windows system, during the upgrade process, attempt to run Puppet manually, which will cause the failure to happen.   Expected Results: Have Puppet check for the upgrade / installation Windows lock file and only proceed if it is not in place. If the lock file is in place when a Puppet run is initiated, it will not proceed and generate a message stating "Aborted, Puppet run in progress". Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Austin Boyd (JIRA)]

Austin Boyd updated an issue

Puppet / PUP-10285 Puppet upgrade should prevent a Puppet run from occurring

Change By: Austin Boyd Zendesk Ticket IDs: 37989 Zendesk Ticket Count: 1

Add Comment

[Austin Boyd (JIRA)]

Austin Boyd updated an issue

Puppet / PUP-10285 Puppet upgrade should prevent a Puppet run from occurring

Change By: Austin Boyd Labels: jira_escalated

Add Comment

[Lucas Young (JIRA)]

Lucas Young updated an issue

Puppet / PUP-10285 Puppet upgrade should prevent a Puppet run from occurring

Change By: Lucas Young Attachment:

Screen Shot 2020-02-07 at 2.29.57 PM.png

Add Comment

[Lucas Young (JIRA)]

[Lucas Young (JIRA)]

[Lucas Young (JIRA)]

[Austin Boyd (JIRA)]

Austin Boyd updated an issue

Puppet / PUP-10285 Puppet upgrade should prevent a Puppet run from occurring

Change By: Austin Boyd Labels: jira_escalated

Add Comment

[Lucas Young (JIRA)]

Lucas Young updated an issue

Puppet / PUP-10285 Puppet upgrade should prevent a Puppet run from occurring

Change By: Lucas Young Comment: A comment with security level 'Developers' was removed.

Add Comment

[Lucas Young (JIRA)]

Lucas Young updated an issue

Puppet / PUP-10285 Puppet upgrade should prevent a Puppet run from occurring

Change By: Lucas Young

Zendesk Ticket IDs: 37989 Zendesk Ticket Count: 1

Add Comment

[Lucas Young (JIRA)]

Lucas Young updated an issue

Puppet / PUP-10285 Puppet upgrade should prevent a Puppet run from occurring

Change By: Lucas Young

Attachment: Screen Shot 2020-02-07 at 2.29.57 PM.png

Add Comment

[Lucas Young (JIRA)]

Lucas Young updated an issue

Puppet / PUP-10285 Puppet upgrade should prevent a Puppet run from occurring Change By: Lucas Young

*Description:* Currently when a user runs Puppet while an agent upgrade is in progress on a Windows system, it will allow Puppet to run, but fail somewhere in the process as a result of the installation.

This process appears to generate various error messages because it will fail at different times in the Puppet run process in an inconsistent manner.

*Steps to reproduce*: Perform an agent upgrade on a Windows system using the standard Windows MSI Installation , during the upgrade process, attempt to run Puppet manually, which will cause the failure to happen.   *Expected Results*: Have Puppet check for the upgrade / installation Windows lock file and only proceed if it is not in place. If the lock file is in place when a Puppet run is initiated, it will not proceed and generate a message stating "Aborted, Puppet run in progress".

Add Comment

[Austin Boyd (JIRA)]

Austin Boyd updated an issue

Puppet / PUP-10285 Puppet upgrade should prevent a Puppet run from occurring

Change By: Austin Boyd Labels: jira_escalated

Add Comment

[Austin Boyd (JIRA)]

Austin Boyd updated an issue

Puppet / PUP-10285 Puppet upgrade should prevent a Puppet run from occurring

Change By: Austin Boyd

Zendesk Ticket IDs: 37989 Zendesk Ticket Count: 1

Add Comment

[Jorie Tappa (JIRA)]

Jorie Tappa updated an issue

Puppet / PUP-10285 Puppet upgrade should prevent a Puppet run from occurring

Change By: Jorie Tappa Team: Night's Watch

Add Comment

[Lucas Young (JIRA)]

Lucas Young updated an issue

Puppet / PUP-10285 Puppet upgrade should prevent a Puppet run from occurring

Change By: Lucas Young

Zendesk Ticket IDs: 37989 Zendesk Ticket Count: 1

Add Comment

[Austin Boyd (JIRA)]

Austin Boyd updated an issue

Puppet / PUP-10285 Puppet upgrade should prevent a Puppet run from occurring

Change By: Austin Boyd Labels: jira_escalated

Add Comment

[Michael Hudson (JIRA)]

Michael Hudson commented on PUP-10285

Re: Puppet upgrade should prevent a Puppet run from occurring Being from a team that initiated the conversation that resulted in this ticket being created, here are my thoughts on this. I am no Windows expert so please take this with a grain of salt. I would prefer that the Windows upgrade be done during a Puppet run instead of triggering a script that does the upgrade outside of a Puppet run, if possible. The way I understand it, this would mean "staging" the newly installed version until it is complete and then taking some action to make that version active (this action may have to be taken after the Puppet run is complete but would hopefully be quick instead of the current process which takes 5+ minutes for the upgrade to complete). Doing this would mean that the Puppet run that triggers the upgrade would stay running while the new version is installed instead of allowing the opportunity that exists now where a new Puppet run could be triggered which would then get killed by the upgrade process. This would help in places where other automation is used to trigger Puppet runs outside of the service scheduled runs.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-10285

Re: Puppet upgrade should prevent a Puppet run from occurring

It's not possible to stage the upgrade and then atomically switch over given that we rely on Windows Installer (MSI) to perform the upgrade. Another complicating issue is Windows does not allow a program to be overwritten while it is running, unlike *nix. In particular we can't overwrite ruby.exe or any of its loaded dlls while it's running. I think the next best thing would be to have the pupet_agent module drop a lock file in place prior to running and cleaning it up when it's done.

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Michael Hudson (Jira)]

Michael Hudson commented on PUP-10285

Re: Puppet upgrade should prevent a Puppet run from occurring

Josh Cooper I am not a Windows expert but not sure I understand why this cannot be done when there are lots of other Windows applications that can upgrade while they are running and then they use the new version when they restart. I know I have seen applications that will do an upgrade in the background and then display a popup that says the application (and sometimes the entire OS) must restart for the new version to load, though if you don't allow the restart the old version still continues to work. I thought I had heard you can do this by locking the current dlls and specifying some flag that the dlls would be replaced after a restart but again I am no expert so maybe I heard wrong. Currently the install script that is triggered after the Puppet run completes already puts a lockfile down so maybe the agent could look for this lockfile when starting to ensure an upgrade is not currently being done. I still think it would be best of the current Puppet run could continue until the upgrade is complete but maybe this is just not possible for some reason.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-10285

Re: Puppet upgrade should prevent a Puppet run from occurring

I am not a Windows expert but not sure I understand why this cannot be done So I agree with your general comment that it would be good to prevent runs from occurring while an upgrade is in progress (and vice-versa), it's just that it can't be done in the way you describe, creating a staging directory and renaming the directory while the puppet ruby process is running.

I thought I had heard you can do this by locking the current dlls and specifying some flag that the dlls would be replaced after a restart but again I am no expert so maybe I heard wrong.

Yep, you can ask windows to move the file when it next reboots (and Windows will store that in the registry in PendingFileRenameOperations), but it's important that puppet-agent be able to upgrade without requiring reboots. All of these reasons are why we have the puppetlabs-puppet_agent module. It spawns a detached process that upgrades the puppet-agent package while puppet/ruby is not running. It does drop a lock file: https://github.com/puppetlabs/puppetlabs-puppet_agent/blob/e85f549f0877387bce97372f137b53fbef60ea12/files/install_puppet.ps1#L271 but it's different than puppet's lock file, which is controlled by the Puppet[:agent_catalog_run_lockfile] setting. Either the module could drop a second lock file in the place that puppet checks, or puppet could check for the installation lockfile that the module drops. I prefer the former, because the module is the one managing the upgrade process, and it already has logic to check if puppet is already running, and to wait for it to finish.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-10285

Puppet upgrade should prevent a Puppet run from occurring

Change By: Josh Cooper Epic Link: PUP-7532

Add Comment

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)