Jira (PUP-10257) Write timestamp to file in environment when beginning compilation

13 views
Skip to first unread message

Maggie Dreyer (JIRA)

unread,
Jan 27, 2020, 6:09:04 PM1/27/20
to puppe...@googlegroups.com
Maggie Dreyer created an issue
 
Puppet / Task PUP-10257
Write timestamp to file in environment when beginning compilation
Issue Type: Task Task
Assignee: Unassigned
Created: 2020/01/27 3:08 PM
Priority: Normal Normal
Reporter: Maggie Dreyer

In order to clean up old versions of the code dir when using the new file-sync layout, we need to know whether or not those versions are still in use for compilation. Since not all systems report access time for files, we need to record this metadata ourselves. Puppet should write the current time to a file in the environment when it begins a compilation for that env. The file-sync client can then later reference this file to decide whether or not to purge the directory.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Henrik Lindberg (JIRA)

unread,
Jan 28, 2020, 11:06:04 AM1/28/20
to puppe...@googlegroups.com
Henrik Lindberg commented on Task PUP-10257
 
Re: Write timestamp to file in environment when beginning compilation

Would this not then create a problem with not knowing if that compile has finished, and would this not also create a race condition (just after checking a compilation starts).

Molly Waggett (JIRA)

unread,
Jan 28, 2020, 12:24:05 PM1/28/20
to puppe...@googlegroups.com

Whatever start time we record, I think the idea was to not purge unless some (yet-to-be-determined) amount of time has passed, where we're confident the compile would be finished.

We considered recording when the compile has finished, but that definitely creates a race condition if we do the check while a compile is in progress and hasn't recorded its end time yet. But I think you're right that we didn't consider the case where we do the time check and then a compile starts before the code dir is actually purged.. We did decide that we'd always keep version n-1, where version n is the most recent for a given environment, in the hopes that this would minimize the chance that we'd purge a version that was still being used.

Does that seem sane to you?

Nick Walker (JIRA)

unread,
Jan 28, 2020, 12:40:04 PM1/28/20
to puppe...@googlegroups.com
Nick Walker commented on Task PUP-10257

I don't quite understand this proposal.  Would it make sense to instead write a timestamp to the n-1 version of code when the n version of code is being deployed?  So basically you write a timestamp indicating when the code was superseded as opposed to writing a timestamp at the beginning of every compile?  

I'd imagine the ideal state is you write down metadata on all the compiles that are currently happening at the time you deploy new code and then during a garbage collection process later you just confirm the compiles that were happening at the time you deployed the new code are no longer running.  My guess would be we don't have a way to know the compiles that were running or a way to confirm they are no longer running?  

 

Molly Waggett (JIRA)

unread,
Jan 28, 2020, 1:36:04 PM1/28/20
to puppe...@googlegroups.com

Just because a new version has been deployed doesn't necessarily mean that older versions aren't still in use, like if you do a bunch of deploys in a short period of time.

Henrik Lindberg (JIRA)

unread,
Jan 28, 2020, 1:54:04 PM1/28/20
to puppe...@googlegroups.com

Nick Walker, I think that makes more sense as well - then there is just an entry barrier that is checked before compiling - I suppose that means "go over there instead" - or is that switch over to another version done based on something else?

To me it seems that a compiling server would be able to keep track of the number of compiles per version that it has ongoing (there are all active in memory, so if it dies they cannot be ongoing). If there are multiple servers acting on the same information it gets more complicated.

Justin Stoller (JIRA)

unread,
Feb 18, 2020, 5:12:05 PM2/18/20
to puppe...@googlegroups.com

I like the idea of file-sync writing a file in versioned environment when that environment is superseded and then cleaning up any environment that was superseded more than X amount of time ago (30 minutes?). This also has the effect of limiting changes to the Puppet compiler (only those running PE will see this artifact).

Nick Walker (JIRA)

unread,
Feb 19, 2020, 11:39:06 AM2/19/20
to puppe...@googlegroups.com
Nick Walker commented on Task PUP-10257

Justin Stoller should we rewrite the ticket description/title and move it to server?

Reply all
Reply to author
Forward
0 new messages