Jira (PUP-10266) Ensure environment cache is not invalidated during a lockless deploy

[Maggie Dreyer (JIRA)]

Maggie Dreyer updated an issue

Puppet / PUP-10266 Ensure environment cache is not invalidated during a lockless deploy Change By: Maggie Dreyer Summary: Ensure environment cache works is not invalidated during a lockless deploy Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Molly Waggett (JIRA)]

Molly Waggett assigned an issue to Molly Waggett

Puppet / PUP-10266 Ensure environment cache is not invalidated during a lockless deploy

Change By: Molly Waggett Assignee: Molly Waggett

Add Comment

[Molly Waggett (JIRA)]

Molly Waggett assigned an issue to Unassigned

[Maggie Dreyer (JIRA)]

Maggie Dreyer updated an issue

Puppet / PUP-10266 Ensure environment cache is not invalidated during a lockless deploy

Change By: Maggie Dreyer We need to ensure that the an environment cache entry persists throughout an agent run. The environment cache gets invalidated (in all JRubies/all threads) by Puppet Server when a compile DELETE request is sent to the {{environment_cache}} endpoint . File-sync hits this endpoint when it finishes doing a deploy, to ensure that Puppet will pick up the newest version of the code. If we re-create an environment during each deploy are not locking the JRuby pool anymore when this happens , a compile running with there is the potential for agent runs to still be in progress that are using an old existing version of the environment could then be accessing new settings/info from cache env . So we don't want to flush out that version until we're sure that it is no longer in use. One potential solution would Furthermore, we want to be able to create brand new use the same caching mechanism when running both with and without versioned environments for each deploy /lockless deploys , e to avoid unnecessary conditional logic that will have to be ripped out later when we eventually stop supporting unversioned environments . g This will probably involve a restructure of the environment cache defined [here|https://github . com/puppetlabs/puppet/blob/master/lib/puppet/environments.rb#L304]. It is currently a mapping of name to {{ production_<sha1> Environment }} , {{production_<sha2>}}, etc objects . as opposed So assuming we need to recreating continue to support keying off of name {{ production name }} on as is, we need another way to store one or more versions associated with each deploy name .

Add Comment

[Maggie Dreyer (JIRA)]

Maggie Dreyer updated an issue

Puppet / PUP-10266 Ensure environment cache is not invalidated during a lockless deploy Change By: Maggie Dreyer

We need to ensure that an environment cache entry persists throughout an agent run. The environment cache gets invalidated (in all JRubies/all threads) by Puppet Server when a DELETE request is sent to the {{environment_cache}} endpoint. File-sync hits this endpoint when it finishes doing a deploy, to ensure that Puppet will pick up the newest version of the code. If we are not locking the JRuby pool anymore when this happens, there is the potential for agent runs to still be in progress that are using an existing version of the env. So we don't want to flush out that version until we're sure that it is no longer in use. Furthermore, we want to be able to use the same caching mechanism when running both with and without versioned environments/lockless deploys, to avoid unnecessary conditional logic that will have to be ripped out later when we eventually stop supporting unversioned environments. This will probably involve a restructure of the environment cache defined [here|https://github.com/puppetlabs/puppet/blob/master/lib/puppet/environments.rb#L304]. It is currently a mapping of {{<env name >}} to {{Environment}} objects. So assuming If we need are going to continue to support saving multiple versions, keying solely off of name is probably insufficient. Ideas: * overloaded versions of {{ name get }} as is, we need another way to store one that either take just the name (and return the "latest" env) or more versions associated with each the name and desired version . * change the key to no longer just be the name, but perhaps a different string like the name and version concatenated.

Add Comment

[Maggie Dreyer (JIRA)]

Maggie Dreyer updated an issue

Puppet / PUP-10266 Ensure environment cache is not invalidated during a lockless deploy Change By: Maggie Dreyer

We need to ensure that an environment cache entry persists throughout an agent run. The environment cache gets invalidated (in all JRubies/all threads) by Puppet Server when a DELETE request is sent to the {{environment_cache}} endpoint. File-sync hits this endpoint when it finishes doing a deploy, to ensure that Puppet will pick up the newest version of the code. If we are not locking the JRuby pool anymore when this happens, there is the potential for agent runs to still be in progress that are using an existing version of the env. So we don't want to flush out that version until we're sure that it is no longer in use. Furthermore, we want to be able to use the same caching mechanism when running both with and without versioned environments/lockless deploys, to avoid unnecessary conditional logic that will have to be ripped out later when we eventually stop supporting unversioned environments.

This will probably involve a restructure of the environment cache defined [here|https://github.com/puppetlabs/puppet/blob/master/lib/puppet/environments.rb#L304]. It is currently a mapping of {{<env name>}} to {{Environment}} objects. If we are going to support saving multiple versions, keying solely off of name is probably insufficient. Ideas: * overloaded versions of {{get}} that either take just the name (and return the "latest" env) or the name and desired version.

* change the key to no longer just be the name, but perhaps a different string like the name and version concatenated.

Only places associated with an agent run are likely to need to know what version of the env to use. So it seems like the first option might be better.

Add Comment

[Maggie Dreyer (JIRA)]

Only places associated with an agent run are likely to need to know what version of the env to use. So it seems like the first option might be better , though this will also impact the {{get_conf}} family of methods, since they call through to {{get}} .

Add Comment

[Maggie Dreyer (JIRA)]

Maggie Dreyer updated an issue

Puppet / PUP-10266 Ensure environment cache is not invalidated during a lockless deploy Change By: Maggie Dreyer

We need to ensure that an environment cache entry persists throughout an agent run. The environment cache gets invalidated (in all JRubies/all threads) by Puppet Server when a DELETE request is sent to the {{environment_cache}} endpoint. File-sync hits this endpoint when it finishes doing a deploy, to ensure that Puppet will pick up the newest version of the code. If we are not locking the JRuby pool anymore when this happens, there is the potential for agent runs to still be in progress that are using an existing version of the env. So we don't want to flush out that version until we're sure that it is no longer in use. Furthermore, we want to be able to use the same caching mechanism when running both with and without versioned environments/lockless deploys, to avoid unnecessary conditional logic that will have to be ripped out later when we eventually stop supporting unversioned environments. This will probably involve a restructure of the environment cache defined [here|https://github.com/puppetlabs/puppet/blob/master/lib/puppet/environments.rb#L304]. It is currently a mapping of {{<env name>}} to {{Environment}} objects. If we are going to support saving multiple versions, keying solely off of name is probably insufficient. Ideas:

* overloaded versions of {{get}} that either take just the name (and return the "latest" env) or the name and desired version , and have the cache be a nested hash . The most recent version of the env should be stored both in a "latest" key so it can be used as a default, and under a key with its version. * change the key to no longer just be the name, but perhaps a different string like the name and version concatenated , and keep the same internal data structure we have now . This requires that if an environment is cached without a version, nothing later will try to access it with a version, and vice versa. Only places associated with an agent run are likely to need to know what version of the env to use. So it seems like the first option might be better, though this will also impact the {{get_conf}} family of methods, since they call through to {{get}}.

Add Comment