Jira (PUP-8014) Create a setting to clear short-lived puppet environments from memory

11 views
Skip to first unread message

Josh Cooper (Jira)

unread,
Oct 13, 2020, 7:03:03 PM10/13/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Puppet / Improvement PUP-8014
Create a setting to clear short-lived puppet environments from memory
Change By: Josh Cooper
Summary: Create an environment_ttl a setting to clear short-lived puppet environments from memory
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Josh Cooper (Jira)

unread,
Oct 13, 2020, 7:07:04 PM10/13/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Release Notes Summary: Adds a new setting ` environment_ttl environment_timeout_mode ` which is controls how the environment cache like the `environment_timeout` setting behaves for values greater than 0 and less than unlimited . However By default , puppet

If
` environment_ttl environment_timeout ` is 0 or unlimited, then there is no change in behavior, environments will evict be never cached or always cached, respectively. For other values, environments from the cache if they haven't been recently used to compile a catalog within the last will be evicted ` environment_ttl environment_timeout ` seconds . This is different from the `environment_timeout` setting which will evict environments at a specific time, regardless of whether the environment has been recently when it was
      was last
used or not .

For the `environment_ttl` setting, a value of `0` will disable caching. This setting can also be set to`unlimited`, which will cache environments until the server is restarted or told to refresh the cache. All other values will result in Puppet server evicting environments that have not been accessed within that ttl.

You should change this setting once your Puppet deployment is doing non-trivial work. We chose the default value of `0` because it lets new users update their code without any extra steps, but it lowers the performance of your Puppet server.

We recommend setting this to a number that will keep your most actively used environments cached, but allow testing environments to fall out of the cache and reduce memory usage. A value of 3 minutes (3m) is a reasonable value.

Once you set `environment_ttl` to a non-zero value, you need to tell Puppet server to read new code from disk using the `environment-cache` API endpoint after you deploy new code. See the docs for the Puppet Server [administrative API](https://puppet.com/docs/puppetserver/latest/admin-api/v1/environment-cache.html).

If the `environment_ttl` setting is explicitly configured in `puppet.conf` then it will override whatever global `environment_timeout` is set in `puppet.conf` and any per-environment `environment_timeout` set in `$codedir/environments/<environment>/environment.conf`.

Josh Cooper (Jira)

unread,
Oct 13, 2020, 7:22:02 PM10/13/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Release Notes Summary: Adds a new setting `environment_timeout_mode` which is controls how the `environment_timeout` setting behaves for values greater than 0 and less than unlimited. By default, which makes it possible to clear short-lived puppet environments from memory, while caching frequently used environments indefinitely. Doing so will reduce Puppet server's memory footprint.

If `environment_timeout` is 0 or unlimited, then there is no change in behavior
. Otherwise , environments will be never cached or always cached if the `environment_timeout` is a duration , respectively. For other values such as 3 minutes , environments then the environment will be evicted `environment_timeout` seconds from when 3 minutes after it was
      was last used
is created .

For the `environment_ttl` setting
However , a value of if ` 0 environment_timeout_mode ` will disable caching. This setting can also be is set to` unlimited from_last_used `, which will cache environments until then the server is restarted or told to refresh the cache. All other values environment will result in Puppet server evicting environments that have not been accessed within that ttl be evicted 3 minutes from when it was last used . We recommend either:

You should change this setting once your Puppet deployment is doing non-trivial work. We chose the default value of Setting ` 0 environment_timeout ` because it lets new users update their code without any extra steps, but it lowers the performance of to unlimited and explicitly refreshing your Puppet server as part of your code deployment process .

We recommend Or setting this `environment_timeout` to a number that will keep your most actively used environments cached, but allow testing environments to fall out of the cache and reduce memory usage. A value of 3 minutes (3m) is a reasonable value. This option requires setting `environment_timeout_mode` to `from_last_used`.

Once you set `
environment_ttl environment_timeout ` to a non-zero value, you need to tell Puppet server to read new code from disk using the `environment-cache` API endpoint after you deploy new code. See the docs for the Puppet Server [administrative API](https://puppet.com/docs/puppetserver/latest/admin-api/v1/environment-cache.html).

If the `environment_ttl` setting is explicitly configured in `puppet.conf` then it will override whatever global `environment_timeout` is set in `puppet.conf` and any per-environment `environment_timeout` set in `$codedir/environments/<environment>/environment.conf`.

Josh Cooper (Jira)

unread,
Oct 13, 2020, 7:26:03 PM10/13/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
h1.  The Problem

When running puppet server you have two levers to adjust how short-lived Puppet environments are cleared from memory.  

1.  {{max-requests-per-instance}} in {{puppetserver.conf}}
2.  {{environment_timeout}} in {{puppet.conf}}/{{environment.conf}}

Each option has it's own cons, because neither addresses the core problem we want to tackle.  We want to clear Puppet environments from memory when they haven't been used for a while and there's no direct way to do that.  

With {{max-requests-per-instance}} we're destroying a whole JRuby which takes out all of the Puppet environments that were ever loaded into it.  

With {{environment_timeout}} we have to clear environments after a certain amount of time even if that environment is getting lots of catalogs compiled against it right now.  

h1.  The solution

- Create a setting that allows for expiring a Puppet environment after it has not been compiled against for a set period of time.  If we were to call this setting {{environment_ttl}} and we set it to {{30m}} then that would mean that if you ran an agent against your test environment and then then no other agents checked into that environment for 30m then the test environment would be marked expired.   
-

-
This gets to the bottom of the problem customers see when deploying short-lived Puppet environments.   
-

-
This setting would be a global setting not a per-environment setting.   
-

We decided to implement a new {{environment_timeout_mode}} setting that controls how {{environment_timeout}} behaves when its value is greater than 0 and less than unlimited. In 6.x, environments will be expired relative to when they are created. If {{environment_timeout_mode}} equals {{from_last_used}}, environments will be expired relative to when they were last used. The {{environment_timeout_mode}} setting affects values of {{environment_timeout}} set globally in {{puppet.conf}} and per-environment {{$codedir/environments/$environment/environment.conf}}.
h1.  Caveats

The setting would presumably interact with the environment_timeout setting.  This setting could only be applied when the environment_timeout is not 0.  If environment_timeout is 0 then this setting would emit a warning and do nothing.  

When the environment_timeout < environment_ttl then environment_timeout would take precedence because it would mark the environment expired before the environment_ttl code had a chance to run.  

When environment_timeout > environment_ttl then this is the primary use case for the setting and environment_ttl finally has a chance to help expire environments more quickly/efficiently than our current settings allow.  

Josh Cooper (Jira)

unread,
Oct 14, 2020, 1:27:03 PM10/14/20
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Oct 14, 2020, 1:27:04 PM10/14/20
to puppe...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages