Default environment_timeout preference

[Owen Rodabaugh]

We've been discussing what the default environment_timeout setting should be. There is general agreement that the current 3 minutes is not great. It's both baffling to new users and does not bring in the full performance benefits.

Two main perspectives on this:

1. Performance should be the primary driver and that the default of unlimited (cache never automatically refreshes) is preferred. This assumes most users have a code deployment workflow and tooling which can be adjusted to include the steps required to update the cache. These steps are either hitting the puppetserver environment cache endpoint, or restarting the service to cause the cache to update.

2. New user experience should be the primary driver and that a default of 0 (caching off) is preferred. This assumes brand new users will be baffled when they create or modify puppet code on the server and do not see it take effect during their test runs. By the time users encounter performance problems they will be more familiar with puppet, and this setting will be found when they dig into tuning.

This setting can be managed per environment, and I'm guessing experienced users will do so. This question is focused on the out of the box defaults.

Appreciate your thoughts.

Owen

[Trevor Vaughan]

Can you use inotify to invalidate the cache via the API call when selected files in your infrastructure change?

It looks like you could do this directly from the core https://launchpad.net/inotify-java. You'll just want to queue them up a bit to not go crazy. 10 seconds should probably do it, but you could make that configurable.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/f524207b-9dff-4e57-a46e-08bd31c640e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[Owen Rodabaugh]

To clarify, I am asking for opinions on whether the default environment_timeout should be 0 or unlimited in future releases of puppet.  The current plan is to default to unlimited. 

I'm concerned that shipping with this default assumes prior experience and will be another hurdle to getting started with puppet. Anecdotally I've heard that a common question in #puppet is "I changed my puppet code, why isn't it showing when I do a test run?".

Conversely setting environment_timeout=0 will result in lower performance, but no need to restart puppet or hit the API to flush a cache to see code changes. The users impacted by this are likely more experienced and would already be managing, or easily able to manage this setting if they had performance concerns or a pre-existing code deployment workflow.

Thanks,

Owen

[Trevor Vaughan]

Oh...in that case I vote for unlimited *if and only if*, I don't have to hack-fu my own script to clear the environment via an API call.

'puppet environment refresh' would be my recommendation.

Thanks,

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/08a028ba-4d8f-4223-8134-45404cd367ce%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Adrien Thebo]

To me, following the principle of least astonishment indicates that caching be disabled by default; it'll work correctly for new users and has no hidden gotchas. When people want to do performance tuning they're probably fairly sophisticated users and can deal with weird cache invalidation issues; since they're opting into this feature they should be prepared to deal with the ramifications.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/08a028ba-4d8f-4223-8134-45404cd367ce%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Adrien Thebo | Puppet Labs

[Eric Shamow]

+1 to this suggestion.

-Eric

-- 
Eric Shamow
Sent with Airmail

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALVJ9SJJyVA5utu6vjSvRoOj1J3HiSxezfJmjL4jK4KWm2F7nA%40mail.gmail.com.

[David Schmitt]

This.

On 2015-03-06 03:42, Adrien Thebo wrote:
> To me, following the principle of least astonishment indicates that
> caching be disabled by default; it'll work correctly for new users and
> has no hidden gotchas. When people want to do performance tuning they're
> probably fairly sophisticated users and can deal with weird cache
> invalidation issues; since they're opting into this feature they should
> be prepared to deal with the ramifications.

Regards, David

--
* Always looking for people I can help with awesome projects *
Twitter: @dev_el_ops G+: https://plus.google.com/+DavidSchmitt
Blog: http://club.black.co.at/log/
LinkedIn: http://at.linkedin.com/in/davidschmitt

[John Bollinger]

On Thursday, March 5, 2015 at 8:42:34 PM UTC-6, Adrien Thebo wrote:

To me, following the principle of least astonishment indicates that caching be disabled by default; it'll work correctly for new users and has no hidden gotchas. When people want to do performance tuning they're probably fairly sophisticated users and can deal with weird cache invalidation issues; since they're opting into this feature they should be prepared to deal with the ramifications.

+1

Perhaps it would make sense for PE to be configured differently out of the box, but between the two options presented, Puppet's built-in default should be no caching.


John

[Reid Vandewiele]

The principle of least astonishment is absolutely what we should be targeting. The use of any kind of timer upon whose ticks behavior changes is in inarguable opposition to this, whether it's 10 seconds, 3 minutes, or 15 minutes. However, I think the use of implementation terms like "caching" in describing the two proposed options clouds the water a bit, as both options result in clear and consistent behavior.

Described without the term "caching", option 1 effectively proposes that puppet-server should read all configuration files on startup (puppet.conf, auth.conf, fileserver.conf, environments/**/*.pp, etc.), and not automatically re-read any of these files unless the user issues an explicit `service puppet-server reload` command.

Described without the term "caching", option 2 effectively proposes that puppet-server should read some configuration files on startup (puppet.conf, auth.conf, fileserver.conf, etc.), and that puppet-server should read directly from disk all relevant files from environments/**/*.pp when compiling a catalog, once per agent request.

Both proposals move strongly away from the problem behavior we have today - a clock-based timeout. Were this a new product, either option seems like it would satisfy principle of least astonishment. The only difference between them, from an astonishment perspective, is that long-time Puppet users are accustomed to behavior resembling option 2 (though in the past implementation it's been more optimized, similar to the inotify suggestion put forth by Trevor).

I don't know that thinking about it this way changes anyone's opinion, but I do want to make sure we aren't getting hung up on implementation terms in considering what the actual proposed behaviors are.

Is having files live-updated (via guaranteed re-read) a value proposition itself? There seem to be some minor benefits to a reload-required behavior. In-progress requests are guaranteed not to get half files from one revision and half files from another if the catalog request timing is particularly bad, since the user won't issue a reload command mid-file-update, and if they do puppet-server will flush in-progress requests anyway. Are those benefits worth considering proposal 2 for?

Does this perspective make sense, or am I missing something?

~Reid

[John Bollinger]

On Friday, March 6, 2015 at 11:36:03 AM UTC-6, Reid Vandewiele wrote:

The principle of least astonishment is absolutely what we should be targeting. The use of any kind of timer upon whose ticks behavior changes is in inarguable opposition to this, whether it's 10 seconds, 3 minutes, or 15 minutes. However, I think the use of implementation terms like "caching" in describing the two proposed options clouds the water a bit, as both options result in clear and consistent behavior.

Described without the term "caching", option 1 effectively proposes that puppet-server should read all configuration files on startup (puppet.conf, auth.conf, fileserver.conf, environments/**/*.pp, etc.), and not automatically re-read any of these files unless the user issues an explicit `service puppet-server reload` command.

Described without the term "caching", option 2 effectively proposes that puppet-server should read some configuration files on startup (puppet.conf, auth.conf, fileserver.conf, etc.), and that puppet-server should read directly from disk all relevant files from environments/**/*.pp when compiling a catalog, once per agent request.

It would also be clear and consistent to say that when a manifest is changed, those changes will start being reflected in catalogs emitted by the master within 3 minutes (or 1 or 20).  The exact timing is not quite as predictable, but the behavior can still be given as a rule, and without using any variant of the word "cache".

 

Both proposals move strongly away from the problem behavior we have today - a clock-based timeout. Were this a new product, either option seems like it would satisfy principle of least astonishment.

Both do move away from the default behavior of 3.7, but I don't see how you can support a claim that either option would provide least astonishment, particularly inasmuch as you also claim that a timeout is more astonishing than either of the other alternatives.  I do not find it self-evident, for example, that most users would be more astonished that Puppet eventually notices manifest changes, than that they have to perform some kind of manual action separate from the change itself to make Puppet notice changes.

 

The only difference between them, from an astonishment perspective, is that long-time Puppet users are accustomed to behavior resembling option 2 (though in the past implementation it's been more optimized, similar to the inotify suggestion put forth by Trevor).

I disagree.  Puppet has always required a restart to recognize changes to the Ruby code of server-side custom components, but that still throws people -- sometimes even people who should know better.  Not every possible behavior is equally good or predictable, even when you start with a clean slate, even when you can state it as a clear and consistent rule.  (And in particular, people seem pre-disposed to expect file changes to be recognized immediately.)

 

I don't know that thinking about it this way changes anyone's opinion, but I do want to make sure we aren't getting hung up on implementation terms in considering what the actual proposed behaviors are.

I don't think the discussion is (or need be) hung up on implementation terms.  I think Adrien had it exactly right that most users will be less surprised by manifest changes taking effect immediately than by them not taking effect until some form of a `notice-my-changes` command is executed.

 

Is having files live-updated (via guaranteed re-read) a value proposition itself?

From a least-astonishment perspective, yes.  From some other perspectives, too.

 

There seem to be some minor benefits to a reload-required behavior. In-progress requests are guaranteed not to get half files from one revision and half files from another if the catalog request timing is particularly bad, since the user won't issue a reload command mid-file-update, and if they do puppet-server will flush in-progress requests anyway.

You are right that those are issues for which reload-required behavior is a win, but not on least-astonishment grounds.  Are you stepping back from your position that "least astonishment is absolutely what we should be targeting"?

John

[Eric Sorenson]

This feels quite a bit like consensus to me, I've filed https://tickets.puppetlabs.com/browse/PUP-4094 to track the change which should hit both in Puppet 3.8 and the Puppet 4 nightlies.

Thank you Owen for kicking off the discussion, and thanks everyone for your thoughtful replies.

Eric Sorenson - eric.s...@puppetlabs.com - freenode #puppet: eric0
puppet platform // coffee // techno // bicycles

[Reid Vandewiele]

As Eric said there seems to be clear consensus and an issue has been opened to make the change. I think it is still be useful for me to respond in detail to John, but just to wrap up the thoughts - not to further advocate for reload behavior. There seem to be good reasons to choose to serve files live as opposed to read-on-start.

On Fri, Mar 6, 2015 at 11:06 AM, John Bollinger <john.bo...@stjude.org> wrote:

On Friday, March 6, 2015 at 11:36:03 AM UTC-6, Reid Vandewiele wrote:

The principle of least astonishment is absolutely what we should be targeting. The use of any kind of timer upon whose ticks behavior changes is in inarguable opposition to this, whether it's 10 seconds, 3 minutes, or 15 minutes. However, I think the use of implementation terms like "caching" in describing the two proposed options clouds the water a bit, as both options result in clear and consistent behavior.

Described without the term "caching", option 1 effectively proposes that puppet-server should read all configuration files on startup (puppet.conf, auth.conf, fileserver.conf, environments/**/*.pp, etc.), and not automatically re-read any of these files unless the user issues an explicit `service puppet-server reload` command.

Described without the term "caching", option 2 effectively proposes that puppet-server should read some configuration files on startup (puppet.conf, auth.conf, fileserver.conf, etc.), and that puppet-server should read directly from disk all relevant files from environments/**/*.pp when compiling a catalog, once per agent request.

It would also be clear and consistent to say that when a manifest is changed, those changes will start being reflected in catalogs emitted by the master within 3 minutes (or 1 or 20).  The exact timing is not quite as predictable, but the behavior can still be given as a rule, and without using any variant of the word "cache".

I'll cede consistent, but I continue to hold that clock-based behavior lacks clarity. The problem is not that the behavior can't be calculated, it's that to a new user experimenting with the product the clock-based behavior hinders their ability to engage in adjust/observe/iterate experimentation. If I'm trying to understand what a product is doing, I want to make a change and observe the result to see if the change I made did what I expected. I want confidence that if I make an adjustment and a change is observed, it is due to my adjustment. This is a feedback loop.

The problem with a clock is that after I make my adjustment, there usually won't be any observed change immediately. But, there will be change later. At best, my feedback loop is delayed and it takes me longer to understand what's going on. More realistically, this kind of inconsistent feedback makes me frustrated with the product and don't have confidence that what I'm doing is having the expected effect. After I figure it out, I can work with it.

 

 

Both proposals move strongly away from the problem behavior we have today - a clock-based timeout. Were this a new product, either option seems like it would satisfy principle of least astonishment.

Both do move away from the default behavior of 3.7, but I don't see how you can support a claim that either option would provide least astonishment, particularly inasmuch as you also claim that a timeout is more astonishing than either of the other alternatives.  I do not find it self-evident, for example, that most users would be more astonished that Puppet eventually notices manifest changes, than that they have to perform some kind of manual action separate from the change itself to make Puppet notice changes.

My assertion that either option provides a least-astonishment experience is indeed built on a belief that clock-based system is Bad (TM), which I first took as a given. I've provided more context for why I believe this to be the case above. Even allowing that this assumption is incorrect, I still believe that option 1 and 2 are much more similar to each other in terms of first-time-user astonishment than either is to clock-based, since both give users consistent feedback.

Users expect to have to restart or refresh most services if their configuration file(s) change. I have been considering puppet manifests similar to configuration files in this way. It sounds like most people would more intuitively consider manifests more like *.php files, or something to be served. The general opinion in this thread supports that. Given that starting point, I can see how it would be less astonishing to people if changes made to those files were immediately impactful, rather than requiring a reload.

 

 

The only difference between them, from an astonishment perspective, is that long-time Puppet users are accustomed to behavior resembling option 2 (though in the past implementation it's been more optimized, similar to the inotify suggestion put forth by Trevor).

I disagree.  Puppet has always required a restart to recognize changes to the Ruby code of server-side custom components, but that still throws people -- sometimes even people who should know better.  Not every possible behavior is equally good or predictable, even when you start with a clean slate, even when you can state it as a clear and consistent rule.  (And in particular, people seem pre-disposed to expect file changes to be recognized immediately.)

 

I don't know that thinking about it this way changes anyone's opinion, but I do want to make sure we aren't getting hung up on implementation terms in considering what the actual proposed behaviors are.

I don't think the discussion is (or need be) hung up on implementation terms.  I think Adrien had it exactly right that most users will be less surprised by manifest changes taking effect immediately than by them not taking effect until some form of a `notice-my-changes` command is executed.

 

Is having files live-updated (via guaranteed re-read) a value proposition itself?

From a least-astonishment perspective, yes.  From some other perspectives, too.

 

There seem to be some minor benefits to a reload-required behavior. In-progress requests are guaranteed not to get half files from one revision and half files from another if the catalog request timing is particularly bad, since the user won't issue a reload command mid-file-update, and if they do puppet-server will flush in-progress requests anyway.

You are right that those are issues for which reload-required behavior is a win, but not on least-astonishment grounds.  Are you stepping back from your position that "least astonishment is absolutely what we should be targeting"?

Not at all. I was however coming from the perspective that both options were equivalent on astonishment-grounds, and so was searching for other differentiators. As the consensus is that the two options are not equivalent on astonishment grounds, these other small differences are not important at this level.

 

John

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-dev/lnbXyHlviT0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/874f8f69-45c6-4034-b7c9-06005dc6f963%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

~Reid