Dealing with transitional states in Puppet

[Reid Vandewiele]

This thread is introducing a simple workaround for an observed limitation in Puppet's ability to automate inelegant but real configuration requirements. The desired outcome is to get feedback on the suitability of the workaround or its approach, how well it fits with Puppet's paradigm, and whether or not people would find it acceptable to implement this kind of pattern in their own environments.

tl;dr: What do people think of https://forge.puppetlabs.com/puppetlabs/transition ?

The Problem:

Puppet has always been tightly focused on a single, final target state. When iterating over resources in the graph, Puppet examines resources one by one and if necessary makes configuration changes to bring them into compliance with that one true target state. It is expected that at the end of the run, every resource in the graph will still be in that target state. This is generally a decent way to model things, but there are some situations where it isn't quite enough. For example, if a running service locks a file (Windows often does this), that file cannot be changed unless the service is stopped. Procedurally, to edit the file one would be expected to stop the service, make the change to the file, and then start the service back up. Similarly, when installing software a procedure may say to download the installation media to a temporary directory, use it to install the software, and finally remove the installation media (as it is no longer necessary to keep it).

Unless the underlying provider has built-in logic that handles those kinds of temporary changes within the context of a single resource, it is very difficult to model these "transitional states" in Puppet. There are a few contexts where it kinda makes sense to extend a provider to encompass a transition state, such as a package provider downloading a source file to keep temporarily, but often times supporting transitional states in the context of a provider would feel like bloat, and not good design.

The Experiment:

What if we had the ability to model transitional states as part of the catalog? Chris Barker, myself and a few others were brainstorming about this a little while back and came up with the idea to insert a resource into the graph that had a kind of reverse-notify behavior, where it would enact a specified state on another resource, temporarily, if and only if another specified resource ahead of it in the graph was going to change. For example:

transition { 'stop myapp service':

  resource   => Service['myapp'],

  attributes => { ensure => stopped },

  prior_to   => File['/etc/myapp/myapp.cfg'],

}

file { '/etc/myapp/myapp.cfg':

  ensure  => file,

  content => 'mycontent',

  notify  => Service['myapp'],

}

service { 'myapp':

  ensure => running,

  enable => true,

}

We implemented a prototype and published it at https://forge.puppetlabs.com/puppetlabs/transition. It's 0.1.0 code, basically first cut, just enough to build out and test the idea, but not all the rough edges are sanded off. There's more detail in the readme on the Forge page.We implemented a prototype and published it at https://forge.puppetlabs.com/puppetlabs/transition. It's 0.1.0 code, basically first cut, just enough to build out and test the idea, but not all the rough edges are sanded off. There's more detail in the readme on the Forge page.

Does this pattern or capability make sense in the general context of Puppet? Is this a decent interim solution for something better currently under development? What do people think of this?

~Reid

[David Schmitt]

On 2014-12-19 22:14, Reid Vandewiele wrote:
> transition { 'stop myapp service':
> resource => Service['myapp'],
> attributes => { ensure => stopped },
> prior_to => File['/etc/myapp/myapp.cfg'],
> }
>
> file { '/etc/myapp/myapp.cfg':
> ensure => file,
> content => 'mycontent',
> notify => Service['myapp'],
> }
>
> service { 'myapp':
> ensure => running,
> enable => true,
> }
>
> We implemented a prototype and published it at
> https://forge.puppetlabs.com/puppetlabs/transition. It's 0.1.0 code,
> basically first cut, just enough to build out and test the idea, but not
> all the rough edges are sanded off. There's more detail in the readme on
> the Forge page.We implemented a prototype and published it at
> https://forge.puppetlabs.com/puppetlabs/transition. It's 0.1.0 code,
> basically first cut, just enough to build out and test the idea, but not
> all the rough edges are sanded off. There's more detail in the readme on
> the Forge page.
>
> Does this pattern or capability make sense in the general context of
> Puppet? Is this a decent interim solution for something better currently
> under development? What do people think of this?

More flexible state management is something that is very much on my mind
in recent times. As other products (like ansible) solve this much better.

Given the restricted malleability of the manifest language, I think your
implementation is already quite advanced and will solve problems.


Regards, David




--
* Always looking for people I can help with awesome projects *
Twitter: @dev_el_ops G+: https://plus.google.com/+DavidSchmitt
Blog: http://club.black.co.at/log/
LinkedIn: http://at.linkedin.com/in/davidschmitt

[Spencer Krum]

This is awesome. I've recommended it to my old work. This is absolutely
necessary when automating some of the hairy enterprise apps I've worked
with.

--
Spencer Krum
ni...@spencerkrum.com

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-dev/5495B94C.8050000%40dasz.at.
> For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

I agree. I hope to find some time to dig into the transaction layer and make some of the catalog manipulation a bit cleaner.

I've done something similar a few times but never thought about doing it this way so I'm very happy with this and hope to contribute!

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/1419108825.1373821.205262217.70C63FFE%40webmail.messagingengine.com.

For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[Eric Shamow]

+1, I think this is terrific.

I need to examine this against my normal workflow, which uses much more bootstrapping (orchestrated hiera checkouts + additional puppet runs) - this seems like it’s a much cleaner way to do the same thing.

I have a back-of-mind worry about a run stages-like level of complexity evolving from this but I suspect we can find a way to guard against that.

-Eric

-- 
Eric Shamow
Sent with Airmail

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoWBAgcc8yit2wGeLTNghPHOjObCcs1pNVN5Ku5LxQCLmA%40mail.gmail.com.

[John Bollinger]

On Friday, December 19, 2014 3:14:23 PM UTC-6, Reid Vandewiele wrote:

This thread is introducing a simple workaround for an observed limitation in Puppet's ability to automate inelegant but real configuration requirements. The desired outcome is to get feedback on the suitability of the workaround or its approach, how well it fits with Puppet's paradigm, and whether or not people would find it acceptable to implement this kind of pattern in their own environments.

tl;dr: What do people think of https://forge.puppetlabs.com/puppetlabs/transition ?

The Problem:

Puppet has always been tightly focused on a single, final target state [...], but there are some situations where it isn't quite enough. For example, if a running service locks a file (Windows often does this), that file cannot be changed unless the service is stopped. Procedurally, to edit the file one would be expected to stop the service, make the change to the file, and then start the service back up.

I agree that this is a longstanding issue, with no particularly good solution up to now: "write a full-blown type and/or provider" is not a realistic recommendation for most people and most requirements.  I'd go so far as to say that it is an inappropriate solution in some cases.

 

What if we had the ability to model transitional states as part of the catalog?

[...]

 

Does this pattern or capability make sense in the general context of Puppet? Is this a decent interim solution for something better currently under development? What do people think of this?

I went looking for holes to poke in this approach, and didn't find any.  I like that it builds on Puppet's core concepts of resources and state, and that it appears to be general enough to be adapted to a wide variety of situations.

Having read the docs but not examined the code, I am curious about management of the Transition type's own state.  How does it "look ahead" to determine whether it is already in sync?  I guess it invokes 'in_sync?' on each of the 'prior_to' resources, or something like that?  Also, I presume a Transition resource fails if it is not initially in sync and it cannot apply the transitional state it describes.  Is that correct?

And does the transitional state copy/inherit anything from the target final state given for the named resource, or does it have only the (explicit) attributes specified in the Transition resource?

I'm also curious about the nature of some of the documented limitations.

With regard to the transitioned resource being of native type, don't defined type instances have a representation in the catalog?  Classes do.  For that matter, are classes excluded from being the transitioned resource?  The more I think about it, the more I foresee potential for difficulties if defined type instances or classes were eligible for transitioning.  On the other hand, I'd be open to the argument that it's ok to offer additional capability to manifest writers in exchange for opening (more) possibilities for shooting themselves in the foot.

With regard to 'noop' parameters, is it your thinking that the transition should not be performed if all the 'prior_to' resources have 'noop' enabled?  What about the transitioned resource?  I'd be inclined to say that the Transition resource *shouldn't* look at 'prior_to' resources' 'noop' parameters.  If 'noop' is being applied on a per-resource basis, then the responsibility should be on the manifest developer to apply 'noop' to the Transition resource where needed.  On the other hand, I think you should consider whether the transitional state should automatically be marked with the same 'noop' value as the final state of the transitioned resource, at least by default.  I haven't reached a conclusion on that myself, but it seems more likely to be appropriate than basing the Transition's noop on the 'prior_to' resources'.

I also have to ask: will this work with Puppet 4?


All my questions notwithstanding, I think this is a really cool and useful idea.  Thanks!


John

[John Bollinger]

On Monday, December 22, 2014 11:11:24 AM UTC-6, John Bollinger wrote:

 

I went looking for holes to poke in this approach, and didn't find any.

Having just posted that, I thought of this: use of a Transition resource makes sense only if all the 'prior_to' resources have a 'before' relationship with the final state of the transitioned resource, so in the event that the transitional state is successfully applied, but one or more of the 'prior_to' resources fails, the final state of the transitioned resource normally will not be applied.

I think I'd place that in the "nothing ventured, nothing gained" category, though.  That such a risk exists should certainly be documented, but it's no reason to nix the idea.  If such a risk is intolerable for some particular resource then no transitional state should be modeled for it.  A full-blown type and provider may be the only viable solution in such a case.


John

[Reid Vandewiele]

On Mon, Dec 22, 2014 at 9:11 AM, John Bollinger <john.bo...@stjude.org> wrote:

[...]

I went looking for holes to poke in this approach, and didn't find any.  I like that it builds on Puppet's core concepts of resources and state, and that it appears to be general enough to be adapted to a wide variety of situations.

Having read the docs but not examined the code, I am curious about management of the Transition type's own state.  How does it "look ahead" to determine whether it is already in sync?  I guess it invokes 'in_sync?' on each of the 'prior_to' resources, or something like that?  Also, I presume a Transition resource fails if it is not initially in sync and it cannot apply the transitional state it describes.  Is that correct?

Effectively, yes. For each prior_to resource the provider will retrieve it from the catalog, call insync?() against each of its properties, and if any are found that are not in sync that triggers the transition. For specifics see https://github.com/puppetlabs/puppetlabs-transition/blob/master/lib/puppet/provider/transition/ruby.rb#L49-L66 (about 16 lines).

 

And does the transitional state copy/inherit anything from the target final state given for the named resource, or does it have only the (explicit) attributes specified in the Transition resource?

Yes. Right now all attribute defaults in the transition resource are inherited from the target final state resource. Now that you mention it I can imagine that this might not always be the desired behavior, so it could make sense to introduce a parameter that affected whether or not the transitional state had this implicit relationship to the target state. The current behavior was mostly chosen for convenience. I'd be curious to think through a specific scenario where the difference was important. One doesn't spring to mind immediately for me. Obviously it's possible already to be explicit about many/all attributes in the transition declaration.

 

I'm also curious about the nature of some of the documented limitations.

With regard to the transitioned resource being of native type, don't defined type instances have a representation in the catalog?  Classes do.  For that matter, are classes excluded from being the transitioned resource?  The more I think about it, the more I foresee potential for difficulties if defined type instances or classes were eligible for transitioning.  On the other hand, I'd be open to the argument that it's ok to offer additional capability to manifest writers in exchange for opening (more) possibilities for shooting themselves in the foot.

Allowing defined types or classes to be eligible for transitioning certainly makes things more complicated. The problem is that defined types and classes could have logic in Puppet code that dictates which resources to create and add to the graph. If statements, case statements, variables - the provider doesn't have access to any of that Puppet code. If a property or parameter on a defined type or a class is set to a different value in Puppet code, it's possible that an entirely different set of resources could be created. This is the root of the native type only limitation. At the time transition operates, none of the Puppet code exists anymore, only the raw catalog itself.

To the best of my knowledge, the graph-level representation of a class is a pair of whits with relationships to each resource from the class. I've been imagining defined types to be the same. This is likely enough to identify which resources came from a particular class or defined type, but I suspect it isn't enough to confidently change a defined type or class parameter. I would appreciate a check though on that from someone more familiar with how the catalog is put together and what information is available.

 

With regard to 'noop' parameters, is it your thinking that the transition should not be performed if all the 'prior_to' resources have 'noop' enabled?  What about the transitioned resource?  I'd be inclined to say that the Transition resource *shouldn't* look at 'prior_to' resources' 'noop' parameters.  If 'noop' is being applied on a per-resource basis, then the responsibility should be on the manifest developer to apply 'noop' to the Transition resource where needed.  On the other hand, I think you should consider whether the transitional state should automatically be marked with the same 'noop' value as the final state of the transitioned resource, at least by default.  I haven't reached a conclusion on that myself, but it seems more likely to be appropriate than basing the Transition's noop on the 'prior_to' resources'.

I think marking the transitional state with the target resource's noop value as default would be the way to go. This is not a design limitation, just work that has yet to be done.

I also have to ask: will this work with Puppet 4?

It should absolutely work with Puppet 4! :-)  If it doesn't I think it'll be because of interface-level changes, and I can't imagine it would take too much work to fix it.

~Reid

[Felix Frank]

On 12/22/2014 07:55 PM, Reid Vandewiele wrote:

And does the transitional state copy/inherit anything from the target final state given for the named resource, or does it have only the (explicit) attributes specified in the Transition resource?

Yes. Right now all attribute defaults in the transition resource are inherited from the target final state resource. Now that you mention it I can imagine that this might not always be the desired behavior, so it could make sense to introduce a parameter that affected whether or not the transitional state had this implicit relationship to the target state. The current behavior was mostly chosen for convenience. I'd be curious to think through a specific scenario where the difference was important. One doesn't spring to mind immediately for me. Obviously it's possible already to be explicit about many/all attributes in the transition declaration.

I would prefer for the transition to operate only on the properties that are specified.

Using your original example, it would be very confusing for the user to observe the transition manage the enable property of Service[myapp]. I have not looked at your code, so I don't know about the specifics and how feasible this behavior is, but this would be my ideal design.

Speaking of properties - transitions for classes and defines don't seem to make sense to me, because they don't have properties, only parameters. Sure, the compiler could try and figure out what property values are influenced by a given parameter, but that seems very complex to me, both in terms of implementation and user experience.

Generally, I will join the chorus - this feature shows amazing promise!

Thanks,
Felix