Ordering without dependencies

[R.I.Pienaar]

hello,

I am looking for a way to influence run order without also doing dependencies.

I have a case where I am removing a bit of software from my system and so as is
typical things need to be done in a different order from creation, crucially I
also do not really care if removal fails. It should just try to remove everything
independent of each other - but at least in a given order so there's some chance
of it working.

There seems to be no way to influence order which does not also imply a requirement
and so today the only way this is achieved is by invoking puppet 3 times with 3
different manifest files which seems horrible but that's the only way I can find
to achieve this as well.


I tried:

exec{"false": before => Exec["true"]}
exec{"true": }

This fails because before implies a requirement.

Surprisingly this also implies a requirement:

exec{"false": notify => Exec["true"]
exec{"true": }

Here the true exec is skipped when false exec fails, I really did not expect
this to be the case

---
R.I.Pienaar

[Craig Dunn]

On Fri, Jul 3, 2015 at 3:10 PM, R.I.Pienaar <r...@devco.net> wrote:

>
> I tried:
>
> exec{"false": before => Exec["true"]}
> exec{"true": }
>
> This fails because before implies a requirement.

If you don't care about a failure, why not always mask it to be true?

exec { '/bin/remove mystuff; /bin/true':
before => Exec['carry_on_regardless'],
}

exec { '/bin/carry_on_regardless': }


?

--
Enviatics | Automation and configuration management
http://www.enviatics.com | @Enviatics
Puppet Training http://www.enviatics.com/training/

[Craig Dunn]

On Fri, Jul 3, 2015 at 3:10 PM, R.I.Pienaar <r...@devco.net> wrote:

> hello,
>
> I am looking for a way to influence run order without also doing dependencies.
>
> I have a case where I am removing a bit of software from my system and so as is
> typical things need to be done in a different order from creation, crucially I
> also do not really care if removal fails. It should just try to remove everything
> independent of each other - but at least in a given order so there's some chance
> of it working.

Theres also a parameter to exec called 'returns' that takes an array,
so if you know what return codes are possible you can do...

exec { '/bin/false':
returns => [ '0', '1' ],
before => Exec['other'],
}

and it will never fail the resource, I think thats cleaner than my
other example.

Craig

[R.I.Pienaar]

----- Original Message -----
> From: "Craig Dunn" <cr...@craigdunn.org>
> To: "puppet-users" <puppet...@googlegroups.com>
> Sent: Friday, July 3, 2015 2:47:53 PM
> Subject: Re: [Puppet Users] Ordering without dependencies

> On Fri, Jul 3, 2015 at 3:10 PM, R.I.Pienaar <r...@devco.net> wrote:
>> hello,
>>
>> I am looking for a way to influence run order without also doing dependencies.
>>
>> I have a case where I am removing a bit of software from my system and so as is
>> typical things need to be done in a different order from creation, crucially I
>> also do not really care if removal fails. It should just try to remove
>> everything
>> independent of each other - but at least in a given order so there's some chance
>> of it working.
>
>
> Theres also a parameter to exec called 'returns' that takes an array,
> so if you know what return codes are possible you can do...
>
> exec { '/bin/false':
> returns => [ '0', '1' ],
> before => Exec['other'],
> }
>
> and it will never fail the resource, I think thats cleaner than my
> other example.

yeah those will work for exec, unfort these include non exec stuff :(

[Christopher Wood]

On Fri, Jul 03, 2015 at 02:10:55PM +0100, R.I.Pienaar wrote:
> hello,
>
> I am looking for a way to influence run order without also doing dependencies.
>
> I have a case where I am removing a bit of software from my system and so as is
> typical things need to be done in a different order from creation, crucially I
> also do not really care if removal fails. It should just try to remove everything
> independent of each other - but at least in a given order so there's some chance
> of it working.

I couldn't say about doing things without dependencies, but for uninstalling stuff I have wrapped everything in a giant if/then or had two different classes doing different things.

class c1 ($un = false) {
if str2bool($un) {
service { 'mys': ensure => stopped, enable => false }
package { 'myp': ensure => absent, }
}
else {
package { 'myp': }
service { 'mys': ensure => running, enable => true, require => Package['myp'] }

}
}

Once managing the service resource fails on every host (due to a lack of init script from package removal) I amend the manifests. Filthy but it works.

> There seems to be no way to influence order which does not also imply a requirement
> and so today the only way this is achieved is by invoking puppet 3 times with 3
> different manifest files which seems horrible but that's the only way I can find
> to achieve this as well.

If you don't care about the service maybe just kill it?

class c2 {
exec {'k': command => 'killall -9 mys || true', path => ['/bin', '/usr/bin'], }
package { 'myp': ensure => absent, require => Exec['k'], }
}

(Using true because an exec fails on a non-zero exit status iirc.)

> I tried:
>
> exec{"false": before => Exec["true"]}
> exec{"true": }
>
> This fails because before implies a requirement.
>
> Surprisingly this also implies a requirement:
>
> exec{"false": notify => Exec["true"]
> exec{"true": }
>
> Here the true exec is skipped when false exec fails, I really did not expect
> this to be the case
>
> ---
> R.I.Pienaar
>

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/886907918.98844.1435929055488.JavaMail.zimbra%40devco.net.
> For more options, visit https://groups.google.com/d/optout.

[R.I.Pienaar]

----- Original Message -----
> From: "Christopher Wood" <christop...@pobox.com>
> To: "puppet-users" <puppet...@googlegroups.com>
> Sent: Friday, July 3, 2015 3:34:29 PM
> Subject: Re: [Puppet Users] Ordering without dependencies

Yeah it's a bit more complex than just a service. This manifest builds an entire
hyperv cluster with n dependant servers, storage volumes, dns entries and so forth
and so on, even virtual machine deployment.

On teardown I don't care what stays and go, just want to try my best to kill things

So the resources in question are varied and custom and spread over multiple hosts.

So rather than a specific solution that involves just killing this or that I am
after ideas for how to do so in a generic way with any kind of resource

[Christopher Wood]

Just rhubarbing along, but what about:

copying all the current environment into another environment
removing all the require/before/subscribe/notify metaparams
removing all the ensure metaparams
add a stack of resource defaults saying ensure => absent
(except for Service, which has true/false running/stopped not present/absent)
(might need to preserve puppet so it's the last to die)
agent runs in the other environment

Service { ensure => false }
Package { ensure => absent }
Hypervserver { ensure => absent }
...

Another thought, grabbing all the resource titles/types from your catalogs and generating a site.pp with ensure => absent as the only parameter for each resource.

Being ridiculous now, a single site.pp with a resources type for each type (and a stub puppet config!) you have:

resources { 't1': purge => true, }
resources { 't2': purge => true, }

On the other hand, since this is HyperV, maybe you can keep your removal manifests to the lower level stuff Destroying all the low-level resources (VMs, storage pools, virtual switches) would take out everything at higher levels too.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/2022577340.109977.1435934562598.JavaMail.zimbra%40devco.net.

[Craig Dunn]

On Fri, Jul 3, 2015 at 4:42 PM, R.I.Pienaar <r...@devco.net> wrote:

> On teardown I don't care what stays and go, just want to try my best to kill things
>
> So the resources in question are varied and custom and spread over multiple hosts.
>
> So rather than a specific solution that involves just killing this or that I am
> after ideas for how to do so in a generic way with any kind of resource

I don't think you'll find a clean way of doing this - you are trying
to take a system that is built entirely around the concept of
enforcing a desired state and telling it to "try it's best". You
could do something batshit crazy with run stages but that would likely
be more trouble than its worth.

[Daniel Dreier]

One relatively generic option would be to switch the ordering mode to use manifest ordering. I've never used it but it's been possible since 3.3ish. I believe you just set ordering=manifest in the agent's puppet.conf. A blog post introducing the functionality is at https://puppetlabs.com/blog/introducing-manifest-ordered-resources.

That would allow you to simply order the resources sequentially in the manifest without establishing hard dependencies, while retaining the option to set dependencies if needed. 

[R.I.Pienaar]

Indeed this is going to be the ultimate solution - once we're on that version of
puppet :(

What I really want is a way for resources to influence sorting only and not make
dependencies.

Manifest ordering does what I want - it orders one resource before another and does
not add relationships. I want parameters for that kind of non relational ordering.

I am though require surprised by the behaviour of notify.

exec{"false": notify => Exec["true"]}
exec{"true": }

Exec["true"] will never run because notify is a dependency.

From a systems view if I had a machine already built with 10 PHP apps and I now
want to upgrade 1 of these apps that notifies a service and there's any kind of
thing wrong with this crappy php app manifest then suddenly my web server service
is unmanaged as it's failing as a side effect.

When the server is still in a position where the web server should be ensured up
or whatever. Notify shouldn't always imply a requirement I guess.

>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to puppet-users...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/918D0F8F-A584-4A1F-8285-7AEBF3453440%40puppetlabs.com.

[Fraser Goffin]

Caveat: I know nothing about your specific infrastructure and application stack setup or how you manage it or the suitability of the business apps you have deployed or .. Well you get the idea, but .... in a VM world, wouldn't you prefer following the immutable server pattern, then uninstalls are moot ?

I appreciate this is not straight-forward, in particular ensuring machine instances are stateless, but it makes this whole category of problem a lot easier to manage.

Regards

Fraser.

P.S. Feedback on your module-data module. Been using it for some time and very much prefer the localisation it offers over uber Hiera. We are moving over to Puppet 4 and preliminary tests appear to show no problems continuing with it. Any reason I should reconsider and/or use the v4 lookup feature ?

[R.I.Pienaar]

----- Original Message -----
> From: "Fraser Goffin" <gof...@gmail.com>
> To: "puppet-users" <puppet...@googlegroups.com>
> Sent: Saturday, July 4, 2015 10:27:36 AM
> Subject: Re: [Puppet Users] Ordering without dependencies

> Caveat: I know nothing about your specific infrastructure and application stack
> setup or how you manage it or the suitability of the business apps you have
> deployed or .. Well you get the idea, but .... in a VM world, wouldn't you
> prefer following the immutable server pattern, then uninstalls are moot ?

Indeed, in my case I am tearing down the entire cluster. This includes:

- vms
- the hyperv cluster
- storage associations
- access lists on the storage servers
- returning ip addresses to a pool
- dns records
- logic network definitions
- servers
- server raid configs
- server razor properties for PXE booting

and it supports combinations of above say cluster stays but server goes etc

its complex so for just VMs sure, but I'm dealing with the entire stack from PXE
boot onward including BIOS settings and firmware. All via Puppet. Chances of every
single thing of this tearing down perfectly in every case is zero so we just want
to keep going and nuke as much as possible - which while I realize this sounds idiotic
is fine in this case. The manifests are constructed dynamically by our orchestrator
so they are one-offs

> P.S. Feedback on your module-data module. Been using it for some time and very
> much prefer the localisation it offers over uber Hiera. We are moving over to
> Puppet 4 and preliminary tests appear to show no problems continuing with it.
> Any reason I should reconsider and/or use the v4 lookup feature ?

I believe PL is implementing a new Hiera which would include the module-data behaviours

Though I did have a pull request from someone who fixed some Puppet 4 stuff so maybe
try master

[jcbollinger]

On Friday, July 3, 2015 at 10:30:59 AM UTC-5, R.I. Pienaar wrote:

What I really want is a way for resources to influence sorting only and not make
dependencies.

Why?

A resource relationship expresses the idea that the resource at one end must be configured before the one at the other end successfully and correctly can be.  If you indeed have such a situation, then you have a genuine dependency that it is useless to ignore.  If you do not have such a situation, then by definition it does not matter in which order you manage the (formally) unrelated resources.

The direction and even existence of relationships is often different for tearing down a (sub)system than it is for building it up, which can be a pain, but I'm not at all seeing what's useful about controlling order of application absent dependencies.

 

Manifest ordering does what I want - it orders one resource before another and does
not add relationships. I want parameters for that kind of non relational ordering.

There are no such metaparameters in current Puppet.

 

I am though require surprised by the behaviour of notify.

  exec{"false": notify => Exec["true"]}
  exec{"true": }

Exec["true"] will never run because notify is a dependency.

Yes, this has been Puppet's behavior since at least since v0.24.  The "notify" and "subscribe" metaparameters have always implied "before" and "require", respectively.  It makes sense to me to some extent, but it is not clear cut as the behavior non-signaling relationships.

John