Resource ordering

[Burkholder, Peter]

I just finished listening to the Configuration Management panel from
OSBridge (on blip.tv).

Near the end of it, Adam Jacob states that Puppet's resource dependency
ordering is non-deterministic,
and that manifests that work fine 19 times will fail the 20th time.

Is this true? I'm puzzled that what Luke considers one of Puppet's
strong suits is derided by
others as its Achille's heel.

-Peter

--
Peter Burkholder
AARP | Web Strategy & Operations | 601 E Street, NW | Washington, DC
20049
email: pburk...@aarp.org | aim: peterbtech | ph: 202-434-3530 | cell:
202-344-7129 |

[Julian Simpson]

Perhaps if you didn't declare the dependencies accurately, some
orderings would work and some orderings would fail. Seems no
different to a declarative build tool in that respect.

Julian.





2009/7/17 Burkholder, Peter <PBurk...@aarp.org>:

--
Julian Simpson
Software Build and Deployment
http://www.build-doctor.com

[David Schmitt]

Burkholder, Peter wrote:
> I just finished listening to the Configuration Management panel from
> OSBridge (on blip.tv).
>
> Near the end of it, Adam Jacob states that Puppet's resource dependency
> ordering is non-deterministic,
> and that manifests that work fine 19 times will fail the 20th time.
>
> Is this true? I'm puzzled that what Luke considers one of Puppet's
> strong suits is derided by
> others as its Achille's heel.

Puppet's ordering of how it'll apply resources is only deterministic up
to the specified dependencies in the manifest. Not requiring more
determinism allows puppet to optimize resource application (in the
future; see e.g. the thread about coalescing package installations). Of
course, this might allow situations like those Adam talks about if the
manifest doesn't state all the implicit dependencies explicitly. Like
trying to start a service before the daemon is installed.

Regards, DavidS

[Bjørn Dyre Dyresen]

2009/7/17 Burkholder, Peter <PBurk...@aarp.org>

I just finished listening to the Configuration Management panel from
OSBridge (on blip.tv).

Near the end of it, Adam Jacob states that Puppet's resource dependency
ordering is non-deterministic,
and that manifests that work fine 19 times will fail the 20th time.

Is this true?   I'm puzzled that what Luke considers one of Puppet's
strong suits is derided by
others as its Achille's heel.

-Peter

If you use a bit of common sense when writing your manifests and set order the same way you would when installing this on the command line it will work 20 out of 20 times. You just set eg the require parameter or before parameter, easy and straight forward. It's not that different from programming either. Calling a function that does not yet exist can easily throw an error.. It's better to have a system letting you set the order explicitly when needed than having a system trying to figure out this in some automagically way and wind up being a pain to use. 

Regards

[Paul Lathrop]

On Fri, Jul 17, 2009 at 6:08 AM, Burkholder, Peter<PBurk...@aarp.org> wrote:
>
> I just finished listening to the Configuration Management panel from
> OSBridge (on blip.tv).
>
> Near the end of it, Adam Jacob states that Puppet's resource dependency
> ordering is non-deterministic,
> and that manifests that work fine 19 times will fail the 20th time.
>
> Is this true?   I'm puzzled that what Luke considers one of Puppet's
> strong suits is derided by
> others as its Achille's heel.

This is true w/o being the whole story. Puppet obeys declared
dependencies, but if you choose not to declare your dependencies, you
are running the risk of things happening in the wrong order. To me,
this is a bug in your manifests, not a bug in Puppet. Adam doesn't
believe this is a good thing. However, I have found that it gives me a
couple advantages: a) it forces me to really think about the order of
events and determine what is actually dependent on what; b) it allows
me to ignore the order of things when it doesn't matter, but often
reveals situations where it *does* matter and I didn't think of it; c)
there is potential for future optimization; d) I find explicitly
declaring dependencies to be preferable to re-arranging lines in a
file and finding things magically working.

--Paul

[James Turnbull]

Paul Lathrop wrote:
>> Is this true? I'm puzzled that what Luke considers one of Puppet's
>> strong suits is derided by
>> others as its Achille's heel.
>
> This is true w/o being the whole story. Puppet obeys declared
> dependencies, but if you choose not to declare your dependencies, you
> are running the risk of things happening in the wrong order. To me,
> this is a bug in your manifests, not a bug in Puppet. Adam doesn't
> believe this is a good thing. However, I have found that it gives me a
> couple advantages: a) it forces me to really think about the order of
> events and determine what is actually dependent on what; b) it allows
> me to ignore the order of things when it doesn't matter, but often
> reveals situations where it *does* matter and I didn't think of it; c)
> there is potential for future optimization; d) I find explicitly
> declaring dependencies to be preferable to re-arranging lines in a
> file and finding things magically working.

Whilst personally immensely biased I strongly agree with Paul. I was
taught (had beaten into me?) as a sys-admin to think through my actions,
plan them and understand sequencing and consequences (old school
mainframe shop). I think that's a critical skill for a sysadmin and if
Puppet forces people to do this when order matters... it's not a bad thing.

Regards

James Turnbull

--
Author of:
* Pro Linux Systems Administration
(http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet
(http://tinyurl.com/pupbook)
* Pro Nagios 2.0
(http://tinyurl.com/pronagios)
* Hardening Linux
(http://tinyurl.com/hardeninglinux)

[Trevor Vaughan]

I'm not biased and I also agree with Paul and James.

While I have had trouble with getting my order right in some cases, it
was generally a failing of the item that I was trying to configure
*not* a problem with Puppet.

It comes down to ordering something in a file implicitly or explicitly
and I actually prefer to tap explicit order for the potential
optimization benefits. I.e. when Ruby 1.9 is in widespread use with
native threads then you'll be able to get much better performance
overall on multi-processor systems because everything that isn't
explicitly serialized can be done in parallel.

Trevor

[James Turnbull]

Burkholder, Peter wrote:
> I just finished listening to the Configuration Management panel from
> OSBridge (on blip.tv).
>
> Near the end of it, Adam Jacob states that Puppet's resource dependency
> ordering is non-deterministic,
> and that manifests that work fine 19 times will fail the 20th time.
>
> Is this true? I'm puzzled that what Luke considers one of Puppet's
> strong suits is derided by
> others as its Achille's heel.
>

There is a change in 0.25.0 that I also should have mentioned because it
impacts this discussion.

In 0.25.0 we've added a 'require' function. The doco is here:

"Evaluate one or more classes, adding the required class as a dependency.

The relationship metaparameters work well for specifying relationships
between individual resources, but they can be clumsy for specifying
relationships between classes. This function is a superset of the
'include' function, adding a class relationship so that the requiring
class depends on the required class.

.. Warning:: using require in place of include can lead to unwanted
dependency cycles. For instance the following manifest, with 'require'
instead of 'include' would produce a nasty dependence cycle, because
notify imposes a before between File[/foo] and Service[foo]::

class myservice {
service { foo: ensure => running }
}

class otherstuff {
include myservice
file { '/foo': notify => Service[foo] }
}
"

This takes some of the (potential) pain out of the ordering by allowing
class level dependencies. This adds dependency resolution higher than
between individual resources. It doesn't solve issues where you haven't
built the right dependencies at a resource level but does provide more
flexibility.

This isn't the same as Chef - as Adam has pointed out Chef has top-down
ordering rather than Puppet's dependency graph - but I think it'll make
life easier for some people.

[Luke Kanies]

On Jul 17, 2009, at 6:08 AM, Burkholder, Peter wrote:

>
> I just finished listening to the Configuration Management panel from
> OSBridge (on blip.tv).
>
> Near the end of it, Adam Jacob states that Puppet's resource
> dependency
> ordering is non-deterministic,
> and that manifests that work fine 19 times will fail the 20th time.
>
> Is this true? I'm puzzled that what Luke considers one of Puppet's
> strong suits is derided by
> others as its Achille's heel.

Dependency ordering is entirely deterministic.

Ordering of unrelated items, just like in any other topological sort
of a graph, is currently nondeterministic.

We could trivially add deterministic ordering for unrelated items:
Just sort any equivalent resources based on name. Ten seconds after
you do this, you'll have people naming resources things like
'00001myservice' because they want to use alphabetic sorting instead
of declared dependencies, and then they'll complain when specified
dependencies get preference.

Basically, once we start saying "unrelated resources will always
happen in a predictable order", we can never change the internal
implementation or add parallelism. There might be some ways to
increase consistency without sacrificing these, but no one besides
Adam has complained much about it, and he was never willing to
actually contribute code to fixing it.

--
It's very hard to predict things . . . Especially the future.
-- Prof. Charles Kelemen, Swarthmore CS Dept.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com

[Trevor Vaughan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

This is going to be a great feature.

Over time, I've been struggling with trying to keep things extremely
modular (perhaps too much so) but still well ordered.

When I can make every class that needs apache just 'require apache', I
will be quite happy.

However, I think that this means that if *anything* in the class fails,
the dependent classes will fail too, is this correct?

Is there any way to make some things fail softly so that they can be
less hard than class-wise fatal?

Thanks,

Trevor

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkpmXc4ACgkQyjMdFR1108BPnwCbBAYZ+kFWaKrORho1NOZK6+Ij
bNQAn2bb0SDw0aofNRH0wKf/fv5iDpzw
=eDIA
-----END PGP SIGNATURE-----

[Luke Kanies]

On Jul 21, 2009, at 5:31 PM, Trevor Vaughan wrote:

>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> This is going to be a great feature.
>
> Over time, I've been struggling with trying to keep things extremely
> modular (perhaps too much so) but still well ordered.
>
> When I can make every class that needs apache just 'require apache', I
> will be quite happy.
>
> However, I think that this means that if *anything* in the class
> fails,
> the dependent classes will fail too, is this correct?

Yes, the dependent classes will be skipped if any resources in the
required classes fail.

>
> Is there any way to make some things fail softly so that they can be
> less hard than class-wise fatal?

Hmm, no one's ever asked before.

It seems reasonable that we could mark some resources as non-fatal,
but what resources would you mark this way?

--
In our civilization, and under our republican form of government,
intelligence is so highly honored that it is rewarded by exemption from
the cares of office. --Ambrose Bierce

[James Turnbull]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Luke Kanies wrote:
> Is there any way to make some things fail softly so that they can be
> less hard than class-wise fatal?
>
>> Hmm, no one's ever asked before.
>
>> It seems reasonable that we could mark some resources as non-fatal,
>> but what resources would you mark this way?

Could you mark an include or require as non-fatal? Would that achieve
the same end?

Regards

James Turnbull

- --

Author of:
* Pro Linux Systems Administration
(http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet
(http://tinyurl.com/pupbook)
* Pro Nagios 2.0
(http://tinyurl.com/pronagios)
* Hardening Linux
(http://tinyurl.com/hardeninglinux)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpmuLgACgkQ9hTGvAxC30DlSgCglgaRghC8SMp+IBfTyBv3OAzS
RLMAoJf/9XyPqLkEFdVBJFNBgu+lyTeH
=NHy2
-----END PGP SIGNATURE-----

[David Schmitt]

Luke Kanies wrote:
> On Jul 21, 2009, at 5:31 PM, Trevor Vaughan wrote:
>> Is there any way to make some things fail softly so that they can be
>> less hard than class-wise fatal?
>
> Hmm, no one's ever asked before.
>
> It seems reasonable that we could mark some resources as non-fatal,
> but what resources would you mark this way?

Probably stuff like setting up logrotate? That's not really critical to
the operation of the webserver. Of course, in an ideal world, that
hypothetical problem with logrotate would be fixed before using the
server in production, but ...

Regards, DavidS

[Trevor Vaughan]

I was thinking of this in a very fine grained way, such as a metaparameter.

The situation that made me think of this is:

class foo {
file { 'super/important':
ensure => 'file',
content => 'something great\n'
}

exec { 'nice to have happen':
command => 'check stuff and set a file',
fail => 'soft'
}
}

class bar {
require 'foo'

# stuff that requires super/important file
}

Basically, the exec in 'foo' is there because it's logical to place it
there, but some of it (the exec) just doesn't matter if it fails or
not.

I see this being most relevant in the case of execs, but could apply
to some situations where everything might be a soft failure except for
a service starting.

Trevor

[Peter Meier]

Hi

> [discussion about soft failing]

hmm I don't see why you'd like to have your environment in a
semi-satisfied state. Either your environment is in the state is
broken or you have to fix it. Everything else is imho just scary...

cheers pete

[Trevor Vaughan]

It's not semi-satisfied in cases where you might be waiting for
cross-system semaphores to complete and you don't want to fire
anything else off until they are.

Trevor

[Nigel Kersten]

On Wed, Jul 22, 2009 at 5:03 AM, Trevor Vaughan<peiri...@gmail.com> wrote:
>
> I was thinking of this in a very fine grained way, such as a metaparameter.
>
> The situation that made me think of this is:
>
> class foo {
>  file { 'super/important':
>    ensure => 'file',
>    content => 'something great\n'
>  }
>
>  exec { 'nice to have happen':
>    command => 'check stuff and set a file',
>    fail => 'soft'
>  }
> }
>
> class bar {
>   require 'foo'
>
>   # stuff that requires super/important file
> }
>
> Basically, the exec in 'foo' is there because it's logical to place it
> there, but some of it (the exec) just doesn't matter if it fails or
> not.
>
> I see this being most relevant in the case of execs, but could apply
> to some situations where everything might be a soft failure except for
> a service starting.

I was thinking about this the other day, that sometimes it would be
useful to be able to specify that you don't care about the return code
of an exec.

Alternatively, being able to specify a list of acceptable return codes
like [ 0, 1, 4] would also give us close to the same functionality as
far as soft failures go with execs.

I'm wracking my brains, and I really can't think of anything other
than execs that I'd want this for personally.

--
Nigel Kersten
nig...@google.com
System Administrator
Google, Inc.

[Trevor Vaughan]

For the most part, execs is it, but sometimes you might use it for a
file or for a custom type.

I suppose that it could be built into each type as necessary with just
as much effectiveness.

In this case though, we might want to set up logging for the exec only
on specific failure states.

I.e. 2 is no logging but 1 is logging.

Trevor