Gmail Calendar Documents Reader Web more »
Recently Visited Groups | Help | Sign in
Google Groups Home
Resource ordering
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  18 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Burkholder, Peter  
View profile  
 More options Jul 17, 9:08 am
From: "Burkholder, Peter" <PBurkhol...@aarp.org>
Date: Fri, 17 Jul 2009 09:08:53 -0400
Local: Fri, Jul 17 2009 9:08 am
Subject: Resource ordering
I just finished listening to the Configuration Management panel from
OSBridge (on blip.tv).

Near the end of it, Adam Jacob states that Puppet's resource dependency
ordering is non-deterministic,
and that manifests that work fine 19 times will fail the 20th time.

Is this true?   I'm puzzled that what Luke considers one of Puppet's
strong suits is derided by
others as its Achille's heel.

-Peter

--
Peter Burkholder
AARP | Web Strategy & Operations | 601 E Street, NW | Washington, DC
20049
email: pburkhol...@aarp.org | aim: peterbtech | ph: 202-434-3530 | cell:
202-344-7129 |


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Julian Simpson  
View profile  
 More options Jul 17, 9:14 am
From: Julian Simpson <simpsonjul...@gmail.com>
Date: Fri, 17 Jul 2009 14:14:39 +0100
Local: Fri, Jul 17 2009 9:14 am
Subject: Re: [Puppet Users] Resource ordering
Perhaps if you didn't declare the dependencies accurately, some
orderings would work and some orderings would fail.  Seems no
different to a declarative build tool in that respect.

Julian.

2009/7/17 Burkholder, Peter <PBurkhol...@aarp.org>:

--
Julian Simpson
Software Build and Deployment
http://www.build-doctor.com

    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
David Schmitt  
View profile  
 More options Jul 17, 9:19 am
From: David Schmitt <da...@dasz.at>
Date: Fri, 17 Jul 2009 15:19:52 +0200
Local: Fri, Jul 17 2009 9:19 am
Subject: Re: [Puppet Users] Resource ordering

Burkholder, Peter wrote:
> I just finished listening to the Configuration Management panel from
> OSBridge (on blip.tv).

> Near the end of it, Adam Jacob states that Puppet's resource dependency
> ordering is non-deterministic,
> and that manifests that work fine 19 times will fail the 20th time.

> Is this true?   I'm puzzled that what Luke considers one of Puppet's
> strong suits is derided by
> others as its Achille's heel.

Puppet's ordering of how it'll apply resources is only deterministic up
to the specified dependencies in the manifest. Not requiring more
determinism allows puppet to optimize resource application (in the
future; see e.g. the thread about coalescing package installations). Of
course, this might allow situations like those Adam talks about if the
manifest doesn't state all the implicit dependencies explicitly. Like
trying to start a service before the daemon is installed.

Regards, DavidS


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Bjørn Dyre Dyresen  
View profile  
 More options Jul 17, 10:25 am
From: Bjørn Dyre Dyresen <bj...@dyresen.net>
Date: Fri, 17 Jul 2009 16:25:52 +0200
Local: Fri, Jul 17 2009 10:25 am
Subject: Re: [Puppet Users] Resource ordering

2009/7/17 Burkholder, Peter <PBurkhol...@aarp.org>

> I just finished listening to the Configuration Management panel from
> OSBridge (on blip.tv).

> Near the end of it, Adam Jacob states that Puppet's resource dependency
> ordering is non-deterministic,
> and that manifests that work fine 19 times will fail the 20th time.

> Is this true?   I'm puzzled that what Luke considers one of Puppet's
> strong suits is derided by
> others as its Achille's heel.

> -Peter

If you use a bit of common sense when writing your manifests and set order
the same way you would when installing this on the command line it will work
20 out of 20 times. You just set eg the require parameter or before
parameter, easy and straight forward. It's not that different from
programming either. Calling a function that does not yet exist can easily
throw an error.. It's better to have a system letting you set the order
explicitly when needed than having a system trying to figure out this in
some automagically way and wind up being a pain to use.

Regards


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Paul Lathrop  
View profile  
 More options Jul 20, 4:42 pm
From: Paul Lathrop <p...@tertiusfamily.net>
Date: Mon, 20 Jul 2009 13:42:42 -0700
Local: Mon, Jul 20 2009 4:42 pm
Subject: Re: [Puppet Users] Resource ordering

On Fri, Jul 17, 2009 at 6:08 AM, Burkholder, Peter<PBurkhol...@aarp.org> wrote:

> I just finished listening to the Configuration Management panel from
> OSBridge (on blip.tv).

> Near the end of it, Adam Jacob states that Puppet's resource dependency
> ordering is non-deterministic,
> and that manifests that work fine 19 times will fail the 20th time.

> Is this true?   I'm puzzled that what Luke considers one of Puppet's
> strong suits is derided by
> others as its Achille's heel.

This is true w/o being the whole story. Puppet obeys declared
dependencies, but if you choose not to declare your dependencies, you
are running the risk of things happening in the wrong order. To me,
this is a bug in your manifests, not a bug in Puppet. Adam doesn't
believe this is a good thing. However, I have found that it gives me a
couple advantages: a) it forces me to really think about the order of
events and determine what is actually dependent on what; b) it allows
me to ignore the order of things when it doesn't matter, but often
reveals situations where it *does* matter and I didn't think of it; c)
there is potential for future optimization; d) I find explicitly
declaring dependencies to be preferable to re-arranging lines in a
file and finding things magically working.

--Paul


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
James Turnbull  
View profile  
 More options Jul 20, 6:17 pm
From: James Turnbull <ja...@lovedthanlost.net>
Date: Tue, 21 Jul 2009 08:17:12 +1000
Local: Mon, Jul 20 2009 6:17 pm
Subject: Re: [Puppet Users] Re: Resource ordering

Whilst personally immensely biased I strongly agree with Paul.  I was
taught (had beaten into me?) as a sys-admin to think through my actions,
plan them and understand sequencing and consequences (old school
mainframe shop).  I think that's a critical skill for a sysadmin and if
Puppet forces people to do this when order matters... it's not a bad thing.

Regards

James Turnbull

--
Author of:
* Pro Linux Systems Administration
(http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet
(http://tinyurl.com/pupbook)
* Pro Nagios 2.0
(http://tinyurl.com/pronagios)
* Hardening Linux
(http://tinyurl.com/hardeninglinux)

  signature.asc
< 1K Download

    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Trevor Vaughan  
View profile  
 More options Jul 21, 5:24 pm
From: Trevor Vaughan <peiriann...@gmail.com>
Date: Tue, 21 Jul 2009 17:24:52 -0400
Local: Tues, Jul 21 2009 5:24 pm
Subject: Re: [Puppet Users] Re: Resource ordering
I'm not biased and I also agree with Paul and James.

While I have had trouble with getting my order right in some cases, it
was generally a failing of the item that I was trying to configure
*not* a problem with Puppet.

It comes down to ordering something in a file implicitly or explicitly
and I actually prefer to tap explicit order for the potential
optimization benefits.  I.e. when Ruby 1.9 is in widespread use with
native threads then you'll be able to get much better performance
overall on multi-processor systems because everything that isn't
explicitly serialized can be done in parallel.

Trevor


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
James Turnbull  
View profile  
 More options Jul 21, 5:31 pm
From: James Turnbull <ja...@lovedthanlost.net>
Date: Wed, 22 Jul 2009 07:31:57 +1000
Local: Tues, Jul 21 2009 5:31 pm
Subject: Re: [Puppet Users] Resource ordering

Burkholder, Peter wrote:
> I just finished listening to the Configuration Management panel from
> OSBridge (on blip.tv).

> Near the end of it, Adam Jacob states that Puppet's resource dependency
> ordering is non-deterministic,
> and that manifests that work fine 19 times will fail the 20th time.

> Is this true?   I'm puzzled that what Luke considers one of Puppet's
> strong suits is derided by
> others as its Achille's heel.

There is a change in 0.25.0 that I also should have mentioned because it
impacts this discussion.

In 0.25.0 we've added a 'require' function.  The doco is here:

"Evaluate one or more classes,  adding the required class as a dependency.

The relationship metaparameters work well for specifying relationships
between individual resources, but they can be clumsy for specifying
relationships between classes.  This function is a superset of the
'include' function, adding a class relationship so that the requiring
class depends on the required class.

.. Warning:: using require in place of include can lead to unwanted
dependency cycles.  For instance the following manifest, with 'require'
instead of 'include'  would produce a nasty dependence cycle, because
notify imposes a before between File[/foo] and Service[foo]::

    class myservice {
       service { foo: ensure => running }
    }

    class otherstuff {
       include myservice
       file { '/foo': notify => Service[foo] }
    }
"

This takes some of the (potential) pain out of the ordering by allowing
class level dependencies.  This adds dependency resolution higher than
between individual resources.  It doesn't solve issues where you haven't
built the right dependencies at a resource level but does provide more
flexibility.

This isn't the same as Chef - as Adam has pointed out Chef has top-down
ordering rather than Puppet's dependency graph - but I think it'll make
life easier for some people.

Regards

James Turnbull

--
Author of:
* Pro Linux Systems Administration
(http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet
(http://tinyurl.com/pupbook)
* Pro Nagios 2.0
(http://tinyurl.com/pronagios)
* Hardening Linux
(http://tinyurl.com/hardeninglinux)

  signature.asc
< 1K Download

    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Luke Kanies  
View profile  
 More options Jul 21, 6:50 pm
From: Luke Kanies <l...@madstop.com>
Date: Tue, 21 Jul 2009 15:50:07 -0700
Local: Tues, Jul 21 2009 6:50 pm
Subject: Re: [Puppet Users] Resource ordering
On Jul 17, 2009, at 6:08 AM, Burkholder, Peter wrote:

> I just finished listening to the Configuration Management panel from
> OSBridge (on blip.tv).

> Near the end of it, Adam Jacob states that Puppet's resource  
> dependency
> ordering is non-deterministic,
> and that manifests that work fine 19 times will fail the 20th time.

> Is this true?   I'm puzzled that what Luke considers one of Puppet's
> strong suits is derided by
> others as its Achille's heel.

Dependency ordering is entirely deterministic.

Ordering of unrelated items, just like in any other topological sort  
of a graph, is currently nondeterministic.

We could trivially add deterministic ordering for unrelated items:  
Just sort any equivalent resources based on name.  Ten seconds after  
you do this, you'll have people naming resources things like  
'00001myservice' because they want to use alphabetic sorting instead  
of declared dependencies, and then they'll complain when specified  
dependencies get preference.

Basically, once we start saying "unrelated resources will always  
happen in a predictable order", we can never change the internal  
implementation or add parallelism.  There might be some ways to  
increase consistency without sacrificing these, but no one besides  
Adam has complained much about it, and he was never willing to  
actually contribute code to fixing it.

--
It's very hard to predict things . . . Especially the future.
     -- Prof. Charles Kelemen, Swarthmore CS Dept.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Trevor Vaughan  
View profile  
 More options Jul 21, 8:31 pm
From: Trevor Vaughan <peiriann...@gmail.com>
Date: Tue, 21 Jul 2009 20:31:14 -0400
Local: Tues, Jul 21 2009 8:31 pm
Subject: Re: [Puppet Users] Re: Resource ordering
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

This is going to be a great feature.

Over time, I've been struggling with trying to keep things extremely
modular (perhaps too much so) but still well ordered.

When I can make every class that needs apache just 'require apache', I
will be quite happy.

However, I think that this means that if *anything* in the class fails,
the dependent classes will fail too, is this correct?

Is there any way to make some things fail softly so that they can be
less hard than class-wise fatal?

Thanks,

Trevor

On 07/21/2009 05:31 PM, James Turnbull wrote:

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkpmXc4ACgkQyjMdFR1108BPnwCbBAYZ+kFWaKrORho1NOZK6+Ij
bNQAn2bb0SDw0aofNRH0wKf/fv5iDpzw
=eDIA
-----END PGP SIGNATURE-----


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Luke Kanies  
View profile  
 More options Jul 22, 2:38 am
From: Luke Kanies <l...@madstop.com>
Date: Tue, 21 Jul 2009 23:38:37 -0700
Local: Wed, Jul 22 2009 2:38 am
Subject: Re: [Puppet Users] Re: Resource ordering
On Jul 21, 2009, at 5:31 PM, Trevor Vaughan wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1

> This is going to be a great feature.

> Over time, I've been struggling with trying to keep things extremely
> modular (perhaps too much so) but still well ordered.

> When I can make every class that needs apache just 'require apache', I
> will be quite happy.

> However, I think that this means that if *anything* in the class  
> fails,
> the dependent classes will fail too, is this correct?

Yes, the dependent classes will be skipped if any resources in the  
required classes fail.

> Is there any way to make some things fail softly so that they can be
> less hard than class-wise fatal?

Hmm, no one's ever asked before.

It seems reasonable that we could mark some resources as non-fatal,  
but what resources would you mark this way?

--
In our civilization, and under our republican form of government,
intelligence is so highly honored that it is rewarded by exemption from
the cares of office. --Ambrose Bierce
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com

    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
James Turnbull  
View profile  
 More options Jul 22, 2:59 am
From: James Turnbull <ja...@lovedthanlost.net>
Date: Wed, 22 Jul 2009 16:59:04 +1000
Local: Wed, Jul 22 2009 2:59 am
Subject: Re: [Puppet Users] Re: Resource ordering
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Luke Kanies wrote:
> Is there any way to make some things fail softly so that they can be
> less hard than class-wise fatal?

>> Hmm, no one's ever asked before.

>> It seems reasonable that we could mark some resources as non-fatal,  
>> but what resources would you mark this way?

Could you mark an include or require as non-fatal?  Would that achieve
the same end?

Regards

James Turnbull

- --
Author of:
* Pro Linux Systems Administration
(http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet
(http://tinyurl.com/pupbook)
* Pro Nagios 2.0
(http://tinyurl.com/pronagios)
* Hardening Linux
(http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpmuLgACgkQ9hTGvAxC30DlSgCglgaRghC8SMp+IBfTyBv3OAzS
RLMAoJf/9XyPqLkEFdVBJFNBgu+lyTeH
=NHy2
-----END PGP SIGNATURE-----


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
David Schmitt  
View profile  
 More options Jul 22, 3:02 am
From: David Schmitt <da...@dasz.at>
Date: Wed, 22 Jul 2009 09:02:32 +0200
Local: Wed, Jul 22 2009 3:02 am
Subject: Re: [Puppet Users] Re: Resource ordering

Luke Kanies wrote:
> On Jul 21, 2009, at 5:31 PM, Trevor Vaughan wrote:
>> Is there any way to make some things fail softly so that they can be
>> less hard than class-wise fatal?

> Hmm, no one's ever asked before.

> It seems reasonable that we could mark some resources as non-fatal,  
> but what resources would you mark this way?

Probably stuff like setting up logrotate? That's not really critical to
the operation of the webserver. Of course, in an ideal world, that
hypothetical problem with logrotate would be fixed before using the
server in production, but ...

Regards, DavidS


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Trevor Vaughan  
View profile  
 More options Jul 22, 8:03 am
From: Trevor Vaughan <peiriann...@gmail.com>
Date: Wed, 22 Jul 2009 08:03:20 -0400
Local: Wed, Jul 22 2009 8:03 am
Subject: Re: [Puppet Users] Re: Resource ordering
I was thinking of this in a very fine grained way, such as a metaparameter.

The situation that made me think of this is:

class foo {
  file { 'super/important':
    ensure => 'file',
    content => 'something great\n'
  }

  exec { 'nice to have happen':
    command => 'check stuff and set a file',
    fail => 'soft'
  }

}

class bar {
   require 'foo'

   # stuff that requires super/important file

}

Basically, the exec in 'foo' is there because it's logical to place it
there, but some of it (the exec) just doesn't matter if it fails or
not.

I see this being most relevant in the case of execs, but could apply
to some situations where everything might be a soft failure except for
a service starting.

Trevor


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Peter Meier  
View profile  
 More options Jul 22, 8:26 am
From: Peter Meier <peter.me...@immerda.ch>
Date: Wed, 22 Jul 2009 14:26:49 +0200
Local: Wed, Jul 22 2009 8:26 am
Subject: Re: [Puppet Users] Re: Resource ordering
Hi

> [discussion about soft failing]

hmm I don't see why you'd like to have your environment in a  
semi-satisfied state. Either your environment is in the state is  
broken or you have to fix it. Everything else is imho just scary...

cheers pete


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Trevor Vaughan  
View profile  
 More options Jul 22, 9:03 am
From: Trevor Vaughan <peiriann...@gmail.com>
Date: Wed, 22 Jul 2009 09:03:03 -0400
Local: Wed, Jul 22 2009 9:03 am
Subject: Re: [Puppet Users] Re: Resource ordering
It's not semi-satisfied in cases where you might be waiting for
cross-system semaphores to complete and you don't want to fire
anything else off until they are.

Trevor


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Nigel Kersten  
View profile  
 More options Jul 22, 9:31 am
From: Nigel Kersten <nig...@google.com>
Date: Wed, 22 Jul 2009 06:31:46 -0700
Local: Wed, Jul 22 2009 9:31 am
Subject: Re: [Puppet Users] Re: Resource ordering

I was thinking about this the other day, that sometimes it would be
useful to be able to specify that you don't care about the return code
of an exec.

Alternatively, being able to specify a list of acceptable return codes
like [ 0, 1, 4] would also give us close to the same functionality as
far as soft failures go with execs.

I'm wracking my brains, and I really can't think of anything other
than execs that I'd want this for personally.

--
Nigel Kersten
nig...@google.com
System Administrator
Google, Inc.

    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Trevor Vaughan  
View profile  
 More options Jul 22, 9:59 am
From: Trevor Vaughan <peiriann...@gmail.com>
Date: Wed, 22 Jul 2009 09:59:57 -0400
Local: Wed, Jul 22 2009 9:59 am
Subject: Re: [Puppet Users] Re: Resource ordering
For the most part, execs is it, but sometimes you might use it for a
file or for a custom type.

I suppose that it could be built into each type as necessary with just
as much effectiveness.

In this case though, we might want to set up logging for the exec only
on specific failure states.

I.e. 2 is no logging but 1 is logging.

Trevor


    Reply to author    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2009 Google