tagging and templates
pc2
I had some hard time with tagging. Finally I found the following
wontfix bug:
http://reductivelabs.com/trac/puppet/ticket/486
I wonder what is the suggested way to achive similar functionality in
a reliable fashion?
--
case tagged(apache) {
true: { package { webmin-apache: ensure => latest } }
}
--
The first tries with defining variables in node definitions instead of
tags gave the same unreliable results. I should also add that I am
trying to set variables for templates this way.
thx,
Bela
Luke Kanies
The only thing I can really say here is that tagging, or maybe
'defined(Class[apache])', is the best you can get, but I know it's not
very good.
Given the number of bugs filed against the language right now, it
looks like it's due for a revamp, but I've got enough other things on
my plate that it doesn't look like it's exactly imminent.
--
Never regret. If it's good, it's wonderful. If it's bad, it's
experience. -- Victoria Holt
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Ashley Penney
Luke Kanies
> This is a must have feature for me as well, so I hope it definitely
> gets onto the todo list.
Well, there are tried and true ways to get things on a todo list in
open source projects: Put them on your own todo list. :)
Or, of course, you can pay for them.
This is, frankly, pretty low on my priority list. This is what
external nodes are for, after all.
--
Never confuse movement with action. -- Ernest Hemingway
Ashley Penney
Digant C Kasundra
wrote:
The notion of setting global variables and then using the variables to
determine how classes should behave is, in my opinion, a broken concept.
There are times when this is appropriate, like the use of facts, but when
you are defining them in the node, I just don't get it. Why not include a
class that explicitly installs the package you want instead of using
generic classes that use case statements to perform non-specific tasks? If
you want webmin-apache, include a class that installs webmin-apache or a
class that includes such a class.
These things will all get easier with the maturity of external node tools
and is, I think, the more proper way to go. But that is just my opinion.
--
Digant C Kasundra <dig...@stanford.edu>
Technical Lead, ITS Unix Systems and Applications, Stanford University
Digant C Kasundra
wrote:
> The only thing I can really say here is that tagging, or maybe
> 'defined(Class[apache])', is the best you can get, but I know it's not
> very good.
>
> Given the number of bugs filed against the language right now, it
> looks like it's due for a revamp, but I've got enough other things on
> my plate that it doesn't look like it's exactly imminent.
In certain areas, perhaps, but on the whole, I disagree. i think people
are trying to force Puppet to do things in a certain way instead of doing
things in a "Puppet way" (i.e. using constructs and capabilities that
already exist). Instead of saying "I want to achieve X" and then seeing
how Puppet can handle that, people are perhaps saying "I want to achieve X,
and the only way I want to achieve that is with solution Y and Puppet can't
do solution Y so it is broken/limited/immature."
Just my two cents. I'm fighting the good fight to keep Puppet from
becoming another scripting or programming language. When that happens, I
feel at least in my institution, it will undo some of the strict discipline
we've been trying to maintain (one of the driving factors behind going with
Puppet in the first place).
jtimberman
> In certain areas, perhaps, but on the whole, I disagree. i think people
> are trying to force Puppet to do things in a certain way instead of doing
> things in a "Puppet way" (i.e. using constructs and capabilities that
> already exist). Instead of saying "I want to achieve X" and then seeing
> how Puppet can handle that, people are perhaps saying "I want to achieve X,
> and the only way I want to achieve that is with solution Y and Puppet can't
> do solution Y so it is broken/limited/immature."
keep things simple as possible. We have a few defined resources that
make use of fairly complex erb templates, and a couple custom facts,
but overall, we've kept things within the capabilities of Puppet.
While it is quite extensible and flexible, as is facter, I don't think
that the "right" approach is to make a custom type, provider or fact
every time I need to solve a configuration issue.
Each tool in a system administrators toolbox fits a specific purpose,
especially on Unix/Linux platforms. Puppet is well suited for many
tasks, but not all. For example, one can't kickstart a system entirely
with Puppet :-).
Evan Hisey
puppet to match my old custom SCM/CMS till I started trying to look at
problems from the puppet language structure. Then alot of things got
easier. Sometimes hard problems went way. Occasionally what should
beeasy is alittle harder, but over all work the with Puppet instead of
against it has made things much saner.
Evan
Ashley Penney
This isn't meant to be an attack, but it's a real problem that I'm struggling to solve cleanly. The other alternative is to always include the statements in every sudoer file on every system, but I feel that's fundementally flawed too.
Paul Lathrop
the statements in every sudoers file on every host. Why isn't this a
good solution?
--Paul
Luke Kanies
I agree with your general point -- keep Puppet's language simple --
but a lot of the things I'm hoping to fix shouldn't add complexity,
they should just shift behaviour so it makes a bit more sense.
In some cases, I might actually remove functionality that never seems
to work right. The tagged method seems to have caused more work than
it's saved, because people can't get the behavoiur they want out of it.
--
Think twice before you speak, and then you may be able to say
something more insulting than if you spoke right out at
once. -- Evan Esar
Luke Kanies
> I'm after the tagging support primarily for templates, in that
> sometimes I need to add extra lines into various files controlled by
> a module (like sudoers) if another module is included on the same
> server. How would you solve that with the current language and no
> external scripts? There appears to be no way to manage the same
> file in multiple places (ignoring what a potential mess that is),
> and only tagging support to check for the existence of a class seems
> to solve this problem. Lukes solution was to add snippets in
> different modules and then concatenate them together, but I think
> it's fundementally cleaner to be able to check this type of tag.
>
> This isn't meant to be an attack, but it's a real problem that I'm
> struggling to solve cleanly. The other alternative is to always
> include the statements in every sudoer file on every system, but I
> feel that's fundementally flawed too.
This is exactly what people mean when they say you're working against
Puppet rather than with it.
Puppet is resource-oriented, not file-oriented. If you model these
sudo lines as resources, you'll have no trouble at all -- the module
that needs to add sudo statements just adds them as resources. If,
however, you try to do the work in a template, then you're always
going to have trouble, because Puppet just doesn't work that way.
This is why I pointed you to the iptables example -- it's the simplest
way to do this in the Puppet language.
Really. I absolutely promise you'll be pleasantly surprised how much
easier your Puppet usage becomes when you switch to thinking about
resources.
We've talked about building some kind of snippet manager into the
language, but we haven't quite gotten around to it yet.
--
The most dangerous strategy is to jump a chasm in two leaps.
-- Benjamin Disraeli
Evan Hisey
> I'm after the tagging support primarily for templates, in that sometimes I
> need to add extra lines into various files controlled by a module (like
> sudoers) if another module is included on the same server. How would you
> solve that with the current language and no external scripts? There appears
> to be no way to manage the same file in multiple places (ignoring what a
> potential mess that is), and only tagging support to check for the existence
> of a class seems to solve this problem. Lukes solution was to add snippets
> in different modules and then concatenate them together, but I think it's
> fundementally cleaner to be able to check this type of tag.
>
> This isn't meant to be an attack, but it's a real problem that I'm
> struggling to solve cleanly. The other alternative is to always include the
> statements in every sudoer file on every system, but I feel that's
> fundementally flawed too.
>
>
them yet, but they do allow you to tinker with the same resources in
multiple places. If exact placement of lines is not an issue, then you
good also look at the line definition on the wiki. it can be a bit
hackish but it works pretty well for some things.
http://reductivelabs.com/trac/puppet/wiki/Recipes/SimpleText
Evan
Digant C Kasundra
wrote:
> I agree with your general point -- keep Puppet's language simple --
> but a lot of the things I'm hoping to fix shouldn't add complexity,
> they should just shift behaviour so it makes a bit more sense.
>
> In some cases, I might actually remove functionality that never seems
> to work right. The tagged method seems to have caused more work than
> it's saved, because people can't get the behavoiur they want out of it.
That definitely makes sense as well.
Digant C Kasundra
<grump...@gmail.com> wrote:
> Each tool in a system administrators toolbox fits a specific purpose,
> especially on Unix/Linux platforms. Puppet is well suited for many
> tasks, but not all. For example, one can't kickstart a system entirely
> with Puppet :-).
I agree. Trying to turn on tool into something completely else doesn't
help either. Your example is valid. We definitely use a different system
to get the base install (Kickstart, as it so happens, for Red Hat, and FAI
for Debian). Almost everything else from there is automated save for
changing the root password. The rest we are building custom types to
manage (like bootstrapping the kerberos keytab file).
Digant C Kasundra
wrote:
> I'll second both of these.I had the some real problems trying to get
> puppet to match my old custom SCM/CMS till I started trying to look at
> problems from the puppet language structure. Then alot of things got
> easier. Sometimes hard problems went way. Occasionally what should
> beeasy is alittle harder, but over all work the with Puppet instead of
> against it has made things much saner.
>
> Evan
As Stephen Colbert would say, "Tip of the hat, for being one of the heroes."
Adam Jacob
to skin this cat. Puppet is a fabulous tool, and any disagreement I
may have is one of implementation, not a criticism of the work that
Luke or anyone else has put in.
This reply is to Luke's message, but it's not targeted at Luke
necessarily. Just a different point of view. :)
Okay, enough dissembling..
On Wed, May 7, 2008 at 1:02 PM, Luke Kanies <lu...@madstop.com> wrote:
> This is exactly what people mean when they say you're working against
> Puppet rather than with it.
>
> Puppet is resource-oriented, not file-oriented. If you model these
> sudo lines as resources, you'll have no trouble at all -- the module
> that needs to add sudo statements just adds them as resources. If,
> however, you try to do the work in a template, then you're always
> going to have trouble, because Puppet just doesn't work that way.
>
> This is why I pointed you to the iptables example -- it's the simplest
> way to do this in the Puppet language.
>
> Really. I absolutely promise you'll be pleasantly surprised how much
> easier your Puppet usage becomes when you switch to thinking about
> resources.
In general, I think it is totally a true statement. For example, we
use a modified version of Digant's iptables assembly script, and it
works great. It fits very, very well with the way I want Puppet
manifests to be.
iptables_rule { "all_established":
ensure => present
}
Extending this to sudo, for example, works nearly as well syntax-wise:
sudoers { "sysadmin-rule":
comment => "Members of the sysadmin group have root",
rule => "%sysadmin ALL=(ALL) ALL"
}
The place where this becomes somewhat a mental block for many, though,
is that this means you'll wind up having sudo rules in places you
might not expect, configuration wise. To use Ashley's example, a rule
that only exists if you have the rhythymx class applied needs to live
in the rhythymx class:
class rhythymx {
sudoers { ...
}
}
Or be included from it:
class sudo::rules::rhythymx {
sudoers { ...
}
}
class rhythymx {
include sudo::rules::rhythymx
}
Any of which will work fine using an approach similar to Digants. (I
have a script that assembles snippets of the file in some semblance of
order)
What this approach *doesn't* do well:
1) It's much harder to determine at a distance what sudoers rules will
be applied to a given host.
In the case of something like iptables rules, this is usually fine --
since the applications I need rules for are being installed with
puppet, having it handle the iptables rules makes total sense. In an
audit-heavy world, though, authorization often matters more than local
firewall rules (I've been audited on access control orders of
magnitude more often than network layer security implementation --
YMMV.)
If this was a template, auditing is easier: I pop open the template
and look. (Arguably, this is about on par with just snagging it off
the system in question)
2) Ordinality is a bitch.
Sudoers is a bad example, but iptables is a good one. I may need to
have a set of rules which must go in a particular order. With
Digant's iptables example, this is possible by naming your resources
in a way that the underlying file system sort order matches the
expected results, or by putting more than one individual rule into the
file that becomes the snippet you want to assemble, and distributing
them all at once.
This can get messy over time, as the number of resources increases.
With a template, this problem essentially disappears... it's in order
because that's the natural state of the config.
3) Sometimes, I just need to know about the world from a viewpoint
puppet doesn't easily give me.
Sudoers is a terrible example for this bullet point, since the
internal syntax is flexible enough to handle almost any situation.
Nagios is the example I always use, but since the example makes Blake
vomit a little in his mouth (Hi, Blake, you're welcome!) I'll switch
to an ssh_host_keys example. Luke uses this example in discussing
export/collect's availability here
http://www.madstop.com/export-and-collection-0-17-0-is-out.html -- go
read it, then come back to this message.
This is the logical end of the "model the world as discrete resources"
point of view. If I have a resource that is declared by the existence
of a given node, but that resource has *meaning* only in the context
of another node, then I need a way for that node to discover the
existence of those external resources. (External to the node who
cares about them, in this case.)
Lets shift the world a bit, to one where you have an external nodes
tool that's tracking just this sort of information. (iClassify does
this, your LDAP node tools could do it, and any custom tool you might
write probably will as well.) In this case, you can write a similar
ssh host key class (forgive the syntax errors, I'm writing from
memory):
class ssh_host_keys {
file { "ssh_host_keys":
path => "/etc/ssh/host_keys",
owner => root,
group => root,
mode => 0644,
content => template("ssh/host_keys.erb")
}
}
If you are using iClassify, the host_keys.erb file might look like:
<% ic = IClassify::Client.new(http://iclassify, username, password) -%>
<% ic.search("ssh_host_key:[* TO *]", [ "ssh_host_key" ]).each do |node| -%>
<%= node.attrib?("ssh_host_key") %>
<% end -%>
(The strange [* TO *] is Lucene search syntax for "everything that has
an ssh_host_key". Next build of iClassify will shorten that to *,
fwiw, along with making node["ssh_host_key"] work.)
(For those of you playing along at home, the thing that makes Blake
vomit in his mouth is how we've jammed some logic in the template,
which is a no-no in maintainable application development land, but
isn't so bad in this case, I think. ;)
Compared to this:
class ssh {
case $sshdsakey {
"": { # ignore empty keys
}
default: {
@sshkey { $hostname: type => dsa, key => $sshdsakey }
}
}
sshkey <||>
}
Along with 34 lines of Ruby driving the sshhostkey provider, and 73
lines (counting comments for both) driving the sshkey type.
To each their own, but to me, the template method is simpler and more
accurately models what I really want:
I need to keep track of everyone's ssh host key. (a fact, that must be
stored somewhere centrally) (iClassify, LDAP, Database)
I need to put them in a file in /etc/ssh/ssh_host_keys. (My File
statement and template)
I'm glad that both ways exist, but I don't think that the
export/collect model is better than a template in this case.
The urge to model sudoers in a single place, and to have that place be
the actual sudoers file you plan on distributing, seems like a totally
rational one to me. Extending the puppet language to enable that sort
of introspection isn't a bad idea, imnsho.
Heretically yours,
Adam
--
HJK Solutions - We Launch Startups - http://www.hjksolutions.com
Adam Jacob, Senior Partner
T: (206) 508-4759 E: ad...@hjksolutions.com
Adam Jacob
>
> I'll second both of these.I had the some real problems trying to get
> puppet to match my old custom SCM/CMS till I started trying to look at
> problems from the puppet language structure. Then alot of things got
> easier. Sometimes hard problems went way. Occasionally what should
> beeasy is alittle harder, but over all work the with Puppet instead of
> against it has made things much saner.
This is great advice to Systems Administrators everywhere. Working
with your tools, instead of forcing them to be something they aren't,
is the right thing.
It's like when you buy furniture from Ikea. They made it easy to
assemble, and yet, every time I buy some, I get to some point where
I'm fighting like mad to made something work. Almost always it's
because I'm doing it wrong... and stepping back and thinking about it
like a mad Danish furniture designer makes things better. (Did that
analogy get out of hand?)
Luke Kanies
>
> I just want to caveat this reply by saying that there are lots of ways
> to skin this cat. Puppet is a fabulous tool, and any disagreement I
> may have is one of implementation, not a criticism of the work that
> Luke or anyone else has put in.
>
> This reply is to Luke's message, but it's not targeted at Luke
> necessarily. Just a different point of view. :)
>
> Okay, enough dissembling..
Dissembling noted, but... We clearly disagree here, but it's a
fundamental philosophical disagreement. It's strange to me that you
(and many in the Puppet community, really) clearly see the value of
resource-based management in some places (like users) but not others
(like sudo rules). I'm just confounded.
I want to make it clear that I have no argument with your
practicality; the fact that you seem to have turned your practical
solutions into validation of a theory is what I can't understand.
> The place where this becomes somewhat a mental block for many, though,
> is that this means you'll wind up having sudo rules in places you
> might not expect, configuration wise. To use Ashley's example, a rule
> that only exists if you have the rhythymx class applied needs to live
> in the rhythymx class:
This is the bit that I don't get at all.
No one is confused that your postgres package has the postgres user;
why would you be confused that your rhythmx module has the rhythmx
sudo permissions?
You don't have a "user" module that selectively installs (or not)
users for everything; why have a "sudo" module that installs all sudo
resources?
Yeah, I agree it's a mental leap, but so was not worrying about
registers, or disk cylinders, or static typing, or basically every
other development in computers.
I completely understand that one could initially find it more
intuitive to group resources by type rather than function; what I
can't understand is why you'd stick with that once you saw there were
other options.
> What this approach *doesn't* do well:
>
> 1) It's much harder to determine at a distance what sudoers rules will
> be applied to a given host.
I don't see how this is the case. You still need to know if a given
host is a rhythmx host, and once you know this, you can either look in
a template, or just look in the rhythmx class.
I could understand if your concern were limited to a single, specific
type -- you want to audit sudo resources, but you don't care about
anything else. Once you start caring about others, though, you are
doing a many to many audit -- matching resources to either classes or
hosts -- and this audit will always be easier when the logic is
simple. I don't think anyone can argue that the logic isn't simplest
when simple booleans (whether a class is included) are the only
decisions involved.
Even simple templates with logic in them are always going to be more
complicated than a class.
But it's worse than that. If you use resources and something like
storeconfigs, you've got basically free auditing of what resources go
where, because you've got a stored inventory of every resource on
every host. You're talking one sql query. Done. Why complicate it
by storing all of your logic and/or semantics in opaque strings?
Sure, you could maybe come up with some magical indexed query on text
strings for a specific sudo text rule, but... why?
>
> In the case of something like iptables rules, this is usually fine --
> since the applications I need rules for are being installed with
> puppet, having it handle the iptables rules makes total sense. In an
> audit-heavy world, though, authorization often matters more than local
> firewall rules (I've been audited on access control orders of
> magnitude more often than network layer security implementation --
> YMMV.)
>
> If this was a template, auditing is easier: I pop open the template
> and look. (Arguably, this is about on par with just snagging it off
> the system in question)
I don't buy it. That's true for one file, but it's not true for
arbitrary resource types, and it scales horribly as your conditionals
increase. If your sudoers template has a conditional for every class
in your system, it's basically unreadable.
I know this is working for you, but that's pretty different from
saying it's the best way to do something.
>
> 2) Ordinality is a bitch.
>
> Sudoers is a bad example, but iptables is a good one. I may need to
> have a set of rules which must go in a particular order. With
> Digant's iptables example, this is possible by naming your resources
> in a way that the underlying file system sort order matches the
> expected results, or by putting more than one individual rule into the
> file that becomes the snippet you want to assemble, and distributing
> them all at once.
All you need is a system that can rely on Puppet's dependency ordering
for its own ordering, and you're done. No, the existing iptables
stuff doesn't provide this, but that doesn't say it's not possible.
>
> This can get messy over time, as the number of resources increases.
> With a template, this problem essentially disappears... it's in order
> because that's the natural state of the config.
Completely linear ordering is simple in templates, in the same way
that linear ordering is simple in traditional init scripts, but the
reason that most systems are switching to dependency-based service
ordering is that linear ordering is actually not a great system. For
simple problems it's easier, but for more complex systems, it starts
to break down pretty quickly. There's a reason why Sun adopted
dependencies for SMF instead of linear ordering.
>
> 3) Sometimes, I just need to know about the world from a viewpoint
> puppet doesn't easily give me.
> [...]
> To each their own, but to me, the template method is simpler and more
> accurately models what I really want:
>
> I need to keep track of everyone's ssh host key. (a fact, that must be
> stored somewhere centrally) (iClassify, LDAP, Database)
> I need to put them in a file in /etc/ssh/ssh_host_keys. (My File
> statement and template)
Again, this just confounds me. Using a template removes all of the
semantics of your ssh key; all that remains is a simple string. Same
with your sudo and iptables rules. Yes, simple text strings are
sufficient to get the job done, in terms of getting bits on disk, and
it will work well for simple problems. But it won't scale mentally --
because you're always thinking of files, you never make that mental
leap to "I don't care about files, just about function".
Tell the truth -- do you still think about what's in /etc/passwd and /
etc/shadow? Do you know what fields are in what order in /etc/
shadow? If not, why not? Is it because you've come to depend almost
entirely on tools that do a better job of capturing its semantics?
Do you directly manage your package databases, or do you rely on
packaging tools? Do you directly manage your services, or do you rely
on service scripts?
If it's better for users, groups, packages, services, and all of the
other "canonical" resources, why isn't it better for all resources?
Is sudo special? Are mount points special?
I just don't buy it. I think it's a limiting view.
>
> I'm glad that both ways exist, but I don't think that the
> export/collect model is better than a template in this case.
>
> The urge to model sudoers in a single place, and to have that place be
> the actual sudoers file you plan on distributing, seems like a totally
> rational one to me. Extending the puppet language to enable that sort
> of introspection isn't a bad idea, imnsho.
I agree it's rational. Wanting to know what register your data is in
is rational. Wanting to know which cylinder your data is in is
rational. Wanting to know which field of /etc/passwd stores the gecos
data is rational.
But none of them suffice for managing modern systems, and most people
have largely left them behind as obsolete techniques.
--
That was just a drill of the emergency y2k system. Had this been a
real emergency, we would've also dumped a bucket of spiders on you and
yelled out "civilization is collapsing!"
Digant C Kasundra
<ad...@hjksolutions.com> wrote:
> Any of which will work fine using an approach similar to Digants. (I
> have a script that assembles snippets of the file in some semblance of
> order)
>
> What this approach *doesn't* do well:
>
> 1) It's much harder to determine at a distance what sudoers rules will
> be applied to a given host.
>
> In the case of something like iptables rules, this is usually fine --
> since the applications I need rules for are being installed with
> puppet, having it handle the iptables rules makes total sense. In an
> audit-heavy world, though, authorization often matters more than local
> firewall rules (I've been audited on access control orders of
> magnitude more often than network layer security implementation --
> YMMV.)
>
> If this was a template, auditing is easier: I pop open the template
> and look. (Arguably, this is about on par with just snagging it off
> the system in question)
Auditing is a good point.
When we include the class for wed administrators, we know that a line will
be added to sudoers to allow members of the wed admin group to start and
stop Apache. I know puppet will do that b/c that's what I told puppet to
do. But to audit, I still have to go look at the sudoers file on the
system. Using a template doesn't help this at all because you're still
going to be relying on a variable of some sort so instead of seeing if a
class is applied to a node, you have to see if a variable is applied to a
node to see if that part of the template will be activated and therefore,
with our local security and auditing practices, I'd *still* have to go look
at the sudoers file to guarantee that Puppet did the right thing.
The template *does* give you one place to look to see what all the possible
blocks are but that too isn't too bad if you use a sudoers module with all
possible sudoers subclasses. I just prefer keeping resources declared
where they are used. To make an extreme example, I wouldn't have a file
module that declared every file I might use and use case statements to drop
them in based on hostname or tags.
>
> 2) Ordinality is a bitch.
>
> Sudoers is a bad example, but iptables is a good one. I may need to
> have a set of rules which must go in a particular order. With
> Digant's iptables example, this is possible by naming your resources
> in a way that the underlying file system sort order matches the
> expected results, or by putting more than one individual rule into the
> file that becomes the snippet you want to assemble, and distributing
> them all at once.
>
> This can get messy over time, as the number of resources increases.
> With a template, this problem essentially disappears... it's in order
> because that's the natural state of the config.
... which is why we're writing a native type for iptables. So, you'd still
use it as a resource type with explicit declarations instead of templates
or the mashing of file fragments. Also, because of the incredible
diversity of servers we model and support, we have 83 possible iptables
fragments. That would be a long template, and Puppet would have to parse
that template each time and then compare it to the existing file with each
run. The current iptables setup is nice b/c it uses a trigger based on
file changes. The iptables native type will be nice b/c it will rely on
(hopefully) faster mechanisms to audit and change iptables rules.
> Along with 34 lines of Ruby driving the sshhostkey provider, and 73
> lines (counting comments for both) driving the sshkey type.
>
> To each their own, but to me, the template method is simpler and more
> accurately models what I really want:
>
> I need to keep track of everyone's ssh host key. (a fact, that must be
> stored somewhere centrally) (iClassify, LDAP, Database)
> I need to put them in a file in /etc/ssh/ssh_host_keys. (My File
> statement and template)
>
> I'm glad that both ways exist, but I don't think that the
> export/collect model is better than a template in this case.
>
> The urge to model sudoers in a single place, and to have that place be
> the actual sudoers file you plan on distributing, seems like a totally
> rational one to me. Extending the puppet language to enable that sort
> of introspection isn't a bad idea, imnsho.
>
> Heretically yours,
> Adam
The problem here is that you're breaking apart facts of a server into
several locations, which is counter to what Puppet tries to do to make
servers look more like distinct objects. External node tools will
certainly be good for these sorts of things but if you have a template for
ssh host keys, sudoers, iptables, etc -- how many places do you have to go
update when you add a host? Remove a host? Change properties of a host?
It is the sort of thing that I feel makes the approach less scalable. It
was definitely one of the things we were trying to get away from when we
chose Puppet and the main driver into why we're planning more and more
native types and metadata types to keep information about a server tightly
coupled to the declaration of the server.
Ashley Penney
... which is why we're writing a native type for iptables. So, you'd still
use it as a resource type with explicit declarations instead of templates
or the mashing of file fragments. Also, because of the incredible
diversity of servers we model and support, we have 83 possible iptables
fragments. That would be a long template, and Puppet would have to parse
that template each time and then compare it to the existing file with each
run. The current iptables setup is nice b/c it uses a trigger based on
file changes. The iptables native type will be nice b/c it will rely on
(hopefully) faster mechanisms to audit and change iptables rules.
Digant C Kasundra
<dig...@stanford.edu> wrote:
> The template *does* give you one place to look to see what all the
> possible blocks are but that too isn't too bad if you use a sudoers
> module with all possible sudoers subclasses.
NOTE: I wouldn't use a sudoers module, but if being able to see where all
sudoers declarations where made, that is a possible way to do it. Frankly,
I still think that's backwards and doesn't add much. I'd still keep sudo
fragment declarations tied logically to where they are used.
It took a while for me to adjust to this type of thinking but when it
clicked it, the whole world made sense. I now get confused more-so if the
apache class, for instance, *doesn't* explicity declare the iptables rules,
sudo rules, etc that apply. If it is a rule for opening ports 80/443, why
wouldn't it be declared in the apache class? Etc.
Digant C Kasundra
<ape...@gmail.com> wrote:
> I suspect that the whole template vs snippit style discussion will fade
> away with time as people write types to meet the needs of the various
> things that must be configured.
I mirror Luke's confusion about this because there are people who don't
want to see more native types. I don't get it, frankly. When I first
started working with Luke on our Puppet deployment about two years ago, one
of the clear goals was always to use definitions where we could as stubs
for future native types.
We finally got someone on board to help us write those native types, but he
just quit so.... I really wish I got more time to work on these sorts of
things.
Luke Kanies
> Auditing is a good point.
>
> When we include the class for wed administrators, we know that a
> line will
> be added to sudoers to allow members of the wed admin group to start
> and
> stop Apache. I know puppet will do that b/c that's what I told
> puppet to
> do. But to audit, I still have to go look at the sudoers file on the
> system. Using a template doesn't help this at all because you're
> still
> going to be relying on a variable of some sort so instead of seeing
> if a
> class is applied to a node, you have to see if a variable is applied
> to a
> node to see if that part of the template will be activated and
> therefore,
> with our local security and auditing practices, I'd *still* have to
> go look
> at the sudoers file to guarantee that Puppet did the right thing.
Just a quick point on this -- one of the other great reasons to use
native resources is high-level auditing. You can use ralsh, or the
APIs behind ralsh, to list your native resources. E.g.:
luke@culain(0) $ ralsh user luke
user { 'luke':
comment => 'Luke Kanies',
uid => '100',
gid => '1000',
shell => '/bin/bash',
ensure => 'present',
password => 'absent',
home => '/home/luke',
groups => ['sysadmin','audio','video','puppet']
}
luke@culain(0) $
Again, that's an API that you can use over the network (and we're
working on a tool that does so). This is actually what the first
PuppetShow did -- queried your clients for the state of resources.
If you use plaintext everywhere, then you have to write your own
parsing anyway to do any kind of analysis. If you've already got
parsers in Puppet, you can use Puppet to do the parsing portion of the
audit, and stick to the analysis in your own tools.
Obviously people haven't spent as much time using this, but I've spent
a lot of time making it possible.
--
Be wary of the man who urges an action in which he himself incurs no
risk. -- Joaquin Setanti
Adam Jacob
The short reply is this one, and it deals with the particulars of
managing sudo. The other is the philosophical reply, which I'll try
and use to illustrate where I believe Luke and I diverge. (It's
pretty far down the philosophical road, honestly.)
So, on to sudo..
On Wed, May 7, 2008 at 9:44 PM, Luke Kanies <lu...@madstop.com> wrote:
> No one is confused that your postgres package has the postgres user;
> why would you be confused that your rhythmx module has the rhythmx
> sudo permissions?
Personally, I think modeling sudo in the way Ashley wanted to is
probably a mistake. As lots of other people have pointed out in this
thread, Sudo already solved this problem for you.
> You don't have a "user" module that selectively installs (or not)
> users for everything; why have a "sudo" module that installs all sudo
> resources?
Agreed.
> I completely understand that one could initially find it more
> intuitive to group resources by type rather than function; what I
> can't understand is why you'd stick with that once you saw there were
> other options.
The philosophy email will get more into that.
>> What this approach *doesn't* do well:
>>
>> 1) It's much harder to determine at a distance what sudoers rules will
>> be applied to a given host.
>
> I don't see how this is the case. You still need to know if a given
> host is a rhythmx host, and once you know this, you can either look in
> a template, or just look in the rhythmx class.
>
> I could understand if your concern were limited to a single, specific
> type -- you want to audit sudo resources, but you don't care about
> anything else. Once you start caring about others, though, you are
> doing a many to many audit -- matching resources to either classes or
> hosts -- and this audit will always be easier when the logic is
> simple. I don't think anyone can argue that the logic isn't simplest
> when simple booleans (whether a class is included) are the only
> decisions involved.
>
> Even simple templates with logic in them are always going to be more
> complicated than a class.
>
> But it's worse than that. If you use resources and something like
> storeconfigs, you've got basically free auditing of what resources go
> where, because you've got a stored inventory of every resource on
> every host. You're talking one sql query. Done. Why complicate it
> by storing all of your logic and/or semantics in opaque strings?
>
> Sure, you could maybe come up with some magical indexed query on text
> strings for a specific sudo text rule, but... why?
I would argue that you probably just need to have a single sudoers
file and be done with it, because Sudo already solved the problem of
giving you a configuration language for managing privilege escalation,
and ported it to every unix platform ever.
Otherwise, I think you, Digant, and me all agree: assemble the sudoers
file from snippets gives you the best return on Utility and
Practicality, if you really need to avoid sudo's built in mechanism
for this sort of thing.
> I don't buy it. That's true for one file, but it's not true for
> arbitrary resource types, and it scales horribly as your conditionals
> increase. If your sudoers template has a conditional for every class
> in your system, it's basically unreadable.
Absolutely. If you are modeling your sudoers file that way, I think
you are probably not leveraging sudo very well.
> I know this is working for you, but that's pretty different from
> saying it's the best way to do something.
I'll try and state, more succinctly, where you and I diverge
philosophically in the next email.
>> 2) Ordinality is a bitch.
>
> All you need is a system that can rely on Puppet's dependency ordering
> for its own ordering, and you're done. No, the existing iptables
> stuff doesn't provide this, but that doesn't say it's not possible.
Sure, you can make things ordered in a Puppet's world.
>>
>> This can get messy over time, as the number of resources increases.
>> With a template, this problem essentially disappears... it's in order
>> because that's the natural state of the config.
>
> Completely linear ordering is simple in templates, in the same way
> that linear ordering is simple in traditional init scripts, but the
> reason that most systems are switching to dependency-based service
> ordering is that linear ordering is actually not a great system. For
> simple problems it's easier, but for more complex systems, it starts
> to break down pretty quickly. There's a reason why Sun adopted
> dependencies for SMF instead of linear ordering.
Comparing the ordinality of a sudoers file to need for dependency
based init replacements is apples and oranges. The switch to
dependency based init systems like Upstart, SMF and launchd was
clearly one based on the improved practical utility of those systems.
It's not because linear ordering sucks: it's because linear ordering
sucks when you have a bunch of things that you could be doing in
parallel, or that you need to reconstruct the dependency tree to
ensure that a particular service has all of the things it needs
available.
I absolutely could write a native provider for sudo that does ordering
using puppets internal require system. I'm just not sure it's
"better".
I'll talk more about this in the philosophy email, because this is one
place we do diverge.
>> 3) Sometimes, I just need to know about the world from a viewpoint
>> puppet doesn't easily give me.
>> [...]
>> To each their own, but to me, the template method is simpler and more
>> accurately models what I really want:
>>
>> I need to keep track of everyone's ssh host key. (a fact, that must be
>> stored somewhere centrally) (iClassify, LDAP, Database)
>> I need to put them in a file in /etc/ssh/ssh_host_keys. (My File
>> statement and template)
>
> Again, this just confounds me. Using a template removes all of the
> semantics of your ssh key; all that remains is a simple string. Same
> with your sudo and iptables rules. Yes, simple text strings are
> sufficient to get the job done, in terms of getting bits on disk, and
> it will work well for simple problems. But it won't scale mentally --
> because you're always thinking of files, you never make that mental
> leap to "I don't care about files, just about function".
I made that leap already. Where we diverge is the definition of
"function". The function I care about is having all of my
ssh_host_keys filled out on every system without any manual
intervention. That is the function. Puppet gives me the leverage to
make that function a reality, but *that is the function*.
The mechanism for delivering that function, no matter what you say, is
sticking a list of keys in a file. So, the question is this one: is
it better to have:
@ssh_host_key { $fqdn:
key => $ssh_dsa_host_key
}
ssh_host_key { <| |> }
Because it delivers some greater semantic value? I don't think it
buys you much of anything at all, in a practical or semantic sense.
Is the goal to have someone who doesn't understand how an
ssh_hosts_key file is built? Then why would they write this at all?
Why would they care?
Puppet's resource model, in my opinion, specifically does *not* scale
well with increased application complexity. I'll talk more about this
in the next email.
> Tell the truth -- do you still think about what's in /etc/passwd and /
> etc/shadow? Do you know what fields are in what order in /etc/
> shadow? If not, why not? Is it because you've come to depend almost
> entirely on tools that do a better job of capturing its semantics?
Absolutely. We disagree that Puppet's semantics are the best fit for
that abstraction in every case or condition.
> Do you directly manage your package databases, or do you rely on
> packaging tools? Do you directly manage your services, or do you rely
> on service scripts?
No and No, obviously.
> If it's better for users, groups, packages, services, and all of the
> other "canonical" resources, why isn't it better for all resources?
> Is sudo special? Are mount points special?
>
> I just don't buy it. I think it's a limiting view.
I have a friend who tells a story about when he was a child. He sat
down for breakfast one morning, and they had run out of milk for his
Captain Crunch. Standing there, wondering what to do (as he had
already poured the cereal into the bowl, without first checking to see
if there was enough milk) he had the thought:
"Hey, we have some Coca-Cola. I like Coca-Cola. I like Captain
Crunch... two great things, that will go great together!"
A huge number of things involving building automated infrastructures
benefit from the resource abstraction. Almost everything that I have
encountered benefits from, at the very least, the building of useful
abstraction from the fundamental building blocks puppet provides
(which are natural fits for the abstraction layer!)
Not everything is best modeled as discrete resources tied to the nodes
which manifest them. Just most things.
> I agree it's rational. Wanting to know what register your data is in
> is rational. Wanting to know which cylinder your data is in is
> rational. Wanting to know which field of /etc/passwd stores the gecos
> data is rational.
>
> But none of them suffice for managing modern systems, and most people
> have largely left them behind as obsolete techniques.
They aren't rational when managing modern systems at scale. You and I
both agree on that point.
Ok, on to philosophy.
Adam Jacob
> with our local security and auditing practices, I'd *still* have to go look
> at the sudoers file to guarantee that Puppet did the right thing.
>
> blocks are but that too isn't too bad if you use a sudoers module with all
> where they are used. To make an extreme example, I wouldn't have a file
> module that declared every file I might use and use case statements to drop
> them in based on hostname or tags.
We are in agreement here, Digant.
>> This can get messy over time, as the number of resources increases.
>> With a template, this problem essentially disappears... it's in order
>> because that's the natural state of the config.
>
> ... which is why we're writing a native type for iptables. So, you'd still
> use it as a resource type with explicit declarations instead of templates
> or the mashing of file fragments. Also, because of the incredible
> diversity of servers we model and support, we have 83 possible iptables
> fragments. That would be a long template, and Puppet would have to parse
> that template each time and then compare it to the existing file with each
> run. The current iptables setup is nice b/c it uses a trigger based on
> file changes. The iptables native type will be nice b/c it will rely on
> (hopefully) faster mechanisms to audit and change iptables rules.
Right. It's either use puppets internal ordering, or expand the
capabilities of your external solution to more elegantly handle the
case. You make a cost benefit analysis, and move on.
> The problem here is that you're breaking apart facts of a server into
> several locations, which is counter to what Puppet tries to do to make
> servers look more like distinct objects. External node tools will
> certainly be good for these sorts of things but if you have a template for
> ssh host keys, sudoers, iptables, etc -- how many places do you have to go
> update when you add a host?
None. We model lots of things this way, and we don't make any changes
at all to our puppet manifest or templates when we add a new host.
> Remove a host? Change properties of a host?
If you change the properites of a host, say in an external node tool,
it just converges on the next run. No changes at all.
> It is the sort of thing that I feel makes the approach less scalable. It
> was definitely one of the things we were trying to get away from when we
> chose Puppet and the main driver into why we're planning more and more
> native types and metadata types to keep information about a server tightly
> coupled to the declaration of the server.
I think that decoupling the information about a server (and more
importantly, the infrastructure as a whole) is a requirement. I need
to know "what" my server will be, and information about it, in a lot
more places than just puppet. I can either re-use that information
within puppet, or I can attempt to expose puppets semantics to those
outside resources. I think it's much, much easier to have puppet
understand how to use that information than make the external tools
grok puppet.
With Puppet, I care more about the "how", not the "what".
jtimberman
> Really. I absolutely promise you'll be pleasantly surprised how much
> easier your Puppet usage becomes when you switch to thinking about
> resources.
I wrote in about an hour to completely deploy a piece of wiki software
that uses several different file and exec resources. Another is a
defined type I added to our yum configuration to handle the yum-
updatesd package and its config file.
-joshua