Classes vs. definitions (#1645)
Luke Kanies
discussion. Here's the original text from the ticket, with my
comments interspersed as necessary:
> I am a newcomer to puppet. This is bad as I don't know all the nuts
and bolts of the framework. But it is good as well as I still have a
"fresh view" of puppet as a tool. So please forgive me if I am telling
nonsense ... Anyhow I'd like to share some "fresh-view" conceptual
thoughts with you.
>
> I am really having conceptual (and practical!!) trouble with the
semantics of puppet's "classes" and "definitions" as they are defined
today. IMO the distinction is quite arbitrary and counter-intuitive.
Just so you know, this has come up multiple times, and I remain
convinced they have separate purposes.
Also, to be clear, I *wholly* encourage this kind of discussion and
recommendation. I've never claimed I have the ultimate vision of
Puppet's language, and I'm always looking for recommendations to make
it better. Of course, I will always defend the existing syntax, and I
will always fight to keep the basic vision I have intact.
The goal of a definition is to function as a composite resource,
allowing multiple Puppet resources to look like a single resource.
This makes it easy to do things like have an exec look like an
iptables rule, for instance. The key to definitions is that they
allow you to create new resource types that can be used with the exact
same syntax as builtin resources.
The point behind classes, however, is to delineate all of the
resources necessary to provide a given service. I often call them
'service classes', rather than just 'classes'.
Given this, there are really two questions:
1) Are these actually two separate purposes?
2) Have I chosen syntax that does a good job of meeting each purpose?
I maintain these are two clearly separate purposes.
However, I tend to think that the syntaxes aren't necessarily that
great, and I definitely agree that we need more functionality for
configurating classes while at the same time the functionality we have
is a bit not-so-great.
> AFAICS there are three main distinctions between classes and
definitions:
>
> 1) classes are singletons from a node perspective, definitions are
instantiable
> 2) definitions receive parameters, classes do not
> 3) classes can declare "class variables" ($class::var), definitions
cannot
These are more syntactical and structural distinctions, but to me, the
distinctions that matter are how they are and should be used.
> I have seen many examples in "recipes" where classes were used like
this:
>
> node abc {
> $globalvar1 = ...
> $globalvar2 = ... }
>
> include xyz
>
> class xyz {
> ...template("consumes <%= globalvar1 %>...")... }
>
> file { ...owner = $globalvar2... }
>
> IMO this is counter to basic principles of information hiding and
modularization. (The most exhaustive discussion of the disadvantages
of such an approach is probably in Steve McConnell's "Code Complete").
If you do not have the most simple classes in your manifests then
puppet invites you to use global variables. I can already see from the
few things I did that this quickly becomes a maintenance nightmare and
makes puppet classes very difficult to use in a more complex setting.
I agree, and I've been trying to find a better syntax for this. I
don't think the answer is to just use definitions, though.
> Definitions accept construction parameters and are therefore able
to hide their inner workings and complexity from outer context. This
makes them more flexible and attractive for modularization than
classes. Definitions are however not meant for usage as node
singletons. IMO it's always bad if you try to coerce a syntactical
construct into a role that it wasn't meant for. Defintions also have
another big disadvantage: They do not allow for instance variables
like classes do (something like $mydefinition["xyz"]::myvar).
>
> Furtheron the distinction between "classes as singletons" and
"definitions as prototypes" seems quite arbitrary to me. What is
called a "class" in puppet is conceptually a class with "automatic
instantiation" (with one "class instance" per node). This is confusing
to me as an OO programmers who considers classes and instances
separate conceptual entities. On the other hand what is called a
"definition" in puppet is conceptually really a class as it defines a
"blueprint" of an object that can be instantiated separately. The
conceptual inconsistency reveals itself in the Class["classname"]
syntax which is a workaround for the fact that puppet classes are
instances of classes as well. Definitions on the other hand can be
referenced quite intuitively with the Definition_name["instance id"]
syntax.
Part of the problem is that Puppet's language isn't a real
"programming" language, but I (possibly foolishly) chose many terms
that mean somewhat different things in other languages. Part of the
reason I did this is I was reusing language in tools like cfengine.
But the point remains: A Puppet class is more of a service class
than something like an OO class; it can also be thought of as the code
that specifies a class of machines -- that is, if you've got a
sendmail-class server, then you get sendmail-related resources.
> IMO the conceptual distinction between classes and definitions is
confusing and unnecessary at the same time. To me it would be much
simpler if such a distinction did not exist. Rather than two resource
containers you could simply have one generic class container without
loosing any functionality and winning flexibility, intuitivity and
maintainability:
>
> Here my propositions:
> 1) Classes should be reframed as universal "blueprints" of resource
containers that can (and must) be instantiated (like definitions today).
How would you instantiate a class?
> 2) All classes may define constructor parameters (like definitions
today).
This seems reasonable and desirable.
> 3) Classes may be defined as node level "singletons" or node level
"prototypes". The distinction between singleton and prototype is well
known to most OO programmers.
How would this distinction be made?
> 4) Classes can define public instance variables that may be
accessed from an outer context (like they do today). The access syntax
would be ($Classname["instance id"]::myvar).
Ugh. Not so fond of this. Really, I'd like to get rid of the
existing syntax, I just knew that something like it was necessary
until we had a better way of providing good encapsulation and
configuration for classes.
And what about singleton classes? Would you support classes without
instance names, and if so, what would that syntax look like?
> 5) Instances of classes can be referenced like definitions today:
Classname["instance id"]
This is just an automatic result of other bits above.
> All this can be introduced in a fully backwards compatible way
(which is certainly a must-have!):
> 1) Definitions will continue to exist but will be deprecated.
Don't definitions provide most of the functionality you want in your
classes? Wouldn't it make more sense to use them as the new 'class'
construct?
> 2) Classes will be node singletons by default.
> 3) Class constructor parameter declaration is optional.
> 4) Singleton classes can still be referenced as Class["xyz"]. This
usage will however be deprecated.
> 5) Class variables can still be referenced as $classname::var. This
usage will however be deprecated.
What about singleton classes classes that don't have multiple instances?
> 6) You can define "inner classes" to replace today's construct of
definitions within class context.
So today's definitions could only ever be defined inside other classes?
> Here is how I'd define the syntax of the ideal resource container
(replacing both classes and definitions), not really BNF but
somehow ;-) :
>
> {singleton|prototype} class <classname> [($par1, [$par2, [...]])] {
> ...
> $instance-var = ...
> ... {singleton|prototype} class <inner class> [($par1, [$par2,
[...]])] { ... }
> }
I can't figure out what you mean here. Can you show me some example
classes?
> The whole thing certainly is not very well thought out yet. But I'd
love to see a discussion starting from here as I am really fighting
with the current syntax...
>
--
Wear the old coat and buy the new book. -- Austin Phelps
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
jerico
thanks for your reply. I really appreciate that you took the time to
respond... and IMO you brought forward some very good arguments! :-)
> Also, to be clear, I *wholly* encourage this kind of discussion and
> recommendation.
> Of course, I will always defend the existing syntax, and I
> will always fight to keep the basic vision I have intact.
> The goal of a definition is to function as a composite resource,
> allowing multiple Puppet resources to look like a single resource.
> resources necessary to provide a given service. I often call them
> 'service classes', rather than just 'classes'.
to someone who worked with cfengine for years. I was aware of
cfengine's semantics and grammar when I made my post. I however
believe that cfengine did very much "organically grow" over the years
rather than being "consistent" or "well thought out". Puppet already
does a *good* job in addressing some inconsistencies in cfengine
though!
> Given this, there are really two questions:
>
> 1) Are these actually two separate purposes?
> 2) Have I chosen syntax that does a good job of meeting each purpose?
>
> I maintain these are two clearly separate purposes.
>
> However, I tend to think that the syntaxes aren't necessarily that
> great, and I definitely agree that we need more functionality for
> configurating classes while at the same time the functionality we have
> is a bit not-so-great.
that encapsulation and information hiding can be enforced with classes
then my main objection to classes would vanish and I as a user would
be completely happy with that, even if you maintained the semantical
distinction between classes and definitions.
Btw: Now that I worked a little more with puppet I see the same
problem with templates. They don't declare the parameters they use.
IMO that's a not-so-good idea. But this is somehow less important in
practice as templates are a lot more "local" than classes are.
I also agree with you that public class variables (or potentially
public attributes of definitions) are not really necessary although I
do not consider them "evil". IMO they serve the purpose of providing
self-documenting parameters to other services (e.g. being able to use
ssh::port in a firewall rule is nice) without breaking encapsulation.
They also help to explicitly identify and declare dependencies between
services (i.e. "automatic require relationships"). And as "definition
attributes" they would provide a nice way of accessing internal read-
only parameters (e.g. "package::platform") which is not possible today
AFAIK. If public class/definition attributes were missing, however, I
could well do without them.
You also said that deprecating syntax constructs is not "backwards
compatibility" and you are right with that! I admit that my "backwards
compatibility" was rather superficial or simply wrong. Adding
"instantiation parameters" to classes and public attributes to
definitions would however be fully backwards compatible I think. I
withdraw the rest of my proposal.
So overall that's already it. If you are short of time then you can
stop reading here and you got all the most important. I think we now
mainly agree on which improvements could be made if someone found the
time and priority to do it.
**************************************************
If you're interested in a more "abstract" discussion of my "ideal"
definition language for a configuration tool, then you may read on. I
think I gave answers to all of your more specific questions here.
You have been warned though! The remainder is more of an article than
of a post. And I won't consider backwards compatibility from now on as
the above practical solution does a good job on that I think.
Anyway: I think that being fully aware of what "would be nice" if
there wasn't backwards compatibility is wonderful for you as a
language designer because it gives you direction in implementation
details even if you cannot implement the "pure" solution for practical
reasons. It also helps users like myself to program into the language
rather than programming in the language (see the book "Code Complete"
again for that distinction) thereby avoiding maintenance problems.
Both are real practical and money-worthy advantages although based on
"conceptual" rather than "implementation" knowledge. I am a
practitioner and not a scientist!!
> Part of the problem is that Puppet's language isn't a real
> "programming" language
procedural/imperative language and is probably not turing complete (no
idea about that, I didn't try the proof!) But maybe this is just an
argument about wording and therefore not so important.
> But the point remains: A Puppet class is more of a service class
> than something like an OO class; it can also be thought of as the code
> that specifies a class of machines -- that is, if you've got a
> sendmail-class server, then you get sendmail-related resources.
describe bundles of configured resources. But that's already it, no
further distinction necessary.
The rest is cfengine heritage which I do not consider valuable when
defining the "ideal" language based on design best-practices. It
certainly is very valuable for cfengine adepts who "upgrade" to
puppet. But that's not the point here (see the practical solution
above for cfengine-think compatible changes).
What you probably want to say is that cfengine or puppet classes are
somehow cross-cutting aspects of nodes (in AOP words) while
definitions are not. But what's the real purpose of AOP? It's just
introducing or modifying procedural methods in bulk to avoid code
duplication. This does not make sense in a language without a
procedural element! My hypothesis is: Any non-procedural language can
be considered class- and aspect-oriented at the same time. If I
introduce a puppet definition into a puppet class then you introduce
some powerful "aspect" into this class using a very concise language
without any code duplication. You could always factor out repetitive
code into puppet classes /or/ definitions without any problems even
when this code will be used in completely different contexts later. So
no required distinction here either.
If you think well then an OO language is nothing other than a hybrid
declarative/imperative language. If you leave out the imperative part
(i.e. methods!) then you have a very nice framework for a fully
declarative language which is very similar to puppet's language. If
you'd even disallow public attributes then you'd get something very
similar to one of those many functional languages "à la
mode" (Haskell, Erlang, ...) these days.
To fully "disillusion" the cfengine heritage: To me a "node" is a
resource as well, I don't see any required semantic difference here to
other resources like a web server software or a firewall. I could also
imagine having higher level clusters of nodes being treated as a
"resource bundle". Why not define a class of nodes with instantiation
parameters and thereby handle a cluster of machines as "one thing"? If
we had a construct like that I could probably eliminate another bunch
of global variables and redundancy in my puppet scripts.
IMO it is completely irrelevant whether you add a resource (node) to a
class (in cfengine parlor) or whether a class bundles resources as
puppet's definitions do. If you agree that a node is a resource and
that aspects and classes are essentially the same in a declarative
language then what's the difference??? You end up with classified (or
bundled) resources, don't you? Just a chicken or egg problem IMO.
The same applies to puppet types. They just abstract away some OS
level resources. To me types are not necessarily semantically
different from classes, nodes or definitions.
So even the node and type constructs become unnecessary if you think
like that. You could express everything you have today in puppet with
resource bundles that take instantiation parameters, allow inheritance
and maybe define public attributes. I bet I could reformulate
everything I can do in puppet today with a simplistic language like
that.
Here another "disillusionment", this time concerning singletons and
prototypes: A singleton is always only a singleton within a certain
context. So in Java or C++ you might do some tricks to define a
singleton. But then you end up with a singleton in the context of a
class loader (Java) or process (C++)! As soon as you work with
application server clusters (Java) or several processes in parallel (C+
+), you have to synchronize all those "singletons" or invent
singletons on cluster level (what some application servers and
parallel processing frameworks effectively do). Think cloud computing
and you're at the next level of abstraction, and so on...
The exact same applies to puppet singletons. Puppet classes may be
singletons on a single node but yes they are prototypes if you
consider a cluster of nodes as an entity in its own right. The fact
that you can safely and implicitly consider classes singletons in
puppet is because you are running puppetd on a node. But what if you
wanted to cluster nodes one day as one "thing" to avoid the current
duplication of node-specific code in puppet? Oups! You'd have to
introduce ... named puppet class instances.
IMO a resource can only be a singleton in the context of a bundle. Not
more not less.
So once we see that we really only have to describe bundled resources
then we can do with one single syntactical construct. Let's call it a
"bundle" for now to avoid all mis-interpretations in the sense of OO
concepts or functional languages or cfengine heritage.
Here a translation table for cfengine and puppet semantics into
"bundle semantics":
cfengine input files (bundle) group cfengine action instances
(resources)
cfengine classes (bundle) group hosts (resources)
puppet nodes (bundle) group puppet classes and definitions (resources)
puppet classes (bundle) group puppet types, classes and definitions
(resources)
puppet definitions (bundle) group puppet types, classes and
definitions (resources)
puppet types (bundle) group OS configuration files, services,
packages, etc. (resources)
Imagine what such a simplification would mean for documentation and
parser implementation. You'd probably be able to through away large
junks of specialised node, class, definition and type code/
documentation. And doing the same thing with less code is always a
good thing. (Wasn't it Mark Twain who apologized that he couldn't
write a shorter text because he didn't have the time to do so? I
apologize too...)
But yes: You cannot do this in a backward compatible way.
Based on this long "foreword" I can quickly answer your remaining
questions:
> How would you instantiate a class?
> instance names, and if so, what would that syntax look like?
If we re-interprete a class as a bundle then we get:
bundle-name { par1 => ..., par2 => ..., ... }
This looks exactly as an instance of today's definition without an
instance name. The instance name is however implicitly defined and
defaults to the containing bundle's name.
For a class in the context of a "node bundle" this would automatically
be the node name. If you nest classes then the node name would
automatically propagate to lower level classes. So this is just a
special case within the general bundle framework now. A very elegant
solution IMO.
Including classes within definitions is possible today. I don't
understand what this really means in your own definition of classes as
node services and definitions as resource bundles. So I won't use it
now. Anyway there is an easy workaround for this in our "bundle
framework" by including the class "bundle" at a higher level. This is
semantically identical and IMO cleaner anyway as it enforces real
encapsulation and disallows code duplication. As we are not concerned
with backwards compatibility here this is ok.
More generally: Any instantiation without name would be considered a
"singleton" instantiation in the context of its bundle. The instance
name would always default to the bundle's instance name. If you try to
instantiate two "name-less" bundles within the same context you'd get
an error. Like that we could even do away with the "singleton/
prototype" keyword that I proposed in my initial post. And you'd get a
very nice upgrade path if you discover all of a sudden that you want
two ssh services running in parallel on one node. Simply transform
your ssh singleton into an ssh prototype by giving it an explicit
name.
> Don't definitions provide most of the functionality you want in your
> classes? Wouldn't it make more sense to use them as the new 'class'
> construct?
exactly my intended target construct. I only retained the "class" word
rather than the "definition" word as it is closer to what is commonly
seen as a class in OO languages and rings the right bell in the head
of somebody trained in OO (of which you probably have as many if not
more as former cfengine users). To keep "the right part" of your
cfengine thinking you might also call it an "aspect" following my
hypothesis of class/aspect equality in declarative languages above .
> > 6) You can define "inner classes" to replace today's construct of
> > definitions within class context.
>
> So today's definitions could only ever be defined inside other classes?
within the top level context. I think that this has become obvious
from the previous discussion.
Hope that I could answer all your questions. I hope that this was at
least an interesting read though maybe not what can be actually
implemented in the foreseeable future.
Florian
Luke Kanies
enough that it could take over my life if I let it.)
On Oct 31, 2008, at 10:33 AM, jerico wrote:
> [...]
>> The goal of a definition is to function as a composite resource,
>> allowing multiple Puppet resources to look like a single resource.
>> The point behind classes, however, is to delineate all of the
>> resources necessary to provide a given service. I often call them
>> 'service classes', rather than just 'classes'.
>
> I think this distinction inherited a lot of cfengine and comes natural
> to someone who worked with cfengine for years. I was aware of
> cfengine's semantics and grammar when I made my post. I however
> believe that cfengine did very much "organically grow" over the years
> rather than being "consistent" or "well thought out". Puppet already
> does a *good* job in addressing some inconsistencies in cfengine
> though!
I agree it's somewhat a hold-over from Cfengine, but it's actually
more closely tied (at least in function, if not terminology) to my
ISconf 3 implementation, which had nodes and types, and each type had
a list of work associated with it (ISconf used standard make stanzas
for work, so it was all shell scripts).
I implemented the same basic idea in Puppet, but instead of a fixed
list of resources per class, you get a more powerful language (ISconf
just supported attributes or stanza names -- no conditionals or
anything crazy like that).
Either way, my previous point still stands -- I'd rather us develop a
comprehensive view of how this should really work, and then work
toward that, rather than just hacking admittedly-imperfect design so
it's slightly more usable.
In other words, I think the low-cost approach (fixing current stuff)
is not the right idea here, and a higher-cost approach (where cost is
mostly design) is the right solution.
>
>> Given this, there are really two questions:
>>
>> 1) Are these actually two separate purposes?
>> 2) Have I chosen syntax that does a good job of meeting each purpose?
>>
>> I maintain these are two clearly separate purposes.
>>
>> However, I tend to think that the syntaxes aren't necessarily that
>> great, and I definitely agree that we need more functionality for
>> configurating classes while at the same time the functionality we
>> have
>> is a bit not-so-great.
>
> If classes gained the ability to declare "instantiation parameters" so
> that encapsulation and information hiding can be enforced with classes
> then my main objection to classes would vanish and I as a user would
> be completely happy with that, even if you maintained the semantical
> distinction between classes and definitions.
This would obviously be straightforward, at this point, in that one
could just use the existing definition syntax for adding parameters,
although it would further confuse the distinction between them.
I'm amenable to this change as a short-term mechanism for adding
attributes to classes, but I would tend to require a change with it:
The '$class::attribute' syntax would only be able to see the
attributes mentioned in the prototype, rather than any internal
variables.
I still think a long-term vision needs to be clarified, though.
>
> Btw: Now that I worked a little more with puppet I see the same
> problem with templates. They don't declare the parameters they use.
> IMO that's a not-so-good idea. But this is somehow less important in
> practice as templates are a lot more "local" than classes are.
Exactly -- templates are inherently local, and really, there's no way
that I know of you could do them any other way. Not unless you wanted
to invent your own templating system, which I think is a real mistake.
>
> I also agree with you that public class variables (or potentially
> public attributes of definitions) are not really necessary although I
> do not consider them "evil". IMO they serve the purpose of providing
> self-documenting parameters to other services (e.g. being able to use
> ssh::port in a firewall rule is nice) without breaking encapsulation.
> They also help to explicitly identify and declare dependencies between
> services (i.e. "automatic require relationships"). And as "definition
> attributes" they would provide a nice way of accessing internal read-
> only parameters (e.g. "package::platform") which is not possible today
> AFAIK. If public class/definition attributes were missing, however, I
> could well do without them.
You'd retain all of this; you just wouldn't have it to *all*
variables, just those that are mentioned in the prototype.
>
> You also said that deprecating syntax constructs is not "backwards
> compatibility" and you are right with that! I admit that my "backwards
> compatibility" was rather superficial or simply wrong. Adding
> "instantiation parameters" to classes and public attributes to
> definitions would however be fully backwards compatible I think. I
> withdraw the rest of my proposal.
>
> So overall that's already it. If you are short of time then you can
> stop reading here and you got all the most important. I think we now
> mainly agree on which improvements could be made if someone found the
> time and priority to do it.
Great. I'll respond to the rest of your email in a different email,
with a new thread heading. Hopefully we can keep this thread trim,
while the other spirals into theory and star-gazing. :)
--
The great tragedy of Science - the slaying of a beautiful hypothesis by
an ugly fact. --Thomas H. Huxley
Luke Kanies
> definition language for a configuration tool, then you may read on. I
> think I gave answers to all of your more specific questions here.
I definitely am. I'm going to trim as much as possible, though, to
make the conversation manageable.
>
> You have been warned though! The remainder is more of an article than
> of a post. And I won't consider backwards compatibility from now on as
> the above practical solution does a good job on that I think.
>
> Anyway: I think that being fully aware of what "would be nice" if
> there wasn't backwards compatibility is wonderful for you as a
> language designer because it gives you direction in implementation
> details even if you cannot implement the "pure" solution for practical
> reasons. It also helps users like myself to program into the language
> rather than programming in the language (see the book "Code Complete"
> again for that distinction) thereby avoiding maintenance problems.
> Both are real practical and money-worthy advantages although based on
> "conceptual" rather than "implementation" knowledge. I am a
> practitioner and not a scientist!!
I don't think backward compatibility is always required, as long-time
members would attest, but I strive for it when possible, and, at the
least, always do my best to be clear when it's absent.
>
>> Part of the problem is that Puppet's language isn't a real
>> "programming" language
>
> I don't agree on that. It might be a specially declarative and less
> procedural/imperative language and is probably not turing complete (no
> idea about that, I didn't try the proof!) But maybe this is just an
> argument about wording and therefore not so important.
I agree it's mostly a terminology point, but it's definitely not
Turing-complete, although Brice might have slipped that in with his
last series of updates. :)
>
>> But the point remains: A Puppet class is more of a service class
>> than something like an OO class; it can also be thought of as the
>> code
>> that specifies a class of machines -- that is, if you've got a
>> sendmail-class server, then you get sendmail-related resources.
>
> I agree with you that it should be the purpose of puppet's language to
> describe bundles of configured resources. But that's already it, no
> further distinction necessary.
>
> The rest is cfengine heritage which I do not consider valuable when
> defining the "ideal" language based on design best-practices. It
> certainly is very valuable for cfengine adepts who "upgrade" to
> puppet. But that's not the point here (see the practical solution
> above for cfengine-think compatible changes).
I disagree with this -- you're coming at it from the implementation,
but I'm thinking about it from the perspective of intent. If
anything, I had to completely invert Cfengine's idea of a class, and
every Cfengine refugee we get has to go through an adoption period
where they rewire their brains.
No one starts with code, even when they think they do -- you always
start with intent. Why are you building this node? What services
should it offer? Why?
To me, this is how you should be able to think about Puppet code:
"This node exists because it's going to be a web server. Web servers
run Apache and MySQL, they need perl installed. It's also on my
network, and every node on my network is a dns client and has sysadmin
users installed on it. The machine is in my Nashville datacenter, and
it should get a router appropriate to that datacenter."
To me, the term 'class' is a convenient way to describe the
configuration associated with a given intent. "Web server" means one
list of resources, "dns client" means another, etc.
So, that's from the direction of specification; now let's talk about
data production. When a client produces a log message, that message
requieres context to be useful. If it fails to install a package, we
need to know why (again, the intent) that package was needed, so we
can then determine what services would be affected. We can also
relate that failure back to code changes, maybe, so it's easier to
resolve the problem network-wide.
>
> What you probably want to say is that cfengine or puppet classes are
> somehow cross-cutting aspects of nodes (in AOP words) while
> definitions are not. But what's the real purpose of AOP? It's just
> introducing or modifying procedural methods in bulk to avoid code
> duplication. This does not make sense in a language without a
> procedural element! My hypothesis is: Any non-procedural language can
> be considered class- and aspect-oriented at the same time. If I
> introduce a puppet definition into a puppet class then you introduce
> some powerful "aspect" into this class using a very concise language
> without any code duplication. You could always factor out repetitive
> code into puppet classes /or/ definitions without any problems even
> when this code will be used in completely different contexts later. So
> no required distinction here either.
I don't think I really understand what you're saying here. Note that
I'm entirely self-trained, so a lot of what you're talking about goes
right over my head. I read a lot of books, but I don't always absorb
or agree.
> To fully "disillusion" the cfengine heritage: To me a "node" is a
> resource as well, I don't see any required semantic difference here to
> other resources like a web server software or a firewall. I could also
> imagine having higher level clusters of nodes being treated as a
> "resource bundle".
I basically agree, with one significant distinction -- nodes have
automatic entry points (i.e., the parser accepts a request from a host
and uses the host's name to look up a node instance). Neither classes
nor definitions have automatic entry.
Otherwise, I agree, and the lack of meaningful distinction is one
thing that led me to want to push nodes outside of Puppet.
> Why not define a class of nodes with instantiation
> parameters and thereby handle a cluster of machines as "one thing"? If
> we had a construct like that I could probably eliminate another bunch
> of global variables and redundancy in my puppet scripts.
This is more of a grouping problem in general -- Puppet can't really
talk about anything that's cross-host, and it's a significant
problem. I think of this as the 'me/it' problem -- Puppet can only
talk about 'me' (what services should I be running?) rather than
'it' (what services should this host or that host be running?). Been
a while since I thought about this aspect, but it's definitely there
and definitely a problem.
How does your 'class of nodes' idea change this?
>
> IMO it is completely irrelevant whether you add a resource (node) to a
> class (in cfengine parlor) or whether a class bundles resources as
> puppet's definitions do. If you agree that a node is a resource and
> that aspects and classes are essentially the same in a declarative
> language then what's the difference??? You end up with classified (or
> bundled) resources, don't you? Just a chicken or egg problem IMO.
I don't really agree that classes and aspects are essentially the
same. In fact, if an aspect is a cross-cutting concern, then Puppet's
classes are actually explicitly not aspects -- they're completely
exclusive. No two classes can overlap at all.
IMO, one of the significant weaknesses of Puppet's language is its
completely lack of ability to specify cross-cutting ideas. The
closest you get is global variables in the top scope.
For instance, if you want to say "All Debian hosts should default to
'aptitude' for package provider", your only real choice is to put that
in your site.pp file. There's no other way to guarantee all of your
classes will get it. There's no 'class' construct you can use that
will intersect all classes.
>
> The same applies to puppet types. They just abstract away some OS
> level resources. To me types are not necessarily semantically
> different from classes, nodes or definitions.
You're kind of losing me here. Resource types theoretically model
some clear resource on the system; if you lose that, you lose a lot,
IMO.
>
> So even the node and type constructs become unnecessary if you think
> like that. You could express everything you have today in puppet with
> resource bundles that take instantiation parameters, allow inheritance
> and maybe define public attributes. I bet I could reformulate
> everything I can do in puppet today with a simplistic language like
> that.
I can't disagree with that, but my question would be, would you have a
"better" language if you did this? And it must be said, "better" here
is defined as "better for its audience". Given that we're not talking
to declarative programmers with CS degrees and 5 years of programming,
but rather are talking to jack-of-all-trades sysadmins, the answer
varies a bit.
And even if you are talking to the programmers (many of whom do use
Puppet), I think a slightly more complex, more explicit language is
often actually a better idea. Every problem can be expressed in LISP,
but that doesn't make LISP the best way to solve every problem. From
what I understand of LISP culture, their whole point is to build a DSL
for every problem and then live there. My goal here is to build a DSL
that provides the most readable and maintainable way of specifying
infrastructure; I'm less concerned with having the simplest language.
>
> Here another "disillusionment", this time concerning singletons and
> prototypes: A singleton is always only a singleton within a certain
> context. So in Java or C++ you might do some tricks to define a
> singleton. But then you end up with a singleton in the context of a
> class loader (Java) or process (C++)! As soon as you work with
> application server clusters (Java) or several processes in parallel
> (C+
> +), you have to synchronize all those "singletons" or invent
> singletons on cluster level (what some application servers and
> parallel processing frameworks effectively do). Think cloud computing
> and you're at the next level of abstraction, and so on...
This is clearly the case -- a given resource must be unique within a
host's catalog, but not usually unique within a network.
>
> The exact same applies to puppet singletons. Puppet classes may be
> singletons on a single node but yes they are prototypes if you
> consider a cluster of nodes as an entity in its own right. The fact
> that you can safely and implicitly consider classes singletons in
> puppet is because you are running puppetd on a node. But what if you
> wanted to cluster nodes one day as one "thing" to avoid the current
> duplication of node-specific code in puppet? Oups! You'd have to
> introduce ... named puppet class instances.
Can you elaborate on this?
>
> IMO a resource can only be a singleton in the context of a bundle. Not
> more not less.
I agree, and that bundle is currently named a 'catalog' in Puppet.
0.25 finally makes this Catalog class the arbiter of singleton-hood.
>
> So once we see that we really only have to describe bundled resources
> then we can do with one single syntactical construct. Let's call it a
> "bundle" for now to avoid all mis-interpretations in the sense of OO
> concepts or functional languages or cfengine heritage.
Urgh, I hate the term, but I'll follow along for now.
>
> Here a translation table for cfengine and puppet semantics into
> "bundle semantics":
>
> cfengine input files (bundle) group cfengine action instances
> (resources)
> cfengine classes (bundle) group hosts (resources)
> puppet nodes (bundle) group puppet classes and definitions (resources)
> puppet classes (bundle) group puppet types, classes and definitions
> (resources)
> puppet definitions (bundle) group puppet types, classes and
> definitions (resources)
> puppet types (bundle) group OS configuration files, services,
> packages, etc. (resources)
>
> Imagine what such a simplification would mean for documentation and
> parser implementation. You'd probably be able to through away large
> junks of specialised node, class, definition and type code/
> documentation. And doing the same thing with less code is always a
> good thing. (Wasn't it Mark Twain who apologized that he couldn't
> write a shorter text because he didn't have the time to do so? I
> apologize too...)
As discussed above, I don't always agree. And you'd be surprised how
little extra code there is right now -- Node and Class are subclasses
of Definition, and it's not that complicated.
>
> But yes: You cannot do this in a backward compatible way.
>
> Based on this long "foreword" I can quickly answer your remaining
> questions:
>
>> How would you instantiate a class?
>> And what about singleton classes? Would you support classes without
>> instance names, and if so, what would that syntax look like?
>> How would [the] distinction [between singletons and prototypes] be
>> made?
>
> If we re-interprete a class as a bundle then we get:
>
> bundle-name { par1 => ..., par2 => ..., ... }
>
> This looks exactly as an instance of today's definition without an
> instance name. The instance name is however implicitly defined and
> defaults to the containing bundle's name.
I actually started out with exactly this syntax, and almost all of the
behaviour you're talking about. I ended up removing it in the end,
but I didn't replace the functionality as I probably should have.
I don't like the idea of declaring class membership via the same
syntax for specifying resources, partially because I think most people
will find that the attributes that are consumed by a class are often
specified in a different location than the code that specifies class
membership.
Take a dns client -- you'd probably have a 'base' class or node that
said every node is a dns client, and your dns client class would
require something like a resolver list and a search path. These
values would be filled in by some Datacenter or Location or whatever
class.
In the current system, this Location class would need to have a
heirarchical relationship to the base class, in order to override as
necessary. However, you'd soon find yourself with 50 subclasses of the
base class, one for each aspect that needed to specify overrides.
Thus, I think any attempt at solving the problems you're describing
should attempt to solve this problem, too, and really, I think this is
a greater problem, because it speaks of something you just can't do
well right now, whereas you're mostly focused on something that's
straightforward but moderately ugly.
I've been thinking of something like an aspect syntax, which could
have maybe attributes and defaults, but no resources. This could then
be included/imported/whatever into any class, such that it could then
configure that class. You might do something like this:
class dnsclient($resolvers,$domain) {
...
}
aspect location {
$resolvers = "..."
$domain = "..."
}
aspect location::nashville inherits location {
$resolvers = "something else"
$domain = "other"
}
class base {
include dnsclient
}
node mynode {
include base
acquire location::nashville
}
Then you just need some kind of validator to make sure that every
class has all of its parameters provided. It might also be a good
idea to differentiate between aspect attributes, which would be
imported when an aspect is 'acquired', and variables, which would not
be.
It seems we also need different ways of specifying relationships
between classes. Right now, a class can use 'include' to declare that
a class should be evaluated, but that's basically it. It'd be nice to
be able to specify a dependency, but it'd also be nice to be able to
somehow merge two classes, akin to the aspect acquisition mentioned
above.
>
> For a class in the context of a "node bundle" this would automatically
> be the node name. If you nest classes then the node name would
> automatically propagate to lower level classes. So this is just a
> special case within the general bundle framework now. A very elegant
> solution IMO.
>
> Including classes within definitions is possible today. I don't
> understand what this really means in your own definition of classes as
> node services and definitions as resource bundles. So I won't use it
> now. Anyway there is an easy workaround for this in our "bundle
> framework" by including the class "bundle" at a higher level. This is
> semantically identical and IMO cleaner anyway as it enforces real
> encapsulation and disallows code duplication. As we are not concerned
> with backwards compatibility here this is ok.
I'm not really sure what you mean in this section.
>
> More generally: Any instantiation without name would be considered a
> "singleton" instantiation in the context of its bundle. The instance
> name would always default to the bundle's instance name. If you try to
> instantiate two "name-less" bundles within the same context you'd get
> an error. Like that we could even do away with the "singleton/
> prototype" keyword that I proposed in my initial post. And you'd get a
> very nice upgrade path if you discover all of a sudden that you want
> two ssh services running in parallel on one node. Simply transform
> your ssh singleton into an ssh prototype by giving it an explicit
> name.
What if you have multiple classes that want to declare that they
require a given class?
Currently this is done by each of them calling 'include' on the class,
but I assume you do away with the 'include' function, and now these
classes have an error.
I don't think this is a good idea -- in this way, classes can be cross-
cutting, in that many classes might care that a given class is
instantiated, so it's important that they can declare this.
>
>> Don't definitions provide most of the functionality you want in your
>> classes? Wouldn't it make more sense to use them as the new 'class'
>> construct?
>
> Yes absolutely! If you add public parameters to definitions, they are
> exactly my intended target construct. I only retained the "class" word
> rather than the "definition" word as it is closer to what is commonly
> seen as a class in OO languages and rings the right bell in the head
> of somebody trained in OO (of which you probably have as many if not
> more as former cfengine users). To keep "the right part" of your
> cfengine thinking you might also call it an "aspect" following my
> hypothesis of class/aspect equality in declarative languages above .
Ok.
>
>>> 6) You can define "inner classes" to replace today's construct of
>>> definitions within class context.
>>
>> So today's definitions could only ever be defined inside other
>> classes?
>
> No. Definitions or "bundles" as I like to call them now, can live
> within the top level context. I think that this has become obvious
> from the previous discussion.
>
> Hope that I could answer all your questions. I hope that this was at
> least an interesting read though maybe not what can be actually
> implemented in the foreseeable future.
None of it's that hard, but I'm not yet convinced it's the right
direction.
--
SELF-EVIDENT, adj. Evident to one's self and to nobody else.
-- Ambrose Bierce
Florian Grandel
> In other words, I think the low-cost approach (fixing current stuff)
> is not the right idea here, and a higher-cost approach (where cost is
> mostly design) is the right solution.
That's ok for me.
> This would obviously be straightforward, at this point, in that one
> could just use the existing definition syntax for adding parameters,
> although it would further confuse the distinction between them.
That's not a problem for me as I don't see a profound difference anyway. ;-)
> I'm amenable to this change as a short-term mechanism for adding
> attributes to classes, but I would tend to require a change with it:
> The '$class::attribute' syntax would only be able to see the
> attributes mentioned in the prototype, rather than any internal
> variables.
What do you mean by prototype? The class definition? (As opposed to a
class instance?) I am used to the distinction "prototype vs. singleton",
that's why I might be confused by your use of the term "prototype" here.
If you mean the class definition by "prototype" then I didn't get your
point. I still see various possibilities to maintain other attributes
than those passed into a class as "parameters":
- You can use "$class::attribute" with constant values, even derived or
somehow calculated values are ok as long as they are constant at
runtime. This means the value can be declared in the class definition
and is otherwise read-only. (My preferred solution)
- You can set the value of "$class::attribute" from "outside". I don't
know what this could be good for because I didn't have a use-case for
this yet. (So to keep it simple and clear I wouldn't do that...)
In both cases you could maintain "independant" class attributes.
>> If public class/definition attributes were missing, however, I
>> could well do without them.
>
> You'd retain all of this; you just wouldn't have it to *all*
> variables, just those that are mentioned in the prototype.
I think we have a misunderstanding here. I didn't mean that public class
attributes need to be taken away. I just wanted to say that if you
didn't introduce public attributes into /definitions/ then I wouldn't mind.
Florian
Luke Kanies
>
> Hi Luke,
>
>> In other words, I think the low-cost approach (fixing current stuff)
>> is not the right idea here, and a higher-cost approach (where cost is
>> mostly design) is the right solution.
>
> That's ok for me.
>
>> This would obviously be straightforward, at this point, in that one
>> could just use the existing definition syntax for adding parameters,
>> although it would further confuse the distinction between them.
>
> That's not a problem for me as I don't see a profound difference
> anyway. ;-)
>
>> I'm amenable to this change as a short-term mechanism for adding
>> attributes to classes, but I would tend to require a change with it:
>> The '$class::attribute' syntax would only be able to see the
>> attributes mentioned in the prototype, rather than any internal
>> variables.
>
> What do you mean by prototype? The class definition? (As opposed to a
> class instance?) I am used to the distinction "prototype vs.
> singleton",
> that's why I might be confused by your use of the term "prototype"
> here.
Sorry, I thought the term 'prototype' was often used for things like
'function prototype', as in, the thing that defines how the function
is used.
Yes, I meant, in your parlance, the class definition.
>
> If you mean the class definition by "prototype" then I didn't get your
> point. I still see various possibilities to maintain other attributes
> than those passed into a class as "parameters":
> - You can use "$class::attribute" with constant values, even derived
> or
> somehow calculated values are ok as long as they are constant at
> runtime. This means the value can be declared in the class definition
> and is otherwise read-only. (My preferred solution)
> - You can set the value of "$class::attribute" from "outside". I don't
> know what this could be good for because I didn't have a use-case for
> this yet. (So to keep it simple and clear I wouldn't do that...)
>
> In both cases you could maintain "independant" class attributes.
If you allow external attributes for classes (that is, attributes set
like those in definitions) you now have two sets of attributes: The
external attributes, which are obviously publicly known, and internal
variables, used for who-knows-what.
What I'm saying is that I'd like to once again hide the latter set of
attributes, if we get a different set of public attributes.
This way the developer has some control over whether every little
variable in a class is exposed.
Either that, or add a keyword that specifies that a given variable is
exposed publicly.
>
>>> If public class/definition attributes were missing, however, I
>>> could well do without them.
>>
>> You'd retain all of this; you just wouldn't have it to *all*
>> variables, just those that are mentioned in the prototype.
>
> I think we have a misunderstanding here. I didn't mean that public
> class
> attributes need to be taken away. I just wanted to say that if you
> didn't introduce public attributes into /definitions/ then I
> wouldn't mind.
I know you didn't mean that, but I'd want that if we added other
public attributes.
--
He is indebted to his memory for his jests and to his imagination for
his facts. --Richard Brinsley Sheridan
Florian Grandel
oups, I saw now that whenever you reply to a mail with the previous
subject, then the discussion subject changes. That was not my intent...
I'll change it back to your new subject with this post.
You didn't like the term "resource-bundle". I think "class" is a good
candidate for future naming of "resource-bundles". Just want to avoid
"class" for now to avoid ambiguity.
> Note that
> I'm entirely self-trained, so a lot of what you're talking about goes
> right over my head.
I am really sorry about that!! I am also entirely self-trained. I think
I just have a strong OO and design pattern background while you may have
more an expert sysadmin background. I need these abstract concepts more
for myself as to "get to grips" with complexity. Our communication gaps
will go away as soon as we communicate with examples about the concepts
we are talking about.
To answer some of your questions I'll give the following example in
"resource-bundle" syntax:
# Define everything that all nodes should have in common.
#
# We inherit from a built-in resource-bundle called "node"
#
# We use inheritance here but that's not necessary, we could
# also use a "mix-in" bundle that is simply included or
# "required" by other bundles.
bundle base-node inherits node {
sudo {...}
syslog-ng {...}
...
}
# Define a resource-bundle that represents a
# web-server node
bundle super-duper-webserver-node inherits base-node {
# Let's include some other bundles specific to this
# node type.
#
# self::ip is a read-only automatic attribute of
# the built-in resource-bundle "node" that we inherit
# (see explanation below), replaces facter
#
# You can see that we don't have any "name" attribute.
# The name is implicitly set to "$name". Therefore
# apache{"$self::name": ip => $self::ip, par2 => ...} would
# be exactly the same as:
apache {ip => $self::ip, par2 => ...}
}
# Now let's instantiate our nodes
super-duper-webserver-node {
"myhost1": ;
"myhost2": ;
...
}
>> I bet I could reformulate
>> everything I can do in puppet today with a simplistic language like
>> that.
>
> I can't disagree with that, but my question would be, would you have a
> "better" language if you did this? And it must be said, "better" here
> is defined as "better for its audience".
You see in my example that "node" is no longer something special. It
probably is a built-in "resource-bundle" like "package", "file" or
whatever that provides read-only instance attributes rather than using
global facter variables that miraculously appear from "nowhere". (This
shows the point about read-only attributes that I made in my response to
the other thread.) Internally facter may provide such attributes, but
does the user have to be concerned with such implementation detail???
The only difference is that now "classes", "definitions", "types" and
"nodes" can all be inherited, parameterized, expanded and whatever, all
the same without any "artificial" distinctions that have to be learned.
I think that this one syntax for all is really much easier to understand
and learn for everybody. You hide away a lot of complexity.
I personally don't believe that a sysadmin "intuitively" thinks about
classes and definitions. To the contrary: The fact that you have to
dedicate a own chapter in the documentation and answer many IRC
questions about when to use classes and when to use definitions seems to
show that the distinction is not really intuitive. I personally think
it's more intuitive to do away with it. Maybe you should start some kind
of poll for first time users?
I think its obvious however that the "bundle-syntax" is as readable as
the current language but at the same time easier to refactor and expand.
> What if you have multiple classes that want to declare that they
> require a given class?
> classes can be cross-
> cutting, in that many classes might care that a given class is
> instantiated, so it's important that they can declare this.
You are right in that I wouldn't allow include statements any more. IMO
the include statement can always be replaced by inheritance, inclusion
of other bundles or "require/before"-relationships.
You can use a "require" relationship at as many points as you like.
Remember that we now can use "require/before" for everything and not
only for "definitions" or "types". I think "require/before" for class
instances makes the "include"-syntax completely redundant.
If you get something as "above/below" (see
http://groups.google.com/group/puppet-dev/browse_thread/thread/1cc72c2de9d1ced7#)
this would further improve the ability to describe real bundle
relationships. I am strongly in favour of such a relationship type.
Give me some examples of use-cases where you think you cannot do without
multiple includes or where the "resource-bundle" syntax would be
counter-intuitive and I'll see whether I have an error in my logic or
whether I can formulate your use-cases in a satisfactory way in "bundle
syntax".
> nodes have
> automatic entry points (i.e., the parser accepts a request from a host
> and uses the host's name to look up a node instance). Neither classes
> nor definitions have automatic entry.
The built-in resource-bundle "node" continues to be an automatic entry
point for puppetd. But IMO that's an unnecessary implementation detail
that can be hidden away from the user.
> To me, the term 'class' is a convenient way to describe the
> configuration associated with a given intent. "Web server" means one
> list of resources, "dns client" means another, etc.
There are many places where the difference between an "intent" and a
"resource" is blurred:
- Is a webserver a resource or an intent?
- Is a subversion server a resource or an intent?
- If I have to add a second database instance to my database server, is
my "database-server class" now suddenly mutating from an intent towards
a resource?
How you respond to these questions is a matter of standpoint. When you
think in terms of an application server, a database instance may be just
a resource. But if you think in terms of a database server it maybe
/the/ "intent".
If you use bundle syntax this blurred distinction simply doesn't matter.
You can show intent where you need to, but you don't have to bother if
the distinction becomes a little blurred or needs to be refactored.
What matters is that "bundle syntax" helps me to express some abstract
concept (e.g. a load-balanced cluster of firewall-protected
web-servers). But it also helps me to iteratively "divide and conquer"
abstract concepts without any conceptual break until I arrive at the OS
level (files, packages, services, etc.). And I can also cluster concepts
into higher-level concepts at any time even if that means that I have to
transform a singleton into a prototype or vice versa. (Currently
refactoring a class into a definition or vice versa is /much/ more work
than that.)
IMO the "bundle-grammar" is simpler while still being able to express
intent. It doesn't reduce expressivity or readability but adds
encapsulation, maintainability and flexibility.
> Puppet can't really
> talk about anything that's cross-host, and it's a significant
> problem. I think of this as the 'me/it' problem -- Puppet can only
> talk about 'me' (what services should I be running?) rather than
> 'it' (what services should this host or that host be running?). Been
> a while since I thought about this aspect, but it's definitely there
> and definitely a problem.
>
> How does your 'class of nodes' idea change this?
>
>> But what if you
>> wanted to cluster nodes one day as one "thing" to avoid the current
>> duplication of node-specific code in puppet? Oups! You'd have to
>> introduce ... named puppet class instances.
>
> Can you elaborate on this?
Now comes my point about "clusters" or "class of nodes". In the
generalized "bundle-syntax" you can say:
bundle full-system-cluster($needs_firewall = true) {
if $self::needs_firewall {
firewall-and-load-balancer-node { "${self::name}-fw": }
} else {
load-balancer-node { "${self::name}-lb": }
}
webserver-node {
"${self::name}-web1": ;
"${self::name}-web2": ;
}
application-server-node {
"${self::name}-app1":
"${self::name}-app2":
}
db-server-node { "${self::name}-db": }
}
And now you can instantiate whole clusters with the same infrastructure
of containing nodes:
full-system-cluster {
"production-cluster": ;
"staging-cluster": ;
"development-cluster": needs_firewall => false;
}
You'd effectively configure 18 hosts (three groups of six) in just three
lines. Isn't that super readable and intuitive, and clearly showing
intent? Try to express the same thing with disparate node declarations
(in a .pp file or as external nodes) and you'll have duplicated code
and/or data en masse.
That's what I meant with "node instances". Sure, puppetd will still
"enter" at the node level. But why not going down /and/ up in the
hierarchy to find out about how to configure our specific node? You
could still derive one single catalog per node.
>> IMO a resource can only be a singleton in the context of a bundle.
>> Not more not less.
>
> I agree, and that bundle is currently named a 'catalog' in Puppet.
> 0.25 finally makes this Catalog class the arbiter of singleton-hood.
> a given resource must be unique within a
> host's catalog, but not usually unique within a network.
I think that my cluster example shows that it may make sense to have
something "unique" at a higher level than node/catalog level to avoid
code duplication in node definitions.
Sure anything that is unique above node level will be unique at node
level as well. But not everything unique at node level must
automatically be "duplicate" at a higher level! My cluster example shows
that "singleton-hood" on a higher than node level can make sense!
> the lack of meaningful distinction [between node and class] is one
> thing that led me to want to push nodes outside of Puppet.
You can continue to do so. I believe that the more generic "bundle
syntax" will help you to simplify and extend your "external definition
provider" implementations. You could provide external parameters not
only on type and node level but also in between or above wherever you
like in the bundle hierarchy. All instantiation can be done externally.
You could keep only bundle definitions in your *.pp files and get all
the rest from an external source.
> When a client produces a log message, that message
> requieres context to be useful. If it fails to install a package, we
> need to know why (again, the intent) that package was needed, so we
> can then determine what services would be affected. We can also
> relate that failure back to code changes, maybe, so it's easier to
> resolve the problem network-wide.
Hm. I don't get your point. Why can you not trace back an error in the
log with "bundles"? The bundles are completely hierarchical. So I don't
see problems here. I think that the error messages won't change at all.
You get the highest level concepts (intents) at the beginning of the log
message and the lowest level resources (OS resources) at the end of the
log message.
>> My hypothesis is: Any non-procedural language can
>> be considered class- and aspect-oriented at the same time.
>
> I don't really agree that classes and aspects are essentially the
> same. In fact, if an aspect is a cross-cutting concern, then Puppet's
> classes are actually explicitly not aspects -- they're completely
> exclusive. No two classes can overlap at all.
I think you are right with what you are saying about aspects. I'll have
to drop my "aspect=class" idea for declarative languages. You
demonstrated a case (the dns example) where the definition of an aspect
makes a difference in practice. Maybe this could be solved quite easily
with parameters as well. But I could think of other situations now where
it makes sense to dynamically mix resources into bundles based on some
dynamic runtime attribute value (e.g. to make a difference between
virtual servers and hardware servers across all definitions).
Although it was me who introduced the term "aspect" here I'd rather drop
the topic from this thread:
- In "bundle syntax" aspects can be worked around in most cases with
parameters, at least for the time being.
- I think aspects are neither incompatible with "resource-bundles" nor
with current syntax.
- Aspects are a feature completely new to puppet. This thread concerns
the re-formulation of existing features. We therefore should discuss
aspects at another place otherwise this thread will further explode. ;-)
> I don't like the idea of declaring class membership via the same
> syntax for specifying resources, partially because I think most people
> will find that the attributes that are consumed by a class are often
> specified in a different location than the code that specifies class
> membership.
Can you give an example? I don't understand that. IMO separating
parameter and class instances is a design error. This is exactly what
makes code error prone and difficult to maintain as it increases the the
amount of code you have to consider when refactoring classes.
If you have to nest bundles at several levels then I consider it good
practice to re-declare required parameters at every level:
bundle shorewall::rule($ip, $port, ...) {
...
file{...template(...using $self::ip and $self::port...)...}
...
}
bundle apache::vhost($ip, $port, ...) {
...
shorewall::rule { ip => $self::ip, port => $self::port, ... }
# This is a good example for a "local singleton" by the way... :-)
# The rule implicitly inherits the name of the enclosing vhost
# If we later need two rules we simply add an explicit name.
...
}
bundle web-server-node inherits node {
...
# As we inherit from the built-in "node" bundle, we automatically
# get facter variables as instance variables and may use them,
# see $self::ip ...
apache::vhost( "${name}-vhost1": ip => $self::ip, port => 80 }
apache::vhost( "${name}-vhost2": ip => $self::ip, port => 443 }
...
}
and so on.
And one final idea concerning templates:
If you made templates a built-in "resource bundle" rather than a
function it would be very easy to explicitly declare parameters.
You could simply say:
template { "/my/template.erb": var1 => ..., var2 => ... }
All variables not explicitly declared will simply not be present in the
template.
This would remove the last occurrence of undeclared dependencies. I
don't see why you couldn't use ruby's template engine internally anyway.
Florian
Florian Grandel
> What I'm saying is that I'd like to once again hide the latter set of
> attributes, if we get a different set of public attributes.
>
> This way the developer has some control over whether every little
> variable in a class is exposed.
>
> Either that, or add a keyword that specifies that a given variable is
> exposed publicly.
I get your point now. Intuitively I prefer the second variant (a
keyword) as it's more flexible and closer to what other languages offer.
And I somehow like the possibility to define public constants over
hard-coding numbers or whatever into parameters.
I think that only constants may be public. Writeable variables' state is
undefined where there is no such thing as a clear temporal execution
order. How do you solve this now? (I am not using writeable public
variables so no experience about that.)
Florian
David Schmitt
> On Oct 31, 2008, at 10:33 AM, jerico wrote:
>> I don't agree on that. It might be a specially declarative and less
>> procedural/imperative language and is probably not turing complete (no
>> idea about that, I didn't try the proof!) But maybe this is just an
>> argument about wording and therefore not so important.
>
> I agree it's mostly a terminology point, but it's definitely not
> Turing-complete, although Brice might have slipped that in with his
> last series of updates. :)
I beg to differ. Puppet manifests have recursion (define) and choice (if
). In my reading that's enough for turing completeness.
Regards, DavidS
Florian Grandel
> I beg to differ. Puppet manifests have recursion (define) and choice (if
> ). In my reading that's enough for turing completeness.
At least one of us who has done his computer science lessons. Thank you! :-D
Florian
Luke Kanies
Is that really recursion, though? Wouldn't we need to be able to pass
resources to the resources, which we can't currently do?
I'd've thought we'd at least need to be able to have the definitions
interact with the resources in some way.
--
To my embarrassment I was born in bed with a lady.
--Wilson Mizner
David Schmitt
> On Nov 5, 2008, at 1:14 AM, David Schmitt wrote:
>
>> Luke Kanies schrieb:
>>> On Oct 31, 2008, at 10:33 AM, jerico wrote:
>>>> I don't agree on that. It might be a specially declarative and less
>>>> procedural/imperative language and is probably not turing complete
>>>> (no
>>>> idea about that, I didn't try the proof!) But maybe this is just an
>>>> argument about wording and therefore not so important.
>>> I agree it's mostly a terminology point, but it's definitely not
>>> Turing-complete, although Brice might have slipped that in with his
>>> last series of updates. :)
>>
>> I beg to differ. Puppet manifests have recursion (define) and choice
>> (if
>> ). In my reading that's enough for turing completeness.
>
>
> Is that really recursion, though? Wouldn't we need to be able to pass
> resources to the resources, which we can't currently do?
>
> I'd've thought we'd at least need to be able to have the definitions
> interact with the resources in some way.
>
Just for the fun of it, a define that creates $name files in /tmp:
define mul() {
if ($name > 0) {
file { "/tmp/$name": ensure => exists }
$next = add($name, -1)
mul { $next: }
}
}
I think there was a recent patch to get the comparison into the if(),
else this could be eumulated by a function too.
Regards, DavidS
Luke Kanies
I *knew* we were in for it when Brice added better conditionals.
Someone queue up that infinite tape... :)
--
Progress isn't made by early risers. It's made by lazy men trying to
find easier ways to do something. --Robert Heinlein
Brice Figureau
you can simplify this with:
$next = $name - 1
>> mul { $next: }
>> }
>> }
> I *knew* we were in for it when Brice added better conditionals.
> Someone queue up that infinite tape... :)
:-)
--
Brice Figureau
Days of Wonder
http://www.daysofwonder.com
Luke Kanies
You can't write to variables from outside of a class, and you can't
use a class's variables until the class has been evaluated, so all
exposed variables are effectively constants once they're set.
--
When a man tells you that he got rich through hard work, ask
him: 'Whose?' --Don Marquis
Luke Kanies
what I think you're recommending, you correct any as necessary, and
we'll see if that can make the discussion a bit more succinct. I'll
then heavily trim your email and respond to the points that I think
are particularly comment-worthy.
My main conclusion here, though, is that you're talking about a
language that bears not a lot of resemblance to Puppet, in the end,
and if this is something you want, then I recommend starting by
essentially forking Puppet's parser and working from there. Either
that, or trying to turn the current DSL.rb into the language you want,
although obviously it's pure ruby.
I'm not saying you shouldn't do these things, just saying that you're
talking about drastic changes to the core of the language for what I
see as questionable benefit.
On to specifics:
Removal of 'class' and 'node':
Node, class, and define are all merged into a single syntactical
element. This element is effectively equivalent to the existing
'define' element, in that it accepts parameters and is unique only for
a given name. That is, you can always create a new instance of this
element, as long as you give it a different name. When used with no
name, the name is automatically set to the name of the element itself.
Node becomes a builtin resource type:
I'm a bit unclear on this. You mention this as a way to continue
providing an automatic entry point for compiling, and you make
reference to turning the facter variables into node attributes. I
don't understand this well enough to know how it'd even work.
'include' goes away:
One no longer includes classes, and instead classes are specified
using the normal resource-specification syntax. I'm pretty unclear on
how you'd do this for the typical what-we-currently-call-classes, with
no unique name and no parameters. Would it be:
webserver { : }
Or maybe:
webserver { webserver: }
This is actually how Puppet started, and I got rid of it because of
how obviously unobvious they are.
You also say something about relationships being used instead of
'include', so I assume you mean there that specifying a relationship
to a class would automatically evaluate that class.
Introduction of the keyword 'self':
I actually don't know what you're doing here, but you're using it in
all of your examples, so I figured I'd point it out.
Templates become resources:
Move templates from a function to a resource to avoid implicit
consumption of variables. I assume this would involve modifying
properties to know how to retrieve a template's content when
necessary. I.e., you'd need to do something like:
template { "whatever": foo => bar }
file { "/my/file": content => Template[whatever] }
You'd need to do this for every property, otherwise you'd lose the
current ability of using templates for any rvalue.
Okay, now for specific (hopefully short) commentary:
On Nov 4, 2008, at 10:09 PM, Florian Grandel wrote:
>
> #
> # We use inheritance here but that's not necessary, we could
> # also use a "mix-in" bundle that is simply included or
> # "required" by other bundles.
> bundle base-node inherits node {
[...]
>
> }
>
> # Define a resource-bundle that represents a
> # web-server node
> bundle super-duper-webserver-node inherits base-node {
[...]
>
> }
>
> # Now let's instantiate our nodes
> super-duper-webserver-node {
> "myhost1": ;
> "myhost2": ;
> ...
> }
Does this mean that we could have multiple subinstances or whatever
you call them for a given node? That is, we could have webserver
{ mynode: } and dbserver { mynode: }?
Also, where are facts here? You've mentioned making them no longer
global constants, but I don't know where they went.
> You see in my example that "node" is no longer something special. It
> probably is a built-in "resource-bundle" like "package", "file" or
> whatever that provides read-only instance attributes rather than using
> global facter variables that miraculously appear from "nowhere". (This
> shows the point about read-only attributes that I made in my
> response to
> the other thread.) Internally facter may provide such attributes, but
> does the user have to be concerned with such implementation detail???
I don't know if you're giving too much or too little credit to the
Puppet user, but trust me, where those facts come from and how they
become available in the language is not just an implementation
detail. It's a critical part of the framework, and most users end up
needing to understand it very well, and I don't see anything in your
recommendation that's changed that.
>
> The only difference is that now "classes", "definitions", "types" and
> "nodes" can all be inherited, parameterized, expanded and whatever,
> all
> the same without any "artificial" distinctions that have to be
> learned.
> I think that this one syntax for all is really much easier to
> understand
> and learn for everybody. You hide away a lot of complexity.
This seems to be the crux of your argument -- that removing
syntactical variety is simpler and hides complexity -- but I'm not
convinced. Puppet's simplicity largely comes from its lack of options
or complexity -- would you say LISP is simpler or more complex than
Puppet? One could argue either way, but I tend toward thinking its
simplicity makes it potentially more complicated to use. Certainly it
raises the bar for the programmer who would be involved.
>
> I personally don't believe that a sysadmin "intuitively" thinks about
> classes and definitions. To the contrary: The fact that you have to
> dedicate a own chapter in the documentation and answer many IRC
> questions about when to use classes and when to use definitions
> seems to
> show that the distinction is not really intuitive. I personally think
> it's more intuitive to do away with it. Maybe you should start some
> kind
> of poll for first time users?
Having spent the last three months attempting to teach my infant
daughters how to nurse, I'm not a big believer in intuitiveness.
Declarative languages aren't intuitive, modeling resources instead of
files isn't intuitive, asynchronous administration (i.e., commit and
wait) isn't intuitive; yet people have learned all of these and
wouldn't give them up, in most cases.
>
> I think its obvious however that the "bundle-syntax" is as readable as
> the current language but at the same time easier to refactor and
> expand.
I think you are missing a lot of corner cases (or maybe not-so-corner)
that make it less so. In particular, the common case of a "class"
with no name or parameter is particularly jarring syntactically:
webserver { : }
For the record, if you look at the earliest versions of Puppet, the
'include' function was just a shortcut for exactly this syntax.
This gets silly on relationships, too; do I do:
require => Webserver[]
or:
require => Webserver[webserver]
or what?
[...]
> Give me some examples of use-cases where you think you cannot do
> without
> multiple includes or where the "resource-bundle" syntax would be
> counter-intuitive and I'll see whether I have an error in my logic or
> whether I can formulate your use-cases in a satisfactory way in
> "bundle
> syntax".
You haven't made it explicit, but I assume you mean that specifying a
relationship to a class would result in automatic evaluation. That
is, I could say 'require => Webserver[foo]' and it would automatically
evaluate that class? Or would I need to evaluate it elsewhere in
order to specify a relationship?
>
>> nodes have
>> automatic entry points (i.e., the parser accepts a request from a
>> host
>> and uses the host's name to look up a node instance). Neither
>> classes
>> nor definitions have automatic entry.
>
> The built-in resource-bundle "node" continues to be an automatic entry
> point for puppetd. But IMO that's an unnecessary implementation detail
> that can be hidden away from the user.
Again, I don't think this is an implementation detail -- it's a
fundamental aspect of how information enters Puppet, and it's
something users really need to undestand.
>
>> To me, the term 'class' is a convenient way to describe the
>> configuration associated with a given intent. "Web server" means one
>> list of resources, "dns client" means another, etc.
>
> There are many places where the difference between an "intent" and a
> "resource" is blurred:
> - Is a webserver a resource or an intent?
> - Is a subversion server a resource or an intent?
> - If I have to add a second database instance to my database server,
> is
> my "database-server class" now suddenly mutating from an intent
> towards
> a resource?
It seems pretty simple: you must be a database server in order to run
database instances. Classes define a service that you provide, and
resources are some aspect of that service.
[...]
I don't think I understand this at all. Given a node name, how does
it know that it's a part of the system cluster? If it's part of the
cluster, how does it know which services from the cluster it provides?
AFAICT, I could do this exact thing with Puppet's classes right now.
>> the lack of meaningful distinction [between node and class] is one
>> thing that led me to want to push nodes outside of Puppet.
>
> You can continue to do so. I believe that the more generic "bundle
> syntax" will help you to simplify and extend your "external definition
> provider" implementations. You could provide external parameters not
> only on type and node level but also in between or above wherever you
> like in the bundle hierarchy. All instantiation can be done
> externally.
> You could keep only bundle definitions in your *.pp files and get all
> the rest from an external source.
This seems to be pretty far out -- the current interface to external
nodes is very simple, and it sounds like you're talking about a pretty
dramatic enhancement to this external interface. I'd say leave that
discussion as a separate problem entirely.
[...trying to hurry to actually finish this email...]
>> I don't like the idea of declaring class membership via the same
>> syntax for specifying resources, partially because I think most
>> people
>> will find that the attributes that are consumed by a class are often
>> specified in a different location than the code that specifies class
>> membership.
>
> Can you give an example? I don't understand that. IMO separating
> parameter and class instances is a design error. This is exactly what
> makes code error prone and difficult to maintain as it increases the
> the
> amount of code you have to consider when refactoring classes.
I can't come up with much in terms of specific examples right now, but
I'm kinda short on time. We can effectively ignore this discussion,
except that I'm already planning on building a web interface that
would make it easy to configure nodes and their classes; this
interface would want to query a class to see what parameters are
required and then require that those params be filled in on the
external node tool. This is a clear separation, and you'd clearly
want it -- I don't want to have to explicitly import those params or
something for the node.
>
> If you have to nest bundles at several levels then I consider it good
> practice to re-declare required parameters at every level:
>
> bundle shorewall::rule($ip, $port, ...) {
> ...
> file{...template(...using $self::ip and $self::port...)...}
> ...
> }
As an aside, where is this '$self' stuff from? You haven't explicitly
mentioned it, but it seems to be in all of your examples.
[...]
> And one final idea concerning templates:
>
> If you made templates a built-in "resource bundle" rather than a
> function it would be very easy to explicitly declare parameters.
>
> You could simply say:
>
> template { "/my/template.erb": var1 => ..., var2 => ... }
>
> All variables not explicitly declared will simply not be present in
> the
> template.
>
> This would remove the last occurrence of undeclared dependencies. I
> don't see why you couldn't use ruby's template engine internally
> anyway.
As mentioned at the top, this would require teaching the Parameter
class how to understand template instances. I guess I'm not entirely
opposed to this, but it seems a bit unnecessary, at the same time.
--
Censorship, like charity, should begin at home; but, unlike charity, it
should end there. --Clare Booth Luce
Florian Grandel
>> I think that only constants may be public. Writeable variables'
>> state is
>> undefined where there is no such thing as a clear temporal execution
>> order. How do you solve this now? (I am not using writeable public
>> variables so no experience about that.)
>
> You can't write to variables from outside of a class, and you can't
> use a class's variables until the class has been evaluated, so all
> exposed variables are effectively constants once they're set.
That makes sense to me. Thanks for the explanation! :-)
Florian
Florian Grandel
Luke Kanies schrieb:
> These emails are getting really long.
I think that's normal for such a generic and complex topic. And I see
that we both are investing a lot of time.
So please don't feel obliged to respond if you are no longer interested
in the discussion for whatever reason. I'd completely understand that
and I'd not at all be offended!!
> My main conclusion here, though, is that you're talking about a
> language that bears not a lot of resemblance to Puppet, in the end,
> and if this is something you want, then I recommend starting by
> essentially forking Puppet's parser and working from there.
I didn't say that I want all this in practice. As a user I'd be
completely happy with a very small change to the language (=class
parameters).
I think we were discussing: "Where could we go if there weren't
compatibility limitations?" My rationale wasn't one of forking puppet
but of discussing some idea of an "ideal" configuration language. The
practical result I hope for is to avoid proliferation of (in my eyes)
"difficult to learn" language constructs or some "bad" puppet recipes
that I saw in the documentation.
> Node, class, and define are all merged into a single syntactical
> element. This element is effectively equivalent to the existing
> 'define' element, in that it accepts parameters and is unique only for
> a given name. That is, you can always create a new instance of this
> element, as long as you give it a different name. When used with no
> name, the name is automatically set to the name of the element itself.
Everything correct, except: The default instance name is not the name of
the element itself but the name of the parent instance.
> Would it be:
>
> webserver { : }
>
> Or maybe:
>
> webserver { webserver: }
>
> This is actually how Puppet started, and I got rid of it because of
> how obviously unobvious they are.
See the examples in my last post... The syntax would be:
bundle webserver_node inherits node {
webserver { par1=>..., par2=>... }
database { par1=>..., par2=>... }
}
Or if you have no parameters then:
bundle webserver_node inherits node {
webserver
database
}
And then in site.pp:
webserver_node { "www.example.com": }
In this example the implicit names of the webserver and database
instances are "web.example.com".
I don't see anything unobvious here.
> You haven't made it explicit, but I assume you mean that specifying a
> relationship to a class would result in automatic evaluation. That
> is, I could say 'require => Webserver[foo]' and it would automatically
> evaluate that class? Or would I need to evaluate it elsewhere in
> order to specify a relationship?
1) Remember that there is no such thing as today's "class" in the
proposed "bundle syntax".
2) You cannot reference anything that has not been instantiated.
3) For better readability and maintainability there shouldn't be
anything being implicitly instantiated.
The following would raise an error:
bundle mynode inherits node {
webserver { require => Firewall }
}
This should work:
bundle mynode inherits node {
firewall
webserver { require => Firewall }
# This is implicitly the same as:
# firewall { "$name": }
# webserver { "$name": require => Firewall["$name"] } !
}
When you think that there is something you cannot do without an include
then please give me a specific example.
> Introduction of the keyword 'self':
>
> I actually don't know what you're doing here, but you're using it in
> all of your examples, so I figured I'd point it out.
It's just a common way to provide separate namespaces for instance
variables and local variables. I used a syntax similar to php, C++ or
java. In Ruby "@..." does the same.
Currently puppet does not distinguish between local variables and
instance variables.
Doing something like
define(...) {
...
$require = 5
...
}
raises an error. This is bad as it may cause maintenance trouble when
introducing new parameters to a definition in puppet that may already
exist as local variables.
> Templates become resources:
>
> Move templates from a function to a resource to avoid implicit
> consumption of variables. I assume this would involve modifying
> properties to know how to retrieve a template's content when
> necessary. I.e., you'd need to do something like:
>
> template { "whatever": foo => bar }
> file { "/my/file": content => Template[whatever] }
>
> You'd need to do this for every property, otherwise you'd lose the
> current ability of using templates for any rvalue.
right.
> Does this mean that we could have multiple subinstances or whatever
> you call them for a given node? That is, we could have webserver
> { mynode: } and dbserver { mynode: }?
No, you cannot have two "subinstances" for the same physical node.
If webserver and dbserver both inherit from node then an instance for a
given physical host can exist for one or the other but not both. If
you'd try to do something like this then puppet should throw an error.
If you want a single node to take over the roles of webserver and
dbserver at the same time then you'd have to define two resource bundles
(webserver, dbserver) that do not inherit from node:
# Resource bundle 1 (aka today's puppet class/definition)
bundle webserver($fqdn) {
.. some resources ..
}
# Resource bundle 2 (aka today's puppet class/definition)
bundle dbserver {
.. some resources ..
}
# Node definition
bundle my_combined_node_type inherits node {
webserver{ fqdn => $self::fqdn } # $self::fqdn is a facter variable!!
dbserver
}
# Node instance (in site.pp)
combined_db_web_node_type { "mynode": }
> Also, where are facts here? You've mentioned making them no longer
> global constants, but I don't know where they went.
Facts are automatically defined as instance variables of any "resource
bundle" that inherits from "node". This is the only way in which facts
are available (see the example above).
Facter variables have to be explicitly passed down from the node type to
any bundle that needs it (see the above example again). This avoids
global variables and improves encapsulation.
> This seems to be the crux of your argument -- that removing
> syntactical variety is simpler and hides complexity -- but I'm not
> convinced. Puppet's simplicity largely comes from its lack of options
> or complexity -- would you say LISP is simpler or more complex than
> Puppet?
The way you state it, it is not correct. Replace "simplicity" by "easy
to learn" and "maintainable" and you are much closer to what I am after.
IMO a language is "easy to learn" if it uses concepts that are already
known to the majority of users when they start to learn it.
I don't want to reduce syntactical variety. I want to...
1) reduce the number of new concepts that one has to learn to learn puppet.
2) reduce duplicate code and global variables to improve maintainability
and extensibility.
* easy to learn:
The distinction definition/class is uncommon outside puppet and has to
be learned by new users. IMO the usage of the term "class" for something
different than an OO class is also "bad" as it puts new users with OO
background (the majority I guess) on the wrong track. A puppet class is
something very different from an OO class while a puppet definition is
something close to an OO class. All this may be obvious to you as you
have lived with these concepts for years. But IMO it is not at all
obvious to new users.
In the syntax I propose it is no longer a problem but a real learning
aid to think of a "bundle" as an OO "class": "Resource bundles" are
conceptually very close to an OO class.
* maintainable:
The current puppet language provides no way to avoid usage of global
variables within classes and templates and produces code duplication on
node level and above (no clusters, no way to define similar nodes by way
of a "node bundle", see the cluster example below).
The language I propose has no problems with that.
* LISP:
I think we agree that LISP is more difficult to learn for the average
sysadmin than puppet. This comes from the sheer number of features
available in LISP and usage of concepts that are probably unknown to the
average sysadmin. LISP may however be easy to learn for a mathematician
(don't know).
If you think that the proposed language is difficult to learn for an
average sysadmin then please make an example. This would make it easier
for me to understand what exactly you don't like.
> One could argue either way, but I tend toward thinking its
> simplicity makes it potentially more complicated to use. Certainly it
> raises the bar for the programmer who would be involved.
Not agreed (see my definition of "simplicity" or "intuitiveness" above).
What is easy to learn depends exclusively on the prior knowledge of the
target audience and not on any inherent property of the language.
> Having spent the last three months attempting to teach my infant
> daughters how to nurse, I'm not a big believer in intuitiveness.
> Declarative languages aren't intuitive, modeling resources instead of
> files isn't intuitive, asynchronous administration (i.e., commit and
> wait) isn't intuitive; yet people have learned all of these and
> wouldn't give them up, in most cases.
You seem to think that only what we can use directly after birth is
really "intuitive". That's a quite narrow definition of "intuitive". But
yes: If you define it like that then my proposed syntax is completely
unintuitive. ;-) You certainly understand that I didn't claim my
proposed language to be understood by a new-born.
What I wanted to say is that the proposed syntax (at least more than
puppet language today) is based on concepts already known to its target
audience (i.e. based on basic OO principles, no mistakable re-definition
of terms already used in another context).
> I think you are missing a lot of corner cases (or maybe not-so-corner)
> that make it less so. In particular, the common case of a "class"
> with no name or parameter is particularly jarring syntactically:
>
> webserver { : }
See my examples in earlier posts and at the beginning of this message. I
think that's a case I've thought about from the beginning.
> This gets silly on relationships, too; do I do:
>
> require => Webserver[]
>
> or:
>
> require => Webserver[webserver]
Neither one nor the other, see the example at the beginning of this message.
Btw: Giving an instance name for what is a class today (=node-level
singleton) is not at all silly when you define clusters of nodes (see
cluster example below).
Send me a reasonably complex pp-file (with lots of "corner cases") and
I'll try to restate them in "bundle syntax". Maybe this helps you to
better judge the power of the proposed language or helps me to
understand what I am missing.
> Again, I don't think this is an implementation detail -- it's a
> fundamental aspect of how information enters Puppet, and it's
> something users really need to undestand.
If you don't agree then please give a specific example again what
exactly you cannot do if you never heard about entry points or the
origin of host information.
Otherwise it is very difficult for me to understand /why/ I need to
understand such information.
>> bundle full-system-cluster($needs_firewall = true) {
>> if $self::needs_firewall {
>> firewall-and-load-balancer-node { "${self::name}-fw": }
>> } else {
>> load-balancer-node { "${self::name}-lb": }
>> }
>>
>> webserver-node {
>> "${self::name}-web1": ;
>> "${self::name}-web2": ;
>> }
>>
>> application-server-node {
>> "${self::name}-app1":
>> "${self::name}-app2":
>> }
>>
>> db-server-node { "${self::name}-db": }
>> }
>>
>> And now you can instantiate whole clusters with the same
>> infrastructure
>> of containing nodes:
>>
>> full-system-cluster {
>> "production-cluster": ;
>> "staging-cluster": ;
>> "development-cluster": needs_firewall => false;
>> }
>>
> I don't think I understand this at all. Given a node name, how does
> it know that it's a part of the system cluster?
The algorithm to construct the configuration catalog for a node is:
- You resolve all variables.
- You produce a full parse tree of the configuration. To do so you'll
have to identify the top level (=root) elements. These are elements that
- You enter with a given node name, say "production-cluster-web1"
- You look for a node instance with this name. At this point you must
make sure that there is only one node instance with this name otherwise
you raise an error. In the cluster example I gave in my post we have
exactly one node instance named "production-cluster-web1".
- Now you can recursively compile all configuration instantiated by the
node into your configuration catalog. This is what you do today.
- Then you look whether the node has a parent instance. If you have a
parse tree of the complete configuration this should be as simple as
looking for child instances. In our case we have a parent instance
called Full-system-cluster["production-cluster"].
- Then you take over all resources defined for the parents into your
configuration catalogue. In our example we do not have any additional
resources in the parent element but there might be some cluster-wide
files or whatever that cannot be attributed to bundles at lower levels.
- You continue to recurse until you reach the root-level element.
That's all.
> If it's part of the
> cluster, how does it know which services from the cluster it provides?
If you follow the example then you'll see that a cluster usually
clusters nodes and not services. If you have some resources defined in a
cluster then they'll be present on all nodes that are defined by this
cluster. Services as you understand them (aka today's classes) are
always defined at node level or below.
> AFAICT, I could do this exact thing with Puppet's classes right now.
Can you? Maybe I am really missing some important puppet concept!
The only equivalent in current puppet syntax I can come up with is:
node "production-cluster-fw" {
class firewall-and-load-balancer-node
}
node "production-cluster-web1" {
class web-server
}
node "production-cluster-web2" {
class web-server
}
node "production-cluster-app1" {
class application-server
}
node "production-cluster-app2" {
class application-server
}
node "production-cluster-db" {
class db-server
}
node "staging-cluster-fw" {
class firewall-and-load-balancer-node
}
node "staging-cluster-web1" {
class web-server
}
node "staging-cluster-web2" {
class web-server
}
node "staging-cluster-app1" {
class application-server
}
node "staging-cluster-app2" {
class application-server
}
node "staging-cluster-db" {
class db-server
}
node "development-cluster-fw" {
class load-balancer-node
}
node "development-cluster-web1" {
class web-server
}
node "development-cluster-web2" {
class web-server
}
node "development-cluster-app1" {
class application-server
}
node "development-cluster-app2" {
class application-server
}
node "development-cluster-db" {
class db-server
}
You can also put all this into external nodes. But AFAIK you still have
to define a "flat" list of nodes that cannot express any hierarchical
relationship or "clusters" of nodes.
How would /you/ do this?
> the current interface to external
> nodes is very simple, and it sounds like you're talking about a pretty
> dramatic enhancement to this external interface. I'd say leave that
> discussion as a separate problem entirely.
ok.
> I can't come up with much in terms of specific examples right now, but
> I'm kinda short on time.
ok.
>> Templates as type:
>
> As mentioned at the top, this would require teaching the Parameter
> class how to understand template instances.
Maybe. I don't know the internals of puppet code. It's just a
possibility and I don't deny that it may be an expensive one.
Florian
Sven Mueller
> Also, to be clear, I *wholly* encourage this kind of discussion and
> recommendation. I've never claimed I have the ultimate vision of
> Puppet's language, and I'm always looking for recommendations to make
> it better. Of course, I will always defend the existing syntax, and I
> will always fight to keep the basic vision I have intact.
Well, one thing I came across that really hinders efficient
configuration management with puppet at my current employer is the
structure of servers and services we have is the write-once variables
thing in puppet. I know this is not directly tied with the
definition/class differentiation, but at least one issue raised in the
thread reminded me of this problem: The use of (not always completely)
global variables in recipes.
To be a bit more specific, we came across this limit three times
already, while still doing the basic system config (not yet any services
besides Kerberos/LDAP), which boil down to puppet not being able to do
this as a programmer would expect:
node base {
$dom="example.local"
include hostsetup
}
node public_server inherits base {
$dom="example.org"
}
node internal_server inherits base {
include ldap
}
....
node mx inherits public_server {
include mailsetup
}
node special_node inherits public_server {
$dom="example.com"
}
(all but the last being templates for use by actual host definitions,
where "class hostsetup" somehow uses $dom)
In other words, overriding some variables in a derived node (or class
for that matter) where the types/templates used in the ancestor actually
access the most specific variable declaration (youngest generation
derived node/class that declares the variable). In the example above,
for node mx, the class nodesetup should see $dom being "example.org"
while for node "special_node", it should see "example.com".
I know there are workarounds for the write-once problem, but they all
add complexity to the manifests.
Actually, what would be even better would be to have multiple
inheritance (but the workaround for this is much smaller in our case),
so that it would be easy to set up templates for
mx[12].dc[12].example.org
db[12].dc[12].example.org
web[12].dc[12].example.org
(i.e. having two (or more) different subdomains, with both having
multiple instances of each server type. Currently, we define the server
types in classes while defining the domain structure in initial node
templates, then combining them into node templates for each
subdomain/server type combination, which are then inherited by the
individual nodes.
Any suggestion for a more elegant solution would be appreciated.
regard,
Sven
Luke Kanies
This is a common problem I haven't been able to find a good solution
for. It's not so much related to the write-once aspect of variables
as it is related to the fact that classes are evaluated immediately,
so the subnode's overriding of a variable has no affect.
>
> In other words, overriding some variables in a derived node (or class
> for that matter) where the types/templates used in the ancestor
> actually
> access the most specific variable declaration (youngest generation
> derived node/class that declares the variable). In the example above,
> for node mx, the class nodesetup should see $dom being "example.org"
> while for node "special_node", it should see "example.com".
>
> I know there are workarounds for the write-once problem, but they all
> add complexity to the manifests.
>
> Actually, what would be even better would be to have multiple
> inheritance (but the workaround for this is much smaller in our case),
> so that it would be easy to set up templates for
>
> mx[12].dc[12].example.org
> db[12].dc[12].example.org
> web[12].dc[12].example.org
> (i.e. having two (or more) different subdomains, with both having
> multiple instances of each server type. Currently, we define the
> server
> types in classes while defining the domain structure in initial node
> templates, then combining them into node templates for each
> subdomain/server type combination, which are then inherited by the
> individual nodes.
>
> Any suggestion for a more elegant solution would be appreciated.
An external node solution would work much better.
--
The real art of conversation is not only to say the right thing at the
right place but to leave unsaid the wrong thing at the tempting
moment. -- Dorothy Nevill
Ohad Levy
Using External nodes parameters solved similar problem for us.
Cheers,
Ohad
Sven Mueller
[overriding variables in derived nodes (and classes?)]
>
> This is a common problem I haven't been able to find a good solution
> for. It's not so much related to the write-once aspect of variables
> as it is related to the fact that classes are evaluated immediately,
> so the subnode's overriding of a variable has no affect.
Hmm, is there a specific reason why classes are evaluated immediately?
IMHO evaluation should start with global entries, followed by the
specific node (following down inheritance), but I'm not sure at all how
this could be achieved with (current) puppet architecture. Actually I
know this won't be easy at all, but it would make puppet so much easier
to use for us.
(Actually, if you had an idea how to solve this, I'm pretty (ie. 75%)
sure I could arrange for you to get paid to implement it)
>> Any suggestion for a more elegant solution would be appreciated.
>
> An external node solution would work much better.
So you mean an external program providing the "host" entries in YAML
format? Hmm, might be a solution.... Will think about it.
Regards,
Sven
Luke Kanies
>
> Luke Kanies schrieb:
> [overriding variables in derived nodes (and classes?)]
>>
>> This is a common problem I haven't been able to find a good solution
>> for. It's not so much related to the write-once aspect of variables
>> as it is related to the fact that classes are evaluated immediately,
>> so the subnode's overriding of a variable has no affect.
>
> Hmm, is there a specific reason why classes are evaluated immediately?
> IMHO evaluation should start with global entries, followed by the
> specific node (following down inheritance), but I'm not sure at all
> how
> this could be achieved with (current) puppet architecture. Actually I
> know this won't be easy at all, but it would make puppet so much
> easier
> to use for us.
> (Actually, if you had an idea how to solve this, I'm pretty (ie. 75%)
> sure I could arrange for you to get paid to implement it)
I have some ideas, but I'm not sure it's possible without either
changing backward compatibility or largely redefining what nodes mean
-- they're currently very similar to classes, and we'd need to
differentiate them considerably, probably by limiting the statements
they allow (no resources) and changing how 'include' works to make it
late-binding somehow.
>
>>> Any suggestion for a more elegant solution would be appreciated.
>>
>> An external node solution would work much better.
>
> So you mean an external program providing the "host" entries in YAML
> format? Hmm, might be a solution.... Will think about it.
If, rather than fixing the node type, your company might be willing to
sponsor a straightforward external nodes app, that might be a better
place to start. I'd like to get one going, and I'll never be able to
deprecate our node syntax until there's a good replacement.
--
True Terror is to wake up one morning and discover that your high
school class is running the country. -- Kurt Vonnegut
Deepak Giridharagopal
"inverting" the inheritance hierarchy of my nodes so that instead of
inheriting, they just duplicate the same includes with a different set of
"global" variables preceding them. It works, but it's clearly suboptimal.
On Mon, Nov 17, 2008 at 12:48 PM, Luke Kanies <lu...@madstop.com> wrote:
>> Any suggestion for a more elegant solution would be appreciated.
>
> An external node solution would work much better.
IIRC, the external nodes feature doesn't support the same capabilities. For
example, I don't believe an external nodes script can use "define". So if you
have defines with attributes that vary from node to node, I don't know how
you'd do it with external nodes in a way that wasn't more trouble than it's
worth.
An external node system that supported all the constructs the puppet language
supports would be great. I'd also like a pony. :) But in the meantime, even
something as basic as allowing external scripts to return puppet code that just
gets parsed and 'eval'-ed, though admittedly ghetto, would make my life a lot
easier.
deepak
Luke Kanies
>
> For the record, I run into this problem all the time. I ended up
> effectively
> "inverting" the inheritance hierarchy of my nodes so that instead of
> inheriting, they just duplicate the same includes with a different
> set of
> "global" variables preceding them. It works, but it's clearly
> suboptimal.
>
> On Mon, Nov 17, 2008 at 12:48 PM, Luke Kanies <lu...@madstop.com>
> wrote:
>>> Any suggestion for a more elegant solution would be appreciated.
>>
>> An external node solution would work much better.
>
> IIRC, the external nodes feature doesn't support the same
> capabilities. For
> example, I don't believe an external nodes script can use "define".
> So if you
> have defines with attributes that vary from node to node, I don't
> know how
> you'd do it with external nodes in a way that wasn't more trouble
> than it's
> worth.
But don't these attributes derive somehow from the attributes in the
external nodes? That is, wouldn't you generally consider these to be
node attributes, rather than than attributes of the resources being
configured, if they differ per node?
>
> An external node system that supported all the constructs the puppet
> language
> supports would be great. I'd also like a pony. :) But in the
> meantime, even
> something as basic as allowing external scripts to return puppet
> code that just
> gets parsed and 'eval'-ed, though admittedly ghetto, would make my
> life a lot
> easier.
What are you hoping to do with this? I'm not entirely opposed, but
I'd prefer not to, if possible. If I could understand the need, maybe
I could solve it without going all the way there.
--
Sabbagh's Second Law:
The biggest problem with communication is the illusion that it
has occurred.
Luke Kanies
wherever possible.
On Nov 10, 2008, at 2:35 PM, Florian Grandel wrote:
>
>> My main conclusion here, though, is that you're talking about a
>> language that bears not a lot of resemblance to Puppet, in the end,
>> and if this is something you want, then I recommend starting by
>> essentially forking Puppet's parser and working from there.
>
> I didn't say that I want all this in practice. As a user I'd be
> completely happy with a very small change to the language (=class
> parameters).
I'm amenable to class parameters, although I don't know how they would
work, as I think classes shouldn't have names. Unless we did
something weird like:
class { webserver: ipaddress => "..." }
So I'm ok with your basic request, but I don't see it as a complete
syntactical proposal yet. Could you produce patches that provided
what you want?
>
> I think we were discussing: "Where could we go if there weren't
> compatibility limitations?" My rationale wasn't one of forking puppet
> but of discussing some idea of an "ideal" configuration language. The
> practical result I hope for is to avoid proliferation of (in my eyes)
> "difficult to learn" language constructs or some "bad" puppet recipes
> that I saw in the documentation.
Ok. FTR, not much I can do about "bad" code.
>> Would it be:
>>
>> webserver { : }
>>
>> Or maybe:
>>
>> webserver { webserver: }
>>
>> This is actually how Puppet started, and I got rid of it because of
>> how obviously unobvious they are.
>
> See the examples in my last post... The syntax would be:
>
> bundle webserver_node inherits node {
> webserver { par1=>..., par2=>... }
> database { par1=>..., par2=>... }
> }
>
> Or if you have no parameters then:
>
> bundle webserver_node inherits node {
> webserver
> database
> }
You've syntactical confusion here, though -- Puppet's current parser
would assume these are functions, not classes. You could try to
autodetect the difference, but I think that would be much worse.
> bundle mynode inherits node {
> firewall
> webserver { require => Firewall }
>
> # This is implicitly the same as:
> # firewall { "$name": }
> # webserver { "$name": require => Firewall["$name"] } !
> }
>
> When you think that there is something you cannot do without an
> include
> then please give me a specific example.
Your 'firewall' statement just can't work without getting rid of
functions, and probably wouldn't work even if we did. Therefore we'd
need to use 'include' or something similar to evaluate the function.
>
>> Introduction of the keyword 'self':
>>
>> I actually don't know what you're doing here, but you're using it in
>> all of your examples, so I figured I'd point it out.
>
> It's just a common way to provide separate namespaces for instance
> variables and local variables. I used a syntax similar to php, C++ or
> java. In Ruby "@..." does the same.
>
> Currently puppet does not distinguish between local variables and
> instance variables.
So you're also proposing adding instance variables to Puppet.
>
> Doing something like
>
> define(...) {
> ...
> $require = 5
> ...
> }
>
> raises an error. This is bad as it may cause maintenance trouble when
> introducing new parameters to a definition in puppet that may already
> exist as local variables.
This certainly does not raise an error:
notify { "Yay": }
define foo($bar) {
$require = Notify["Yay"]
notify { "$name $bar": }
}
foo { a: bar => b }
=> notice: Yay
notice: //Notify[Yay]/message: defined 'message' as 'Yay'
notice: a b
notice: //Foo[a]/Notify[a b]/message: defined 'message' as 'a b'
>> Does this mean that we could have multiple subinstances or whatever
>> you call them for a given node? That is, we could have webserver
>> { mynode: } and dbserver { mynode: }?
>
> No, you cannot have two "subinstances" for the same physical node.
>
> If webserver and dbserver both inherit from node then an instance
> for a
> given physical host can exist for one or the other but not both. If
> you'd try to do something like this then puppet should throw an error.
In other words, because the facts are only available in the node, you
can only ever have one class on the system that has access to the
facts. I don't think that will work.
> Facter variables have to be explicitly passed down from the node
> type to
> any bundle that needs it (see the above example again). This avoids
> global variables and improves encapsulation.
They're still as global as the current facts, they just start one
scope further down the tree. So slightly less global, I guess, but
people would just build their configs so they were effectively global.
>
>> This seems to be the crux of your argument -- that removing
>> syntactical variety is simpler and hides complexity -- but I'm not
>> convinced. Puppet's simplicity largely comes from its lack of
>> options
>> or complexity -- would you say LISP is simpler or more complex than
>> Puppet?
>
> The way you state it, it is not correct. Replace "simplicity" by "easy
> to learn" and "maintainable" and you are much closer to what I am
> after.
> IMO a language is "easy to learn" if it uses concepts that are already
> known to the majority of users when they start to learn it.
>
> I don't want to reduce syntactical variety. I want to...
> 1) reduce the number of new concepts that one has to learn to learn
> puppet.
> 2) reduce duplicate code and global variables to improve
> maintainability
> and extensibility.
I'm willing to give the benefit of the doubt, but I don't agree with
your conclusions about simplicity. Or rather, I don't think you've
proposed a complete, workable system. If you could come up with code
that provided this functionality, I'd be much more convinced.
>> I think you are missing a lot of corner cases (or maybe not-so-
>> corner)
>> that make it less so. In particular, the common case of a "class"
>> with no name or parameter is particularly jarring syntactically:
>>
>> webserver { : }
>
> See my examples in earlier posts and at the beginning of this
> message. I
> think that's a case I've thought about from the beginning.
Again, the syntax you've proposed conflicts with an existing syntax,
that for functions.
>
>> This gets silly on relationships, too; do I do:
>>
>> require => Webserver[]
>>
>> or:
>>
>> require => Webserver[webserver]
>
> Neither one nor the other, see the example at the beginning of this
> message.
This was 'require => Webserver'? I guess that's ok.
At this point, I think it's a question of code. I know that's a big
ask, but I think you can't see the complications from your proposals
until you go to implement them.
--
A little government and a little luck are necessary in life, but only a
fool trusts either of them. -- P. J. O'Rourke
Florian Grandel
> Could you produce patches that provided
> what you want?
No, unfortunately not. I am currently working on other projects that
have priority. I wrote that in my first post to not deceive you
afterwards.
> class { webserver: ipaddress => "..." }
>
> So I'm ok with your basic request, but I don't see it as a complete
> syntactical proposal yet.
The syntax I proposed for "class instantiation" was:
webserver { ipaddress => "..." }
Or maybe
include webserver { ipaddress => "..." }
to keep backwards compatibility.
Class definition would be:
class webserver ($ipaddress) {...}
This syntax doesn't clash with functions, does it?
The difficulty I see is that classes included at several places "count"
only once. So this causes problems here because you could have the same
class included twice with different parameters.
To me this doesn't matter as I never include the same class twice. But
others probably want to continue being able to do so.
I only see the following options:
1) You throw an exception if the parameters for two includes of the same
class are not the same at runtime.
2) You "preconfigure" classes once with the above syntax somewhere in
global context and then include them without repeating the parameters
wherever you like.
I don't like either of the solutions very much. But that probably has
more to do with the fact that I find puppet's concept of classes
"difficult". So if one of the above two solutions is in line with the
current concept of classes then I'd implement it that way.
> You've syntactical confusion here, though -- Puppet's current parser
> would assume these are functions, not classes. You could try to
> autodetect the difference, but I think that would be much worse.
Well this can be easily resolved by requiring () for functions and
something like a "new" keyword for the instantiation of "bundles". I
certainly won't have to prove OO conceptual or practical viability. To
me it is more a matter of conviction to get puppet closer to a simple
and restricted OO language than a technical problem.
I have found workarounds and conventions for doing most of the things I
propose in puppet's current syntax here and now. I am "abusing" some
puppet concepts a little to get there but I get an easier to maintain
system in return. So there is no doubt that what I am proposing works
for me.
It all comes down to one question: Does "pure" OO provide a "better"
framework to represent configured resources or not?
If I have not convinced you by now I won't be able to do so in further
posts. I can't add any new arguments. I am just repeating myself. This
is not helpful.
To me this discussion was useful to consolidate my thoughts. I am using
puppet quite differently now. I hope that you got something out of it as
well. Maybe we should just leave it that way to not further waste your time?
Florian
Luke Kanies
>
> Hi Luke,
>
>> Could you produce patches that provided
>> what you want?
>
> No, unfortunately not. I am currently working on other projects that
> have priority. I wrote that in my first post to not deceive you
> afterwards.
Ah, that's right. I asked again because implementation often provides
key information on the complexity of an idea, and I think it would in
this case.
>
>> class { webserver: ipaddress => "..." }
>>
>> So I'm ok with your basic request, but I don't see it as a complete
>> syntactical proposal yet.
>
> The syntax I proposed for "class instantiation" was:
>
> webserver { ipaddress => "..." }
>
> Or maybe
>
> include webserver { ipaddress => "..." }
>
> to keep backwards compatibility.
>
> Class definition would be:
>
> class webserver ($ipaddress) {...}
>
> This syntax doesn't clash with functions, does it?
>
> The difficulty I see is that classes included at several places
> "count"
> only once. So this causes problems here because you could have the
> same
> class included twice with different parameters.
>
> To me this doesn't matter as I never include the same class twice. But
> others probably want to continue being able to do so.
This is the same problem that convinced me to remove exactly this
syntax a couple of years ago.
>
> I only see the following options:
> 1) You throw an exception if the parameters for two includes of the
> same
> class are not the same at runtime.
> 2) You "preconfigure" classes once with the above syntax somewhere in
> global context and then include them without repeating the parameters
> wherever you like.
>
> I don't like either of the solutions very much. But that probably has
> more to do with the fact that I find puppet's concept of classes
> "difficult". So if one of the above two solutions is in line with the
> current concept of classes then I'd implement it that way.
The problem doesn't really change if you switch to your concept of
bundles, as far as I can see.
Okay. At the least, I think this conversation has gone as far as it
can without implementation, so until someone has the cycles to start
developing this, there's not much point in continuing.
I'll continue trying to come up with a workable way of solving the
class parameter problem, and if I do, I'll get it implemented.
--
Smoking is one of the leading causes of statistics. -- Fletcher Knebel
Florian Grandel
> The problem doesn't really change if you switch to your concept of
> bundles, as far as I can see.
In my concept there is no such thing as "multiple inserts". Multiple
uses of the same bundle create different objects (as in every OO
language...). Remember that one of my main ideas was to remove the
distinction between classes and definitions. Bundles are very similar to
definitions.
But this is all theoretical and doesn't help you to find a solution for
the practical class parameter problem, sorry. :-(
I agree that we have come as far as possible without coding something.
Florian