ShadowPuppet
jesse newland
Like Luke mentioned a while back [1], I've been working on a project
at Rails Machine called Moonshine [2] [3], which is largely based on
Puppet. While Moonshine as a whole isn't ready yet, I've gone ahead
and extracted the Puppet parts into its own project: ShadowPuppet.
ShadowPuppet is a re-implementation of Puppet::DSL designed for use
by, well, Rubyists. There's actually not very much *DSL* in
ShadowPuppet at all: manifests are Classes, groups of resources are
instance methods, and 'plugins' are implemented as Modules.
Source: http://github.com/railsmachine/shadow_puppet
Rdoc: http://railsmachine.github.com/shadow_puppet/
Sample Manifest: http://github.com/railsmachine/shadow_puppet/blob/master/examples/foo.rb
There are a few differences worth mentioning between the traditional
Puppet approach to how ShadowPuppet works:
# ShadowPuppet doesn't have puppetmaster
# You execute a ShadowPuppet manifest in one of two ways:
## With the included 'shadow_puppet' binary.
## Creating an instance of ShadowPuppet::Manifest and calling the
'execute' method on that instance.
We'll be blogging about this soon [4], but I wanted to share this with
puppet-dev first to get some feedback from you all. Thoughts?
1: http://groups.google.com/group/puppet-dev/msg/9ddfcfbb782d4a11
2: http://files.jnewland.com/moonshine.pdf
3: http://blog.railsmachine.com/articles/2009/01/16/moonshine-configuration-management-and-deployment/
4: http://blog.railsmachine.com/
Regards -
Jesse Newland
---
je...@jnewland.com
404.216.1093
http://jnewland.com/
http://twitter.com/jnewland
Luke Kanies
>
> Hey all -
>
> Like Luke mentioned a while back [1], I've been working on a project
> at Rails Machine called Moonshine [2] [3], which is largely based on
> Puppet. While Moonshine as a whole isn't ready yet, I've gone ahead
> and extracted the Puppet parts into its own project: ShadowPuppet.
>
> ShadowPuppet is a re-implementation of Puppet::DSL designed for use
> by, well, Rubyists. There's actually not very much *DSL* in
> ShadowPuppet at all: manifests are Classes, groups of resources are
> instance methods, and 'plugins' are implemented as Modules.
>
> Source: http://github.com/railsmachine/shadow_puppet
> Rdoc: http://railsmachine.github.com/shadow_puppet/
> Sample Manifest: http://github.com/railsmachine/shadow_puppet/blob/master/examples/foo.rb
(This is long; please bear with me.)
Summary:
I propose that we build/adopt a pure-ruby DSL (hereafter called the
'internal DSL'), and that this DSL be built so that it can be used to
write Puppet manifests in pure ruby, but also so it can be used by
Puppet's own parser ('external DSL' from here on). The ultimate goal
of this internal DSL would be to replace the existing DSL in the RAL,
used for specifying resource types (i.e., the 'file' et al that you
know and love).
The development process would be gradual: We'd first create the DSL
for direct usage, then begin porting the parser over to use it, and,
once we were comfortable it was correct, begin porting the RAL over to
it.
In the process, our language-level defined resources and classes and
such would get loads of extra functionality, like parameter
validation, builtin documentation, and more.
Here goes.
=============================
So, I've been stewing on ShadowPuppet since you published it, and I
think I have a bit of a plan for how we can both integrate it into
Puppet and reduce code overlap.
First, a bit of a simplification. ShadowPuppet focuses on creating
pure Ruby classes to do Puppet work, and puts all of the resource
definitions (and any related code) into methods; here's a shortened
example:
class Foo < ShadowPuppet::Manifest
recipe :some_gems
def some_gems
package 'rails', :ensure => :updated, :provider => :gem
package 'railsmachine', :ensure => '1.0.5', :provider
=> :gem, :require => package('capistrano')
package 'capistrano', :ensure => :updated, :provider => :gem
end
end
Jesse and I discussed this in person; I find this a bit clumsy, at
least for the default case where you've only got one recipe. However,
you can easily put a simple wrapper around it, so it looked like this:
puppetclass :foo do
package 'rails', :ensure => :updated, :provider => :gem
package 'railsmachine', :ensure => '1.0.5', :provider
=> :gem, :require => package('capistrano')
package 'capistrano', :ensure => :updated, :provider => :gem
end
Internally, it would just create the class, and turn the block into a
default method, then call 'recipe' with that method name.
To me, this is important, because it makes it easy to define new
Puppet classes inside other methods, which is critical for what I'm
proposing. I'd recommend this wrapper, and any similar wrappers that
might get created, be put into some kind of module that can be easily
included on other classes.
The main thing I'd like to see is both the internal DSL and the
external DSL use the same classes, and, effectively, the same code for
defining the major functional elements of a manifest. For example,
our parser creates a class in the language something like this (with
the variables converted to strings, and rearranged to be more readable):
@astset.definitions[name] = AST::HostClass.new(
:namespace => "foo::bar",
:code => options[:code],
:parser => self,
:classname => "baz",
:doc => options[:doc]
)
Some of this is internal bookkeeping (the reference to the parser, for
instance), some of it is in ShadowPuppet (the name, and maybe the
namespace), and some of it is missing (the docs).
We could envision, however, this being called just about exactly like
the 'puppetclass' above:
puppetclass "foo::bar::baz", :doc => options[:doc] do
...
end
Now, this is where I go off the deep end a bit.
The *model* of a class, or a defined resource type, or a node, can be
exactly the same in the internal and external DSLs. They'd all
support docs, they'd all have names, namespaces, and arguments would
be supported in a similar way when supported.
The difference between them is whatever's in that block of code passed
to the 'puppetclass' call. When using the internal DSL, that block
has a bunch of Ruby code. Note that we don't know what that ruby code
does -- it's essentially an opaque chunk of code that Ruby turns into
resources. When using the external DSL, that block has a bunch of AST
code. Again, it's opaque -- we just know that our parser turns into
resources.
And here's the key: The majority of the code in Puppet's classes,
definitions, and nodes is unrelated that opaque chunk of code - we
just pass it to someone else and expect resources to happen. The code
is instead all about managing arguments, namespaces, collisions,
scopes, etc. - i.e., the model. Since it makes sense for the two DSLs
to share this model, it makes sense for them to share whatever code
they can.
Puppet would then just track file extensions -- .rb means use the
internal DSL, and .pp means use the external DSL. You can write in
what you want to write in, you can seamlessly transition from simple
Puppet code to complete Ruby code, and our burden of code maintenance
is low.
I think this is pretty cool, but this isn't quite the angels-singing-
from-on-high revelation I had afterward.
I've been wanting to refactor Puppet's internal RAL API for, well, a
long time. It appears to be code that only its father could love, and
even my affection is wearing thin. The key realization, though, is
that it is itself a DSL for defining resource types and their
parameters.
Isn't that a problem we have in the language right now? Isn't that
what we do when we use 'define'?
So the question becomes, if we create a new internal DSL for defining
resource types, classes, etc, how would it differ -- in the model --
from Puppet's existing RAL?
And I would claim that it would not. In other words, ShadowPuppet's
internal DSL would make *three* places where the same models would be
used -- internal DSL, external DSL, and the RAL -- yet aren't
currently planning on using the same code base.
To me, this represents a huge opportunity. We can build this new
internal DSL, with the support we need for defining resource types,
their arguments, and all that, and once we're comfortable with this
new DSL, we switch Puppet's RAL to use it. Then, the only difference
between a pure-ruby defined resource type and a builtin resource type
would be that the builtin resource type has providers associated with
it, while the defined resource type generates other resources.
Think about this: You build your defined resource types, but they
work like defined resources do today -- they accept arguments and
create other resources. Once you've got the model all figured out,
you decide you're tired of using 'exec' and 'file' resources or
whatever to do your work, so you create a provider to do it. And now
here's the key: Your resource type barely changes -- you translate
the resources it generates into a provider, and you're done. No
changes to the parameters.
Let's go through a simple iteration of a resource type, as a means of
demonstrating the DSL. Take a basic hook for defining our type:
resource_type :user do
# have a couple of execs that do what you want
end
Of course, you want to accept arguments to your user -- each one is
different, after all:
resource_type :user, :arguments => [:ensure, :shell] { ... }
Hmm, that's good, but I want a default value for the shell; ok:
resource_type :user, :arguments => {:ensure => nil, :shell => "/usr/
bin/bash"} do
...
end
Well, actually, I want validation on these values. For instance,
there are only two valid values to 'ensure':
resource_type :user, :arguments => {
:ensure => [:absent, :present],
:shell => "/usr/bin/bash"
} do
...
end
This works, but it starts to get a bit hideous when you've got, say,
6-8 parameters, and they want to do some basic validation:
resource_type :user, :arguments => {
:ensure => {
:allowed_values => [:absent, :present]
},
:shell => {
:default => "/usr/bin/bash",
:allowed_values => %w{/usr/bin/bash /usr/bin/sh ...}
}
} do ... end
Okay, so we could maybe support an alternate syntax (I'm pointedly
*not* recommending this syntax, just using it as an example):
resource_type :user, :inline => false do
parameter(:ensure) do
allowed_values :absent, :present
end
parameter(:shell) do
default "/usr/bin/bash"
allowed_values %w{...}
end
resources do
...original resource code goes here...
end
end
Here I've declared that the resource type's arguments aren't specified
inline, assuming that the default would be the earlier, one-line
definitions, but supporting these more complicated definitions via
this switch.
Now let's look at what a simple before and after might look like, when
switching from defined resources to builtin:
resource_type :user, :inline => false do
parameter(:ensure) do
allowed_values :absent, :present
end
parameter(:shell) do
default "/usr/bin/bash"
allowed_values %w{...}
end
resources do
if self.ensure == :absent
exec "rm-#{name}",
:command => "/usr/bin/userdel #{name}",
:onlyif => "finger #{name}"
else
exec "add-#{name}",
:command => "/usr/bin/useradd #{name}",
:unless => "finger #{name}"
end
end
Would convert to:
# Resource type
resource_type :user, :inline => false do
parameter(:ensure) do
allowed_values :absent, :present
end
parameter(:shell) do
default "/usr/bin/bash"
allowed_values %w{...}
end
end
# Provider
resource_type(:user).provides :useradd do
commands :userdel, :useradd
def ensure=(value)
if value == :absent
userdel name # using the same command magic we have now
else
useradd name
end
end
end
See how our Resource Type hasn't changed, just the code used to
actually do the work?
So, the questions are:
* Do we actually want the above functionality (validation, etc.) when
defining resource types, classes, etc., in the language?
* Does it make sense to develop this DSL as something that can
eventually (even if not immediately) be the basis for the internal and
external DSLs *and* the RAL itself?
To me, the answers to both of these questions are an emphatic yes!,
but I'd love other feedback.
For the record, I've gotten mostly *crickets* when discussing this
individually, so I expect I'm a bit off the deep end here.
--
Men never do evil so completely and cheerfully as when they do it from a
religious conviction. --Blaise Pascal
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Steven Jenkins
Disclaimer: I believe the people writing the code should make the
decisions. By that metric, my input counts very, very little, if anything.
Luke Kanies wrote:
...
> Okay, so we could maybe support an alternate syntax (I'm pointedly
> *not* recommending this syntax, just using it as an example):
>
...
<Lots of details deleted>
3 layers? (internal DSL, external DSL, and RAL) From an architectural
viewpoint, that seems too many. Can you elaborate on why you think you
need 3? (as opposed to, say, 2 or 4?) In other words, what
distinguishes one layer from the other?
>
> Now let's look at what a simple before and after might look like, when
> switching from defined resources to builtin:
Your examples in this mail seem to be about
- providing default values
- providing enumerated types (which you call 'validation', but the only
validation being done is making sure values are members of a
pre-determined set, which is an enumerated type)
- working out the syntax and semantics of an evaluation engine that can
move code from one of the 3 implementation languages into a 4th (the
actual internal Ruby runtime).
...
> See how our Resource Type hasn't changed, just the code used to
> actually do the work?
>
This particular example is really a type of macro expansion/rewriting
rules. It seems an exercise in 'I don't like this syntax, but this
other one seems better'. That's good, but I feel like I'm missing
something critical there (i.e., the 'why?').
> So, the questions are:
>
> * Do we actually want the above functionality (validation, etc.) when
> defining resource types, classes, etc., in the language?
I would tentatively say 'yes', here. Default values and enumerated
types are useful programming constructs.
>
> * Does it make sense to develop this DSL as something that can
> eventually (even if not immediately) be the basis for the internal and
> external DSLs *and* the RAL itself?
>
I don't quite get the point of this part. I'm hesitant in part because
I think trying to design 3 languages (internal, external, and RAL) is
really, really hard.
> To me, the answers to both of these questions are an emphatic yes!,
> but I'd love other feedback.
>
> For the record, I've gotten mostly *crickets* when discussing this
> individually, so I expect I'm a bit off the deep end here.
>
Many of those crickets live in my yard, so your admonition for people to
chime in is well-aimed and well-received, at least by me.
Steven Jenkins
End Point Corporation
Luke Kanies
>
>
> Disclaimer: I believe the people writing the code should make the
> decisions. By that metric, my input counts very, very little, if
> anything.
>
> Luke Kanies wrote:
> ...
>> Okay, so we could maybe support an alternate syntax (I'm pointedly
>> *not* recommending this syntax, just using it as an example):
>>
> ...
>
> <Lots of details deleted>
>
> 3 layers? (internal DSL, external DSL, and RAL) From an
> architectural
> viewpoint, that seems too many. Can you elaborate on why you think
> you
> need 3? (as opposed to, say, 2 or 4?) In other words, what
> distinguishes one layer from the other?
I probably should have started with a higher-level summary, I suppose.
The basic situation right now is that we have a language and a
library. The language is an external DSL, but some want an internal
DSL.
The language and library were developed as separate subsystems because
I didn't see how they could be anything else.
The main point of my post was actually that it *shouldn't* be three
layers, but given the current system would need to be modeled that way.
I want 1 layer, but as it stands, we'd have three - the library
itself, the external DSL, and the internal DSL.
>> Now let's look at what a simple before and after might look like,
>> when
>> switching from defined resources to builtin:
>
> Your examples in this mail seem to be about
>
> - providing default values
> - providing enumerated types (which you call 'validation', but the
> only
> validation being done is making sure values are members of a
> pre-determined set, which is an enumerated type)
Sure, I just didn't feel like adding a regex-based validator, or a
code-based one.
>
> - working out the syntax and semantics of an evaluation engine that
> can
> move code from one of the 3 implementation languages into a 4th (the
> actual internal Ruby runtime).
Sure, that seems a reasonable way to think about it, although again,
the goal is that we'd only have 1 implementation language - the whole
point is reducing to one from three.
And the providers don't use the same model at all, so I don't think
they qualify as an equivalent level to the resource specification
languages.
>
> ...
>> See how our Resource Type hasn't changed, just the code used to
>> actually do the work?
>>
>
> This particular example is really a type of macro expansion/rewriting
> rules. It seems an exercise in 'I don't like this syntax, but this
> other one seems better'. That's good, but I feel like I'm missing
> something critical there (i.e., the 'why?').
It's not the awesomest example, I'm sure. And you're probably right
that we could envision the providers as having a similar model to the
resource types I talk about in my example.
There's context missing from my example that matters, though -- you'd
tend to have multiple providers for a given resource type, so the
resource type's job is to define the model (what parameters are
allowed, what values do they allow, etc.) and the provider's job is to
implement a given kind of behaviour.
In terms of the transition I tried to demonstrate, the key is that it
went from Puppet resources to pure Ruby, thus starting above the
abstraction layer and ending up below it, and the interesting bit is
that the resource model didn't change, only the implementation for
doing the work.
Is that a sufficient explanataion?
>
>> So, the questions are:
>>
>> * Do we actually want the above functionality (validation, etc.) when
>> defining resource types, classes, etc., in the language?
>
> I would tentatively say 'yes', here. Default values and enumerated
> types are useful programming constructs.
>
>>
>> * Does it make sense to develop this DSL as something that can
>> eventually (even if not immediately) be the basis for the internal
>> and
>> external DSLs *and* the RAL itself?
>>
>
> I don't quite get the point of this part. I'm hesitant in part
> because
> I think trying to design 3 languages (internal, external, and RAL) is
> really, really hard.
Right. And maintaining them is hard.
So, really, the question is, can we do it in one language instead of 3?
>
>> To me, the answers to both of these questions are an emphatic yes!,
>> but I'd love other feedback.
>>
>> For the record, I've gotten mostly *crickets* when discussing this
>> individually, so I expect I'm a bit off the deep end here.
>>
>
> Many of those crickets live in my yard, so your admonition for
> people to
> chime in is well-aimed and well-received, at least by me.
Heh, I think it's more my wide-eyed declarations that this is really
awesome. "Um, that's just not that exciting." -- everyone
--
The easiest way to figure the cost of living is to take your income and
add ten percent.
James Turnbull
Hash: SHA1
Now I might be the only one but I like Puppet's DSL. :)
I think it's an excellent value add and an easy entry point. I don't
think the rush to do a Ruby-based DSL - al la Chef - is a good fit for
many users. Especially non-web/non-Ruby enterprise customers who don't
want to learn Ruby or a Ruby-based DSL.
I would be very reluctant to support anything that did away with the DSL
entirely.
Regards
James Turnbull
- --
Author of:
* Pulling Strings with Puppet
(http://www.amazon.com/gp/product/1590599780/)
* Pro Nagios 2.0
(http://www.amazon.com/gp/product/1590596099/)
* Hardening Linux
(http://www.amazon.com/gp/product/1590594444/)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFJkocN9hTGvAxC30ARAg60AKCgK7D5l4+cMuvTIB7yKtXaMZVEawCgqj8c
gcJvT0/Nkk0iF8bOMr607TY=
=4cyD
-----END PGP SIGNATURE-----
Peter Meier
> Now I might be the only one but I like Puppet's DSL. :)
>
> I think it's an excellent value add and an easy entry point. I don't
> think the rush to do a Ruby-based DSL - al la Chef - is a good fit for
> many users. Especially non-web/non-Ruby enterprise customers who don't
> want to learn Ruby or a Ruby-based DSL.
>
> I would be very reluctant to support anything that did away with the DSL
> entirely.
I'd like to second that. The external DSL has to stay! :)
I didn't understand that we'd like to get rid of it, but in all the
ongoing discussions it might be good to state once that the external
DSL is really nice and that it is an advantage to have a restricted
external DSL.
It's always the question if you can live with the trade offs of an
external DSL or that you simply start programming in the host language
itself if you're using an internal DSL, hence the internal DSL becomes
ambiguous.
I would never say that the external DSL is the key/a problem of puppet
and assume that not many people would disagree on that.
However I sometimes came to the point where I had to admit, that my
defines got a way to complex and some of the complexity was due to the
language restrictions. In discussions we then came to the point that
this might have been the point to write a custom type, where we could
program in pure ruby. However this also would have had its
disadvantages, as many parts of the defines were really simple in the
external DSL and the implementation of that in types would be far more
clumsy and complex.
So what we then missed is an easy way to address some of the
restrictions we have in the external DSL, but still would have the
power of the puppet language.
Writing custom types is quite easy, however most of the time it's far
more elegant and faster to write a simple define.
Looking at how many custom types are out there, I assume that somehow
it's still a way to complex to write them. I might be wrong, but it's
the current intention/feeling I have.
So long I think Luke's current proposal would address all these issues
and I really like it. I think it would be nice to get that kind of
behavior and it would extend the possibilities a lot, as well make
writing modules a lot easier.
The main goal imho is to not loose the power of the external DSL, but
to give easier possibilities to address certain restrictions in pure
ruby.
cheers pete
Luke Kanies
> So what we then missed is an easy way to address some of the
> restrictions we have in the external DSL, but still would have the
> power of the puppet language.
Right - the goal would be to keep the external DSL while adding an
internal DSL, without adding significant code overhead to maintain
them separately.
In fact, I'd like to go a bit further than that -- I'd like the parser
system to support pure-data formats like YAML and JSON. I don't know
exactly how this would integrate with the current system, but
certainly I think it should be easy to store resource information in
multiple formats.
And while I agree that many people want to stick with the external
DSL, I've got multiple customers where all of the Puppet development
is done by their developers, not their sysadmins (the devs write the
code to manage their own apps). These developers would generally much
prefer to use ruby, from what they're saying. Even the internal DSL
gives you decent protection, because it only runs on the server (in
most cases) - you can't just arbitrarily do whatever you want.
>
> Writing custom types is quite easy, however most of the time it's far
> more elegant and faster to write a simple define.
> Looking at how many custom types are out there, I assume that somehow
> it's still a way to complex to write them. I might be wrong, but it's
> the current intention/feeling I have.
I think it's more a lack of clarity than unnecessary complexity, but
then, I wrote it, so it never seems complex to me.
However, there have been enough complaints about it that I've been
planning for ages to refactor the API for the RAL. The goal here
would be to iterate on the language side until we came up with an API
we liked for resource type specification (i.e., specifying defined
resources and classes and nodes), and once we're happy with it,
migrate the RAL API to use that same interface.
>
> So long I think Luke's current proposal would address all these issues
> and I really like it. I think it would be nice to get that kind of
> behavior and it would extend the possibilities a lot, as well make
> writing modules a lot easier.
> The main goal imho is to not loose the power of the external DSL, but
> to give easier possibilities to address certain restrictions in pure
> ruby.
Exactly. It's about adding the cheapest, most forward-looking way,
not subtracting.
--
Trying to determine what is going on in the world by reading
newspapers is like trying to tell the time by watching the second
hand of a clock. --Ben Hecht
Brice Figureau
On Tue, 2009-02-10 at 19:12 -0600, Luke Kanies wrote:
> On Jan 26, 2009, at 4:41 PM, jesse newland wrote:
>
> I propose that we build/adopt a pure-ruby DSL (hereafter called the
> 'internal DSL'), and that this DSL be built so that it can be used to
> write Puppet manifests in pure ruby, but also so it can be used by
> Puppet's own parser ('external DSL' from here on). The ultimate goal
> of this internal DSL would be to replace the existing DSL in the RAL,
> used for specifying resource types (i.e., the 'file' et al that you
> know and love).
>
> The development process would be gradual: We'd first create the DSL
> for direct usage, then begin porting the parser over to use it, and,
> once we were comfortable it was correct, begin porting the RAL over to
> it.
>
> In the process, our language-level defined resources and classes and
> such would get loads of extra functionality, like parameter
> validation, builtin documentation, and more.
>
[huge snip]
> So, the questions are:
>
> * Do we actually want the above functionality (validation, etc.)
> when
> defining resource types, classes, etc., in the language?
Actually you as a developper want this, but I'm not sure the usual
sysadmin want to bother with this when writing his manifests.
I still think this is a cool feature to have, but I'm not even sure I'd
use it, after all in my modest environment, I'm the only puppet manifest
writer, so I know how my defines are working and what they want as
input.
Something absolutely not related that I'd love to have as a Puppet user
(and not with my developper hat on), would be to have the possibility to
define a public api for a module (ie a kind of parametrized class, where
you say this module/class needs those entries...). But that's a complete
different story which has been already discussed here :-)
> * Does it make sense to develop this DSL as something that can
> eventually (even if not immediately) be the basis for the internal
> and
> external DSLs *and* the RAL itself?
I understand what you want to achieve (or at least I think I
understand). That would indeed be great to have a common stable layer on
which everything would be built (I don't really see how the RAL can be
built on top of this, but I know you have a better vision than mine, so
I trust you on this).
The thing I'm not sure would be usefull for puppet end-users (and
usually users have different views from the developpers) is the ruby
DSL. You know, we're doing products for them, so you have to compare the
added benefit of this layer, against the benefit for your users, and the
time you won't have for other features they need or request.
So, I'm absolutely not against a move toward this direction (and far
from that in fact), as long we have the guarantee that we'll keep the
external DSL in place (which I understand is your plan), because I like
the safeguards it puts (ie to force you to write manifests in the puppet
sense and not against it). It also has the advantage that you can't
fiddle with the master internals from the language, which I guess the
pure ruby DSL would allow.
> To me, the answers to both of these questions are an emphatic yes!,
> but I'd love other feedback.
>
> For the record, I've gotten mostly *crickets* when discussing this
> individually, so I expect I'm a bit off the deep end here.
Being non English native sometimes has some disadvantages, I have
difficulties to parse the above sentence :-)
So, to summarize my point of view:
* I'm not sure current puppet users want a ruby DSL (but, again, frankly
I don't know what they want. Some might want ruby everywhere, otherwise
Chef wouldn't have been created). Maybe your survey (which I didn't fill
btw) can answer this.
* Do it as long as the time invested is worth the result (ie you don't
have to delay a super feature to put in place something that is roughly
equivalent of what your users already have now).
* Do it if you feel that you can attract a different population (ie
programmers in need of a deployement system for instance) than the one
forming the actual puppet user community (which I guess is mostly
sysadmins), and as such grow your user base (which is of course your
ultimate goal to maintain a sustainable business :-))
Anyway, if you need help for the implementation, I'm ready to contribute
as usual ;-)
--
Brice Figureau
My Blog: http://www.masterzen.fr/
Jesse Newland
aflame the existing DSL. I personally use it to manage the
configuration on hundreds of servers. It's awesome and has, as the
saying goes [1], gotten me to the pub by 4pm plenty of times.
On Feb 11, 2009, at 9:45 AM, Luke Kanies wrote:
> I've got multiple customers where all of the Puppet development
> is done by their developers, not their sysadmins (the devs write the
> code to manage their own apps). These developers would generally much
> prefer to use ruby, from what they're saying.
This is the core reason we expanded upon the Puppet Ruby DSL in
Moonshine/ShadowPuppet - to reduce the barrier to entry to idempotent
system configuration management by presenting a certain set of users
(i.e. Rails Developers) with access to the wonderful Puppet "library"
via a set of tools they are familiar with.
Luke, the use of one internal DSL for in-Ruby resource creation,
external DSL parsing, and the RAL sounds like both an excellent plan
and an huge undertaking. I'm happy to help with the implementation of
this in any way I can.
- Jesse
1: http://www.slideshare.net/littleidea/systems-building-systems-a-puppet-story
---
Jesse Newland
http://jnewland.com/
http://twitter.com/jnewland
Luke Kanies
>
> On Tue, 2009-02-10 at 19:12 -0600, Luke Kanies wrote:
> [huge snip]
>
>> So, the questions are:
>>
>> * Do we actually want the above functionality (validation, etc.)
>> when
>> defining resource types, classes, etc., in the language?
>
> Actually you as a developper want this, but I'm not sure the usual
> sysadmin want to bother with this when writing his manifests.
Yeah, I left a lot of this discussion out. There are two ways to look
at this:
First, it wouldn't be targeted at your regular sysadmin, it'd be
targeted at people who already find the existing DSL too limiting,
either because it can't iterate fast enough or because it's missing a
lot of common language functionality.
Second, however, it allows us to keep the current DSL from acquiring
90% of the functionality of other languages. There's been a
disturbing trend recently of writing functions that just import Ruby
functionality into Puppet. I call this 'disturbing', because Ruby is
OO and Puppet is decidedly not, which means they mostly get imported
as functions or new syntaxes. This puts us on a collision course to
either look like PHP (functions) or perl (syntaxes) in a few years,
neither of which is pretty or maintainable.
With this internal DSL we can say, hey, you want simple stuff that
works well and provides guarantees, use the external DSL, but if you
want to get all complicated and want regexes and iteration and other
(actually quite normal) stuff, use the internal DSL.
This way we don't end up with 500 functions and 650 statement types in
a few years.
>
> I still think this is a cool feature to have, but I'm not even sure
> I'd
> use it, after all in my modest environment, I'm the only puppet
> manifest
> writer, so I know how my defines are working and what they want as
> input.
>
> Something absolutely not related that I'd love to have as a Puppet
> user
> (and not with my developper hat on), would be to have the
> possibility to
> define a public api for a module (ie a kind of parametrized class,
> where
> you say this module/class needs those entries...). But that's a
> complete
> different story which has been already discussed here :-)
I'm actually not sure that it is. Right now, Puppet's only conception
of class dependencies is that you can call 'include' from within the
class, but that's a compile-time dependency; it's just as reasonable
to have a parse time dependency, something that might look like:
class foo requires bar {
...
}
Now, I'm not suggesting this syntax, but whatever we did, we'd be
extending both the internal model of classes and definitions -- what
features they have and how we can use them -- *and* adding a new
syntax, and, of course, these would need to be done in tandem. Every
tweak to the model would result in a (probably not backward
compatible) tweak to the syntax.
Instead, we can put ourselves in a situation where we can iterate the
model in the internal DSL, not worrying about a syntax initially, and
once we've got the model nailed, then we wrap an external syntax
around it. This would make it *much* faster to iterate on the model,
and, I think, faster to provide new syntaxes.
So yeah, we're going to have module-level dependencies at some point,
specified in a pure-data file of some kind, but I don't think they'll
get you all the way there.
>
>> * Does it make sense to develop this DSL as something that can
>> eventually (even if not immediately) be the basis for the internal
>> and
>> external DSLs *and* the RAL itself?
>
> I understand what you want to achieve (or at least I think I
> understand). That would indeed be great to have a common stable
> layer on
> which everything would be built (I don't really see how the RAL can be
> built on top of this, but I know you have a better vision than mine,
> so
> I trust you on this).
>
> The thing I'm not sure would be usefull for puppet end-users (and
> usually users have different views from the developpers) is the ruby
> DSL. You know, we're doing products for them, so you have to compare
> the
> added benefit of this layer, against the benefit for your users, and
> the
> time you won't have for other features they need or request.
I agree, on worrying about relative benefits. Fortunately, we've
already got a company (RailsMachine) who's putting development effort
behind their goals in this area (and they're explicitly developing for
ruby developers, so an external DSL won't cut it for them), so
hopefully we can at least partially rely on them, and I'm also talking
to a couple of customers who seem interested enough in pure-ruby
manifests that they're willing to fund developer time here, so it's
not really a zero-sum game.
And don't forget that this also brings what I think is significant
clarity to the whole system, and with that clarity comes both
inspiration and greater maintainability. Until I started thinking
about this, I didn't realize that the definition, hostclass, and node
AST classes *aren't* AST classes at all. If you look in those
classes, you'll see that I've essentially extracted every bit of AST
behaviour from them, and any remaining is a result of lack of clarity.
So really, the next logical step is to pull them out of the AST
hierarchy. I've always found that clarity in the model has a huge
beneficial affect on maintainability, and having multiple DSLs relying
on the same backend code will definitely bring clarity to the model.
Once we've extracted them, we can start relying on them for the basis
of an internal DSL. Then, as mentioned, we can iterate on their model
without changing the syntax, and once we're happy with things like
dependencies and whatever else needs to be added, we can develop
syntaxes that concisely express those new behaviours.
>
> So, I'm absolutely not against a move toward this direction (and far
> from that in fact), as long we have the guarantee that we'll keep the
> external DSL in place (which I understand is your plan), because I
> like
> the safeguards it puts (ie to force you to write manifests in the
> puppet
> sense and not against it). It also has the advantage that you can't
> fiddle with the master internals from the language, which I guess the
> pure ruby DSL would allow.
An internal DSL would allow some mucking around, but that probably
wouldn't happen as much as you think, and, generally, I'd expect to
see that mucking around take two forms: Complete hackery that no one
would admit exists, much less publish, and actual extensions to the
model that should probably find their way into the model itself.
>
>> To me, the answers to both of these questions are an emphatic yes!,
>> but I'd love other feedback.
>>
>> For the record, I've gotten mostly *crickets* when discussing this
>> individually, so I expect I'm a bit off the deep end here.
>
> Being non English native sometimes has some disadvantages, I have
> difficulties to parse the above sentence :-)
Heh. I think it's a cartoon thing -- someone says something profound,
expecting a loud, enthusiastic response, and instead all they hear is
one lone cricket in the corner; the cricket is meant to indicate an
essentially deafening silence. It's probably more an Americanism than
an English thing. I suppose the 'deep end' phrase is similarly
nonportable. :/
>
> So, to summarize my point of view:
> * I'm not sure current puppet users want a ruby DSL (but, again,
> frankly
> I don't know what they want. Some might want ruby everywhere,
> otherwise
> Chef wouldn't have been created). Maybe your survey (which I didn't
> fill
> btw) can answer this.
For the record, I agree - most users today wouldn't switch, and I
think most people who call themselves sysadmins today would use the
external DSL. But we're getting a lot more developers to write
sysadmin code with Puppet, and many of them would tend to use the
internal DSL. And I think there are actually a lot of sysadmins who
chafe at the limitations of the external DSL, and those users probably
tend to be some of the most prolific Puppet developers, so it pays to
enable their creative output.
And really, if it's relatively inexpensive (and I think I've laid out
a plan here where it is), you can essentially throw both out there and
let the market decide.
>
> * Do it as long as the time invested is worth the result (ie you don't
> have to delay a super feature to put in place something that is
> roughly
> equivalent of what your users already have now).
Right. That's one of the reasons for laying out this multi-step
plan: I don't want to find us two years down the road just working on
this. I think there are multiple short-term steps we can take toward
accomplishing this, where each step gives us benefits and costs us
little.
>
> * Do it if you feel that you can attract a different population (ie
> programmers in need of a deployement system for instance) than the one
> forming the actual puppet user community (which I guess is mostly
> sysadmins), and as such grow your user base (which is of course your
> ultimate goal to maintain a sustainable business :-))
I don't really think of it as growing the base so much as making it so
anyone who needs to express system manifests can use Puppet to express
those manifests. And, actually, I think it's a great long-term
direction for refactoring our RAL API. I think that bit was lost on
most people, but if I'm right about this, we'll look up in a year or
two and see a seamless model from writing manifests to writing the
code that manages your actual system.
>
> Anyway, if you need help for the implementation, I'm ready to
> contribute
> as usual ;-)
Awesome. Want to extract the definition, hostclass, and node classes
from AST? :)
Really, though, a question does come up: Where would the results of
this extraction live? In puppet/parser, I guess, but puppet/parser/
model, or something silly like that? No idea.
--
The truth is that there is nothing noble in being superior to somebody
else. The only real nobility is in being superior to your former self.
-- Whitney Young
Luke Kanies
>
> Just to be clear, none of this is about removing / dropping / setting
> aflame the existing DSL. I personally use it to manage the
> configuration on hundreds of servers. It's awesome and has, as the
> saying goes [1], gotten me to the pub by 4pm plenty of times.
In case anyone's curious, that's how Jorge Castro (now at Canonical)
always sold people on Puppet. We stole the slogan from him.
>
> On Feb 11, 2009, at 9:45 AM, Luke Kanies wrote:
>> I've got multiple customers where all of the Puppet development
>> is done by their developers, not their sysadmins (the devs write the
>> code to manage their own apps). These developers would generally
>> much
>> prefer to use ruby, from what they're saying.
>
>
> This is the core reason we expanded upon the Puppet Ruby DSL in
> Moonshine/ShadowPuppet - to reduce the barrier to entry to idempotent
> system configuration management by presenting a certain set of users
> (i.e. Rails Developers) with access to the wonderful Puppet "library"
> via a set of tools they are familiar with.
I really like this focus you have on a specific group of people -- you
know they're already using Ruby, and you know that they're interested
in internal DSLs that can better express the problems they're trying
to solve.
>
> Luke, the use of one internal DSL for in-Ruby resource creation,
> external DSL parsing, and the RAL sounds like both an excellent plan
> and an huge undertaking. I'm happy to help with the implementation of
> this in any way I can.
I really hope it's not a huge undertaking, and I actually belive it's
not, at least leaving out the RAL bits for now (because of their key
backward compatibility concerns). Current versions of Puppet make
this even easier; it would have been harder a year ago.
There's what I might call a symmetry internally in Puppet: Usage of
classes, definitions, and nodes is modeled as resources, and these
things in turn create more resources.
Take something like this:
define foo {
notify { "foo $name": }
}
foo { bar: }
This is two statements: Create the 'foo' defined resource type
(really, we should start calling them 'composite' resources, since
they're just a wrapper around one or more resources), and create an
instance of that resource. The resource type itself gets put into a
table that makes it easy to look it up by name. The 'foo' resource
gets put into our resource graph.
We then, via an iterative process, look up unevaluated resources in
the graph. This process finds the 'foo' resource and calls 'evaluate'
on it. This method knows how to look up the 'foo' resource type and
call the right methods on it. Check out
Puppet::Parser::Resource#evaluate.
The same is true of nodes and classes, except that their resource type
is 'Node' and 'Class', respectively, and they (currently) never accept
arguments.
The key to pulling Puppet and ShadowPuppet together is getting
ShadowPuppet to use a similar system: You've either got resource
types or classes in a table, or you've got resources in a graph that
know how to look up and evaluate their types in that table.
The next step is to unify the instances that get stored in those tables.
I doubt I made it much clearer there, but my main point, the basic
model is actually both simple and clear, and as long as the two tools
use the same model, it should be straightforward to bring them together.
--
I think that's how Chicago got started. A bunch of people in New York
said, 'Gee, I'm enjoying the crime and the poverty, but it just isn't
cold enough. Let's go west.' --Richard Jeni
Paul Lathrop
> * Do we actually want the above functionality (validation, etc.) when
> defining resource types, classes, etc., in the language?
+1
> * Does it make sense to develop this DSL as something that can
> eventually (even if not immediately) be the basis for the internal and
> external DSLs *and* the RAL itself?
Yes, yes, and more yes!
> To me, the answers to both of these questions are an emphatic yes!,
> but I'd love other feedback.
>
> For the record, I've gotten mostly *crickets* when discussing this
> individually, so I expect I'm a bit off the deep end here.
FTR I think this is made of awesome. Especially since I have these
pipe dreams of taking a step up the stack and managing whole clusters
with Puppet. I dream of adding things to load balancers, setting up
masters before slaves, etc. and I think that this is a good first step
on the (long) journey towards that dream.
--Paul
James Turnbull
Hash: SHA1
Paul Lathrop wrote:
> On Tue, Feb 10, 2009 at 5:12 PM, Luke Kanies <lu...@madstop.com> wrote:
>> * Do we actually want the above functionality (validation, etc.) when
>> defining resource types, classes, etc., in the language?
>
> +1
Ditto +100000
>> * Does it make sense to develop this DSL as something that can
>> eventually (even if not immediately) be the basis for the internal and
>> external DSLs *and* the RAL itself?
>
> Yes, yes, and more yes!
If this is plan then me == happy.
>> To me, the answers to both of these questions are an emphatic yes!,
>> but I'd love other feedback.
>>
>> For the record, I've gotten mostly *crickets* when discussing this
>> individually, so I expect I'm a bit off the deep end here.
>
> FTR I think this is made of awesome. Especially since I have these
> pipe dreams of taking a step up the stack and managing whole clusters
> with Puppet. I dream of adding things to load balancers, setting up
> masters before slaves, etc. and I think that this is a good first step
> on the (long) journey towards that dream.
>
Why does my brain play "Land of Hope and Glory" right after this
paragraph. I chose a bad week to give up drinking.
Regards
James
- --
Author of:
* Pulling Strings with Puppet
(http://www.amazon.com/gp/product/1590599780/)
* Pro Nagios 2.0
(http://www.amazon.com/gp/product/1590596099/)
* Hardening Linux
(http://www.amazon.com/gp/product/1590594444/)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFJk0Vu9hTGvAxC30ARAnLVAKDL4b7jJE8Ss1UJZRQYgdIywc8yiACdGr5I
hyh6n7c0ZDZBYYVScLrmGb8=
=mVl2
-----END PGP SIGNATURE-----
Luke Kanies
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Paul Lathrop wrote:
>> On Tue, Feb 10, 2009 at 5:12 PM, Luke Kanies <lu...@madstop.com>
>> wrote:
>>> * Do we actually want the above functionality (validation, etc.)
>>> when
>>> defining resource types, classes, etc., in the language?
>>
>> +1
>
> Ditto +100000
>
>>> * Does it make sense to develop this DSL as something that can
>>> eventually (even if not immediately) be the basis for the internal
>>> and
>>> external DSLs *and* the RAL itself?
>>
>> Yes, yes, and more yes!
>
> If this is plan then me == happy.
Be happy, as that's the plan.
>
>>> To me, the answers to both of these questions are an emphatic yes!,
>>> but I'd love other feedback.
>>>
>>> For the record, I've gotten mostly *crickets* when discussing this
>>> individually, so I expect I'm a bit off the deep end here.
>>
>> FTR I think this is made of awesome. Especially since I have these
>> pipe dreams of taking a step up the stack and managing whole clusters
>> with Puppet. I dream of adding things to load balancers, setting up
>> masters before slaves, etc. and I think that this is a good first
>> step
>> on the (long) journey towards that dream.
>>
>
> Why does my brain play "Land of Hope and Glory" right after this
> paragraph. I chose a bad week to give up drinking.
There's a good week to give up drinking?
--
A person's maturity consists in having found again the seriousness one
had as a child, at play. --Friedrich Nietzsche
Luke Kanies
Good to hear.
--
He attacked everything in life with a mix of extraordinary genius and
naive incompetence, and it was often difficult to tell which was
which. -- Douglas Adams
Jos Backus
> So, the questions are:
>
> * Do we actually want the above functionality (validation, etc.) when
> defining resource types, classes, etc., in the language?
>
> * Does it make sense to develop this DSL as something that can
> eventually (even if not immediately) be the basis for the internal and
> external DSLs *and* the RAL itself?
>
> To me, the answers to both of these questions are an emphatic yes!,
> but I'd love other feedback.
+1
--
Jos Backus
jos at catnook.com
Nigel Kersten
I can't think of any reason why we wouldn't want this so far....