Like Luke mentioned a while back [1], I've been working on a project
at Rails Machine called Moonshine [2] [3], which is largely based on
Puppet. While Moonshine as a whole isn't ready yet, I've gone ahead
and extracted the Puppet parts into its own project: ShadowPuppet.

ShadowPuppet is a re-implementation of Puppet::DSL designed for use
by, well, Rubyists. There's actually not very much *DSL* in
ShadowPuppet at all: manifests are Classes, groups of resources are
instance methods, and 'plugins' are implemented as Modules.

Source: http://github.com/railsmachine/shadow_puppet
Rdoc: http://railsmachine.github.com/shadow_puppet/
Sample Manifest: http://github.com/railsmachine/shadow_puppet/blob/master/examples/foo.rb

There are a few differences worth mentioning between the traditional
Puppet approach to how ShadowPuppet works:

# ShadowPuppet doesn't have puppetmaster
# You execute a ShadowPuppet manifest in one of two ways:
## With the included 'shadow_puppet' binary.
## Creating an instance of ShadowPuppet::Manifest and calling the
'execute' method on that instance.

We'll be blogging about this soon [4], but I wanted to share this with
puppet-dev first to get some feedback from you all. Thoughts?

1: http://groups.google.com/group/puppet-dev/msg/9ddfcfbb782d4a11
2: http://files.jnewland.com/moonshine.pdf
3: http://blog.railsmachine.com/articles/2009/01/16/moonshine-configurat...
4: http://blog.railsmachine.com/

Regards -

Summary:

I propose that we build/adopt a pure-ruby DSL (hereafter called the  
'internal DSL'), and that this DSL be built so that it can be used to  
write Puppet manifests in pure ruby, but also so it can be used by  
Puppet's own parser ('external DSL' from here on).  The ultimate goal  
of this internal DSL would be to replace the existing DSL in the RAL,  
used for specifying resource types (i.e., the 'file' et al that you  
know and love).

The development process would be gradual:  We'd first create the DSL  
for direct usage, then begin porting the parser over to use it, and,  
once we were comfortable it was correct, begin porting the RAL over to  
it.

In the process, our language-level defined resources and classes and  
such would get loads of extra functionality, like parameter  
validation, builtin documentation, and more.

Here goes.

=============================

So, I've been stewing on ShadowPuppet since you published it, and I  
think I have a bit of a plan for how we can both integrate it into  
Puppet and reduce code overlap.

First, a bit of a simplification.  ShadowPuppet focuses on creating  
pure Ruby classes to do Puppet work, and puts all of the resource  
definitions (and any related code) into methods; here's a shortened  
example:

class Foo < ShadowPuppet::Manifest
   recipe :some_gems

   def some_gems
     package 'rails', :ensure => :updated, :provider => :gem
     package 'railsmachine', :ensure => '1.0.5', :provider  
=> :gem, :require => package('capistrano')
     package 'capistrano', :ensure => :updated, :provider => :gem
   end
end

Jesse and I discussed this in person; I find this a bit clumsy, at  
least for the default case where you've only got one recipe.  However,  
you can easily put a simple wrapper around it, so it looked like this:

   puppetclass :foo do
     package 'rails', :ensure => :updated, :provider => :gem
     package 'railsmachine', :ensure => '1.0.5', :provider  
=> :gem, :require => package('capistrano')
     package 'capistrano', :ensure => :updated, :provider => :gem
   end

Internally, it would just create the class, and turn the block into a  
default method, then call 'recipe' with that method name.

To me, this is important, because it makes it easy to define new  
Puppet classes inside other methods, which is critical for what I'm  
proposing.  I'd recommend this wrapper, and any similar wrappers that  
might get created, be put into some kind of module that can be easily  
included on other classes.

The main thing I'd like to see is both the internal DSL and the  
external DSL use the same classes, and, effectively, the same code for  
defining the major functional elements of a manifest.  For example,  
our parser creates a class in the language something like this (with  
the variables converted to strings, and rearranged to be more readable):

         @astset.definitions[name] = AST::HostClass.new(
             :namespace => "foo::bar",
             :code => options[:code],
             :parser => self,
             :classname => "baz",
             :doc => options[:doc]
         )

Some of this is internal bookkeeping (the reference to the parser, for  
instance), some of it is in ShadowPuppet (the name, and maybe the  
namespace), and some of it is missing (the docs).

We could envision, however, this being called just about exactly like  
the 'puppetclass' above:

   puppetclass "foo::bar::baz", :doc => options[:doc] do
     ...
   end

Now, this is where I go off the deep end a bit.

The *model* of a class, or a defined resource type, or a node, can be  
exactly the same in the internal and external DSLs.  They'd all  
support docs, they'd all have names, namespaces, and arguments would  
be supported in a similar way when supported.

The difference between them is whatever's in that block of code passed  
to the 'puppetclass' call.  When using the internal DSL, that block  
has a bunch of Ruby code.  Note that we don't know what that ruby code  
does -- it's essentially an opaque chunk of code that Ruby turns into  
resources.  When using the external DSL, that block has a bunch of AST  
code.  Again, it's opaque -- we just know that our parser turns into  
resources.

And here's the key:  The majority of the code in Puppet's classes,  
definitions, and nodes is unrelated that opaque chunk of code - we  
just pass it to someone else and expect resources to happen.  The code  
is instead all about managing arguments, namespaces, collisions,  
scopes, etc. - i.e., the model.  Since it makes sense for the two DSLs  
to share this model, it makes sense for them to share whatever code  
they can.

Puppet would then just track file extensions -- .rb means use the  
internal DSL, and .pp means use the external DSL.  You can write in  
what you want to write in, you can seamlessly transition from simple  
Puppet code to complete Ruby code, and our burden of code maintenance  
is low.

I think this is pretty cool, but this isn't quite the angels-singing-
from-on-high revelation I had afterward.

I've been wanting to refactor Puppet's internal RAL API for, well, a  
long time.  It appears to be code that only its father could love, and  
even my affection is wearing thin.  The key realization, though, is  
that it is itself a DSL for defining resource types and their  
parameters.

Isn't that a problem we have in the language right now?  Isn't that  
what we do when we use 'define'?

So the question becomes, if we create a new internal DSL for defining  
resource types, classes, etc, how would it differ -- in the model --  
from Puppet's existing RAL?

And I would claim that it would not.  In other words, ShadowPuppet's  
internal DSL would make *three* places where the same models would be  
used -- internal DSL, external DSL, and the RAL -- yet aren't  
currently planning on using the same code base.

To me, this represents a huge opportunity.  We can build this new  
internal DSL, with the support we need for defining resource types,  
their arguments, and all that, and once we're comfortable with this  
new DSL, we switch Puppet's RAL to use it.  Then, the only difference  
between a pure-ruby defined resource type and a builtin resource type  
would be that the builtin resource type has providers associated with  
it, while the defined resource type generates other resources.

Think about this:  You build your defined resource types, but they  
work like defined resources do today -- they accept arguments and  
create other resources.  Once you've got the model all figured out,  
you decide you're tired of using 'exec' and 'file' resources or  
whatever to do your work, so you create a provider to do it.  And now  
here's the key:  Your resource type barely changes -- you translate  
the resources it generates into a provider, and you're done.  No  
changes to the parameters.

Let's go through a simple iteration of a resource type, as a means of  
demonstrating the DSL.  Take a basic hook for defining our type:

   resource_type :user do
     # have a couple of execs that do what you want
   end

Of course, you want to accept arguments to your user -- each one is  
different, after all:

   resource_type :user, :arguments => [:ensure, :shell] { ... }

Hmm, that's good, but I want a default value for the shell; ok:

   resource_type :user, :arguments => {:ensure => nil, :shell => "/usr/
bin/bash"} do
   ...
   end

Well, actually, I want validation on these values.  For instance,  
there are only two valid values to 'ensure':

   resource_type :user, :arguments => {
     :ensure => [:absent, :present],
     :shell => "/usr/bin/bash"
   } do
     ...
   end

This works, but it starts to get a bit hideous when you've got, say,  
6-8 parameters, and they want to do some basic validation:

   resource_type :user, :arguments => {
     :ensure => {
       :allowed_values => [:absent, :present]
     },
     :shell => {
       :default => "/usr/bin/bash",
       :allowed_values => %w{/usr/bin/bash /usr/bin/sh ...}
     }
   } do ... end

Okay, so we could maybe support an alternate syntax (I'm pointedly  
*not* recommending this syntax, just using it as an example):

   resource_type :user, :inline => false do

     parameter(:ensure) do
       allowed_values :absent, :present
     end

     parameter(:shell) do
       default "/usr/bin/bash"
       allowed_values %w{...}
     end

     resources do
       ...original resource code goes here...
     end
end

Here I've declared that the resource type's arguments aren't specified  
inline, assuming that the default would be the earlier, one-line  
definitions, but supporting these more complicated definitions via  
this switch.

Now let's look at what a simple before and after might look like, when  
switching from defined resources to builtin:

   resource_type :user, :inline => false do

     parameter(:ensure) do
       allowed_values :absent, :present
     end

     parameter(:shell) do
       default "/usr/bin/bash"
       allowed_values %w{...}
     end

Disclaimer: I believe the people writing the code should make the
decisions.  By that metric, my input counts very, very little, if anything.

<Lots of details deleted>

3 layers?  (internal DSL, external DSL, and RAL)  From an architectural
viewpoint, that seems too many.  Can you elaborate on why you think you
need 3?  (as opposed to, say, 2 or 4?)  In other words, what
distinguishes one layer from the other?

- providing default values
- providing enumerated types (which you call 'validation', but the only
validation being done is making sure values are members of a
pre-determined set, which is an enumerated type)
- working out the syntax and semantics of an evaluation engine that can
move code from one of the 3 implementation languages into a 4th (the
actual internal Ruby runtime).

...

The basic situation right now is that we have a language and a  
library.  The language is an external DSL, but some want an internal  
DSL.

The language and library were developed as separate subsystems because  
I didn't see how they could be anything else.

The main point of my post was actually that it *shouldn't* be three  
layers, but given the current system would need to be modeled that way.

I want 1 layer, but as it stands, we'd have three - the library  
itself, the external DSL, and the internal DSL.

And the providers don't use the same model at all, so I don't think  
they qualify as an equivalent level to the resource specification  
languages.

There's context missing from my example that matters, though -- you'd  
tend to have multiple providers for a given resource type, so the  
resource type's job is to define the model (what parameters are  
allowed, what values do they allow, etc.) and the provider's job is to  
implement a given kind of behaviour.

In terms of the transition I tried to demonstrate, the key is that it  
went from Puppet resources to pure Ruby, thus starting above the  
abstraction layer and ending up below it, and the interesting bit is  
that the resource model didn't change, only the implementation for  
doing the work.

Is that a sufficient explanataion?

So, really, the question is, can we do it in one language instead of 3?

Now I might be the only one but I like Puppet's DSL. :)

I think it's an excellent value add and an easy entry point.  I don't
think the rush to do a Ruby-based DSL - al la Chef - is a good fit for
many users. Especially non-web/non-Ruby enterprise customers who don't
want to learn Ruby or a Ruby-based DSL.

I would be very reluctant to support anything that did away with the DSL
entirely.

Regards

James Turnbull

- --
Author of:
* Pulling Strings with Puppet
(http://www.amazon.com/gp/product/1590599780/)
* Pro Nagios 2.0
(http://www.amazon.com/gp/product/1590596099/)
* Hardening Linux
(http://www.amazon.com/gp/product/1590594444/)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

I would never say that the external DSL is the key/a problem of puppet  
and assume that not many people would disagree on that.

However I sometimes came to the point where I had to admit, that my  
defines got a way to complex and some of the complexity was due to the  
language restrictions. In discussions we then came to the point that  
this might have been the point to write a custom type, where we could  
program in pure ruby. However this also would have had its  
disadvantages, as many parts of the defines were really simple in the  
external DSL and the implementation of that in types would be far more  
clumsy and complex.
So what we then missed is an easy way to address some of the  
restrictions we have in the external DSL, but still would have the  
power of the puppet language.
Writing custom types is quite easy, however most of the time it's far  
more elegant and faster to write a simple define.
Looking at how many custom types are out there, I assume that somehow  
it's still a way to complex to write them. I might be wrong, but it's  
the current intention/feeling I have.

So long I think Luke's current proposal would address all these issues  
and I really like it. I think it would be nice to get that kind of  
behavior and it would extend the possibilities a lot, as well make  
writing modules a lot easier.
The main goal imho is to not loose the power of the external DSL, but  
to give easier possibilities to address certain restrictions in pure  
ruby.

In fact, I'd like to go a bit further than that -- I'd like the parser  
system to support pure-data formats like YAML and JSON.  I don't know  
exactly how this would integrate with the current system, but  
certainly I think it should be easy to store resource information in  
multiple formats.

And while I agree that many people want to stick with the external  
DSL, I've got multiple customers where all of the Puppet development  
is done by their developers, not their sysadmins (the devs write the  
code to manage their own apps).  These developers would generally much  
prefer to use ruby, from what they're saying.  Even the internal DSL  
gives you decent protection, because it only runs on the server (in  
most cases) - you can't just arbitrarily do whatever you want.

However, there have been enough complaints about it that I've been  
planning for ages to refactor the API for the RAL.  The goal here  
would be to iterate on the language side until we came up with an API  
we liked for resource type specification (i.e., specifying defined  
resources and classes and nodes), and once we're happy with it,  
migrate the RAL API to use that same interface.

I still think this is a cool feature to have, but I'm not even sure I'd
use it, after all in my modest environment, I'm the only puppet manifest
writer, so I know how my defines are working and what they want as
input.

Something absolutely not related that I'd love to have as a Puppet user
(and not with my developper hat on), would be to have the possibility to
define a public api for a module (ie a kind of parametrized class, where
you say this module/class needs those entries...). But that's a complete
different story which has been already discussed here :-)

The thing I'm not sure would be usefull for puppet end-users (and
usually users have different views from the developpers) is the ruby
DSL. You know, we're doing products for them, so you have to compare the
added benefit of this layer, against the benefit for your users, and the
time you won't have for other features they need or request.

So, I'm absolutely not against a move toward this direction (and far
from that in fact), as long we have the guarantee that we'll keep the
external DSL in place (which I understand is your plan), because I like
the safeguards it puts (ie to force you to write manifests in the puppet
sense and not against it). It also has the advantage that you can't
fiddle with the master internals from the language, which I guess the
pure ruby DSL would allow.

So, to summarize my point of view:
* I'm not sure current puppet users want a ruby DSL (but, again, frankly
I don't know what they want. Some might want ruby everywhere, otherwise
Chef wouldn't have been created). Maybe your survey (which I didn't fill
btw) can answer this.

* Do it as long as the time invested is worth the result (ie you don't
have to delay a super feature to put in place something that is roughly
equivalent of what your users already have now).

* Do it if you feel that you can attract a different population (ie
programmers in need of a deployement system for instance) than the one
forming the actual puppet user community (which I guess is mostly
sysadmins), and as such grow your user base (which is of course your
ultimate goal to maintain a sustainable business :-))

On Feb 11, 2009, at 9:45 AM, Luke Kanies wrote:

Luke, the use of one internal DSL for in-Ruby resource creation,  
external DSL parsing, and the RAL sounds like both an excellent plan  
and an huge undertaking. I'm happy to help with the implementation of  
this in any way I can.

- Jesse

1: http://www.slideshare.net/littleidea/systems-building-systems-a-puppe...

First, it wouldn't be targeted at your regular sysadmin, it'd be  
targeted at people who already find the existing DSL too limiting,  
either because it can't iterate fast enough or because it's missing a  
lot of common language functionality.

Second, however, it allows us to keep the current DSL from acquiring  
90% of the functionality of other languages.  There's been a  
disturbing trend recently of writing functions that just import Ruby  
functionality into Puppet.  I call this 'disturbing', because Ruby is  
OO and Puppet is decidedly not, which means they mostly get imported  
as functions or new syntaxes.  This puts us on a collision course to  
either look like PHP (functions) or perl (syntaxes) in a few years,  
neither of which is pretty or maintainable.

With this internal DSL we can say, hey, you want simple stuff that  
works well and provides guarantees, use the external DSL, but if you  
want to get all complicated and want regexes and iteration and other  
(actually quite normal) stuff, use the internal DSL.

This way we don't end up with 500 functions and 650 statement types in  
a few years.

   class foo requires bar {
     ...
   }

Now, I'm not suggesting this syntax, but whatever we did, we'd be  
extending both the internal model of classes and definitions -- what  
features they have and how we can use them -- *and* adding a new  
syntax, and, of course, these would need to be done in tandem.  Every  
tweak to the model would result in a (probably not backward  
compatible) tweak to the syntax.

Instead, we can put ourselves in a situation where we can iterate the  
model in the internal DSL, not worrying about a syntax initially, and  
once we've got the model nailed, then we wrap an external syntax  
around it.  This would make it *much* faster to iterate on the model,  
and, I think, faster to provide new syntaxes.

So yeah, we're going to have module-level dependencies at some point,  
specified in a pure-data file of some kind, but I don't think they'll  
get you all the way there.

And don't forget that this also brings what I think is significant  
clarity to the whole system, and with that clarity comes both  
inspiration and greater maintainability.  Until I started thinking  
about this, I didn't realize that the definition, hostclass, and node  
AST classes *aren't* AST classes at all.  If you look in those  
classes, you'll see that I've essentially extracted every bit of AST  
behaviour from them, and any remaining is a result of lack of clarity.

So really, the next logical step is to pull them out of the AST  
hierarchy.  I've always found that clarity in the model has a huge  
beneficial affect on maintainability, and having multiple DSLs relying  
on the same backend code will definitely bring clarity to the model.

Once we've extracted them, we can start relying on them for the basis  
of an internal DSL.  Then, as mentioned, we can iterate on their model  
without changing the syntax, and once we're happy with things like  
dependencies and whatever else needs to be added, we can develop  
syntaxes that concisely express those new behaviours.

And really, if it's relatively inexpensive (and I think I've laid out  
a plan here where it is), you can essentially throw both out there and  
let the market decide.

There's what I might call a symmetry internally in Puppet:  Usage of  
classes, definitions, and nodes is modeled as resources, and these  
things in turn create more resources.

Take something like this:

   define foo {
     notify { "foo $name": }
   }

   foo { bar: }

This is two statements:  Create the 'foo' defined resource type  
(really, we should start calling them 'composite' resources, since  
they're just a wrapper around one or more resources), and create an  
instance of that resource.  The resource type itself gets put into a  
table that makes it easy to look it up by name.  The 'foo' resource  
gets put into our resource graph.

We then, via an iterative process, look up unevaluated resources in  
the graph.  This process finds the 'foo' resource and calls 'evaluate'  
on it.  This method knows how to look up the 'foo' resource type and  
call the right methods on it.  Check out  
Puppet::Parser::Resource#evaluate.

The same is true of nodes and classes, except that their resource type  
is 'Node' and 'Class', respectively, and they (currently) never accept  
arguments.

The key to pulling Puppet and ShadowPuppet together is getting  
ShadowPuppet to use a similar system:  You've either got resource  
types or classes in a table, or you've got resources in a graph that  
know how to look up and evaluate their types in that table.

The next step is to unify the instances that get stored in those tables.

I doubt I made it much clearer there, but my main point, the basic  
model is actually both simple and clear, and as long as the two tools  
use the same model, it should be straightforward to bring them together.

Regards

James

- --
Author of:
* Pulling Strings with Puppet
(http://www.amazon.com/gp/product/1590599780/)
* Pro Nagios 2.0
(http://www.amazon.com/gp/product/1590596099/)
* Hardening Linux
(http://www.amazon.com/gp/product/1590594444/)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org