RFC: Nodify: A simple external nodes app

[Luke Kanies]

Hi all,

This email is seeking feedback on a proposed replacement for the
builtin node syntax in Puppet. It proposes an external node source
that will ship with Puppet, to eventually deprecate the current 'node'
construct entirely. I'm specifically seeking feedback on the following:

* Should it be part of Puppet (vs. a separate tool)?
* Is an internal DSL (see below) acceptable?
* Is there any unnecessary functionality included?
* Is any obvious or critical functionality missing?
* If you don't want to replace the current 'node' type with this, why
not?

I've been talking for more than a year about deprecating the in-
language node definitions, instead moving to some kind of external
nodes. My intention has always been to support flexibility, allowing
people to collect their information however they want, but at the same
time it's obvious that I need to provide something as simple as the
current solution.

So, I'm thinking of writing a very simple internal DSL for defining
nodes. "Internal DSL" means that the configurations would be written
in pure Ruby, rather than my writing a separate parser and lexer.
This should do two helpful things for us: It should help m get the
tool out the door that much faster, and it should make it easier for
others to enhance and extend the app.

Because the node's job is pretty simple, this DSL will be really simple.

There are (currently) only three things we need to know about a node,
other than its name:

* The classes it's a member of
* Its parameters, including facts
* Its environment

Everything else, including things like inheritance, is basically a
question of application modeling, rather than output.

My initial pass would just do everything it could to look and act like
Puppet's current nodes, with only a couple of important behaviour
exceptions.

This is more of a "what do you think?" email; there's currently no
implementation. It's simple enough that I'll probably take a day this
week and knock it out, but the intention of this email is just to
thwack the bushes a bit and see what runs out.

Here's a basic node:

node :base do
environment 'production'
self[:location] = "bna"

member_of :one, :two, :three
end

This exercises, in a minimal form, all three of the core data types:

* Set the environment to 'production'
* Set a 'location' parameter
* Declare that the node is a member of the three listed classes

All nodes would get the node's facts from Puppet as a starting point.
In the beginning, at least, this would not be configurable.

You can easily declare multiple nodes at once, and/or with parent nodes:

node :one, "two.madstop.com", "three", :inherits => :base do
...
end

At this point, I'm planning on changing the matching rules so that all
matching nodes get evaluated, because...

I'd also support regex-named nodes:

node %r{blah} do
...
end

This way, you could have multiple node declarations that added classes
or parameters or whatever based on matching the node name against a
regex.

The only other semi-significant feature I'm planning on adding is some
ability to declare that certain classes are automatic based on
parameter settings:

node :base do
autoclass :operatingsystem, :environment, :location
end

This would automatically add a class whose name is the value of the
parameter. So, if this is a production Debian machine in Nashville
(whose airport code is 'BNA' for some reason), you'd get 'production',
'debian', and 'bna' classes automatically. This works well with the
current behaviour in Puppet's external nodes, because it will
basically ignore any unknown classes returned by an external nodes app
(although some have recommended that this behaviour change).

Lastly, it would include a new executable, 'nodify', which you could
use to inspect any node at any time.

I can easily think of lots of other interesting features to add, but
this is the core to me. Specific names and terms are obviously still
subject to change, but this is close enough. I'm hoping to take some
"fun" time tonight and hack this up, including some specs laying out
the behaviours in a useful way. I'll be publishing it in my personal
repo in a branch named 'nodify'.

Before I get started, though, I'd love some feedback on the basics.

--
Anyone who considers arithmatical methods of producing random digits
is, of course, in a state of sin. --John Von Neumann
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com

[Arjuna Christensen]

A rubyfied app for external nodes would be very cool - this would allow us (me) to build and or define nodes based on other external resources programmatically.

+1

Arjuna Christensen | Systems Engineer 
Maximum Internet Ltd
7a Parkhead Pl, Albany, North Shore, 0632 | PO Box 8006, Auckland, 1150, NZ
DDI: + 64 9 913 9683 | Ph: +64 9 915 1825 | Fax:: +64 9 300 7227
arjuna.ch...@maxnet.co.nz| www.maxnet.co.nz
________________________________
Maxnet | mission critical internet
________________________________
This email (including any attachments) is confidential and intended only for the person to whom it is addressed.
If you have received this email in error, please notify the sender immediately and erase all copies of this message
and attachments. The views expressed in this email do not necessarily reflect those held by Maxnet.

[Brian Finney]

Was a solution ever found for specifying resources directly from
external node sources.

The common example being able to say:

node 'blah' {
ip{
"eth0":
address =>'192.168.100.100'
"eth1":
address =>'192.168.200.200'

[Luke Kanies]

On Mar 31, 2008, at 4:26 PM, Brian Finney wrote:

> Was a solution ever found for specifying resources directly from
> external node sources.
>
> The common example being able to say:
>
> node 'blah' {
> ip{
> "eth0":
> address =>'192.168.100.100'
> "eth1":
> address =>'192.168.200.200'
> }
> }

This will never be possible, just because it conflates node
classification with resource specification.

However, my proposed tool would make it downright easy to do this,
basically: Just add an automatic class for the hostname, and then you
can just define a class named after the host, which the host would
then automatically include. I assume that would work for you.

--
Work is not always required. There is such a thing as sacred idleness.
-- George MacDonald

[Brian Finney]

So this tool would automatically include class $hostname_network?
Then you would have:

class bah_network{

ip{
"eth0":
address =>'192.168.100.100'
"eth1":
address =>'192.168.200.200'
}
}

class humbug_network{...}

I suppose it works, feels like it defeats the main benefits sense you
would now have a class for every node, but no one has been able to
come up with a better solution.

Works for me.

Thanks
Brian

[Brian Finney]

days almost over...

now that I've reread the original email you would have

node :base do
autoclass :hostname
end

and the classes would be
class bah {...}
class humbug {...}

any thoughts on allowing either prefix or postfix?

It's backwards compatibly enough for me, though I still wish there
where some super magical extremely elegant solution that would also
solve world hunger :-).

Thanks
Brian

[Luke Kanies]

On Mar 31, 2008, at 5:17 PM, Brian Finney wrote:

> So this tool would automatically include class $hostname_network?

Something like that; I'm not sure exactly how it'd be set up. At the
least, I'm thinking of the ability to automatically convert the value
of a parameter (like $hostname or $operatingsystem) to a class.

It should certainly be easy to just directly include a normal value:

node :mynode do
belongs_to "#{self[:hostname]}_network"
end

That gets the exact behaviour you want, with the class including both
the hostname and network. In general, this'd probably be better:

node :mynode do
belongs_to "network::#{self[:hostname]}"
end

I might try to support multiple inheritance, too, with a strict
collision model (i.e., two parent nodes couldn't set the same
parameter), but it probable makes more sense to just support some kind
of module-like mixin.

>
> Then you would have:
>
> class bah_network{
> ip{
> "eth0":
> address =>'192.168.100.100'
> "eth1":
> address =>'192.168.200.200'
> }
> }
>
> class humbug_network{...}
>
> I suppose it works, feels like it defeats the main benefits sense you
> would now have a class for every node, but no one has been able to
> come up with a better solution.

Not really. Remember that the whole point of external nodes is to
separate the problem of node classification from resource specification.

Many organizations have an essentially trivial node classification
problem -- they just statically declare a bunch of information and
walk away. For them, the current situation is mostly adequate, and
they don't mind mixing classification and resource specification.

For other organizations, though, the problems are much more
complicated, and they need to create automatic classes based on things
like hostname, location, datacenter, etc. Most companies I've worked
with included metadata in the hostnames, and they need to
automatically extract that metadata and turn it into class
membership. Also, many people want to declare parameters in a base
node and then override those parameters per-location or per-node, such
as setting a different router or dns resolver. This isn't possible
with Puppet today, because Puppet has an inheritance model that works
well for resources but not well for parameters. This new tool (and
those like it) are an opportunity to provide a more appropriate
inheritance model, and to clearly say things like "this host gets its
router by being a member of the 'chicago' class".

Hmm, that reminds me; I need some much better way to associate
parameters with class membership. Just having nodes and parent nodes
isn't sufficient.

Hmmm....

--
Millions long for immortality who do not know what to do with
themselves on a rainy Sunday afternoon. -- Susan Ertz

[James Turnbull]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Luke Kanies wrote:
| Hi all,
|
| This email is seeking feedback on a proposed replacement for the
| builtin node syntax in Puppet. It proposes an external node source
| that will ship with Puppet, to eventually deprecate the current 'node'
| construct entirely. I'm specifically seeking feedback on the following:
|

I have no issues with a replacement as such BUT... I am still of the
opinion (as you'd be aware from previous conversations) that you don't
deprecate the existing node syntax. The existing node syntax is perfect
for small-medium set-ups and is quick and easy to understand. I
recommend freezing its functionality and only focussing on the
replacement but I don't recommend doing away it for quite some time - if
ever.

Cheers

James

- --
James Turnbull (ja...@lovedthanlost.net)
- --
Author of:
- - Pulling Strings with Puppet
(http://www.amazon.com/gp/product/1590599780/)
- - Pro Nagios 2.0
(http://www.amazon.com/gp/product/1590596099/)
- - Hardening Linux
(http://www.amazon.com/gp/product/1590594444/)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH8Wst9hTGvAxC30ARAlhXAKC28QGgJuqgRmeJD3qh1s6wQhgu7gCcCS37
9eX6SYY1gvvcCfWPknGhzrA=
=5Q9E
-----END PGP SIGNATURE-----

[Brian Finney]

wouldn't aliasing node to class and giving the new tool a default config of:

node :default do
autoclass :hostname
end

provide backward compatibility to current node files?

Though actually would probably need something like

node :default do
autoclass :hostname default belongs_to "default"
end

to provide for default nodes, I'd imagine the ability to add a default
if desired could also be useful in general. Though I don't think the
external node api allows for this (if I remember correctly)

[Luke Kanies]

On Mar 31, 2008, at 5:52 PM, James Turnbull wrote:

> I have no issues with a replacement as such BUT... I am still of the
> opinion (as you'd be aware from previous conversations) that you don't
> deprecate the existing node syntax. The existing node syntax is
> perfect
> for small-medium set-ups and is quick and easy to understand. I
> recommend freezing its functionality and only focussing on the
> replacement but I don't recommend doing away it for quite some time
> - if
> ever.

The problem is that it's actually not perfect. It seems like nearly
every new Puppet user tries to set parameters in the parent node and
override them in the subnode, and that just doesn't work.

It *might* be possible to modify node behaviour so that this worked,
but in that case, there's just no need for a simple external nodes tool.

IMO, either we build something like this nodify, or we keep the
current node syntax; I don't think it's reasonable to do both.

--
The leader of Jamestown was "John Smith" (not his real name), under
whose direction the colony engaged in a number of activities,
primarily related to starving. -- Dave Barry, "Dave Barry Slept Here"

[Luke Kanies]

On Mar 31, 2008, at 6:06 PM, Brian Finney wrote:

> wouldn't aliasing node to class and giving the new tool a default
> config of:
>
> node :default do
> autoclass :hostname
> end
>
> provide backward compatibility to current node files?

Yes, the tool would have support for a default node, just like the
current system.

>
> Though actually would probably need something like
>
> node :default do
> autoclass :hostname default belongs_to "default"
> end

These behaviours don't exist right now, so I wouldn't add them. I
expect that over time the community would find a certain list of
automatic class memberships would make sense, which could then be added.

>
> to provide for default nodes, I'd imagine the ability to add a default
> if desired could also be useful in general. Though I don't think the
> external node api allows for this (if I remember correctly)

Anyone using an external node app is expected to implement their own
support for defaults and inheritance. This way Puppet isn't enforcing
its model on other applications.

--
The real art of conversation is not only to say the right thing at the
right place but to leave unsaid the wrong thing at the tempting
moment. -- Dorothy Nevill

[James Turnbull]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Luke Kanies wrote:
| IMO, either we build something like this nodify, or we keep the
| current node syntax; I don't think it's reasonable to do both.

Why am I so tempted to tell you to just keep the existing node syntax...

James

- --
James Turnbull (ja...@lovedthanlost.net)
- --
Author of:
- - Pulling Strings with Puppet
(http://www.amazon.com/gp/product/1590599780/)
- - Pro Nagios 2.0
(http://www.amazon.com/gp/product/1590596099/)
- - Hardening Linux
(http://www.amazon.com/gp/product/1590594444/)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH8gXw9hTGvAxC30ARAmozAJoDVmdqy1LgD6Fa1Gwg2BrSQJznngCeLPdJ
D5/hs+4Xo9bDKBWclz9QtA4=
=PASP
-----END PGP SIGNATURE-----

[Mathieu Sauve-Frankel]

> I have no issues with a replacement as such BUT... I am still of the
> opinion (as you'd be aware from previous conversations) that you don't
> deprecate the existing node syntax. The existing node syntax is perfect
> for small-medium set-ups and is quick and easy to understand. I
> recommend freezing its functionality and only focussing on the
> replacement but I don't recommend doing away it for quite some time - if
> ever.

I couldn't possibly agree more with James here. The current syntax is not
broken, please don't "fix" it

--
Mathieu Sauve-Frankel

[Luke Kanies]

But... it is broken. You don't quite realize how many people show up
looking for different behaviour, and get pretty upset when nodes don't
behave like they expect, with the ability to override parameters set
in parent nodes.

I guess I should put it to the wider community and see what others
think, though.

I'm still kinda leaning away from keeping this syntax, at least in the
form it's in. I know that makes me more tyrant than community leader,
but I may just push it this one time.

--
A child can go only so far in life without potty training. It is not
mere coincidence that six of the last seven presidents were potty
trained, not to mention nearly half of the nation's state legislators.
-- Dave Barry

[Matthew malthouse]

On 01/04/2008, Luke Kanies <lu...@madstop.com> wrote:

I'm still kinda leaning away from keeping this syntax, at least in the
form it's in.  I know that makes me more tyrant than community leader,
but I may just push it this one time.

I'd not dispute the brokenness, if you say it's broke then broke it is Mr George Louis Tyrant.

But when I think of the work that the change will require against the advantages to be had I find myself flicking through the diary to 4th and 14th July.

Matthew

[Steven Jenkins]

How hard would it be to create a tool to convert from the existing
format to the new-yet-to-be-defined format? That's a route I'd
suggest.

Steven

[Adam Jacob]

On Mon, Mar 31, 2008 at 1:46 PM, Luke Kanies <lu...@madstop.com> wrote:
> Here's a basic node:
>
> node :base do
> environment 'production'
> self[:location] = "bna"
>
> member_of :one, :two, :three
> end

Can I recommend you do this instead:

node "base" do
environment 'production'

location "bna"

member_of :one, :two, :three
end

Essentially, have a method_missing that sets up the attributes for
you, masquerading the "self[:foo]" stuff. If you hate that, then:

node "base" do
environment 'production'

set "location", "bna"

member_of :one, :two, :three
end

I would be careful with using symbols in the node definition, since
you won't be able to use the most common node setting (a hostname or
fqdn) in all situations. Support it, but make sure people understand
they can just use strings there all the time.

I'm also in the "don't deprecate the regular node syntax" camp, since
the rest of puppet's language isn't a ruby pidgin. If you're going to
say that people ought to use the declarative puppet language, I think
it makes sense to let them use only that.

Regards,
Adam

--
HJK Solutions - We Launch Startups - http://www.hjksolutions.com
Adam Jacob, Senior Partner
T: (206) 508-4759 E: ad...@hjksolutions.com

[Blake Barnett]

I think I'm with Steven on this one, I say kill the current node
syntax, but have a tool available to convert existing nodes to The New
Way.

-Blake

[Luke Kanies]

On Apr 1, 2008, at 12:34 PM, Blake Barnett wrote:

> I think I'm with Steven on this one, I say kill the current node
> syntax, but have a tool available to convert existing nodes to The New
> Way.

If I did go through with the separate app, I would add this converter.

--
If all the world's a stage, I want to operate the trap door.
-- Paul Beatty

[Luke Kanies]

On Apr 1, 2008, at 12:19 PM, Adam Jacob wrote:

>
> On Mon, Mar 31, 2008 at 1:46 PM, Luke Kanies <lu...@madstop.com> wrote:
>> Here's a basic node:
>>
>> node :base do
>> environment 'production'
>> self[:location] = "bna"
>>
>> member_of :one, :two, :three
>> end
>
> Can I recommend you do this instead:
>
> node "base" do
> environment 'production'
> location "bna"
>
> member_of :one, :two, :three
> end
>
> Essentially, have a method_missing that sets up the attributes for
> you, masquerading the "self[:foo]" stuff. If you hate that, then:
>
> node "base" do
> environment 'production'
>
> set "location", "bna"
>
> member_of :one, :two, :three
> end

I've thought about both of these approaches. I generally dislike
using method_missing, as it gets messy quickly. I'm undecided on a
set/get method; certainly it would be easy to provide both that and
the :[] method.

>
> I would be careful with using symbols in the node definition, since
> you won't be able to use the most common node setting (a hostname or
> fqdn) in all situations. Support it, but make sure people understand
> they can just use strings there all the time.

Definitely.

>
> I'm also in the "don't deprecate the regular node syntax" camp, since
> the rest of puppet's language isn't a ruby pidgin. If you're going to
> say that people ought to use the declarative puppet language, I think
> it makes sense to let them use only that.

It depends on the extent to which I want people to think of this as
the same app as Puppet or a different.

One of my primary goals here is to force a recognition that node
classification and the related inventory mgmt tasks are a separate
problem deserving of a separate application. The more people believe
Puppet should do this, plus whatever wacky stuff comes along in the
future, the harder it's going to be to build small, maintainable,
stand-alone tools, which has always been my goal.

It really comes down to a philosophical shift. I want node
classification to be a separate problem, solved by a separate
application. It's a change, but I think it's an important change. I
want it to be obvious to people that they get to choose between
competing tools for this problem, not between "the Puppet way" and
some other way. The Puppet way involves using something else to solve
this problem.

--
A government that robs Peter to pay Paul can always depend on the
support of Paul. -- George Bernard Shaw

[Luke Kanies]

On Apr 1, 2008, at 10:36 AM, Matthew malthouse wrote:

> I'd not dispute the brokenness, if you say it's broke then broke it
> is Mr George Louis Tyrant.
>
> But when I think of the work that the change will require against
> the advantages to be had I find myself flicking through the diary to
> 4th and 14th July.

Hey, I'm not taxing you, just making your tools better. I'm an
enlightened despot, really. :)

I'm just trying to think of a way to make this work (keeping the
existing nodes around), and I just can't. You can't freeze code
without deprecating it, really, and if Puppet retains this
functionality, then I'll always be fighting the perception that it's
the supported way to do things.

I'm still convinced the best way is to provide a simple, relatively
stand-alone application that replaces the current node functionality,
with a converter.

--
It's a small world, but I wouldn't want to paint it.
-- Stephen Wright

[Adam Jacob]

On Tue, Apr 1, 2008 at 11:58 AM, Luke Kanies <lu...@madstop.com> wrote:
> > node "base" do
> > environment 'production'
> > location "bna"
> >
> > member_of :one, :two, :three
> > end
>

> I've thought about both of these approaches. I generally dislike
> using method_missing, as it gets messy quickly. I'm undecided on a
> set/get method; certainly it would be easy to provide both that and
> the :[] method.

It might not be so bad if you actually have the call to:

node "foo" do
..
end

Instantiate a new Node object, then do instance_eval on the block you
passed to it. That would solve a lot of the outside pollution you get
with method_missing in the raw pidgin interpreter. I only harp on it
because I think:

node "foo" do
something "value"
end

Is a lot easier on the eyes than:

node "foo" do
self["something"] = "value"
end

In the method_missing case, you'll still have to provide some
mechanism for people to deal with reserved words..

node "foo" do
attribute["method_of"] = "value"
end

In fact, having attribute be an accessor to the same hash you would
have kept in self[] might be a good middle way.

> It depends on the extent to which I want people to think of this as
> the same app as Puppet or a different.
>
> One of my primary goals here is to force a recognition that node
> classification and the related inventory mgmt tasks are a separate
> problem deserving of a separate application. The more people believe
> Puppet should do this, plus whatever wacky stuff comes along in the
> future, the harder it's going to be to build small, maintainable,
> stand-alone tools, which has always been my goal.
>
> It really comes down to a philosophical shift. I want node
> classification to be a separate problem, solved by a separate
> application. It's a change, but I think it's an important change. I
> want it to be obvious to people that they get to choose between
> competing tools for this problem, not between "the Puppet way" and
> some other way. The Puppet way involves using something else to solve
> this problem.

Hey, I'm in the same philosophical camp. Puppet as a tool needs node
classification, though, so the path needs to be really explicit.

[David Lutterkort]

On Tue, 2008-04-01 at 09:59 -0500, Luke Kanies wrote:
> > I couldn't possibly agree more with James here. The current syntax
> > is not
> > broken, please don't "fix" it

> I'm still kinda leaning away from keeping this syntax, at least in the

> form it's in. I know that makes me more tyrant than community leader,
> but I may just push it this one time.

Seeing how nodify isn't implemented yet, isn't this a bit premature ? I
think, too, that the current syntax should stay there as an option as
long as possible. It doesn't sound like keeping it will cause any
trouble for implementing/using nodify.

Once nodify has been in use for a while, we'd be in a much better
position to judge how useful the old syntax still is.

David

[Luke Kanies]

That does make the most sense -- get nodify working and relatively
stable, then look at deprecation.

I don't think I'd announce nodify and the deprecation in the same
release; that would be too much. I *do* think I'd do my best to hang
the spectre of deprecation over nodes, but it wouldn't be quite there
yet.

--
Get forgiveness now -- tomorrow you may no longer feel guilty.

[David Lutterkort]

On Mon, 2008-03-31 at 20:19 -0500, Luke Kanies wrote:
> The problem is that it's actually not perfect. It seems like nearly
> every new Puppet user tries to set parameters in the parent node and
> override them in the subnode, and that just doesn't work.

But that's an implementation problem, not a syntax problem. It seems to
me the main arguments for nodify are (or should be IMHO)
* make it possible to access data sources that you can't get to
from the puppet language to do node classification
* allow much more complex computations for node classification
than what you can do in puppet's language
* separate the classification functionality out from puppet for
others to use (though I am a little hazy on how that would work
in practice)

David

[Luke Kanies]

On Apr 1, 2008, at 3:20 PM, Adam Jacob wrote:

>
> On Tue, Apr 1, 2008 at 11:58 AM, Luke Kanies <lu...@madstop.com> wrote:
>>> node "base" do
>>> environment 'production'
>>> location "bna"
>>>
>>> member_of :one, :two, :three
>>> end
>>
>> I've thought about both of these approaches. I generally dislike
>> using method_missing, as it gets messy quickly. I'm undecided on a
>> set/get method; certainly it would be easy to provide both that and
>> the :[] method.
>
> It might not be so bad if you actually have the call to:
>
> node "foo" do
> ..
> end
>
> Instantiate a new Node object, then do instance_eval on the block you
> passed to it. That would solve a lot of the outside pollution you get
> with method_missing in the raw pidgin interpreter. I only harp on it
> because I think:

This is definitely how I was planning on doing it (although I'm most
likely going to use the block to dynamically define a method on each
node, since most nodes would be able to run multiple times, so you
weren't reparsing the file for every new node configuration).

>
> node "foo" do
> something "value"
> end
>
> Is a lot easier on the eyes than:
>
> node "foo" do
> self["something"] = "value"
> end
>
> In the method_missing case, you'll still have to provide some
> mechanism for people to deal with reserved words..
>
> node "foo" do
> attribute["method_of"] = "value"
> end
>
> In fact, having attribute be an accessor to the same hash you would
> have kept in self[] might be a good middle way.

Yeah, and it does make thread safety a bit easier than 'self[]', which
is a concern.

> Hey, I'm in the same philosophical camp. Puppet as a tool needs node
> classification, though, so the path needs to be really explicit.

That I certainly agree with; uncertainty is deadly.

--
It is absurd to divide people into good and bad. People are either
charming or tedious. -- Oscar Wilde

[Luke Kanies]

On Apr 1, 2008, at 4:27 PM, David Lutterkort wrote:
>
> On Mon, 2008-03-31 at 20:19 -0500, Luke Kanies wrote:
>> The problem is that it's actually not perfect. It seems like nearly
>> every new Puppet user tries to set parameters in the parent node and
>> override them in the subnode, and that just doesn't work.
>
> But that's an implementation problem, not a syntax problem.

Well, it's more semantics than either (but, of course, semantics are
defined by the implementation). People expect node inheritance to
work differently than class inheritance does. I don't think it's
reasonable to have two inheritance semantics in the same tool.

> It seems to
> me the main arguments for nodify are (or should be IMHO)
> * make it possible to access data sources that you can't get to
> from the puppet language to do node classification
> * allow much more complex computations for node classification
> than what you can do in puppet's language
> * separate the classification functionality out from puppet for
> others to use (though I am a little hazy on how that would work
> in practice)

I think these are mostly general external node features. Nodify's job
is just to be the simplest fully-functional external node application,
and thus the default one that everyone starts with.

It has some additional benefits because of how we'd be implementing it
(regexes for name matching, and ease of compiling node information on
demand without handling the catalog at all), but those benefits aren't
really why I want to create it.

To me, nodify's entire reason for existence is to move *everyone*
toward using external nodes.

--
SELF-EVIDENT, adj. Evident to one's self and to nobody else.
-- Ambrose Bierce

[David Lutterkort]

On Mon, 2008-03-31 at 15:46 -0500, Luke Kanies wrote:
> * Should it be part of Puppet (vs. a separate tool)?

I don't see much of a gain from tying it closely to Puppet - of course,
you'd probably want it on every puppetmaster, but that's more a
packaging issue.

At the same time, I am not sure how much it will help other tools;
probably best to keep it separate from puppet implementation-wise, but
narrowly focused on puppet's needs until some other tool wants to use it
and has other needs.

> * Is an internal DSL (see below) acceptable?

Seems to make the most sense, since one of the drivers for this is to
enable classification based on all kinds of 'stuff' that you'd need a
full programming language to get to.

> node :base do
> environment 'production'

> self[:location] = "bna"

>
> member_of :one, :two, :three
> end

Echoing what others have said, I'd prefer it if parameters weren't set
using 'self[]' ... something like 'parameters[:location]' or even
'parameter :location "bna"' seems cleaner.

David

[Luke Kanies]

On Apr 1, 2008, at 4:37 PM, David Lutterkort wrote:
>
> On Mon, 2008-03-31 at 15:46 -0500, Luke Kanies wrote:
>> * Should it be part of Puppet (vs. a separate tool)?
>
> I don't see much of a gain from tying it closely to Puppet - of
> course,
> you'd probably want it on every puppetmaster, but that's more a
> packaging issue.

I guess that's kind of my question, though -- if I just include it
with Puppet, the packaging is easy. If I don't, then each packager
will need to package an additional application.

Really, though, it seems that most people are using packages for
Puppet, so we can rely on the package dependencies to bring in nodify
with puppet-server. It's a higher packaging burden, but not a lot.

>
> At the same time, I am not sure how much it will help other tools;
> probably best to keep it separate from puppet implementation-wise, but
> narrowly focused on puppet's needs until some other tool wants to
> use it
> and has other needs.

So maybe it should be a separate tool in a separate repo, like Facter?

>
>> * Is an internal DSL (see below) acceptable?
>
> Seems to make the most sense, since one of the drivers for this is to
> enable classification based on all kinds of 'stuff' that you'd need a
> full programming language to get to.

Okay.

>
>> node :base do
>> environment 'production'
>> self[:location] = "bna"
>>
>> member_of :one, :two, :three
>> end
>
> Echoing what others have said, I'd prefer it if parameters weren't set
> using 'self[]' ... something like 'parameters[:location]' or even
> 'parameter :location "bna"' seems cleaner.

Alright, sounds like the self[] is out. I'll probably pick something
like parameters[] or set :param, "value", depending on which is
shorter and/or makes more sense. It'll always be easy to change, at
least until people start using it. :)

--
A censor is a man who knows more than he thinks you ought to.
-- Granville Hicks

[Paul Lathrop]

Hi Luke,

Comments inline:

On Mon, Mar 31, 2008 at 1:46 PM, Luke Kanies <lu...@madstop.com> wrote:
> * Should it be part of Puppet (vs. a separate tool)?

I think it should be a separate tool, to help maintain the logical
separation. That said, once internal nodes are deprecated (should you
choose to do so), puppetmaster should 'depend' on this, to ensure that
there is *something* available in a default install.

> * Is an internal DSL (see below) acceptable?

An internal DSL is acceptable to me.

> * Is there any unnecessary functionality included?
> * Is any obvious or critical functionality missing?

As long as it entirely duplicates the functionality of 'node', I would
be fine leaving other 'features' to further development (or even tell
folks to roll their own as is done now).

> * If you don't want to replace the current 'node' type with this, why
> not?

Let me echo others in saying that if you choose to replace the 'node'
collection type, you should do so at *least* one release after nodify.
I would envision a several-release deprecation cycle to ensure nodify
can cleanly replace 'node'. After that, I'm happy to see 'node'
relegated to history. Keep in mind, however, that since 0.23.x is
immortalized in Debian Etch, and they'll probably be horribly behind
when the next release hits as well, that we'll be fielding questions
about 'node' for a LONG time.

> Here's a basic node:
>

> node :base do
> environment 'production'
> self[:location] = "bna"
>
> member_of :one, :two, :three
> end

I want to echo other folks' dislike of the self[] syntax here.

> At this point, I'm planning on changing the matching rules so that all
> matching nodes get evaluated, because...
>
> I'd also support regex-named nodes:
>
> node %r{blah} do
> ...
> end

Awesome.

> The only other semi-significant feature I'm planning on adding is some
> ability to declare that certain classes are automatic based on
> parameter settings:
>
> node :base do
> autoclass :operatingsystem, :environment, :location
> end

Also awesome.

Regards,
Paul

[Luke Kanies]

On Apr 1, 2008, at 5:13 PM, Paul Lathrop wrote:

>
> Hi Luke,
>
> Comments inline:
>
> On Mon, Mar 31, 2008 at 1:46 PM, Luke Kanies <lu...@madstop.com> wrote:
>> * Should it be part of Puppet (vs. a separate tool)?
>
> I think it should be a separate tool, to help maintain the logical
> separation. That said, once internal nodes are deprecated (should you
> choose to do so), puppetmaster should 'depend' on this, to ensure that
> there is *something* available in a default install.

Yep, I agree. And, obviously, good errors when it is missing
somehow. :)

> As long as it entirely duplicates the functionality of 'node', I would
> be fine leaving other 'features' to further development (or even tell
> folks to roll their own as is done now).

I think it will make sense to develop nodify over the years, but it
hopefully won't need much functionality.

>
>> * If you don't want to replace the current 'node' type with this, why
>> not?
>
> Let me echo others in saying that if you choose to replace the 'node'
> collection type, you should do so at *least* one release after nodify.
> I would envision a several-release deprecation cycle to ensure nodify
> can cleanly replace 'node'. After that, I'm happy to see 'node'
> relegated to history. Keep in mind, however, that since 0.23.x is
> immortalized in Debian Etch, and they'll probably be horribly behind
> when the next release hits as well, that we'll be fielding questions
> about 'node' for a LONG time.

Yeah, sorry; I clearly didn't make it clear that deprecation would be
gradual.

And I fully understand how long we'll be supporting Debian people,
unfortunately.

> I want to echo other folks' dislike of the self[] syntax here.

Ok.

--
To succeed in the world it is not enough to be stupid, you must also
be well-mannered. -- Voltaire

[Mathieu Sauve-Frankel]

On Tue, Apr 01, 2008 at 04:27:28PM -0500, Luke Kanies wrote:
>
> On Apr 1, 2008, at 4:22 PM, David Lutterkort wrote:
> >
> > On Tue, 2008-04-01 at 09:59 -0500, Luke Kanies wrote:
> >
> >> I'm still kinda leaning away from keeping this syntax, at least in
> >> the
> >> form it's in. I know that makes me more tyrant than community
> >> leader,
> >> but I may just push it this one time.
> >
> > Seeing how nodify isn't implemented yet, isn't this a bit
> > premature ? I
> > think, too, that the current syntax should stay there as an option as
> > long as possible. It doesn't sound like keeping it will cause any
> > trouble for implementing/using nodify.
> >
> > Once nodify has been in use for a while, we'd be in a much better
> > position to judge how useful the old syntax still is.
>
>
> That does make the most sense -- get nodify working and relatively
> stable, then look at deprecation.

Sorry for the tone of my mail yesterday, but this is really the only
kind of assurance that I am looking for.

The current model certainly has issues, and I am not arguing against that.
Specifically I am concerned about replacing something that is stable with
known limitations for something experimental and unproven.

We are heavily investing in puppet at work and will have over 200 nodes
using it by the end of this year. We are quite concerned with
stability at this point. We are still running 0.22.4 in prod and
are currently planning deploying 0.24.4 in our new environment.

We have not felt comfortable deploying any of the other release up until
now.

--
Mathieu Sauve-Frankel

[Luke Kanies]

On Apr 1, 2008, at 10:17 PM, Mathieu Sauve-Frankel wrote:

> Sorry for the tone of my mail yesterday, but this is really the only
> kind of assurance that I am looking for.
>
> The current model certainly has issues, and I am not arguing against
> that.
> Specifically I am concerned about replacing something that is stable
> with
> known limitations for something experimental and unproven.

I completely understand.

>
> We are heavily investing in puppet at work and will have over 200
> nodes
> using it by the end of this year. We are quite concerned with
> stability at this point. We are still running 0.22.4 in prod and
> are currently planning deploying 0.24.4 in our new environment.
>
> We have not felt comfortable deploying any of the other release up
> until
> now.

I think this is reasonable. Any project with as small a signficant
developer base as Puppet currently has, especially with as large a
scope as it has, should encourage conservative deployment.

I hope it's obvious that I'm working hard at enhancing stability, and
at getting more involvement from others, which also helps considerably.

--
To be positive: To be mistaken at the top of one's voice.
-- Ambrose Bierce

[David Schmitt]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wednesday 02 April 2008, Paul Lathrop wrote:
> Hi Luke,
>
> Comments inline:
>
> On Mon, Mar 31, 2008 at 1:46 PM, Luke Kanies <lu...@madstop.com> wrote:
> > * Should it be part of Puppet (vs. a separate tool)?
>
> I think it should be a separate tool, to help maintain the logical
> separation. That said, once internal nodes are deprecated (should you
> choose to do so), puppetmaster should 'depend' on this, to ensure that
> there is *something* available in a default install.

+1

> > * Is an internal DSL (see below) acceptable?
>
> An internal DSL is acceptable to me.

+1

> > * Is there any unnecessary functionality included?
> > * Is any obvious or critical functionality missing?
>
> As long as it entirely duplicates the functionality of 'node', I would
> be fine leaving other 'features' to further development (or even tell
> folks to roll their own as is done now).

One recurring theme I met with all my (two: hosting, campus infrastructure)
puppet projects is the need to configure specific resources for hosts, e.g.
ip adresses as already mentioned or things like "put this $web_app on this
$path in this $virtual_host" where the info is coming from sources which
cannot write manifests directly (c.f. customers)

> > * If you don't want to replace the current 'node' type with this, why
> > not?
>
> Let me echo others in saying that if you choose to replace the 'node'
> collection type, you should do so at *least* one release after nodify.
> I would envision a several-release deprecation cycle to ensure nodify
> can cleanly replace 'node'. After that, I'm happy to see 'node'
> relegated to history. Keep in mind, however, that since 0.23.x is
> immortalized in Debian Etch, and they'll probably be horribly behind
> when the next release hits as well, that we'll be fielding questions
> about 'node' for a LONG time.

Etch has 0.20.1 and lenny has currently 0.24.1 and will receive 0.24.4 in 6
days if nothing untoward happens.

http://packages.debian.org/puppet
http://packages.qa.debian.org/p/puppet.html -> "Testing Status"

> > At this point, I'm planning on changing the matching rules so that all
> > matching nodes get evaluated, because...
> >
> > I'd also support regex-named nodes:
> >
> > node %r{blah} do
> > ...
> > end
>
> Awesome.

+1

> > The only other semi-significant feature I'm planning on adding is some
> > ability to declare that certain classes are automatic based on
> > parameter settings:
> >
> > node :base do
> > autoclass :operatingsystem, :environment, :location
> > end
>
> Also awesome.

+1

Regards, DavidS

- --
The primary freedom of open source is not the freedom from cost, but the free-
dom to shape software to do what you want. This freedom is /never/ exercised
without cost, but is available /at all/ only by accepting the very different
costs associated with open source, costs not in money, but in time and effort.
- -- http://www.schierer.org/~luke/log/20070710-1129/on-forks-and-forking
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFH86qh/Pp1N6Uzh0URArURAJ9WPyJdLywpg9rQPZ6TawL8OthNeQCdHl6x
Al7Hlc+adxmUAriXyomg0c8=
=X777
-----END PGP SIGNATURE-----

[jtimberman]

On Mar 31, 4:52 pm, James Turnbull <ja...@lovedthanlost.net> wrote:
> I have no issues with a replacement as such BUT... I am still of the
> opinion (as you'd be aware from previous conversations) that you don't
> deprecate the existing node syntax. The existing node syntax is perfect
> for small-medium set-ups and is quick and easy to understand. I
> recommend freezing its functionality and only focussing on the
> replacement but I don't recommend doing away it for quite some time - if
> ever.

I second this.

We're using the current node { } syntax internal to puppet, not
external node classification. It suits our needs just fine and I see
no reason to change. We do have a complex environment, but I've worked
pretty hard over the last several months building our configuration to
make it easy to use and deploy.

External node classification or an external tool complicates this more
than necessary for our configuration.