Definitions in External Nodes

[Douglas Garstang]

Anyone know if there's a plan to allow definitions to be used in
external nodes? Not having that ability is a major pain in the ass. It
really means that all that really works with external nodes is really
simple cases of a single piece of software being installed and
configured, like apache for example.

I have a number of web packages that I need to install on a system.
Using external nodes, I have to explicitly set everything as a
variable, and then have a specialised class to handle that, which
means that each time I add a new customer, I have to add a new class.
Uhm... new logic to handle new customers.... not very scalable.

Doug.

[Rob McBroom]

On Aug 26, 2010, at 12:58 PM, Douglas Garstang wrote:

> Anyone know if there's a plan to allow definitions to be used in
> external nodes? Not having that ability is a major pain in the ass. It
> really means that all that really works with external nodes is really
> simple cases of a single piece of software being installed and
> configured, like apache for example.

I've used external nodes all along and never found this to be a limitation. Classes can do much much more than one thing, so I'm not sure what you mean.

External nodes can have classes and variables. Classes can call definitions. Facts and variables can be passed to those definitions.

What additional functionality would you get?

--
Rob McBroom
<http://www.skurfer.com/>

[Douglas Garstang]

I mean that you need to create a new class for each customer you add.
That does not scale.

Doug.

[Nigel Kersten]

Why can't you have a class that instantiates your defined type?

>
> Doug.
>
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>
>

--
nigel

[Thomas Bellman]

Nigel Kersten wrote:

> On Sat, Aug 28, 2010 at 9:04 AM, Douglas Garstang
> <doug.g...@gmail.com> wrote:

>> I mean that you need to create a new class for each customer you add.
>> That does not scale.
>
> Why can't you have a class that instantiates your defined type?

I don't know Douglas' circumstances, but assume you have a web
hotel. You then probably have a database of your customers,
telling you things like which domain(s) they have, what features
they pay for (static pages only, CGI scripts, mod_python,
database access, and so on), and how much storage and bandwidth
they are paying for.

To map all these customers onto your physical machines, you write
a Puppet define that is used something like this:

virtual_web_host {
"www.example.com":
avg_bandwidth => 4096, # Mbyte/month
peak_bandwidth => 2000, # Kbit/s
storage => 1536, # Mbyte disk space
cgi => true,
mod_python => false,
postgresql => true;
}

Since you have your customers in a database, it would be nice to
use an external nodes script to basically generate one such
resource for each customer domain, and it will probably use some
information in the database to balance the virtual hosts over the
physical machines. But external nodes scripts *can't* generate
resources. You of course don't want to write a new class for
each customer, because if you are a largish web hotel you may get
dozens of new customers per day (and possibly loose as many), so
that would be too much work.

But external node classifiers *can't* generate resources. They
can only use resources that already exist in the manifests, and
affect them by setting variables.

There is a way to work around this. You can let the external
node classifier set a variable with a value like this:

$vhosts = [
"dom=www.example.com|avgbw=4096|peakbw=2000|storage=1536|cgi=true|...",
"dom=www.example.org|avgbw=65536|peakbw=512|storage=600|cgi=false|...",
"dom=www.example.net|avgbw=512|peakbw=100|storage=2500|cgi=true|...",
...
]

(although the script needs to output it using YAML syntax, of
course), and tell Puppet to include a class "all-vhosts". The
manifest files would contain code looking something like this:

class all-vhosts
{
ext_virtual_web_host {
$vhosts:
;
}
}

define ext_virtual_web_host()
{
$domain = extract_field('dom', $name)
$avgbw = extract_field('avgbw', $name)
$peakbw = extract_field('peakbw', $name)
$storage = extract_field('storage', $name)
$cgi = extract_field('cgi', $name)
$mod_python = extract_field('mod_python', $name)
$postgresql = extract_field('postgresql', $name)

virtual_web_host {
$domain:
avg_bandwidth => $avgbw,
peak_bandwidth => $peakbw,
storage => $storage,
cgi => $cgi,
mod_python => $mod_python,
postgresql => $postgresql;
}
}

And then a custom function extract_field() for parsing the
strings. Doable, but not very pretty...

Those strings are of course a kind of dictionary/hash/mapping,
and Puppet 2.6 has support for dictionaries directly. But that
would only help you if 2.6 allows you to have a dictionary as the
title/name for a resource; I haven't checked, but I would guess
that you can only use strings for titles/names.

It would be much easier if external node classifiers could define
resources directly, instead of having to do something like the
above. But I seem to remember Luke saying that they are planning
to add that functionality sometime in the future; I don't think
they have set a timeline for it yet, though

[Douglas Garstang]

Thomas, thanks for the detailed description. Nigel, Thomas pretty much
nailed the issue on the head. It's not practical to add a new class
every time you add a customer. The extract_field customer function
doesn't exist yet, and I'm not a ruby programmer, so it won't exist
any time soon. Even if it did, it's an ugly hack.

Doug.

[Jeff McCune]

On Thu, Aug 26, 2010 at 9:58 AM, Douglas Garstang
<doug.g...@gmail.com> wrote:
> Anyone know if there's a plan to allow definitions to be used in
> external nodes? Not having that ability is a major pain in the ass. It
> really means that all that really works with external nodes is really
> simple cases of a single piece of software being installed and
> configured, like apache for example.

This is feature [#2408]. Please watch the ticket and paste your use
cases there.

http://projects.puppetlabs.com/issues/2408

Thanks,
-Jeff

[Nigel Kersten]

This is much clearer now.

Can you not solve this problem with extlookup now rather than having
to write custom functions? Either key the relevant data off a base
variable, or set the key to lookup info for in the external node
provider?

I like this approach because it keeps the data and the model nicely
separated, rather than whacking data into the external node provider.

[Nigel Kersten]

On Mon, Aug 30, 2010 at 5:35 PM, Alan Sparks <asp...@doublesparks.net> wrote:

>  On 8/30/2010 9:44 AM, Nigel Kersten wrote:
>> This is much clearer now.
>>
>> Can you not solve this problem with extlookup now rather than having
>> to write custom functions?  Either key the relevant data off a base
>> variable, or set the key to lookup info for in the external node
>> provider?
>>
>> I like this approach because it keeps the data and the model nicely
>> separated, rather than whacking data into the external node provider.
>>
>

> If the extlookup() function supported other than just CSV files now, it
> would be more attractive.  My instance, most of this information has to
> be derived in the external node classifier, this would entail some sort
> of periodic extraction to generate the extra data source.  Besides,
> isn't the node classifier intended to provide the driving data (e.g.,
> parameters) for the models?  As well as calling out the nodes classes?

I'm not disagreeing with you. It could be better, just pointing out an option.

I have custom functions for pulling data out of yaml files, and
strongly prefer to keep that data separate to the actual code in my
node classifier, so it doesn't really matter to me whether the data is
directly interrogated by the node classifier, or pulled out by
functions inside manifests based upon a key that the node classifier
sets.

> -Alan

[Alan Sparks]

On 8/30/2010 9:44 AM, Nigel Kersten wrote:

> This is much clearer now.
>
> Can you not solve this problem with extlookup now rather than having
> to write custom functions? Either key the relevant data off a base
> variable, or set the key to lookup info for in the external node
> provider?
>
> I like this approach because it keeps the data and the model nicely
> separated, rather than whacking data into the external node provider.
>

If the extlookup() function supported other than just CSV files now, it

would be more attractive. My instance, most of this information has to
be derived in the external node classifier, this would entail some sort
of periodic extraction to generate the extra data source. Besides,
isn't the node classifier intended to provide the driving data (e.g.,
parameters) for the models? As well as calling out the nodes classes?

-Alan

[James Turnbull]

Alan Sparks wrote:
>
> If the extlookup() function supported other than just CSV files now, it
> would be more attractive. My instance, most of this information has to
> be derived in the external node classifier, this would entail some sort
> of periodic extraction to generate the extra data source. Besides,
> isn't the node classifier intended to provide the driving data (e.g.,
> parameters) for the models? As well as calling out the nodes classes?
> -Alan
>
>

On the roadmap for extlookup are JSON and YAML inputs.

That's targeted at 2.7.

Regards

James Turnbull

--
Puppet Labs - http://www.puppetlabs.com
C: 503-734-8571