Systems Provisioning

[Douglas Garstang]

I'm wondering what people are doing systems provisioning with, ie the
process that gets puppet installed onto a system, running for the
first time, and also the handling of certificate signing and so forth.
I don't see this topic discussed much.

The mc-provision tools at
https://github.com/ripienaar/mcollective-server-provisioner don't seem
to be actively developed anymore, or at least I wasn't able to find
enough documentation to be able to effectively make use of it.

Doug

[James A. Peltier]

We have some custom written tools that look after management of our systems since we're integrating with a very large Active Directory in an environment that requires us to pre-stage machines into certain locations. This tool generates a puppet manifest that manages the generation and removal of puppet keys from a central mirror server that all of our clients fetch as part of our post configuration.

Essentially, the hosts ensure that they have a proper FQDN, then they fetch the puppet certificates from the mirror server and proceed to run puppet and configure themselves.

You can also use other tools like The Foreman to provision machines. MCollective is certainly actively developed but perhaps that module is not.

--
James A. Peltier
Manager, IT Services - Research Computing Group
Simon Fraser University - Burnaby Campus
Phone : 778-782-6573
Fax : 778-782-3045
E-Mail : jpel...@sfu.ca
Website : http://www.sfu.ca/itservices
http://blogs.sfu.ca/people/jpeltier

Success is to be measured not so much by the position that one has reached
in life but as by the obstacles they have overcome. - Booker T. Washington

[Dick Davies]

We use Cobbler for our Centos/RHEL kickstarts (around 150+ nodes, mainly VMs)
and a very simple .ks that adds puppetd. on first boot it starts up
and we manually
sign the CSR on the puppet master.

Historically, we didn't want tight integration between provisioning and CM as we
were new to both and if one turned out to be a lemon we didn't want to have to
abandon both.

We only provision a few nodes a day max. so that works ok. Lately I've been
thinking of replacing Cobbler with The Foreman, which has better
Puppet integration.
(our team are happy with editing nodes.pp and Hiera and have never
really needed anything
like advanced storeconfig etc. but we build a lot of servers for
external teams and being
able to expose config options etc. is the real driver for Foreman).

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>

[Dom]

We're doing a simple combination of PXE boot, kickstart, and internal repos, which then hands off to puppet.  We also use some rvc for vmware provisioning, which is mostly what happens in our environment.

Not many people appear to be talking about razor.  That, plus puppetdb, seems to offer a way to better handle bare metal provisioning and inventory.  Since we're getting into hadoop for a couple projects which will use a collection of bare metal, it looks compelling, but we haven't used it yet.

-Domenick

[Jakov Sosic]

On 09/16/2012 11:31 AM, Dick Davies wrote:
> We use Cobbler for our Centos/RHEL kickstarts (around 150+ nodes, mainly VMs)
> and a very simple .ks that adds puppetd. on first boot it starts up
> and we manually sign the CSR on the puppet master.

+1

except that we allow autosigning from certain subnets.

> Historically, we didn't want tight integration between provisioning and CM as we
> were new to both and if one turned out to be a lemon we didn't want to have to
> abandon both.

We started with both together at the same time :)

> We only provision a few nodes a day max. so that works ok. Lately I've been
> thinking of replacing Cobbler with The Foreman, which has better
> Puppet integration.

What does foreman provide better than cobbler?



--
Jakov Sosic
www.srce.unizg.hr

[Keiran Sweet]

Hi There,

I manage a relatively large RHEL environment, we handle provisioning as follows:

- PXE + Kickstart to bootstrap and install the base OS + Puppet client onto the platform, be it VMWare or bare metal

- Kickstart post scripts put a basic puppet configuration file in place on the host, and a number of the values for things such as environment and puppetmaster come from Foreman's Macro's, this allows values in the ENC to flow into the kickstart files before your first puppet run.

We then run in the %post section of the kickstart file the following:

- A Puppet run that bootstraps the puppet client using tags ie,  --tags puppet::client

- A full puppet run via puppet agent -tov which applys the SOE to the platform

That provides on first boot a fully configured RHEL server that includes all our additional software and customisations in about 3-5 minutes (not including POST)

In regards to certs, we have a relatively open autosign.conf on our build networks, so we can provision servers , physical or virtual quite quickly by just hitting F12 for a network boot. I am sure there are some cleaner/more secure things we can do provisioning wise, however these have been slightly hindered by the RHN Satellite server i've been slowly pulling out of the environment at the same time, as it had the potential to break things if i wasnt careful.

ENC wise, I can't recommend Foreman enough, version 1.x is just brilliant, you can see the macros it can provide here:  http://theforeman.org/projects/foreman/wiki/TemplateWriting

Hope this helps,

K

[Glenn Poston]

Similar setup here: pxe boot, etc. but we keep a pxe booted sparse base image available to clone. And use a script to to do the clone so that we can provision more quickly (saves us the pxe boot time). The script also interacts with puppet dashboard to set the machine role. We can provision a new machine in less than 4 minutes.

[Douglas Garstang]

I probably should have been clearer with my question. I was more
interested in how people are managing certificates? Even if you use
autosign, you still need to clean certificates manually.

Doug.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/NrKmbHHiaq8J.

>
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.g...@gmail.com
Cell: +1-805-340-5627

[Luke Bigum]

If you want the least amount of headache at the cost of security, here
is a sanitised extract from my kickstarts:

#LB: attempt to revoke and delete the certificate for this hostname, this should stop us having
#to manually clean off every cert.
curl -k -X PUT -H "Content-Type: text/pson" --data '{"desired_state":"revoked"}' https://puppet:8140/production/certificate_status/$HOSTNAME
curl -k -X DELETE -H "Accept: pson" https://puppet:8140/production/certificate_status/$HOSTNAME
#LB: run Puppet, our hostname should be set correctly by now
puppet agent --test --pluginsync --report --environment testing


You will need this in auth.conf on your master:

#allow hosts to delete their own certificates
#path /certificate_status/([^/]+)$
path ~ /certificate_status/([^/]+)$
auth any
allow $1

Hope that helps,

-Luke

Luke Bigum
Senior Systems Engineer

Information Systems
Ph: +44 (0) 20 3192 2520
luke....@lmax.com | http://www.lmax.com
LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN


FX and CFDs are leveraged products that can result in losses exceeding
your deposit. They are not suitable for everyone so please ensure you
fully understand the risks involved. The information in this email is not
directed at residents of the United States of America or any other
jurisdiction where trading in CFDs and/or FX is restricted or prohibited
by local laws or regulations.

The information in this email and any attachment is confidential and is
intended only for the named recipient(s). The email may not be disclosed
or used by any person other than the addressee, nor may it be copied in
any way. If you are not the intended recipient please notify the sender
immediately and delete any copies of this message. Any unauthorised
copying, disclosure or distribution of the material in this e-mail is
strictly forbidden.

LMAX operates a multilateral trading facility. Authorised and regulated
by the Financial Services Authority (firm registration number 509778) and
is registered in England and Wales (number 06505809).
Our registered address is Yellow Building, 1A Nicholas Road, London, W11
4AN.

[Peter Bukowinski]

On Sep 17, 2012, at 2:16 PM, Douglas Garstang wrote:

> I probably should have been clearer with my question. I was more
> interested in how people are managing certificates? Even if you use
> autosign, you still need to clean certificates manually.
>
> Doug.

Doug,

We autosign certs for hosts in our datacenter (based on the subdomain wildcard *.domain.org) and manually sign certs for desktops. All our datacenterhosts are set to try network booting first, so it's easy to redeploy any of them at any time. I wrote a re/deployment script that automates all the necessary deployment steps for linux hosts in our datacenter:

1. It lists the available PXE configs and links the
host's address to the one you select.
2. It reboots the host using ipmi.
3. It schedules the removal of the PXE link (so the
host doesn't stay in a permanent install cycle.)
4. It cleans the host's existing puppet cert.

I have this script up on github, if you want to see how I'm doing it. It pushes the limits of bash sanity, but it works well.

https://github.com/pmbuko/misc-scripts/blob/master/deployserv.sh

--
Peter Bukowinski

[R.I.Pienaar]

----- Original Message -----
> From: "Douglas Garstang" <doug.g...@gmail.com>
> To: puppet...@googlegroups.com
> Sent: Monday, September 17, 2012 7:16:02 PM
> Subject: Re: [Puppet Users] Re: Systems Provisioning
>
> I probably should have been clearer with my question. I was more
> interested in how people are managing certificates? Even if you use
> autosign, you still need to clean certificates manually.

so maybe the question is what would your desired behavior be?
what happens with certificates really is a site specific question
influenced by policy, deployment, frequency of rebuilds, acceptable
time taken to do rebuilds etc.

As you've seen from the mco provisioner you've been playing with
its reasonably feasible to implement any behavior you might like
with a bit of code. If you were to note a n step process that
you want machines to go through on first boot then translating
that into a agent + something to drive the agent really isn't
that hard.

The mco provisioner captures what I needed at the time and also
fed back a bunch of requirements into the mcollective development
which would make re-writing it a lot easier today. It's no doubt
too complex for your needs or not a good fit - though it does
allow you to customize exactly what it does in 2 ways:

- you can provide your own agent that implements different logic
for each action. The actions are effectively named API end points
but you can decide what the logic is in each case. You did not
like the bootstrap environment, no biggie, just rewrite that
action to do what you want

- you can enable and disable any of the steps to meet your needs
so sticking with the bootstrap thing as an example only, you
can configure it to just skip that step entirely. stages and
chaining did not exist when this provisioner was written so it
had a complicated extra step to achieve the same.

This worked fine, I had a single provisioner running provisioning
both physical machines and cloud instances - complete with different
logic in the cloud from that in the datacenter by just deploying
different agents leaving the provisioner to just call the agent
in the desired order. My setup was quite complex as I had many
masters in many locations and had to make a lot of decisions upfront
before I could even know what master would be responsible for a
specific node.

I can think of many other ways to approach this problem though.

The question about what to do with certs is purely site specific,
its clear from the answers in this thread that for many auto signing
is acceptable, others disagree. You have to decide what your desired
behavior is based on local policy etc and you're almost certainly
going to have to write some code to match up puppet, kickstart and
all the other moving parts with that.

Puppet, MCollective and other tools just exist to make it easier for
you to reach your goals with a little bit of glue in between - be it
ENCs, scripts, cronjobs or a more complex ever present observer like
the mco provisioner. It's all just building blocks that u need to
combine into a whole.

[Dick Davies]

Don't know it that well yet, but the bits that are appealing to me are
the dashboard
functionality (for puppet reporting) and the 'Smart Proxy' concept to
drive external
services as part of provisioning. That takes care of the puppet
signing issue but in
our case we have bespoke system that runs our DHCP/DNS servers.

Cobbler can manage BIND/isc-dhpcd, but we have to click through an external
webapp to reserve the IP and FQDN up front. A small REST app in front of that
looks like it should be able to allow automation with a Foreman setup.

[Nick]

Hi,

A have a question related to this thread. I'd like to ask, is the Foreman smart
proxy amenable to being configured using Puppet, to the extent of being able to
define the systems that are PXE booted within Puppet? Or is there some other
existing means of configuring PXE booted systems from Puppet?

Cheers,

N

[Glenn Poston]

We have one script that is called to bootstrap and provision new VMs. There's another one for destroying. These scripts are also responsible for logging into the puppet master (via ssh), polling for the cert to be created (puppet cert list) and then issuing the sign/clean commands.

[Jakov Sosic]

I don't know about foreman, but I have written cobbler puppet module
with 4 custom types (distro, profile, repo, system), so we manage
cobbler entirely through puppet. It's more readable and easier to edit
properties. For example, this is how a typical system looks like:


cobblersystem { 'typical-node':
ensure => present,
profile => 'CentOS-6.3-x86_64',
interface => { 'eth0' => { mac => '00:AA:BB:CC:DD:EE', ip_address
=> '192.168.1.99', subnet => '255.255.255.0', static => '1' }, },
netboot => false,
gateway => '192.168.1.1',
require => Service[$cobbler::service_name],
}


--
Jakov Sosic
www.srce.unizg.hr

[Dan White]

Is this module posted somewhere public ?
Looks interesting enough to try out.

[Jakov Sosic]

On 10/13/2012 05:55 PM, Dan White wrote:
> Is this module posted somewhere public ?
> Looks interesting enough to try out.

It's not posted yet but it will be soon. I'm currently rewriting some
providers from CLI cobbler to XMLRPC calls, and I'm not yet fully
satisfied with integration with puppetlabs/apache module. I had to
modify apache module to make it work...


Also I have some minor issues, like first run adds system without
interfaces to cobbler, and subsequent run add interfaces. So you have to
run it twice to properly add the system.

Maybe someone can point out is there a way to fix this, because I've
implemented interfaces as property and not param, so provider has
separate methods for checking/modifying current state, and I don't quite
get it why it doesn't run seamlessly in the first run

Also, I don't have documentation ready yet :)



Although if you want I can send you current snapshot and help with
ongoing issues that occur. And ofcourse you don't have to use the whole
module, you can use just custom types if you wish.

[Stefan Schulte]

On Sat, Oct 13, 2012 at 10:50:05PM +0200, Jakov Sosic wrote:
> On 10/13/2012 05:55 PM, Dan White wrote:
> > Is this module posted somewhere public ?
> > Looks interesting enough to try out.
>
> It's not posted yet but it will be soon. I'm currently rewriting some
> providers from CLI cobbler to XMLRPC calls, and I'm not yet fully
> satisfied with integration with puppetlabs/apache module. I had to
> modify apache module to make it work...
>
>
> Also I have some minor issues, like first run adds system without
> interfaces to cobbler, and subsequent run add interfaces. So you have to
> run it twice to properly add the system.
>
> Maybe someone can point out is there a way to fix this, because I've
> implemented interfaces as property and not param, so provider has
> separate methods for checking/modifying current state, and I don't quite
> get it why it doesn't run seamlessly in the first run
>

If puppet has to sync ensure it will not sync any other property. That
means if your type defines "ensurable" or you have defined an ensure
property manually and your system is not yet present, your create method
is called and puppet expects the create method to create your system with
interfaces.

Otherwise you will see the described behaviour:

1) First run: Puppet finds out ensure is out of sync (is absent, should
be present) and calls create
2) Second run: Puppet finds out interfaces is out of sync and and calls
interfaces= (or whatever method you have defined for that)

(see lib/puppet/transaction/resource_harness.rb#perform_changes)

-Stefan

[Jakov Sosic]

On 10/13/2012 11:17 PM, Stefan Schulte wrote:
> If puppet has to sync ensure it will not sync any other property. That
> means if your type defines "ensurable" or you have defined an ensure
> property manually and your system is not yet present, your create method
> is called and puppet expects the create method to create your system with
> interfaces.

My understanding of the problem was along those lines too. Now I have
confirmation...

> Otherwise you will see the described behaviour:
>
> 1) First run: Puppet finds out ensure is out of sync (is absent, should
> be present) and calls create
> 2) Second run: Puppet finds out interfaces is out of sync and and calls
> interfaces= (or whatever method you have defined for that)

I am already detecting in my create method if 'system' is added to
'cobbler' (if it's present on the machine agent is running on), and if
it is, I choose to edit it rather then to try to create it again.

So, can I just call method "interface=", if I detect I have to create
the 'system', or is there any way to solve this issue?

Problem is that cobbler is used for provisioning (PXE+installation), so
when you add new 'system' to cobbler, you are going to try to install
the machine immediately afterwards (in 99% of cases), so two puppet runs
in this case really stand in the way of usability of the whole idea of
puppetizing Cobbler setup. So I would love to circumvent it somehow if
possible.


Thank you

[Stefan Schulte]

On Sat, Oct 13, 2012 at 11:52:49PM +0200, Jakov Sosic wrote:
> On 10/13/2012 11:17 PM, Stefan Schulte wrote:
> > If puppet has to sync ensure it will not sync any other property. That
> > means if your type defines "ensurable" or you have defined an ensure
> > property manually and your system is not yet present, your create method
> > is called and puppet expects the create method to create your system with
> > interfaces.
>
> My understanding of the problem was along those lines too. Now I have
> confirmation...
>
>
> > Otherwise you will see the described behaviour:
> >
> > 1) First run: Puppet finds out ensure is out of sync (is absent, should
> > be present) and calls create
> > 2) Second run: Puppet finds out interfaces is out of sync and and calls
> > interfaces= (or whatever method you have defined for that)
>
>
> I am already detecting in my create method if 'system' is added to
> 'cobbler' (if it's present on the machine agent is running on), and if
> it is, I choose to edit it rather then to try to create it again.

If you use "ensurable", puppet will expect the provider to have an
exists? method and if that returns true your create method will not be
called so you do not have to check the existance in the create method

again.

>
> So, can I just call method "interface=", if I detect I have to create
> the 'system', or is there any way to solve this issue?

If adding interfaces to a new host is exactly the same as changing
interfaces of an already existing host you can call

interface = resource[:interface] if resoure[:interface]

Is your provider dealing with files or does it execute commands? When
dealing with files I find it is often easier to do all the work in the
flush method (which is only called if it is implemented by the provider)
and the other set methods are only updating the @property_hash hash.

So if you want to share your provider code I am always interested ;-)

-Stefan

[Jakov Sosic]

On 10/14/2012 03:01 PM, Stefan Schulte wrote:
> If you use "ensurable", puppet will expect the provider to have an
> exists? method and if that returns true your create method will not be
> called so you do not have to check the existance in the create method
> again.

But then I would have to change all params to properties and add
appropriate methods to provider, but that would both slow it down and
complicate it more... I don't know if it's worth it just for the sake of
log message stating "property changed from A to B" instead of
"cobblersystem created" on every param change.

> If adding interfaces to a new host is exactly the same as changing
> interfaces of an already existing host you can call
>
> interface = resource[:interface] if resoure[:interface]

I'll try that.

> Is your provider dealing with files or does it execute commands? When
> dealing with files I find it is often easier to do all the work in the
> flush method (which is only called if it is implemented by the provider)
> and the other set methods are only updating the @property_hash hash.
>
> So if you want to share your provider code I am always interested ;-)

Offcourse, I have nothing to hide. Here's the current code:

http://pastebin.com/f7GFU2qp



--
Jakov Sosic
www.srce.unizg.hr

[Jakov Sosic]

On 10/14/2012 04:20 PM, Jakov Sosic wrote:
> On 10/14/2012 03:01 PM, Stefan Schulte wrote:
>> If you use "ensurable", puppet will expect the provider to have an
>> exists? method and if that returns true your create method will not be
>> called so you do not have to check the existance in the create method
>> again.
>
> But then I would have to change all params to properties and add
> appropriate methods to provider, but that would both slow it down and
> complicate it more... I don't know if it's worth it just for the sake of
> log message stating "property changed from A to B" instead of
> "cobblersystem created" on every param change.
>
>
>> If adding interfaces to a new host is exactly the same as changing
>> interfaces of an already existing host you can call
>>
>> interface = resource[:interface] if resoure[:interface]
>
> I'll try that.
>

First of all I renamed "interface" to "interfaces", and then I've found
solution for adding the property in first run, here is what code looks like:

# add interfaces
unless self.interfaces == @resource.should(:interfaces)
self.interfaces = @resource.should(:interfaces)
end

It's documented here:
http://projects.puppetlabs.com/projects/puppet/wiki/Development_Complete_Resource_Example

I forgot about documentation :)

So I guess I could be rewriting all of my params to properties now, hm...


--
Jakov Sosic
www.srce.unizg.hr

[Stefan Schulte]

On Sun, Oct 14, 2012 at 04:20:09PM +0200, Jakov Sosic wrote:
> On 10/14/2012 03:01 PM, Stefan Schulte wrote:
> > If you use "ensurable", puppet will expect the provider to have an
> > exists? method and if that returns true your create method will not be
> > called so you do not have to check the existance in the create method
> > again.
>
> But then I would have to change all params to properties and add
> appropriate methods to provider, but that would both slow it down and
> complicate it more... I don't know if it's worth it just for the sake of
> log message stating "property changed from A to B" instead of
> "cobblersystem created" on every param change.
>

Is it possible to run a command to get all the desired information about
every systems at once? This way you can implement a prefetch pattern.
Basically you create provider instances for each system at once and
write the current values in the @property_hash hash. Then your get methods
just return the cached value which scales pretty well.

>
> > If adding interfaces to a new host is exactly the same as changing
> > interfaces of an already existing host you can call
> >
> > interface = resource[:interface] if resoure[:interface]
>
> I'll try that.
>
>
> > Is your provider dealing with files or does it execute commands? When
> > dealing with files I find it is often easier to do all the work in the
> > flush method (which is only called if it is implemented by the provider)
> > and the other set methods are only updating the @property_hash hash.
> >
> > So if you want to share your provider code I am always interested ;-)
>
> Offcourse, I have nothing to hide. Here's the current code:
>
> http://pastebin.com/f7GFU2qp
>
>
>
> --
> Jakov Sosic
> www.srce.unizg.hr
>

[Jakov Sosic]

On 10/15/2012 09:13 AM, Stefan Schulte wrote:

> Is it possible to run a command to get all the desired information about
> every systems at once? This way you can implement a prefetch pattern.
> Basically you create provider instances for each system at once and
> write the current values in the @property_hash hash. Then your get methods
> just return the cached value which scales pretty well.

Yes it is possible to get all systems at once. It's what the XMLRPC call
does:

xmlrpcresult = cobblerserver.call("get_systems")

I already do that, but I select only one system, examine the hash, and
if I notice differences from current puppet settings, approach to
running CLI commands editing that system.

Your idea would require top to bottom rewrite of provider, and I don't
have currently time for it... also, what are the benefits?

Currently I am planning to change params to properties (every param that
is not needed for creation of system can be managed as property - it's
more native way as I figured it out now).

[Stefan Schulte]

The fact that your exists? method does not really answer the question if
a resource is present or absent is a bit strange. And inside the create
method you are basically reimplementing properties with parameters. If
something has to be checked for correctness it should be a property.
Otherwise it is a parameter. Like the service resource: enable is a
property because it can be out of sync. hasstatus is a parameter because
it cannot be out of sync but only changes the behaviour of the provider

Your main concern against properties if I got you correctly was about
speed because puppet would run one query for each property. One way around
that is to implement a query method that will query all properties at once
and store them in a hash (@property_hash). Every get-method now check
if @property_hash[:some_property] does already exist and return that
value if it does or run the query method that would populate the
@property_hash hash.

Another speed improvement is to implement an `instances` and `prefetch`
method. That has the benefit that puppet does "react" on such methods
if they are implemented:

* you are able to run "puppet resource cobblersystem" on the command
line to get the current configuration of all systems (that depends on
an instances classmethod)
* you can use the resources type to purge unmanaged systems

resources { 'cobblersystem':
purge => true
}
* the prefetch method is automatically called by puppet if implemented
to create provider instances
* your get methods become trivial

-Stefan

[Jakov Sosic]

On 10/15/2012 07:23 PM, Stefan Schulte wrote:

> The fact that your exists? method does not really answer the question if
> a resource is present or absent is a bit strange. And inside the create
> method you are basically reimplementing properties with parameters. If
> something has to be checked for correctness it should be a property.
> Otherwise it is a parameter. Like the service resource: enable is a
> property because it can be out of sync. hasstatus is a parameter because
> it cannot be out of sync but only changes the behaviour of the provider

OK, I've figured that out through this conversation...

Now this is somewhat fixed code:

http://pastebin.com/q0TBX4KB

I've moved some params to properties.

> Your main concern against properties if I got you correctly was about
> speed because puppet would run one query for each property. One way around
> that is to implement a query method that will query all properties at once
> and store them in a hash (@property_hash). Every get-method now check
> if @property_hash[:some_property] does already exist and return that
> value if it does or run the query method that would populate the
> @property_hash hash.

That sounds interesting, and more important it seems to me that complete
rewrite is not necessary in this case. Do you have some examples of this
idea?

> Another speed improvement is to implement an `instances` and `prefetch`
> method. That has the benefit that puppet does "react" on such methods
> if they are implemented:
>
> * you are able to run "puppet resource cobblersystem" on the command
> line to get the current configuration of all systems (that depends on
> an instances classmethod)
> * you can use the resources type to purge unmanaged systems
>
> resources { 'cobblersystem':
> purge => true
> }
> * the prefetch method is automatically called by puppet if implemented
> to create provider instances
> * your get methods become trivial

Wow, sounds very interesting.

I would plea for possible examples :)

[Nan Liu]

You can look for other examples in puppet source code or puppet
modules with custom providers. Simply search for self.instances.

Here's the rough draft about self.instances from the provider chapter
for Puppet Types and Provider. The first two chapters are available as
early preview for O'Reilly Safari users. We are looking forward to
getting some feedback as we flush out more content. Also we are
looking for a technical reviewer if anyone is interested.

Thanks,

Nan

[Stefan Schulte]

The instances method is a class method and has to return an array of
providers. So this often looks like this

def self.instances
systems = []
my_fancy_command.each_line do |line|
somehow_split_line_into_different_fields_like_name_and_interfaces
systems << new(
:name => name,
:interfaces => interfaces,
:ensure => :present
)
end
systems
end

One important thing: If you create a new provider instance you can pass
a hash (like I did in new(:name => name, :interfaces => interface)) and
this hash is stored in the member variable @property_hash of that new
provider.

An example of a simple instances method:
https://github.com/stschulte/puppet-rpmkey/blob/master/lib/puppet/provider/rpmkey/rpm.rb

The rpmkey type can make sure that a certain gpg key is imported into
rpm. To get the currently installed keys the provider runs

rpm -q gpg-key

This command can either return with a non zero exit code (no packages
found) in case we have zero keys or it will print one line per key.
For each line a provider instance is added to the array that is finally
returned.

prefetch:
The prefetch method is called by puppet for each providerclass that
implements such a method (see lib/puppet/transaction.rb#prefetch). The
prefetch method is called with a hash of every resource that is defined
in the user's manifest (=every resource puppet should manage). The
hash will have the form resource[:name] as a key and resource as the
value. What the prefetch method can do now is create provider instances
and bind the provider instances to resources. A common prefetch method
that is also shown in the rpm provider for rpmkey:

def self.prefetch(resources)
instances.each do |prov|
if resource = resources[prov.name]
resource.provider = prov
end
end
end

The prefetch method first calls instances that will return a list of
every key that is currently present. Then I check if that key is also
managed by puppet. If the lookup succeeds (the key is indeed managed by
puppet), I'll bind the provider to the resource. At this point the provider
instance already has @property_hash[:ensure] set, so when puppet later
handles the different rpmkey resources and asks exists? I can simply
return the cached value.

def exists?
get(:ensure) != :absent
end

Note: get(:ensure) is implemented in lib/provider.rb as

def get(param)
@property_hash[param.intern] || :absent
end

A more complex provider that has to manage multiple properties:
https://github.com/stschulte/puppetlabs-solaris/blob/feature/master/projects/lib/puppet/provider/project/projadd.rb

The instances method is more complicated here but it basically does the
same thing: return an array of provider instances. You'll notice that I
do not implement any method to get the current values of the properties.
The methodcall "mk_resource_methods" already creates these for me.
"mk_resource_methods" will create a get method for every property that
is essentially (if your property is named interfaces)

def interfaces
@property_hash[:interfaces] || :absent
end

I hope that helps. An example of a query method can be found in the
package provider in puppet core. The package provider is a bit special
here because it does not trust its prefetched values. This is because
one resource can change the state of another resource:

package { "mod_ssl": ensure => installed }
package { "httpd": ensure => installed }

The ensure property of "httpd" can change in the middle of a puppetrun (if
mod_ssl is installed first httpd will be installed as a dependency)
so the prefetched value of :absent can be obsolete when puppet actually
handles the resource Package["httpd"]. For you cobblersystem type I do
not think that one resource may influence another resource right?

-Stefan

[Jakov Sosic]

On 10/15/2012 11:44 PM, Stefan Schulte wrote:

> The instances method is a class method and has to return an array of
> providers. So this often looks like this
>
> def self.instances
> systems = []
> my_fancy_command.each_line do |line|
> somehow_split_line_into_different_fields_like_name_and_interfaces
> systems << new(
> :name => name,
> :interfaces => interfaces,
> :ensure => :present
> )
> end
> systems
> end

Huh, so I have to build whole resources in self.instances, with all the
possible properties?

> This command can either return with a non zero exit code (no packages
> found) in case we have zero keys or it will print one line per key.
> For each line a provider instance is added to the array that is finally
> returned.

Got it from the upper code.

> I hope that helps. An example of a query method can be found in the
> package provider in puppet core. The package provider is a bit special
> here because it does not trust its prefetched values. This is because
> one resource can change the state of another resource:
>
> package { "mod_ssl": ensure => installed }
> package { "httpd": ensure => installed }
>
> The ensure property of "httpd" can change in the middle of a puppetrun (if
> mod_ssl is installed first httpd will be installed as a dependency)
> so the prefetched value of :absent can be obsolete when puppet actually
> handles the resource Package["httpd"]. For you cobblersystem type I do
> not think that one resource may influence another resource right?

Yeah, you are right, one cobblersystem instance cannot influence
creation or modification of other instance.

But now that you mention, I do have one more question in this regard. Is
it possible for some property to create auto-require with some other
resource? Except from cobblersystem I do have 3 another custom types
(cobblerprofile, cobblerdistro and cobblerrepo). So, for example:

cobblerprofile { 'foo':
ensure => present,
}
cobblersystem { 'bar':
ensure => present,
profile => 'foo',
require => Cobblerprofile['foo'],
}

Is there a way to skip the last require, and force it through profile
property in the type/provider code?


Thank you so much for the great explanation. I will save it and try to
code it first on my simpler types like cobblerprofile and if I succeed
will jump into fire and rewrite cobblersystem.

ty.

[Nan Liu]

Since you already specify the profile, in your cobblersystem resource
type add autorequire.

autorequire(:cobblerprofile) do
self[:profile]
end

Thanks,

Nan

[Jakov Sosic]

On 10/15/2012 07:23 PM, Stefan Schulte wrote:

> Another speed improvement is to implement an `instances` and `prefetch`
> method. That has the benefit that puppet does "react" on such methods
> if they are implemented:
>
> * you are able to run "puppet resource cobblersystem" on the command
> line to get the current configuration of all systems (that depends on
> an instances classmethod)
> * you can use the resources type to purge unmanaged systems
>
> resources { 'cobblersystem':
> purge => true
> }
> * the prefetch method is automatically called by puppet if implemented
> to create provider instances
> * your get methods become trivial

Thank both you and Nan Liu.

As the holidays approached I finally got a free day to work on this :)
I've rewritten 3 out of 4 my custom providers to self.instances schema.

I will test this and report back performance improvements on puppet
agents run with lots of resources of custom type I wrote, after I
rewrite the final provider.

And code looks much cleaner too! :) Once more a big thank you both :)


--
Jakov Sosic
www.srce.unizg.hr

[Jakov Sosic]

On 12/24/2012 01:59 AM, Jakov Sosic wrote:

> Thank both you and Nan Liu.
>
> As the holidays approached I finally got a free day to work on this :)
> I've rewritten 3 out of 4 my custom providers to self.instances schema.
>
> I will test this and report back performance improvements on puppet
> agents run with lots of resources of custom type I wrote, after I
> rewrite the final provider.
>
> And code looks much cleaner too! :) Once more a big thank you both :)

OK, I've finally published Cobbler module for Puppet :)

http://forge.puppetlabs.com/jsosic

https://bitbucket.org/jsosic/puppet-cobbler



And here is the most complex provider I have so far:

https://bitbucket.org/jsosic/puppet-cobbler/src/da3cdd684314360dcc5c633f374310f59f87ec42/lib/puppet/provider/cobblersystem/system.rb?at=default



And now about performance increase after rewriting my providers to use
XMLRPC one-call-per provider (self.instances) instead of few cmdline
calls for every resources. These are the agent run times:


Site1 (22 x cobblersystem)
- before: 68.1s
- after: 9.7s

Site2 (19 x cobblersystem)
- before: 54.0s
- after: 10.0s

Site3: (11 x cobblersystem)
- before: 40.9s
- after: 9.7s

Site4: (22 x cobblersystem)
- before: 80.1s
- after: 10.6s

So I guess this changes move it from O(n) to O(1) :)



--
Jakov Sosic
www.srce.unizg.hr