Custom Facts

[Douglas Garstang]

So, after reading this doc:

http://projects.puppetlabs.com/projects/1/wiki/Adding_Facts

it's not readily apparent to me how I can create facts that are derived from variables defined at the node level in puppet, rather than at the O/S level on the client. I want to be able to set a variable that defines a server type in the node, and access that. Anyone?

Doug.

[Aaron Grewell]

You can't.  The whole purpose of facts is to allow Puppet to make decisions based on client data.  Variables=server, facts=client.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Nigel Kersten]

On Thu, Sep 1, 2011 at 4:31 PM, Aaron Grewell <aaron....@gmail.com> wrote:

You can't.  The whole purpose of facts is to allow Puppet to make decisions based on client data.  Variables=server, facts=client.

Yep. However, puppet manifests do allow you to access both and make decisions based on their values.

class base {

  if $operatingsystem == "debian" and $server_location == "botswana" {

    include aptclient::botswana

  }

}

 

On Thu, Sep 1, 2011 at 3:41 PM, Douglas Garstang <doug.g...@gmail.com> wrote:

So, after reading this doc:

http://projects.puppetlabs.com/projects/1/wiki/Adding_Facts

it's not readily apparent to me how I can create facts that are derived from variables defined at the node level in puppet, rather than at the O/S level on the client. I want to be able to set a variable that defines a server type in the node, and access that. Anyone?

Doug.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--
Nigel Kersten
Product Manager, Puppet Labs

Join us for PuppetConf 

<http://www.bit.ly/puppetconfsig>

Sept 22/23 Portland, Oregon, USA.

[Douglas Garstang]

On Thu, Sep 1, 2011 at 4:31 PM, Aaron Grewell <aaron....@gmail.com> wrote:

You can't.  The whole purpose of facts is to allow Puppet to make decisions based on client data.  Variables=server, facts=client.

Seriously??? I can edit mcollective's server.cfg file and change the factsource from facter to yaml, and then have puppet populate facts.yaml, which I think is actually easier and more manageable, but someone here has already gone and used custom facts, which means I have to use facter as the _only_ fact source. That's pretty lame.

So, if I am stuck with this, how would I set a class of server for a particular node? Would I need to have puppet push out a text file containing the server class to the client, and then write a custom fact that reads the file from the client's OS? That seems like a much more complicated approach than setting 'server_class = web_server' in the node on the puppet master, and then having puppet put that in facts.yaml.

Doug

[Aaron Grewell]

Create a variable for server_class in your node definition.

[Douglas Garstang]

On Thu, Sep 1, 2011 at 4:56 PM, Aaron Grewell <aaron....@gmail.com> wrote:

Create a variable for server_class in your node definition.

And then....? 

 

[Aaron Grewell]

And then assign classes based on that.  Standard if logic or case statement as you prefer.

--

[Douglas Garstang]

On Thu, Sep 1, 2011 at 4:58 PM, Aaron Grewell <aaron....@gmail.com> wrote:

And then assign classes based on that.  Standard if logic or case statement as you prefer.

On Thu, Sep 1, 2011 at 4:57 PM, Douglas Garstang <doug.g...@gmail.com> wrote:

On Thu, Sep 1, 2011 at 4:56 PM, Aaron Grewell <aaron....@gmail.com> wrote:

Create a variable for server_class in your node definition.

And then....? 

 

This isn't for logic within puppet manifests. In this particular scenario, it's for setting a server class on each node so that I can use mcollective against servers matching that class. For example, if I wanted to stop apache on the web servers, I can call mc-service httpd restart -F "hostclass=webserver" etc.

Doug.

[Aaron Grewell]

Ah.  I don't know how mcollective works so I'm not sure how to address that.  If there's a YAML data file involved then you could push it with Puppet...

[Douglas Garstang]

On Thu, Sep 1, 2011 at 5:05 PM, Aaron Grewell <aaron....@gmail.com> wrote:

Ah.  I don't know how mcollective works so I'm not sure how to address that.  If there's a YAML data file involved then you could push it with Puppet...

Yes, mcollective can read a YAML file, which by default is /etc/mcollective/facts.yaml. However, if I go down that path, it can't read custom facts. If I use custom facts, I can't find a way to set a variable in a node, and then access it in the custom fact.

Doug.

 

[Nigel Kersten]

http://docs.puppetlabs.com/mcollective/reference/ui/filters.html 

$ mco rpc --help . . . Host Filters -W, --with FILTER Combined classes and facts filter -F, --wf, --with-fact fact=val Match hosts with a certain fact -C, --wc, --with-class CLASS Match hosts with a certain config management class -A, --wa, --with-agent AGENT Match hosts with a certain agent 

-I, --wi, --with-identity IDENT Match hosts with a certain configured identity

 

You want to match a class, not a fact.

MCollective can match based on Puppet classes already by reading classes.txt which contains all the classes from the latest catalog.

[Aaron Grewell]

This is just me Googling wildly into the air but are you using this approach?

http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/FactsFacterYAML

If I'm reading the code correctly (somewhat questionable, my Ruby-foo is weak) it should return all top-scope variables and facts.

--

[Douglas Garstang]

Tried that once before... I forget why.... but it didn't work so well.

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.g...@gmail.com
Cell: +1-805-340-5627

[Douglas Garstang]

Thanks Nigel, but there's no mention of classes.txt on that page.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Douglas Garstang]

Nigel,

Are you referring to the 'Class Filters' section on http://docs.puppetlabs.com/mcollective/reference/integration/puppet.html ?

If so, that's not really the same thing. Classes.txt lists classes included on a server. That's not what I am looking for. Life isn't always as simple as assuming that all machines within a certain functional group include the same class. You'd think so, but hey, I work for a startup, so not everything is engineered that well.

All I want to be able to do is set a variable on a per node basis that defines a functional grouping, ie a class of server, and query that with mcollective. Should not be difficult to do.

Doug.

[Nigel Kersten]

On Thu, Sep 1, 2011 at 5:40 PM, Douglas Garstang <doug.g...@gmail.com> wrote:

Nigel,

Are you referring to the 'Class Filters' section on http://docs.puppetlabs.com/mcollective/reference/integration/puppet.html ?

If so, that's not really the same thing. Classes.txt lists classes included on a server. That's not what I am looking for. Life isn't always as simple as assuming that all machines within a certain functional group include the same class. You'd think so, but hey, I work for a startup, so not everything is engineered that well.

That's not an immense amount of engineering.... and if your functional groups don't actually map well to classes, that's a sign of a problem that you should be resolving as it's almost certainly causing wasted effort.

 

All I want to be able to do is set a variable on a per node basis that defines a functional grouping, ie a class of server, and query that with mcollective. Should not be difficult to do.

There are many hoops you could jump through to achieve this like making a file that your node parameters get pushed to in a template, and then make facts that query that file... 

but honestly, work with the system and not against it. Make some functional classes that correspond to the node parameters you're talking about, and include them.

I can't remember off the top of my head what happens with completely empty classes, they may not actually get written out to classes.txt, but I'd test that first. If that fails, stick some dummy resources in, but your best bet is to start making functional classes mapping to the subset of nodes you want to pick out, and organize your existing classes by function where appropriate.

--

[Douglas Garstang]

I really appreciate your reply, but I've inherited a mess that isn't easy to fix, and the most appropriate solution for right now isn't always the "best" solution. When the boss says "GET IT DONE", you don't have time to refactor someone elses mess and do it the right way. I try and make my questions as specific as possible so that I can get options that are as specific to the problem at hand as possible.

[Al @ Lab42]

All I want to be able to do is set a variable on a per node basis that defines a functional grouping, ie a class of server, and query that with mcollective. Should not be difficult to do.

It isn't and it seems to me that you already know how to do it, just do not try to assign that variable as a custom facts (it that doesn't show up in the scope lookup) and set it in puppet manifests (or in the ENC), then set factsource = yaml in mcollective config file and populate it with something like:

    file {"/etc/mcollective/facts.yaml":

        owner    => root,

        group    => root,

        mode     => 400,

        loglevel => debug,  # this is needed to avoid it being logged and reported on every run

        # avoid including highly-dynamic facts as they will cause unnecessary template writes

        content  => inline_template("<%= scope.to_hash.reject { |k,v| k.to_s =~ /(uptime.*|path|timestamp|free|.*password.*|.*psk.*|.*key)/ }.to_yaml %>"),

        require => Package["mcollective"],

    }

I generally use this, in mcollective for custom variables like $role and $role_group that are assigned on role classes that are included by nodes (one node, one role).

The $role_group is useful, for me, to group all the machines (of different roles) that cuncur to provide a service for users (they have all to work for the service to be usable).

And trust me, when you can check in few seconds, whatever is relevant to a site or a service offered to your users with something like:

mc-puppi check -F role_group=appstore

you are happy, but that's another story.

My2c

al

[Matt Moor]

On 2/09/11 3:03 PM, Douglas Garstang wrote:
> I really appreciate your reply, but I've inherited a mess that isn't
> easy to fix, and the most appropriate solution for right now isn't
> always the "best" solution. When the boss says "GET IT DONE", you
> don't have time to refactor someone elses mess and do it the right
> way. I try and make my questions as specific as possible so that I can
> get options that are as specific to the problem at hand as possible.

Class is an unbelievably overloaded term, but I'm gathering you mean it
in the context of identifying a group of machines, and will assume so
for the purposes of this post. I've hacked around this problem a couple
of ways in the past:

- Derive the class from the FQDN

Previous organisations have had FQDN structures like
sitename-app-01.staging.mycompany.net, and I've just written a simple
fact to extract the class from that; e.g. staging-app. I don't really
recommend this approach, as it can be brittle, it overloads DNS, and
it's not very granular. But it worked.

- Use a "helper" file

If you've got the node classification available on the puppetmaster
(e.g. node foo { $class=abc }), I've used a template to write a file
with the class details out somewhere useful like
/etc/mycompany/machine_class.

This guy seems to have taken this idea and run with it:
http://nuknad.com/2011/02/11/self-classifying-puppet-nodes/

If I was wanting to do this the "right" way, I'd be looking at hooking
mcollective up to an external node classifier in some way - that way
both Puppet and MCollective are referencing the same source of "truth".
No tips on how to actually to do this - it's just where I'd start looking.

Cheers,

Matt

[Aaron Grewell]

In case you decide to go with an ENC, here's the one we use.  It takes a YAML file for each host providing environment, classes, and parameters to be used.  All node-specific data is kept in the same YAML file and either provided via the ENC or pulled in via Hiera at the class level.

#-----------------------------------------
#!/usr/bin/ruby
# Modified by Aaron Grewell <agr...@rei.com>
# Originally by Gary Larizza
# http://glarizza.posterous.com

require 'yaml'
require 'puppet'

# Intitialize Variables
environments = ["development", "testing", "qa", "production"]
prefix = "/usr/share/puppet/environments"
default = {'classes' => []}
parameters = {}
yaml_output = {}
yamlfile = nil

# Check to see if the Node YAML file exists.
# If it exists, set the classes variable to that fact's value
begin
  environments.each do |environment|
    nodefile = "#{prefix}/#{environment}/nodes/#{ARGV[0]}.yaml"
    if File::exists?( nodefile )
      yamlfile = YAML::load_file(nodefile)
    end
  end

  if yamlfile.nil?
    puts "ERROR: Node YAML file was not found!"
    exit(1)
  end

  environment = yamlfile["environment"]
  classes = yamlfile["classes"]
  parameters = yamlfile["parameters"]
end

# Output our classes and environment values to YAML for Puppet
yaml_output =  {'classes' => classes, 'environment' => environment, 'parameters' => parameters}
print yaml_output.to_yaml
#-----------------------------------------

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.

To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.

[Douglas Garstang]

On Thu, Sep 1, 2011 at 10:30 PM, Matt Moor <m...@imprecise.org> wrote:

Hi Matt. :)

Unfortunately, using the FQDN to determine the class of server here isn't feasible. The DNS namespace is pretty flat. As for the helper file option, I can add a variable to the node definition, which is what I was hoping to do. I can then template the value, and write it to a file on the target. However, at this point, using a custom fact to retrieve the value from the local O/S seems a bit heavy handed. Maybe that's what I'll have to end up using.

At the end of the day, this all seems to be a limitation with mcollective. It can only read facts from one source, either facter or a yaml file. I believe facter is the current source, and people have written scripts that rely on that. I can write a custom fact, but.... then the issue becomes that I don't know if it's possible for a custom fact written in ruby and dropped into $module/lib/puppet can access variables defined in the node.

Doug.

[R.I.Pienaar]

----- Original Message -----
>
> At the end of the day, this all seems to be a limitation with
> mcollective. It can only read facts from one source, either facter
> or a yaml file. I believe facter is the current source, and people
> have written scripts that rely on that. I can write a custom fact,
> but.... then the issue becomes that I don't know if it's possible
> for a custom fact written in ruby and dropped into
> $module/lib/puppet can access variables defined in the node.
>

using the yaml file built by a template method that was linked to
any variables you set at the node level, top scope or in the class
that creates the yaml file is put in the yaml file.

Therefore you can use these files in mcollective filters, so if you
just set variables in the node to do whatever you want they will show
up in the yaml file and be usable.

As the wiki says:

"plus it lets you get any in-scope variables (for example, parameters from your external node classifier) available as mcollective filters for free."

http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/FactsFacterYAML

Mcollective thus gets a combination of facts, variables etc all in one
place. You can even create many yaml files and it will read the lot for
you, one local the node, one by puppet, maybe even with facter -y etc.

and if all this fails, you can trivially write your own fact source that
does whatever you want.

[Douglas Garstang]

--

So, I think I had used this approach once before. But, now.... after implementing that, yes, facts.yaml seems to have all the facter variables as well as variables defined in the node, which is generally what I want.

However, running mc-inventory shows no facts, and doing something like this:

mc-service puppet status -F "hosttype=proxy"

where hosttype=proxy does appear in the yaml file, returns no data. I have set the factsource to yaml in mcollective's server.cfg and restarted it.

Douglas.

 

[Douglas Garstang]

Eh. Now it's working. Weird.

[Alessandro Franceschi]

Btw, Rip or anyone, any suggestion on how to sort alphabetically the
scope variables so that the generated yaml doesn't change at (almost)
every puppet run?

content => inline_template("<%= scope.to_hash.reject { |k,v| k.to_s

=~ /(uptime_seconds|timestamp|free)/ }.to_yaml %>")

On 2 Set, 20:26, "R.I.Pienaar" <r...@devco.net> wrote:
> ----- Original Message -----
>
> > At the end of the day, this all seems to be a limitation with
> > mcollective. It can only read facts from one source, either facter
> > or a yaml file. I believe facter is the current source, and people
> > have written scripts that rely on that. I can write a custom fact,
> > but.... then the issue becomes that I don't know if it's possible
> > for a custom fact written in ruby and dropped into
> > $module/lib/puppet can access variables defined in the node.
>
> using the yaml file built by a template method that was linked to
> any variables you set at the node level, top scope or in the class
> that creates the yaml file is put in the yaml file.
>
> Therefore you can use these files in mcollective filters, so if you
> just set variables in the node to do whatever you want they will show
> up in the yaml file and be usable.
>
> As the wiki says:
>
> "plus it lets you get any in-scope variables (for example, parameters from your external node classifier) available as mcollective filters for free."
>

> http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/Fact...

[R.I.Pienaar]

----- Original Message -----
> Btw, Rip or anyone, any suggestion on how to sort alphabetically the
> scope variables so that the generated yaml doesn't change at (almost)
> every puppet run?
>
> content => inline_template("<%= scope.to_hash.reject { |k,v| k.to_s
> =~ /(uptime_seconds|timestamp|free)/ }.to_yaml %>")

no, unfortunately the usual method for doing this with yaml doesnt work
in puppet, probably because of their yaml not being normal yaml.

http://snippets.dzone.com/posts/show/5811

maybe one of the devs can say, I tried various methods non that I didnt
consider a hack would work.

I also filed a bug to see if certain resources can be excluded from reporting
but this got no love