Puppet, Facter and Mcollective

[Tony C]

I'm not sure if this is the right group or not, but i'll start here.

I have Puppet enterprise 2.0, playing around with custom facts.

I have noticed that adding a custom fact to any module will distribute
that fact to all machines, regardless if they are assigned to that
module or not. Is there a way around this, or is this just by design?

I have the pluginsync and mcollective setup to use /etc/puppetlabs/
mcollective/facts.yaml.

server.cfg
# Facts
factsource = yaml
plugin.yaml = /etc/puppetlabs/mcollective/facts.yaml

Without doing anything, after a few minutes, my facts.yaml gets
populated with the new custom fact, even without following this guide:

http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/FactsFacterYAML


and mco find -F myfact will work as well, not at first, but after a
minute or 2. why is that?

[Gary Larizza]

Hey there,

Check your cron with crontab -l and you'll see a line that calls "/opt/puppet/sbin/refresh-mcollective-metadata" every 15 minutes.  That's the process that updates the facts.yaml file with your current custom facts.  You can trigger it whenever you'd like :)

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--

Gary Larizza

Professional Services Engineer

Puppet Labs

[Tony C]

Awesome Gary, Thanks for the reply.

Will this also populate parameters assigned via an ENC (my case would
be the puppet dashboard)?

Thanks again.

On Feb 16, 3:33 pm, Gary Larizza <g...@puppetlabs.com> wrote:
> Hey there,
>
> Check your cron with crontab -l and you'll see a line that calls
> "/opt/puppet/sbin/refresh-mcollective-metadata" every 15 minutes.  That's
> the process that updates the facts.yaml file with your current custom
> facts.  You can trigger it whenever you'd like :)
>
>
>
>
>
>
>
>
>
> On Thu, Feb 16, 2012 at 3:19 PM, Tony C <tonyjch...@gmail.com> wrote:
> > I'm not sure if this is the right group or not, but i'll start here.
>
> > I have Puppet enterprise 2.0, playing around with custom facts.
>
> > I have noticed that adding a custom fact to any module will distribute
> > that fact to all machines, regardless if they are assigned to that
> > module or not. Is there a way around this, or is this just by design?
>
> > I have the pluginsync and mcollective setup to use /etc/puppetlabs/
> > mcollective/facts.yaml.
>
> > server.cfg
> > # Facts
> > factsource = yaml
> > plugin.yaml = /etc/puppetlabs/mcollective/facts.yaml
>
> > Without doing anything, after a few minutes, my facts.yaml gets
> > populated with the new custom fact, even without following this guide:
>

> >http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/Fact...

[Gary Larizza]

Tony,

No.  Those don't become custom fact values, so client's aren't aware of them.

[Tony C]

I see, but following this page does allow for that

http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/FactsFacterYAML

I just tested it. So in my enterprise, i would much rather control the
facts.yaml using a module of the above listed link and a puppet run,
than allowing a cron job to do this. Also, the parameters defined in
the ENC really help me carve out my infrastructure nicely for my
puppet manifests, as well as the ability to run queries in mcollective
using these parameters. A good examle would be a group in the
Dashboard called PRODUCTION, with a parameter for site=LA


How do people typically manage that puppet generated crontab?

[Gary Larizza]

Our cron entry is declared in the pe_mcollective module that we ship in our modulepath --> /opt/puppet/share/puppet/modules/pe_mcollective/manifests/metadata.pp

I'll answer this more in-depth when I get a free second to expound on it, but I'm a bit tied-up at the moment :)

[Tony C]

No worries Gary. Thanks for your help already. I'm definitely on your
time so just chime in if / when you have a second.

On Feb 16, 4:00 pm, Gary Larizza <g...@puppetlabs.com> wrote:
> Our cron entry is declared in the pe_mcollective module that we ship in our
> modulepath
> --> /opt/puppet/share/puppet/modules/pe_mcollective/manifests/metadata.pp
>
> I'll answer this more in-depth when I get a free second to expound on it,
> but I'm a bit tied-up at the moment :)
>
>
>
>
>
>
>
>
>
> On Thu, Feb 16, 2012 at 3:52 PM, Tony C <tonyjch...@gmail.com> wrote:
> > I see, but following this page does allow for that
>

> >http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/Fact...

[Nan Liu]

On Thu, Feb 16, 2012 at 3:19 PM, Tony C <tonyj...@gmail.com> wrote:
> I'm not sure if this is the right group or not, but i'll start here.
>
> I have Puppet enterprise 2.0, playing around with custom facts.
>
> I have noticed that adding a custom fact to any module will distribute
> that fact to all machines, regardless if they are assigned to that
> module or not. Is there a way around this, or is this just by design?

Gary already pointed out the cron job. I'm not aware of an easy way to
perform limited pluginsync, it's either all or nothing. The reason
this is not possible, puppet need facts to compile catalog to know
what modules belong to a node, and puppet can't compile without facts,
so chicken and egg. For example, puppet can't know if it should
pluginsync my_fact if it's in my_module with the following code:

if $my_fact {include my_module}

HTH,

Nan

[Nan Liu]

On Thu, Feb 16, 2012 at 3:52 PM, Tony C <tonyj...@gmail.com> wrote:
> I see, but following this page does allow for that
>
> http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/FactsFacterYAML
>
> I just tested it. So in my enterprise, i would much rather control the
> facts.yaml using a module of the above listed link and a puppet run,
> than allowing a cron job to do this. Also, the parameters defined in
> the ENC really help me carve out my infrastructure nicely for my
> puppet manifests, as well as the ability to run queries in mcollective
> using these parameters. A good examle would be a group in the
> Dashboard called PRODUCTION, with a parameter for site=LA

I can see scenarios where I wouldn't want to dump the entire ENC
content since there's confidential settings though relevant for that
system, I don't want to make discoverable through mco inventory.

Since you are using PE, another options you can take advantage of
facts.d directory, and write out custom facts in key=value format in
txt file under /etc/puppetlabs/facter/facts.d/

See the puppet_enterprise_installer.txt for example, and this will be
a fairly straightforward file resource using a template.

Thanks,

Nan

[Tony C]

Thanks Nan.

I supposed every one's environment is different. For the facts.d, how
does that work exactly?

On Feb 16, 4:31 pm, Nan Liu <n...@puppetlabs.com> wrote:

> On Thu, Feb 16, 2012 at 3:52 PM, Tony C <tonyjch...@gmail.com> wrote:
> > I see, but following this page does allow for that
>

> >http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/Fact...

[Gary Larizza]

Let me see if I can help on a couple of things:

1.) Facts.dot.d is from this repo --> https://github.com/ripienaar/facter-facts/tree/master/facts-dot-d  Check it out.  You can specify YAML, JSON, Shell, or even a single plain text file where you write 'factname = factval' and they turn into Facter facts.  The facts-dot-d fact just reads a file, parses it, and returns what it parses as Facter facts.  It's cool if you have a central source like a CMDB to scrape for facts, but if you're getting started you might want to start and play with custom facts.

2.) Facts.yaml  So, MCollective, using the YAML agent, will read a YAML file for its facts/inventory information.  Because running facter every time MCollective is called incurs some cost, reading a flat-file is much faster for a tool like MCollective.  That's why we use it.  We also want to make sure the facts are reasonably up-to-date for when you do Live Management or MCollective queries, so we run that cron job every 15 minutes.  You can disable the job if you want and MAKE it so the facts are only updated according to Puppet runs - that's up to how you want to manage your environment.  Yes, that inline_template will also return variables IN SCOPE to the facts.yaml file, but that's something of an 'added bonus' rather than a dependable feature.  If you WANT that sort of flexibility, you should look to creating custom facts or doing some sort of external data lookup (by querying the Console's REST API, for example).  You could also scrape the MySQL DB backend for that information.  It's entirely up to you.  There are many options available to you, but I'm not sure what you're ultimately wanting to achieve?

Hope that helped a bit :)

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Greg]

As far as I know thats true... One option to limit facts is to use
confine to limit where its gets run.

For example, here is a fact that is clearly only applicable for
Solaris hosts:

Facter.add("obpversion") do
confine :kernel => :sunos
setcode do
%x{/usr/sbin/prtconf -V}.chomp.split(" ")[1]
end
end

Whilst this won't stop it from being downloaded, it will mean that the
code will only be run on hosts that meet the requirements.

Hope that helps...



On Feb 17, 11:23 am, Nan Liu <n...@puppetlabs.com> wrote:

[Tony C]

Gary and Greg, wow, thanks a whole lot. I've been reading the same
things you typed, in different posts, but for some reason after
reading your posts, the light bulb went off and almost everything came
together for me. How do I use facts.d? I understand what it does, but
how can I leverage puppet and facter to

Bottom line, it all depends on our requirements. I basically want to
be able to query Mcollective based on facts, such as env, site, and
application. The place i work uses weblogic and they build out a brand
new WLS domain for every application set, so I want to do something
like (excuse my pseudo code here, not a developer)

if app = MyAppSet1,
if site = LA
if env=Dev
do some action

At first I thought I could use custom facts but shot that idea down
because of how it distributes to every puppet client. Now that I have
read about confine, I maybe able to pull off some combination of
confine and group parameters specified in the Puppet Console?

I'm going to play with this today and see what I come up with. I think
there are a few different ways of pulling off what I need, and I do
thank everyone who has put in the time to reply.

- Tony

[Gary Larizza]

Tony,

You might want to look at Hiera and doing this INSIDE Puppet instead of MC.  MC is awesome for orchestration and ad-hoc triggering of jobs, but it sounds like what YOU want to do is ensure a final state of a machine based on its role, environment, and site.

Hiera allows you to specify a hierarchy (such as node-specific, then role, environment, and so on all the way down to a global file that has all your defaults in it) and return data based on the hierarchy that is pertinent to your node.  For example, if you wanted to ensure that all App-nodes had Tomcat, you could have Hiera search through the hierarchy for a 'classes' parameter and return an array of ALL the classes that were declared in every level of the hierarchy.  At the app-node level of the hierarchy, you would have 'tomcat' returned as a value for the 'classes' parameter.  This would allow you to dynamically classify ALL app-nodes to include the tomcat class without having to change a bunch of individual node declarations in Puppet.  It would also let you limit/change behavior based on their environment, role, and so on.  Does this make sense?

Check out the blog post here --> http://puppetlabs.com/blog/first-look-installing-and-using-hiera/ and the repositories at http://github.com/puppetlabs/hiera and http://github.com/puppetlabs/hiera-puppet

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Tony C]

Hey Gary, Thanks for that, I read the link, as well as this article
written by you

http://puppetlabs.com/blog/the-problem-with-separating-data-from-puppet-code/

I think I have 2 use cases here. I need mcollective to do ad-hoc
triggering of jobs and some orchestration based on some criteria, like
role, env and site. I also want my puppet manifests to control the
state of a machine, after this orchestration based on the criteria
used to trigger and ad-hoc job, just as you stated.

For example, for mcollective

role = tomcat
tomcat_installed = no
env=dev
designated_app = MyApp1
then install Tomcat

After the installation, I would want puppet to run, and if those same
criteria matches the logic in my puppet manifest, deploy the proper
config files needed to power that tomcat, in that site, for that
particular application.


So I think it comes down to, where do I put the source of truth. If
hiera uses yaml in the backend, can I use Hiera's yaml file to power /
etc/puppetlabs/mcollective/facts.yaml? I really like the Puppet
Console. Easy to setup, easy to use.

I think I understand how Hiera works. When doing a search, it will
basically look down the chain and return values that will match
whatever you asked for, even if that value is not in the immediate
"level" you are searching against.

My apologies if I am all over the place. Just really amped to getting
this thing off the ground. I feel like I'm right there, I just need to
make a decision.

On Feb 17, 12:19 pm, Gary Larizza <g...@puppetlabs.com> wrote:
> Tony,
>
> You might want to look at Hiera and doing this INSIDE Puppet instead of MC.
>  MC is awesome for orchestration and ad-hoc triggering of jobs, but it
> sounds like what YOU want to do is ensure a final state of a machine based
> on its role, environment, and site.
>
> Hiera allows you to specify a hierarchy (such as node-specific, then role,
> environment, and so on all the way down to a global file that has all your
> defaults in it) and return data based on the hierarchy that is pertinent to
> your node.  For example, if you wanted to ensure that all App-nodes had
> Tomcat, you could have Hiera search through the hierarchy for a 'classes'
> parameter and return an array of ALL the classes that were declared in
> every level of the hierarchy.  At the app-node level of the hierarchy, you
> would have 'tomcat' returned as a value for the 'classes' parameter.  This
> would allow you to dynamically classify ALL app-nodes to include the tomcat
> class without having to change a bunch of individual node declarations in
> Puppet.  It would also let you limit/change behavior based on their
> environment, role, and so on.  Does this make sense?
>
> Check out the blog post here -->http://puppetlabs.com/blog/first-look-installing-and-using-hiera/and the

> repositories athttp://github.com/puppetlabs/hieraandhttp://github.com/puppetlabs/hiera-puppet

[Tony C]

I just read another article in the mcollective-users group and re-read
what you put, and see how I have this wrong.

Puppet can do everything I need as far as

> role = tomcat
> tomcat_installed = no
> env=dev
> designated_app = MyApp1
> then install Tomcat

and the apply the config files.

Mcollective should be used for things like, I want to deploy a new
app, use mcollective to shutdown these specific servers, deploy and
bring them back up, etc etc etc

On Feb 17, 1:13 pm, Tony C <tonyjch...@gmail.com> wrote:
> Hey Gary, Thanks for that, I read the link, as well as this article
> written by you
>

> http://puppetlabs.com/blog/the-problem-with-separating-data-from-pupp...

> > repositories athttp://github.com/puppetlabs/hieraandhttp://github.com/puppetlabs/hie...