Making a "role" fact work

[gareth.hu...@gmail.com]

Hi All,

I'm trying to add a new fact to puppet, which is the role the host has - I'm using the popular roles/profiles methodology, so all nodes have exactly one role::<something> class.  I want to get the <something> into a fact.

I've managed to get it working easily enough, but either not automatically or only on subsequent puppet runs (ie not the first).  This makes perfect sense, given the work flow of collect-facts -> compile catalog -> send-catalog-to-host.

My next attempt is to write a server-side function that can analyse the catalog after it's been built, extract the class names, find one that matches /^role::/ and bob's your uncle.

I'm having trouble getting started though - can anyone help me out with how I might get access to the catalog?  Is there a variable I can lookup, or a function or something?

Is the catalog even available, or are functions run before compilation?  Is there any way I can access a list of classes being applied?

I don't want to require something be done by developers (for example, putting the logic it in role.pp and require all roles to inherit that), as that leaves too much room for mistakes.  I really want it to be automatically done on all hosts, and available from the first run.

I'm not married to parsing the catalog - the end goal here is just getting the fact - any other ideas welcome.

Thoughts?

[Gareth Humphries]

Follow-up - I've managed to extract the node definition which includes the top level classes ("settings" and "role") - I'm presently working on how to extract the data I need from that.

I lookup the node with:

  node=Puppet::Node.indirection.find( lookupvar('fqdn') )

Which works, and if i pp() it I can see it has an attribute: @known_resource_types=TypeCollection{:hostclasses=>["", "settings", "role::gareth"], ..., ... }

So, now I just need to figure out the syntax to extract that data from within my function.  Getting closer!

[Gavin Williams]

Gareth

I've been working through the same issue internally here for a new Cloud based platform, and came up with the solution of pre-loading the machine with the role at the point the machine is provisioned. 

Then when it first checks in with Puppet it's already for the fact present... 

Might be another option?

Cheers

Gav

[Gareth Humphries]

Thanks Gav,

It's a good idea, though on the surface I don't think it will work for us (we're trying to spin stuff up from a gold image using an ENC, and I think having an extra magic step in between is a greater evil than explicit declarations), but you've got me thinking down some other lines of automation.

[Gareth Humphries]

I've made some good progress, and am agonisingly close.

I have the below code which extracts the role into a variable (ugly, but it works).  I had kinda assumed that Facter.Add would work server-side, but I've discovered that it doesn't.  So now, my problem is simple how to make a string calculated in a function available beyond that function?  A fact would be great, a scoped variable would be fine, anything that exposes it in a way that hiera.yaml can get it's grubby little hands on it.

Any ideas?

Code - in dire need of betterification, but it will hopefully save other people a few hours of trial-and-error.

module Puppet::Parser::Functions

  newfunction(:get_role, :type => :rvalue) do |args|

    @roles=[]

    node=Puppet::Node.indirection.find( lookupvar('fqdn') )

    @role_lines=node.inspect.sub( /^.*?"role::/, "").split( /"role::/ )

    @role_lines.each do |rolestr|

      @roles.push rolestr.sub( /".*/, "")

    end

    @role=@roles.max_by(&:length).chomp

#    Facter.add( :tl_role ) do

#      setcode do

#        @role

#      end

#    end

#    $tl_role=@role

    return @role

  end

end

On Thursday, January 28, 2016 at 3:17:00 PM UTC, Gareth Humphries wrote:

[Dirk Heinrichs]

Am 29.01.2016 um 13:03 schrieb Gareth Humphries:

Any ideas?

Can you derive the role from the hostname? I do something similar to determine the location of a machine based on IP address range. So I have lib/facter/location.rb in one of my modules, with the following content:

# Set location of a host by examining its IP address

Facter.add(:location) do
    setcode do
        address = Facter.value(:ipaddress)
        case address
        when /^10\.11\.\d+\.\d+$/
            'Here'
        when /^10\.12\.\d+\.\d+$/
            'There'
        when /^10\.13\.\d+\.\d+$/
            'Somewhere Else'
        else
            'Unknown Location'
        end
    end
end

You could derive the role fact from the hostname, if you name your machines after their purpose.

HTH...

    Dirk

--

Dirk Heinrichs, Senior Systems Engineer, Engineering Solutions
Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach
Tel: +49 2226 1596666 (Ansage) 1149
Email: d...@recommind.com
Skype: dirk.heinrichs.recommind
www.recommind.com

[Gareth Humphries]

Hi Dirk,

That's a good idea, thanks.  In our case we have a lot of legacy hosts and no good naming convention - combined with an eye to a future of cloud-bases services (with undefined hostnames), and I think in our case that approach won't be maintainable.

A good solution without those constraints though.

Gareth

[jcbollinger]

On Friday, January 29, 2016 at 3:29:29 AM UTC-6, Gareth Humphries wrote:

Thanks Gav,

It's a good idea, though on the surface I don't think it will work for us (we're trying to spin stuff up from a gold image using an ENC, and I think having an extra magic step in between is a greater evil than explicit declarations), but you've got me thinking down some other lines of automation.

If you're relying on an ENC, then isn't that ENC assigning roles to machines?  In that case, why do you need a fact?  The ENC can inject top-level variables that you can use exactly as you would use facts.

More generally, if you have centralized knowledge of what role each machine should have, then you should serve it centrally instead of storing it on nodes and relying on them to feed it back correctly.  Hiera would be another option for doing that.

I'm really having trouble understanding why you are approaching the problem as you describe.  If indeed you have a way to assign a role class to your nodes without relying on the fact you're trying to create, then I don't see why you need the fact.  Moreover, I don't see why you need to analyze catalogs to extract the value for such a fact, instead of making Puppet itself manage the fact.  If you rely on external facts for the purpose, you can have your role classes manage an ordinary file on agent-side file system to make that node provide any given fact on subsequent runs.  As a side effect, this could even prevent assigning multiple role classes to the same node.  Indeed, this is one path by which you could arrange for nodes to provide the desired fact with their very first catalog request, though again, I don't understand what purpose the fact is supposed to serve.

John

[Gareth Humphries]

ENC is the end game, but we have legacy hosts this has to work on.  Right now I have site.pp which has a list of unpleasant regexes and an 'include role::<whatever>' stanza for each.  I could put '$role=<something> ; include role::$role' in each of them instead, but I would have to do that in every single case, which I'm trying to avoid.

External facts work, but not on the first run.  Because facts get loaded before catalog compilation, the host doesn't know what to set that fact to until it already has a catalog -  a little bit chicken-and-egg.

If i'm relying on the role fact to get data out of hiera, I need that fact available first run or compilation will fail.

Perhaps the set-the-variable-everywhere approach is going to be the solution, i was just hoping to find a way that doesn't require that.

[R.I.Pienaar]

----- Original Message -----
> From: "Gareth Humphries" <gareth.hu...@gmail.com>
> To: "puppet-users" <puppet...@googlegroups.com>
> Sent: Friday, January 29, 2016 4:50:50 PM
> Subject: [Puppet Users] Re: Making a "role" fact work

> ENC is the end game, but we have legacy hosts this has to work on. Right
> now I have site.pp which has a list of unpleasant regexes and an 'include
> role::<whatever>' stanza for each. I could put '$role=<something> ;
> include role::$role' in each of them instead, but I would have to do that
> in every single case, which I'm trying to avoid.
>
> External facts work, but not on the first run. Because facts get loaded
> before catalog compilation, the host doesn't know what to set that fact to
> until it already has a catalog - a little bit chicken-and-egg.
> If i'm relying on the role fact to get data out of hiera, I need that fact
> available first run or compilation will fail.

facts are available from first run.

Plugin sync happens first and then it requests a catalog including the newly
received facts.

[Luke Bigum]

This might be relevant:

https://groups.google.com/forum/#!searchin/puppet-users/luke$20bigum|sort:date/puppet-users/XWAcm152cyQ/P_rpi50XBAAJ

The ENC above inserts a top scope variable into a node's manifest, designed to be used as a "role" Fact. It reads from one of two YAML files, either explicit hostnames or hostname regex matches. It might meet your requirement half way before you get a "proper" ENC in place.

[Gareth Humphries]

Normally, yes, but in this case it needs a class list locally to populate the fact, and it doesn't have that until after a run.

Facts get sent to the master, where classes are allocated and a catalog compiled, then the catalog sent back to the client.  Without the initial catalog, the client has no knowledge of the classes, and can't set the fact accordingly.

If I could set the fact on the server I'd be away, but I haven't yet found a way to get that working.

This is proving harder than initially thought.  I think I might have to shelve it.   :-(

[Gareth Humphries]

Thanks Luke, I'll give that a whirl.  May prove helpful indeed!

[R.I.Pienaar]

----- Original Message -----
> From: "Gareth Humphries" <gareth.hu...@gmail.com>
> To: "puppet-users" <puppet...@googlegroups.com>
> Sent: Friday, January 29, 2016 5:14:12 PM
> Subject: Re: [Puppet Users] Re: Making a "role" fact work

> Normally, yes, but in this case it needs a class list locally to populate
> the fact, and it doesn't have that until after a run.
> Facts get sent to the master, where classes are allocated and a catalog
> compiled, then the catalog sent back to the client. Without the initial
> catalog, the client has no knowledge of the classes, and can't set the fact
> accordingly.
>
> If I could set the fact on the server I'd be away, but I haven't yet found
> a way to get that working.
>
>
> This is proving harder than initially thought. I think I might have to
> shelve it. :-(

so that's not a problem with facts, it's a problem with your fact :)

[Jason Slagle]

On 1/29/16, 11:14 AM, "Gareth Humphries" <puppet...@googlegroups.com on behalf of gareth.hu...@gmail.com> wrote:

Normally, yes, but in this case it needs a class list locally to populate the fact, and it doesn't have that until after a run.

Facts get sent to the master, where classes are allocated and a catalog compiled, then the catalog sent back to the client.  Without the initial catalog, the client has no knowledge of the classes, and can't set the fact accordingly.

If I could set the fact on the server I'd be away, but I haven't yet found a way to get that working.

This is proving harder than initially thought.  I think I might have to shelve it.   :-(

I’m also having a hard time understanding the end goal.

It seems like you have information based on the classes assigned to a host that determine it’s role.  You want to use those classes to set a role fact on the node.

What I don’t understand is that if you have a defined set of logic you can apply to determine the role, who you need a role fact at all?  What in the classes is required for you to determine this?

Jason

[Gareth Humphries]

Absolutely.  I'm trying to do something weird, and that's why I'm failing.  Let there be no doubt about that.

[jcbollinger]

On Friday, January 29, 2016 at 9:50:50 AM UTC-6, Gareth Humphries wrote:

ENC is the end game, but we have legacy hosts this has to work on.  Right now I have site.pp which has a list of unpleasant regexes and an 'include role::<whatever>' stanza for each.  I could put '$role=<something> ; include role::$role' in each of them instead, but I would have to do that in every single case, which I'm trying to avoid.

I think if you had put the same amount of effort into porting those legacy node regexes to an ENC then you'd probably be done by now.

In fact (no pun intended), you could perhaps even more easily pull the node regexes out to a top-level statement or block at the beginning of your site.pp that assigns a value to the role variable.

 

External facts work, but not on the first run.

As far as I know, external facts work even on the first run, just like all other facts.  What would not work on the first run is using Puppet to manage the external fact file as a normal file, but you can bootstrap that process by dropping the appropriate file into place manually.  You could even drop in a standard file that assigns a role that means "I don't know my own role", and let Puppet take it from there by updating that file as appropriate (and probably not doing much else on that first run).  As I describe below, however, even that's probably unnecessary.

In any case, something along these lines would still be a great deal better than parsing the node's catalog to compute the fact value, as you started out describing.  You could seed nodes with initial catalogs, just as you could seed them with external fact files, to work around the first run problem, but an external fact file would be much simpler to build and maintain, and the external fact facility is a public interface exposed by Puppet.  The catalog file format is an internal interface.

 

 Because facts get loaded before catalog compilation, the host doesn't know what to set that fact to until it already has a catalog -  a little bit chicken-and-egg.

If i'm relying on the role fact to get data out of hiera, I need that fact available first run or compilation will fail.

Nonsense.  In the first place, let's be clear: the catalog builder knows only about the variable $::role.  It neither knows nor cares how that variable is assigned its value (if indeed it is assigned any value), so you do not specifically need a fact.  In the second place, it cannot be too hard to condition evaluation of your site manifest on whether $::role has been assigned a non-empty value.  You could certainly make it build and deliver a bootstrapping catalog in the event that no $::role variable is defined.

Moreover, relying on your nodes themselves to tell you what configuration they are supposed to have is dreadful, at least from a security perspective.  This exact problem is why ENCs override nodes with respect to environment assignment.

 

Perhaps the set-the-variable-everywhere approach is going to be the solution, i was just hoping to find a way that doesn't require that.

I suppose by that you mean putting an assignment statement in each node block.  I don't think that's actually so bad, but it's also not among the things I proposed.  Making your ENC do the job would probably be best, and I still don't understand why that should be a big problem -- even if you have to write a simple ENC from scratch.  If all it has to do is assign roles to nodes then your existing site manifest already has most of the logic you need.


John