how to conditionally add users to a virtualized group?

[Jo Rhett]

I'm fighting with a ticklish issue.  We have some groups and users that only belong on some systems. So we made all users virtual and then realize them in classes specific to those system types.  This works quite well for the users, but not for the groups. When you specify a user, you have to list all the groups they are in. 

groups => ['support',ops','dev'],

 Obviously some groups aren't realized on all systems, so this produces an error when usermod is run.

'/usr/sbin/usermod -G support,ops,dev jrhett' returned 6: usermod: unknown group dev
usermod: unknown group dev

So I tried to get smarter, and put logic to add the group to each member under the appropriate class

Class users::dev inherits users { 

User['jrhett'] { groups +> ['dev'] }

}

This works… almost. It works for all instances where the user is only subclassed once. But if I do the same technique in multiple classes I get 

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Parameter 'groups' is already set on User_and_key[jrhett] by #<Puppet::Resource::Type:0x7f4feed2d828> at /etc/puppet/modules/users/manifests/support.pp:22; cannot redefine at /etc/puppet/modules/users/manifests/dev.pp:27 on node s2-d1.company.com

So how can this be achieved, short of using an exec with an unless doing another exec to determine if the group exists?

-- 

Jo Rhett

Net Consonance : net philanthropy to improve open source and internet projects.

[Denmat]

Puppet users and groups are fiddly. My current not implemented thinking is to use ldap and manage pam_groups via puppet on the hosts to get the granularity. 

More thinking out loud than anything else.

Den

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Christopher Wood]

I use nss-pam-ldapd and pam_ldap depending on the system, using an ldap filter to allow only certain groups per system. I prefer nss-pam-ldapd.

nss-pam-ldapd:

CentOS 6
Debian 6
Ubuntu 10.04

pam_ldap:

CentOS 5
FreeBSD 9

(Solaris is more like pam_ldap in configuration, but fairly unique.)

The manifests to deal with the above are essentially OS-specific.

On Thu, Jul 12, 2012 at 05:52:24PM +1000, Denmat wrote:
> Puppet users and groups are fiddly. My current not implemented thinking is
> to use ldap and manage pam_groups via puppet on the hosts to get the
> granularity. 
> More thinking out loud than anything else.
> Den
>

> On 12/07/2012, at 6:03, Jo Rhett <[1]jrh...@netconsonance.com> wrote:
>
> I'm fighting with a ticklish issue.  We have some groups and users that
> only belong on some systems. So we made all users virtual and then
> realize them in classes specific to those system types.  This works
> quite well for the users, but not for the groups. When you specify a
> user, you have to list all the groups they are in. 
> groups => ['support',ops','dev'],
>  Obviously some groups aren't realized on all systems, so this produces
> an error when usermod is run.
> '/usr/sbin/usermod -G support,ops,dev jrhett' returned 6:
> usermod: unknown group dev
> usermod: unknown group dev
> So I tried to get smarter, and put logic to add the group to each member
> under the appropriate class
> Class users::dev inherits users { 
> User['jrhett'] { groups +> ['dev'] }
> }
> This works� almost. It works for all instances where the user is only
> subclassed once. But if I do the same technique in multiple classes I
> get 
> err: Could not retrieve catalog from remote server: Error 400 on SERVER:
> Parameter 'groups' is already set on User_and_key[jrhett] by
> #<Puppet::Resource::Type:0x7f4feed2d828> at
> /etc/puppet/modules/users/manifests/support.pp:22; cannot redefine at
> /etc/puppet/modules/users/manifests/dev.pp:27 on node

> [2]s2-d1.company.com

> So how can this be achieved, short of using an exec with an unless doing
> another exec to determine if the group exists?
> -- 
> Jo Rhett
> Net Consonance : net philanthropy to improve open source and internet
> projects.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.

> To post to this group, send email to [3]puppet...@googlegroups.com.

> To unsubscribe from this group, send email to

> [4]puppet-users...@googlegroups.com.

> For more options, visit this group at

> [5]http://groups.google.com/group/puppet-users?hl=en.

>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>

> References
>
> Visible links
> 1. mailto:jrh...@netconsonance.com
> 2. http://s2-d1.company.com/
> 3. mailto:puppet...@googlegroups.com
> 4. mailto:puppet-users...@googlegroups.com
> 5. http://groups.google.com/group/puppet-users?hl=en

[Felix Frank]

Hi,

On 07/11/2012 10:03 PM, Jo Rhett wrote:
> So I tried to get smarter, and put logic to add the group to each member
> under the appropriate class
> Class users::dev inherits users {
> User['jrhett'] { groups +> ['dev'] }
> }
>

> This works� almost. It works for all instances where the user is only

> subclassed once. But if I do the same technique in multiple classes I get

sound approach, but I've hit this wall a couple of times as well.

I've resorted to horrors that would add items to array variables that
are declared in a central, well-known class, and use the final value for
the resources in question. Depending on how much flexibility is
required, this may not be feasible at all.

Perhaps hiera can be used to do something clever here?

Cheers,
Felix

[jcbollinger]

If it is the case that each user always has the same potential secondary groups, and you need to narrow the actual secondary groups to those that are actually present, then I think you could do it without too much pain.  The main ingredients would be a list (array) of the groups that are supposed to be present, and a custom function that forms the intersection of two arrays.  (Or you could use an inline template and split(), but yuck!)

Hiera would probably provide a good means for building the list of available groups, which you could then use not only to filter user definitions but also to drive virtual group realization.  Here's a skeleton of how it might work:

class auth::constants {
  $available_groups = hiera('groups')
}

class auth::groups::virtual {
  # Virtual group declarations, such as
  @group { 'dev':
    gid => 4242,
    ensure => present
  }
}

define auth::concrete_group () {
  include 'auth::groups::virtual'
  realize Group[$name]
}

class auth::groups {
  include 'auth::constants'

  auth::concrete_group { $auth::constants::available_groups: }
}

class auth::users::virtual {
  include 'auth::constants'

  # Virtual user declarations, such as
  @user { 'jbolling':
    uid => 4200,
    gid => 4200,
    groups => intersect(['dev', 'support', 'ops'], $auth::constants::available_groups)
  }
}

A few bits are omitted, most notably user realization.  The main concept is to declare what you want in the first place, rather than throwing up something and trying to tweak it afterward, or trying to build values incrementally.  The latter two approaches tends to work poorly in Puppet (with certain caveats).

Note also that the above is completely hypothetical.  I think it would work, but it's not based on anything I have actually implemented.


John

[Jo Rhett]

That's great if you have centralized and co-hosted infrastructure and are willing to accept the dependancy. Given that this is a small need for a small number of users on a very small amount of systems (like 3 out of hundreds) without a centralized backbone between them, implementing LDAP makes little sense.

[Jo Rhett]

On Jul 12, 2012, at 4:30 AM, Felix Frank wrote:

On 07/11/2012 10:03 PM, Jo Rhett wrote:

So I tried to get smarter, and put logic to add the group to each member

under the appropriate class

Class users::dev inherits users {

User['jrhett'] { groups +> ['dev'] }

}

This works… almost. It works for all instances where the user is only

subclassed once. But if I do the same technique in multiple classes I get

sound approach, but I've hit this wall a couple of times as well.

I've resorted to horrors that would add items to array variables that
are declared in a central, well-known class, and use the final value for
the resources in question. Depending on how much flexibility is
required, this may not be feasible at all.

Hm. That might work, but seems even uglier :(

Perhaps hiera can be used to do something clever here?

This is actually something that hiera seems perfect for, but we simply don't have any backend dataset from which to derive hiera data at this time. That is going to change, and I'm looking forward to having hiera access at that point.

[Jo Rhett]

On Jul 12, 2012, at 6:46 AM, jcbollinger wrote:

If it is the case that each user always has the same potential secondary groups, and you need to narrow the actual secondary groups to those that are actually present, then I think you could do it without too much pain.  The main ingredients would be a list (array) of the groups that are supposed to be present, and a custom function that forms the intersection of two arrays.  (Or you could use an inline template and split(), but yuck!)

Hiera would probably provide a good means for building the list of available groups, which you could then use not only to filter user definitions but also to drive virtual group realization.  Here's a skeleton of how it might work:

class auth::constants {
  $available_groups = hiera('groups')
}

Interesting idea, but depends on an external datasource that tells us which groups are valid.  Since all of these groups are already defined in puppet, I simply don't see the value of managing intersections of data between a hiera data source and puppet.

  # Virtual user declarations, such as
  @user { 'jbolling':
    uid => 4200,
    gid => 4200,
    groups => intersect(['dev', 'support', 'ops'], $auth::constants::available_groups)
  }
}

I think the intersect idea is valid, as long as I can find out if a parameter is realized or not.  Basically, write a function that removes from the array any group which isn't realized. This removes any need for heira.  However I'm poking around and the docs don't show any methods to determine if something has been realized or not.

If I am reading this right, intersect is provided by stdlib, right?  So I really just need to write a function to determine if something is realized or not. I suspect this is going to fall back to the same issues as defined() unless I can delay execution until the end.

[jcbollinger]

On Thursday, July 12, 2012 1:42:28 PM UTC-5, Jo wrote:

On Jul 12, 2012, at 6:46 AM, jcbollinger wrote:

If it is the case that each user always has the same potential secondary groups, and you need to narrow the actual secondary groups to those that are actually present, then I think you could do it without too much pain.  The main ingredients would be a list (array) of the groups that are supposed to be present, and a custom function that forms the intersection of two arrays.  (Or you could use an inline template and split(), but yuck!)

Hiera would probably provide a good means for building the list of available groups, which you could then use not only to filter user definitions but also to drive virtual group realization.  Here's a skeleton of how it might work:

class auth::constants {
  $available_groups = hiera('groups')
}

Interesting idea, but depends on an external datasource that tells us which groups are valid.  Since all of these groups are already defined in puppet, I simply don't see the value of managing intersections of data between a hiera data source and puppet.

No, it doesn't depend on an external datasource; rather, It depends on up-front knowledge of which groups are supposed to be realized for the node.  Although I proposed using an external datasource to provide that data, it could just as well be provided by an ENC or determined via DSL code based on conditionals, node facts, etc.  Even class parameters.

 

  # Virtual user declarations, such as
  @user { 'jbolling':
    uid => 4200,
    gid => 4200,
    groups => intersect(['dev', 'support', 'ops'], $auth::constants::available_groups)
  }
}

I think the intersect idea is valid, as long as I can find out if a parameter is realized or not.  Basically, write a function that removes from the array any group which isn't realized. This removes any need for heira.  However I'm poking around and the docs don't show any methods to determine if something has been realized or not.

If I am reading this right, intersect is provided by stdlib, right?

If so, then I'm somehow overlooking it.  My suggestion and expectation was that you would create it yourself, but it seems sufficiently general-purpose that you might find something suitable already made.  You can also, of course, jerry-rig something based on inline_template().
 

 So I really just need to write a function to determine if something is realized or not. I suspect this is going to fall back to the same issues as defined() unless I can delay execution until the end.

I would avoid that variation on this approach if at all possible.  You would sidestep multiple pitfalls if you could determine up front, based on node name and facts, which groups are supposed to be present, instead of attempting to determine after the fact which were realized.  Indeed, you might even find it convenient to use that information to drive group realization.  If nothing else, doing so would ensure that users aren't assigned to secondary groups that don't get realized.


John

[Jo Rhett]

On Jul 12, 2012, at 2:26 PM, jcbollinger wrote:

I would avoid that variation on this approach if at all possible.  You would sidestep multiple pitfalls if you could determine up front, based on node name and facts, which groups are supposed to be present, instead of attempting to determine after the fact which were realized.  Indeed, you might even find it convenient to use that information to drive group realization.

If nothing else, doing so would ensure that users aren't assigned to secondary groups that don't get realized.

This is what policy as expressed in the puppet manifests does. I don't see how to avoid the unrealized problem here.

What's funny is that you are expressing exactly what puppet does today, but it appears you are suggesting that I need to create another data source and mirror the information out of puppet manifests into that for comparison purposes. Huh?

I'm a bit baffled by the fairly constant suggestion by people here that I keep spreading out the places where information is stored. The point is to centralize the data, not provide more sources to grow inconsistent with each other.

[jcbollinger]

Relying on a single source of information is exactly what what I have suggested you do, specifically by using an up-front group list both to filter users' declared secondary groups and to drive which groups get realized.  I have described that three times now, and it's included in the example code I posted earlier.  You can populate such a list by whatever means you want and from whatever source you want, and you can store it wherever you want, so long as you produce the entire list before any part of it is needed.

So no, I'm not suggesting you mirror information from your puppet manifests.  Rather, I am suggesting that you move implicit information out of your manifests to someplace more accessible.  Study my example if you still don't understand what I mean by that.  The "someplace" where the information lands could be an explicit expression elsewhere in your manifests, or it could be external, as seems best to you. The information implicitly encoded in the structure of your manifests and/or developed during compilation is inherently difficult to use from within the manifests themselves, and if you insist on using it anyway then you're choosing to be stuck in an uncomfortable position.

More generally, people recommending various possible data sources to you -- hiera, ENC, etc. -- are not implying that you should "spread out" your data.  That's a function of your own manifest designs and how you use the data.  You do a disservice to those volunteering their help to you by criticizing them for deficiencies in your imagined applications of their suggestions.


John

[Felix Frank]

On 07/13/2012 05:02 PM, jcbollinger wrote:
> More generally, people recommending various possible data sources to you
> -- hiera, ENC, etc. -- are not implying that you should "spread out"
> your data. That's a function of your own manifest designs and how you
> use the data. You do a disservice to those volunteering their help to

> you by criticizing them for deficiencies in /your imagined applications/
> of their suggestions.

Though he did put it quite bluntly, I do believe that Jo has a point.

The thing is, I generally want my manifests to be clever about some
things. When I include my mysql development class, I may want to realize
a couple of users and groups as a result (hypothetically speaking - I
have no such class nor such requirements, but there are other things in
this vein).

I would tell hiera to have puppet include the mysql development class,
not each single user and group. That would strike me as silly.

So there *is* value in constructing information inside the manifest from
outside information, or at least that is my firm belief.

Configuring roles and other settings of larger granularity is what I
want to do in hiera. Picking single user accounts - probably not so much.

Cheers,
Felix

[jcbollinger]

On Friday, July 13, 2012 10:17:25 AM UTC-5, Felix.Frank wrote:

On 07/13/2012 05:02 PM, jcbollinger wrote:
> More generally, people recommending various possible data sources to you
> -- hiera, ENC, etc. -- are not implying that you should "spread out"
> your data.  That's a function of your own manifest designs and how you
> use the data.  You do a disservice to those volunteering their help to
> you by criticizing them for deficiencies in /your imagined applications/
> of their suggestions.

Though he did put it quite bluntly, I do believe that Jo has a point.

Do you mean that it would be useful to have a reliable way for manifests to extract information about declarations appearing some unspecified elsewhere?  I don't dispute that, but the fact is that Puppet does not have such a mechanism, and I don't see any likelihood that it will have one soon.  The point is that that limitation does not create a need for data duplication, despite Jo's assertions to the contrary.
 

The thing is, I generally want my manifests to be clever about some
things. When I include my mysql development class, I may want to realize
a couple of users and groups as a result (hypothetically speaking - I
have no such class nor such requirements, but there are other things in
this vein).

I don't mean to suggest that Puppet already does the best it is possible to do in this area.  Indeed, my comments to Jo are entirely about working within Puppet's current constraints, not about wishlist features.

Nevertheless, the essential problem is a hard one: to determine what declarations satisfying certain criteria will have been made by the end of catalog compilation.  That's the underlying problem with using defined(), using a hypothetical realized() function, building values cooperatively, and perhaps other potentially useful things.  It would be convenient to be able to do those safely and reliably, but solving the key problem would likely require a fundamental change in the manifest compiler's architecture.
 

I would tell hiera to have puppet include the mysql development class,
not each single user and group. That would strike me as silly.

Sure, but I'm not seeing how that relates.  A more parallel situation would be if in addition, some unrelated class wanted to be able to determine which users and groups the mysql development class had declared.  As Puppet now stands, the best way would be for the mysql development class to provide that data in class variables, or else to have obtained it from some shared source in the first place.  The point is that neither of those options requires that data to be duplicated in the structure of the class.

John

[Jo Rhett]

On Jul 13, 2012, at 8:02 AM, jcbollinger wrote:

Relying on a single source of information is exactly what what I have suggested you do, specifically by using an up-front group list both to filter users' declared secondary groups and to drive which groups get realized.  I have described that three times now, and it's included in the example code I posted earlier.  You can populate such a list by whatever means you want and from whatever source you want, and you can store it wherever you want, so long as you produce the entire list before any part of it is needed.

I did not see that from what you showed. Your example didn't show how to aggregate or use the data at all. I saw six classes that were significantly more complex and appeared to require defining the data in multiple places. There was certainly no obvious way this would reduce my data sources.

So no, I'm not suggesting you mirror information from your puppet manifests.  Rather, I am suggesting that you move implicit information out of your manifests to someplace more accessible.  Study my example if you still don't understand what I mean by that.  The "someplace" where the information lands could be an explicit expression elsewhere in your manifests, or it could be external, as seems best to you. The information implicitly encoded in the structure of your manifests and/or developed during compilation is inherently difficult to use from within the manifests themselves, and if you insist on using it anyway then you're choosing to be stuck in an uncomfortable position.

I hear what you are saying, but I really don't see how your example makes this idea clear. I saw multiple sets of classes relying on each other's data in an unreadable manner.

I would argue that even if it does do what I meant, the very fact that I couldn't read it to understand this ensures nobody else here has a chance at maintaining it. "More complex" is not a desired trait here.

In general I see ENCs as eventually providing a way to simplify the data input, but that's not what I've seen recommended or demonstrated. The case for ENCs would be made a lot stronger if some good examples of ways to simply via the use of ENCs were posted.

More generally, people recommending various possible data sources to you -- hiera, ENC, etc. -- are not implying that you should "spread out" your data.  That's a function of your own manifest designs and how you use the data.  You do a disservice to those volunteering their help to you by criticizing them for deficiencies in your imagined applications of their suggestions.

I could go back and make a line by line review of every single time people have told me that I should take data from the puppet manifests and reinforce it / control it via data from an ENC. There hasn't been a single situation where someone said what you are suggesting -- hey, pull this all out of puppet and incorporate it this way (x) so that you can get what you want.  It has always been how to do half the job in puppet and half the job in something else, and manually manage the dependancies between the two.

[Felix Frank]

John,

On 07/13/2012 06:38 PM, jcbollinger wrote:
>
> I would tell hiera to have puppet include the mysql development class,
> not each single user and group. That would strike me as silly.
>
>
> Sure, but I'm not seeing how that relates. A more parallel situation
> would be if in addition, some unrelated class wanted to be able to
> determine which users and groups the mysql development class had
> declared. As Puppet now stands, the best way would be for the mysql
> development class to provide that data in class variables, or else to
> have obtained it from some shared source in the first place. The point
> is that neither of those options requires that data to be duplicated in
> the structure of the class.

I'm not sure I concur with your conclusion, nor with what you boil the
problem at hand down to.

Puppet *does* have means to implement business logic that adds resources
as implications of roles or aspects, and that is by the singleton nature
of classes. You can add users via implication of different possible
aspects of nodes by having the "aspect" class include the approprate
user class(es) for example.

Puppet even allows to make adjustments to such implied resources by
means of subclasses. If a node has a very certain role, the respective
class will include a subclass of some other feature to specialize its
resources appropriately.

All this is according to what puppet has so far been designed to do.

The problem at hand can be handled to a degree with these language
features, as the OP has successfully done already. The approach has
scaling issues, so a workaround is currently needed. I don't see why the
approach should be discarded as "not currently implemented" out of hand,
seeing as the better part of it *is* in fact possible.

Felix

[jcbollinger]

Felix,

On Monday, July 16, 2012 2:44:58 AM UTC-5, Felix.Frank wrote:

John,

On 07/13/2012 06:38 PM, jcbollinger wrote:
>
>     I would tell hiera to have puppet include the mysql development class,
>     not each single user and group. That would strike me as silly.
>
>
> Sure, but I'm not seeing how that relates.  A more parallel situation
> would be if in addition, some unrelated class wanted to be able to
> determine which users and groups the mysql development class had
> declared.  As Puppet now stands, the best way would be for the mysql
> development class to provide that data in class variables, or else to
> have obtained it from some shared source in the first place.  The point
> is that neither of those options requires that data to be duplicated in
> the structure of the class.

I'm not sure I concur with your conclusion, nor with what you boil the
problem at hand down to.

I didn't define the problem, Jo did.  He defined at least two different problems, in fact, so perhaps that's where you and I have gotten out of sync.  The first problem was to build the value of a resource property based on declarations in multiple unrelated classes.  Jo then went on to object to some proposed alternative solutions on the basis that they introduced duplication of data already implicit in his class and resource declarations, making avoiding such duplication the second problem.
 

Puppet *does* have means to implement business logic that adds resources
as implications of roles or aspects, and that is by the singleton nature
of classes. You can add users via implication of different possible
aspects of nodes by having the "aspect" class include the approprate
user class(es) for example.

Puppet even allows to make adjustments to such implied resources by
means of subclasses. If a node has a very certain role, the respective
class will include a subclass of some other feature to specialize its
resources appropriately.

All this is according to what puppet has so far been designed to do.

The problem at hand can be handled to a degree with these language
features, as the OP has successfully done already. The approach has
scaling issues, so a workaround is currently needed. I don't see why the
approach should be discarded as "not currently implemented" out of hand,
seeing as the better part of it *is* in fact possible.

The fact that the approach doesn't work isn't sufficient?  Until now, my focus has been on something that Jo could actually use, rather than on advocacy for new Puppet features.  If you have a good way to make that approach work for Jo, with current Puppet, then I would be genuinely interested in hearing it.  I suppose Jo would be even more so.

I'm not suggesting that Puppet shouldn't have a feature that allowed Jo to do what he wants, approximately how he wants, without parse-order dependencies or other serious drawbacks.  I would love it if there were such a feature.  In fact, I think it would serve a lot of people well, and allow clearer, simpler manifests.

That feature isn't available today, as far as I know, but I have an idea how it might look.  In fact, I had it about six months ago.  Remember resource constraints (https://groups.google.com/forum/?fromgroups#!topic/puppet-users/Fvl0aOe4RPE)?  The idea arose as an approach to cross-module dependencies, but it would address this problem as well.  In fact, this problem is pretty similar to a cross-module dependency issue, and might even be one.


John

[Felix Frank]

On 07/16/2012 04:14 PM, jcbollinger wrote:
> The fact that the approach doesn't work isn't sufficient? Until now, my
> focus has been on something that Jo could actually use, rather than on
> advocacy for new Puppet features. If you have a good way to make that
> approach work for Jo, with current Puppet, then I would be genuinely
> interested in hearing it. I suppose Jo would be even more so.

I cannot, of course, but I do sympathize with Jo's notion that in order
to solve the apparently small problem of making resource overrides
scale, he is now required to rework most if not all of his manifests to
play with a hiera based approach.
I'm not saying there is a simpler solution right now. And we've already
agreed there should be:

> I'm not suggesting that Puppet shouldn't have a feature that allowed Jo
> to do what he wants, approximately how he wants, without parse-order
> dependencies or other serious drawbacks. I would love it if there were
> such a feature. In fact, I think it would serve a lot of people well,
> and allow clearer, simpler manifests.
>
> That feature isn't available today, as far as I know, but I have an idea
> how it might look. In fact, I had it about six months ago. Remember
> resource constraints
> (https://groups.google.com/forum/?fromgroups#!topic/puppet-users/Fvl0aOe4RPE)?
> The idea arose as an approach to cross-module dependencies, but it would
> address this problem as well. In fact, this problem is pretty similar
> to a cross-module dependency issue, and might even be one.

I do. Sometimes even wondered if that made it to some pin-board down at
puppetlabs ;-)

But I'm not sure they're the best solution for the problem at hand. My
understanding is that you would have each class declare constraints on
given users' group memberships, e.g. user jo needs to be in the devs group.
The only real advantage I see here is that, yes, you can make the
compilation fail if your hiera data is not consistent with your business
logic. That doesn't really simplify the way to get to the desired catalog.

Am I missing a wholly different implied intention of your's here?

Cheers,
Felix

[Jo Rhett]

On Jul 16, 2012, at 8:42 AM, Felix Frank wrote:

I cannot, of course, but I do sympathize with Jo's notion that in order
to solve the apparently small problem of making resource overrides
scale, he is now required to rework most if not all of his manifests to
play with a hiera based approach.

Well, more matter of factly, that shifting to a hiera-based approach would require us to manage very carefully the balance of data between puppet and hiera, and manage by eyeball the dependencies between the two. There is considerable resistance to this idea here.

If it was possible to put all the user and group information in hiera and then put the assignment/management of that information into puppet then we could probably manage that. But having to edit "this host gets the sql server info" in puppet and then "these users get put in mysql group on this host" in hiera is completely nonfunctional, and I've seen no examples of ways to bridge that gap.

[Christopher Wood]

(inline)

Possibly something like the following pseudocode example? The main point being to only include a puppet class if there's a certain piece of data in hiera.

node default {
if hiera('usemysql') {
include mysql::service
}
if hiera_array('users') {
include users
}
}

(I haven't tested the above myself. We're still not using hiera at work, more's the pity.)

> -- 
> Jo Rhett
> Net Consonance : net philanthropy to improve open source and internet
> projects.
>

[Jo Rhett]

On Jul 16, 2012, at 11:55 AM, Christopher Wood wrote:

Possibly something like the following pseudocode example? The main point being to only include a puppet class if there's a certain piece of data in hiera.

node default {
 if hiera('usemysql') {
   include mysql::service
 }
 if hiera_array('users') {
   include users
 }
}

I'm not sure how this would work.  So you're now talking about putting all the if/then logic inside hiera?

(I haven't tested the above myself. We're still not using hiera at work, more's the pity.)

We aren't, because we have no external datasource for this stuff and every example we've seen (like yours above) indicates that we're going to have to put half of the logic engine of puppet inside the data source, which means it needs to be a very complex thing that enforces the structure and somehow ties it with the puppet logic. Our analysis so far is that to implement hiera we're going to have to write our own software platform which manages hiera data and writes out puppet policies on the fly when the data changes.

[Felix Frank]

On 07/16/2012 10:29 PM, Jo Rhett wrote:
> We aren't, because we have no external datasource for this stuff and
> every example we've seen (like yours above) indicates that we're going
> to have to put half of the logic engine of puppet inside the data
> source, which means it needs to be a very complex thing that enforces
> the structure and somehow ties it with the puppet logic. Our analysis so
> far is that to implement hiera we're going to have to write our own
> software platform which manages hiera data and writes out puppet
> policies on the fly when the data changes.

Not quite. I believe that the canonical approach is to move your node ->
roles relation into hiera. This way you need little "individual"
manifest code per node.
You certainly need a means to manage hiera's datastores, but I don't
think generating manifests is required.

Anyway, as participants in this thread seem to agree, the approach is
limited insofar that puppet has some shortcomings in the area of
constructing complex derived data (e.g. groups for given users) from
complex and disjoint pieces of input data.

Since you're going to end up with some sort of kludge anyway (imho), I
believe it's perfectly fine for you to forego the hiera backend at this
time and apply the kludge inside your comfort zone, for what it's worth.

Cheers,
Felix

[jcbollinger]

Apparently so.  I don't want to drag this thread off into a rehash of the constraints idea, but one of the central ideas is that it allows cooperative specification of resource properties.  Constraints -- as I envision them -- are not a dynamic validation feature, but rather an indirect, deferred declaration feature.  In many cases, explicit resource declarations could be replaced by one or more constraints on the same resource, which could appear anywhere in the manifest set.  Everything gets resolved after all resources are compiled.

I'll say no more about that here, but if anyone wants to discuss it further then I'd be likely to respond to a new thread on that topic.


John

[Jo Rhett]

On Jul 17, 2012, at 12:54 AM, Felix Frank wrote:

On 07/16/2012 10:29 PM, Jo Rhett wrote:

We aren't, because we have no external datasource for this stuff and

every example we've seen (like yours above) indicates that we're going

to have to put half of the logic engine of puppet inside the data

source, which means it needs to be a very complex thing that enforces

the structure and somehow ties it with the puppet logic. Our analysis so

far is that to implement hiera we're going to have to write our own

software platform which manages hiera data and writes out puppet

policies on the fly when the data changes.

Not quite. I believe that the canonical approach is to move your node ->
roles relation into hiera. This way you need little "individual"
manifest code per node.
You certainly need a means to manage hiera's datastores, but I don't
think generating manifests is required.

Is this not the epitome of diverse and redundant dependancies? I can't edit my hiera data without evaluating puppet manifests, I can't edit the puppet manifests without editing the hiera data…

In short, I'll look at hiera as soon as I have time to build out a whole new infrastructure for data management. And trust me, free time is something I have lots of. (not)

[Jo Rhett]

On Jul 17, 2012, at 6:30 AM, jcbollinger wrote:

Apparently so.  I don't want to drag this thread off into a rehash of the constraints idea, but one of the central ideas is that it allows cooperative specification of resource properties.  Constraints -- as I envision them -- are not a dynamic validation feature, but rather an indirect, deferred declaration feature.  In many cases, explicit resource declarations could be replaced by one or more constraints on the same resource, which could appear anywhere in the manifest set.  Everything gets resolved after all resources are compiled.

Sounds like treating hiera data as virtualized to me (and sounds like a functional way to deal with the issues we are discussing).  How would you implement this today?

I'll say no more about that here, but if anyone wants to discuss it further then I'd be likely to respond to a new thread on that topic.

Seems like a thread that you should name, unless you want a thread labelled "Ask John" … ;-)

[jcbollinger]

On Tuesday, July 17, 2012 12:35:34 PM UTC-5, Jo wrote:

On Jul 17, 2012, at 6:30 AM, jcbollinger wrote:

Apparently so.  I don't want to drag this thread off into a rehash of the constraints idea, but one of the central ideas is that it allows cooperative specification of resource properties.  Constraints -- as I envision them -- are not a dynamic validation feature, but rather an indirect, deferred declaration feature.  In many cases, explicit resource declarations could be replaced by one or more constraints on the same resource, which could appear anywhere in the manifest set.  Everything gets resolved after all resources are compiled.

Sounds like treating hiera data as virtualized to me (and sounds like a functional way to deal with the issues we are discussing).  How would you implement this today?

And now I'm breaking my promise, but it is your thread...

The resource constraints idea has no direct relationship to hiera.  There's no reason why the two shouldn't interoperate, but neither depends on the other.

I am not at all sure there is a clean way to build it on the (intended) extension points available today, however.  As I conceive it, it involves a catalog tweaking step between initial compilation and delivery to the agent, and as such it would require modifications to the Puppet core.  Perhaps one could approximate it, however.  The most likely way I can think of to do that involves custom functions.

At least two functions would be needed: one to cache data somewhere in the master (invoked potentially many times) and one to read back and apply that data (probably called fewer times, and very late in the compilation -- such as in a final run stage).  I apologize for how abstract and vague that description is, but there is a great deal more design effort needed than I am prepared to exert at the moment.

John

[Jo Rhett]

On Jul 17, 2012, at 12:42 PM, jcbollinger wrote:

I apologize for how abstract and vague that description is, but there is a great deal more design effort needed than I am prepared to exert at the moment.

No worries. It confirms what is necessary for me right now which is that it's not something I should be testing anytime soon ;-)

[Felix Frank]

On 07/17/2012 07:31 PM, Jo Rhett wrote:
>
> Is this not the epitome of diverse and redundant dependancies? I can't
> edit my hiera data without evaluating puppet manifests, I can't edit the

> puppet manifests without editing the hiera data�

Hmm, by "managing your hiera data" I wasn't implying that puppet should
manage them. I don't think that can work.

Rather, if you're not intending to scribble YAML files by hand (which is
entirely possible), you would have to write a web frontend or similar.

On 07/17/2012 09:42 PM, jcbollinger wrote:
> At least two functions would be needed: one to cache data somewhere in
> the master (invoked potentially many times) and one to read back and
> apply that data (probably called fewer times, and very late in the
> compilation -- such as in a final run stage). I apologize for how
> abstract and vague that description is, but there is a great deal more
> design effort needed than I am prepared to exert at the moment.

This seems reminiscent of the "futures" idea that puppetlabs bounced
around a year back or so. Implementing such things is really a lot
harder than it sounds, because you need to make sure that your compiler
does not take x^n (x raised to the power of n) loops, where n is e.g.
the number of resources in your manifest.

I may be overthinking this, though. Also this branch of discussion
probably belongs on the dev list. Kudos for thinking this up in the
first place, anyhow.

Cheers,
Felix

[Jo Rhett]

On 07/17/2012 07:31 PM, Jo Rhett wrote:

Is this not the epitome of diverse and redundant dependancies? I can't

edit my hiera data without evaluating puppet manifests, I can't edit the

puppet manifests without editing the hiera data…

On Jul 18, 2012, at 1:19 AM, Felix Frank wrote:

Rather, if you're not intending to scribble YAML files by hand (which is
entirely possible), you would have to write a web frontend or similar.

Understood. The problem is that every example I've seen to date you have to know the puppet module code well to edit the data. The value of a front-end would be to allow users other than ruby coders to manage the data. I haven't seen any examples with enough differentiation for that.

Ultimately I guess you build an entire ecosystem where you tie the front-end data management code to the puppet manifests in git and update both at the same time--but that's one hell of an infrastructure creation that I don't have enough free time for.  In short, I believe that hiera is only useful for companies who already have a large information schema from which to draw their data, and you are only editing the hiera adapters to get the data as you update the puppet manifests.  Anyone else has a huge project to get to "useful".