Uber Frustration with Puppet.

[Douglas Garstang]

Boy, am I frustrated. I'm about ready to throw puppet out the window
here. I'm trying to configure glusterfs, and you know, it kinda made
sense to separate the data from the manifests, so I went ahead and put
this into a YAML file, which hiera loads...

glusterfs_volumes:
gfsvol01:
volume_name: gfsvol01
master_node: gfs01.us1.xxx.com # Make sure only one node
runs the gluster commands.
nodes:
- name: gfs01.us1.xxx.com
bricks:
- device: /dev/bcvg/disk1
brick_name: /var/bricks/gfsvol01-0
- device: /dev/bcvg/disk2
brick_name: /var/bricks/gfsvol01-1
- name: gfs02.us1.xxx.com
bricks:
- device: /dev/bcvg/disk3
brick_name: /var/bricks/gfsvol01-0
- device: /dev/bcvg/disk4
brick_name: /var/bricks/gfsvol01-1

For the last couple of days I have been dealing with the inadequacies
of puppet in dealing with working with this kind of data structure.
You can't easily iterate through it, and every time you do, you have
to write a new definition that takes an array. The whole thing ends up
turning into a giant complicated mess.

I tried writing some custom functions in ruby that do things like,
return a list of nodes for a volume, or return a list of bricks for a
node, but it really irks me that I have to keep writing ruby scripts
for this (since ruby makes my eyes bleed).

So... what are my options here? Aren't we supposed to strive for
separating the manifest from the data? I could probably get away with
a few definitions that take a set of parameters. However, when the
time comes to say, add a new node to the cluster, we have to modify
the manifest. At one point, I had this working so that all you had to
do was add a node to the yaml file, make ZERO changes to the manifest
file, and after running puppet, it would add the node to the cluster.

It may make my life easier if I flatten the yaml file, but then I'm
changing the data to suit the limitations of the DSL.

At this point, I'm very close to simply sticking with the yaml file,
have puppet push that out to the clients, write some python scripts to
do all the magic (reading the yaml file), and have puppet run those
scripts with Exec {}.

Is proper array/hash iteration ever going to be added to puppet?

Doug.

[Erik Dalén]

Have you had a look at the functions in puppetlabs-stdlib?
There's a bunch of functions for dealing with hashes and iterating
over them there, keys() for example could be helpful in this case.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>



--
Erik Dalén

[David Schmitt]

My limited understanding from what I've read about hiera, is that
node-specific data should go to node-specific files, which are to be
configured as first priority lookup. That way you can strip away most of
the layers of your YAML.

I know not enough about glusterfs or your setup to advise you on the
internal structure, but from your YAML it seems as if you're trying to
configure a mapping between physical disks and "bricks". Maybe this can
be calculated instead of hardcoded? That would take care of the final
layer of your YAML.

Finally, the config info exchange between your glusterfs-nodes and your
glusterfs-master should almost certainly be done by exporting and
collecting the necessary resources. Since the advent of puppetdb,
there's no reason at all anymore to avoid storedconfigs.


D.


PS: having verbs in a resource name (like "add_brick") indicates that
you're not yet fully grasping the declarative style of puppet. Or should
the brick be added each time puppet runs?

[jcbollinger]

On Thursday, August 30, 2012 6:40:11 AM UTC-5, Erik Dalén wrote:

Have you had a look at the functions in puppetlabs-stdlib?
There's a bunch of functions for dealing with hashes and iterating
over them there, keys() for example could be helpful in this case.

Also, when you have the choice, I think it usually works better to design your manifests first and then fit the data structure to them, as opposed to designing the data structure first and trying to fit manifests to that structure.

Also, if you're having trouble modeling a complex resource, then that may be a sign that what you really need is a custom type (I know, bleeding eyes and all that, but still...).


John

[Jason Slagle]

Doug,

I'm not terribly familiar with gluster, but that thing in your hand
certainly looks square and you're trying to fit it in that round hole.

It looks to me like you've created an arbitrary service-centric data
model, and are then attempting to smash on the tool (puppet) to make it
work, and becoming frustrated with it. I liken this to me designing
some data structure with pointers, then venting that BASIC can't handle
it [1].

As has been pointed out on several occasions on this list recently, most
of us seem to structure our data in a host-centric model, not a service
centric model. This would flatten out the below structure quite a bit.
It also would likely lead to duplication of data, which is what I
suspect you're trying to avoid. If you're keen on writing a python
script, you could most likely write one to consume the below YAML file
and spit out a node-centric YAML that could more easily be used by the
puppet DSL.

Something like this maybe (No attempt has been made to make this validate)?

gluster_master_volumes: [gfsvol01]
gluster_bricks:
/var/bricks/gfsvol01-0:
device: /dev/bcvg/disk1
/var/bricks/gfsvol01-1:
device: /dev/bcvg/disk2

Then you should be able to use hiera_hash and creates_resources on
gluster_bricks.

Another option would be to write a custom type to handle the gluster
components. Since types are written in Ruby, you should be able to use
the data in a more native way, which would make looping over the hashes
or arrays much easier.

If you need this to work as is, I suggest contacting Puppet for
professional services - those guys over there are really awesome, and
I'm sure they can whip something up that can consume the below data
structure and do what you want with it.

Jason

[1] Ok, I'm sure one of you will point out some basic implementation
with pointers