The Future - ENCs vs Hiera?

[Jason Antman]

After digressing into this discussion on #puppet last night, I was
wondering what the community feelings are on ENCs vs Hiera...

I know that Dashboard/Console still exists, but have heard rumors (for
years) of it being either replaced by something else, or totally
rewritten. Then there's Foreman, and a bunch of us with homegrown ENCs.
On the other hand, it seems that the bleeding edge of the community, or
many of those who are most active and knowledgeable, eschew the ENC idea
and generally respond with "use Hiera".

- Am I right that a community consensus seems to be forming around "ENCs
are bad" or "Don't use ENCs", use Hiera instead, essentially for
everyone other than Puppet Enterprise users who get a supported Console
install?
- If so, is this because of some actual failing of ENCs, or is it
because the currently available ENCs fail in certain areas?

I'm a fan of ENCs. I'm just starting work on my... third... that will
hopefully become a real thing, not just some in-house script. I like the
promise of being able to integrate report processing and *certain*
configuration on one screen in a web interface, hook it up to other
tools, give fine-grained access control (i.e. on the parameter level,
theoretically), and (unlike Hiera) easily see every class/param applied
to my node in one place. I also like - really, need - the fact that I
can change configuration data in my ENC via its API, from Jenkins or
arbitrary cli scripts...

But as I'm starting down the road of writing *another* ENC - hopefully
"a good one" this time - and more and more people whose advice and
knowledge I've respected for years say "use Hiera", I'm starting to
wonder if either (a) I'm totally missing something, or (b) the community
is increasingly ignoring a subset of us who, in fact, don't want every
bit of configuration data committed to a git repo and/or like web UIs
for certain things and/or need to manipulate that data via an API.

I'm sure (hope) I'll get more than a few informed responses to this, and
probably a bit of a holy war too. But I'm starting to question the way
I've looked at this for a long time (and ENC is a good thing).

Thanks,
Jason / jantman

[Peter Meier]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> But as I'm starting down the road of writing *another* ENC -
> hopefully "a good one" this time - and more and more people whose
> advice and knowledge I've respected for years say "use Hiera", I'm
> starting to wonder if either (a) I'm totally missing something, or
> (b) the community is increasingly ignoring a subset of us who, in
> fact, don't want every bit of configuration data committed to a git
> repo and/or like web UIs for certain things and/or need to
> manipulate that data via an API.

What you need to do is to write a tool that can act as an interface
for a datastore for your server data.

Your datastore then can then be used either by an ENC or hiera (write
your own backend, dump the data in a supported hiera format, ...).

Hiera has various advantages over an ENC and all the things that you
can do with an ENC are also possible within hiera, but not the other
way round.

Hence, if you build your interface the way that it can be used by both
techniques you're on the safe side and very flexible (you could even
use it in a mix mode). And actually if you think a little bit further,
it's actually not a big thing to support both ways of handing data
over to puppet.

~pete

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlLILDIACgkQbwltcAfKi3+27gCgitWOzIViRTUAgZnwMzA+V27p
rGEAoJT/AJmuPnGpD5J4q2YIdSloghhz
=wb5g
-----END PGP SIGNATURE-----

[Jason Antman]

I guess I hit a bit of a nomenclature issue here. By "ENC" I didn't just mean the node_terminus script, but the whole app - i.e. "ENC" as in the way Dashboard is an ENC...

Though your comment does remind me that Hiera can now be more than "a directory full of YAML files"... I guess I'll start doing some research into hiera backends, and what they can (could possibly) do for me...

Thanks
-Jason

[Jose Luis Ledesma]

I'm in the process of setting up puppet in my company and I don't like at all the idea of having some thousand of files for describe the nodes. And, despite the well known limitations, I like a lot the idea of having an enc like ldap.

Regards,

[Jason Slagle]

On 01/04/2014 01:15 PM, Jose Luis Ledesma wrote:
> I'm in the process of setting up puppet in my company and I don't like at all the idea of having some thousand of files for describe the nodes. And, despite the well known limitations, I like a lot the idea of having an enc like ldap.

So use this or a derivative:

https://github.com/hunner/hiera-ldap

I think the point is what was previously mentioned - everything you can
do with the ENC you can do with hiera (I think environments may be the
one exception). You just need to write a hiera backend instead of an enc.

I actually suspect it'd be trivial to write a hiera backend that could
just consume enc data. It'd be a fairly limited backend since it'd
really only work with certname, but it could be done.

Jason

[Jose Luis Ledesma]

Enforcing environment from the master is a must for us, and I don't really see the point of configuring hiera against LDAP when puppet supports it natively. Do you know some benefit I'm missing?

[Jason Edgecombe]

FYI, the next generation of RedHat satellite (Subscription Manager?) are
based on Katello and Foreman.

http://www.katello.org/

Jason

[simon c]

For myself, I can see that Hiera is really the way to go, (with a good front end), however for where
 I am now on the puppet rollout, asking other admins to create even simple json/yaml files for
each node would not get anywhere.

So, I've got a very simple node terminus ENC script that parses flat text files to groups &
"roles" and anything more complex goes either goes into hiera, or into a node specific
manifest class file.

I quite liked the puppet dashboard as an ENC, but there was no way to audit who
changed what when, which would be unacceptable in our environment.

[jcbollinger]

On Saturday, January 4, 2014 7:46:58 AM UTC-6, Jason Antman wrote:

After digressing into this discussion on #puppet last night, I was
wondering what the community feelings are on ENCs vs Hiera...

I know that Dashboard/Console still exists, but have heard rumors (for
years) of it being either replaced by something else, or totally
rewritten. Then there's Foreman, and a bunch of us with homegrown ENCs.
On the other hand, it seems that the bleeding edge of the community, or
many of those who are most active and knowledgeable, eschew the ENC idea
and generally respond with "use Hiera".

- Am I right that a community consensus seems to be forming around "ENCs
are bad" or "Don't use ENCs", use Hiera instead, essentially for
everyone other than Puppet Enterprise users who get a supported Console
install?
- If so, is this because of some actual failing of ENCs, or is it
because the currently available ENCs fail in certain areas?

I can't speak to any community consensus, but I'll give you my own two cents.

• I do not think there's anything inherently wrong with GUI configuration control panels.  On the contrary, such things can be very valuable.  In that expansive sense I have no objection whatever to ENCs.
• To the best of my knowledge, all widely-available GUI configuration panels for Puppet leverage the node_terminus extension point to provide node data.
  
• A node_terminus script (the strict sense of "external node classifier") provides a few capabilities that hiera does not.  In particular, you can use an ENC to set the node environment, and to set top-scope variables without any corresponding manifest code.  Technically, an ENC can also classify nodes without any manifest code, but you only need one line of manifest code to enable node classification via hiera.
  
• On the other hand, I strongly recommend providing class parameter customizations via hiera and automated data binding.  It reduces opportunities for breakage associated with Puppet's requirement that if a class declaration provides parameter values then it must be the first declaration of its target class to be evaluated.  There is no inherent reason, however, why a GUI configuration panel could not use hiera to provide class parameters, and even to feed node classification itself to Puppet.
• I do not use an ENC myself.  I do not find the advantages to be particularly compelling for my own relatively modest needs, and I certainly don't find them sufficient to justify me building my own GUI configuration application.
  

 

John

[Felix Frank]

This really speaks to me.

I haven't dabbled in ENCs as of yet, but my natural response to the
question of hiera vs. ENC would be to use the right tool for the
respective job. If the added complexity of using both is acceptable, do
go ahead and use both.

If I got my history correct, hiera was not intended to be an ENC
alternative or replacement. It did replace extlookup, which in turn was
kind of a crutch for people who didn't want to invest the effort of
creating a full fledged ENC.

Personally, I agree with most commentors that hiera should not be a
reason to deprecate the ENC concept altogether.

Cheers,
Felix

[R.I.Pienaar]

----- Original Message -----
> From: "Felix Frank" <felix...@alumni.tu-berlin.de>
> To: puppet...@googlegroups.com
> Sent: Tuesday, January 7, 2014 11:08:38 AM
> Subject: Re: [Puppet Users] Re: The Future - ENCs vs Hiera?
>
> This really speaks to me.
>
> I haven't dabbled in ENCs as of yet, but my natural response to the
> question of hiera vs. ENC would be to use the right tool for the
> respective job. If the added complexity of using both is acceptable, do
> go ahead and use both.
>
> If I got my history correct, hiera was not intended to be an ENC
> alternative or replacement. It did replace extlookup, which in turn was
> kind of a crutch for people who didn't want to invest the effort of
> creating a full fledged ENC.
>
> Personally, I agree with most commentors that hiera should not be a
> reason to deprecate the ENC concept altogether.

Hiera as you say replaced extlookup, primarily by making the backends pluggable
and introducing the merging behaviours as a result of Puppet gaining more data
types.

Hiera is a tool that aims to be good at solving the common data modelling problem
many of us have. Our machines and environments map well onto its hierarchical
nature and so it gives us the ability put data in a way that maps well to the
natural structure of compute environments.

When Puppet 3 added the automatic class param lookup feature it make hiera a
lot more powerful in concept and really brought the old ENC like hiera_include
to full power so today, for most people, a hiera ENC works fine.

Given a custom backend that talks to your DB that has a web frontend that
administers the data in question you can get very close to what you can with
an ENC in terms of friendly UI and such.

Notice how I said it's aim is to solve data problems many of us have - not all.
It's not intended to solve all data problems for all people. There will always
be cases where your environment is too complex or just not hierarchical or whatever.

These tend to be people with special needs who would generally not have a problem
with doing their own development to solve a problem. So the ENC interface is
valuable to this class of user, they have a powerful plugin method to do whatever
they like without having to fit into what hiera prefers.

Totally think there's a place for both types of people, if hiera saves 80% of us
from writing their own ENC (deceptively complex process) then it's been a
success, but it's aim is not to replace all ENCs

>
> Cheers,
> Felix
>
> On 01/06/2014 10:09 AM, simon c wrote:
> > For myself, I can see that Hiera is really the way to go, (with a good
> > front end), however for where
> > I am now on the puppet rollout, asking other admins to create even
> > simple json/yaml files for
> > each node would not get anywhere.
> >
> > So, I've got a very simple node terminus ENC script that parses flat
> > text files to groups &
> > "roles" and anything more complex goes either goes into hiera, or into a
> > node specific
> > manifest class file.
> >
> > I quite liked the puppet dashboard as an ENC, but there was no way to
> > audit who
> > changed what when, which would be unacceptable in our environment.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/52CBE036.8020203%40alumni.tu-berlin.de.
> For more options, visit https://groups.google.com/groups/opt_out.
>

[Arnaud Gomes-do-Vale]

Last time I checked the native LDAP terminus did not support
parametrized classes. This alone justified writing my own ENC, which was
initially just an improved version of the LDAP terminus (this was before
Hiera even existed, let alone was supported out of the box).

--
Arnaud

[Johan De Wit]

Hi Arnoud,

I'm very interested in your improvement. Is it available for the
public somewhere ?

Grts

Jo

--
Johan De Wit

Open Source Consultant

Red Hat Certified Engineer (805008667232363)
Puppet Certified Professional 2013 (PCP0000006)
_________________________________________________________

Open-Future Phone +32 (0)2/255 70 70
Zavelstraat 72 Fax +32 (0)2/255 70 71
3071 KORTENBERG Mobile +32 (0)474/42 40 73
BELGIUM http://www.open-future.be
_________________________________________________________


Next Events:
Puppet Advanced Training | http://www.open-future.be/puppet-advanced-training-7-till-9th-january
Puppet Fundamentals Training | http://www.open-future.be/puppet-fundamentals-training-4-till-6th-february
Subscribe to our newsletter | http://eepurl.com/BUG8H

[Jose Luis Ledesma]

True, but with a profiles/roles design + hiera you shouldn't need to parametrize classes in the LDAP.

Regards,

[Nathan Valentine]

Keep in mind that Hiera can support multiple backends and not all of them mean manipulating JSON or YAML. For instance, I'm a proponent of something like this:

On Sat, Jan 4, 2014 at 10:15 AM, Jose Luis Ledesma <joseluis...@gmail.com> wrote:

I'm in the process of setting up puppet in my company and I don't like at all the idea of having some thousand of files for describe the nodes. And, despite the well known limitations, I like a lot the idea of having an enc like ldap.

Regards,

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/cbae58f9-3e14-472e-906d-ce7a3be355af%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

---
Nathan Valentine, Puppet Labs Professional Services

Have you seen : http://ask.puppetlabs.com , http://puppetlabs.com/learn ?

[Nathan Valentine]

Sorry, bitten by the Gmail interface on that last one...

I'm a proponent of something like this:

http://www.craigdunn.org/2012/11/puppet-data-from-couchdb-using-hiera-http/

with classification data that doesn't change much ( 'require ::site::profile::base' ) perhaps living in YAML and specific one-off tweaks to a node's config like perhaps an override of the default NTP server coming from CouchDB. Much easier for the less Puppet-y people on a team to login and twiddle things via CouchDB web interface. 

[Arnaud Gomes-do-Vale]

Johan De Wit <Jo...@open-future.be> writes:

> I'm very interested in your improvement. Is it available for the
> public somewhere ?

No, there is just too much site-specific stuff hardcoded in
there. Basically, it:

- grabs the host's entry in LDAP, or bombs out if it doesn't exist
- extracts the puppetclass attribute and adds a few implicit classes: a
common base class, a few additional classes for NRPE plugins derived
from the host's Nagios configuration (which is also in LDAP), maybe
a couple of other classes for managing local users and groups, and so
on
- extracts the puppetvar attribute
- for each puppetvar, simply add it to the list of parameters if it does
not contain the string "::"
- if the puppetvar does contain the string "::", it is of the form
class::parameter (or module::class::parameter or whatever); add it to
the relevant class parameters list (or bomb out if the class is not
included)

Add in a lot of special cases, mix, and output as YAML when you're
done. :-) By the way, it's written in Python as I already had a few
local LDAP helper libraries but that's an implementation detail.

--
Arnaud