A GUI tool for Hiera

[desertkun]

Hello, everyone.

I have made a small useful open source project for Puppet/Hiera, so I hope you can excuse me a bit of advertisement of it for greater good.

https://github.com/desertkun/hiera-editor

Basically it takes editing Hiera configurations to a new level. 

It parses modules with puppet-strings to extract class information like field names, types and doc strings, and retrieves default values of class fields 

by doing best-effort compilation (with puppet-parser) of Puppet AST on your machine. So no more typos and less of "commit-deploy-check" cycles.

The goal of the project is to help manage servers with Puppet to those who far away from the back-end, including Puppet itself, 

like "I need to deploy nglinx but I have installed debian for the first time". So if you have a project that complex that requires Puppet to deploy it, having some

tool to introduce Puppet to end users of your project might improve the learning curve.

Would really appreciate any input on the idea, including concerns like "there's no need for this" as I just have made the project public and still not sure if I should continue.

Regards.

[Peter Berghold]

I don't want to sound harsh but I'm going to be blunt.   I could not get this thing to work for me on Linux (Ubuntu) even when doing an "npm audit fix" followed by an "npm audit fix --force" followed by an "npm install" 

Very disappointing to me.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1eb43a86-8f65-42d6-910f-9d45a8789256%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Peter L. Berghold                       Salty....@gmail.com

http://science-fiction.berghold.net

[desertkun]

Sorry for that. The thing is the Linux build support is not yet implemented completely. It misses packaged version of ruby for linux. That's not that hard on electron, it's just I haven't yet paid attention to it. I will try to implement that as soon as possible.

[desertkun]

I have included Linux binary to the latest release, and also completed the Linux support for building from source (might need to install zlib1g-dev not sure if that was my local issue).

On Tuesday, January 8, 2019 at 10:45:57 PM UTC+2, Salty Old Cowdawg wrote:

[Karsten Heymann]

Hi Alex,

that sounds definitely very interesting, especially for our coworkers that aren't that deep into puppet and for which unerstanding hiera regularly is the most complicated part of the setup. Some questions and remarks:

* Have you thoght about integrating support for hiera-enc?

* Some kind of git commit function for the changes done would be a neat feature. This would certain parts of our coworkers to stay completely in this kind of editor as they probably only interact with hiera data. If you are interested I have some ideas how this could be implemented in a safe and convenient way.

* Is there support for changing a value at every depth of the configured hiera hierachy?

* Does this include module-level data directories?

* Are you aware of jerakia (http://jerakia.io/). We don't use it yet but are considering it to be able to use our "hiera" data from ansible too. But I think it's a very interesting project.

* Is the hieraresources part optional? We don't want to use hiera to define arbitrary resources as it would work around the way we define roles and profiles.

So thank you for starting this, we are definitely interested in what progress you will make.

Best regards

Karsten

--

[desertkun]

especially for our coworkers that aren't that deep into puppet and for which unerstanding hiera regularly is the most complicated part of the setup.

 

Thanks for the feedback, that is exactly the goal of the project. It makes me happy to know that someone sees potential in this.

 

* Have you thoght about integrating support for hiera-enc?

 

Wasn’t aware of it, so I need time to investigate what it does.

 

* Some kind of git commit function for the changes done would be a neat feature. This would certain parts of our coworkers to stay completely in this kind of editor as they probably only interact with hiera data. If you are interested I have some ideas how this could be implemented in a safe and convenient way.

 

Yes, that is definitely planned. I want it to the level the editor would highlight classes/yaml files/directories with green/blue/red colors to indicate staging status. Obviously, with commit/discard buttons. I also want to implement actual module installation (assuming everyone just adds them as submodules to their repo).

 

* Is there support for changing a value at every depth of the configured hiera hierachy?

 

Not quite sure about that, it can edit classes (their fields) and resources (and their fields), would appreciate an example of what you mean.

 

* Does this include module-level data directories?

 

Not as of yet.

 

* Are you aware of jerakia (http://jerakia.io/). We don't use it yet but are considering it to be able to use our "hiera" data from ansible too. But I think it's a very interesting project.

 

I am not but I am pretty sure we have different goals. I want make a tool for those who have no idea what Puppet is but need to use it.

 

* Is the hieraresources part optional? We don't want to use hiera to define arbitrary resources as it would work around the way we define roles and profiles.

 

It is, you can just do hiera_include('classes') instead if you only need to manage classes. But as long as you won’t create any resource in the tool it won’t populate the `resources` hash and thus the second line hiera_resources('resources')  of the hieraresources module won’t do anything. I implemented this because there is a case when you actually need your instances of profiles several times.

 

From: Karsten Heymann
Sent: Wednesday, January 9, 2019 13:25
To: puppet...@googlegroups.com
Subject: Re: [Puppet Users] A GUI tool for Hiera

 

Hi Alex,

 

that sounds definitely very interesting, especially for our coworkers that aren't that deep into puppet and for which unerstanding hiera regularly is the most complicated part of the setup. Some questions and remarks:

 

* Have you thoght about integrating support for hiera-enc?

* Some kind of git commit function for the changes done would be a neat feature. This would certain parts of our coworkers to stay completely in this kind of editor as they probably only interact with hiera data. If you are interested I have some ideas how this could be implemented in a safe and convenient way.

* Is there support for changing a value at every depth of the configured hiera hierachy?

* Does this include module-level data directories?

* Are you aware of jerakia (http://jerakia.io/). We don't use it yet but are considering it to be able to use our "hiera" data from ansible too. But I think it's a very interesting project.

* Is the hieraresources part optional? We don't want to use hiera to define arbitrary resources as it would work around the way we define roles and profiles.

 

So thank you for starting this, we are definitely interested in what progress you will make.

 

Best regards

Karsten

 

Am Di., 8. Jan. 2019 um 18:57 Uhr schrieb desertkun <dese...@gmail.com>:

Hello, everyone.

 

I have made a small useful open source project for Puppet/Hiera, so I hope you can excuse me a bit of advertisement of it for greater good.

 

https://github.com/desertkun/hiera-editor

 

Basically it takes editing Hiera configurations to a new level. 

It parses modules with puppet-strings to extract class information like field names, types and doc strings, and retrieves default values of class fields 

by doing best-effort compilation (with puppet-parser) of Puppet AST on your machine. So no more typos and less of "commit-deploy-check" cycles.

 

 

The goal of the project is to help manage servers with Puppet to those who far away from the back-end, including Puppet itself, 

like "I need to deploy nglinx but I have installed debian for the first time". So if you have a project that complex that requires Puppet to deploy it, having some

tool to introduce Puppet to end users of your project might improve the learning curve.

 

Would really appreciate any input on the idea, including concerns like "there's no need for this" as I just have made the project public and still not sure if I should continue.

 

Regards.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1eb43a86-8f65-42d6-910f-9d45a8789256%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/YYFYaDphwbU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAL017hCAj_XsOfG%3D7aaJAV56JbBXm96U1p32DqCXW1GXK--Z0Q%40mail.gmail.com.

[Karsten Heymann]

Am Mi., 9. Jan. 2019 um 12:48 Uhr schrieb desertkun <dese...@gmail.com>:

* Have you thoght about integrating support for hiera-enc?

 

Wasn’t aware of it, so I need time to investigate what it does.

It allows to encrypt entries (value) in hiera so that they don't end up in clear text for example in the git history of the puppet code. 

 

* Some kind of git commit function for the changes done would be a neat feature. This would certain parts of our coworkers to stay completely in this kind of editor as they probably only interact with hiera data. If you are interested I have some ideas how this could be implemented in a safe and convenient way.

 

Yes, that is definitely planned. I want it to the level the editor would highlight classes/yaml files/directories with green/blue/red colors to indicate staging status. Obviously, with commit/discard buttons. I also want to implement actual module installation (assuming everyone just adds them as submodules to their repo). 

I would think that nowadays using a Puppetfile is the preferred way to install modules.  

* Is there support for changing a value at every depth of the configured hiera hierachy?

 

Not quite sure about that, it can edit classes (their fields) and resources (and their fields), would appreciate an example of what you mean.

The way we use hiera is that we have the following hierachy definded in hiera.yaml (this is still hiera 3, but the principle stays the same with hiera 5+):

hiera.yaml:

:hierarchy:

  - "hostname/%{fqdn}"

  - "roles_%{trusted.extensions.pp_apptier}/%{trusted.extensions.pp_role}"

  - "roles_%{trusted.extensions.pp_apptier}/default"

  - "roles_common/%{trusted.extensions.pp_role}"

  - common

The hiera data folder in our environments looks this way:

$ tree -d

.

├── hostname

├── roles_common

├── roles_labor

├── roles_prod

└── roles_staging

With yaml files in every folder and a common.yaml in the main folder. Now when in the puppet code a variable is looked up, the hierachy list above is evaluated and checked on which level a variable is set.

Example: A webserver with the hostname web-01.mydomain, the apptier staging and the role web uses the class profiles::webserver and in there looks up the variable profiles::webserver::servername.  Now when compiling the catalog the following hiera files are sequentially checked if they contain a "profiles::webserver::servername: whatever" variable:

hostname/web-01.mydomain.yaml

roles_staging/web.yaml

roles_staging/default.yaml

roles_common/web.yaml

common.yaml

Depending if I want to set a variable only for a specific server, for all staging web server, for all staging servers, for all prod/lab/staging web servers or for every system known, the variable can be edited on any of these levels. This is the main benefit of using hiera, you can define a variable as general and as specific as needed without repeating yourself. The main feature I would expect from a hiera editor would be to make this layered setup as approachable and transparent as possible, 

  

* Are you aware of jerakia (http://jerakia.io/). We don't use it yet but are considering it to be able to use our "hiera" data from ansible too. But I think it's a very interesting project.

 

I am not but I am pretty sure we have different goals. I want make a tool for those who have no idea what Puppet is but need to use it.

This wasn't ment as a competing project, more as a hint to a possible alternative backend that could be supported additionally to hiera.

  

* Is the hieraresources part optional? We don't want to use hiera to define arbitrary resources as it would work around the way we define roles and profiles.

 

It is, you can just do hiera_include('classes') instead if you only need to manage classes. But as long as you won’t create any resource in the tool it won’t populate the `resources` hash and thus the second line hiera_resources('resources')  of the hieraresources module won’t do anything. I implemented this because there is a case when you actually need your instances of profiles several times.

 

I think we use hiera in a different way. We don't load any classes from hiera at all, instead we use an enc script for this. So the only purpose hiera does serve us is to set the variables for the profiles loaded by the roles our servers use. Every server "automatically" loads roles::<tier>::<role>.pp and in this file all the server's profiles are included ('contain'd to be precise). But I understand that loading classes not from hiera makes it much more difficult to decide which variables are used by a server.

This all isn't ment to say that our solution is any better than what you seem to propose, but just as a heads up that I think it's quite important that you document the assumptions you have about how a user uses puppet/hiera in order to be able to use your tool.

Best regards,

Karsten

[desertkun]

Thank you for such detailed feedback. It all makes sense now.

I have pushed an update that makes the editor to support hierarchy. It now shows on what level of hierarchy each property of the class is defined (by color-coding)

You can also switch to desired level in the drop-down, and make changes on that level.

It also now evaluates actual pp files and shows resolved classes, so you can use Hiera Editor with puppet-resolved classes.

Unfortunately, a connection to Puppet Server is a requirement now (it needs to download facts and node list). But I've tried to make that process as less painful as possible. 

Couple questions.

1. You have facts like "trusted.extensions.pp_role" in your examples. Could you please elaborate on those so I can also support them?
   
2. How would you like to see hiera-enc working? To automatically decrypt/encrypt values, we need the keys. Those are essential not only to change values, but also to accurately evaluate properties (for example, a condition that fails if password too short). Hiera-enc states that you should really keep the keys on the Puppet Server only. How about password-encrypted keys? You can use them within the editor, but you have to enter a master password each time the editor is launched.

[glenn...@puppet.com]

This is VERY VERY interesting...

I'm one of the people you maintains the VSCode Puppet extension, and there seems to be some nice cross over here. (I see you develop using VSCode so hopefully you're aware of it :-) )

In particular offering a GUI experience to modifying Hiera data, combined with the work we've done in undestanding control repos and puppet manifests.

Also, it looks like you made a Puppet AST parser in Javascript which is awesome.  One of my goals would be to create a "NodeJS only" puppet extension (Cut down version from the full extension) so it could be used by products like coder.com and one of things I'm missing is a tokeniser/ast for Puppet language

I'll be poking around the codebase for a while and see where we can integrate this!

Thanks,

Glenn.

[desertkun]

Fill free to steal it. 

I could try to cut off the AST parser to an external repository so you would be able to mess around with it. For example, it misses a lot of builtin functions  at the moment. However, it parses a produced output from puppet-strings so good luck with that, you'd have to run some ruby with js.

[desertkun]

 

However, it parses a produced output from puppet-strings

Excuse me with spam, I actually meant puppet-parser instead.

Regards.