Module use with Foreman ENC

[Sean Alderman]

Hi,
  Thanks for setting up the google group.  I was curious if you have any hints on how to go about using your modules with Foreman as the ENC.  I've had good luck with your modules when I was using puppet alone, but I must admit, I'm struggling to understand how to leverage and configure your modules through host groups and environments in Foreman.

  For starters, when looking at a host group, I can select puppet classes that should be assigned to hosts in that group.  I select openssh, which expands to show all the sub-classes(?) like openssh::params and openssh::spec.  I assume I only need to select the top level class, but I'm not sure.  Secondly, I'm struggling to understand how to specify class parameters for your modules as parameters in Foreman.  Foreman parameters seem to be simple name,value hashes.  For example, I'd like to set PermitRootLogin without-password instead of no (as is the default on RHEL based servers).  I could edit the kickstart post script to make this change on every system built by foreman, but given that foreman is not necessarily going to build all systems, that's problematic.

  Perhaps this is more of a question for the foreman group, if so, I can take it there.  I was hoping you might have played with it a bit.

Thanks!

[Alessandro Franceschi]

Hi,
good question for which there is (I think) a quite good answer out of the box.

If you have to use an Example42 module with an ENC just include the main class (for example openssh) and set as variables all the parameters of the class you want to manage, using the class name as prefix.
For example, to provide a custom template for openssh, just set this variable (you may do it at global, hostgroup or host level, overwriting the parameter are needed):
openssh_template = 'site/openssh/sshd.config.erb'

This is equivalent to:
class { 'openssh':
  template =>  'site/openssh/sshd.config.erb',
}


For parameters that expect an hash, like the options parameter where you could set an arbitrary openssh parameter like PermitRootLogin, I think (haven't tried) that you can just place the hash as value for the relevant parameter, for example:
openssh_options => { PermitRootLogin => no ... }
Then, of course, on your openssh template you will have to use the options_lookup function to get the value of PermitRootLogin
(let me know it that actually works).

Note finally that for some parameters (the ones that have 'global' as second argument passed to the params_lookup function) you can also set a site-wide variable which is used for all the classes.
For example, given that in all the modules you have:

[...]
  $monitor             = params_lookup( 'monitor' , 'global' ),

[...]
you can set, on Foreman (or any ENC):
monitor = true
to enable monitoring for all the modules, and eventually:
openssh_monitor = false
to disable it only for openssh.

Hope this answer your question.

al

[Sean Alderman]

Alessandro,
  Thanks.  I think I've got this working now.  There seems to be a trick with list values, they just need to be set as an comma separated list, no brackets or quotes as those end up in the configuration file on the target host.  See attached image, specifically resolver_search.

  I haven't tried another openssh option in the hash list, but the other problem with that is that I need to insert an authorized key at build time for root to be accessible immediately after the system load.  Currently I'm using the console to do that manually, but I'd like to get to a point where I can associate a  public ssh key with the foreman user who owns the server and stuff that user's key into root's authorized_keys file on build.

  So I guess I put the cart before the horse in asking the question about the foreman parameters.

[Alessandro Franceschi]

Thank you very much for the info, I think it will be useful for future reference.
If you can make a test with different key-values pairs in the hash, just to be sure that they all work as expected, and report here the result, it would be great.

For root's or other users ssh key management you may find interesting the sshkey_source in the user::managed define ( https://github.com/example42/puppet-user/blob/master/manifests/managed.pp ) which basically wraps the user type adding some options to manage collateral users' related resources.

[Sean Alderman]

Ok, so following your suggestions on the hash...

Within Foreman:
openssh_options = { PermitRootLogin => without-passwd, Banner => /etc/udwarning }
openssh_template = openssh/sshd_config.erb

Template looks like (only the appropriate lines):
PermitRootLogin <%= scope.function_options_lookup(['PermitRootLogin','no'])%>
Banner <%= scope.function_options_lookup(['Banner','none']) %>

And the output in the config file on the target host is:
PermitRootLogin PermitRootLogin
Banner Banner

Am I misunderstanding your function_options_lookup?  First parameter in the list is the hash key, if it exists, else a default value?  It's almost like its taking the quoted string as a literal.

Here's the output of node.rb on the foreman/puppet server.  Looks ok, no?

puppet]# ./node.rb tomcat1.int.localnet
---
  environment: production
  parameters:
    resolver_search: "int.localnet,localnet,servers.udayton.edu"
    puppet_ca: firewall.int.localnet
    openssh_firewall: "true"
    resolver_dns_servers: "192.168.100.1"
    monitor_tool: "nagios, puppi"
    openssh_options: "{ PermitRootLogin => without-passwd, Banner => /etc/udwarning }"
    owner_name: "Admin User"
    owner_email: "root@localhost"
    puppi: "yes"
    openssh_firewall_src: "192.168.100.0/24"
    foreman_env: production
    hostgroup: tomcat/tomcat-prod
    nrpe_allowed_hosts: "127.0.0.1, 192.168.100.2"
    openssh_firewall_dst: "$ipaddress_eth0"
    root_pw: "$1$fOfzM8YE$VUkCXmXBrLDfc8Y6ffNz5."
    domainname: "Local Internal Network"
    puppetmaster: firewall.int.localnet
    monitor: "yes"
    openssh_template: openssh/sshd_config.erb
  classes:
    ud-files:
    tomcat:
    nrpe:
    autofs:
    "apache::ssl":
    sudo:
    apache:
    puppi:
    openssh:
    logrotate:
    resolver:
    iptables:
    rsyslog:

[Alessandro Franceschi]

On Thursday, May 30, 2013 8:52:17 PM UTC+2, Sean Alderman wrote:

Ok, so following your suggestions on the hash...

Within Foreman:
openssh_options = { PermitRootLogin => without-passwd, Banner => /etc/udwarning }
openssh_template = openssh/sshd_config.erb

Template looks like (only the appropriate lines):
PermitRootLogin <%= scope.function_options_lookup(['PermitRootLogin','no'])%>
Banner <%= scope.function_options_lookup(['Banner','none']) %>

This is all correct.

 

And the output in the config file on the target host is:
PermitRootLogin PermitRootLogin
Banner Banner

This is definitively wrong, I expect, obviously, something like:

PermitRootLogin without-passwd

Banner /etc/udwarning

 

Am I misunderstanding your function_options_lookup?  First parameter in the list is the hash key, if it exists, else a default value?  It's almost like its taking the quoted string as a literal.

First parameter is the hash key, second is the default value if no key is found and third, optional, is the name of the parameter name from which to get the hash (default: options).
 

Here's the output of node.rb on the foreman/puppet server.  Looks ok, no?

Yes. Gotta make some tests.
Can you eventually see what happens when you set in /etc/puppet/manifests/site.pp :
$openssh_options = { PermitRootLogin => without-passwd, Banner => /etc/udwarning }
$openssh_template = openssh/sshd_config.erb

 and remove those entries from Foreman?

Just to see if it may be relate to the integration with Foreman or what...

[Alessandro Franceschi]

Yes. Gotta make some tests.
Can you eventually see what happens when you set in /etc/puppet/manifests/site.pp :
$openssh_options = { PermitRootLogin => without-passwd, Banner => /etc/udwarning }
$openssh_template = openssh/sshd_config.erb

 and remove those entries from Foreman?

It seems it's a bug of options_lookup :-( as it happens also without Foreman. Have to investigate it...
Btw, a note, you are not supposed to modify anything in the module (that's the whole point of reusability) so I would expect all the custom templates or static files for configurations to be places in a custom "site" module, where you place all you custom stuff (so that you can more easily keep the module aligned to the upstream version).

So I would set something like:
openssh_template = site/openssh/sshd_config.erb
and place the template in your site module.

 

[Sean Alderman]

In my real environment (this is a lab inside virtualbox), I have an openssh template that looks like:

<default RHEL config file>
# Values managed by puppet...
<% options.each do |key, value| -%>
<%= key %> <%= value %>
<% end -%>

If I switch over to that template, and use the openssh_options parameter from inside foreman I get:
# Values managed by puppet...
{ 'PermitRootLogin' => 'without-password', 'Banner' => '/etc/udwarning' }

When I move that parameter to site.pp:
# Empty site.pp required (puppet #15106, foreman #1708)
$openssh_options = { 'PermitRootLogin' => 'without-password', 'Banner' => '/etc/udwarning' }

after a puppet run I get :
# Values managed by puppet...
Banner /etc/udwarning
PermitRootLogin without-password

One thing I noticed here it it looks like the internal data structure is different (see my emphasized text)...

info: Applying configuration version '1369942071'
--- /etc/ssh/sshd_config        2013-05-30 19:22:41.000000000 +0000
+++ /tmp/puppet-file20130530-11360-8cp8zb-0     2013-05-30 19:27:57.000000000 +0000
@@ -136,5 +136,6 @@
 #      ForceCommand cvs server

 # Values managed by puppet...
-{ 'PermitRootLogin' => 'without-password', 'Banner' => '/etc/udwarning' }
+Banner /etc/udwarning
+PermitRootLogin without-password

info: FileBucket adding {md5}437901fef42db1e7d4702b44b579f791
info: /File[openssh.conf]: Filebucketed /etc/ssh/sshd_config to puppet with sum 437901fef42db1e7d4702b44b579f791
notice: /File[openssh.conf]/content: content changed '{md5}437901fef42db1e7d4702b44b579f791' to '{md5}04e24380209912a0ad296d0ec4c3ee53'
info: /File[openssh.conf]: Scheduling refresh of Service[openssh]
notice: /Stage[main]/Openssh/Service[openssh]/ensure: ensure changed 'stopped' to 'running'
info: /Stage[main]/Openssh/Service[openssh]: Unscheduling refresh on Service[openssh]
--- /etc/puppi/data/standard_openssh.yml        2013-05-30 19:22:43.000000000 +0000
+++ /tmp/puppet-file20130530-11360-1rwjj3a-0    2013-05-30 19:27:58.000000000 +0000
@@ -11,7 +11,9 @@
   bool_audit_only: false
   bool_disableboot: false
   debug: false
-  options: "{ 'PermitRootLogin' => 'without-password', 'Banner' => '/etc/udwarning' }"
+  options:
+    Banner: /etc/udwarning
+    PermitRootLogin: without-password
   puppi_helper: standard
   bool_disable: false
   config_file_mode: "0600"

[Alessandro Franceschi]

Ok, I checked better and I was actually using a badly formatted hash :

This is correct:

$openssh_options = { 'PermitRootLogin' => 'without-passwd', 'Banner' => '/etc/udwarning' }

I firstly messed up with quotes , making me think that there was a bug in options_lookup.

Looking at the diff it seems clear that Foreman is passing the parameter as string and not as hash. 
I've to investigate this on the Foreman side, in the meantime try to see if setting exactly the above value in foreman it works.

[Sean Alderman]

Apparently this has to do with Foreman converting stuff into yaml and in the process the value looses it's ruby hash structure, at least according to the Foreman-Users group.

There's also an interesting thing that Foreman shows the "smart class parameter" called options as a string.  I could perhaps override it to be a hash.  So maybe this is fixable inside Foreman and has more to do with the Puppet Smart Proxy and how it imports parametrized classes.

[Sean Alderman]

Ok, I've made progress.

In finding this: http://projects.theforeman.org/projects/foreman/wiki/Parameterized_class_support

I successfully overridden the Foreman Puppet Class / openssh / Smart Class Parameter / options to a Hash type and given it these default options at the class level.  The default type was String and the default value was params_lookup("options").  Not sure why, but whatever.  If I override the type and specify a yaml or json hash syntax the erb template seems to work.  It's ugly, but works.

---
  environment: production
  classes:
    puppi:
    logrotate:
    autofs:
    ud-files:
    openssh:
      options:
        Banner: /etc/udwarning
        PermitRootLogin: without-password
    iptables:
    sudo:
    "apache::ssl":
    apache:
    tomcat:
    nrpe:
    resolver:
    rsyslog:
  parameters:
    puppi: "yes"
    puppet_ca: firewall.int.localnet
    resolver_dns_servers: "192.168.100.1"
    openssh_firewall_src: "192.168.100.0/24"
    domainname: "Local Internal Network"
    openssh_template: openssh/orig-sshd_config.erb
    resolver_search: "int.localnet,localnet,servers.udayton.edu"
    openssh_firewall_dst: "$ipaddress_eth0"
    owner_name: "Admin User"
    foreman_env: production

    nrpe_allowed_hosts: "127.0.0.1, 192.168.100.2"
    owner_email: "root@localhost"
    monitor: "yes"
    openssh_firewall: "true"
    monitor_tool: "nagios, puppi"
    hostgroup: tomcat/tomcat-prod
    puppetmaster: firewall.int.localnet

[Alessandro Franceschi]

This is interesting.
I suppose you found as default value params_lookup("options") because that's actually the default value of the parameter and Foreman gets it from the same class.
For who might read this in the future, what's exactly the value you placed on the foreman form ?

 openssh: 
      options: 
        Banner: /etc/udwarning
        PermitRootLogin: without-password

???
If so is REALLY ugly :-)

I try to involve Ohad on this, so that we have a really authoritative point of view...

[Sean Alderman]

ARG!  Groups just blasted my response when I went to post it, saying I needed to reload my browser to get a new version of the app. *sigh*.

Ok the process I followed was to goto the smart class parameter for the openssh module and manually override the type and default value to a has and use the YAML syntax for the key/value pairs as the default value.

Basically that looks like so:

Browse to Puppet Classes / <module> / Smart Class Parameter tab
Click options
Check Override
In Type, select Hash
In Default value input:
key1: value1
key2: value2

In my specific case for openssh that looked like this:
PermitRootLogin: without-password
Banner: /etc/udwarning

I did not try the JSON syntax which apparently is also acceptable.  Both differ from the Puppet hash syntax.

The UGLY factor is that in order to utilize your modules parameters that are Puppet Hash types, foreman admin must override each one changing the Smart Class Parameter type from string to hash and setting a valid default value.  Options being the most useful so far for me, but you have written 76 modules which utilize this options hash, each would need overriding to make work as you've designed with foreman.

It would appear that I can override the Smart Class Parameter anywhere it appears later - host group, host, domain, etc, but with out changing it from string to hash, that does not become possible.  I'd much rather just be able to make a Foreman Parameter named <module>_options and give it a value that will later become a hash right out of the box...that would be beautiful - not ugly.  :)

Does that make sense?

[Alessandro Franceschi]

I think, but I'm not sure, that Foreman automatically detects the type of the parameter based on the default param.
Here the default is the string params_lookup("options") so it can't guess that it expects an hash.
It would be nice if Foreman would try to process the function params_lookup (or any other function eventually used for the default value) but I frankly don'y know if this is possible. 

[Sean Alderman]

I didn't mean to imply that I understand such to be the case, I'm making an assumption since the function call doesn't appear to look like any other yaml data type.

I might troll around github and see if I can find a module to play with that might leverage a parameter that foreman will notice as a hash.

[Alessandro Franceschi]

Well, you can just try to change the default value :
params_lookup('options')
with :
{ }

and see what happens to the default value type (I think you might have to reimport the class in foreman)

Also some attempts with an empty array:
[]
or also with predefined, not empty, hashes and arrays could be interesting to make.