puppet recipes

[Asif Iqbal]

Hi

I am looking for recipe or some hints to a recipe that can help me
achieve the following

I have about 300 servers of different functions. To make it easy I
decided to keep multiple group dirs based on the
function and have hosts,passwd,users,sudoers file located inside those
function dirs, like the following. In this
example dns is the function of the hosts listed w/ fqdn in the hosts
file. The passwd and shadow are going to be
same as the /etc/passwd and /etc/shadow file for all these hosts, same
for sudeors. users is list of users. may have no purpose
right now.

(root)@puppetmaster:/path/to/groups# ls -lR dns/
dns/:
total 11
-rw------- 1 root other 1 Aug 23 2005 hosts
-r--r--r-- 1 root other 33 Aug 22 2005 passwd
-r-------- 1 root other 31 Aug 22 2005 shadow
-r--r----- 1 root root 546 Aug 27 2005 sudoers
-rw-r--r-- 1 root other 152 Feb 21 2006 users

currently, I have a test site.pp like this

# site.pp

node basenode {
case $hostname {
puppet-test: {}
default: {}
}
}

node 'puppet-test' {
include dns
include sudo
}

class dns_users {
@user { "testuser":
ensure => "present",
uid => "102",
gid => "1",
comment => "test user",
home => "/home/testuser",
shell => "/bin/bash",
managehome => "true",
}
}

class dns {
include dns_users
realize(
User["testuser"]
)
}

class sudo {
file { sudoers: # a common title for all platforms
path => $operatingsystem ? {
solaris => "/opt/csw/etc/sudoers",
default => "/etc/sudoers"
},
owner => root,
group => root,
mode => 440,
source => "puppet:///sudo/sudoers"
}
}

Instead of creating 300 manifests and that many more users in the
class and/or @users I like to see if there is maybe a template can be
created.
So when the puppet client comes to puppetmaster, based on the fqdn of
the host it will be assigned as part of a group. Then based on the
assigned
group it will receive specific sudoers file and a list of users will
be created based on the values in passwd and shadow files.

Looking for recipe to achieve that.

Thanks

--
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[Teyo Tyree]

Hey Asif,

On Wed, Jul 15, 2009 at 12:51 PM, Asif Iqbal <vad...@gmail.com> wrote:

Hi

I am looking for recipe or some hints to a recipe that can help me
achieve the following

I have about 300 servers of different functions. To make it easy I
decided to keep multiple group dirs based on the
function and have hosts,passwd,users,sudoers file located inside those
function dirs, like the following.

What do you mean by group dirs in this context? I am assuming you me host groups base on node function.  For clarity, I will call them functional groups.

In this
example dns is the function of the hosts listed w/ fqdn in the hosts
file. The passwd and shadow are going to be
same as the /etc/passwd and /etc/shadow file for all these hosts, same
for sudeors.  users is list of users. may have no purpose
right now.

So, we are talking about a dns functional group based on the FQDN.  In general, I avoid using metadata in the FQDN as a means to classify a given node.  Classification is a human assignment, so I just classify using my node tool (site.pp or external) as the database instead of some conditional statement base on FQDN.  I know this is unorthodox, but I have good reason for despising metadata based hostnames. ( Hostnames make a sorry database! Rant available upon request. )

Secondly,  just for a simplification you can use a single sudoers file for all of your host.  You can specify access based on host groups in the sudoers file itself.  There are some cases (security domains) where you may want to avoid this, but in general I use one sudoers to rule them all.

(root)@puppetmaster:/path/to/groups# ls -lR dns/
dns/:
total 11
-rw-------    1 root     other           1 Aug 23  2005 hosts
-r--r--r--    1 root     other          33 Aug 22  2005 passwd
-r--------    1 root     other          31 Aug 22  2005 shadow
-r--r-----    1 root     root          546 Aug 27  2005 sudoers
-rw-r--r--    1 root     other         152 Feb 21  2006 users

Ok, here is the Puppety part and it is really about organization and reuse.  Forget this host group organizational structure.  It is going to be nothing but trouble in the long run.  Lets think of classes instead as a way to specify configurations via composition and inheritance and lets use modules exclusively.  Explicitly lets create two module paths:

/path/to/modules/dist:

Is where you will build small reusable modules that will be used to compose class that classify your services. And...

/path/to/modules/site

is where you will build larger modules and create complex composite configurations.  Here you will include classes from the dist path. I would avoid including site classes in the classes defined in the dist path.  I like to have the dependencies flow one way.

Ok, so in the site module path lets create a module called acme.  And reorganize based on this structure:

/path/to/modules/site/acme 

currently, I have a test site.pp like this

# site.pp

node basenode {
       case $hostname {
               puppet-test: {}
               default: {}
       }
}

K,  I would avoid doing the condition stuff here.   Instead if we have a node foo lets just assign it the base class acme from our acme module.  This will make our site.pp compatible with an external nodes tool.

node foo { acme: }

On a side note, no need for client server when if we are testing.  Just checkout the dev branch of your puppet modules on the test node, use the puppet executable and pass it a test.pp that includes the classes that you want to test like so:

 

puppet --debug --modulepath=/path/to/modules/dist:/path/to/modules/site test.pp

This is how I training people to develop their puppet code in our classes.  Try it; you'll like it!

Alright, so here we go refactoring this we would have a acme::dns class in our acme module that would include or inherit all the smaller classes that are needed to setup a DNS host.

node 'puppet-test' {
               include dns
               include sudo
               }

So our node definition would now look like...

node 'puppet-test.fqdn.org' { include acme::dns } 

Again, I prefer simple assignment.  Essentially, one class included per node.  I do all the specification that is role based in classes.  If an individual host needs specific parameters set, I either handle that logically inside the classes or assign parameters to the host (External Nodes supports this better).  This allows me to test composite classes as part of the module development process.

We would refactor this class then: 

class dns_users {
       @user { "testuser":
               ensure => "present",
               uid     => "102",
               gid => "1",
               comment => "test user",
               home => "/home/testuser",
               shell => "/bin/bash",
               managehome => "true",
       }
}

Becoming a single virtusers class:

class acme::virtusers {

    @user{

        testuser:

            ensure => "present",

            uid     => "102",
            gid => "1",
            comment => "test user",
            home => "/home/testuser",
            shell => "/bin/bash",

            managehome => "true",

            tag => dns;  # Yup a tag!

        foouser:

            ensure => "present",

            uid     => "103",

            gid => "1",
            comment => "test user",

            home => "/home/foouser",

            shell => "/bin/bash",

            managehome => "true",

            tag => webserver;  # MMMM tags!

     }

 }

We include acme:virtuser in the acme base class.  Then when you want to include a user or set of users in a given role, for example in class acme::dns, just realize that user or set of users based on a tag.  In this case, we are going to use the dns tag that we set using the tag metaparameter.

class acme {

   ...

   include virtusers

}

class acme::dns {

     include acme  # this includes all the base stuff including acme::virtusers

     User <| tag == dns|>  #this realizes the virtual users tag with dns

}

We replaced this with acme::dns above.

class dns {
       include dns_users
       realize(
               User["testuser"]
       )
}

I would expect a generic reusable bind or dns module in the dist module path that handles installing packages or any other resources you may want to manage.  As we develo, a shareable module repo, this directory may be populated with reusable modules sourced from the interwebs and maintained by the community.

class sudo {
   file { sudoers: # a common title for all platforms
       path => $operatingsystem ? {
           solaris => "/opt/csw/etc/sudoers",
           default => "/etc/sudoers"
       },
       owner => root,
       group => root,
       mode => 440,
       source => "puppet:///sudo/sudoers"
   }
}

This would go in a generic sudoers module in dist.  If you really really need to use a different sudo for each role then inheritance is your friend.

class acme::dns::sudo inherits sudo {

    ...

    File['sudoers': source => puppet:///acme/dns/sudoers]  # Simply overriding the source parameter.

}

and in acme::dns:

class acme::dns {

    include acme::dns::sudo

}

   

Instead of creating 300 manifests and that many more users in the
class and/or @users I like to see if there is maybe a template can be
created.
So when the puppet client comes to puppetmaster, based on the fqdn of
the host it will be assigned as part of a group. Then based on the
assigned

So with the refactor that I have suggested, you should be able to avoid the duplication and make the code manageable.  I still would avoid the FQDN based class assignment if possible though.  I am prejudiced.

group it will receive specific sudoers file and a list of users will
be created based on the values in passwd and shadow files.

Looking for recipe to achieve that.

Thanks

 

 Code reuse, clean organization, reduction of conditionals, and simple assignment... FTW.

--
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

Cheers,

Teyo

--
Teyo Tyree :: www.reductivelabs.com

[Asif Iqbal]

Hi Teyo,

I seem to be lost in your explanations. BTW, I do not need to use fqdn.

I realized, I started looking for a recipe that will be very
complicated for a beginner like me.
So I think I should start small and simple and it may grow to a
solution that will be really useful to others.

Lets start w/ real basic.

I have 300 hosts. I like a push a user to about 100 hosts (dns
resolver type hosts) out of 300 total.

How do I set that up within puppet ?

(sorry for the top post. I like to ignore the complex recipe, at least
for me, and may go back to it eventually but gradually)

[David Schmitt]

Asif Iqbal wrote:
> So I think I should start small and simple and it may grow to a
> solution that will be really useful to others.
>
> Lets start w/ real basic.
>
> I have 300 hosts. I like a push a user to about 100 hosts (dns
> resolver type hosts) out of 300 total.
>
> How do I set that up within puppet ?

The very simplest stuff:

| node "dns1", ..., "dns100" {
| user { "foo": ... }
| }

That's of course very trivial. The next steps would be to put the user
into his own class/module where you can encapsulate the user and his
environment (ssh key, shell configuration, ...) and use an external
nodes classifier[1] to find your nodes instead of typing them all out.

You can read many more examples on the wiki [2] and [3]. Also look at
the references linked from the documentation main page[4].

Regards, DavidS

[1] http://reductivelabs.com/trac/puppet/wiki/ExternalNodes
[2] http://reductivelabs.com/trac/puppet/wiki/PuppetModules
[3] http://reductivelabs.com/trac/puppet/wiki/Recipes
[4] http://reductivelabs.com/trac/puppet/wiki/DocumentationStart

[Asif Iqbal]

On Wed, Jul 29, 2009 at 1:57 AM, David Schmitt<da...@dasz.at> wrote:
>
> Asif Iqbal wrote:
>> So I think I should start small and simple and it may grow to a
>> solution that will be really useful to others.
>>
>> Lets start w/ real basic.
>>
>> I have 300 hosts. I like a push a user to about 100 hosts (dns
>> resolver type hosts) out of 300 total.
>>
>> How do I set that up within puppet ?
>
> The very simplest stuff:
>
> | node "dns1", ..., "dns100" {
> |       user { "foo": ... }
> | }

this recipe worked perfect. I have seen the links you posted below and
I like to use them
slowly. I will move to that direction gradually.

For now, the user account created perfectly. Here is the complete recipe

(root)@sys-ubuntu:/etc/puppet/manifests# cat site.pp
# site.pp
# the .pp extension is default and not needed to add

node "puppet-client1","puppet-client2",..."puppet-client10" {

user { "testuser":
ensure => "present",
uid => "102",
gid => "1",
comment => "test user",

home => "/export/home/testuser",

shell => "/bin/bash",
managehome => "true",
}
}

How do I add this user to User_Alias TESTUSERS in the sudoers file on
all these hosts?
Without puppet I would ssh in to all the hosts and run `visudo' and
add the user in that User_Alias.

I looked at the puppet recipe where sudeors file is kept in puppet
server and is pushed to
the puppet clients. For this I need to edit the sudoers file and my
recipe depends on it.
I like it more dynamic. I want puppet client to run the visudo and
append the user in User_Alias.
This way even if my environment grows I don't have to manage multiple
sudoers file on puppet master.

I am going to be using puppet mainly to manage user accounts (password
and group files) and sudoers
file of various formats. Once I am comfortable with it, I will plan to
use it to install packages and
then down the road may be install patches as well. Most of my servers
are solaris.

>
> That's of course very trivial. The next steps would be to put the user
> into his own class/module where you can encapsulate the user and his
> environment (ssh key, shell configuration, ...) and use an external
> nodes classifier[1] to find your nodes instead of typing them all out.
>
> You can read many more examples on the wiki [2] and [3]. Also look at
> the references linked from the documentation main page[4].
>
>
>
> Regards, DavidS
>
>
>
> [1] http://reductivelabs.com/trac/puppet/wiki/ExternalNodes
> [2] http://reductivelabs.com/trac/puppet/wiki/PuppetModules
> [3] http://reductivelabs.com/trac/puppet/wiki/Recipes
> [4] http://reductivelabs.com/trac/puppet/wiki/DocumentationStart
>
>
>
>
>
> >
>

[David Schmitt]

Since there is currently no native sudo type I know of, I'd recommend
using the concatenated_file and concatenated_file_part defines[1] from
my "common" module[2]. Using them you can build your sudoers file on the
nodes from a locally editable header and various parts from your manifests:

class sudo {
concatenated_file { "/etc/sudoers": }
}

class admin1 {
user { admin1: }
concatenated_file_part {
"admin1":
dir => "/etc/sudoers.d",
content => "..."
}
}

node ... {
include admin1
}

Regards, DavidS

[1]http://git.black.co.at/?p=module-common;a=blob;f=manifests/defines/concatenated_file.pp;hb=HEAD
[2]http://git.black.co.at/?p=module-common

[Asif Iqbal]

I am little lost. I dont see my user `testuser' here. I guess `admin1'
could be `testuser' instead if I want
to be consistent with my initial recipe?

Also what would go in the content => .. so that I can append user
`testuser' to the following entry

User_Alias TESTUSERS = user1, user2

in the sudoers file

>
>
>
>
> Regards, DavidS
>
> [1]http://git.black.co.at/?p=module-common;a=blob;f=manifests/defines/concatenated_file.pp;hb=HEAD
> [2]http://git.black.co.at/?p=module-common
>
> >
>

[Asif Iqbal]

On Wed, Jul 29, 2009 at 1:57 AM, David Schmitt<da...@dasz.at> wrote:
>

> Asif Iqbal wrote:
>> So I think I should start small and simple and it may grow to a
>> solution that will be really useful to others.
>>
>> Lets start w/ real basic.
>>
>> I have 300 hosts. I like a push a user to about 100 hosts (dns
>> resolver type hosts) out of 300 total.
>>
>> How do I set that up within puppet ?
>
> The very simplest stuff:
>
> | node "dns1", ..., "dns100" {
> |       user { "foo": ... }
> | }

I tried to expand on it and I have setup a recipe which is not really
working yet

node basenode {
$userlist = [ "user1", "user2", "user3", .. , "user50" ]
$passwd = [ "hashkey1", "hashkey2", .. , "hashkey50" ]
$fullname = [ "fname1 lname1", "fname2 lname2",... , "fname50 lname50" ]
$uids = [ "101", "102", ... , "150" ]
user { $userlist:
ensure => "present",
uid => $uids,
gid => "1",
comment => $fullname,
home => "/export/home/$userlist",
password => $passwd,

shell => "/bin/bash",
managehome => "true",
}
}

node "node01" inherits basenode {}
....
node "node100" inherits basenode {}

>
> That's of course very trivial. The next steps would be to put the user
> into his own class/module where you can encapsulate the user and his
> environment (ssh key, shell configuration, ...) and use an external
> nodes classifier[1] to find your nodes instead of typing them all out.
>
> You can read many more examples on the wiki [2] and [3]. Also look at
> the references linked from the documentation main page[4].
>
>
>
> Regards, DavidS
>
>
>
> [1] http://reductivelabs.com/trac/puppet/wiki/ExternalNodes
> [2] http://reductivelabs.com/trac/puppet/wiki/PuppetModules
> [3] http://reductivelabs.com/trac/puppet/wiki/Recipes
> [4] http://reductivelabs.com/trac/puppet/wiki/DocumentationStart
>
>
>
>
>
> >
>

[David Schmitt]

That's not how it is intended to work. You'll need to create a define to
handle such "parallel" arrays:

define custom_user($passwd, $fullname) {
user { "user${name}":
ensure => present,
uid => 100 + $name,
gid => 1,
comment => $fullname,
home => "/export/home/${name}",

password => $passwd,
shell => "/bin/bash",
managehome => true,
}
}

custom_user {
"1":
passwd => "hashkey1",
fullname => "fname1 lname1";
"2":
passwd => ...
}

See
http://reductivelabs.com/trac/puppet/wiki/LanguageTutorial#definitions
for details on definitions.

Regards, DavidS

[Asif Iqbal]

Here is my updated recipe

#site.pp

class newuser {
define newu ( $uid , gid = 1, $fullname ) {
exec { "/usr/sbin/useradd -m -d /home/$name -c $fullname -u $uid -g
$gid -s /bin/bash $name": }
}

define newid ( $uid , gid = 1, $passwd, $fullname ) {

user { $name:
ensure => present,

uid => "$uid",
gid => "$gid",
comment => "$fullname",
home => "/export/home/$name",
password => "$passwd",

shell => "/bin/bash",
managehome => "true",
}
}

define addgrp ( $groups ) {
exec { "/usr/sbin/usermod -G $groups $name": }
}
}

newuser::newu {
"testuser":

uid => "102",
gid => "1",

fullname => "test user",
}

newuser::newid {
"testu2":
uid => "103",
gid => "10",
passwd => "XyZ123ZyX12",
fullname => "test2 user2",
}

newuser::addgrp {
"testu2":
groups => ["sysadmin", "developer"]
}

node basenode {
include newuser
}

node default inherits basenode {}

It worked somewhat. The group names "sysadmin" and "developer" became
one group "sysadmindeveloper"

I was trying to use the groups array to call the addgrp definition
multiple times.

Also it failed to create user `testuser' using Newuser::Newu

deatils here

debug: Loaded state in 0.00 seconds
debug: //Newuser::Newid[testu2]/User[testu2]: Changing ensure
debug: //Newuser::Newid[testu2]/User[testu2]: 1 change(s)
debug: User[testu2](provider=user_role_add): Executing
'/usr/sbin/useradd -u 103 -s /bin/bash -g 10 -c test2 user2 -d
/export/home/testu2 -m testu2'
notice: //Newuser::Newid[testu2]/User[testu2]/ensure: created
debug: //Newuser::Addgrp[testu2]/Exec[/usr/sbin/usermod -G
sysadmindeveloper testu2]: Changing returns
debug: //Newuser::Addgrp[testu2]/Exec[/usr/sbin/usermod -G
sysadmindeveloper testu2]: 1 change(s)
debug: //Newuser::Addgrp[testu2]/Exec[/usr/sbin/usermod -G
sysadmindeveloper testu2]: Executing '/usr/sbin/usermod -G
sysadmindeveloper testu2'
debug: Executing '/usr/sbin/usermod -G sysadmindeveloper testu2'
err: //Newuser::Addgrp[testu2]/Exec[/usr/sbin/usermod -G
sysadmindeveloper testu2]/returns: change from notrun to 0 failed:
/usr/sbin/usermod -G sysadmindeveloper testu2 returned 3 instead of 0
at /etc/puppet/manifests/site.pp:22
debug: //Newuser::Newu[testuser]/Exec[/usr/sbin/useradd -m -d
/home/testuser -c test user -u 102 -g 1 -s /bin/bash testuser]:
Changing returns
debug: //Newuser::Newu[testuser]/Exec[/usr/sbin/useradd -m -d
/home/testuser -c test user -u 102 -g 1 -s /bin/bash testuser]: 1
change(s)
debug: //Newuser::Newu[testuser]/Exec[/usr/sbin/useradd -m -d
/home/testuser -c test user -u 102 -g 1 -s /bin/bash testuser]:
Executing '/usr/sbin/useradd -m -d /home/testuser -c test user -u 102
-g 1 -s /bin/bash testuser'
debug: Executing '/usr/sbin/useradd -m -d /home/testuser -c test user
-u 102 -g 1 -s /bin/bash testuser'
err: //Newuser::Newu[testuser]/Exec[/usr/sbin/useradd -m -d
/home/testuser -c test user -u 102 -g 1 -s /bin/bash
testuser]/returns: change from notrun to 0 failed: /usr/sbin/useradd
-m -d /home/testuser -c test user -u 102 -g 1 -s /bin/bash testuser
returned 2 instead of 0 at /etc/puppet/manifests/site.pp:5
debug: Finishing transaction 74491050 with 3 changes
debug: Storing state
debug: Stored state in 0.02 seconds
notice: Finished catalog run in 0.41 seconds

>
> See
> http://reductivelabs.com/trac/puppet/wiki/LanguageTutorial#definitions
> for details on definitions.
>
>
> Regards, DavidS

[Asif Iqbal]

Fixed it

define addgrp ( $uname ) {
exec { "/usr/sbin/groupadd $title": }
exec { "/usr/sbin/usermod -G $title $uname": }
}

groups = [ "sysadmin", "developr"]
newuser::addgrp { $groups: uname => "testu2" }

Now I need to explore the external node script to push multiple users.

I have the list of users available in a flat file.

Then I need to find a way to send the same list of users to multiple
hosts

[Larry Ludwig]

Why are you using exec type for user and group, when these types
already exist? I didn't read the complete thread so I donno if this
was discussed. The exec type should always be used as the last resort.

-L

--
Larry Ludwig
Reductive Labs

[Asif Iqbal]

On Tue, Aug 4, 2009 at 3:44 PM, Larry Ludwig<la...@reductivelabs.com> wrote:
>
>
> Why are you using exec type for user and group, when these types
> already exist?  I didn't read the complete thread so I donno if this

I am still learning.

So I guess I could modify the newuser::newid like following

class newuser {
.......

define newid ( $uid , gid = 1, $passwd, $fullname ) {
user { $name:
ensure => present,
uid => "$uid",
gid => "$gid",

groups => [ "sysadmin" , "developr" ]

comment => "$fullname",
home => "/export/home/$name",
password => "$passwd",
shell => "/bin/bash",
managehome => "true",
}
}

.........

}

> was discussed.  The exec type should always be used as the last resort.

it is better not to read the whole thread.. since it started with me
being very confused about how
puppet works to now when I have some idea, heh

Now I am studying to see how to create say 50 users on say 100 hosts
using external node.

>
> -L
>
> --
> Larry Ludwig
> Reductive Labs
>
>
> >
>