Creating multiple directories based on variables or array

[Ugo Bellavance]

Hi,

I'd need to create a number of directories in a fasion similar to that:

/var/extlogs/$stage/$client-$application

For example, if stages are integration, staging, production and clients are client1 and client2 (there are many morem in reality), I'd like to have puppet create

/var/extlogs/integration/client1-application1

/var/extlogs/integration/client1-application2

/var/extlogs/integration/client2-application1

/var/extlogs/integration/client2-application2

/var/extlogs/staging/client1-application1

/var/extlogs/staging/client1-application2

/var/extlogs/staging/client2-application1

/var/extlogs/staging/client2-application2

/var/extlogs/production/client1-application1

/var/extlogs/production/client1-application2

/var/extlogs/production/client2-application1

/var/extlogs/production/client2-application1

I've created a class that eases the job:

define lsyncd::createdestdirs (

    $application      = '',

    $client           = '',

    $envstage         = '',

) {

  file {

    '/var/extlogs':

      ensure => directory,

      owner  => "root",

      group  => "root",

      mode   => 0755,

  }

  file {

    "/var/extlogs/${envstage}":

      ensure => directory,

      owner  => "root",

      group  => "root",

      mode   => 0755,

      require => File['/var/extlogs'],

  }

  file {

    "/var/extlogs/${envstage}/${client}-${application}":

      ensure => directory,

      owner  => "root",

      group  => "root",

      mode   => 0755,

      require => File["/var/www/extlogs/${envstage}"],

  }

}

I can simply call something like

  lsyncd::createdestdirs { 'client1-application1':

    application => 'application1',

    client      => 'client1',

    envstage    => 'production',

    }

to create one directory, but would there be a way to have all the directories created in one directive, by setting $application, $client, and $stage in an array?

I'm currently using puppet 2.7.25 and I'm not using hiera or puppetdb.

Thanks,

Ugo

[Dan White]

Here's a snippet from my user management define that shows how I deal with making a directory that might be many levels deep : 

      # create the homedir-dir

      exec { "mkdir-${username}":

        command => "/bin/mkdir -p ${homedirdir} ; /bin/chgrp ${gname} ${homedirdir}",

        unless  => "/usr/bin/test -d ${homedirdir}",

        require => Group[$gname],

      }

      # create the homedir

      file { $userhome :

        ensure  => directory,

        require => [

          User[$username],

          Exec["mkdir-${username}"],

        ]

      }

Note the "chgrp" command stacked in the command line.  It did not want to work gracefully in the "file" resource.

“Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.” (Bill Waterson: Calvin & Hobbes)

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/c9182f72-ae82-4a89-95e1-e297dc74aa5a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan White]

OOOOPS !  For got one bit that goes in front of the other two lines :

  # account creation requires that the homedir directory exists

  $homedirdir = dirname ( $userhome )

Ignore the comment about the "chgrp" command.  I forgot that it is for the parent directory.

“Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.” (Bill Waterson: Calvin & Hobbes)

[jcbollinger]

On Tuesday, October 13, 2015 at 12:20:05 PM UTC-5, Ugo Bellavance wrote:

Hi,

I'd need to create a number of directories in a fasion similar to that:

/var/extlogs/$stage/$client-$application

For example, if stages are integration, staging, production and clients are client1 and client2 (there are many morem in reality), I'd like to have puppet create

/var/extlogs/integration/client1-application1

/var/extlogs/integration/client1-application2

/var/extlogs/integration/client2-application1

/var/extlogs/integration/client2-application2

/var/extlogs/staging/client1-application1

/var/extlogs/staging/client1-application2

/var/extlogs/staging/client2-application1

/var/extlogs/staging/client2-application2

/var/extlogs/production/client1-application1

/var/extlogs/production/client1-application2

/var/extlogs/production/client2-application1

/var/extlogs/production/client2-application1

I've created a class that eases the job:

Well no, you've created a defined type.  It's very important to understand the difference.

 

define lsyncd::createdestdirs (

    $application      = '',

    $client           = '',

    $envstage         = '',

) {

  file {

    '/var/extlogs':

      ensure => directory,

      owner  => "root",

      group  => "root",

      mode   => 0755,

  }

This is not actually working for you as written.  At least, not for more than one combination of 'application', 'client', and 'envstage' per node.  You would end up with duplicate declarations at least of File['/var/extlogs'], at least.

 

  file {

    "/var/extlogs/${envstage}":

      ensure => directory,

      owner  => "root",

      group  => "root",

      mode   => 0755,

      require => File['/var/extlogs'],

  }

Note that although it is by no means wrong to declare that dependency on the parent directory, Puppet will generate it automatically for you if you omit it.  I rely on that in my example code below.

 

[...] would there be a way to have all the directories created in one directive, by setting $application, $client, and $stage in an array?

So you want to create a directory for every combination of $application, $client, and $stage values?  Historically, that sort of thing has been difficult, but there are good ways to handle it in modern Puppet.

 

I'm currently using puppet 2.7.25 and I'm not using hiera or puppetdb.

Unfortunately, you're stuck in "historically".  Way back in "historically", in fact: the latest Puppet is 4.2.  This problem can still be handled, but it's not as clean or easy.  Moreover, although puppetdb is not really relevant to the question, Hiera is.  In Puppet 2.7, you have four alternatives for providing data for your classes to operate upon (your various 'client', etc. arrays):

• you can build the data directly into your classes; or
  
• you can parameterize your classes and use resource-like class declarations; or
  
• you can look up your data via an external service, with hiera() and extlookup() being the most common; or
• you can declare them as variables in a scope visible from your class, such as at node scope or even top scope.
  

Use of resource-like class declarations has always been problematic, ever since parameterized classes were first introduced.  There are a few specialized cases were it is relatively safe, but I rarely recommend resource-like class declarations to anyone.  Since Puppet 2 provided no alternative, parameterized classes were very problematic until the introduction of automated data binding (via Hiera) in Puppet 3.

Extlookup() does not provide for structured data (e.g. arrays), and you're not using Hiera.  If you want to be able to have different nodes with different clients, stages, or applications, then you cannot encode these data directly into the class that uses them.

Let's say, therefore, that you will declare

$applications = [ 'application1', 'application2' ]
$clients      = [ 'client1', 'client2' ]
$envstages    = [ 'integration', 'staging', 'production' ]

at top scope.  You could do that via an ENC, but you could also just put such variable definitions in your site.pp or any other manifest `import'ed by it.  You might then have a class (really a class) that manages the overall directory set, a defined type whose multiple instances will each manage one of the stage directories, and a helper defined type whose instances manage the ultimate directories for each stage:

class lsyncd::destdirs {
  $basedir = '/var/extlogs'

  # resource defaults save a bit of typing:
  File {

      owner  => 'root',

      group  => 'root',

      mode   => '0755'

  }
   
  # The overall base directory

  file { $basedir: ensure => 'directory' }

  # Since (if) $envstages is an array, this declares one
  # lsyncd::stagedir for each element:

  lsyncd::stagedir { ${envstages}: basedir => $basedir }

}

define lsyncd::stagedir($basedir) {
  # The name of the directory for the stage managed by
  # this resource:
  $stagebase = "${basedir}/${title}"

  # Manage the directory; resource defaults from
  # lsyncd::destdirs are still in effect here
  file { $stagebase: ensure => 'directory' }

  # One resource for this stage for each client:
  lsyncd::stageclient { ${client}: stagebase => $stagebase }
}

define lsyncd::stageclient ($stagebase) {
  # regsubst() uses a regex substitution to create one result
  # string -- in this case a directory name -- for each element
  # of $applications.  The title and parameter are interpolated
  # into the results to form the target directory names.
  $appdirs = regsubst($applications, '.*', "${stagebase}/${title}-\1")

  # Manage the ultimate directories; resource defaults from
  # lsyncd::destdirs are still in effect here
  file { ${appdirs}: ensure => 'directory' }
}


John

[Ugo Bellavance]

Thanks for your awesome, complete answer John.

Ugo

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/Xd0Zx9LPa7k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/38280ea6-3e2c-43f9-986b-f90aa79fbd42%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Ugo Bellavance (ug...@lubik.ca)