Re: Have Class Only Perform Actions When There Is Work To Do (i.e. Making Them Idempotent)

[Ellison Marks]

If you're using this wget module

https://github.com/liquidstate/puppet-wget/blob/master/manifests/init.pp

It's set to not run so long as the destination is still there, so /tmp/Python-${version}.tgz needs to stick around. I'm not sure which archive module you're using, but as long as you have that require there, it should follow.

On Friday, October 26, 2012 6:55:32 AM UTC-7, Dave Mankoff wrote:

Howdy. I feel like I am missing something really simply with regards to the way that Puppet works and I am wondering if someone can point me in the write direction.

I have written a class that downloads, uncompresses, compiles, and installs Python from source. So far so good. The problem is that it only needs to do this once, when Python is not already in place (or some other custom indicator of the Python version). I have my 3 calls to exec doing their checks just fine, but my calls to wget::fetch and archive::untar both fire during every apply. Specifically, archive::untar takes about 30 seconds to run and I'd prefer it if it only ran conditionally. 

What is the best way to make sure that this code:

  wget::fetch { "python-${version}":

    source => "http://python.org/ftp/python/${version}/Python-${version}.tgz",

    destination => "/tmp/Python-${version}.tgz",

  }

  archive::untar {"/tmp/python-${version}":

    source => "/tmp/Python-${version}.tgz",

    compression => 'gz',

    rootdir => "Python-${version}",

    require => Wget::Fetch["python-${version}"],

  }

only runs when some condition is met? I can easily put a custom file in place to look for, but how do I make these commands dependent on its absence? I tried making such a file and subscribing to it, but these commands still ran each time.

[Dave Mankoff]

Yeah, the wget seems to be working. The archive library I am using is:

http://forge.puppetlabs.com/bobsh/archive

It runs irrespective of whether or not wget actually fetches anything, however.

At this point, it seems that the exec type is the only thing that can actually conditionally run (unless, onlyif, creates, etc). I created a bash script in my templates directory that contains the wget and tar commands. I now copy that bash script out and only conditionally run it, which seems to defeat the whole purpose.

[Ellison Marks]

Ah, right, my mistake, the require won't guarantee anything in this case, as the other command is successful, even if it does nothing. It's early...

However, looking at that module it seems like it's trying to use refreshonly, but it doesn't look like it will be effective the way it's using it. You might try adding

creates => $name,

under the

exec {$exec_name:

resource in untar.pp. This would make it dependent on the name given to archive::untar, /tmp/python-${version} in this case.

[Tim Mooney]

In regard to: [Puppet Users] Have Class Only Perform Actions When There Is...:

> Howdy. I feel like I am missing something really simply with regards to the
> way that Puppet works and I am wondering if someone can point me in the
> write direction.
>
> I have written a class that downloads, uncompresses, compiles, and installs
> Python from source. So far so good.

I would highly recommend you just package your custom python and install
it using a package management system, rather than doing what you're doing.
Depending on what host OS you're using, it's not too difficult, and it
works a lot better with puppet *and* you get all the benefits of having
the package installed through a more coherent means.

> The problem is that it only needs to do
> this once, when Python is not already in place (or some other custom
> indicator of the Python version). I have my 3 calls to exec doing their
> checks just fine, but my calls to wget::fetch and archive::untar both fire
> during every apply. Specifically, archive::untar takes about 30 seconds to
> run and I'd prefer it if it only ran conditionally.
>
> What is the best way to make sure that this code:
>
> wget::fetch { "python-${version}":
> source =>
> "http://python.org/ftp/python/${version}/Python-${version}.tgz",
> destination => "/tmp/Python-${version}.tgz",
> }
>
> archive::untar {"/tmp/python-${version}":
> source => "/tmp/Python-${version}.tgz",
> compression => 'gz',
> rootdir => "Python-${version}",
> require => Wget::Fetch["python-${version}"],
> }

As you're discovering, these kind of exec chains, where the first part
of the chain involves temporary files, don't really fit into the puppet
paradigm very well. About the best you can do is something like

exec { '/usr/local/sbin/install-python-if-necessary.sh':
source => 'http://your_module/install-python-if-necessary.sh',
creates => '/your/python/lib/dir',
}

and then bury all the fetch/extract/configure/compile/install logic
in the shell script, which puppet will make certain is always present
on the system. It will only execute it if /your/python/lib/dir is not
present.

But if you're going to build fetch/extract/configure/compile/install logic
into a shell script, you're probably 85% of the way to packaging the
software appropriately anyway.

Tim
--
Tim Mooney Tim.M...@ndsu.edu
Enterprise Computing & Infrastructure 701-231-1076 (Voice)
Room 242-J6, IACC Building 701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164

[Dave Mankoff]

On Friday, October 26, 2012 2:31:56 PM UTC-4, Tim Mooney wrote:

In regard to: [Puppet Users] Have Class Only Perform Actions When There Is...:

I would highly recommend you just package your custom python and install
it using a package management system, rather than doing what you're doing.
Depending on what host OS you're using, it's not too difficult, and it
works a lot better with puppet *and* you get all the benefits of having
the package installed through a more coherent means.

As you're discovering, these kind of exec chains, where the first part
of the chain involves temporary files, don't really fit into the puppet
paradigm very well.  About the best you can do is something like

         exec { '/usr/local/sbin/install-python-if-necessary.sh':
                 source  => 'http://your_module/install-python-if-necessary.sh',
                 creates => '/your/python/lib/dir',
         }

and then bury all the fetch/extract/configure/compile/install logic
in the shell script, which puppet will make certain is always present
on the system.  It will only execute it if /your/python/lib/dir is not
present.

But if you're going to build fetch/extract/configure/compile/install logic
into a shell script, you're probably 85% of the way to packaging the
software appropriately anyway.

Interesting. It sounds like you're actually advocating _for_ the bash script approach. I wanted to avoid package management systems only because they are way more complicated than a basic install of python requires:

  wget python.tgz

  tar -xzvf python.tgz

  cd python

  ./configure --prefix=/install/path

  make

  make test

  make install

I wanted to make (I have made?) a simple "python" class that accepts a python version number, downloads it, and runs those steps, irrespective of the base Linux flavor. I don't know much of anything about deb or rpm files other than that they are more complicated than that; and I don't want to have to package up and maintain several different python versions for several different package managers. 

[Stefan Schulte]

On Fri, Oct 26, 2012 at 06:55:32AM -0700, Dave Mankoff wrote:
> Howdy. I feel like I am missing something really simply with regards to the
> way that Puppet works and I am wondering if someone can point me in the
> write direction.
>
> I have written a class that downloads, uncompresses, compiles, and installs

> Python from source. So far so good. The problem is that it only needs to do

> this once, when Python is not already in place (or some other custom
> indicator of the Python version). I have my 3 calls to exec doing their
> checks just fine, but my calls to wget::fetch and archive::untar both fire
> during every apply. Specifically, archive::untar takes about 30 seconds to
> run and I'd prefer it if it only ran conditionally.
>
> What is the best way to make sure that this code:
>
> wget::fetch { "python-${version}":
> source =>
> "http://python.org/ftp/python/${version}/Python-${version}.tgz",
> destination => "/tmp/Python-${version}.tgz",
> }
>
> archive::untar {"/tmp/python-${version}":
> source => "/tmp/Python-${version}.tgz",
> compression => 'gz',
> rootdir => "Python-${version}",
> require => Wget::Fetch["python-${version}"],
> }
>

> only runs when some condition is met? I can easily put a custom file in
> place to look for, but how do I make these commands dependent on its
> absence? I tried making such a file and subscribing to it, but these
> commands still ran each time.
>

You don't tell us how wget::fetch is implemented so I can only guess
that there is an exec resource in there? The wget::fetch resource is
*always* evaluated so you have to make sure that the exec resource inside
does not do anything (the exec resource has a `creates` parameter you
can point to a file. If this file is present the command specified by
the `command� parameter will not run).

Is there a reason why you do not install python as a package or build a
package your own?

-Stefan

[dave]

You don't tell us how wget::fetch is implemented so I can only guess
that there is an exec resource in there? The wget::fetch resource is
*always* evaluated so you have to make sure that the exec resource inside
does not do anything (the exec resource has a `creates` parameter you
can point to a file. If this file is present the command specified by
the `command´ parameter will not run).

The wget package is: http://forge.puppetlabs.com/maestrodev/wget

The archive package is: http://forge.puppetlabs.com/bobsh/archive

 

Is there a reason why you do not install python as a package or build a
package your own?

I wanted a python class where I could declare the version I wanted and have it download and make-d. Most the package management systems I've seen come with 2.6 (or older!) by default. 

I didn't want to create my own package for reasons I listed in another email: extra complication (python is really easy to make from source) and extra maintenance (would have to maintain different python versions for every package management system I wanted to support). I really just wanted to create a simple puppet script to download and install python in a consistent fashion on whatever system I happened to be setting up.

-dave

[Tim Mooney]

In regard to: Re: [Puppet Users] Have Class Only Perform Actions When There...:

>
>
> On Friday, October 26, 2012 2:31:56 PM UTC-4, Tim Mooney wrote:
>>
>> In regard to: [Puppet Users] Have Class Only Perform Actions When There
>> Is...:
>>
>> I would highly recommend you just package your custom python and install
>> it using a package management system, rather than doing what you're doing.
>> Depending on what host OS you're using, it's not too difficult, and it
>> works a lot better with puppet *and* you get all the benefits of having
>> the package installed through a more coherent means.
>>
>> As you're discovering, these kind of exec chains, where the first part
>> of the chain involves temporary files, don't really fit into the puppet
>> paradigm very well. About the best you can do is something like
>>
>> exec { '/usr/local/sbin/install-python-if-necessary.sh':
>> source => '
>> http://your_module/install-python-if-necessary.sh',
>> creates => '/your/python/lib/dir',
>> }
>>
>> and then bury all the fetch/extract/configure/compile/install logic
>> in the shell script, which puppet will make certain is always present
>> on the system. It will only execute it if /your/python/lib/dir is not
>> present.
>>
>> But if you're going to build fetch/extract/configure/compile/install logic
>> into a shell script, you're probably 85% of the way to packaging the
>> software appropriately anyway.
>>
>>
> Interesting. It sounds like you're actually advocating _for_ the bash
> script approach.

Absolutely not. I think the package management system is definitely the
way to go. If you're bound and determined to not do it that way, and
you want to avoid having the pre-cursor exec's fire in your
fetch/extract/compile chain, then the shell script is really your
best option.

Otherwise, the wget fetch will fire any time that
/tmp/Python-${version}.tgz is not present. That won't be present if
your /tmp cleaner gets rid of it. Ditto for the extract step.

I'm assuming that none of these systems are multi-user systems, where you
might have less-than-trustworthy people with local access on the system.
If that's not the case, I definitely wouldn't be having the first step
of a multi-exec chain depend on a file in /tmp.

> I wanted to avoid package management systems only because
> they are way more complicated than a basic install of python requires:
>
> wget python.tgz
> tar -xzvf python.tgz
> cd python
> ./configure --prefix=/install/path
> make
> make test
> make install
>
> I wanted to make (I have made?) a simple "python" class that accepts a
> python version number, downloads it, and runs those steps, irrespective of
> the base Linux flavor. I don't know much of anything about deb or rpm files
> other than that they are more complicated than that; and I don't want to
> have to package up and maintain several different python versions for
> several different package managers.

You don't have to create your package from whole cloth, though. You
download the source format for the package management system, extract
the recipe file, make a couple of changes for your local install, and
then build the package.

In the case of an RPM, that means starting with the source RPM (.srpm) for
the python that comes from your CentOS/RHEL/RHEL-derived vendor. You
would extract the "spec" file (the recipe), tweak the %_prefix, the
Version, and probably not much else.

If you're still against packaging it, there's external repos you might
consider, to get packages that someone else has created. That might not
meet your needs, though.

Ultimately, use puppet however it works best for your organization. Just
be advised that some things fit better into the puppet paradigm than
others.

[Aaron Grewell]

>>>
>>> I would highly recommend you just package your custom python and install
>>> it using a package management system, rather than doing what you're
>>> doing.

In this case you really ought to consider packaging, but there's
always *something* that doesn't work that way for whatever reason
(badly-wrapped vendor software is a favorite here). IMHO a tarball
installer is a necessary evil in certain situations. It'll probably
take a bit of tweaking to get it working in your environment , but
this works well for me.

# Define: packages::tar::install
#
# This define installs tar-based packages, including making sure they're
# only installed once and performing cleanup after the installation.
#
# Sample Usage:
# packages::tar::install { 'vmware-solaris-tools':
# package => 'vmware-solaris-tools-381511.tar.gz', # name of tarball
# repo => 'http://hostname/path', # ftp or http path minus filename
# dir => 'vmware-tools-distrib', # top-level directory in the tarball
# installer => 'vmware-install.pl', # name of install script
# options => '-d', # options to pass to install script
# tmpdir => '/opt/tmp', # dir to extract tarball into
# }
define packages::tar::install($repo,$package,$dir,$installer,$options,$tmpdir='/var/tmp',
$timeout='600',$dotdir='/opt/puppet/libexec'){

exec { "wget -O $tmpdir/$package $repo/$package":
unless => "/usr/bin/test -f ${dotdir}/.${package}",
path => ["/opt/csw/bin","/usr/bin"],
alias => "wget_${package}",
require => File["$tmpdir"],
}

exec { "gunzip -c $tmpdir/$package | tar xf - ":
unless => "/usr/bin/test -f ${dotdir}/.${package}",
path => ["/bin","/usr/bin","/usr/sbin"],
alias => "untar_${package}",
cwd => "$tmpdir",
require => Exec["wget_${package}"],
}

exec { "$tmpdir/$dir/$installer $options":
unless => "/usr/bin/test -f ${dotdir}/.${package}",
cwd => "$tmpdir/$dir",
alias => "install_${package}",
timeout => "$timeout",
require => Exec["untar_${package}"],
}

exec { "touch ${dotdir}/.${package}":
path => ["/bin","/usr/bin"],
unless => "/usr/bin/test -f ${dotdir}/.${package}",
alias => "${package}_dotfile",
require => Exec["install_${package}"],
}

exec { "rm -rf $tmpdir/$dir":
path => ["/bin","/usr/bin"],
onlyif => "/usr/bin/test -d $tmpdir/$dir",
cwd => "$tmpdir",
alias => "rm_${dir}",
require => Exec["install_${package}"],
}

exec { "rm -f $tmpdir/$package":
path => ["/bin","/usr/bin"],
onlyif => "/usr/bin/test -f $tmpdir/$package",
cwd => "$tmpdir",
alias => "rm_${package}",
require => Exec["install_${package}"],
}
}

[joe dhonny]

Hi Dave,

Am 26.10.2012 20:50, schrieb Dave Mankoff:
> Interesting. It sounds like you're actually advocating _for_ the bash
> script approach. I wanted to avoid package management systems only because
> they are way more complicated than a basic install of python requires:
>
> wget python.tgz
> tar -xzvf python.tgz
> cd python
> ./configure --prefix=/install/path
> make
> make test
> make install
>
> I wanted to make (I have made?) a simple "python" class that accepts a
> python version number, downloads it, and runs those steps, irrespective of
> the base Linux flavor.

Fetch a copy of the Puppet 2.7 Cookbook and read pages 130 and 131,
there's a recipe that does exactly what you want.

> I don't know much of anything about deb or rpm files
> other than that they are more complicated than that; and I don't want to
> have to package up and maintain several different python versions for
> several different package managers.

For deb files it's pretty easy: install "checkinstall" on your desktop,
type "checkinstall make install" instead of "make install" and your
package is ready to go.

The "alien" package comes in handy to from time to time, as it converts
.tgz to .deb or .rpm

hth

[jcbollinger]

On Friday, October 26, 2012 1:50:53 PM UTC-5, Dave Mankoff wrote:

[...] I wanted to avoid package management systems only because they are way more complicated than a basic install of python requires:

  wget python.tgz

  tar -xzvf python.tgz

  cd python

  ./configure --prefix=/install/path

  make

  make test

  make install

I wanted to make (I have made?) a simple "python" class that accepts a python version number, downloads it, and runs those steps, irrespective of the base Linux flavor. I don't know much of anything about deb or rpm files other than that they are more complicated than that; and I don't want to have to package up and maintain several different python versions for several different package managers. 

And I am startled that you actually want to install anything on your systems without using your package manager.  Aside from making it pretty darn easy to install software, your package manager makes it easy to

• know what software is installed, including for automated tools such as Puppet,
  
• ensure that your software's dependencies are installed, and remain installed,
• have your software automatically installed as a dependency of other software,
  
• know what files belong to each package, and identify files that don't belong to any package,
• ensure that every install of your software is the same (to the extent that the same package is used everywhere)
  
• prevent conflicts between packages,
• cleanly remove everything that belongs to a package (init scripts? man pages? documentation? config files? you can't in general rely on everything to be in the same place)
  
• optionally, to automate additional management tasks that should attend installation / upgrade / removal of the software
  
• avoid having development tools installed on most systems
  

I've probably missed a couple.  Software packaging is a major win, very much offsetting the cost of building your own packages where you need to do so.

There is a bit of a learning curve for building your own packages, multiplied by the number of packaging systems you actually need to deal with, but it's really not that hard.  If you're just packaging a different version of something for which your distro already provides packages, then as Tim said, you can probably get away with just making a few changes to the existing packaging instructions.


John

[John Warburton]

On 30 October 2012 01:59, jcbollinger <John.Bo...@stjude.org> wrote:
> On Friday, October 26, 2012 1:50:53 PM UTC-5, Dave Mankoff wrote:
>>
>> [...] I wanted to avoid package management systems only because they are
>> way more complicated than a basic install of python requires:

> There is a bit of a learning curve for building your own packages,
> multiplied by the number of packaging systems you actually need to deal

https://github.com/jordansissel/fpm