Manage delivery and execution of RPMs not in YUM

[warron.french]

Hello all,

can someone please advise me on a proper set of syntax (a file to look at) for an example to follow to solve the following challenge:

1. I have 2 deliver 2 *.rpm files that are not in a YUM repository, so I dropped them into the files directory of my module path.
2. I need to be able to execute each of them either together, or    A.rpm before B.rpm
3. Then execute a shell script that requires the 2 RPMs to be in place before that happens.
   

I am starting to get into slightly more complicated modules, instead of simply delivering basic ASCII text files using  content => template('modulename/some.erb').

I just need an example that is know to provide proper execution, proper syntax, and something I can learn from correctly.  I am still building the foundation of my understanding, so troubleshooting someone else's code isn't going to be too good for my development yet.

Thank you in advance,

--------------------------
Warron French

[Garrett Honeycutt]

On 3/2/17 9:58 AM, warron.french wrote:
> Hello all,
> can someone please advise me on a proper set of syntax (a file to look
> at) for an example to follow to solve the following challenge:
>

> 1. I have 2 deliver 2 *.rpm files that are not in a YUM repository, so

> I dropped them into the files directory of my module path.

> 2. I need to be able to execute each of them either together, or
> _A.rpm before B.rpm_
> 3. __Then execute a shell script that requires the 2 RPMs to be in

> place before that happens.
>
> I am starting to get into slightly more complicated modules, instead of

> simply delivering basic ASCII text files using *content =>
> template('modulename/some.erb')*.

>
> I just need an example that is know to provide proper execution, proper
> syntax, and something I can learn from correctly. I am still building
> the foundation of my understanding, so troubleshooting someone else's
> code isn't going to be too good for my development yet.
>
>
> Thank you in advance,
> --------------------------
> Warron French
>

Hi Warron,

What you want to accomplish is a bad idea and you should use a yum repo
and definitely not check in binary data with your modules. You could at
least store the rpm's somewhere and then download them from that
canonical source. Take a look at Artifactory which can help with where
to store things such as your random rpm's.

Sometimes you have to automate what you have before you build something
better. Suggest writing an exec resource that can handle what you are
trying to do. The key here is to have two commands. One that checks to
see if you are already in the desired state and another to get you to
the desired state. Figure that out without Puppet and once you have
those commands, you can write a manifest.

Best regards,
-g

--
Garrett Honeycutt
@learnpuppet
Puppet Training with LearnPuppet.com
Mobile: +1.206.414.8658

[warron.french]

Garrett, thanks.

So, to clarify for myself in terms of a BEST practice are you declaring "don't deliver RPMs as part of the payload of the Puppet Module?"  I just got that part working.  :-/  I don't mind correction, but I don't want to go down the rabbit hole.

Secondly, using an exec resource to implement the RPMs?

Perhaps something like this...

exec { 'install_cctk_rpms':
         creates => '/opt/dell/dcc/cctk',
         command => 'yum localinstall -y A.rpm B.rpm',
         returns => '0',
}

I have never written an exec resource declaration before.  Can you tell me if the exec syntax is correct, and that it is also what you meant for having to commands; I can combine them into a single yum localinstall -y  command correct?

Thanks Garrett.

--------------------------
Warron French

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/4fc045bb-3e5f-f9d4-88a6-688ca3e3436b%40garretthoneycutt.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Grimberg]

Repositories should _never_ contain binary objects. The only exception I
ever allow my developers is graphical assets related to websites.

On 03/02/2017 08:48 AM, warron.french wrote:
> Garrett, thanks.
>
> So, to clarify for myself in terms of a BEST practice are you declaring

> "don't deliver RPMs as part of the payload of the Puppet Module?" *I
> just got that part working. :-/* I don't mind correction, but I don't

> want to go down the rabbit hole.
>
> Secondly, using an exec resource to implement the RPMs?
>
> Perhaps something like this...
>
> exec { 'install_cctk_rpms':
> creates => '/opt/dell/dcc/cctk',
> command => 'yum localinstall -y A.rpm B.rpm',
> returns => '0',
> }
>
> I have never written an exec resource declaration before. Can you tell
> me if the exec syntax is correct, and that it is also what you meant for

> having to commands; I can combine them into a single *yum localinstall
> -y *command correct?

> Mobile: +1.206.414.8658 <tel:%2B1.206.414.8658>

>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to puppet-users...@googlegroups.com
> <mailto:puppet-users%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/4fc045bb-3e5f-f9d4-88a6-688ca3e3436b%40garretthoneycutt.com

> <https://groups.google.com/d/msgid/puppet-users/4fc045bb-3e5f-f9d4-88a6-688ca3e3436b%40garretthoneycutt.com>.

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/CAJdJdQkCQJ%2BXE2th_OLyu7%2BZyJDROfyht9UC906_JXRn%3D0Q7Dg%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-users/CAJdJdQkCQJ%2BXE2th_OLyu7%2BZyJDROfyht9UC906_JXRn%3D0Q7Dg%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[James Pryor]

Warron,
Correct. Garret is right. It is not best to deliver RPMs as part of the payload-contents of the Puppet Module, and do not store RPMs in a git repo (or whatever version control system you might be using).

Yes, "yum localinstall" exists and but RPMs are best delivered via the OS's native packaging tools like yum (or dnf) for the Red Hat family of Linux distribution. Even if some independent software vendor creates and offers RPMs for their product, those RPMs should be placed in a yum repo either by the vendor hosting it, or by the sysadmin locally hosting it.

I have a quick and dirty example of creating a Yum repo from a directory of RPMs, and then using yum command to install from that repo:
https://gist.github.com/jjpryor/67b2a4822069f57e852c3cb71bc778e3#create-a-yum-repo-from-directory-of-rpms

What that quick and dirty example does not contain is setting up HTTPS, and GPG key management, and GPG signing the RPMs so that authenticity and integrity can be guaranteed. Those steps are usually required in following RPM & YUM best practices for the greater internet.

So the above should solve #1 from your numbered list in your opening email.

If #1 is solved via a Yum repo, then #2 is solved like so:
[root@localhost myinstallmodule]# puppet --version
3.6.2
root@localhost myinstallmodule]# tree
.
|-- manifests
|   `-- init.pp
|-- metadata.json
|-- Rakefile
|-- README.md
|-- spec
|   |-- classes
|   |   `-- init_spec.rb
|   `-- spec_helper.rb
`-- tests
    `-- init.pp

4 directories, 7 files
[root@localhost myinstallmodule]# cat manifests/init.pp
class myinstallmodule {
  package { 'mesecond':
    ensure => present,
  }
  package { 'mefirst':
    ensure => present,
  }
  Package['mefirst'] -> Package['mesecond']
}
[root@localhost myinstallmodule]# cat tests/init.pp
include myinstallmodule
[root@localhost myinstallmodule]# puppet apply tests/init.pp
Notice: Compiled catalog for localhost in environment production in 0.26 seconds
Notice: /Stage[main]/Myinstallmodule/Package[mefirst]/ensure: created
Notice: /Stage[main]/Myinstallmodule/Package[mesecond]/ensure: created
Notice: Finished catalog run in 4.76 seconds

I hope this helps.

James

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJdJdQkCQJ%2BXE2th_OLyu7%2BZyJDROfyht9UC906_JXRn%3D0Q7Dg%40mail.gmail.com.

[James Pryor]

Warron,
And if my previous email solves #1 & #2, then #3 is an addition of an exec resource, then adding it at the end of the dependency chain:

[root@localhost myinstallmodule]# cat manifests/init.pp
class myinstallmodule {

  exec { 'dothething':
    command => 'echo "look it works!"; logger look it works',
    path    => '/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin',

  }
  package { 'mesecond':
    ensure => present,
  }
  package { 'mefirst':
    ensure => present,
  }

  Package['mefirst'] -> Package['mesecond'] -> Exec['dothething']
}
[root@localhost myinstallmodule]# puppet apply tests/init.pp --verbose
Notice: Compiled catalog for localhost in environment production in 0.28 seconds
Info: Applying configuration version '1488482217'

Notice: /Stage[main]/Myinstallmodule/Package[mefirst]/ensure: created
Notice: /Stage[main]/Myinstallmodule/Package[mesecond]/ensure: created

Notice: /Stage[main]/Myinstallmodule/Exec[dothething]/returns: executed successfully
Notice: Finished catalog run in 4.37 seconds
[root@localhost myinstallmodule]# tail /var/log/messages
Mar  2 14:16:59 localhost yum[13827]: Installed: mefirst-1.0.0-1.el7.x86_64
Mar  2 14:17:01 localhost yum[13840]: Installed: mesecond-1.0.0-1.el7.x86_64
Mar  2 14:17:02 localhost vagrant: look it works

I hope this has been fruitful and must bow out as I don't have any more spare-time to give to this list today, sorry.

Regards,

James

[Rich Burroughs]

There are a number of reasons it's not a great idea to put them in the module, but one is that if you start sticking binary artifacts into your Puppet code, the size of the repos(s) will grow a lot and it will be much slower to clone them. Also it's just not how people expect things to work.

Say you onboard a new member of your team and they need to deal with that RPM -- your Puppet module is probably the last place they would ever look for it. Beyond the technical reasons, I think this is an important one. The less snowflakey you can make your automation, the easier it's going to be for new folks to learn to work with it.

Lots of good suggestions in the thread: setting up a Yum repo, or using an artifact repository like Artifactory or Nexus. At my last shop we used Spacewalk:

http://spacewalk.redhat.com/

At my first job where I did Puppet like 4-5 years ago we had Solaris hosts that used a lot of custom packages. We served those up with just a vanilla HTTP server and used wget to pull them down in our Puppet code. Pretty ugly but still better than checking them in with the modules themselves.

Whenever you can leverage your OS's native packaging system, you're going to make it easier on yourself and your coworkers.

Rich

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJdJdQkTwF4HTh8r54HZ_aNittaLZ7UMEKEZTEgpaJ9vU8R5mw%40mail.gmail.com.

[Rob Nelson]

For clarity, because I had to reread this twice to get the context: VERSION CONTROL repositories should never contain binary objects.

Just want to differentiate that from yum repositories.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/4c9c1b1a-377a-8579-c7d3-a36e4488eebd%40bardicgrove.org.

For more options, visit https://groups.google.com/d/optout.

--

Rob Nelson

[Rob Nelson]

We should note that yumrepo is a native type in puppet that you can use to manage the remote repo information on nodes, and there's a (from memory) palli/createrepo module to create and maintain the yum repo itself. It's not that difficult to add createrepo to a role and set up a node as your internal yum repo if you'd rather start there instead of spacewalk or artifactory and the like. 

--

Rob Nelson

[Dan White]

+1
To manage an RPM not in yum, put it into yum.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/4fc045bb-3e5f-f9d4-88a6-688ca3e3436b%40garretthoneycutt.com.

[Michael Watters]

This would be the ideal but you *can* use the rpm provider when needed.  For example:

package { 'jdk':

    ensure  => installed,

    provider => 'rpm',

    source  => '/pub/oracle/jdk-8-linux-x64.rpm',

}

This will install the rpm using the defined source path.  In our environment the /pub directory is available to all nodes via nfs.

[John Gelnaw]

On Friday, March 3, 2017 at 8:31:00 AM UTC-5, Michael Watters wrote:

This would be the ideal but you *can* use the rpm provider when needed.  For example:

package { 'jdk':

    ensure  => installed,

    provider => 'rpm',

    source  => '/pub/oracle/jdk-8-linux-x64.rpm',

}

This will install the rpm using the defined source path.  In our environment the /pub directory is available to all nodes via nfs

But it won't handle a number of issues in the original post-- RPM 'a' before 'b', and invoke script 'c' once packages a and b are installed.  Of course, you can do that with a number of puppet metaparameters.

I agree with everyone else that the "Right Way" is to create a yum repo, and install the packages via the package provider.  You can use notify / subscribe to control sequencing, or use the chaining arrows.

However, a long time ago, I solved this problem the wrong way-- and if you're going to do it the wrong way, you should at least do it right.  ;)

My original use was installing 2 .rpm or .deb files (depending on OS), without storing binaries in my git repo (the amount of pain it took to excise the vmware-tools installer out of my git repo was... enlightening.  If wisdom comes from making mistakes, I could compete with Solomon).  I had a second use case of a set of lab workstations installing an arbitrary number of RPM files containing various custom software applications, and that turned out to be something I'm still using this method for today.

I created a second "mount point" in puppet via fileserver.conf, pointing to a location outside the puppet git tree, so I could use "puppet:///downloads/<directory>" as a source, synchronized that directory to a local directory on the workstation using "ensure => directory", "recurse => true", and "purge => true", then had that resource notify an exec of "yum install -y *rpm"-- because yum will automatically handle dependencies, sequence, and upgrades.

Then any time you drop a new rpm into the directory on the puppet server, the client automatically downloads and installs and/or upgrades the packages.

Not the "Right Way", no, but it does work, and takes less work to add new packages (my way, drop new RPM in location accessible only by puppet-- yum/package way, add package to http:// accessible repo, update repo metadata, and (if needed) add package to node catalog if it's a new package vs. an upgraded existing package).

[jcbollinger]

On Sunday, March 5, 2017 at 6:56:11 PM UTC-6, John Gelnaw wrote:

I created a second "mount point" in puppet via fileserver.conf, pointing to a location outside the puppet git tree, so I could use "puppet:///downloads/<directory>" as a source, synchronized that directory to a local directory on the workstation using "ensure => directory", "recurse => true", and "purge => true", then had that resource notify an exec of "yum install -y *rpm"-- because yum will automatically handle dependencies, sequence, and upgrades.

Then any time you drop a new rpm into the directory on the puppet server, the client automatically downloads and installs and/or upgrades the packages.

Not the "Right Way", no, but it does work, and takes less work to add new packages (my way, drop new RPM in location accessible only by puppet-- yum/package way, add package to http:// accessible repo, update repo metadata, and (if needed) add package to node catalog if it's a new package vs. an upgraded existing package).

That's a viable option.  I'd say that its main advantage is avoiding any need to update your manifest set or Hiera data when you want to add a new package to the group.  Even that isn't a big win, however: in my environment, it's a one-liner to add a package to my big list of local packages to manage.  Nevertheless, it is one fewer piece to get aligned in the right direction, and there is some value in that.

All the costs of such an approach should be taken into account, however.  One of the more obvious ones is that the full set of RPMs will be maintained locally on each client machine. This could be mitigated by putting the packages on a network drive, so as to avoid any downloading at all, but that does come with its own trade offs.

Another cost is somewhat increased security exposure. If someone can obtain sufficient privilege to put their own RPM into the directory containing the local RPM copies, then Puppet will install it along with all the others.  That constitutes a privilege escalation attack if it takes more privilege to install packages than it does to drop files in the target directory.

Additionally, this approach does not afford an easy way to remove packages, though it may be that package removal is rarely needed.


John

[John Gelnaw]

Except the directory gets forcibly sync'd with the puppet server before any RPM's are processed.  Any locally added files not present on the server will be zapped.

 

Additionally, this approach does not afford an easy way to remove packages, though it may be that package removal is rarely needed.

Same approach as yum/package-- "package { 'foo' : ensure => absent }", although one needs to remember to remove it from the server side first.  ;)

[warron.french]

Hi James, I am working in an environment that uses RH Satellite; based on everyone's input though I will get the RPMs added into a 3rd party repository.

Thanks for the inputs everyone.  I have deployed the module already, but I will redesign it with this separate report as suggested.

\\Warron French from mobile

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAOsPUcgdRkna9zWoO_J_oQHKuVwcGqnCvWLu-jraqW60OG-ZmQ%40mail.gmail.com.