Using Package and Yumrepo together? Why is yumrepo creating a corrupt .repo file?

[Stefan Lasiewski]

Hello Everyone,

I am running Puppet 3.8 on CentOS 6. I'm trying to install a yum repository from an RPM package, as well as modify the contents of the .repo file after the package is installed.

My example manifest is below. In my `yumrepo {'remi-safe':}` class, I require `require => Package['remi-release'],` so that the Package will be installed before the yumrepo class is called.

# Install the REMI repo for some PHP software
class stefanl::remi (
  $ensure   = 'latest',
  $enabled  = true,
  $priority = '80',
) {

  package { 'remi-release':
    ensure  => $ensure,
  }

  yumrepo { 'remi-safe':
    enabled  => $enabled,
    priority => $priority,
    require  => Package['remi-release'],
  }

}

Instead, what seems to happen is that:

1. `yumrepo {'remi-safe':}` is called first, creates a file named `/etc/yum.repos.d/remi-safe.repo`
2. Then `package { 'remi-release':}` is called. I guess that since `/etc/yum.repos.d/remi-safe.repo` was already created, remi-release doesn't bother to install /etc/yum.repos.d/remi-safe.repo from the RPM, and I end up with a useless .repo file.

[root@webhost yum.repos.d]# cat remi-safe.repo
[remi-safe]
enabled=true
priority=80
[root@webhost yum.repos.d]#

I'm a little confused why this is happening. Can I use `yumrepo` to modify a file installed by `package`?

Thank you,

-= Stefan

[Thomas Müller]

I suspect the yumrepo type removes lines not provided in the puppet manifest. So the rpm remi-release  package adds the *.repo file and yumrepo deletes lines? You might add the other settings to the yumrepo type too.

Why would you wan't to manage the yumrepo file with puppet if its already installed by the remo-release rpm? If the "remi-release" package does not add other things beside the repo file you could just not install the package and use the yumrepo type "standalone".

- Thomas

 

[jcbollinger]

On Thursday, November 12, 2015 at 7:40:30 PM UTC-6, Stefan Lasiewski wrote:

Hello Everyone,

I am running Puppet 3.8 on CentOS 6. I'm trying to install a yum repository from an RPM package, as well as modify the contents of the .repo file after the package is installed.

My example manifest is below. In my `yumrepo {'remi-safe':}` class, I require `require => Package['remi-release'],` so that the Package will be installed before the yumrepo class is called.

# Install the REMI repo for some PHP software
class stefanl::remi (
  $ensure   = 'latest',
  $enabled  = true,
  $priority = '80',
) {

  package { 'remi-release':
    ensure  => $ensure,
  }

  yumrepo { 'remi-safe':
    enabled  => $enabled,
    priority => $priority,
    require  => Package['remi-release'],
  }

}

Instead, what seems to happen is that:

1. `yumrepo {'remi-safe':}` is called first, creates a file named `/etc/yum.repos.d/remi-safe.repo`
2. Then `package { 'remi-release':}` is called. I guess that since `/etc/yum.repos.d/remi-safe.repo` was already created, remi-release doesn't bother to install /etc/yum.repos.d/remi-safe.repo from the RPM, and I end up with a useless .repo file.

That chain of events seems unlikely.  In the first place, I'm not prepared to believe that when applying a catalog corresponding to the declarations you presented, the agent fails to successfully apply Package['remi-release'] before it applies Yumrepo['remi-safe'].  If you truly observe such behavior -- via the agent's log output, for example -- then it follows that the catalog applied was not built based on those declarations.

In the second place, it would be surprising, but not impossible, for RPM installation to behave as you suggest.  Ordinarily, one would expect one or more .repo files to be packaged as files in the RPM, in which case successful package installation will definitely result in those files being installed on the system.

Under some circumstances, however, the RPM's version of the files might be installed with an additional .rpmnew extension to avoid replacing pre-existing versions of the same file.  If you happen to also be doing something like purging unmanaged files from /etc/yum.repos.d then perhaps you would overlook that.  Alternatively, if the latest available version of package remi-release were already installed on the system, then the Package resource would have no effect, even if files belonging to the package no longer match their packaged versions.

Also, use of files in /etc/yum.repos.d is only one alternative for installing a repo.  Repo descriptions can also be installed directly in /etc/yum.conf, though that somewhat problematic approach is not commonly used.  More commonly, multiple repositories -- normally related to each other -- can be recorded in the same file in /etc/yum.repos.d.
 

[root@webhost yum.repos.d]# cat remi-safe.repo
[remi-safe]
enabled=true
priority=80
[root@webhost yum.repos.d]#

I'm a little confused why this is happening. Can I use `yumrepo` to modify a file installed by `package`?

Yes, you can.  I do so, albeit not with the same version of Puppet.

Your confusion arises first and foremost because what you have concluded is happening is surely not what's actually happening.  Things to check:

• Is the class you presented being applied to the affected node?  Successfully?
  
• Is the RPM in question indeed installed?  If so, then
• Does it pass verification (rpm --verify remi-release)?
• Is /etc/yum.repos.d/remi-safe.repo in its file list (rpm -ql remi-release)?
• If you start with the package not installed and no relevant repo declaration in place, and from that state install the package manually (yum install remi-release), what changes result?
• Are there any in-scope resource defaults for Yumrepo resources?
  

It is conceivable that your Yumrepo resource is clobbering unmanaged properties of the managed repository.  That would constitute a bug and a regression, and you can test for it by installing the release package manually, verifying it good, and then applying the specified class to the node and observing damage to the repo definition.  There are any number of other possibilities, but I decline to speculate further.


John

[Stefan Lasiewski]

On Fri, Nov 13, 2015 at 1:45 AM, Thomas Müller <tho...@chaschperli.ch> wrote:

I suspect the yumrepo type removes lines not provided in the puppet manifest.

I also use this same formula for several other Yum repos, such as mysql-community, EPEL, Software Collections and Repoforge. Yumrepo doesn't remove lines in these other cases, so I'm pretty sure it isn't removing lines for these Remi repositories either.

 

Why would you wan't to manage the yumrepo file with puppet if its already installed by the remo-release rpm?

The 'remi-release' RPM provides most of what I need, such as the base .repo files, the yum keys, etc. However, I also want to tweak a few settings-- I'd like to enable and disable certain repos, and set their Priority for yum-priorities. This is a pretty common thing to do with Yum.

You might wonder why do I not use `yumrepo` exclusively? I prefer to follow a principle of least change for my systems. Since the RPM does most of the automation for me, I'd prefer to use that and use `yumrepo` to apply the smallest number of changes that I need. If the upstream vendor makes changes to their repositories, a new RPM will apply those changes. I've been bitten by a few Puppet Forge modules where the `yumrepo` definitions didn't stay up to date with the upstream vendor's changes.

Thanks,

-= Stefan

--

Stefan Lasiewski                         Email: ste...@nersc.gov
Computer System Engineer III    Email: slasi...@lbl.gov
Networking, Security, and Servers Group

National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory

[Stefan Lasiewski]

On Fri, Nov 13, 2015 at 6:36 AM, jcbollinger <John.Bo...@stjude.org> wrote:

That chain of events seems unlikely.  In the first place, I'm not prepared to believe that when applying a catalog corresponding to the declarations you presented, the agent fails to successfully apply Package['remi-release'] before it applies Yumrepo['remi-safe'].  If you truly observe such behavior -- via the agent's log output, for example -- then it follows that the catalog applied was not built based on those declarations.

Thanks for that sanity check. Now that I read the logs more closely, I was mistaken.

 

Under some circumstances, however, the RPM's version of the files might be installed with an additional .rpmnew extension to avoid replacing pre-existing versions of the same file.

That is my understanding as well. However, these .rpmnew files aren't being created either, which is strange. If I remove the Yumrepo resource from my manifest, the .rpmnew files will be created if I create a remo-safe.repo file by hand.

It is conceivable that your Yumrepo resource is clobbering unmanaged properties of the managed repository.  That would constitute a bug and a regression, and you can test for it by installing the release package manually, verifying it good, and then applying the specified class to the node and observing damage to the repo definition.  There are any number of other possibilities, but I decline to speculate further.

I think I ran into a longstanding bug with the Yumrepo resource type:

* https://tickets.puppetlabs.com/browse/PUP-723 "Due to prefetching, Yumrepo clobbers any definition that it does not create"

* Historical data is in https://projects.puppetlabs.com/issues/1238

At the head of my log output I see that Puppet is prefetching some Yum resources:

Info: Applying configuration version '1447376917'

Debug: Prefetching yum resources for package

...

... and further down, it executes the Yumrepo type with cached content:

Debug: Executing '/usr/bin/yum -d 0 -e 0 -y list remi-release'

Debug: Package[remi-release](provider=yum): Ensuring => latest

Debug: Executing '/usr/bin/yum -d 0 -e 0 -y install remi-release'

Notice: /Stage[main]/stefanl_yum::Remi/Package[remi-release]/ensure: created

Debug: /Stage[main]/stefanl_yum::Remi/Package[remi-release]: The container Class[stefanl_yum::Remi] will propagate my refresh event

Notice: /Stage[main]/stefanl_yum::Remi/Yumrepo[remi-safe]/ensure: created

Debug: /Stage[main]/stefanl_yum::Remi/Yumrepo[remi-safe]: The container Class[stefanl_yum::Remi] will propagate my refresh event

From reading the bug, it looks like what is actually happening here might be:

1. yumrepo prefetches a yum resource. At this point, the /etc/yum.repos.d/remi*.repo files don't exist, so yumrepo assumes that the files won't exist later.

2. Later, Package[remi-release] is created

3. Then, Yumrepo[remi-safe] is created. But Yumrepo doesn't check for the existence of the remi-safe.repo file first. Instead of adding the parameters to the existing file, it replaces the whole file with the cached content from #1, which is surprising.

So, my manifest *does* install things in the order I was expecting. Yumrepo also adds an unexpected surprise at the end.

That's my guess anyways. Thanks for the tips John.

-= Stefan

[Stefan Lasiewski]

Unfortunately, I do not have a workaround, short of "Don't use the Yumrepo and Package types together".

Does anyone else have a better way to workaround this bug?

-= Stefan

On Friday, November 13, 2015 at 9:58:26 AM UTC-8, Stefan Lasiewski wrote:

[Stefan Lasiewski]

I'm trying Augeas as a workaround. This seems to work as expected-- the Package is installed first, and the file is modified second.

class stefanl::remi (
                                                               
  $ensure   = 'latest',                                                                    
  $enabled  = true,                                                                        
  $priority = '80',                                                                      

) {                                                                                        
  package { 'remi-release':                                                                
    ensure  => $ensure,                                                                    
  }

  augeas { 'remi-safe':                                                                    
    context => '/files/etc/yum.repos.d/remi-safe.repo/remi-safe',                          
    changes => [                                                                            
      "set enable   ${enabled}",                                                            
      "set priority ${priority}",                                                          
    ],                                                                                      
    require => Package['remi-release'],                                                    
  }                                                                                        

}

-= Stefan