Run stages of virtual resources

[David Arroyo]

Our site has several dozen yum repositories. Pushing all yum repositories to all servers isn't practical; it hurts performance, some repositories are OS-specific, and some repositories cause conflicts with each other (we have a ruby187 repo and a ruby 193 repo, for example).

In our current setup, we have one module with all our yumrepos defined virtually:

class yumrepos {
@yumrepo{'puppet':
…
}
@yumrepo{'python26':
…
}
…
}

And our various modules realize those resources as needed:

class puppet(...) {
realize Yumrepo['puppet']
…
}

However, this requires every package definition to require the Yumrepo resource. I can ease the pain with resource defaults, but it doesn't go away completely. I have found on puppet 2.7 that virtual resources are evaluated in the run stage they are defined in, not the run stage they are realized in, so that I can do in site.pp:

stage{'package-setup': before => Stage['main'] }
class{'yumrepos': stage => 'package-setup' }

Then all yum repositories that a node will use are on the machine before any packages are installed. Is this a kosher use of run stages? Am I going to be surprised by something I didn't consider? I have only tested this behavior in Puppet 2.7 and don't know if it is subject to change in later releases. How do others handle this problem?

-David

[james.e...@fasthosts.com]

How about chaining the resources, ala http://docs.puppetlabs.com/puppet/2.7/reference/lang_relationships.html#chaining-arrows.

Yumrepo <| |> -> Package<| |>

This declared in site.pp should apply globally to all nodes and would avoid the use of run stages (if I understand it correctly).

J

[jcbollinger]

On Tuesday, December 31, 2013 10:47:33 AM UTC-6, David Arroyo wrote:

Our site has several dozen yum repositories. Pushing all yum repositories to all servers isn't practical; it hurts performance, some repositories are OS-specific, and some repositories cause conflicts with each other (we have a ruby187 repo and a ruby 193 repo, for example).

In our current setup, we have one module with all our yumrepos defined virtually:

        class yumrepos {
          @yumrepo{'puppet':
            …
          }
          @yumrepo{'python26':
            …
          }
          …
        }

And our various modules realize those resources as needed:

        class puppet(...) {
          realize Yumrepo['puppet']
          …
        }

However, this requires every package definition to require the Yumrepo resource. I can ease the pain with resource defaults, but it doesn't go away completely. I have found on puppet 2.7 that virtual resources are evaluated in the run stage they are defined in, not the run stage they are realized in, so that I can do in site.pp:

        stage{'package-setup': before => Stage['main'] }
        class{'yumrepos': stage => 'package-setup' }

Then all yum repositories that a node will use are on the machine before any packages are installed. Is this a kosher use of run stages?

It is exactly the sort of thing that run stages are intended for.

 

Am I going to be surprised by something I didn't consider? I have only tested this behavior in Puppet 2.7 and don't know if it is subject to change in later releases. How do others handle this problem?

Run stages remain available in Puppet 3, and I see no reason to fear that they will go away in the foreseeable future.  As for surprises, however, you are wise to be wary.  Run stages are a pretty heavy tool, and they can indeed cause trouble if used carelessly or excessively.  The most common form of trouble they are involved in is resource dependency cycles, and that might happen even in your limited use if you have repositories that are managed via packages.  With care and restraint, however, it should be possible to use run stages as you suggest.


John

[jcbollinger]

On Thursday, January 2, 2014 2:18:06 AM UTC-6, james.e...@fasthosts.com wrote:

How about chaining the resources, ala http://docs.puppetlabs.com/puppet/2.7/reference/lang_relationships.html#chaining-arrows.

Yumrepo <| |> -> Package<| |>

This declared in site.pp should apply globally to all nodes and would avoid the use of run stages (if I understand it correctly).

It would indeed avoid the stated need for run stages.  It would also, however, realize every virtual Yumrepo and every virtual Package declared for each node, which is contrary to David's intent and purpose.

It might be possible to declare tags on repos, and on packages if necessary, and to use them to filter the repositories to be realized and chained.  However, if the use of stages is restricted to applying Yumrepos first then I think that would be clean and reasonably safe.  I say this as a person who is not particularly enamored of run stages in general.


John

[Joseph Swick]

On 01/02/2014 03:18 AM, james.e...@fasthosts.com wrote:
> How about chaining the resources, ala
> http://docs.puppetlabs.com/puppet/2.7/reference/lang_relationships.html#chaining-arrows
> .
>
> Yumrepo <| |> -> Package<| |>
>
> This declared in site.pp should apply globally to all nodes and would avoid
> the use of run stages (if I understand it correctly).
>
> J
>

One caveat to this is that if you are defining any packages (or the
YumRepos) virtually and then adding them to modules with 'realize', the
resource collector will realize all of the virtual packages, regardless
whether if you're realizing them in a module or not. This behavior is
documented in the resource chaining documentation.

I ran into this personally with a couple of custom modules that we use
virtual packages with so that we don't get duplicate resource errors
when managing various packages. My case was very similar, I had a
custom Yum repo I wanted to ensure that was put in place before puppet
tried to install a package out of it, so I had defined the chaining
within the module for the yum repo of:

Yumrepo['CustomRepo'] -> Package <| name == 'CustomPackage' |>

However, when I moved the custom package into our virtual package
resources, that package started getting realized on machines that didn't
need it, but I had forgotten that I had done the above resource
chaining. Fortunately, we did provide a way to require repos with our
virtual package definitions, so I was able to remove the resource
chaining and still have the desired result.

--
Joseph Swick <joseph...@meltwater.com>
Operations Engineer
Meltwater Group

[james.e...@fasthosts.com]

Yeah I didn't spot this in the docs before I posted.

Thanks for correcting me :)

J