Puppet newbie, Apache vHosts, and trying to do it the right way

[Ben Ruset]

Alright, this is definitely a Puppet 101 question, and I am sure that the Puppet docs have an answer for this and I am just dense and can't find it. 

I'm trying to teach myself Puppet, with the goal of managing the VPS I run with it. Additionally, when I get a better grasp on things, I want to deploy it at work. So while my home deployment will be small, I want to learn how to "do it right."

I have a vanilla Puppet server and an agent. Both are managed through Puppet.

Right now, on my agent node, I'm configuring Apache. I've got code like this in my nodes.pp:

###

node 'hostname' {

  include 'ntp'

  include 'sudo'

  class { 'vim':

    opt_misc => ['nu','fo=tcq','nocompatible','modeline','tabstop=2','expandtab','softtabstop=2','shiftwidth=2','backspace=indent,eol,start'],

  }

  class { 'apache':

    serveradmin => 'whatever',

  }

  apache::vhost{ 'www.example.com':

    default_vhost   => true,

    docroot         => '/var/www/html/www.example.com',

    logroot         => '/var/log/httpd/www.example.com',

    access_log_file => 'access_log',

    error_log_file  => 'error_log',

    override        => 'all',

  }

  some more vhosts

  ...

}

###

It seems like best practice is to put all of these configs in some separate directory/file so I could end up with something like:

###

node 'hostname' {

  include 'ntp'

  include 'sudo'

  class { 'vim':

    opt_misc => $std_vim_ops

  }

  include 'vhost1'

  include 'vhost2'

}

###

Now I guess I can make modules for each vhost, but that doesn't seem right. How would I go about doing this? (Bear in mind that my second block of code is just a rough approximation and I don't really know the right way to write out what I want to do.)

Thanks for your patience. I've been searching for answers and not really understanding what I come up with.

[Christopher Wood]

Short version: read up on hiera, data bindings, and create_resources.

Details:

It sounds like you might want to look into hiera (baked into puppet 3 and higher) and the create_resources function. Specifically that you list the variable portions of things in your puppet configuration in hiera, and then use the create_resources function to configure an arbitrary number of those things. That way each vhost is another data entry which you can more easily parse and validate than the equivalent puppet code.

Quick example:

a) your hiera data file

---
apache::vhosts::vhosts:
'www.example.com':
'default_vhost': true
'docroot': /var/www/html/www.example.com
'logroot': /var/log/httpd/www.example.com
'access_log_file': access_log
'error_log_file': error_log
'override': all
'www.otherhost.com':
'default_vhost': false
'docroot': /var/www/html/www.otherhost.com
'logroot': /var/log/httpd/www.otherhost.com
'access_log_file': access_log
'error_log_file': error_log
'override': all

b) a define representing one vhost, note variables named like yaml keys

define apache::onevhost (
$default_vhost,
$docroot,
$logroot,
$access_log_file,
$error_log_file,
$override,
) {
# your resource declarations here
}

c) a class wrapping that define and the create_resources function, also with some chaining/collecting so that the resources are applied in the right order

class apache::vhosts ( $vhosts ) {
include apache
create_resources('apache::onevhost', $vhosts)
Class['Apache'] -> Apache::Onevhost <| |>
}

d) in your node definition (or better, in a profile included in a role) you would include the apache::vhosts class

include ::apache::vhosts


Here's the read-until-eyes-bleed set of links:


http://docs.puppetlabs.com/hiera/1/
https://ask.puppetlabs.com/question/117/how-can-i-use-data-bindings-in-puppet-3/
http://docs.puppetlabs.com/puppet/latest/reference/lang_collectors.html
http://docs.puppetlabs.com/references/latest/function.html#createresources
http://docs.puppetlabs.com/puppet/latest/reference/lang_functions.html
(str2bool from stdlib could be important when declaring true/false in yaml)
http://docs.puppetlabs.com/guides/scope_and_puppet.html
http://www.craigdunn.org/2012/05/239/
http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-1/
http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/
https://ask.puppetlabs.com/question/1655/an-end-to-end-roleprofile-example-using-hiera/


Now all that said, the optimal way to write up your home/work puppet setup is the one that makes your life easier. The above represents my opinion and no more. Your mileage may definitely vary and that's perfectly fine too.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [1]puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> [2]https://groups.google.com/d/msgid/puppet-users/41e9486d-c072-4e14-8ee6-d91804277f74%40googlegroups.com.
> For more options, visit [3]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:puppet-users...@googlegroups.com
> 2. https://groups.google.com/d/msgid/puppet-users/41e9486d-c072-4e14-8ee6-d91804277f74%40googlegroups.com?utm_medium=email&utm_source=footer
> 3. https://groups.google.com/d/optout

[Ben Ruset]

Thanks for the answer. I grasp the fundamentals of what you're saying, but right now I'm a little leery of delving into Hiera when I don't really know the fundamentals of Puppet itself yet. My reading through those docs would only confuse me more. (Not that those docs are probably not good - it's just that I'm not sure I'm the target audience - yet.)

How would somebody accomplish this before Hiera was around? 

[Christopher Wood]

Before hiera you might have a define specifying what a vhost is, and then have a class with the defines all listed. You would include that class in your node definition, ensuring that this class was applied after the basic apache class. Sometimes people would put the defines right in the node, but putting stuff directly in the node got discouraged over time. It's basically what you did but having the defines in a class rather than in the node.

define thing::one ( $param1, $param2 ) {
# resources here
}

class thing::ones {
thing::one { 'first':
param1 => 'string one',
param2 => 'string two',
}
thing::one { 'second':
param1 => 'string three',
param2 => 'string four',
}
}

node 'hostname' {
include thing
include thing::ones

}




On Sun, Jun 22, 2014 at 06:23:05PM -0700, Ben Ruset wrote:
> Thanks for the answer. I grasp the fundamentals of what you're saying, but
> right now I'm a little leery of delving into Hiera when I don't really
> know the fundamentals of Puppet itself yet. My reading through those docs
> would only confuse me more. (Not that those docs are probably not good -
> it's just that I'm not sure I'm the target audience - yet.)
> How would somebody accomplish this before Hiera was around? 
>
> On Sunday, June 22, 2014 2:45:52 PM UTC-4, Christopher Wood wrote:
>
> Short version: read up on hiera, data bindings, and create_resources.
>
> Details:
>
> It sounds like you might want to look into hiera (baked into puppet 3
> and higher) and the create_resources function. Specifically that you
> list the variable portions of things in your puppet configuration in
> hiera, and then use the create_resources function to configure an
> arbitrary number of those things. That way each vhost is another data
> entry which you can more easily parse and validate than the equivalent
> puppet code.
>
> Quick example:
>
> a) your hiera data file
>
> ---
> apache::vhosts::vhosts:

>   '[1]www.example.com':
>     'default_vhost': true
>     'docroot': /var/www/html/[2]www.example.com
>     'logroot': /var/log/httpd/[3]www.example.com

>     'access_log_file': access_log
>     'error_log_file': error_log
>     'override': all

>   '[4]www.otherhost.com':
>     'default_vhost': false
>     'docroot': /var/www/html/[5]www.otherhost.com
>     'logroot': /var/log/httpd/[6]www.otherhost.com

>     'access_log_file': access_log
>     'error_log_file': error_log
>     'override': all
>
> b) a define representing one vhost, note variables named like yaml keys
>
> define apache::onevhost (
>   $default_vhost,
>   $docroot,
>   $logroot,
>   $access_log_file,
>   $error_log_file,
>   $override,
> ) {
>   # your resource declarations here
> }
>
> c) a class wrapping that define and the create_resources function, also
> with some chaining/collecting so that the resources are applied in the
> right order
>
> class apache::vhosts ( $vhosts ) {
>   include apache
>   create_resources('apache::onevhost', $vhosts)
>   Class['Apache'] -> Apache::Onevhost <| |>
> }
>
> d) in your node definition (or better, in a profile included in a role)
> you would include the apache::vhosts class
>
> include ::apache::vhosts
>
> Here's the read-until-eyes-bleed set of links:
>

> [7]http://docs.puppetlabs.com/hiera/1/
> [8]https://ask.puppetlabs.com/question/117/how-can-i-use-data-bindings-in-puppet-3/
> [9]http://docs.puppetlabs.com/puppet/latest/reference/lang_collectors.html
> [10]http://docs.puppetlabs.com/references/latest/function.html#createresources
> [11]http://docs.puppetlabs.com/puppet/latest/reference/lang_functions.html

>   (str2bool from stdlib could be important when declaring true/false in
> yaml)

> [12]http://docs.puppetlabs.com/guides/scope_and_puppet.html
> [13]http://www.craigdunn.org/2012/05/239/
> [14]http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-1/
> [15]http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/
> [16]https://ask.puppetlabs.com/question/1655/an-end-to-end-roleprofile-example-using-hiera/

>
> Now all that said, the optimal way to write up your home/work puppet
> setup is the one that makes your life easier. The above represents my
> opinion and no more. Your mileage may definitely vary and that's
> perfectly fine too.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to [17]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [18]https://groups.google.com/d/msgid/puppet-users/ca8e184e-e8a7-4b35-b9b6-e43ca8cd899e%40googlegroups.com.
> For more options, visit [19]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. http://www.example.com/
> 2. http://www.example.com/
> 3. http://www.example.com/
> 4. http://www.otherhost.com/
> 5. http://www.otherhost.com/
> 6. http://www.otherhost.com/
> 7. http://docs.puppetlabs.com/hiera/1/
> 8. https://ask.puppetlabs.com/question/117/how-can-i-use-data-bindings-in-puppet-3/
> 9. http://docs.puppetlabs.com/puppet/latest/reference/lang_collectors.html
> 10. http://docs.puppetlabs.com/references/latest/function.html#createresources
> 11. http://docs.puppetlabs.com/puppet/latest/reference/lang_functions.html
> 12. http://docs.puppetlabs.com/guides/scope_and_puppet.html
> 13. http://www.craigdunn.org/2012/05/239/
> 14. http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-1/
> 15. http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/
> 16. https://ask.puppetlabs.com/question/1655/an-end-to-end-roleprofile-example-using-hiera/
> 17. mailto:puppet-users...@googlegroups.com
> 18. https://groups.google.com/d/msgid/puppet-users/ca8e184e-e8a7-4b35-b9b6-e43ca8cd899e%40googlegroups.com?utm_medium=email&utm_source=footer
> 19. https://groups.google.com/d/optout

[Ben Ruset]

Ah, okay this makes more sense for me. So I know that I can stick this in the same file with the node definitions, but there's got to be a better place for it to go. Where would the "proper" place be for it to go?

Many thanks for your responses. They've been very helpful.

On Sun, Jun 22, 2014 at 9:52 PM, Christopher Wood <christop...@pobox.com> wrote:

Before hiera you might have a define specifying what a vhost is, and then have a class with the defines all listed. You would include that class in your node definition, ensuring that this class was applied after the basic apache class. Sometimes people would put the defines right in the node, but putting stuff directly in the node got discouraged over time. It's basically what you did but having the defines in a class rather than in the node.

<snip> 

[Christopher Wood]

Perhaps read about modules? if you specify a class name (apache, apache::vhosts) that class is automatically loaded from the relevant module. Just like putting stuff directly in node definitions has correctly gone out of style, so has defining classes/defines outside of modules.

http://docs.puppetlabs.com/learning/modules1.html

http://docs.puppetlabs.com/puppet/latest/reference/modules_fundamentals.html

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to [2]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [3]https://groups.google.com/d/msgid/puppet-users/CADBQ%3DdaY4NX3c%2ByKX_WzHQX5AfwHVbvs9kpPJwh2-ZMkBu8Rmw%40mail.gmail.com.
> For more options, visit [4]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:christop...@pobox.com
> 2. mailto:puppet-users...@googlegroups.com
> 3. https://groups.google.com/d/msgid/puppet-users/CADBQ%3DdaY4NX3c%2ByKX_WzHQX5AfwHVbvs9kpPJwh2-ZMkBu8Rmw%40mail.gmail.com?utm_medium=email&utm_source=footer
> 4. https://groups.google.com/d/optout

[Ben Ruset]

Well, so I didn't know if it was "good form" to modify modules to hold site-specific configuration items.

I think part of my frustration is that I haven't been able to find a doc, or maybe one doesn't exist, that shows what a decent "production" Puppet environment might look like. I think it'd be easier for folks like me who are super new to it to be able to look and see how someone might have implemented it. Obviously there's a million ways to accomplish an implementation, but at least it'd be great to see something as a starting point.

[Christopher Wood]

I think the good form these days is to create generic modules and keep the site-specific data in hiera (not that I'm good at keeping up with fashion). Once you've gone through the Learning Puppet series of docs you'll probably have more context to the roles/profiles pattern and other links.

http://docs.puppetlabs.com/learning/index.html
http://docs.puppetlabs.com/learning/introduction.html

As to a document regarding a canonical best practice, there won't be such a thing as long as different environments have such disparate requirements and personalities driving them.

For you, try creating a simple module to do anything, then including that module in a node, and then adjusting class parameters via hiera. That should set you up fairly well for expansion.

On Mon, Jun 23, 2014 at 02:50:27PM -0400, Ben Ruset wrote:
> Well, so I didn't know if it was "good form" to modify modules to hold
> site-specific configuration items.
> I think part of my frustration is that I haven't been able to find a doc,
> or maybe one doesn't exist, that shows what a decent "production" Puppet
> environment might look like. I think it'd be easier for folks like me who
> are super new to it to be able to look and see how someone might have
> implemented it. Obviously there's a million ways to accomplish an
> implementation, but at least it'd be great to see something as a starting
> point.
>
> On Mon, Jun 23, 2014 at 1:53 PM, Christopher Wood
> <[1]christop...@pobox.com> wrote:
>
> Perhaps read about modules? if you specify a class name (apache,
> apache::vhosts) that class is automatically loaded from the relevant
> module. Just like putting stuff directly in node definitions has
> correctly gone out of style, so has defining classes/defines outside of
> modules.
>

> [2]http://docs.puppetlabs.com/learning/modules1.html
>
> [3]http://docs.puppetlabs.com/puppet/latest/reference/modules_fundamentals.html

>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to [4]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [5]https://groups.google.com/d/msgid/puppet-users/CADBQ%3DdYHGBUi5kD1S2LANZ_o0fZxHPKHii4H53VSmpak0_Ur4Q%40mail.gmail.com.
> For more options, visit [6]https://groups.google.com/d/optout.

>
> References
>
> Visible links
> 1. mailto:christop...@pobox.com

> 2. http://docs.puppetlabs.com/learning/modules1.html
> 3. http://docs.puppetlabs.com/puppet/latest/reference/modules_fundamentals.html
> 4. mailto:puppet-users...@googlegroups.com
> 5. https://groups.google.com/d/msgid/puppet-users/CADBQ%3DdYHGBUi5kD1S2LANZ_o0fZxHPKHii4H53VSmpak0_Ur4Q%40mail.gmail.com?utm_medium=email&utm_source=footer
> 6. https://groups.google.com/d/optout

[Ramin K]

On 6/23/2014 4:05 AM, Ben Ruset wrote:
> Ah, okay this makes more sense for me. So I know that I can stick this
> in the same file with the node definitions, but there's got to be a
> better place for it to go. Where would the "proper" place be for it to go?
>
> Many thanks for your responses. They've been very helpful.

Simplest solution might be to organize around role/profiles without
doing all the fancy create_resources Hiera stuff. However I would take
this time to learn Hiera. It's much easier to start with Hiera than to
graph it on later. My 2000+ line commit last summer assures me of this.
Using Hiera helps you separate data from code which makes you write less
code and the code you do right tends to suck less.

That said if you still want to get it up and running without going down
that path it might look a bit like the following.

node site-frontend01 {
include role::frontend
}

node site-middleware01 {
include role::middleware
}

class role::frontend {
include profile::apache
include profile::base
include profile::frontend
include profile::php
}

class role::middleware {
include profile::apache
include profile::base
include profile::java
include profile::middleware
include profile::varnish
}

class profile::frontend {
include ::apache
include ::sitecode

apache::vhost { 'www.example.com':
blah => stuff
}
apache::vhost { 'www.notexample.com':
blah => stuff
}
}

class profile::middleware {
include ::apache
include ::appcode

apache::vhost { 'app.example.com':
blah => stuff,
port => 8080,
}
apache::vhost { 'app.notexample.com':
blah => stuff,
port => 8081,
}
}

[José Luis Ledesma]

Imho trying to learn puppet without hiera nowadays is like trying to learn ski without poles. It is possible, but it is not the best way.

Regards,

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/53A86B04.9070202%40badapple.net.
For more options, visit https://groups.google.com/d/optout.

[Rich Burroughs]

Yeah I think part of the difficulty is that there are multiple ways to do things and not everyone agrees on what is the best way. There are some groups that have put their entire Puppet code base on GitHub. I bet you could find some by Googling or searching on GitHub. That's one of the nice things about Hiera -- you separate any sensitive data from the code, and so it's not a problem to share the code with others if you choose to.

One of the best blogs I have found is Gary Larizza's:

http://garylarizza.com

Gary works at Puppet Labs and has a ton of great information there.

Also the second edition of Pro Puppet has a lot of good information on things like Hiera and writing modules with specific examples. That would be a good place to start, in my opinion.

Rich

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CADBQ%3DdYHGBUi5kD1S2LANZ_o0fZxHPKHii4H53VSmpak0_Ur4Q%40mail.gmail.com.

[Ramin K]

You'll get no arguement from me on those points. However I recall it
taking several months to wrap my head around my first Puppet install
circa 0.24. Perhaps building manifests without Hiera is the "wax the
car, paint the fence" exercise that sets the new user up for their
config mgmt epiphany in the future?

Ramin

On 6/24/2014 12:05 AM, José Luis Ledesma wrote:
> Imho trying to learn puppet without hiera nowadays is like trying to
> learn ski without poles. It is possible, but it is not the best way.
>
> Regards,
>
> El 24/06/2014 02:29, "Ramin K" <ramin...@badapple.net

> <mailto:ramin...@badapple.net>> escribió:

> apache::vhost { 'www.example.com <http://www.example.com>':
> blah => stuff
> }
> apache::vhost { 'www.notexample.com <http://www.notexample.com>':

> blah => stuff
> }
> }
>
> class profile::middleware {
> include ::apache
> include ::appcode
>

> apache::vhost { 'app.example.com <http://app.example.com>':

> blah => stuff,
> port => 8080,
> }

> apache::vhost { 'app.notexample.com <http://app.notexample.com>':

[Ben Ruset]

My rationale for it is exactly that. Focus on learning how Puppet itself works, and then learn the add-ons after the fact. On top of that, Hiera isn't a requirement for use - you can just as easily use something like Foreman (which, from what I've seen of it so far looks really good.) But even in my learning environment I'm not using Foreman yet because I want to focus on the absolute basics first.

[Rich Burroughs]

I'd really recommend that second edition for Pro Puppet for you if you are starting and want to learn to Puppet the right way. If I remember right doing Apache vhosts is an example they use in the book even.

The problem is that there are certain types of tasks that it's going to be able to do without some extra source for data, whether it's Hiera, an ENC like Foreman, or custom facts. You are just going to need to get to data that's not going to be available. Yes you can do things like override parameters for parameterized classes in site.pp but you did say you want to learn to do things the right way :)

Also, I mentioned Gary's blog, he had a great post where he talked about some of these same issues, comparing using Hiera to params.pp and roles/profiles:

http://garylarizza.com/blog/2013/12/08/when-to-hiera/

It may be worth reading even if some of it ends up being advanced for where you're at. Or bookmark it for later.

Rich

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CADBQ%3Dda3eUSE8oYYmZPRHzTTux9aH5iCvv%3DRGb2TZHYCKDNTjg%40mail.gmail.com.