Seeking some Puppet advice for a newbie (specifically Virtualmin/CSF related)

[Laurence Cope]

HI

Forgive me if this is such a newbie issue. I am learning Puppet to manage several servers we have. The tutorials on manifests etc. seem a bit simplistic for I want, if what I want can be achieved with Puppet. 

Our servers are web hosting servers. We use Virtualmin (and Webmin) to manage the server and virtual servers. installing Virtualmin takes care of installing all the software required for virtual servers and websites, like Apache MySQL, PHP, BIND, Postfix etc. So we dont install those separately. Virtualmin does it. 

To setup a server we create the VPS, install Virtualmin, configure it, install CSF (Firewall), configure it and then our monitoring system agent. This is the minimum. We may want to then install something like Varnish perhaps, or anything else. At the moment I copy another server and delete its virtual servers as I have everything configured. But in time the configs just move away from each other. So I also want a way to manage all the configs on all servers, so they are the same. CSF in particular. 

I wondered if all this can be done in Puppet. I have no idea if it can because I cannot find any examples of this. 

I wondered if this is very possible with Puppet so I can continue to learn, OR if not then I will give up now as I already spent a while on this. 

Thanks

[Jakov Sosic]

On 11/19/2012 08:57 PM, Laurence Cope wrote:

> I wondered if this is very possible with Puppet so I can continue to
> learn, OR if not then I will give up now as I already spent a while on
> this.

Yes, you can manage firewall and everything else with puppet. Just write
your own modules or use existing ones and you'll be fine. Prepare to
learn and to be slow in the beginning with deploying servers - like 2-3
times slower than installing all manually. But it will pay of in the
end, you'll see.

[Laurence Cope]

Thanks. But my main stumbling block is installing Virtualmin. I am not sure it's a "package". How do I know? What defines a package?

I could use exec to install it I think maybe. But what about the questions it asks during the manual install? Can Puppet answer manual questions asked during install?

Thanks

[Laurence Cope]

Whe I ask what is a package, I mean is a package installed using yum in a repo. And if so, is it any repo installed on the server, or the default ones or what? For packages not in the default repo but somewhere else I guess I add that repo using puppet first then install the package?

I don't know if Virtualmin is available as a package hence asking about exec.

[Ilya Dmitrichenko]

Hi,

Yes, you probably want to do something like this:

class virtualmin::install {

  $version = '1.2.3-foo'

  exec { 'get installer script':

   command => 'wget http://software.virtualmin.com/gpl/scripts/install.sh',

   cwd => '/tmp',

   creates => '/tmp/install.sh',

  }

  file { '/tmp/install.sh':

    mode => 0755,

    # checksum => 'the_checksum_of_that_script_if_you_feed_paranoid'

  }

  exec { 'install it now':

    command => '/tmp/install.sh',

    unless => 'test -x /usr/local/virtualmin/bin/foo && /usr/local/virtualmin/bin/foo --version | grep $version',

    require => File['/tmp/install.sh'],

  }

}

hope this kind of makes sense and please excuse any errors this pseudocode might give you :)

Cheers,

-- 

Ilya 

[Laurence Cope]

Thanks Ilya, I will give this a try. 

BUT if I remember correctly, the install process will ask input during the install, to set some options... can Puppet handle entering options during installations? 

Thanks

[Jakov Sosic]

On 11/20/2012 10:44 AM, Laurence Cope wrote:
> Thanks Ilya, I will give this a try.
>
> BUT if I remember correctly, the install process will ask input during
> the install, to set some options... can Puppet handle entering options
> during installations?

Puppet is not a shell script, it's a state-ensuring tool. So with puppet
you are forcing your system to be in the state you want it to be. So,
you should avoid running 'installation scripts'.

Best way to solve your problem would be to distribute virtualmin through
distro native packages, and later just modify configuration files
according to your needs.

[Laurence Cope]

So does that mean I still have to manually setup server and install what I need, then just use Puppet to keep them all configured the way I want them? 

So I would still copy my virtual servers to create a new one (because it takes too long to setup a virtual server from scratch and install everything I need). 

I was hoping to find some sort of "server setup" automation tool so I can create a new CentOS VM for example, then just have something install all the software I need automatically. 

Thanks

[Matthew Burgess]

On Tue, Nov 20, 2012 at 10:40 AM, Laurence Cope
<amityweb...@gmail.com> wrote:
> So does that mean I still have to manually setup server and install what I
> need, then just use Puppet to keep them all configured the way I want them?
>
> So I would still copy my virtual servers to create a new one (because it
> takes too long to setup a virtual server from scratch and install everything
> I need).
>
> I was hoping to find some sort of "server setup" automation tool so I can
> create a new CentOS VM for example, then just have something install all the
> software I need automatically.

Puppet can certainly do the 'install all the software I need
automatically' bit for you, but its and your job is made much easier
by asking/telling it to install software in your OSes 'native'
packaging format. As you mention CentOS, that would be RPM. RPMs for
Virtualmin can be found at http://www.webmin.com/vdownload.html. In
order to have Puppet install that for you, you will need to create
your own Yum repository, have Puppet configure yum to make use of that
repo, then create a manifest that installs the package.

Hope this helps,

Matt.

>
> Thanks
>
>
> On Monday, November 19, 2012 7:57:33 PM UTC, Laurence Cope wrote:
>>
>> HI
>>
>> Forgive me if this is such a newbie issue. I am learning Puppet to manage
>> several servers we have. The tutorials on manifests etc. seem a bit
>> simplistic for I want, if what I want can be achieved with Puppet.
>>
>> Our servers are web hosting servers. We use Virtualmin (and Webmin) to
>> manage the server and virtual servers. installing Virtualmin takes care of
>> installing all the software required for virtual servers and websites, like
>> Apache MySQL, PHP, BIND, Postfix etc. So we dont install those separately.
>> Virtualmin does it.
>>
>> To setup a server we create the VPS, install Virtualmin, configure it,
>> install CSF (Firewall), configure it and then our monitoring system agent.
>> This is the minimum. We may want to then install something like Varnish
>> perhaps, or anything else. At the moment I copy another server and delete
>> its virtual servers as I have everything configured. But in time the configs
>> just move away from each other. So I also want a way to manage all the
>> configs on all servers, so they are the same. CSF in particular.
>>
>> I wondered if all this can be done in Puppet. I have no idea if it can
>> because I cannot find any examples of this.
>>
>> I wondered if this is very possible with Puppet so I can continue to
>> learn, OR if not then I will give up now as I already spent a while on this.
>>
>> Thanks
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/fsdRJsMHrT8J.
>
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.

[Laurence Cope]

you will need to create your own Yum repository, have Puppet configure yum to make use of that repo, then create a manifest that installs the package.

Ah right... this bit helps a lot. never thought of creating an own repo, that makes sense now. so if its in a repo puppet can do it. I will look into that, and also request Virtualmin do it because I asked them about this on their forum, but they had no experience with Puppet. Makes sense for it to come from a repo they manage. 

thanks

[Matthew Burgess]

On Tue, Nov 20, 2012 at 1:54 PM, Laurence Cope
<amityweb...@gmail.com> wrote:
>> you will need to create your own Yum repository, have Puppet configure yum
>> to make use of that repo, then create a manifest that installs the package.
>
> Ah right... this bit helps a lot. never thought of creating an own repo,
> that makes sense now. so if its in a repo puppet can do it. I will look into
> that, and also request Virtualmin do it because I asked them about this on
> their forum, but they had no experience with Puppet. Makes sense for it to
> come from a repo they manage.

I generally favour my own private yum repositories rather than
upstream repositories for the following reasons:

1) Most client environments I've worked in have no Internet access; or
if they do it'll just be 1 or 2 servers, of which neither will be my
Puppet/Yum server.
2) I can control what versions of which packages are installed when
the Puppet manifest states 'ensure=>latest'. With a public
repository, I'd be at the mercy of the upstream vendor; as soon as
they release a new package all of my systems would be upgraded with no
testing/staging possible (there was a fairly recent thread in this
group when PuppetLabs released puppet-3.0 into the same repository as
puppet-2.x. It caught a lot of folks out). This can be mitigated, of
course, by using 'ensure=>installed', but then that means upgrades are
painful.
3) It's quicker; your internal network should be much faster and more
reliable than going out to the wider Internet.

Thanks,

Matt.

[Tim Mooney]

In regard to: Re: [Puppet Users] Re: Seeking some Puppet advice for a...:

> On Tue, Nov 20, 2012 at 1:54 PM, Laurence Cope
> <amityweb...@gmail.com> wrote:
>>> you will need to create your own Yum repository, have Puppet configure yum
>>> to make use of that repo, then create a manifest that installs the package.
>>
>> Ah right... this bit helps a lot. never thought of creating an own repo,
>> that makes sense now. so if its in a repo puppet can do it. I will look into
>> that, and also request Virtualmin do it because I asked them about this on
>> their forum, but they had no experience with Puppet. Makes sense for it to
>> come from a repo they manage.
>
> I generally favour my own private yum repositories rather than
> upstream repositories for the following reasons:

+1, for all the reasons Matt mentioned.

Making your own repo may seem daunting, but it's not bad at all. If you
have an internal server that's running http and has enough disk space to
store your RPMs, you're already most of the way there.

Tim
--
Tim Mooney Tim.M...@ndsu.edu
Enterprise Computing & Infrastructure 701-231-1076 (Voice)
Room 242-J6, IACC Building 701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164

[Martijn]

Puppet can't provide input when a script prompts for it, but usually there are alternative ways to install the software. Most install scripts only untar the software to a directory, ask some questions and write some configuration files based on your answers. Most popular software has some way to automate installation.

Install the software on a test-VM, and see what the end-result is. Which files are created, which daemons are started, which config files are modified. That's what you need to recreate with Puppet. You don't need to recreate all the steps the installer does. You simply need to achieve the same end-result.

Either: 

1. The install script may allow command-line parameters to specify the various settings. Simply provide those parameters when executing the script via Puppet.
2. If the installation is not too complicated you can skip the installation script and perform all the steps via Puppet, i.e. untar to a directory, drop a config file somewhere in /etc, configure the service to start.
   
3. You can create your own package that performs the steps that the install script would and simply install the package via Puppet.

For simple installations I would choose option 2, especially since you don't have your own package repo yet. For more complex installations, I'd go for 3.

Might be worth making your own repo anyway, since it's quite a good way to install custom software in a structured way. All those tools you drop in /usr/local could easily be packaged, providing you with all the advantages a packaging system has. Also, Puppet knows how to deal with packages, so your Puppet manifests will be simpler as well.

Regards, Martijn Heemels

Op dinsdag 20 november 2012 10:44:01 UTC+1 schreef Laurence Cope het volgende:

[Jakov Sosic]

On 11/20/2012 03:12 PM, Matthew Burgess wrote:
> (there was a fairly recent thread in this
> group when PuppetLabs released puppet-3.0 into the same repository as
> puppet-2.x. It caught a lot of folks out).

I didn't use ensure=>latest and didn't plan to upgrade so soo, but was
also in trouble because of Cobbler using external puppet repo for
deploying puppet onto freshly installed machines... offcourse every one
of them was puppet agent 3.0 :D So I decided it was much easier for me
to upgrade master then to implement other solutions.

> This can be mitigated, of
> course, by using 'ensure=>installed', but then that means upgrades are
> painful.

Why? You can always use cexec or mcollective with some kind of custom
plugin for doing those sort of things.


--
Jakov Sosic
www.srce.unizg.hr

[Matthew Burgess]

I knew I should have qualified that statement: It's painful because
I've not had to do it
yet because I use my own repos and couldn't immediately think of a
non-painful way of upgrading packages :-)

Thanks for the MCollective pointer; it's been on my list of things to
investigate for a while now!

Regards,

Matt.

[Nikola Petrov]

You can actually install in a custom directory and build a package from
that pretty easily. Look at directory provider from
https://github.com/jordansissel/fpm on how to do that. Of course you
will then want to tweak specific configuration options like IP address
through puppet.

Hope that helps :)

Best, Nikola

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/AxVrJPPnfDgJ.