Autoscaling with Puppet

[de]

I have a specific need to use Puppet to Autoscale a few applications I have deployed on EC2. I'm using Puppet now to manage and create instances and saw a neat video on this subject. I'm missing the a piece of the puzzle. I'm assuming I should use Cloudwatch to send my puppetmaster a notification to then create a new instance with the cloudformation module. 

Does that sound about right? I looked into cloudformation, but that doesn't seem right.

Can someone point me in the right direction.

Thanks.

[Tim Dunphy]

I use scalr for this purpose..


http://scalr.net/

I think they did away with the free version of it recently. But if you
ask me what they're asking is well worth it! Autoscales up and down
based on server load. Works with both AWS and rackspace cloud.

-tim

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/ps2yIUWdTMQJ.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.



--
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

[Lorenzo Salvadorini]

http://scalr.net/

I think they did away with the free version of it recently. But if you
ask me what they're asking is well worth it! Autoscales up and down
based on server load. Works with both AWS and rackspace cloud.

-tim

On Sun, May 13, 2012 at 9:28 PM, de <dust...@gmail.com> wrote:
> I have a specific need to use Puppet to Autoscale a few applications I have
> deployed on EC2. I'm using Puppet now to manage and create instances and saw
> a neat video on this subject. I'm missing the a piece of the puzzle. I'm

> assuming I should use CloudWatch to send my puppetmaster a notification to

> then create a new instance with the cloudformation module.

To create autoscaling webfarms I use Scalr too. As Tim reported is a powerful cloud orchestration tool, available as an online service or as an opensource software (PHP/Python/JS) that you can install on your servers. 

I must confirm that there is no more free trial for the service and the pricing is based on the number of hosts you manage.

 > Does that sound about right? I looked into cloudformation, but that doesn't

> seem right.

I haven't tried it but I suppose that Cloudformation has some orchestration feature too: I think you can define an AutoScalingGroup  and have CloudFormation starts a new host based on a predefined AMI when loadaverage of the hosts in the group reach some predefined level; you can choose a different metric if you prefer, like a webpage response time. Then I suppose you have to update your ELB configuration and probably something else ... All these tasks can be configured with a tool like Scalr through a web interface and this simplifies your life, but it's something to pay for, while cloudformation is a free feature on AWS, for what I remember.

Hope this helps,

L.

[Brian Gupta]

Can you provide a link to the video? I'm guessing the workflow could be much simpler, in that you could have an autoscaling group, that spins up instances that are based on AMIs that are either preconfigured for puppet, or have the cloud-init info passed to them to get puppet up and running.

-Brian

On Sun, May 13, 2012 at 9:28 PM, de <dust...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/ps2yIUWdTMQJ.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--

[de]

Thanks everyone. Here's the video I mentioned if you're interested. He makes it look really simple.

http://vimeo.com/27335693

To clarify, how can I tell Puppet(master) about this new instance in my autoscale group when it becomes available? I'd like to automate running puppet master to configure new nodes. 

Thanks.

-Brian

To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--

[de]

Scalr looks okay. I'll look into this. What I really meant was how does Puppet(master) detect a new instance is available in your autoscale group and deploy your modules. I guess that has more to do with notifications than autoscale, but thought I'd clarify. 

Thanks.

[Steven Nemetz]

Generally puppet doesn't "detect" new systems as such. The new systems tell the puppet server about themselves.

The way this is handled is your base instance install would include puppet. So, as the system boots up puppet would be running. The puppet agent will contact the master. The master would have autosigning setup, so it will automatically exchange certs with the new system. Then the behavior would continue as defined in the puppet configuration.

Steven

---

Date: Mon, 14 May 2012 11:30:04 -0700
From: dust...@gmail.com
To: puppet...@googlegroups.com
Subject: Re: [Puppet Users] Autoscaling with Puppet

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/U41DUaMkB3sJ.

[de]

This might be what I was looking for. Havent' yet looked into this.

https://github.com/ccaum/puppet-autoami

[Jeff McCune]

On Mon, Jun 4, 2012 at 1:01 PM, Matthew <mat...@sympoz.com> wrote:
> I'm in the same boat here. I have a puppet master server that provides
> config to a few different classes of machines. I'm struggling with a way to
> have instances from one autoscale group come up, and attach to a
> "webgroup-a" or "webgroup-b" puppet configuration.
>
> Is there a way for a puppet client to pass along some bit of information to
> the master, other than a hostname, to identify itself as having booted into
> a particular class, and thus to receive particular configuration?

This sounds like a custom fact.

If you're doing this from a script a quick and dirty (but not ideal
long term since it's not clear where this fact came from) is to simply
export an environment variable on the agent:

export FACTER_FACT_AUTOSCALE_GROUP=webgroup-b
puppet agent --test

When the puppet agent connects to the master there will be a fact
named fact_autoscale_group with the value of "webgroup-b".

I've used this before to signal to the master how the agent should
configure a new Puppet master worker behind a load balancer.

-Jeff

[Matthew Boeckman]

Thanks, that's very helpful. 

What are other users doing with autoscale to define their nodes.pp? A wildcard match? Basing replies off a custom fact like this? I'm already looking at using a non-fqdn based certname via user-data... but that general part in autoscale is a bit obscured for me. 

Or, phrased differently, if Amazon is init'ing the instance creation request from one of my AMI's, how do I get a config in place on the puppet side to match the host. I'm thinking that's certname and a custom fact, but am open to other ideas.

-Matthew

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--

Matthew Boeckman Director - Technical Operations

p: 303.997.0345 x832 mat...@sympoz.com

www.sympoz.com | www.craftsy.com

Sympoz, Inc. 2150 W. 29th Ave., Suite 400 Denver‚ CO 80211 Toll Free: (888) 979.6769

[Lorenzo Salvadorini]

> What are other users doing with autoscale to define their nodes.pp? A wildcard match? Basing replies off a custom
> fact like this? I'm already looking at using a non-fqdn based certname via user-data... but that general part in
> autoscale is a bit obscured for me.

I use the certname (=FQDN) to match the right configuration in
nodes.pp. The right name is set at boot-time by the cloud orchestrator
(Scalr). While Puppet knows only the configuration it gives to each
role (webserver or mysql server or mysql slave...), the orchestrator
knows the exact status of the farm (how many instances of each role).
So if you have a farm composed by two autoscaling "roles", web & db, i
change the hostname from ec2-xx-xx-xx-xx to something like:

- myservice-web01
- myservice-web02                  match myservice-web0[0-9] in site.pp
- myservice-web03

- myservice-db-master             match myservice-db-master in site.pp

- myservice-db-slave01
- myservice-db-slave02            match myservice-db-slave0[0-9] in site.pp
- myservice-db-slave03

A simple advantage of this approach is that I don't need to build, pay
and maintain a customized AMI for each role in each farm, I can simply
start from a fresh OS at boot time.

[Stephen Gran]

Hello,

On Tue, 2012-06-05 at 08:43 -0600, Matthew Boeckman wrote:
> Thanks, that's very helpful.
>
>
> What are other users doing with autoscale to define their nodes.pp? A
> wildcard match? Basing replies off a custom fact like this? I'm
> already looking at using a non-fqdn based certname via user-data...
> but that general part in autoscale is a bit obscured for me.

When you create the autoscaling group, one of the things you do is
create a launch config. In this launch config you can add scripts to
the userdata that will then be run by the node on bootup (via
cloud-init). This is very helpful for injecting custom data into the
node for use when it connects to the puppetmaster.

> Or, phrased differently, if Amazon is init'ing the instance creation
> request from one of my AMI's, how do I get a config in place on the
> puppet side to match the host. I'm thinking that's certname and a
> custom fact, but am open to other ideas.

Cheers,
--
Stephen Gran
Senior Systems Integrator - guardian.co.uk

Please consider the environment before printing this email.
------------------------------------------------------------------
Visit guardian.co.uk - newspaper of the year

www.guardian.co.uk www.observer.co.uk www.guardiannews.com

On your mobile, visit m.guardian.co.uk or download the Guardian
iPhone app www.guardian.co.uk/iphone

To save up to 30% when you subscribe to the Guardian and the Observer
visit www.guardian.co.uk/subscriber
---------------------------------------------------------------------
This e-mail and all attachments are confidential and may also
be privileged. If you are not the named recipient, please notify
the sender and delete the e-mail and all attachments immediately.
Do not disclose the contents to another person. You may not use
the information for any purpose, or store, or copy, it in any way.

Guardian News & Media Limited is not liable for any computer
viruses or other material transmitted with or as part of this
e-mail. You should employ virus checking software.

Guardian News & Media Limited

A member of Guardian Media Group plc
Registered Office
PO Box 68164
Kings Place
90 York Way
London
N1P 2AP

Registered in England Number 908396