Scaling Puppet with Git

[diez]

Good morning!

I found an interesting article about scaling puppet with git here.

http://bitfieldconsulting.com/scaling-puppet-with-distributed-version-control

Hope it could help you.

Greetings Diez

[John Arundel]

On Mon, Nov 29, 2010 at 8:12 AM, diez <flat...@stresstiming.de> wrote:
> Good morning!
>
> I found an interesting article about scaling puppet with git here.
>
> http://bitfieldconsulting.com/scaling-puppet-with-distributed-version-control

Has anyone actually done this? It's an interesting idea, but I haven't
heard of it being put into practice.

Regards,
John
--
Bitfield Consulting: we make software that makes things work
http://bitfieldconsulting.com/

[Joe Miller]

I tried something similar, but my environment had too many isolated networks separated by different VPNs to implement the automated post-commit hooks in the article.

What I would do instead was:  a) login to each server (eg: with capistrano),  b) run git clone or git pull to checkout the entire puppet repo, c) run puppet locally.

It's actually fairly fast (except for the initial git clone.)  Ultimately, however, I ended up going back to the puppetmaster model and implementing Passenger, which was just as fast if not faster.   Keep in mind this was with about 30-40 machines at 3 locations, and I have no data on how the git method scales.

One thing that I did find out was that the standard sshd config needs to be changed to allow rapid logins by up'ing the MaxStartups value in sshd_config.  Without this change my capistrano job would randomly fail on a few hosts because sshd on the git server was throttling ssh connections.  I think the default is 10 or less.

--
You received this message because you are subscribed to the Google Groups "Agile System Administration" group.
To post to this group, send email to agile-system-...@googlegroups.com.
To unsubscribe from this group, send email to agile-system-admini...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/agile-system-administration?hl=en.

[Ryan Dooley]

So after reading this post, I think it'll be something that I will be trying out at Lookout. I probably won't be able to get around to switching up the infrastructure here until after the new year though.

Cheers,
Ryan

[ranjib dey]

why not chef? chef scales pretty well ....this seems to be overkill..

[Ryan Dooley]

We already have a pretty heavy investment of puppet.  The investment is not so large that it's not impossible to back out of but what we have works for us and we have processes built around puppet.

Cheers,

Ryan

[Lindsay Holmwood]

On Dec 1, 1:33 am, John Arundel <j...@bitfieldconsulting.com> wrote:

> On Mon, Nov 29, 2010 at 8:12 AM, diez <flatl...@stresstiming.de> wrote:
>
> Has anyone actually done this? It's an interesting idea, but I haven't
> heard of it being put into practice.
>

I have, and took it one step further by freezing Puppet into the repo
as a git submodule, so we could A/B test new versions of Puppet across
our infrastructure.

I wrote a wrapper tool around the process that handled checking out
the repo, checking out Puppet as a submodule, setting up the command
line and load path, and running Puppet.

Held it all together with Cap and it worked quite well for managing
~100 machines.

Lindsay

[Ryan Dooley]

So that's pretty sweet. Any chance of a write up or github gist ;-) I'd happyly contribute to that :D

Cheers,
Ryan

[Joe Miller]

Lindsay, I'm also interested in hearing more, if you are willing to share, such as code snippets.

Also, you talk in the past tense.  Did you abandon this approach?  If so, why?

[Lindsay Holmwood]

Unfortunately it's all IP that belongs to a former employer. I can
certainly talk to them about open sourcing it.

If you wanted to re-implement it, it took me about about 4 days build
and deploy it.

Lindsay

[Lindsay Holmwood]

Good news! My former employer has ok'd me to open source it provided I
make it work with Puppet 2.6. :-)

I'll see if I can find some time to hack on it over the next few
days.

Lindsay

[Ryan Dooley]

That's awesome news!

Cheers,
Ryan

[John Arundel]

On Wed, Dec 1, 2010 at 5:56 AM, Ryan Dooley <ryan....@gmail.com> wrote:
> That's awesome news!
>
> Cheers,
> Ryan

Seconded… and when you publish it, I can add a link from the article.

[Lindsay Holmwood]

On 2 December 2010 13:29, John Arundel <jo...@bitfieldconsulting.com> wrote:
>
> Seconded… and when you publish it, I can add a link from the article.
>

Open sourced at:

http://github.com/railsmachine/rump

Slides from a recent presentation I did on Rump at:

http://www.slideshare.net/auxesis/rump-making-puppetmasterless-puppet-meaty

Enjoy!

Lindsay