Scaling Puppet with Git

346 views
Skip to first unread message

diez

unread,
Nov 29, 2010, 3:12:40 AM11/29/10
to agile-system-...@googlegroups.com
Good morning!

I found an interesting article about scaling puppet with git here.

http://bitfieldconsulting.com/scaling-puppet-with-distributed-version-control

Hope it could help you.

Greetings Diez

John Arundel

unread,
Nov 30, 2010, 9:33:22 AM11/30/10
to agile-system-...@googlegroups.com
On Mon, Nov 29, 2010 at 8:12 AM, diez <flat...@stresstiming.de> wrote:
> Good morning!
>
> I found an interesting article about scaling puppet with git here.
>
> http://bitfieldconsulting.com/scaling-puppet-with-distributed-version-control

Has anyone actually done this? It's an interesting idea, but I haven't
heard of it being put into practice.

Regards,
John
--
Bitfield Consulting: we make software that makes things work
http://bitfieldconsulting.com/

Joe Miller

unread,
Nov 30, 2010, 10:00:45 AM11/30/10
to agile-system-...@googlegroups.com
I tried something similar, but my environment had too many isolated networks separated by different VPNs to implement the automated post-commit hooks in the article.

What I would do instead was:  a) login to each server (eg: with capistrano),  b) run git clone or git pull to checkout the entire puppet repo, c) run puppet locally.

It's actually fairly fast (except for the initial git clone.)  Ultimately, however, I ended up going back to the puppetmaster model and implementing Passenger, which was just as fast if not faster.   Keep in mind this was with about 30-40 machines at 3 locations, and I have no data on how the git method scales.

One thing that I did find out was that the standard sshd config needs to be changed to allow rapid logins by up'ing the MaxStartups value in sshd_config.  Without this change my capistrano job would randomly fail on a few hosts because sshd on the git server was throttling ssh connections.  I think the default is 10 or less.


--
You received this message because you are subscribed to the Google Groups "Agile System Administration" group.
To post to this group, send email to agile-system-...@googlegroups.com.
To unsubscribe from this group, send email to agile-system-admini...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/agile-system-administration?hl=en.


Ryan Dooley

unread,
Nov 30, 2010, 11:29:02 AM11/30/10
to agile-system-...@googlegroups.com
So after reading this post, I think it'll be something that I will be trying out at Lookout. I probably won't be able to get around to switching up the infrastructure here until after the new year though.

Cheers,
Ryan

ranjib dey

unread,
Nov 30, 2010, 1:15:17 PM11/30/10
to agile-system-...@googlegroups.com
why not chef? chef scales pretty well ....this seems to be overkill..

Ryan Dooley

unread,
Nov 30, 2010, 3:28:43 PM11/30/10
to agile-system-...@googlegroups.com

We already have a pretty heavy investment of puppet.  The investment is not so large that it's not impossible to back out of but what we have works for us and we have processes built around puppet.

Cheers,
Ryan

Lindsay Holmwood

unread,
Nov 30, 2010, 6:50:47 PM11/30/10
to Agile System Administration
On Dec 1, 1:33 am, John Arundel <j...@bitfieldconsulting.com> wrote:
> On Mon, Nov 29, 2010 at 8:12 AM, diez <flatl...@stresstiming.de> wrote:
>
> Has anyone actually done this? It's an interesting idea, but I haven't
> heard of it being put into practice.
>

I have, and took it one step further by freezing Puppet into the repo
as a git submodule, so we could A/B test new versions of Puppet across
our infrastructure.

I wrote a wrapper tool around the process that handled checking out
the repo, checking out Puppet as a submodule, setting up the command
line and load path, and running Puppet.

Held it all together with Cap and it worked quite well for managing
~100 machines.

Lindsay

Ryan Dooley

unread,
Nov 30, 2010, 7:28:26 PM11/30/10
to agile-system-...@googlegroups.com
So that's pretty sweet. Any chance of a write up or github gist ;-) I'd happyly contribute to that :D

Cheers,
Ryan

Joe Miller

unread,
Nov 30, 2010, 7:39:26 PM11/30/10
to agile-system-...@googlegroups.com
Lindsay, I'm also interested in hearing more, if you are willing to share, such as code snippets.

Also, you talk in the past tense.  Did you abandon this approach?  If so, why?

Lindsay Holmwood

unread,
Nov 30, 2010, 7:51:37 PM11/30/10
to agile-system-...@googlegroups.com
Unfortunately it's all IP that belongs to a former employer. I can
certainly talk to them about open sourcing it.

If you wanted to re-implement it, it took me about about 4 days build
and deploy it.

Lindsay

Lindsay Holmwood

unread,
Nov 30, 2010, 11:57:59 PM11/30/10
to Agile System Administration
Good news! My former employer has ok'd me to open source it provided I
make it work with Puppet 2.6. :-)

I'll see if I can find some time to hack on it over the next few
days.

Lindsay

Ryan Dooley

unread,
Dec 1, 2010, 12:56:00 AM12/1/10
to agile-system-...@googlegroups.com
That's awesome news!

Cheers,
Ryan

John Arundel

unread,
Dec 2, 2010, 7:29:36 AM12/2/10
to agile-system-...@googlegroups.com
On Wed, Dec 1, 2010 at 5:56 AM, Ryan Dooley <ryan....@gmail.com> wrote:
> That's awesome news!
>
> Cheers,
> Ryan

Seconded… and when you publish it, I can add a link from the article.

Lindsay Holmwood

unread,
Apr 29, 2011, 4:28:22 AM4/29/11
to agile-system-...@googlegroups.com
On 2 December 2010 13:29, John Arundel <jo...@bitfieldconsulting.com> wrote:
>
> Seconded… and when you publish it, I can add a link from the article.
>

Open sourced at:

http://github.com/railsmachine/rump

Slides from a recent presentation I did on Rump at:

http://www.slideshare.net/auxesis/rump-making-puppetmasterless-puppet-meaty

Enjoy!

Lindsay

Reply all
Reply to author
Forward
0 new messages