Disable git clone/pull for specific roles

[StuartM]

Hi,

I have multiple roles in the deploy.rb file, however, I would like to
remove some roles from doing a git clone/pull (not sure which one it
does).

The reason for this is due to it taking it quite a long time for 15+
machines, so my intentions is to run the normal git clone/pull on a
single machine, then create a Debian package and distribute that
instead.

Any help would be greatly appreciated.

Thanks

Stuart

[Lee Hambley]

Stuart,

The only option is `:no_release => true` on the role/server declaration, but beware this does more than disable the clone/checkout - it also disables anything related to source code (think… database server)

If there's a real problem checking our your source from Git, you should perhaps look at the remote_cache strategy, which should reduce your checkout time. Or perhaps the murder gem from Twitter, which uses torrent to distribute the code (very fast.) between machines; and is already rather well integrated with Capistrano.

An alternative would be to come up with your own deploy strategy (not that tough, take a look in the sources) - and use something like `:once`, (old thread here might help: http://www.mail-archive.com/capis...@googlegroups.com/msg06388.html ) - to nominate just one server to do the checkout, then you can run whatever you need on the other machines to distribute the code.

I'd dearly love to see a decent "checkout and share between servers" deployment strategy; but there's some good reasons why this doesn't exist in the defaults, so YMMV.

- Lee

--
* You received this message because you are subscribed to the Google Groups "Capistrano" group.
* To post to this group, send email to capis...@googlegroups.com
* To unsubscribe from this group, send email to capistrano+...@googlegroups.com For more options, visit this group at http://groups.google.com/group/capistrano?hl=en

[Michael Richardson]

>>>>> "Lee" == Lee Hambley <lee.h...@gmail.com> writes:
Lee> I'd dearly love to see a decent "checkout and share between
Lee> servers" deployment strategy; but there's some good reasons why
Lee> this doesn't exist in the defaults, so YMMV.

One of the things I would like is to be able to give git (and git
submodule) a --references option. That would basically cache everything
in some place. Any old checkout would do, but best if it got blessed in
some way.

I will remind that I also wrote a deploy via dpkg awhile ago.

I found it very useful to make deployments to customers who ran the
system behind ugly firewalls that could do HTTP-only, and I had to
access via putty from an RDP session... (so no ssh-agent, and certainly
no outgoing port-22).

--
] He who is tired of Weird Al is tired of life! | firewalls [
] Michael Richardson, Sandelman Software Works, Ottawa, ON |net architect[
] m...@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[
Kyoto Plus: watch the video <http://www.youtube.com/watch?v=kzx1ycLXQSE>
then sign the petition.

[StuartM]

Hi Lee!

Thanks for the response, very grateful.

I'll give `:no_release => true` a try, I do actually use this for our
MongoDB, but as you said it probably does some other things as well.

I like the idea of creating a .deb package and letting apt manage the
packages, then using Capistrano to orchestrate the "apt-get update"
and stop/start of services.

We did look into remote caches of GitHub and nearly started work on
it, but it didn't solve the issue that quite an intensive processes is
happening every deployment for N servers - The deb package seems a
cleaner, more manageable solution.

I'll try the above and report back.

Thanks again.

Stuart

On Aug 15, 3:14 pm, Lee Hambley <lee.hamb...@gmail.com> wrote:
> Stuart,
>
> The only option is `:no_release => true` on the role/server declaration, but
> beware this does more than disable the clone/checkout - it also disables
> anything related to source code (think… database server)
>
> If there's a real problem checking our your source from Git, you should
> perhaps look at the remote_cache strategy, which should reduce your checkout
> time. Or perhaps the murder gem from Twitter, which uses torrent to
> distribute the code (very fast.) between machines; and is already rather
> well integrated with Capistrano.
>
> An alternative would be to come up with your own deploy strategy (not that
> tough, take a look in the sources) - and use something like `:once`, (old

> thread here might help:http://www.mail-archive.com/capis...@googlegroups.com/msg06388.html) - to

> nominate just one server to do the checkout, then you can run whatever you
> need on the other machines to distribute the code.
>
> I'd dearly love to see a decent "checkout and share between servers"
> deployment strategy; but there's some good reasons why this doesn't exist in
> the defaults, so YMMV.
>
> - Lee
>

[Lee Hambley]

One of the things I would like is to be able to give git (and git submodule) a --references option.

That fits with this, which is pending to be pulled; https://github.com/capistrano/capistrano/pull/80 - Where can I read about “--references”, there's nothing obvious in the man pages?

 

I will remind that I also wrote a deploy via dpkg awhile ago.

Where does that live?

[Lee Hambley]

Stuart,

I'd refute the part about using a platform-specific package management tool to deploy your application! If we're being strict about it - why not package your source code as a Gem, and host your own ruby gems repository on the server? (I'm assuming this is a Ruby app)

You'd benefit from packaged code, and an easy and convenient installation on the rest of your cluster?

Might be a long shot, but more and more frequently (owing to Ruby 1.9.2/3 load time disaster) I'm breaking my projects apart this way, even if i'm still using Bundler's pseudo-git-gems functionality, and not really building Gems.)

I'd think about that before dpkg, but if Michael already built something; we could certainly look into that if demand is increasing.

- Lee

[StuartM]

> I'd refute the part about using a platform-specific package management tool
> to deploy your application! If we're being strict about it - why not package
> your source code as a Gem, and host your own ruby gems repository on the
> server? (I'm assuming this is a Ruby app)

Totally agree, and we would do that if we were using 100% Ruby -
however, we have several NodeJS apps (and some Django) in the mix as
well, so creating a deb/rpm/tar.gz and bundling them all together
gives us the flexibility to deploy the entire codebase (in one
package) to every machine in our architecture regardless of it's role.
As a result, everything is the same apart from what we decide to
execute on each machine.

[Michael Richardson]

>>>>> "Lee" == Lee Hambley <lee.h...@gmail.com> writes:

>> One of the things I would like is to be able to give git (and git
>> submodule) a --references option.

Lee> That fits with this, which is pending to be pulled;
Lee> https://github.com/capistrano/capistrano/pull/80 - Where can I
Lee> read about �--references�, there's nothing obvious in the man
Lee> pages?

Yes, I think #80 would mostly solve it, but I'd like to automate it a
bit more, because creating the directory the first time

man git-clone

--reference <repository>
If the reference repository is on the local machine, automatically
setup .git/objects/info/alternates to obtain objects from the
reference repository. Using an already existing repository as an
alternate will require fewer objects to be copied from the
repository being cloned, reducing network and local storage costs.

NOTE: see the NOTE for the --shared option.

>> I will remind that I also wrote a deploy via dpkg awhile ago.

Lee> Where does that live?

It was done awhile ago, likely needs to be updated:
https://github.com/mcr/capistrano

[Michael Richardson]

>>>>> "StuartM" == StuartM <stuar...@gmail.com> writes:
StuartM> I'll give `:no_release => true` a try, I do actually use
StuartM> this for our MongoDB, but as you said it probably does some
StuartM> other things as well.

StuartM> I like the idea of creating a .deb package and letting apt
StuartM> manage the packages, then using Capistrano to orchestrate
StuartM> the "apt-get update" and stop/start of services.

For me, the major plus, if one can get all of one's gem's repackaged as
dpkg, is that upgrades to appliances (where the customer pulls...) make
more sense.
(Also, any gem's that have native code require a compiler on the target
system, which is annoying from a minimal security profile point of view)

StuartM> We did look into remote caches of GitHub and nearly started
StuartM> work on it, but it didn't solve the issue that quite an
StuartM> intensive processes is happening every deployment for N
StuartM> servers - The deb package seems a cleaner, more manageable
StuartM> solution.

So you mean, you had a server, not github.com, which was the master for
your repository. You are still pulling a lot of data from it.
--references would be a much bigger win network and disk-wise.

Deploy-via-dpkg means that you don't ship any history at all.
You might also want to try the copy methods, but I don't know if they
have been integrated with git-submodules before.