Any advice using capistrano and git to deploy sites with heavy flash assets?

[Chad A.]

Hi,

We are currently experimenting with using both git and capistrano to
deploy our websites. Here's the situation. Both our flash developers
and backend developers use the same git repository (don't really
believe in submodules). Now only certain flash pieces need to be
deployed - the rest are binary assets used to build the flash
pieces.

Using git and capistrano, when we cap deploy, capistrano deploys all
files in the repo, even those that don't need to be - those binary
flash assets. This is not capistrano's fault. It's git. As far as I
know, there is no way to 'pull' only one part of a git repo (unlike
svn).

So I'm wondering if any other cap users out there have found a way not
to deploy certain parts of their git repos. I've considered writing a
custom capistrano task that deletes everything that should not be
deployed once it's on the server, but I fear that might interfere with
the our deploy_via, :remote cache setting.

Any advice would be greatly appreciated. Thanks

Chad

[Lee Hambley]

Hi Chad,

I'm afraid the usecase you are mentioning isn't possible with Git - this isn't a limitation on our end; rather one that the Git mindset has.

You can read a little more here; though - this topic polarises Git users more than the emacs/vim argument:– http://stackoverflow.com/questions/600079/is-there-any-way-to-clone-a-git-repositorys-sub-directory-only

We have a patch, that works ( actually a hack, it deploys all the files to the server, but only copies one directory to the release directory ) you can find that here; it might suit your needs but, it only works with :remote_cache... and it comes without any warranty:– https://capistrano.lighthouseapp.com/projects/8716/tickets/91

There's actually a good reason we can't implement this in the other deployment strategies - mostly because this is accepted to be a limitation... if that is the right word of Git itself.

They have no interest in this feature, and we can't support something like this that isn't supported; to the Git community creating new repositories is so inexpensive that you should be using two repositories.

That said, I have had limited success (I didn't really peruse it, but it worked pretty well) using two branches without common ancestry - in my case this took the form of a `design` branch, and a `development` branch, design had all my binary resources; wire-frames, mocks, photoshop files, etc... and the development had all the code, of course -- you can take a look at the idea of maintaing two branches with no common ancestry here; this is the next-best solution for splitting one repository into two:– https://wincent.com/wiki/Creating_independent_branches_with_Git

Hope this helps Chad!

-- Lee Hambley

Twitter: @leehambley | @capistranorb
Blog: http://lee.hambley.name/
Working with Rails: http://is.gd/1s5W1

2009/11/4 Chad A. <cal...@neomantic.com>

[Chad A.]

Thanks Lee, for the solid advice. I suspected as much - the issue
falls squarely on git's shoulders. It will make git a more difficult
sell to my organization.

Thanks again,

Chad

On Nov 4, 4:28 pm, Lee Hambley <lee.hamb...@gmail.com> wrote:
> Hi Chad,
>
> I'm afraid the usecase you are mentioning isn't possible with Git - this
> isn't a limitation on our end; rather one that the Git mindset has.
>
> You can read a little more here; though - this topic polarises Git users

> more than the emacs/vim argument:–http://stackoverflow.com/questions/600079/is-there-any-way-to-clone-a...

>
> We have a patch, that works ( actually a hack, it deploys all the files to
> the server, but only copies one directory to the release directory ) you can
> find that here; it might suit your needs but, it only works with

> :remote_cache... and it *comes without any warranty*:–https://capistrano.lighthouseapp.com/projects/8716/tickets/91

>
> There's actually a good reason we can't implement this in the other
> deployment strategies - mostly because this is accepted to be a
> limitation... if that is the right word of Git itself.
>
> They have no interest in this feature, and we can't support something like
> this that isn't supported; to the Git community creating new repositories is
> so inexpensive that you should be using two repositories.
>
> That said, I have had limited success (I didn't really peruse it, but it
> worked pretty well) using two branches without common ancestry - in my case
> this took the form of a `design` branch, and a `development` branch, design
> had all my binary resources; wire-frames, mocks, photoshop files, etc... and
> the development had all the code, of course -- you can take a look at the
> idea of maintaing two branches with no common ancestry here; this is the
> next-best solution for splitting one repository into two:–https://wincent.com/wiki/Creating_independent_branches_with_Git
>
> Hope this helps Chad!
>
> -- Lee Hambley
>
> Twitter: @leehambley | @capistranorb
> Blog:http://lee.hambley.name/
> Working with Rails:http://is.gd/1s5W1
>

> 2009/11/4 Chad A. <calb...@neomantic.com>

[Lee Hambley]

Chad, 

I should add that the branch notion is the correct way to do this; under your circumstances - however this 

We have a patch, that works ( actually a hack, it deploys all the files to the server, but only copies one directory to the release directory ) you can find that here; it might suit your needs but, it only works with :remote_cache... and it comes without any warranty:– https://capistrano.lighthouseapp.com/projects/8716/tickets/91

Is a perfectly usable solution; and the :remote_cache strategy keeps a copy of the repository on the server so that a new checkout only performs a pull to get the new changes, then can copy individual directories out to the release directories.

-- Lee Hambley

Twitter: @leehambley | @capistranorb
Blog: http://lee.hambley.name/
Working with Rails: http://is.gd/1s5W1

2009/11/5 Chad A. <cal...@neomantic.com>

[skrat]

We've been having similar issue, and proven solution is to NOT put any
big binary stuff into git repo, but having on a CDN, and having a URL
rewriting script which will update paths in source to those of CDN.

Alternative solution could be to put those into some folder in /shared
and make symlinks in after deploy task

You will also get faster git workflow as storing large binaries means
lengthy pulls

On Nov 5, 10:35 am, Lee Hambley <lee.hamb...@gmail.com> wrote:
> Chad,
>
> I should add that the branch notion is the correct way to do this; under
> your circumstances - however this
>
> We have a patch, that works ( actually a hack, it deploys all the files to
>
> > the server, but only copies one directory to the release directory ) you can
> > find that here; it might suit your needs but, it only works with
> > :remote_cache... and it comes without any warranty:–https://capistrano.
> > lighthouseapp.com/projects/8716/tickets/91
>
> Is a perfectly usable solution; and the :remote_cache strategy keeps a copy
> of the repository on the server so that a new checkout only performs a pull
> to get the new changes, then can copy individual directories out to the
> release directories.
>
> -- Lee Hambley
>
> Twitter: @leehambley | @capistranorb
> Blog:http://lee.hambley.name/
> Working with Rails:http://is.gd/1s5W1
>

> 2009/11/5 Chad A. <calb...@neomantic.com>

[Lee Hambley]

Skrat, they should be in version control somewhere.

Chad to elaborate on my point from earlier - you should try and sell to your employers that sharing one repository between development and design is a disaster waiting to happen.

Consider that most [designers, I apologise!] designers tend to have not-so-strong a technical background, and that they probably struggle a little with version control systems; there are solutions [1] of course [2] - but they are often sub-optimal.

Also, there is security - you should consider that your revision control system should be one of the most secure parts of your infrastructure; people who use Github are trusting… and you should be mindful to make sure this is approached carefully; naturally this also expands to security within your organisation;– at my current employer, I have quite a free reign of access, but I am still denied read access to a number of repositories here... "need to know" springs to mind… so it makes perfect sense to consider that your designers / flash engineers don't need access to the code base where you store details of your cookie/session configuration, maybe apache configurations; worse… firewall configuration!

Analysing these issues is important, and often makes great business sense to separate these things out; maliciousness aside, there's still a great deal of damage that could be done if your designers/front-end engineers have the power to corrupt/damage/etc your code repository.

I'd go as far to say that you should consider having separate deploys (sure, you'll often run them at the same time) - but do you want the overhead of deploying nMb of binary assets next time you need to hot-patch something your tests missed? 

So, in summary;

• You should really consider this to be a cue to split your codebase up
• You should be version controlling everything (even if this means you use Mecurial/Darcs for binary resources (apparently they handle them better) - and Git for your code.

[1] Easy Git - http://www.gnome.org/~newren/eg/

[2] GitX - http://gitx.frim.nl/

-- Lee Hambley

Twitter: @leehambley | @capistranorb
Blog: http://lee.hambley.name/
Working with Rails: http://is.gd/1s5W1

2009/11/5 skrat <dusan.m...@gmail.com>

[skrat]

Well, this is not my issue, we already solved it, and surprise! - yes
we have separate deploys for this stuff

> deploying *n*Mb of binary assets next time you need to hot-patch something

> your tests missed?
>
> So, in summary;
>

>    - You should *really* consider this to be a cue to split your codebase up
>    - You should be version controlling everything (even if this means you

>    use Mecurial/Darcs for binary resources (apparently they handle them better)
>    - and Git for your code.
>
> [1] Easy Git -http://www.gnome.org/~newren/eg/

> [2] GitX -http://gitx.frim.nl/

>
> -- Lee Hambley
>
> Twitter: @leehambley | @capistranorb
> Blog:http://lee.hambley.name/
> Working with Rails:http://is.gd/1s5W1
>

> 2009/11/5 skrat <dusan.malia...@gmail.com>

[Rafael G.]

A bit late, but I want to said that I agree with Lee.

I was investigating and asking to experienced users(I'm not an expert
git user) about a clean method to do it, and none give me a simple
solution because as said Lee it isn't supported by Git community.

I invite you to make a plugin with this patch that work with the other
strategies :)

Regards!

Lee Hambley wrote:
> Hi Chad,
>
> I'm afraid the usecase you are mentioning isn't possible with Git -
> this isn't a limitation on our end; rather one that the Git mindset has.
>
> You can read a little more here; though - this topic polarises Git
> users more than the emacs/vim
> argument:– http://stackoverflow.com/questions/600079/is-there-any-way-to-clone-a-git-repositorys-sub-directory-only
>
> We have a patch, that works ( actually a hack, it deploys all the
> files to the server, but only copies one directory to the release
> directory ) you can find that here; it might suit your needs but, it

> only works with :remote_cache... and it /comes without any warranty/:–

> https://capistrano.lighthouseapp.com/projects/8716/tickets/91
>
> There's actually a good reason we can't implement this in the other
> deployment strategies - mostly because this is accepted to be a
> limitation... if that is the right word of Git itself.
>
> They have no interest in this feature, and we can't support something
> like this that isn't supported; to the Git community creating new
> repositories is so inexpensive that you should be using two repositories.
>
> That said, I have had limited success (I didn't really peruse it, but
> it worked pretty well) using two branches without common ancestry - in
> my case this took the form of a `design` branch, and a `development`
> branch, design had all my binary resources; wire-frames, mocks,
> photoshop files, etc... and the development had all the code, of
> course -- you can take a look at the idea of maintaing two branches
> with no common ancestry here; this is the next-best solution for
> splitting one repository into
> two:– https://wincent.com/wiki/Creating_independent_branches_with_Git
>
> Hope this helps Chad!
>
> -- Lee Hambley
>
> Twitter: @leehambley | @capistranorb
> Blog: http://lee.hambley.name/
> Working with Rails: http://is.gd/1s5W1
>
>

> 2009/11/4 Chad A. <cal...@neomantic.com <mailto:cal...@neomantic.com>>

>
>
> Hi,
>
> We are currently experimenting with using both git and capistrano to
> deploy our websites. Here's the situation. Both our flash developers
> and backend developers use the same git repository (don't really
> believe in submodules). Now only certain flash pieces need to be
> deployed - the rest are binary assets used to build the flash
> pieces.
>
> Using git and capistrano, when we cap deploy, capistrano deploys all
> files in the repo, even those that don't need to be - those binary
> flash assets. This is not capistrano's fault. It's git. As far as I
> know, there is no way to 'pull' only one part of a git repo (unlike
> svn).
>
> So I'm wondering if any other cap users out there have found a way not
> to deploy certain parts of their git repos. I've considered writing a
> custom capistrano task that deletes everything that should not be
> deployed once it's on the server, but I fear that might interfere with
> the our deploy_via, :remote cache setting.
>
> Any advice would be greatly appreciated. Thanks
>
> Chad
>
>
>
>
> >

--
Rafa

[Richie Vos]

last Thursday I sent this directly to Chad on accident and he was nice
enough to forward it back to me for re-posting to the list.

We have the same situation on my site. We're a rails back-end with
flash/flex front-ends. It seems like there might be 2 issues here. 1
is getting the build files of your app out of your rails project and 2
deploying the final generated swfs to the rails project.

Our approach is we have 2 completely separate repositories. 1 for our
flex stuff, 1 for our rails stuff (we actually have more than 2, but
for simplicity assume 2).

In development mode we setup directories like so:

flash/flash_project_and_all_build_files
rails/app
rails/public
...

When building our swfs we set the tools (FlexBuilder for us) to put
the swf in rails/public. We do not store our development swfs in
version control. If you're going to use git, you don't want to do
this. Assuming they change frequently and are committed frequently,
you're going to end up with a ginormous repository. It's a pain.

When deploying to production our process then is (we automated this
with cap):
1. Checkout the production branch in rails
2. Build the latest swf from flash
3. Copy it into rails/public
4. Commit it to that branch
5. Push the production branch (which has the swf in it)

Personally, I'm hoping to move away from even committing a production
swf to our rails source (you can always rebuild it if you need it) and
instead just have part of the deployment either copy the swf to the
production box, or push it to s3 and serve it from there (have to deal
with some cross-domain stuff then).

This process works pretty well for us and leaves us with a lean-mean
rails repo with a fast deployment.

Another option here is git submodules. Also, you can clone 1 git repo
into another if you want your flash files in your rails dev directory,
but are willing to do commits separately.

git clone ..rails.git rails/
cd rails/
git clone flash.git flash/
echo flash >> .gitignore

will set you up with
rails
rails/flash

Completely different option:
When deploying, in cap, if you use the remote_cache strategy you can
set :copy_exclude, to exclude particular files from your deployment.
If you can keep your flash development files all together in say
rails/
flash-dev all you'd have to do is:

set :copy_exclude, ['rails/flash-dev'] and they won't be in your
releases.