Bundle Install hanging, using a lot of memory

[bob]

Using bundler 1.0.0.rc.1
I am having a problem getting bundler to install gems since updating
to rc 1 this morning.

I have run bundle update, bundle install, and bundle package, in that
order on my local machine. They all work fine. The problem is when I
try to deploy my app and run bundler on the remote server.

When I was using bundler 0.9 bundle install --without test worked well
for me.

In attempts to get bundler to install the gems I have tried running
the following commands.
bundle install --without test
bundle install vendor/bundle
bundle install vendor --disable-shared-gems
All of those commands run for a long time ( several minutes ) and use
up all the available RAM (~400MB) and get into swap. Bundler has
never successfully terminated, I have always killed the process after
5 minutes or so of making no progress.

I am running Ruby 1.8.7, RubyGems 1.3.7, and Bundler 1.0.0.rc.1 on a
Fedora box.

Why is Bundler running so long, taking so much memory, and what can I
do to make it work?

[bob]

I finally tried

bundle install vendor/bundle --local

and that ran without incident. Seems like a pretty big change from
0.9, as far as needing new flags to get it to work.

[mateo]

On Jul 28, 10:23 pm, bob <larr...@gmail.com> wrote:

I've run into the same issue, rc.1 worked fine for me a couple of days
ago, but now runs out of memory; rc.2 is not any better. --local
doesn't work.

Anybody have any ideas on how to get this to work? I'm using
capistrano.

[mateo]

On Aug 1, 4:02 pm, mateo <mateo.mur...@gmail.com> wrote:

> I've run into the same issue, rc.1 worked fine for me a couple of days
> ago, but now runs out of memory; rc.2 is not any better. --local
> doesn't work.

As it turns out, there was simply less memory available on my staging
box yesterday than the day before, and killing a couple of processes
helped. Bundler is definitely using excessive amounts of memory to do
its work though, and a fix for this would be greatly appreciated.

[Yehuda Katz]

We have fixed and applied a patch written by Kyle Shank (http://github.com/carlhuda/bundler/commit/43fe70ef22a6443865207936d0b4277169e19d4a) which significantly improves memory usage (by 2-3x). It will be in the next release :)

We will also be pursuing other avenues (including working with Gemcutter and further profiling bundler itself).

You can mitigate your problems even further by:

• running `bundle cache` in development. this will put the .gem files you need into your vendor/cache directly, and remove the need to hit rubygems.org on deploy
• use the `--production` flag in deployment. among other things, this will ensure that we do not hit rubygems.org if all the gems you need are stored in vendor/cache

Yehuda Katz
Architect | Engine Yard
(ph) 718.877.1325

--
You received this message because you are subscribed to the Google Groups "ruby-bundler" group.
To post to this group, send email to ruby-b...@googlegroups.com.
To unsubscribe from this group, send email to ruby-bundler...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/ruby-bundler?hl=en.

[Mateo Murphy]

On 2-Aug-10, at 6:22 PM, Yehuda Katz wrote:

> We have fixed and applied a patch written by Kyle Shank (http://github.com/carlhuda/bundler/commit/43fe70ef22a6443865207936d0b4277169e19d4a
> ) which significantly improves memory usage (by 2-3x). It will be in
> the next release :)

Great news, thanks!

>
> We will also be pursuing other avenues (including working with
> Gemcutter and further profiling bundler itself).
>
> You can mitigate your problems even further by:
> • running `bundle cache` in development. this will put the .gem
> files you need into your vendor/cache directly, and remove the need
> to hit rubygems.org on deploy

this involves checking in vendor/cache into source control as well,
yes? how does this handle compiled gems?

> • use the `--production` flag in deployment. among other things,

> this will ensure that we do not hit rubygems.org if all the gems you
> need are stored in vendor/cache

regarding this, since I'm using capistrano I would need to symlink
vendor/cache into shared/ is well, yes?

thanks for the help

[Andre Arko]

On Aug 2, 2010, at 3:32 PM, Mateo Murphy wrote:
> this involves checking in vendor/cache into source control as well, yes? how does this handle compiled gems?

Yes. Compiled gems are just that, compiled when they are installed. The .gem files stored in vendor/cache are not compiled yet.

> regarding this, since I'm using capistrano I would need to symlink vendor/cache into shared/ is well, yes?

Well, if vendor/cache is in your source control, I'm not sure why you would need to symlink it into shared.

— Andre

[Mateo Murphy]

On 2-Aug-10, at 6:41 PM, Andre Arko wrote:

>
> On Aug 2, 2010, at 3:32 PM, Mateo Murphy wrote:
>> this involves checking in vendor/cache into source control as well,
>> yes? how does this handle compiled gems?
>
> Yes. Compiled gems are just that, compiled when they are installed.
> The .gem files stored in vendor/cache are not compiled yet.

right, that makes sense

>
>> regarding this, since I'm using capistrano I would need to symlink
>> vendor/cache into shared/ is well, yes?
>
> Well, if vendor/cache is in your source control, I'm not sure why
> you would need to symlink it into shared.

Sorry, I was assuming that --production would also cache gems, rather
than having vendor/cache checked into source control (which I'd rather
not do, if possible).

Seems to me that having a shared cache would be useful as a production
stratregy? It could be done manually by running "bundle install --
production" followed by "bundle cache"

[Yehuda Katz]

Yehuda Katz
Architect | Engine Yard
(ph) 718.877.1325

You wouldn't need to do that. Instead, you could simply symlink vendor/bundle to a shared location, and then run bundle install. This will share the installation location across deploys for the same application. If you don't make any changes to your Gemfile, running bundle install --production on every deploy will be a noop, so you should not need to store the .gem files as well.

[Yehuda Katz]

Yehuda Katz
Architect | Engine Yard
(ph) 718.877.1325

On Mon, Aug 2, 2010 at 4:06 PM, Mateo Murphy <mateo....@gmail.com> wrote:

Seems to me that having a shared cache would be useful as a production stratregy? It could be done manually by running "bundle install --production" followed by "bundle cache"

You wouldn't need to do that. Instead, you could simply symlink vendor/bundle to a shared location, and then run bundle install. This will share the installation location across deploys for the same application. If you don't make any changes to your Gemfile, running bundle install --production on every deploy will be a noop, so you should not need to store the .gem files as well.

 

[Mateo Murphy]

On 2-Aug-10, at 8:50 PM, Yehuda Katz wrote:

> You wouldn't need to do that. Instead, you could simply symlink
> vendor/bundle to a shared location, and then run bundle install.
> This will share the installation location across deploys for the
> same application. If you don't make any changes to your Gemfile,
> running bundle install --production on every deploy will be a noop,
> so you should not need to store the .gem files as well.

Of course.. I guess I hadn't thought out my question properly.

Thanks for the help