Slow `bundle install`, "Fetching source index..."

[Mark McGranaghan]

Hi All,

I'm really enjoying the dependency resolution and isolation that
Bundler provides, but I'm having some problems with very long install
times. In particular, running `bundle install` often takes several
minutes to run, most of it spent "Fetching source index for http://rubygems.org/".
I'm wondering if I'm doing something wrong, or if not why it takes so
long and how we might improve the situation.

Here is an example workflow from a recent app I worked on:

# start with just a Gemfile in a clean environment
$ bundle -v
Bundler version 1.0.0.rc.5

$ bundle config
Settings are listed in order of priority. The top value will be used.

$ ls
Gemfile

$ cat Gemfile
source :gemcutter

gem "sinatra", "1.0"
gem "json_pure", "1.4.3"
gem "stalker", "0.4.2"
gem "sequel", "3.11.0"
gem "pg", "0.9.0"
gem "memcached", "0.19.7"
gem "rake", "0.8.7"
gem "thin", "1.2.7"
gem "uuidtools", "2.1.1"
gem "system_timer", "1.0"
gem "rack", "1.2.1"
gem "activesupport", "2.3.8"
gem "aws", "2.3.13"
gem "rest-client", "1.6.0"
gem "exceptional", "2.0.15"
gem "pony", "1.0"
gem "newrelic_rpm", "2.12.3"
gem "log4r", "1.1.8"
gem "rake", "0.8.7"
gem "warden", "0.10.7"

# install for the first time, without a lock file
$ time bundle install
Fetching source index for http://rubygems.org/
Using rake (0.8.7)
Using activesupport (2.3.8)
Using http_connection (1.3.0)
Using uuidtools (2.1.1)
Using xml-simple (1.0.12)
Using aws (2.3.13)
Using beanstalk-client (1.1.0)
Using daemons (1.1.0)
Using eventmachine (0.12.10)
Using exceptional (2.0.15)
Using json_pure (1.4.3)
Using log4r (1.1.8)
Using mime-types (1.16)
Using polyglot (0.3.1)
Using treetop (1.4.8)
Using mail (2.2.5)
Using memcached (0.19.7)
Using newrelic_rpm (2.12.3)
Using pg (0.9.0)
Using pony (1.0)
Using rack (1.2.1)
Using rest-client (1.6.0)
Using sequel (3.11.0)
Using sinatra (1.0)
Using stalker (0.4.2)
Using system_timer (1.0)
Using thin (1.2.7)
Using warden (0.10.7)
Using bundler (1.0.0.rc.5)
Your bundle is complete! Use `bundle show [gemname]` to see where a
bundled gem is installed.

real 2m41.885s
user 1m55.330s
sys 0m37.416s

# try installing again right away
$ time bundle install
...
real 0m3.406s
user 0m2.682s
sys 0m0.677s

# add a dependency
$ echo 'gem "warden-googleapps" -v 0.1.1' >> Gemfile

# install again
$ bundle install
...
Installing warden-googleapps (0.1.1)
Using bundler (1.0.0.rc.5)
Your bundle is complete! Use `bundle show [gemname]` to see where a
bundled gem is installed.

real 2m41.358s
user 1m58.116s
sys 0m38.596s

Out of the three `bundle install`s issued above, the first and third
seem very slow, taking on the order of three minutes.

I'm especially surprised that the first one takes so long since I had
all of the required gems available locally.

I'm also surprised that the third one took so long since only 1 of the
gems (the one added by with `echo`) was not installed locally.

Am I doing something wrong that is causing these `bundle install`s to
take so long? If not, why is "Fetching source index for http://rubygems.org/"
necessary and why does it take so long? If this is known and expected
behavior, how can I help to make Bundler faster in these cases?

[Wincent Colaiuta]

On 22 ago, 07:36, Mark McGranaghan <mmcgr...@gmail.com> wrote:
>
> I'm really enjoying the dependency resolution and isolation that
> Bundler provides, but I'm having some problems with very long install
> times. In particular, running `bundle install` often takes several

> minutes to run, most of it spent "Fetching source index forhttp://rubygems.org/".

> I'm wondering if I'm doing something wrong, or if not why it takes so
> long and how we might improve the situation.

There's been talk about this on the gemcutter group:

http://groups.google.com/group/gemcutter/browse_thread/thread/53be5eb64be79a86

I also previously filed a related ticket in the Bundler issue tracker:

http://github.com/carlhuda/bundler/issues/505

Cheers,
Wincent

[Alex Chaffee]

Given that Bundler often stalls out "Fetching source index for
http://rubygems.org", any thoughts about an option to cache it on the
local machine?

e.g. store it in ~/.bundle/source_index.marshal
and if you call "bundle install" with a "--use-cache" option it'll
look there, and if it's not too stale (say, 4 hours old or so) then
use it instead of trying to fetch from rubygems. Or if rubygems takes
more than a few seconds to reply.

It would be easy to implement, but I'm not interested in submitting a
patch if Carlhuda aren't ok with it.

On Aug 23, 2:56 am, Wincent Colaiuta <w...@wincent.com> wrote:
> On 22 ago, 07:36, Mark McGranaghan <mmcgr...@gmail.com> wrote:
>
>
>
> > I'm really enjoying the dependency resolution and isolation that
> > Bundler provides, but I'm having some problems with very long install
> > times. In particular, running `bundle install` often takes several
> > minutes to run, most of it spent "Fetching source index forhttp://rubygems.org/".
> > I'm wondering if I'm doing something wrong, or if not why it takes so
> > long and how we might improve the situation.
>
> There's been talk about this on the gemcutter group:
>

> http://groups.google.com/group/gemcutter/browse_thread/thread/53be5eb...

[Andre Arko]

On Aug 30, 2010, at 12:06 PM, Alex Chaffee wrote:

> Given that Bundler often stalls out "Fetching source index for
> http://rubygems.org", any thoughts about an option to cache it on the
> local machine?

The install command has a --local option already, and gemcutter just added a "resolver" endpoint that returns only the indexes of the gems that bundler actually needs. We will be using that endpoint by 1.1, and it is _way, way, way_ faster. :)

— Andre

[Alex Chaffee]

On Mon, Aug 30, 2010 at 5:47 PM, Andre Arko <an...@arko.net> wrote:
> On Aug 30, 2010, at 12:06 PM, Alex Chaffee wrote:
>
>> Given that Bundler often stalls out "Fetching source index for
>> http://rubygems.org", any thoughts about an option to cache it on the
>> local machine?
>
> The install command has a --local option already

Ah! Excellent. It's not in 0.9.4 so I hadn't seen it.

> , and gemcutter just added a "resolver" endpoint that returns only the indexes of the gems that bundler actually needs. We will be using that endpoint by 1.1, and it is _way, way, way_ faster. :)
>

Also excellent. :-)

- A

--
Alex Chaffee - al...@stinky.com - http://alexch.github.com
Stalk me: http://friendfeed.com/alexch | http://twitter.com/alexch |
http://alexch.tumblr.com

[Alex Chaffee]

On Mon, Aug 30, 2010 at 5:47 PM, Andre Arko <an...@arko.net> wrote:

Six months have now passed, and "Fetching source index" still feels
slow. Is the new endpoint in use yet? And if so, is it living up to
its promise?

Either way, I'd still like to lobby for a source index cache feature.
The --local option doesn't quite do it since I want to cache the
source index, not the gems themselves. That way if a bundle requires a
gem I don't have locally, it will go out and download it, but it won't
bother reloading the source index unless (a) i tell it to, or (b) a
certain amount of time has passed since the last time it loaded it.

[Adam Soltys]

Nick Quaranto (qrush), the gemcutter maintainer, has written up an
excellent blog post about this issue:
http://robots.thoughtbot.com/post/2729333530/fetching-source-index-for-http-rubygems-org

On Feb 17, 11:28 am, Alex Chaffee <a...@stinky.com> wrote:
> On Mon, Aug 30, 2010 at 5:47 PM, Andre Arko <an...@arko.net> wrote:
> > On Aug 30, 2010, at 12:06 PM, Alex Chaffee wrote:
>
> >> Given that Bundler often stalls out "Fetching source index for
> >>http://rubygems.org", any thoughts about an option to cache it on the
> >> local machine?
>
> > The install command has a --local option already, and gemcutter just added a "resolver" endpoint that returns only the indexes of the gems that bundler actually needs. We will be using that endpoint by 1.1, and it is _way, way, way_ faster. :)
>
> Six months have now passed, and "Fetching source index" still feels
> slow. Is the new endpoint in use yet? And if so, is it living up to
> its promise?
>
> Either way, I'd still like to lobby for a source index cache feature.
> The --local option doesn't quite do it since I want to cache the
> source index, not the gems themselves. That way if a bundle requires a
> gem I don't have locally, it will go out and download it, but it won't
> bother reloading the source index unless (a) i tell it to, or (b) a
> certain amount of time has passed since the last time it loaded it.
>
> --

> Alex Chaffee - a...@stinky.com -http://alexch.github.com