[Rubygems-developers] Gemcutter
Nick Quaranto
which aims to deliver a fresh take on gem hosting for the Ruby community.
The purpose of the project is three-fold:
* Create more transparent and accessible project pages
* Enable the community to improve and enhance the site
* Provide a better API for dealing with gems
Gemcutter is now up and running on Heroku, and the gem serving is done
entirely through S3. Patches are beginning to roll in from contributors, and
a redesign is in progress at http://staging.gemcutter.org thanks to some
designer time donated from Thoughtbot.
I would like to emphasize that I've worked hard to make this an open, honest
effort towards improving a core part of the Ruby world that's easily
accessible to every Ruby developer. The overall goal of the project is to
become the default gem host for RubyGems. However, this can't happen without
the support of the RubyGems and RubyForge team.
Here's what I propose:
1) Redirect gems.rubyforge.org to gemcutter.org for gem serving. (all gems
are currently mirrored from RubyForge and are ready for consumption)
2) Make gemcutter.org the default gem host for the next major release of
RubyGems.
3) Keep RubyForge online since many projects depend on its bug trackers,
mailing lists, and more.
Please note I'm not suggesting that all of these happen *this instant*, I
just want to open up discussion about making this happen and what would be
involved. Your thoughts and comments would be appreciated.
-Nick Quaranto
_______________________________________________
Rubygems-developers mailing list
http://rubyforge.org/projects/rubygems
Rubygems-...@rubyforge.org
http://rubyforge.org/mailman/listinfo/rubygems-developers
John Barnette
On Aug 26, 2009, at 1:42 PM, Nick Quaranto wrote:
> 1) Redirect gems.rubyforge.org to gemcutter.org for gem serving.
> (all gems
> are currently mirrored from RubyForge and are ready for consumption)
RubyForge currently has a pretty good mirror system for supporting Gem
downloads. I expect Tom could get your some figures on how much
bandwidth they use. If Gemcutter was the default RG source, how would
you deal with this? Are you planning on eating S3 costs yourself? Are
you thinking about using something like CloudFront, since S3 itself
isn't strictly a CDN?
~ j.
Chad Woolley
> Please note I'm not suggesting that all of these happen *this instant*, I
> just want to open up discussion about making this happen and what would be
> involved. Your thoughts and comments would be appreciated.
To be honest, this seems Not Ready For Prime Time:
* Pagination/sorting is wrong and broken
* Project pages contain obviously broken/incorrect links (version and
homepage links)
* Project pages seem to have much less information than I have on
RubyForge. No release notes, individual file links (.gem, .tar, etc),
etc... (broken and less != transparent and accessible)
* I uploaded a gem to RubyForge yesterday just before midnight, it is
still not mirrored sixteen hours later (and it was available almost
immediately on Rubyforge, which shows their mirror process is working
very well)
That's just what I found in one minute of poking around, it doesn't
give me a good feeling for the overall quality and stability of the
effort. I'm also concerned about long-term support. No offense to
you, but Tom Copeland is a machine, he is always there to support
Rubyforge, year after year...
In general, if it Ain't Broken, Don't Fix It - and think especially
hard if the replacement isn't a no-brainer out-of-the-park homerun...
Is there a reason we can't get many of these benefits from
RubyForge.org? For example, your API seems like a good idea, but why
can't it be a wrapper or enhancement to the current RubyGems/RubyForge
ecosystem? A Gem plugin, even?
Finally, I don't see why there needs to be a change to the canonical
source, since Rubyforge isn't even really one anymore. That ship has
already sailed, many people don't even publish gems to RubyForge, they
just use Github or their own servers. If anything there needs to be
better support for multiple sources (There are good ideas here not yet
implemented). The multiple-naming problem is a bigger but related
issue as Eric discussed
(http://blog.segment7.net/articles/2009/02/04/a-rubygems-github-proposal)...
-- Chad
Ryan Davis
On Aug 26, 2009, at 14:57 , John Barnette wrote:
> Nick,
>
> On Aug 26, 2009, at 1:42 PM, Nick Quaranto wrote:
>> 1) Redirect gems.rubyforge.org to gemcutter.org for gem serving.
>> (all gems
>> are currently mirrored from RubyForge and are ready for consumption)
>
> RubyForge currently has a pretty good mirror system for supporting
> Gem downloads. I expect Tom could get your some figures on how much
> bandwidth they use. If Gemcutter was the default RG source, how
> would you deal with this? Are you planning on eating S3 costs
> yourself? Are you thinking about using something like CloudFront,
> since S3 itself isn't strictly a CDN?
(possibly) even better, I'm pretty sure we could get Tom convinced
that gemcutter is a good idea and we should migrate rubyforge to it.
Luis Lavena
>
> On Aug 26, 2009, at 14:57 , John Barnette wrote:
>
>> Nick,
>>
>> On Aug 26, 2009, at 1:42 PM, Nick Quaranto wrote:
>>>
>>> 1) Redirect gems.rubyforge.org to gemcutter.org for gem serving. (all
>>> gems
>>> are currently mirrored from RubyForge and are ready for consumption)
>>
>> RubyForge currently has a pretty good mirror system for supporting Gem
>> downloads. I expect Tom could get your some figures on how much bandwidth
>> they use. If Gemcutter was the default RG source, how would you deal with
>> this? Are you planning on eating S3 costs yourself? Are you thinking about
>> using something like CloudFront, since S3 itself isn't strictly a CDN?
>
> (possibly) even better, I'm pretty sure we could get Tom convinced that
> gemcutter is a good idea and we should migrate rubyforge to it.
>
The thing is that RubyForge is more than just gems.
Take as example the installers for Ruby for Windows:
http://rubyforge.org/frs/?group_id=167&release_id=38052
I believe that GemCutter API and interface has potential, but as was
previously mentioned, the support and cost of backing it up raises
some concerns.
Taking in consideration there are lot of gems with more than 10K of
downloads, the incurring charge of running that form S3 and adding
CloudFront CDN on top will need to be payed my someone.
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
Nick Quaranto
hasn't come up. My main concern is keeping the gap between publishing
a gem and being able to install it low.
As for the costs, I'd like to approach a non-profit organization such
as RubyCentral to help deal with them. I originally sent an email out
to them first about supporting the project a few weeks ago, but I
haven't heard anything back. I'm interested to hear their opinions on
the site as well.
I would love to get an estimate on bandwidth since the costs are easy
to calculate for S3. So far with over 2,000 gem downloads and
uploading all of the gems from RubyForge over the past 2 months it's
been less than $5.
Nick Quaranto
finished project yet. There's a full fledged redesign in progress that is
being pushed out to: http://staging.gemcutter.org Check out the preview post
here too:
http://robots.thoughtbot.com/post/165832471/gemcutter-org-redesign-preview
I'd also like to pull in all of the READMEs for each gem and show them at
the bottom of the gem page and get them indexed for searching. The API is
already being used by the gem plugins provided by the gemcutter gem...that's
how `gem push` works for instance.
-Nick
> (http://blog.segment7.net/articles/2009/02/04/a-rubygems-github-proposal).<http://blog.segment7.net/articles/2009/02/04/a-rubygems-github-proposal%29.>
> ..
>
> -- Chad
> _______________________________________________
> Rubygems-developers mailing list
> http://rubyforge.org/projects/rubygems
> Rubygems-...@rubyforge.org<http://rubyforge.org/projects/rubygems%0ARubygems...@rubyforge.org>
> http://rubyforge.org/mailman/listinfo/rubygems-developers
Luis Lavena
> I'll take a look into CloudFront, I just haven't yet since the need
> hasn't come up. My main concern is keeping the gap between publishing
> a gem and being able to install it low.
>
CloudFront is a transparent CDN, you only need to subscribe to the
service and enable it for the S3 bucket you're using.
Then, change the URL to instead of using S3 bucket directly uses the
CNAME of the CloudFront enabled bucket.
old url: http://my-bucket.s3.amazonaws.com/path/to/my/gem.gem
new url: http://assets.gemcutter.org/path/to/my/gem.gem
Or something similar.
> I would love to get an estimate on bandwidth since the costs are easy
> to calculate for S3. So far with over 2,000 gem downloads and
> uploading all of the gems from RubyForge over the past 2 months it's
> been less than $5.
>
Taking in consideration RubyForge has lot of mirrors, collecting real
life numbers for it would be really great.
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
Aaron Patterson
> On Wed, Aug 26, 2009 at 1:42 PM, Nick Quaranto<ni...@quaran.to> wrote:
> > Please note I'm not suggesting that all of these happen *this instant*, I
> > just want to open up discussion about making this happen and what would be
> > involved. Your thoughts and comments would be appreciated.
>
> To be honest, this seems Not Ready For Prime Time:
>
> * Pagination/sorting is wrong and broken
> * Project pages contain obviously broken/incorrect links (version and
> homepage links)
> * Project pages seem to have much less information than I have on
> RubyForge. No release notes, individual file links (.gem, .tar, etc),
> etc... (broken and less != transparent and accessible)
> * I uploaded a gem to RubyForge yesterday just before midnight, it is
> still not mirrored sixteen hours later (and it was available almost
> immediately on Rubyforge, which shows their mirror process is working
> very well)
>
> That's just what I found in one minute of poking around, it doesn't
> give me a good feeling for the overall quality and stability of the
> effort. I'm also concerned about long-term support. No offense to
> you, but Tom Copeland is a machine, he is always there to support
> Rubyforge, year after year...
>
> In general, if it Ain't Broken, Don't Fix It - and think especially
> hard if the replacement isn't a no-brainer out-of-the-park homerun...
I didn't think this was supposed to be a replacement for rubyforge, but
a replacement for the rubyforge gem server.
> Is there a reason we can't get many of these benefits from
> RubyForge.org? For example, your API seems like a good idea, but why
> can't it be a wrapper or enhancement to the current RubyGems/RubyForge
> ecosystem? A Gem plugin, even?
>
> Finally, I don't see why there needs to be a change to the canonical
> source, since Rubyforge isn't even really one anymore. That ship has
> already sailed, many people don't even publish gems to RubyForge, they
> just use Github or their own servers. If anything there needs to be
> better support for multiple sources (There are good ideas here not yet
> implemented). The multiple-naming problem is a bigger but related
> issue as Eric discussed
If people aren't willing to publish to the canonical source, it sounds like
something is Is Broken, so we Should Fix It.
--
Aaron Patterson
http://tenderlovemaking.com/
Ryan Davis
On Aug 26, 2009, at 17:16 , Luis Lavena wrote:
>> (possibly) even better, I'm pretty sure we could get Tom convinced
>> that
>> gemcutter is a good idea and we should migrate rubyforge to it.
>>
>
> The thing is that RubyForge is more than just gems.
I wasn't suggesting that gemcutter take over rubyforge, but that
rubyforge migrate to using gemcutter for gem services. With a little
bit of work, it wouldn't be hard to do.
> Taking in consideration there are lot of gems with more than 10K of
> downloads, the incurring charge of running that form S3 and adding
> CloudFront CDN on top will need to be payed my someone.
There is no incurred charge if S3 is not involved. That is one benefit
of having it brought under the wing of ruby central / rubyforge.
Tom Copeland
On Aug 26, 2009, at 5:57 PM, John Barnette wrote:
> Nick,
>
> On Aug 26, 2009, at 1:42 PM, Nick Quaranto wrote:
>> 1) Redirect gems.rubyforge.org to gemcutter.org for gem serving.
>> (all gems
>> are currently mirrored from RubyForge and are ready for consumption)
>
> RubyForge currently has a pretty good mirror system for supporting
> Gem downloads. I expect Tom could get your some figures on how much
> bandwidth they use.
I just ran webalizer on the Apache log from the Bytemark mirror and it
did 14M gems of gem traffic on Monday. Figure Bytemark serves about a
quarter of the traffic, that's about 60M/day. File downloads eat a
little more bandwidth - that Bytemark mirror did 49M yesterday of
files... it's serving about 60% of the traffic, so files account for
80M/day, maybe. And of course there are spikes here and there.
Yours,
Tom
Tom Copeland
On Aug 26, 2009, at 6:21 PM, Nick Quaranto wrote:
>
> As for the costs, I'd like to approach a non-profit organization such
> as RubyCentral to help deal with them. I originally sent an email out
> to them first about supporting the project a few weeks ago, but I
> haven't heard anything back. I'm interested to hear their opinions on
> the site as well.
It would definitely be good to get Chad or Rich or David to weigh in
since they're on the board of Ruby Central, and Ruby Central rides
herd on RubyForge.
Yours,
tom
Tom Copeland
On Aug 26, 2009, at 9:22 PM, Tom Copeland wrote:
>
> On Aug 26, 2009, at 5:57 PM, John Barnette wrote:
>
>> Nick,
>>
>> On Aug 26, 2009, at 1:42 PM, Nick Quaranto wrote:
>>> 1) Redirect gems.rubyforge.org to gemcutter.org for gem serving.
>>> (all gems
>>> are currently mirrored from RubyForge and are ready for consumption)
>>
>> RubyForge currently has a pretty good mirror system for supporting
>> Gem downloads. I expect Tom could get your some figures on how much
>> bandwidth they use.
>
> I just ran webalizer on the Apache log from the Bytemark mirror and
> it did 14M gems of gem traffic on Monday. Figure Bytemark serves
> about a quarter of the traffic, that's about 60M/day. File
> downloads eat a little more bandwidth - that Bytemark mirror did 49M
> yesterday of files... it's serving about 60% of the traffic, so
> files account for 80M/day, maybe. And of course there are spikes
> here and there.
Oh, also, space-wise there are 6GB of gems and 29G of files.
Yours,
tom
Nick Quaranto
14mb/day, is less than 1GB a month. For S3 that's less than $1/month just
for data transfer out, and a little more with CloudFront. It doesn't seem
like this would be a problem unless if we were in the range of
terabytes/month. As for hosting files on Gemcutter, I'm open to that if
that's what is necessary for the transition. 80 mb/day is still a trivial
amount for S3 or CloudFront to host, cost-wise.
If I interpreted your findings wrong Tom, let me know.
-Nick
Tom Copeland
On Aug 26, 2009, at 10:27 PM, Nick Quaranto wrote:
> So I'm trying out: http://calculator.s3.amazonaws.com/calc5.html
>
> 14mb/day, is less than 1GB a month. For S3 that's less than $1/month
> just
> for data transfer out, and a little more with CloudFront. It doesn't
> seem
> like this would be a problem unless if we were in the range of
> terabytes/month. As for hosting files on Gemcutter, I'm open to that
> if
> that's what is necessary for the transition. 80 mb/day is still a
> trivial
> amount for S3 or CloudFront to host, cost-wise.
>
> If I interpreted your findings wrong Tom, let me know.
Nah, that sounds right. Chad and I were actually talking today about
having RubyForge use S3 for serving gems and files...
Yours,
tom
Tom Copeland
On Aug 26, 2009, at 11:12 PM, Tom Copeland wrote:
>
> On Aug 26, 2009, at 10:27 PM, Nick Quaranto wrote:
>
>> So I'm trying out: http://calculator.s3.amazonaws.com/calc5.html
>>
>> 14mb/day, is less than 1GB a month. For S3 that's less than $1/
>> month just
>> for data transfer out, and a little more with CloudFront. It
>> doesn't seem
>> like this would be a problem unless if we were in the range of
>> terabytes/month. As for hosting files on Gemcutter, I'm open to
>> that if
>> that's what is necessary for the transition. 80 mb/day is still a
>> trivial
>> amount for S3 or CloudFront to host, cost-wise.
Oh, one more thing - serving up the gem indexes (yaml, Marshal.4.8,
Marshal.4.8.Z, etc) also accounts for 100 MB or so a day. The yaml
file is about 60MB of that. Anyhow, it's still a pretty small amount
S3-wise...
Nick Quaranto
Gemcutter. Since you need at least RubyGems 1.3.3 to use the gem plugins, I
figured supporting the legacy ones shouldn't be a priority as the project
started. Currently I'm only generating and serving up the modern indexes
(and these are undergoing some serious refactoring and rework internally as
well).
It would be interesting to watch the traffic of the legacy indexes vs the
modern indexes to see if it's even worth it to support those going forward.
At the very least, we could drop the legacy indexes into S3 and not update
them.
-Nick
On Wed, Aug 26, 2009 at 11:59 PM, Tom Copeland <t...@infoether.com> wrote:
>
> On Aug 26, 2009, at 11:12 PM, Tom Copeland wrote:
>
>
>> On Aug 26, 2009, at 10:27 PM, Nick Quaranto wrote:
>>
>> So I'm trying out: http://calculator.s3.amazonaws.com/calc5.html
>>>
>>> 14mb/day, is less than 1GB a month. For S3 that's less than $1/month just
Tom Copeland
On Aug 27, 2009, at 12:08 AM, Nick Quaranto wrote:
> Just a note here: I haven't added support for the legacy indexes for
> in
> Gemcutter. Since you need at least RubyGems 1.3.3 to use the gem
> plugins, I
> figured supporting the legacy ones shouldn't be a priority as the
> project
> started. Currently I'm only generating and serving up the modern
> indexes
> (and these are undergoing some serious refactoring and rework
> internally as
> well).
>
> It would be interesting to watch the traffic of the legacy indexes
> vs the
> modern indexes to see if it's even worth it to support those going
> forward.
> At the very least, we could drop the legacy indexes into S3 and not
> update
> them.
Here's the full stats for gems.rubyforge.org (not the mirrors, just
the main domain) for Wednesday:
http://infoether.com/~tom/gems.rubyforge.org.logs/usage_200908.html
6K hits on the old-timey yaml index. We only rebuild it once a day...
I should put it up on S3 and do a rewrite rule for just that one file...
Yours,
Tom
James Tucker
On 26 Aug 2009, at 23:21, Nick Quaranto wrote:
> I'll take a look into CloudFront, I just haven't yet since the need
> hasn't come up. My main concern is keeping the gap between publishing
> a gem and being able to install it low.
I really can't stress enough that it would be possible to integrate
these requirements with rubyforge. I'm not saying "no", I'm just
pointing out, that it could have been done just with the gem plugin
targeting rubyforge.
Ryan Davis
On Aug 26, 2009, at 21:08 , Nick Quaranto wrote:
> Just a note here: I haven't added support for the legacy indexes for
> in
> Gemcutter. Since you need at least RubyGems 1.3.3 to use the gem
> plugins, I
> figured supporting the legacy ones shouldn't be a priority as the
> project
> started. Currently I'm only generating and serving up the modern
> indexes
> (and these are undergoing some serious refactoring and rework
> internally as
> well).
Agreed. Cut and run.