Re: [gemcutter] Proposal: Extract Bundler API from Rubygems.org

[Evan Phoenix]

If the webhook mechanism is used to propagate data, there will be a delay. This might confuse users.

Thusly, we may have to look into adding a more explicit mechanism to pass data to the new app (something I'd planned on adding anyway).

 - Evan

-- 
Evan Phoenix // ev...@phx.io

On Thursday, October 18, 2012 at 12:27 PM, André Arko wrote:

Somewhat predictably, I feel strongly about the availability of the Bundler API. As a result of the recent downtime (due to load issues), I've come up with a plan that I think can increase the reliability of the Bundler API while removing the burden of maintaining its infrastructure from the Rubygems.org team.

I would like to create a new project that implements only the Bundler Dependency API. It would, at least initially, implement only the Rubygems dependency API that lives at `/api/v1/dependencies`. I think it would be feasible to keep it up to date with rubygems.org in near-realtime using the existing webhook system.

If desired, we could add a rubygems.org subdomain dedicated to this app, or we could host it at a separate domain. Either way, requests to rubygems.org could be redirected to the new service via 302 redirects. The redirects could even be served by the rubygems.org frontend server, so the Rails app would no longer have to respond to dependency requests.

Heroku has volunteered to provide hosting for this API app, but it should be feasible to host it anywhere. Long term, it could even be possible to host it on multiple services for higher availability.

I'll post here again once I have a working Bundler API hosted on Heroku, but getting this live for everyone to use will obviously require at least a little bit of ops work on rubygems.org. I'm happy to do that work myself, or work with someone who already has access to get things going.

Thanks,
André

[Nick Quaranto]

The web hook delay is a a minute or so at most, probably around the same time for the indexes to update and push to S3. I don't think the "delay" is reason enough to throw this proposal out. Making the APIs better (read: yanked gems get a web hook as well, a longstanding issue) is really the work involved here.

-Nick

[Evan Phoenix]

I'm happy to build rubygems.org as a federation of services running in different places. It would be trivial to add a separate web hook mechanism that is high priority that we use to push updates out, which should reduce the delay even further.

Additionally, I'd prefer that clients still hit rubygems.org/api/v1/dependencies and we have the frontends perform either a backend proxy or a 304 redirection to something like api-dep.rubygems.org which could be running on heroku. That gives us the flexibility to easily move things around without having to change any client configs.

-- 
Evan Phoenix // ev...@phx.io

[Nick Quaranto]

Hurray! Huge huge +1. Let's talk this over in the pow-wow.

-Nick

[Jordon Bedwell]

On Fri, Oct 19, 2012 at 12:35 PM, Evan Phoenix <ev...@phx.io> wrote:
> Additionally, I'd prefer that clients still hit
> rubygems.org/api/v1/dependencies and we have the frontends perform either a
> backend proxy or a 304 redirection to something like api-dep.rubygems.org
> which could be running on heroku. That gives us the flexibility to easily
> move things around without having to change any client configs.

I think it would be pretty hefty to redirect people to a new URI or
URL because many people don't account for that in their code (that is
if their code requires they manually handle redirects like I've had to
with things like Net::HTTP when building my own wrappers.) Though at
the same time, it's not much effort to account for those types of
things.

[Evan Phoenix]

See below.

-- 
Evan Phoenix // ev...@phx.io

Rubygems and Bundler handle it properly. I don't think that there are really an users of the API outside of those 2 code bases, so doing a redirect should be fine. We'd basically need to communicate that anyone using rubygems.org resources need to accommodate redirects.