Somewhat predictably, I feel strongly about the availability of the Bundler API. As a result of the recent downtime (due to load issues), I've come up with a plan that I think can increase the reliability of the Bundler API while removing the burden of maintaining its infrastructure from the Rubygems.org team.
I would like to create a new project that implements only the Bundler Dependency API. It would, at least initially, implement only the Rubygems dependency API that lives at `/api/v1/dependencies`. I think it would be feasible to keep it up to date with rubygems.org in near-realtime using the existing webhook system.
If desired, we could add a rubygems.org subdomain dedicated to this app, or we could host it at a separate domain. Either way, requests to rubygems.org could be redirected to the new service via 302 redirects. The redirects could even be served by the rubygems.org frontend server, so the Rails app would no longer have to respond to dependency requests.
Heroku has volunteered to provide hosting for this API app, but it should be feasible to host it anywhere. Long term, it could even be possible to host it on multiple services for higher availability.
I'll post here again once I have a working Bundler API hosted on Heroku, but getting this live for everyone to use will obviously require at least a little bit of ops work on rubygems.org. I'm happy to do that work myself, or work with someone who already has access to get things going.
On Thursday, October 18, 2012 at 12:27 PM, André Arko wrote:
> Somewhat predictably, I feel strongly about the availability of the Bundler API. As a result of the recent downtime (due to load issues), I've come up with a plan that I think can increase the reliability of the Bundler API while removing the burden of maintaining its infrastructure from the Rubygems.org (http://Rubygems.org) team. I would like to create a new project that implements only the Bundler Dependency API. It would, at least initially, implement only the Rubygems dependency API that lives at `/api/v1/dependencies`. I think it would be feasible to keep it up to date with rubygems.org (http://rubygems.org) in near-realtime using the existing webhook system. If desired, we could add a rubygems.org (http://rubygems.org) subdomain dedicated to this app, or we could host it at a separate domain. Either way, requests to rubygems.org (http://rubygems.org) could be redirected to the new service via 302 redirects. The redirects could even be served by the rubygems.org (http://rubygems.org) frontend server, so the Rails app would no longer have to respond to dependency requests. Heroku has volunteered to provide hosting for this API app, but it should be feasible to host it anywhere. Long term, it could even be possible to host it on multiple services for higher availability. I'll post here again once I have a working Bundler API hosted on Heroku, but getting this live for everyone to use will obviously require at least a little bit of ops work on rubygems.org (http://rubygems.org). I'm happy to do that work myself, or work with someone who already has access to get things going. Thanks, André
The web hook delay is a a minute or so at most, probably around the same time for the indexes to update and push to S3. I don't think the "delay" is reason enough to throw this proposal out. Making the APIs better (read: yanked gems get a web hook as well, a longstanding issue) is really the work involved here.
On Thursday, October 18, 2012 at 3:35 PM, Evan Phoenix wrote:
> If the webhook mechanism is used to propagate data, there will be a delay. This might confuse users.
> Thusly, we may have to look into adding a more explicit mechanism to pass data to the new app (something I'd planned on adding anyway).
> On Thursday, October 18, 2012 at 12:27 PM, André Arko wrote:
> > Somewhat predictably, I feel strongly about the availability of the Bundler API. As a result of the recent downtime (due to load issues), I've come up with a plan that I think can increase the reliability of the Bundler API while removing the burden of maintaining its infrastructure from the Rubygems.org (http://Rubygems.org) team. I would like to create a new project that implements only the Bundler Dependency API. It would, at least initially, implement only the Rubygems dependency API that lives at `/api/v1/dependencies`. I think it would be feasible to keep it up to date with rubygems.org (http://rubygems.org) in near-realtime using the existing webhook system. If desired, we could add a rubygems.org (http://rubygems.org) subdomain dedicated to this app, or we could host it at a separate domain. Either way, requests to rubygems.org (http://rubygems.org) could be redirected to the new service via 302 redirects. The redirects could even be served by the rubygems.org (http://rubygems.org) frontend server, so the Rails app would no longer have to respond to dependency requests. Heroku has volunteered to provide hosting for this API app, but it should be feasible to host it anywhere. Long term, it could even be possible to host it on multiple services for higher availability. I'll post here again once I have a working Bundler API hosted on Heroku, but getting this live for everyone to use will obviously require at least a little bit of ops work on rubygems.org (http://rubygems.org). I'm happy to do that work myself, or work with someone who already has access to get things going. Thanks, André
I'm happy to build rubygems.org as a federation of services running in different places. It would be trivial to add a separate web hook mechanism that is high priority that we use to push updates out, which should reduce the delay even further.
Additionally, I'd prefer that clients still hit rubygems.org/api/v1/dependencies and we have the frontends perform either a backend proxy or a 304 redirection to something like api-dep.rubygems.org which could be running on heroku. That gives us the flexibility to easily move things around without having to change any client configs.
On Thursday, October 18, 2012 at 12:40 PM, Nick Quaranto wrote:
> The web hook delay is a a minute or so at most, probably around the same time for the indexes to update and push to S3. I don't think the "delay" is reason enough to throw this proposal out. Making the APIs better (read: yanked gems get a web hook as well, a longstanding issue) is really the work involved here.
> -Nick
> On Thursday, October 18, 2012 at 3:35 PM, Evan Phoenix wrote:
> > If the webhook mechanism is used to propagate data, there will be a delay. This might confuse users.
> > Thusly, we may have to look into adding a more explicit mechanism to pass data to the new app (something I'd planned on adding anyway).
> > On Thursday, October 18, 2012 at 12:27 PM, André Arko wrote:
> > > Somewhat predictably, I feel strongly about the availability of the Bundler API. As a result of the recent downtime (due to load issues), I've come up with a plan that I think can increase the reliability of the Bundler API while removing the burden of maintaining its infrastructure from the Rubygems.org (http://Rubygems.org) team. I would like to create a new project that implements only the Bundler Dependency API. It would, at least initially, implement only the Rubygems dependency API that lives at `/api/v1/dependencies`. I think it would be feasible to keep it up to date with rubygems.org (http://rubygems.org) in near-realtime using the existing webhook system. If desired, we could add a rubygems.org (http://rubygems.org) subdomain dedicated to this app, or we could host it at a separate domain. Either way, requests to rubygems.org (http://rubygems.org) could be redirected to the new service via 302 redirects. The redirects could even be served by the rubygems.org (http://rubygems.org) frontend server, so the Rails app would no longer have to respond to dependency requests. Heroku has volunteered to provide hosting for this API app, but it should be feasible to host it anywhere. Long term, it could even be possible to host it on multiple services for higher availability. I'll post here again once I have a working Bundler API hosted on Heroku, but getting this live for everyone to use will obviously require at least a little bit of ops work on rubygems.org (http://rubygems.org). I'm happy to do that work myself, or work with someone who already has access to get things going. Thanks, André
On Friday, October 19, 2012 at 1:35 PM, Evan Phoenix wrote:
> I'm happy to build rubygems.org (http://rubygems.org) as a federation of services running in different places. It would be trivial to add a separate web hook mechanism that is high priority that we use to push updates out, which should reduce the delay even further.
> Additionally, I'd prefer that clients still hit rubygems.org/api/v1/dependencies (http://rubygems.org/api/v1/dependencies) and we have the frontends perform either a backend proxy or a 304 redirection to something like api-dep.rubygems.org (http://api-dep.rubygems.org) which could be running on heroku. That gives us the flexibility to easily move things around without having to change any client configs.
> On Thursday, October 18, 2012 at 12:40 PM, Nick Quaranto wrote:
> > The web hook delay is a a minute or so at most, probably around the same time for the indexes to update and push to S3. I don't think the "delay" is reason enough to throw this proposal out. Making the APIs better (read: yanked gems get a web hook as well, a longstanding issue) is really the work involved here.
> > -Nick
> > On Thursday, October 18, 2012 at 3:35 PM, Evan Phoenix wrote:
> > > If the webhook mechanism is used to propagate data, there will be a delay. This might confuse users.
> > > Thusly, we may have to look into adding a more explicit mechanism to pass data to the new app (something I'd planned on adding anyway).
> > > On Thursday, October 18, 2012 at 12:27 PM, André Arko wrote:
> > > > Somewhat predictably, I feel strongly about the availability of the Bundler API. As a result of the recent downtime (due to load issues), I've come up with a plan that I think can increase the reliability of the Bundler API while removing the burden of maintaining its infrastructure from the Rubygems.org (http://Rubygems.org) team. I would like to create a new project that implements only the Bundler Dependency API. It would, at least initially, implement only the Rubygems dependency API that lives at `/api/v1/dependencies`. I think it would be feasible to keep it up to date with rubygems.org (http://rubygems.org) in near-realtime using the existing webhook system. If desired, we could add a rubygems.org (http://rubygems.org) subdomain dedicated to this app, or we could host it at a separate domain. Either way, requests to rubygems.org (http://rubygems.org) could be redirected to the new service via 302 redirects. The redirects could even be served by the rubygems.org (http://rubygems.org) frontend server, so the Rails app would no longer have to respond to dependency requests. Heroku has volunteered to provide hosting for this API app, but it should be feasible to host it anywhere. Long term, it could even be possible to host it on multiple services for higher availability. I'll post here again once I have a working Bundler API hosted on Heroku, but getting this live for everyone to use will obviously require at least a little bit of ops work on rubygems.org (http://rubygems.org). I'm happy to do that work myself, or work with someone who already has access to get things going. Thanks, André
On Fri, Oct 19, 2012 at 12:35 PM, Evan Phoenix <e...@phx.io> wrote:
> Additionally, I'd prefer that clients still hit
> rubygems.org/api/v1/dependencies and we have the frontends perform either a
> backend proxy or a 304 redirection to something like api-dep.rubygems.org
> which could be running on heroku. That gives us the flexibility to easily
> move things around without having to change any client configs.
I think it would be pretty hefty to redirect people to a new URI or
URL because many people don't account for that in their code (that is
if their code requires they manually handle redirects like I've had to
with things like Net::HTTP when building my own wrappers.) Though at
the same time, it's not much effort to account for those types of
things.
On Friday, October 19, 2012 at 10:50 AM, Jordon Bedwell wrote:
> On Fri, Oct 19, 2012 at 12:35 PM, Evan Phoenix <e...@phx.io (mailto:e...@phx.io)> wrote:
> > Additionally, I'd prefer that clients still hit
> > rubygems.org/api/v1/dependencies (http://rubygems.org/api/v1/dependencies) and we have the frontends perform either a
> > backend proxy or a 304 redirection to something like api-dep.rubygems.org (http://api-dep.rubygems.org)
> > which could be running on heroku. That gives us the flexibility to easily
> > move things around without having to change any client configs.
> I think it would be pretty hefty to redirect people to a new URI or
> URL because many people don't account for that in their code (that is
> if their code requires they manually handle redirects like I've had to
> with things like Net::HTTP when building my own wrappers.) Though at
> the same time, it's not much effort to account for those types of
> things.
Rubygems and Bundler handle it properly. I don't think that there are really an users of the API outside of those 2 code bases, so doing a redirect should be fine. We'd basically need to communicate that anyone using rubygems.org resources need to accommodate redirects.
Thanks for the update! Do you have any ideas or details on the web hook? I guess we'd need both new gems as well as yanks. I'm currently working on an update mechanism to keep our copy of pg to date to match the index. Can we look into a polling solution as well? What would we do if the push fails or the receiver goes down?
> On Friday, October 19, 2012 at 10:50 AM, Jordon Bedwell wrote:
> On Fri, Oct 19, 2012 at 12:35 PM, Evan Phoenix <ev...@phx.io <javascript:>> > wrote:
> Additionally, I'd prefer that clients still hit > rubygems.org/api/v1/dependencies and we have the frontends perform either > a > backend proxy or a 304 redirection to something like api-dep.rubygems.org > which could be running on heroku. That gives us the flexibility to easily > move things around without having to change any client configs.
> I think it would be pretty hefty to redirect people to a new URI or > URL because many people don't account for that in their code (that is > if their code requires they manually handle redirects like I've had to > with things like Net::HTTP when building my own wrappers.) Though at > the same time, it's not much effort to account for those types of > things.
> Rubygems and Bundler handle it properly. I don't think that there are > really an users of the API outside of those 2 code bases, so doing a > redirect should be fine. We'd basically need to communicate that anyone > using rubygems.org resources need to accommodate redirects.
