ANN An update on recent Bintray account disruptions

[Michael Klishin]

Dear RabbitMQ community,

As some of you may have noticed, our Bintray organization has been rate limited twice

in the last week. This is an update with the latest news and how our team plans to avoid

such disruptions in the future.

## Why is RabbitMQ's Org Rate Limited?

We've been heavy users of Bintray in the last year. We host all kinds of packages

there, and some of them are relatively large. Bintray has certain limits open source

accounts cannot hit. When that happens, the entire org is rate limited.

Next let's clarify what "rate limited" means on Bintray. Unfortunately it's not throttling:

a rate limited account is more or less blocked: no new versions can be published (makes sense)

and no artifacts can be downloaded. The latter part is very disruptive to the users

who whose to use Bintray.

Our org uses the free plan for open source projects and has been blocked before,

primarily for running over the storage quota. We reduced our usage by only keeping

a few most recent development builds on Bintray. That helped for a period of time

but as the number of sub-projects and our general usage of Bintray both grew, we

eventually started hitting the limit again. That was the first blocking incident last week.

The second one was to do with "API rate limiting". It's an explanation provided by

Bintray and we don't know what that really means. Likely some of our packages

experienced a higher than usual download rate. Bintray has since increased the

API rate limit for our account, which our team appreciates.

## Why is this a Recurring Event?

Since we have a reasonably detailed understanding of what kind of limits we hit,

how come the issue's been recurring? The answer is: Bintray provides no way for

open source accounts to monitor their quota. No API endpoints we could use, no

UI indicator. This feature is available on the Enterprise plan which

is the most expensive plan Bintray has and a complete overkill for RabbitMQ.

Recently we managed to convince some members of the team at JFrog/Bintray to make those monitoring

features available for open source accounts. We cannot know if or when it's going

to ship and the issue is very pressing: we've been promoting Bintray as the best

option for direct downloads as well as Debian and Yum repositories.

After all, it's a nice service that offers a CDN, a decent API and supports every

package type we need to distribute, all for free if your project is open source.

So there has to be a plan that involves more than "just wait and add monitoring".

## Our Plan for Bintray

Our team has agreed that the only short term solution is to reduce our use of Bintray.

A reasonable % of our users consume our releases from Bintray, it's a nice

service in many ways and moving away from it would take more time than our

team can spare on infrastructure. We chose Bintray to not have to worry about

infrastructure to begin with!

So besides working with Bintray to introduce quota monitoring we will promote

GitHub as the primary source of direct downloads; GitHub doesn't provide certain features

Bintray has for artifact distribution but it also doesn't block downloads of our stuff.

For Debian and RPM repositories, we will make sure that everything that's available from Bintray

is also available from Package Cloud. Debian packages of Erlang currently aren't.

Bintray will continue being a first class citizen when it comes to Debian and RPM packages.

We will research alternatives that are not Package Cloud (since we would like to not be

overly dependent on a single service).

To reduce the chances of hitting Bintray's storage quota before monitoring is in place

we'll stop distributing some less popular but large packages to Bintray:

 * The MacOS standalone package

 * The source tarball

 * Windows packages as ZIP archives (.tar.xz files can be used on Windows just fine these days)

## What You Can Do Today

As a RabbitMQ user who uses Bintray, you can do a few things today:

 * If you use direct downloads, switch to GitHub and you are done

 * If you use Debian or RPM packages, investigate if Package Cloud would work for you; their ways of

   repository setup are a bit unorthodox compared to Bintray but otherwise they are very much

   functionally equivalent.

We'll post future updates to this thread.

Cheers.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ