Upcoming changes to Clojars

[Toby Crawley]

Howdy folks! We're in the process of making some changes to the
Clojars infrastructure to make it more robust, reliable, and easier to
maintain. One of those changes will be to provide a single canonical
repository instead of serving artifacts from two locations.


## A Tale of Two Repositories

You can currently access the Clojars repository in two ways:

- via the "on-disk" repository at https://clojars.org/repo/
- via the "cdn-fronted" repository at https://repo.clojars.org/

The artifacts for these two repositories are stored in different
places. The on-disk repository serves from the disk of the clojars.org
server. The cdn-fronted repository serves from a Rackspace Cloudfiles
repository. When a deploy occurs, it is first written to disk, then
uploaded to Cloudfiles. The intent was for the repositoriesitories to
be identical, but that hasn't always been the case - we would
occasionally fail to upload artifacts to the Cloudfiles repository,
leaving the two repositories out of sync. We've done recent work[1] to
mitigate that, but would like to move to having a single,
Cloudfiles/blobstore-based repository.


## The proposed changes

In order to have a single repository, we plan to make the following
changes:

- Any GET requests to the on-disk repository at
https://clojars.org/repo/<path> will be redirected to the
cdn-fronted repository at https://repo.clojars.org/<path>
- Deploys will no longer write to the on-disk repository, just to the
cdn-fronted one


## Timeline

We would like to make these changes within the next few weeks, but may
delay that based on your feedback (see below).


## How this will affect you

If you are using a modern version of lein (2.8.0 or newer), boot
(2.7.0 or newer), or tools.deps.alpha (0.5.425 or newer), you are
already using the cdn-fronted repository, and should see no issues
with dependency resolution or deployment.

The cdn-fronted repository uses some TLS features that aren't
supported by older java versions by default. If you are using an older
version of Java (pre-Java 8), you may be using older build tools or
have configured newer versions of the tools to point to the on-disk
repository to avoid this. With this change, you may no longer be able
to resolve dependencies. If this affects you, we would love to hear
about it! Please comment on the issue where this change is being
discussed[2].

Once we remove the on-disk repository, rsync'ing a copy of the
repository will no longer be a supported feature. Based on our rsync
logs, it looks like we've had ~100k rsync connections over the last
month from ~100 ip addresses. But the bulk of those requests were over
a few scattered days where a single ip would connect every 2-5
*seconds*. If you are using the rsync feature to maintain a copy of
the repository, please comment on the issue where this change is being
discussed[3] to let us know your use case and we'll see if we can come
up with an alternate solution.

[1]: https://groups.google.com/forum/#!topic/clojars-maintainers/FXcV6O-FjiA
[2]: https://github.com/clojars/clojars-web/issues/734
[3]: https://github.com/clojars/clojars-web/issues/735

[Sean Corfield]

Will the upload URL or process be changing?

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/CAA3HuyZtb1jOmNgYA%2BcLpvcxk-Yce74G4KRGKctbA6xuc-x19w%40mail.gmail.com.

--

Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
World Singles Networks, LLC. -- https://worldsinglesnetworks.com/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

[Toby Crawley]

Thanks for asking Sean. The upload URL and process will *not* be
changing. This change should have no impact for 99% of Clojars users.

- Toby

> To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/CAD4thx_8VczEZP8ExJCRNs0A%2BXsEqu-PMJYQAiF86U07qWZ%2B5A%40mail.gmail.com.

[Toby Crawley]

We haven't gotten any objections to these changes to Clojars, so have
implemented them.

The changes are:
- rsyncing of the Clojars repo is now disabled (see
https://github.com/clojars/clojars-web/issues/735)
- GET or HEAD requests to the on-disk repo will now redirect to the
CDN-fronted repo (see
https://github.com/clojars/clojars-web/issues/734). This means that
GET https://clojars.org/repo/whatever will return a 302 Found status
code with a location header pointing to
https://repo.clojars.org/whatever. We'll change that to a 301 Moved
Permanently once this change has had time to settle and we don't have
any issues reported that causes us to revert the change. Resolving
dependencies should continue to work, since aether (the lib used by
boot/lein/maven/tools.deps to resolve dependencies) should follow
redirects correctly. Deploying to Clojars should be unaffected by this
change - you can still deploy to https://clojars.org/repo/ or
https://repo.clojars.org.

If you (or someone you love) is negatively affected by either of these
changes, please let us know by commenting on one of the above issues.

Thanks!
- Toby