[ANN] Clojars Releases repository

[Phil Hagelberg]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hello folks.

I'm happy to announce that the new Clojars releases repository is open
for business.

With the releases repository we are aiming for a middle ground in
between the anything-goes nature of the current repository and the
bureaucracy of the Central repository. There are a few different things
motivating this:

* You should be able to pull in stable versions of Clojure repositories
without adding a snapshots repo to your configuration since this has
adverse affects upon dependency resolution time and can introduce
unexpected results in the context of version ranges.

* We want to ensure that the new repository doesn't contain anything
that's missing crucial metadata such as the project's URL,
description, or license.

* It needs to be possible to verify that your dependencies were
published by a trusted source, so the releases repository only
accepts artifacts that have been properly signed.

What does this mean for you? If you have been deploying your libraries
with a Leiningen 2 preview, there isn't much to do--Leiningen will warn
you if you are missing the necessary metadata, and it will sign releases
before sending them to Clojars. All that's necessary is for you to log
into clojars.org and paste your public key in the "PGP Public Key" field
in your profile.

If you don't have a key yet, generate one with `gpg --gen-key`. The
default settings are pretty good, though I'd recommend making it expire
in a year or two. Next find your key ID. It's the 8-character part after
the slash on the line beginning with "pub":

$ gpg --list-keys

↓↓↓↓↓↓↓↓
pub 2048R/77E77DDC 2011-07-17 [expires: 2014-07-16]
uid Phil Hagelberg <techn...@gmail.com>
sub 2048R/39EFEE7D 2011-07-17

Then you can show it with `gpg --export -a $KEY_ID`. Grab that
(including the "-----BEGIN PGP PUBLIC KEY BLOCK-----" parts) and paste
it into your Clojars profile.

Once you have done this you can redeploy to trigger promotion to the
releases repo if your jar is qualified, or you can visit the jar page in
the Clojars web UI (while logged in) to see if there are reasons it's
not qualified. Note that deployments actually go to the classic
repository originally, so `lein deploy clojars` should do the trick. If
something is amiss here please let us know either on this thread or in
the #leiningen channel on freenode; it hasn't been tested widely yet and
is still fairly new code.

The Releases repository is the final missing piece of the puzzle for a
final release of Leiningen 2. But the time isn't yet right because
version 2 will only check Central and the Clojars Releases repo by
default. So since the new Releases repo only has a handful of jars, it
would be a jarring transition to switch at this point. That's why we're
hoping library maintainers can do what's necessary to ensure their
libraries make it into the new repository.

Thanks!

- -Phil
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQEcBAEBAgAGBQJQqOjnAAoJEK9We5d3533cUEoH/2UC4n+RlbMCgP5J/Aj9JwXN
Z7yWHP/tHqcqCUoawY0B8cdYtMjgCLzjZ0pLkQ+TnAiWHucS5O3D2MHuPLxxZ1L/
DWDppR4iHkDUp81KQWxd57FWo7dseYkMmfOyLiuO3Ma2KvklP1Ue2wUdMvFo8UN1
b3fPF+1SnQNCIQA9k8rof4NFD1FlyhEvRflFgj0vDyy9Of80OUcEYPXVseNosV5Y
oZ1ELTkkCPSVabf/NwEZdN77xxA89uU5k2HkX5uA+/2yGlN7NHgcpud5AyHexUta
iRgC1taRcp+LEtRzY2ACMyIx4Tt1j6es1byIVbj7kMkjEPIyRXon/Synpe8LgkY=
=GMbM
-----END PGP SIGNATURE-----

[Jim - FooBar();]

Followed the instructions below exactly but clojars says 'Invalid PGP
public key'...

any clues?

Jim

[Nelson Morris]

I've just deployed a new clojars version. The previous one was a bit
strict on the whitespace (thanks Lee Hinman).

Make sure to include both the "-----BEGIN PGP PUBLIC KEY BLOCK-----"
and "-----END PGP PUBLIC KEY BLOCK-----".

-
Nelson Morris

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

[Jim - FooBar();]

On 18/11/12 14:39, Nelson Morris wrote:

The previous one was a bit
strict on the whitespace

I just pasted the same with no wxtra white-space and now I'm getting

Invalid anti-forgery token

my god what is happening?

Jim

[Jim - FooBar();]

Ok I managed to push my jar successfully, but i got this at the end:

Could not transfer artifact enclog:enclog:pom:0.5.8 from/to clojars (https://clojars.org/repo/): Access denied to: https://clojars.org/repo/enclog/enclog/0.5.8/enclog-0.5.8.pom, ReasonPhrase:Forbidden.
Failed to deploy artifacts: Could not transfer artifact enclog:enclog:pom:0.5.8 from/to clojars (https://clojars.org/repo/): Access denied to: https://clojars.org/repo/enclog/enclog/0.5.8/enclog-0.5.8.pom, ReasonPhrase:Forbidden.

Is this important?

Jim

[Nelson Morris]

The "Invalid anti-forgery token" message is a unfortunate side effect
of interaction with sessions and restarting the server. It should
disappear if the profile page is refreshed.

enclog 0.5.8 appears in the releases repo, so everything is ok. I
have a theory as to why that message occurred and will see what I can
track down for the future. Unfortunately, I'd expect a possibility of
this occurring for any redeployment of artifacts with signatures
already in the "classic" repo.

Thanks for signing and feedback about the issues.

[Jim - FooBar();]

Yes that's obviously important because now I can't fetch the jar!  Doing lein2 repl in a project that depends on that jar gives me:

Could not transfer artifact enclog:enclog:pom:0.5.8 from/to clojars (https://clojars.org/repo/): Checksum validation failed, no checksums available from the repository
Check :dependencies and :repositories for typos.
It's possible the specified jar is not in any repository.
If so, see "Free-floating Jars" under http://j.mp/repeatability
...
...
However clojars says that the jar is promoted...

Can anyone shine some light?

Jim

[Jim - FooBar();]

On 18/11/12 15:14, Nelson Morris wrote:
> enclog 0.5.8 appears in the releases repo, so everything is ok.

No, unfortunately everything is not ok...fetching the jar from a project
results in:

Could not transfer artifact enclog:enclog:pom:0.5.8 from/to clojars

(https://clojars.org/repo/): Checksum validation failed, no checksums
available from the repository
Check :dependencies and :repositories for typos.
It's possible the specified jar is not in any repository.
If so, see "Free-floating Jars" under http://j.mp/repeatability

etc etc (exceptions)

Jim

[Nelson Morris]

Yeah, i had checked the releases not expected the "classic" repo to
loose it. Fixed manually.

[Sean Corfield]

On Sun, Nov 18, 2012 at 5:56 AM, Phil Hagelberg <ph...@hagelb.org> wrote:

If you don't have a key yet, generate one with `gpg --gen-key`. The
default settings are pretty good, though I'd recommend making it expire
in a year or two. Next find your key ID. It's the 8-character part after
the slash on the line beginning with "pub":

As I said at the conj, I'm looking forward to the documentation explaining how to install and use gpg since it's not provided by default on either Mac OS X or Windows.

Then you can show it with `gpg --export -a $KEY_ID`.

$KEY_ID? (again, as I noted at the conj, without good documentation on the Leiningen site for this, folks won't necessarily know what this is or why they need to do all of this, especially the web of trust stuff you discussed and key exchanges / publishing etc).

The Releases repository is the final missing piece of the puzzle for a
final release of Leiningen 2. But the time isn't yet right because
version 2 will only check Central and the Clojars Releases repo by
default. So since the new Releases repo only has a handful of jars, it
would be a jarring transition to switch at this point. That's why we're
hoping library maintainers can do what's necessary to ensure their
libraries make it into the new repository.

So if the status quo persists and Mac and Windows users don't bother to install gpg, the Clojars process will stay exactly as it is? In other words, we can simply ignore the whole gpg issue and continue with things just as we do today and it won't break? Will users of Clojars projects be required to install and use gpg?

(I'm not arguing against encryption or signing - just trying to a) point out that I think the vast majority of Clojure library developers probably don't have gpg installed and b) establish what is _required_ vs _optional_ and figure out what your plans are regarding existing Clojars projects and users)

-- 

Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
World Singles, LLC. -- http://worldsingles.com/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

[Phil Hagelberg]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sean Corfield <seanco...@gmail.com> writes:

> Then you can show it with `gpg --export -a $KEY_ID`.
>
>
> $KEY_ID? (again, as I noted at the conj, without good documentation on
> the Leiningen site for this, folks won't necessarily know what this is

> or why they need to do all of this)

Perhaps it would be helpful if you could explain in more detail what it
is about the provided explanation that you found confusing?

> > If you don't have a key yet, generate one with `gpg --gen-key`. The
> > default settings are pretty good, though I'd recommend making it expire
> > in a year or two. Next find your key ID. It's the 8-character part after
> > the slash on the line beginning with "pub":
> >

> > $ gpg --list-keys
> >
> > ↓↓↓↓↓↓↓↓
> > pub 2048R/77E77DDC 2011-07-17 [expires: 2014-07-16]
> > uid Phil Hagelberg <techn...@gmail.com>
> > sub 2048R/39EFEE7D 2011-07-17

> So if the status quo persists and Mac and Windows users don't bother
> to install gpg, the Clojars process will stay exactly as it is? In
> other words, we can simply ignore the whole gpg issue and continue
> with things just as we do today and it won't break? Will users of
> Clojars projects be required to install and use gpg?

If you turn off :sign-releases inside your :repositories entry when
deploying libraries everything will work for you as before. But your
libraries won't qualify for the Releases repo in this case. So once your
users upgrade to Leiningen 2.0.0 they will have to include a separate
:repositories entry for the classic repo to indicate that they are OK
with pulling in dependencies that don't meet the higher standards of the
new repo.

> (I'm not arguing against encryption or signing - just trying to a)
> point out that I think the vast majority of Clojure library developers
> probably don't have gpg installed and b) establish what is _required_
> vs _optional_ and figure out what your plans are regarding existing
> Clojars projects and users)

Indeed, the root problem is this notion that you can be a professional
software developer and remain ignorant of how public-key crypto works.
So collecting improved documentation and educational resources is going
to need to be a priority. I'll do what I can to put together good general
resources but will need help covering systems like Windows and OS X that
make things more difficult.

But I should emphasize that signing is only necessary for library
authors, and verifying the signatures will always be optional.

- -Phil
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQEcBAEBAgAGBQJQqnGSAAoJEK9We5d3533cMtEH/jJYEjkhymnB2sz6eWP5C5Wy
k5E6SXXSoOOyPPMYHZPsW9DedHRFpNi7bhZ8zYXlioKrRuKPEgUjjbM/oj9FB4oo
akJrf1cbR/rG9AoQn2aYiZAVVFQyWPCbieqdZRYyf6toxVAaDi3OJ3iDRX89NZvf
FPo/LbruJq32MJWNXo4PqZ9dq01K0Cs2ljCt9WLgzf/niKrwSi8tFC43NVH4k26t
fjH7UxHq6k8xs5tFpyXl4xZkc5rzoa85sRJE799R4+NA7IKoSseGSCrT6g0Ev6oy
IO0q4bz9Rc8Je9JZ5IV7Jpd4+kLp67cTCuXXnsqlIG9srkrDH4Q1VLH0Fp8JWpA=
=Ma81
-----END PGP SIGNATURE-----

[Jeff Heon]

As a starting point, the gpg website features native installers for both Windows and Mac OS.

http://www.gnupg.org

[Sean Corfield]

On Mon, Nov 19, 2012 at 9:51 AM, Phil Hagelberg <ph...@hagelb.org> wrote:

Perhaps it would be helpful if you could explain in more detail what it
is about the provided explanation that you found confusing?

In the first step you use an actual example, then switch to $KEY_ID without explanation, instead of again showing an actual example. At the conj, you just put up slides without any indication of what $KEY_ID was or where it could be found.

If you turn off :sign-releases inside your :repositories entry when
deploying libraries everything will work for you as before. But your
libraries won't qualify for the Releases repo in this case. So once your
users upgrade to Leiningen 2.0.0 they will have to include a separate
:repositories entry for the classic repo to indicate that they are OK
with pulling in dependencies that don't meet the higher standards of the
new repo.

So the choices are:

* follow the signing path (install and learn gpg etc), users don't need to do anything

* ignore the signing path, Leiningen will refuse to upload your libraries?

* explicitly turn off signing, users will be forced to change project.clj

Which means this isn't really an optional change: Leiningen is forcing signing on the community.

Again, I'm not arguing against it, I just want to be clear about whether we have a status quo option (we don't) so we must change.

Indeed, the root problem is this notion that you can be a professional

software developer and remain ignorant of how public-key crypto works.

Are you saying that all those people who don't have gpg or similar installed are unprofessional? It seems that such a statement would insult a very large number of software developers.

So collecting improved documentation and educational resources is going
to need to be a priority. I'll do what I can to put together good general
resources but will need help covering systems like Windows and OS X that
make things more difficult.

Perhaps you could run Windows and OS X in VMs on your Linux machine so you can experience what it is like and write about it from the perspective of a newbie on those OSes? The Windows experience for Clojure is already sub-par compared to OS X and Linux (although it has improved over time) and this is another Linux-centric change. OS X has been sufficiently Linux-y in the past to have escaped change but now is also on the other side of this particular fence. Have you considered adding keygen to Leiningen so that it can bridge that divide, as it does for every other aspect of the project automation process? (well, barring the initial curl/wget issue on Windows which can be mitigated by downloading the JAR directly)

-- 

[Lee Hinman]

Jeff Heon writes:

> As a starting point, the gpg website features native installers for both Windows and Mac OS.
>
> http://www.gnupg.org

And for OSX:

https://www.gpgtools.org/

; Lee

[jamii]

I now have the same problem as Jim (with https://clojars.org/strucjure/versions/0.3.2). I do have a gpg key set up and previous deploys claimed to be signing (although they are list as unsigned now).

What was the fix in this case?

[Nurullah Akkaya]

Is there a particular reason for not using Bouncy Castle[1]. Instead of forcing users to install gpg lein can generate and/or upload the key.

[1] http://www.bouncycastle.org/

--
Nurullah Akkaya
http://nakkaya.com

[Phil Hagelberg]

Nurullah Akkaya <nuru...@nakkaya.com> writes:

> Is there a particular reason for not using Bouncy Castle[1]. Instead
> of forcing users to install gpg lein can generate and/or upload the
> key.

Yeah, we intended to use that originally, but Bouncy Castle's PGP
support is awful beyond words. It's effectively undocumented, and the
classes it exposes really only make sense if you have the OpenPGP RFC
memorized.

-Phil

[Sean Corfield]

On Mon, Nov 19, 2012 at 2:28 PM, Phil Hagelberg <ph...@hagelb.org> wrote:

Yeah, we intended to use that originally, but Bouncy Castle's PGP

support is awful beyond words. It's effectively undocumented, and the
classes it exposes really only make sense if you have the OpenPGP RFC
memorized.

Ugh! :( And there are no other reasonable options?

-- 

[Sean Corfield]

FWIW, after setting up a public key etc and using lein deploy clojars to push congomongo 0.3.3 (successfully with one key), I am also getting the error about transferring the POM:

Sending congomongo/congomongo/0.3.3/congomongo-0.3.3.pom.asc (1k)

    to https://clojars.org/repo/

Sending congomongo/congomongo/0.3.3/congomongo-0.3.3.jar.asc (1k)

    to https://clojars.org/repo/

Sending congomongo/congomongo/0.3.3/congomongo-0.3.3.jar (15k)

    to https://clojars.org/repo/

Sending congomongo/congomongo/0.3.3/congomongo-0.3.3.pom (3k)

    to https://clojars.org/repo/

Could not transfer artifact congomongo:congomongo:pom:0.3.3 from/to clojars (https://clojars.org/repo/): Access denied to: https://clojars.org/repo/congomongo/congomongo/0.3.3/congomongo-0.3.3.pom, ReasonPhrase:Forbidden.

Failed to deploy artifacts: Could not transfer artifact congomongo:congomongo:pom:0.3.3 from/to clojars (https://clojars.org/repo/): Access denied to: https://clojars.org/repo/congomongo/congomongo/0.3.3/congomongo-0.3.3.pom, ReasonPhrase:Forbidden.

I still seem to be able to pull the library into a project and Clojars says it has been promoted (after my first successful try - with a different key / user ID).

[Phil Hagelberg]

Sean Corfield <seanco...@gmail.com> writes:

> Are you saying that all those people who don't have gpg or similar
> installed are unprofessional? It seems that such a statement would
> insult a very large number of software developers.

It's one thing to not have gotten around to learning something; it's
another thing entirely to ignore a technology and not have any intention
of getting familiar with it. Someone who writes software for a living
without understanding how to securely share secrets over email *and is
perfectly happy with that fact* is doing something wrong.

> Perhaps you could run Windows and OS X in VMs on your Linux machine so
> you can experience what it is like and write about it from the
> perspective of a newbie on those OSes?

That's actually illegal to do with OS X. Anyway, the problem with
Windows isn't that we don't know what's broken; it's that nobody with
the skills to fix it has volunteered to help.

> Have you considered adding keygen to Leiningen so that it can bridge
> that divide, as it does for every other aspect of the project
> automation process?

I tried to do this; see my response to Nurullah Akkaya. I wish it were
feasible, but it is not.

-Phil

[Sean Corfield]

On Mon, Nov 19, 2012 at 10:32 PM, Phil Hagelberg <ph...@hagelb.org> wrote:

Someone who writes software for a living

without understanding how to securely share secrets over email *and is
perfectly happy with that fact* is doing something wrong.

Thanx for that clarification :)

 

That's actually illegal to do with OS X.

They still don't allow you to run it in a VM if you bought a copy? Dang, I thought they'd actually fixed that silliness. Good to know.

 

Windows isn't that we don't know what's broken; it's that nobody with

the skills to fix it has volunteered to help.

Well, I'm buying a Windows 8 ultrabook convertible in the next few weeks and plan to use it for Clojure development while I'm on the road so I'll have quite the incentive to help...

-- 

[Peter Taoussanis]

I'd caution anyone against trying to redeploy their libraries right now since there seems to be some serious unresolved issues. I just tried a redeploy myself and am also getting the "ReasonPhrase:Forbidden" error.

Unfortunately this seems to leave the repo in a bad state, since dependency pull requests now come back with a "Checksum validation failed".

> I still seem to be able to pull the library into a project and Clojars says it has been promoted (after my first successful try - with a different key / user ID).

Sean, are you sure it's working if the dependency isn't already in your .m2 cache?

- Peter Taoussanis

[Peter Taoussanis]

BTW for those of you running into the ReasonPhrase:Forbidden error, it seems as if the old lein-clojars can be used to restore a repo to a working (unsigned) state.

[Sean Corfield]

I removed congomongo completely from my local repo and lein repl seemed to pull it back down with no problems. Tested it on two machines. So it seems the repo on Clojars is OK for me - except that I can't redeploy the POM?

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

[Wes Freeman]

+1 on the checksum validation error and ReasonPhrase:Forbidden on the pom. Using leiningen preview10. I updated my version number to not have SNAPSHOT, so maybe that's why it's not working, compared to Sean's?

Regardless of the current bug, thanks for working on this functionality, Phil (and whoever else contributed to it). It felt like clojars (along with leiningen) was a great convenience, but was a bit weak on the security side, being too open--and this closes that gap significantly.

Quick OSX/Homebrew tutorial (this is what I did):
brew install gpg
gpg --gen-key

# the following command is sufficient if you've just installed gpg and only have one key, otherwise you should probably already know how to figure out which key you want
gpg --export -a 

(copy the resulting text into your clojars profile)

lein deploy clojars

Not too hard, right? (Admittedly, gpg is one of those homebrew recipes that requires symlinking some stuff in /usr/local/share/locale/ to finish linking; my usual work around is to "chown root:wheel" the homebrew executable to "sudo brew link gpg", and then reset it.)

Wes

On Tue, Nov 20, 2012 at 12:22 AM, Peter Taoussanis <ptaou...@gmail.com> wrote:

--

[Wes Freeman]

On Monday, November 19, 2012 12:53:32 PM UTC-5, Phil Hagelberg wrote:

If you turn off :sign-releases inside your :repositories entry when
deploying libraries everything will work for you as before. But your
libraries won't qualify for the Releases repo in this case. So once your
users upgrade to Leiningen 2.0.0 they will have to include a separate
:repositories entry for the classic repo to indicate that they are OK
with pulling in dependencies that don't meet the higher standards of the
new repo.

Can you give some example syntax for :repositories and ":sign-releases false" specifically for clojars? I was hoping to test deploy without signing and then try again with signing, to see if that fixes my project.

Thanks,

Wes

[Wes Freeman]

Ok, if you do a signed deploy, you can scp the pom and jar over to get it working, directly after.

Like:

scp pom.xml target/clojurithms-0.1.0.jar clo...@clojars.org:/repo/clojurithms/clojurithms/0.1.0/

--

[Phil Hagelberg]

Wes Freeman <freem...@gmail.com> writes:

> Can you give some example syntax for :repositories and ":sign-releases
> false" specifically for clojars? I was hoping to test deploy without
> signing and then try again with signing, to see if that fixes my
> project.

Sure; it would just be something like this:

:repositories [["clojars" {:url "https://clojars.org/repo"
:sign-releases false}]]

-Phil

[Phil Hagelberg]

Peter Taoussanis <ptaou...@gmail.com> writes:

> I'd caution anyone against trying to redeploy their libraries right
> now since there seems to be some serious unresolved issues. I just
> tried a redeploy myself and am also getting the
> "ReasonPhrase:Forbidden" error.

Sorry about this. We've tracked it down to concurrency issues with the
underlying sqlite database driver. I have a fix that reduces the
contention by having an explicit "promote" button instead of
automatically promoting upon deployment, but a proper fix will be more
involved; might need to port to a new DB. My fix is currently blocked on
an anti-forgery issue, but I hope to have it out tomorrow.

> Unfortunately this seems to leave the repo in a bad state, since
> dependency pull requests now come back with a "Checksum validation
> failed".

If you have to deploy before then I would recommend using an scp-based
deployment.

Sorry for the inconvenience.

-Phil

[Peter Taoussanis]

Sorry for the inconvenience.

No problem at all Phil. You're doing us all a huge service with all your work on this; teething issues are to be expected. Just wanted to suggest library authors proceed cautiously for now to avoid unknowingly breaking dependents in the wild.

[John Gabriele]

On Sunday, November 18, 2012 4:46:51 PM UTC-5, Sean Corfield wrote:

On Sun, Nov 18, 2012 at 5:56 AM, Phil Hagelberg <ph...@hagelb.org> wrote:

If you don't have a key yet, generate one with `gpg --gen-key`. The
default settings are pretty good, though I'd recommend making it expire
in a year or two. Next find your key ID. It's the 8-character part after
the slash on the line beginning with "pub":

As I said at the conj, I'm looking forward to the documentation explaining how to install and use gpg since it's not provided by default on either Mac OS X or Windows.

Then you can show it with `gpg --export -a $KEY_ID`.

$KEY_ID? (again, as I noted at the conj, without good documentation on the Leiningen site for this, folks won't necessarily know what this is or why they need to do all of this, especially the web of trust stuff you discussed and key exchanges / publishing etc).

Regarding the documentation issue, I wrote this today https://github.com/clojuredocs/cds/blob/master/articles/ecosystem/libraries_authoring.md . It should make it online to clojure-doc.org soon. I included the instructions Phil gave here, and went through the process with a trivial example lib to make sure I included any other details I ran across.

Thank you to the folks on #leiningen for helping clarify some of the points covered in that doc.

---John