Overwriting deployed artifacts

[Phil Hagelberg]

Hello folks.

I am considering pushing off the switch to primarily using the Releases
repository to later than the 2.0.0 release of Leiningen since it's
proving to be more challenging than foreseen. One thing that could make
a good compromise is sticking with the regular Clojars repo but changing
the policy so that each deployed release is immutable.

From the reaction I got asking on IRC, most people don't even realize
that you're currently allowed to redeploy over an existing artifact.
This leads me to believe that such a change would not be very
disruptive. The only time I've done this is when I made a mistake in a
deployment and wanted to correct it quickly before anyone noticed. While
it can be a bit embarrassing to have a mistaken deploy out there,
bumping the bugfix version number is easy to do.

In addition, the jar deletion process has been formalized now with
GitHub issues, so if something sensitive is deployed it can be addressed
relatively quickly without requiring a redeploy.

The main legitimate case for redeploying is when you have an old release
that you want to go back and sign. While it'd be technically possible to
go download the old version, sign it, and simply deploy the signature,
it's currently much easier to redeploy the whole thing as a unit. So I
think blocking the redeployment of signed artifacts is probably the way
to go.

Thoughts?

-Phil

[Toby Crawley]

This sounds like a good plan - being able to alter released versions has
always struck me as a little dangerous, and has bitten me in the past
(though it wasn't a clojars hosted artifact).

Redeploying unsigned artifacts as signed is fine as well, assuming that
some day all non-snapshot artifacts will have to be signed on initial
deployment (which I believe is where we are heading, right?).

- Toby

--
Toby Crawley
http://immutant.org | http://torquebox.org

[Dan Larkin]

+1

[Nelson Morris]

I like the immutable change.

It shouldn't be too hard to make a plugin for lein that will download
previous pom/jars and then create signatures and send just those to
clojars.

[Alex Osborne]

On 8/01/13 11:49 AM, Phil Hagelberg wrote:
> From the reaction I got asking on IRC, most people don't even realize
> that you're currently allowed to redeploy over an existing artifact.
> This leads me to believe that such a change would not be very
> disruptive. The only time I've done this is when I made a mistake in a
> deployment and wanted to correct it quickly before anyone noticed. While
> it can be a bit embarrassing to have a mistaken deploy out there,
> bumping the bugfix version number is easy to do.

And prevents any confusion over local cached versions. Redeploying was
never really an intentional feature, it's just a case that wasn't
covered properly in the original design.

[Phil Hagelberg]

In light of the recent verification efforts around the possibility of
compromise, I'd like to revisit this question. I'm not sure that it's
worth the special case of allowing redeployment of unsigned artifacts
since it just complicates things and makes verification more difficult.

The downside of disallowing redeployment across the board is that adding
a signature would involve bumping the version number. Personally I feel
like this is not overly burdensome, but I'm not sure whether others will
feel the same.

One alternative would be to deploy just the signature files of a given
jar/pom up to Clojars; this would allow signing artifacts that are
currently unsigned without compromising on immutability. I am hesitant
to recommend this (though of course we can't really prevent it) since I
believe it would encourage people to pull down artifacts from Clojars
and sign those rather than signing artifacts they trust because they've
just generated them directly from source.

Nelson brought up the fact that since HTTP deploys are not
transactional, you could have a situation in which the jar gets deployed
and the pom and signatures do not. This is definitely a consideration
here. My hope is that it's rare enough that we can rely on manual
intervention to set these cases right, but I don't have any data to back
this up. We could probably determine how often deploys are interrupted
From the logs though, so perhaps that's a good place to start.

Thoughts?

-Phil

[Nelson Morris]

I'm in favor of simply rejecting the PUT requests of any non maven-metadata.xml file that already exists. This would mean it would be possible to upload signatures significantly later, but I can't think of a non-time based way to make a cutoff.

We could handle failed deploys similar to delete requests.

Also, ssh deployment would require a similar mechanism of checking for the file in the repo.

[Alex Osborne]

+1 Let's see how it goes. I've never much liked redeploys but they have served a useful purpose on occasion.

--
You received this message because you are subscribed to the Google Groups "clojars-maintainers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojars-maintai...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Paul Stadig]

Correct me if I'm wrong, but is it not the case that the only time you would *not* want to overwrite an artifact if is you have a jar *and* a valid signature. In all other cases there aren't any guarantees. What about rejecting uploads if there is already a jar and valid signature (where 'valid' could mean signed by any key, not necessarily an authorized key)? This would allow for transactional failure with uploading separate files. It would also not require a user to download a jar from clojars sign it and upload just the signature.

Another possibility would be to send back a successful response (but not actually overwrite anything) if there's already a file on disk with the same etag. Essentially making uploads idempotent. Although with this approach, if an upload got truncated because of a network error, then there would be no way to fix it. You'd have to resort to bumping the version number.

[Phil Hagelberg]

Paul Stadig writes:

> Correct me if I'm wrong, but is it not the case that the only time you
> would *not* want to overwrite an artifact if is you have a jar *and* a
> valid signature. In all other cases there aren't any guarantees. What about
> rejecting uploads if there is already a jar and valid signature (where
> 'valid' could mean signed by any key, not necessarily an authorized key)?
> This would allow for transactional failure with uploading separate files.
> It would also not require a user to download a jar from clojars sign it and
> upload just the signature.

Sorry; I didn't really provide the context for my last suggestion. In
the event of a possible security incident, it's very valuable to be able
to compare checksums of the current repository with historical values.
We wouldn't have to manually verify that all changes were legitimate
redeploys by the original author like we did last week, since there
would be no such thing as a legitimate redeploy.

When I first proposed blocking redeploys, we were pursuing a much more
aggressive rollout schedule for the releases repository. Given that it
was dropped from the lein2 release, we're having a much more gradual
time of easing into it where all deploys for a nice long period will be
signed unless it's explicitly disabled. So in my eyes the need for
signing already-deployed artifacts is much reduced since once the
releases repository becomes the default it will only be very old
(or opted-out) artifacts that are unsigned.

Does that make sense?

> Another possibility would be to send back a successful response (but not
> actually overwrite anything) if there's already a file on disk with the
> same etag. Essentially making uploads idempotent. Although with this
> approach, if an upload got truncated because of a network error, then there
> would be no way to fix it. You'd have to resort to bumping the version
> number.

This would be doable, but the problem is that jars generated by
Leiningen include timestamps on them, so redoing a `lein deploy clojars`
will not re-send the same bytes unfortunately.

-Phil

[Paul Stadig]

On Mon, Apr 22, 2013 at 12:44 PM, Phil Hagelberg <ph...@hagelb.org> wrote:

So in my eyes the need for
signing already-deployed artifacts is much reduced since once the
releases repository becomes the default it will only be very old
(or opted-out) artifacts that are unsigned.

Does that make sense?

Yeah that makes sense. I got distracted when you mentioned uploading a signature after having uploaded a jar. So it would seem my second suggestion would be useful.

 

> Another possibility would be to send back a successful response (but not
> actually overwrite anything) if there's already a file on disk with the
> same etag. Essentially making uploads idempotent. Although with this
> approach, if an upload got truncated because of a network error, then there
> would be no way to fix it. You'd have to resort to bumping the version
> number.

This would be doable, but the problem is that jars generated by
Leiningen include timestamps on them, so redoing a `lein deploy clojars`
will not re-send the same bytes unfortunately.

Then again maybe not. :(