gem yank

[John Trupiano]

Just creating a thread to discuss the specifics of this feature. I've
got an implementation of the feature up and running on the yank branch
of my fork: http://github.com/jtrupiano/gemcutter/commits/yank

Outstanding concerns at this time that could use some input (put forth
by Nick):

1) Deleting a RubyGem means that dependencies might break. For
example, I blew away the 'zentest' gem and that broke a LOT of gems.
2) Really need to make sure ownerships, downloads, etc are all cleaned
up after one has been deleted. Not sure if that's happening now.
3) What should happen when a rubygem's last version is taken away?
What if they just wanted to revert one pushed gem and someone came in
the meantime and pushed something else over it? Too bad for the
original author? Maybe there should be a grace period?

Nick, on a separate note, I found a large majority of the gem tests
failing due to a badly mocked FakeWeb URI's. Was I the only one
seeing these failures? This commit cleans them up:
http://github.com/jtrupiano/gemcutter/commit/26b02238f87e51e1054e18f17217ef18d1b90e1b

-John

[Nick Quaranto]

Over the past few weeks I've been moving the gemcutter gem plugins into http://github.com/qrush/rubygems since they're going to get merged into RubyGems proper....as such I probably should have deleted the gem/ folder from gemcutter to avoid any other confusion. If the tests were out of sync I'm sorry :/

But yes, those concerns still exist and will have to get resolved before gem yank is released/ready.

-Nick

[John Trupiano]

On Wed, Nov 25, 2009 at 5:32 PM, John Trupiano <jtru...@gmail.com> wrote:

1) Deleting a RubyGem means that dependencies might break. For
example, I blew away the 'zentest' gem and that broke a LOT of gems.

I'm not sure what the best thing to do here is. My use case is to remove a gem almost immediately after I released it. If someone has already upped their dependency to that version, well I don't think there's a whole lot we can do. (see a real example of this happening with timecophttp://github.com/jnunemaker/mongomapper/commit/18ef8b6cfc61b0b9d5e08452c6ee9e603badc096 ) What's probably more worrisome is "retiring" old gems. As your library marches along, you might just not want to support the old versions of the gem that others may be depending on. If no one has any good ideas, maybe we simply don't allow the removal of a gem that other gems depend on until someone comes up with a better idea?

 

2) Really need to make sure ownerships, downloads, etc are all cleaned
up after one has been deleted. Not sure if that's happening now.

The only database-specific code that gets run in this entire feature is version.destroy. Looking at the Version model, you do have :dependent => :destroys for both dependencies and downloads. Unless there's something else we need to clean up, this probably isn't a concern.

 

3) What should happen when a rubygem's last version is taken away?
What if they just wanted to revert one pushed gem and someone came in
the meantime and pushed something else over it? Too bad for the
original author? Maybe there should be a grace period?

 

Honestly, I think we just leave the window open. What really are the chances that someone gets screwed by this? btw, what is the current status, can a gem exist on gemcutter without any releases? 

OK, well these testcase fixes are all isolated to a single commit that you can ignore when merging this fork in if appropriate.  In either case, I think you'll want to reference my commits to the gem/ directory when porting the plugin commands over to your fork.  I believe you had previously told me that I should leave that integration/port to you.  Is that correct?

-John

[John Trupiano]

> >> 1) Deleting a RubyGem means that dependencies might break. For
> >> example, I blew away the 'zentest' gem and that broke a LOT of gems.
>
> I'm not sure what the best thing to do here is. My use case is to remove a
> gem almost immediately after I released it. If someone has already upped
> their dependency to that version, well I don't think there's a whole lot we

> can do. (see a real example of this happening with timecophttp://github.com/jnunemaker/mongomapper/commit/18ef8b6cfc61b0b9d5e08...

> )
> What's probably more worrisome is "retiring" old gems. As your library
> marches along, you might just not want to support the old versions of the
> gem that others may be depending on. If no one has any good ideas, maybe we
> simply don't allow the removal of a gem that other gems depend on until
> someone comes up with a better idea?

What if removing a gem that others gem depended on required you to set
a force flag. A sample shell session might look like:

$> gem yank timecop -v0.3.2

Unable to yank gem because other gems depend on it.

*** DEPENDENT GEMS ***

mongomapper (0.3.5)
rails-caddy (0.0.8)

$> gem yank timecop -v0.3.2 -f

Gem successfully yanked.

[David Dollar]

Building and understanding the dependency graph of all gems isn't
really feasible right now. For example, if a gem specified a
dependency of example-gem >= 1.0 should I be able to yank 1.0.1? What
if there is nothing lower than 1.0.1 that satisfies the dependency?

[John Trupiano]

On Tue, Dec 1, 2009 at 12:54 PM, David Dollar <ddo...@gmail.com> wrote:

Building and understanding the dependency graph of all gems isn't
really feasible right now. For example, if a gem specified a
dependency of  example-gem >= 1.0 should I be able to yank 1.0.1? What
if there is nothing lower than 1.0.1 that satisfies the dependency?

So are you suggesting that we ignore this problem?  It's important to note that the owner of the gem will be the only person who can remove it. So really it's the owner's prerogative if they want to remove a gem that other gems depend upon.

I can get behind this behavior (after all, I won't have to write any code to leave it as is), but it was an issue that Nick suggested had to be resolved before we could merge the feature in.

-John

 

[David Dollar]

I personally think that if we're going to have gem yank, it should just be up to the owner with no checks. However, I just wanted to raise the issue that if that's a must-have check on this feature, then it's going to be a technical challenge.

[Douglas Campos]

We should remember the _why disappearance.... IMHO we'd check yank
1.0.1 only if 1.0.2 exists, satisfying the dep. (that would work with
a force switch + grace period)

--
Douglas Campos (qmx)
+55 11 7626 5959
+55 11 6762 5959

[Ryan Grove]

On Tue, Dec 1, 2009 at 4:41 PM, Douglas Campos <dou...@theros.info> wrote:
> We should remember the _why disappearance.... IMHO we'd check yank
> 1.0.1 only if 1.0.2 exists, satisfying the dep. (that would work with
> a force switch + grace period)

Why should Gemcutter (or anyone) get to dictate what I can or can't do
with code that I've written and released? I'm sure nobody was pleased
to see _why disappear and take all his code with him, but he was
certainly within his rights to do so.

- Ryan

[David Dollar]

I agree. If we're going to add gem yank, there shouldn't be any limitations on it.

[Douglas Campos]

>> Ryan wrote

>> Why should Gemcutter (or anyone) get to dictate what I can or can't do
>> with code that I've written and released? I'm sure nobody was pleased
>> to see _why disappear and take all his code with him, but he was
>> certainly within his rights to do so.

Hey, please let's not start a flamewar =) I'm thinking at the newbs
now, and protecting the ecosystem - a broken dependency is bad -
that's why I talked about a grace period. This would prevent
drunkiness, etc.. But still retains the power at the owner.

BTW, this is just a suggestion. Your point is highly valid - I was
just thinking in something like gmail's undo.

Cheers!

[Ryan Grove]

On Tue, Dec 1, 2009 at 5:15 PM, Douglas Campos <dou...@theros.info> wrote:
> Hey, please let's not start a flamewar =)  I'm thinking at the newbs
> now, and protecting the ecosystem - a broken dependency is bad -
> that's why I talked about a grace period. This would prevent
> drunkiness, etc.. But still retains the power at the owner.

Sorry, I didn't mean to sound inflammatory.

I don't know about a grace period, but I wouldn't be opposed to a
clearly-worded warning and confirmation step to ensure that the person
who just ran `gem yank` understands that they might be causing
problems for other gem authors and to verify that they *really* want
to delete their gem in spite of this.

- Ryan

[Chad Woolley]

On Tue, Dec 1, 2009 at 6:00 PM, Ryan Grove <ry...@wonko.com> wrote:
> Why should Gemcutter (or anyone) get to dictate what I can or can't do
> with code that I've written and released?

Because it is open source, and by publishing it you (should) have an
implicit responsibility not to break people who depend upon you. The
only valid reasons I can think of to yank a gem is a severe security
hole. A bug isn't a good reason, because I may depend upon the
library for other non-buggy functionality, and could care less about
any given bug. In fact, unless it is a very severe security hole
(e.g. compromises host system) I'd say it should still stay out there.

In other words, I think yank has the potential to do much more harm
than good when people yank dependencies willy-nilly for questionable
reasons. If there's something wrong with a gem, release a new version
with the fix. If you don't, then let the users gems which depend on
you make the decision whether to use it or not, don't screw them by
deleting a dependency and forcing them to fork or re-release.

As for the _why analogy, he just deleted his master repos and sites,
but all his source is still around, just owned by other people. Most
relevant to this discussion is the fact that all releases of all his
gems are still available in the canonical RubyGems repo. He didn't
delete those (or break anyone who depended on them).

-- Chad

[Nick Quaranto]

Gemcutter is not a source code repo.

My main motivation for this is that I usually get 2-3 requests daily to delete gems someone didn't mean to push. I think there's plenty of gem mirrors abound if we really have lost a gem many depend on.

Perhaps once a gem has been marked as deleted, that 'namespace' should be marked in some way so others can't push to it anymore. Basically it becomes a dead man's zone, but the records would still hang around in our DB. If someone really wants to take over a gem they can still contact me about it.

-Nick

[Anuj Dutta]

2009/12/2 Ryan Grove <ry...@wonko.com>

Hello all,

A warning message sounds just fine in this case with a confirmation step that seeks user approval.

Anuj

--
Anuj DUTTA

[Ryan Grove]

On Tue, Dec 1, 2009 at 6:36 PM, Chad Woolley <thewoo...@gmail.com> wrote:
>> Why should Gemcutter (or anyone) get to dictate what I can or can't do
>> with code that I've written and released?
>
> Because it is open source, and by publishing it you (should) have an
> implicit responsibility not to break people who depend upon you.

While I agree that breaking things people depend on would be a lame
thing for me to do, I disagree that Gemcutter or any other service
should have the right to tell me that I can't delete something I've
chosen to publish freely. This would be akin to Blogger refusing to
allow me to delete a post from my blog that other people have linked
to, or Twitter refusing to allow me to delete a tweet.

I also agree that `gem yank` could cause harm if used thoughtlessly,
but it's not Gemcutter's job to police my motivations. I may want to
delete a gem because I want to vanish from the face of the earth and
break all the software that's come to depend on my code, or I may want
to delete it because I ran `gem push` prematurely, or a hundred other
reasons.

Regardless of my motivations, it's my gem, and I should have control
over it. If you're worried that I'll yank it and ruin your day, all
you need to do is download a copy.

- Ryan

[John Trupiano]

I just want to reiterate my use case.

I push a new patch release that fixes a bug in the past release.  In just a short period of time, a user of my gem emails me telling me that I've introduced an unrelated regression.  I am not in a position to hunt down the cause of the latest regression.  I have three choices now:

1) Leave the broken gem up there until I fix it.

2) Yank that version of the gem (which has only been up a short while) until I have time get a new release out.

3) Re-release the previous version with a higher patch release number than the broken one.

Scenario 1 will lead to other people unwittingly installing a broken version of my gem (anyone who runs gem install mygem will get the broken version).  

Scenario 2, while making that version 'disappear', will prevent others from losing time when they check for a new version.  I see it as my responsibility as a gem maintainer to communicate the removal of that gem version to my users.

Scenario 3 just feels wrong.  Now, my releases look like: 0.3.1 (good but with a bug), 0.3.2 (useless), 0.3.3 (exactly the same as 0.3.1).

The primary concern here appears to be we as a community don't trust gem maintainers to be responsible with their decision to use gem yank.  Perhaps we could add a "yanked" bitfield to the rubygems table.  When set, that version of a gem can only be installed by explicitly asking for that version.  A plain old gem install mygem will install the most recent non-yanked version of the gem.

-John

[Chad Woolley]

On Wed, Dec 2, 2009 at 5:38 AM, John Trupiano <jtru...@gmail.com> wrote:
> The primary concern here appears to be we as a community don't trust gem
> maintainers to be responsible with their decision to use gem yank.  Perhaps
> we could add a "yanked" bitfield to the rubygems table.  When set, that
> version of a gem can only be installed by explicitly asking for that
> version.  A plain old gem install mygem will install the most recent
> non-yanked version of the gem.

My main concern is not the "remove broken version" case you describe
(which is a legitimate uses and wouldn't break dependencies), but
entirely removing all versions of a gem (perhaps maliciously or
petulantly or ignorantly).

Given that, your proposal above of the 'yanked' flag which would hide
the gem unless you explicitly request is sounds like a great idea.

-- Chad

[John Trupiano]

On Dec 2, 10:08 am, Chad Woolley <thewoolley...@gmail.com> wrote:

> On Wed, Dec 2, 2009 at 5:38 AM, John Trupiano <jtrupi...@gmail.com> wrote:
> > The primary concern here appears to be we as a community don't trust gem
> > maintainers to be responsible with their decision to use gem yank.  Perhaps
> > we could add a "yanked" bitfield to the rubygems table.  When set, that
> > version of a gem can only be installed by explicitly asking for that
> > version.  A plain old gem install mygem will install the most recent
> > non-yanked version of the gem.
>

> Given that, your proposal above of the 'yanked' flag which would hide
> the gem unless you explicitly request is sounds like a great idea.
>
> -- Chad

I suppose that would be 2 +1's for a yanked flag on release versions.
How do others feel? Nick, do you think this scheme appropriately
addresses your concerns about breaking dependencies?

> > 3) What should happen when a rubygem's last version is taken away?
> >> What if they just wanted to revert one pushed gem and someone came in
> >> the meantime and pushed something else over it? Too bad for the
> >> original author? Maybe there should be a grace period?
>

> Honestly, I think we just leave the window open. What really are the chances
> that someone gets screwed by this? btw, what is the current status, can a
> gem exist on gemcutter without any releases?

Does anyone have thoughts on the third concern from the initial post?

-John

[David Dollar]

A "yanked" flag would have to be implemented at the Rubygems level.

* Indexes would have to be made aware which gems are "yanked"

* Does the gem only get installed if I do   gem install foo -v 1.1 ?

* What if I do  >= 1.0, <= 1.2 and that's the only gem that matches ?

I don't think it's a good idea to mess with Rubygems proper over this. In addition, I have strong ethical objections to telling people what they can or can't do with their own code. On top of that, there is no guarantee that a given gem will even be maintained under a license which gives Gemcutter the legal right to stop them from deleting their own gem.

All of that being said, if we don't give gem owners the ability (with a big fat warning and an admonition about what a big meanie they are) to remove their own gems, Nick is going to be fielding just as many requests to do it manually as he is today.

- David

[Chad Woolley]

On Thu, Dec 3, 2009 at 10:48 AM, David Dollar <ddo...@gmail.com> wrote:
> addition, I have strong ethical objections to telling people what they can
> or can't do with their own code.

I don't. I do strongly object to people deleting dependencies out
from under me, though...

> On top of that, there is no guarantee that
> a given gem will even be maintained under a license which gives Gemcutter
> the legal right to stop them from deleting their own gem.

If the license for a given version allows people to copy the code
(which it must, otherwise, why would they publish it), then
rubygems.org (and anyone else who has downloaded it) has rights to
keep the copy.

> All of that being said, if we don't give gem owners the ability (with a big
> fat warning and an admonition about what a big meanie they are) to remove
> their own gems, Nick is going to be fielding just as many requests to do it
> manually as he is today.

I disagree. The yank flag will prevent anyone from seeing a yanked
gem unless they really want to. In that situation, the only people
bugging Nick should be ones who insist that their code be completely
purged from the public gem repo, which I think should be very rare
(see comments on licenses above).

-- Chad

[David Dollar]

Having a "yank" flag is not possible with Rubygems as it currently exists.At the end of the day, Gemcutter is basically a pretty front-end to generatign a gem index. A gem index looks like (a Marshal.dump'd, gzip'd version of) this:

[["_", #<Gem::Version "1.2">, "ruby"], 

["a2ws", #<Gem::Version "0.1.9">, "ruby"], 

["aaalex-off_github", #<Gem::Version "1.1.2">, "ruby"], 

["aalib-ruby", #<Gem::Version "0.7.1">, "ruby"],

["aargvark", #<Gem::Version "0.0.15">, "ruby"],

["aasm", #<Gem::Version "2.1.3">, "ruby"]]

["gemname", version, "platform"]

The Rubygems command-line utility downloads this index and uses it to determine which gems are available, and which gems fill dependencies. The "gem" command line client never interacts with Gemcutter itself, so any "yanked" property would have to be stored in the index itself which means significant changes in Rubygems itself.

- David

[Ariejan de Vroom]

Personally, I think it should be possible to for an owner to yank his
gems. This does have implications, as we've all seen with the _why
incident. However, there are a number of reasons why anyone wants to
remove a gem.

* They didn't intent to push anything in the first place.
* There's a fenomenal bug or security hole in a gem that does damage.
* The gem is obsolete or has been renamed.

The gem owner should have the rights to yank gems. If he does yank all
versions, the name should become available again for others to push
to. This would allow me to remove all my gems and have someone else
take over.

It'd also be nice if with the yank, the owner must specify a reason
for yanking the gem. If someone does an install and the gem no longer
exists, the reason could be displayed. E.g. you have to use another
gem or there were severe security implications.

Personally, I think the ruby community will be able to use the yank
feature wisely.

- Ariejan

On Dec 3, 7:54 pm, David Dollar <ddol...@gmail.com> wrote:
> Having a "yank" flag is not possible with Rubygems as it currently exists.At
> the end of the day, Gemcutter is basically a pretty front-end to generatign
> a gem index. A gem index looks like (a Marshal.dump'd, gzip'd version of)
> this:
>
> [["_", #<Gem::Version "1.2">, "ruby"],
> ["a2ws", #<Gem::Version "0.1.9">, "ruby"],
> ["aaalex-off_github", #<Gem::Version "1.1.2">, "ruby"],
> ["aalib-ruby", #<Gem::Version "0.7.1">, "ruby"],
> ["aargvark", #<Gem::Version "0.0.15">, "ruby"],
> ["aasm", #<Gem::Version "2.1.3">, "ruby"]]
>
> ["gemname", version, "platform"]
>
> The Rubygems command-line utility downloads this index and uses it to
> determine which gems are available, and which gems fill dependencies. The
> "gem" command line client never interacts with Gemcutter itself, so any
> "yanked" property would have to be stored in the index itself which means
> significant changes in Rubygems itself.
>
> - David
>

> On Thu, Dec 3, 2009 at 1:36 PM, Chad Woolley <thewoolley...@gmail.com>wrote: