Github vs Gemcutter, versions, etc

[Justin French]

Hi,

Can anyone point me towards a blog post or discussion or suggestions
on how to maintain gems on Github and Gemcutter? I have a reasonably
popular gem hosted on Github (justinfrench-formtastic), and I'd rather
not have to publish the two gems individually forever or leave people
behind on a dead gem.

I guess I'm wondering if Gemcutter has a post-commit hook that Github
can use — push a new gemspec to Github, and Gemcutter gets a new gem
too (or something like that).

Maybe another way to go would be some kind of "Github migrate" that
can go and make canonical gems from the ones I've already published on
Github.

Otherwise, I guess my plan is:

* start publishing to both with the next version
* change the documentation to point people towards the gemcutter
version
* figure out a way to warn/notify people using the Github version to
shift over
* stop building the Github gem at some point

If anyone else has done some thinking on this, more than keen to hear
it!

[Nick Quaranto]

GitHub's gem builder is currently disabled, and Gemcutter does not and will never build gems for you. I feel this approach to gem server management is flawed and I tried to avoid that at all costs as I built out this project.

As for going forward, I'm hopefully going to import the github gems and get a migration process down for github in the near future. if you're willing to help on this please let me know!

There's also discussion about gem forking in general going on, we're trying to solve this for *all* of rubygems: http://wiki.github.com/qrush/gemcutter/fork-support

-Nick

[Dan Croak]

Justin,

I know Formtastic is using Jeweler, which has Gemcutter support baked in:

http://github.com/technicalpickles/jeweler

I don't have a solution to the Github-to-Gemcutter migration but I do
think we'll also probably be moving our gems to Gemcutter "officially"
and dropping Github support at some point. A clear migration path
would be nice, and the rubygems.org move is probably the moment.

Dan

On Wed, Oct 7, 2009 at 5:39 PM, Justin French <jus...@indent.com.au> wrote:
>

--
Dan Croak
@Croaky

[Nick Quaranto]

Here's what I've been brainstorming for the github migration:

* gem migrate your-github-user --github (otherwise default to rubyforge)
* This then pulls out your github.user and github.token, sends it to Gemcutter. If either aren't there, prompt for it.
* Gemcutter then hits the github api and sees if it can get to some hidden info: http://develop.github.com/p/users.html
* If it can, you get ownership for every github gem that was imported under your user.

Sadly, I can't think of a way to do this without committing to some arbitrary repo or sending the api key to Gemcutter (where we wouldn't store it).

-Nick

[trans]

On Oct 7, 5:50 pm, Nick Quaranto <n...@quaran.to> wrote:
> Here's what I've been brainstorming for the github migration:
>
> * gem migrate your-github-user --github (otherwise default to rubyforge)
> * This then pulls out your github.user and github.token, sends it to
> Gemcutter. If either aren't there, prompt for it.
> * Gemcutter then hits the github api and sees if it can get to some hidden
> info:http://develop.github.com/p/users.html
> * If it can, you get ownership for every github gem that was imported under
> your user.
>
> Sadly, I can't think of a way to do this without committing to some
> arbitrary repo or sending the api key to Gemcutter (where we wouldn't store
> it).

You're going to have a mess on your hands. There are already github
repos with the same names under different users that are completely
different programs. Not to mention, the name conflicts between
rubyforge and github that will arise. You're trying to make Gemcutter
the official gem source by subordinating it to Rubyforge and Github.
It's a contradiction and the deeper you get the more it is going to
prove itself "gem hell".

As for gem forks. Ultimately I imagine it must be up to Ruygems team
to decide if they wish to support the concept formally. I will suggest
however that drbrain is on the right track: "alpha portions can be
appended to versions and mark the gems as “prerelease” right now. (gem
“rake”, “0.8.7.drbrain”)"

I note that forks are effectively equivalent to pre-releases. Forks
don't generally stay forks for very long. Either they are merged to
master or they evolve into new projects (with new names). In the cases
where someone is purposefully maintaining forks as a means of
customization (say pre-customer customizations), that I think is a
poor approach --use branches or in program conditionals. In cases
where a fork is being maintained for a specific use and presumably b/c
the fork is unsuitable and/or the maintainer of the master repo simply
won't accept the patch, then either be satisfied to install via --
prerelease, or maintain a renamed plugin --one of the nice things
about Ruby is that all code is pluggable by it's dynamic nature.

[Nick Quaranto]

trans, I think it might be best to clarify the direction we're going in here. The idea is that users can sign up for a subdomain, and we'll give them a blank index to work with. You can register trans.gemcutter.org, and it's essentially your own playground that you can push anything to.

I feel it's up to Gemcutter to educate people on the real usage of these:

1) Subdomain use should be infrequent. It's for trial/forked gems that shouldn't be relied on for production.
2) Use prerelease versions in the main repo if at all possible (1.0.0pre, etc)
3) Don't add someone's subdomain as a source, unless you can completely trust anything they toss there. (like, your own for example)
4) http://rubygems.org is the main, canonical repo that you can trust.
5) Start looking into gem signing/cert since it's *the* way we can really trust gems: http://blog.segment7.net/articles/2009/02/04/a-rubygems-github-proposal

As for the GitHub migration, we'll probably import forked gems into subdomains and let people migrate them over using the API. This way, the main index won't be polluted with tons of forked gems. At least, that's my plan so far...none of this has been finalized yet but I think we're pretty set on this way to fix gem forks.

-Nick

[trans]

On Oct 7, 11:31 pm, Nick Quaranto <n...@quaran.to> wrote:
> trans, I think it might be best to clarify the direction we're going in
> here. The idea is that users can sign up for a subdomain, and we'll give
> them a blank index to work with. You can register trans.gemcutter.org, and
> it's essentially your own playground that you can push anything to.
>
> I feel it's up to Gemcutter to educate people on the real usage of these:
>
> 1) Subdomain use should be infrequent. It's for trial/forked gems that
> shouldn't be relied on for production.
> 2) Use prerelease versions in the main repo if at all possible (1.0.0pre,
> etc)
> 3) Don't add someone's subdomain as a source, unless you can completely
> trust anything they toss there. (like, your own for example)

> 4)http://rubygems.orgis the main, canonical repo that you can trust.

> 5) Start looking into gem signing/cert since it's *the* way we can really
> trust gems:http://blog.segment7.net/articles/2009/02/04/a-rubygems-github-proposal

Ah. So instead of altering gem names for forks, each user will have
there own "mini" gem source?

[Nick Quaranto]

Exactly.

-Nick

[Jon Yurek]

My concern with the subdomain is that if you have a forked gem that
depends on another forked gem, simply specifying the source on the
command line (e.g. sudo gem install someoneelse-fakefs --source
http://someoneelse.gemcutter.org) does not currently allow fetching
all the dependencies correctly. It would require a change in rubygems
to allow that. And if we're changing rubygems, why not just add real
namspacing support right in the gemspec?

On Oct 8, 9:08 am, Nick Quaranto <n...@quaran.to> wrote:
> Exactly.
>
> -Nick
>

> On Thu, Oct 8, 2009 at 2:32 AM, trans <transf...@gmail.com> wrote:
>
> > On Oct 7, 11:31 pm, Nick Quaranto <n...@quaran.to> wrote:
> > > trans, I think it might be best to clarify the direction we're going in
> > > here. The idea is that users can sign up for a subdomain, and we'll give
> > > them a blank index to work with. You can register trans.gemcutter.org,
> > and
> > > it's essentially your own playground that you can push anything to.
>
> > > I feel it's up to Gemcutter to educate people on the real usage of these:
>
> > > 1) Subdomain use should be infrequent. It's for trial/forked gems that
> > > shouldn't be relied on for production.
> > > 2) Use prerelease versions in the main repo if at all possible (1.0.0pre,
> > > etc)
> > > 3) Don't add someone's subdomain as a source, unless you can completely
> > > trust anything they toss there. (like, your own for example)

> > > 4)http://rubygems.orgisthe main, canonical repo that you can trust.

[Nick Quaranto]

We're actually not changing RubyGems at all for this to work... from what I've been told, adding a source to the gemspec (or even a namespace) would be yet another huge undertaking, and could end up in a massive change in workflow. Perhaps I'll rope in one of the RubyGems committers to give a better explanation of this.

-Nick

[Jonathan Yurek]

To be honest, that confuses me, since I've tried adding gems that
depends on other gems not in my source list and it completely fails. I
understand that adding a namespace to the gemspec would be hard, but
it seems like the end result would be cleaner and more
straightforward, IMHO. Using subdomains feels a bit kludgy.

--
Jonathan Yurek, Founder and CTO
thoughtbot, inc.
organic brains. digital solutions.

617.482.1300 x114
http://www.thoughtbot.com/

[David Dollar]

I think the general idea is that gem forks should be a huge special
case. I'm somewhat of the mind personally that making gem forks too
'easy' causes a great deal of unnecessary fragmentation in the
community. It seems reasonable to me, that if your project is going to
depend on a gem fork, that the dependency resolution not be automatic,
and your installation instructions can tell the user how to get the
forked dependency.

If a fork is going to be long-term, or a true alternative, it should
probably be reregistered under a new name as a different project.

I know this opinion won't be shared by all, but it seems like easy
forking can start to impose a significant cost on the community
itself. Ever try to figure out which version of delayed_job to use?

[Jonathan Yurek]

That's a really good point, and I hadn't really thought of it like that.

--

[Wayne E. Seguin]

On Oct 8, 2:58 pm, David Dollar <ddol...@gmail.com> wrote:
> I think the general idea is that gem forks should be a huge special  
> case. I'm somewhat of the mind personally that making gem forks too  
> 'easy' causes a great deal of unnecessary fragmentation in the  
> community. It seems reasonable to me, that if your project is going to  
> depend on a gem fork, that the dependency resolution not be automatic,  
> and your installation instructions can tell the user how to get the  
> forked dependency.
>
> If a fork is going to be long-term, or a true alternative, it should  
> probably be reregistered under a new name as a different project.
>
> I know this opinion won't be shared by all, but it seems like easy  
> forking can start to impose a significant cost on the community  
> itself. Ever try to figure out which version of delayed_job to use?

I completely agree with David's perspective on this issue. So my vote
is for what he suggests.

~Wayne

[Gabe da Silveira]

On Oct 12, 10:02 am, "Wayne E. Seguin" <wayneeseg...@gmail.com> wrote:

> I completely agree with David's perspective on this issue. So my vote
> is for what he suggests.

+1 on David's comment as well. Gems with their versioning feel like
they should be canonical, whereas forking on github is so incredibly
easy for quick hacks and exploration. Throw gems into the mix and you
have something that is neither easy nor canonical.

[jdwy]

Yes, by all means. Forking should be difficult and a PITA. That way,
when I patch a something, it will be difficult to distribute and I
will have to hack around the gem system to get the patch to my
coworkers. It'll be just like old times on sourceforge, where projects
would sit derelict for years while willing contributors tried in vain
to get the reins to the canonical version. It'll be great!

Github-style easy gem building of forks is a core feature of our
thriving community, mess with the mojo at our collective peril.

[Nick Quaranto]

jdwy,

The subdomain solution we've come up with is just as easy as doing a `gem push` of your forked gem to your own subdomain. We're still working out the implementation details of this.

I'm not sure what you're missing here. Have you read this post yet? http://litanyagainstfear.com/blog/2009/10/09/on-gem-forking/

-Nick

[jdwy]

Don't think I'd read that at the time, I thought subdomains was still
up in the air.

I appreciate that you guys are thinking about this. I think where we
differ most is in your belief that things will always converge back to
a mainline gem. You use the term 'hopefully'. What exactly is the
fallback scenario? When _why disappears (eg) are we all left adding
the new maintainers subdomain? And what after that? I think the
decentralization of gems and correlation of code to gem was a big win
for the ecosystem in the long term and the github style was a graceful
solution.

We'll see, maybe I'm wrong, grabbing out domain will certainly help
with the basic bug fix use case.

On Oct 21, 5:47 pm, Nick Quaranto <n...@quaran.to> wrote:
> jdwy,
>
> The subdomain solution we've come up with is just as easy as doing a `gem
> push` of your forked gem to your own subdomain. We're still working out the
> implementation details of this.
>

> I'm not sure what you're missing here. Have you read this post yet?http://litanyagainstfear.com/blog/2009/10/09/on-gem-forking/

[Florian Gilcher]

Hi,

we just discussed this internally and came up with some issues.
Mainly, offering such system will be just asking for people wanting to
build such constructs and thus just generate frustration (especially,
because the dependency-managment in rubygems is really simple).

The problem is that you are discussing the case where such a gem is
the head of the dependency chain. Lets consider the following case: we
use an (imaginary) web stack that has the (imaginary) gem "mail-
delivery" as a dependency. Now, "mail-delivery" is cool, but there are
some customizations we need to apply to it. Those are incompatible
with the ideas that A has about the gem, so we have a reason for a
friendly fork. As my party B uses and supports this gem, there is a
reason for us to only use "mail-delivery" installed from http://b.rubygems.org
. Because we and others use the Web Stack W pretty heavily, but with
exception to our customizations, we have a reason to transparently
replace A by B.

Because Rubygems doesn't remember the source where it installed gems
from and cannot be pinned on specific sources for specific gems, we
suddenly have an update race.

There are other cool ideas that come to mind when talking about the
idea of having "your own" rubygems repository. For example: how cool
would it be to have a gem-repository that can be tuned to your liking?
Not including some gems while having some others? Basically handling a
stack as a repos. For example a repository of gems that are known to
work with 1.9, Jruby, Rubinius? Or one with gems that are older than 2
years and can be installed on a Debian system without fearing the
wrath of Lenny? On core, there was the discussion about a "blessed"
repository"... (Granted, all this could be handled by hosting seperate
Gemcutter instances)

I think that the idea of subdomains "that should be made easy", but
should still remain "a special case" is just asking for dangerous
ideas. It basically has the same problems as prefixing every gem with
a name. And the fact that the discussion between 2 people just came up
with some ideas that totally work against your ideas shows that it is
a problem that should be addressed with some care. Especially, because
every tiny bit of clunky flexibility will be used to full extend in
the ruby world. This could mar the beautiful simplicity of the system
while not adding much to it. This is basically running ahead of the
capabilities of the client.

If, on the other hand, someone is willing to take the endeavor and
actually tune rubygems for handling such complexity, I think that a
far better system can be derived.

Regards,
Florian Gilcher

[John Barnette]

On Fri, Nov 6, 2009 at 11:47 AM, Florian Gilcher <f...@andersground.net> wrote:
> Because Rubygems doesn't remember the source where it installed gems
> from and cannot be pinned on specific sources for specific gems, we
> suddenly have an update race.

This is by design. You might want to think really hard about this
before you decide that gems should be pinned to their originating
repo.

> If, on the other hand, someone is willing to take the endeavor and
> actually tune rubygems for handling such complexity, I think that a
> far better system can be derived.

I'm a RubyGems committer, and my RG repo is available for forking on
GitHub (jbarnette/rubygems). I welcome pull requests, but I think most
of the issues you've raised are straw men. Code talks, though, and I'd
be happy if you showed me the error of my ways.

~ j.

[Florian Gilcher]

You misunderstood me. I am not opposed to how RubyGems actually handles
things. It is a discussion to make, especially because handling it would
introduce a whole lot of complexity (both to handling and to the end-
user).
But it is still true that this decision - be it conscious or otherwise
- is
the reason for this race.

The point I am making is the following:
allowing subdomains as an "easy" feature for everyone would expose that
problem or make the subdomain feature an endeavor that is not really
worth it.
Having subdomains that could act as full gem repositories would be much
more interesting, because it would not expose this behavior as a flaw as
well as give the same benefits.

My problem is that this discussion is too much about intention and
proper
use, which are things you _cannot code_.

Regards,
Florian Gilcher