switching to git

[Bess Sadler]

Dear Blacklight Community,

The committers on the project have been discussing for some time now a move to git from subversion for source code management. The time has come to make that switch. Moving to git will allow us to more easily branch and merge changes, which will make it easier to have a greater number of people working on the code base at once without stepping on each other's toes. It will also make it easier for folks who aren't official committers to the project to make code contributions, and for us to accept them and merge them into the core code base. 

The new source code repository will be available at http://github.com/projectblacklight

We will make the switch on Wednesday the 9th of September. Before that time we hope to have re-written the documentation for checking out source code, re-configured our hudson continuous build system to use git instead of svn, and re-written our capistrano deploy scripts that come with the demo app, but some of this may lag behind the actual switch. Please bear with us as we make the change, and help us out by reporting any places where we've missed documentation updates.

If you haven't used git yet, consider setting up a git account now and playing around with it. You can get a free account at github.com, and if you follow projectblacklight you can choose to be notified of any changes that occur to that project. If you're currently on our subversion commit changes list, github services will be replacing that list, and I'll send out a separate email about how to make sure you continue receiving notifications of code commits. If you need an introduction to using git, I highly recommend Pro Git by Scott Chacon, which is available both in print and online at http://progit.org/. 

Thanks for being such a fantastic community!

Bess Sadler

release manager for project blacklight

[Jonathan Rochkind]

Awesome, this is great.

When the plugin is in svn, I believe that, without actually checking it
out from svn, there is some way to use standard Rails rake tasks to
'install' the plugin from svn, and then later to 'update' to latest
version, etc? It was on my list to investigate actually HOW to do this,
since I hadn't figured it out yet, but believed it was possible. rake
gem update or something?

Is there a similar way to do this when the thing is in git? It would be
awesome if any updated instructions could cover this too. Unless I'm
completely confused and what I'm talking about doens't make any sense,
which happens to me from time to time. :)

Jonathan

[Vernon Chapman (g8tor)]

Hello All,

I am going to be working on the move to GitHub.
While fretting over the move, something dawned on me.

With the current proposed setups

(blacklight-demo with plugin) http://github.com/projectblacklight/blacklight-demo/tree/master
(blacklight-plugin only) http://github.com/projectblacklight/blacklight-plugin/tree/master

a potential user would only be able to get the master branch or trunk as you will.

I think it would be major turn off if I had to go someplace else to
download the current release. So I've been toying with the following plan of action

1- export everything from the svn repo
2- remove branches,tags etc. (all other cruft)
3- move releases/[demo-app | plugin]/* -> tags
4- rename all tags to just a version nummber i.e 2.2 or 2.3
5- import trunk/tags/branches into a local svn repo
6- use svn2git to convert to a git repo
7- clone from local git repo to break from svn
8- switch to newly cloned repo
9- remove the current [remote origin] section in .git/config
10- git remote add git@git...........
11- git push origin master

I've tried it on my personal account under the
http://github.com/chapman/blacklight/tree/master repository
and now the releases are git tags.

Does this sound correct? Is this the correct useage of git tags?
Am I on the right path or am I totaly off base?

Things to note here.

1 - The new tags/releases should mirror each other, i.e. in the blacklight-demo and blacklight-plugin repositories there should be matching tags.In this case that would leave us with a 2.2 & 2.3 tag in each

2 - The newly created tags will remain as is. i.e the blacklight-demo will still have the plugin installed by default. The plugin will be at whatever version it was when that tag/release was originally created.

Thanks

Vernon a.k.a. g8tor

P.S.

IT'S GRATE 2 BE A FLORIDA GATOR!

[Ron DuPlain]

On Wed, Sep 2, 2009 at 3:33 PM, Vernon Chapman (g8tor) <chapma...@gmail.com> wrote:

1- export everything from the svn repo
2- remove branches,tags etc. (all other cruft)
3- move releases/[demo-app | plugin]/* ->  tags
4- rename all tags to just a version nummber i.e 2.2 or 2.3
5- import trunk/tags/branches into a local svn repo
6- use svn2git to convert to a git repo
7- clone from local git repo to break from svn
8- switch to newly cloned repo
9- remove the current [remote origin] section in .git/config
10- git remote add git@git...........
11- git push origin master

You may need to:
12 - git push --tags

http://github.com/guides/push-tags-to-github
 

I've tried it on my personal account under the
http://github.com/chapman/blacklight/tree/master repository
and now the releases are git tags.

Does this sound correct? Is this the correct useage of git tags?
Am I on the right path or am I totaly off base?

I'd say your on the right path.

Your test repository tags worked for me.  To test, I ran:

$ git clone git://github.com/chapman/blacklight.git
Initialized empty Git repository in ...
...
$ cd blacklight/
$ git tag
0.2
2.0
2.1
2.2
2.3
$

From there, I could `git checkout 2.0`.

Things to note here.

1 -  The new tags/releases should mirror each other, i.e. in the blacklight-demo and blacklight-plugin repositories there should be matching tags.In this case that would leave us with a 2.2 & 2.3 tag in each

2 - The newly created tags will remain as is. i.e the blacklight-demo will still have the plugin installed by default. The plugin will be at whatever version it was when that tag/release was originally created.

Note that there are lightweight tags and annotated tags.  You'll want annotated tags (which it looks like you've used).  You might consider also signing your tags.

http://progit.org/book/ch2-6.html

 

Thanks

Vernon a.k.a. g8tor

P.S.

IT'S GRATE 2 BE A FLORIDA GATOR!

Will you be using GitHub for building RubyGems for the blacklight plugin?
http://gems.github.com/

I'm guessing that's a benefit/reason of why you'd host at github.
Good luck in your transition,

Ron


PS - Hi, I'm Ron.  I'm working with Solr at NRAO in Charlottesville, and I've been very interested in Blacklight for a few months now.  I met some of you at beCamp back in May.  If I'm on IRC, I'll be rduplain.

[Vernon Chapman (g8tor)]

Ron,

Thanks you are absolutely correct.
Other than that how does that sound?

I'm about to try it again under my account I have blacklight-demo
andblacklight-plugin.
I'll try that later and see how it works

Vern

> <http://github.com/chapman/blacklight.git>

[Ron DuPlain]

On Wed, Sep 2, 2009 at 6:30 PM, Vernon Chapman
(g8tor)<chapma...@gmail.com> wrote:
>
> Ron,
>
> Thanks you are absolutely correct.
> Other than that how does that sound?
>

Are you planning on migrating the entire commit history?

I have not used svn2git, but I understand it uses the git-svn package.
I have used git-svn and you can migrate an entire history of svn commits to git.

However, with projectblacklight this will make a git repository that
is over 200MB, getting close to the 300MB github free account limit.
I tried `git svn clone --use-log-author -s
http://blacklight.rubyforge.org/svn blacklight` and I hit 244MB (which
could be condensed a bit with git repack and gc). A repository that
is 70%+ of the free account limit could be a bit of a concern when
forking on github.

More than half of the repository is vendor packages, and storing
various revisions over time adds up. I'm sure there's a way to rebase
some of the commits to keep the repository size down, but it's almost
always a bad idea to change history after pushing to a public
repository location. So, while you are migrating, you might want to
consider:

Do you want to preserve every commit, or just enough to support the
tags and branches you want to keep?

Also, at some point in your migration, you may want to copy over svn ignores.
I'd be happy to help in anyway I can.

Ron

[Bess Sadler]

On 6-Sep-09, at 9:30 PM, Ron DuPlain wrote:

> Do you want to preserve every commit, or just enough to support the
> tags and branches you want to keep?

My vote is to only keep enough to support the tags and branches we
want to keep. I can't see a reason to keep every commit.

> Also, at some point in your migration, you may want to copy over svn
> ignores.
> I'd be happy to help in anyway I can.

Thank you, Ron. It's great to see a new face in the community. Welcome!

Bess

>
> Ron

[Vernon Chapman (g8tor)]

Hi All,

I think Ron makes some very good points about the history and the size
of the projects.

I agree with Bess we should keep as much of the history as needed or as
possible to
support the tags that we are keeping which at the moment are 2.2 and
2.3. I also agree with Bess
welcome to the community Ron.

Ron, I attempted to use a straight git-svn clone, but that presented
some problems.

First, was that the way git-svn handled the tags, it made them branches
i.e. tags/blacklight-2.X it didn't make them git tags
Second was that it kept a "link" back to the svn repository. I think we
are trying to leave svn behind. As I said above, I think we probably
should make an attempt to move the supporting history to support the
tags we are keeping but I'm not sure how we could do that mainly because
of #2 below.

With all that I'm a little confused, so Bess , Ron or (Insert your name
here) please help.

1 - Isn't git svn for bidirectional operations between a Subversion
tree and git repo?
Doesn't that keep them linked with the svn tree being the central
repository still?
I thought the idea was that we were making a clean break from svn?

2 - (Ron) I'm not sure how we would keep the history, being that at
current the two projects are
combined in the svn trunk. I'm not sure how we would separate the
histories so that blacklight-demo history went with
blacklight-demo etc..

3 - Ron makes a great point about the size of the repository and the
vendorized gems. I was thinking that we could save some space on the
repository if we made most of those vendorized gems submodules of
the blacklight-plugin. Any objections concerns? I have been able
to locate git repos for 11 of the 16 vendorized gems here is the list:

rack
rdoc
haml
marc
ruby-xslt

git://github.com/binarylogic/authlogic.git
git://github.com/taf2/curb.git
git://github.com/mislav/hanna.git
git://github.com/mislav/will_paginate.git
git://github.com/mwmitchell/material_girl.git
git://github.com/mwmitchell/rsolr.git
git://github.com/mwmitchell/rsolr-ext.git
git://github.com/tenderlove/nokogiri.git
git://github.com/dchelimsky/rspec.git
git://github.com/dchelimsky/rspec-rails.git
git://github.com/brynary/webrat.git

4 - (Ron) I was never a big svn user so If you would like to handle the
copying over of the svn ignores that would be great. I'm not sure what
you might need from me but just let me know.

Whatever the outcome, I would like to have a finalized plan soon or we
can push the move back (I'd rather not but I'm one person )until we have
a better plan that everyone can live with.

Thanks

Vern

P.S. How about those GATORS!

[Vernon Chapman (g8tor)]

Hello All,

T minus a little less than 24 hours before we actually move to GitHub.
(I think).

I just wanted to reply to Ron's email about the subversion commit logs.
I believe that
Ron and Bess make good arguments for keeping at the very least enough of
the svn.log
to support the releases that will be moved to git. (I believed we
decided we would move the
2.2 & 2.3 releases of both the demo and the plugin)

I guess my issue is how do we do that? The fact is that the two projects
have been linked since
the initial import on 2008-01-28 10:58:18 -0500 (Mon, 28 Jan 2008), with
almost two years of
commits it would be a painstaking job to extract the logs for each
project (demo & plugin).

In my opinion, we should simply export the log using svn log or svn log
-v zip it up place it in
the docs folder and name the file blacklight-subversion.log (or
something) put a note in the README
and make a clean break from svn. That way if someone really is
interested in some historical decision
etc, it is right in the project files. I did a test and svn log yields
a file of 404k while svn log -v yields a
file of 5.5M.

In my opinion doing it this way gives us the best opportunity to make a
clean break from subversion,
clean up the cruft in the project and get a fresh start on GitHub.
Frankly, I believe it would be way too
much work for very little reward.

On another note, I want to make sure I'm thinking of this correctly.
According to the README in the svn root, the branches folder contains

* code for specific releases, should only be used for fixing minor
bugs within a release and

* code for refactoring efforts that will be merged back into trunk

At this point these bug fixes etc have been merged into trunk, so there
is no reason to bring these over.
If I'm wrong please let me know.

I believe that we are going to take the official releases from the
svn/releases directory and make them the git tags,
so that people can checkout a specific version. As I stated above we
decided to bring over versions 2.2 & 2.3 since there
are corresponding releases for both the demo-app and plugin. (In theory
we could make a 2.0 & 2.1 release for the plugin by
exporting the vendor/plugin/blacklight from the demo-app's 2.X release.)

Some other question:
1 - What will become of the archive (clearly this does not belong in
the new repositories?)
2 - Are the svn/blacklight_import and svn/dl_importer candidate for a
third/fourth repository?
3 - What about the svn/javascript directory, I think theses files
are already included in the plugin so no need to pull them over as well?


Finally, I plan on doing this late night tonight so when we wake up in
the morning it will already be done. However, I'm going to need some
clarity and direction from the commiters and community as a whole. I
will be making test runs of the procedure most of the day, so
I need some sense of what people want to do.

Thanks

Vernon

P.S.

Sorry for the lengthy brain dump post it was a long weekend.
Oh and IT'S GREAT 2 BE A FLORIDA GATOR!

[Matt Mitchell]

To begin, why don't we just get trunk (and a log file) over to github? Do a clean svn export and test that everyone can pull/push etc.. From there, honestly I don't see why we would need to put anything other than trunk there. Our current/past releases and tags should just stay on rubyforge no? What would be the value in bringing them over to github? I'm probably missing something :)

The javascripts directory can stay on rubygforge. We're not using any of that stuff. The importer directories, that's very UVa specific so I can't imagine a need for that going to github.

Matt

[Vernon Chapman (g8tor)]

Matt,

I appreciate the quick response on this.

I'm not opposed to just bringing over the trunk & releases and leaving
everything
else on rubyforge.

I think the main benefit of bringing over the releases is that ,
potential adopters may not
be comfortable working on the bleeding edge. I think it gives potential
adopters a sort
of safety net, because they know that these tags/releases have already
been tested in the wild.
Secondly, if I had to go track down an official release of a project on
a secondary site
that would totally turn me off (then again I'm lazy as they come).

One other thing, Ron did make a good point about the size of the repos,
what do you think about
making 11 or the 16 vendorized gems submodules of the blacklight-plugin?
That way we save space
on our free account. I know this will add more steps to the setup
process for potential adopters but I think
the space savings are worth it.

Anyone else have any comment on the vendorized gems and space issues?
Here is a list of potential submodules
we could add to blacklight-plugin. The first 5 I have been unable to
locate a git repo for, and the rest are the public clone
urls for the 11 candidates:

rack
rdoc
haml
marc
ruby-xslt

git://github.com/binarylogic/authlogic.git
git://github.com/taf2/curb.git
git://github.com/mislav/hanna.git
git://github.com/mislav/will_paginate.git
git://github.com/mwmitchell/material_girl.git
git://github.com/mwmitchell/rsolr.git
git://github.com/mwmitchell/rsolr-ext.git
git://github.com/tenderlove/nokogiri.git
git://github.com/dchelimsky/rspec.git
git://github.com/dchelimsky/rspec-rails.git
git://github.com/brynary/webrat.git

Thanks

Vernon

GO GATORS!

[Ron DuPlain]

On Tue, Sep 8, 2009 at 9:54 AM, Matt Mitchell<good...@gmail.com> wrote:
> To begin, why don't we just get trunk (and a log file) over to github? Do a
> clean svn export and test that everyone can pull/push etc.. From there,
> honestly I don't see why we would need to put anything other than trunk
> there. Our current/past releases and tags should just stay on rubyforge no?
> What would be the value in bringing them over to github? I'm probably
> missing something :)
>

It's useful to be able to diff (or otherwise browse changes) against
releases. All git commands with <commit> arguments can take tag names
instead of sha commit names. This is neat for commands like:

* `git diff 2.0` - get a diff on all files since 2.0 release
* `git checkout 2.0` - get a working tree of the 2.0 release

So instead of just getting a single commit trunk, you could have a few
tagged commits for the releases of interest, to provide git
interaction against them.

-Ron

[Matt Mitchell]

OK the new user scenario makes sense. Good point.

I've never used submodules before. Is that essentially creating a reference to another repo? Something like svn externals?

For the gems, I really don't know. I guess I'm looking at that big list of vendorized gems and am thinking the same thing could be accomplished by specifying the gem+version as a dependency, then remove it from the repo. But I know we've already had this discussion and clearly we all agreed to vendorize this stuff.

Matt

[Ross Singer]

I ran into problems with the hulking size of the BL vendor directory
on Heroku, as well (I wound up just completely ripping it out -- you
specify the gem versions, anyway).

Perhaps rather than bundling all of the dependencies in the project
itself we can have a rake task that vendors the gems for the actual
installation?

-Ross.

[Matt Mitchell]

Ron, thanks for this explanation.

Matt

On Tue, Sep 8, 2009 at 10:43 AM, Ron DuPlain <ron.d...@gmail.com> wrote:

[Matt Mitchell]

Makes me wonder if we should NOT vendorize gems, and use a Rails Template (http://railscasts.com/episodes/148-app-templates-in-rails-2-3) for installing Blacklight instead. My co-workers and I have been working with Rails Templates lately and they're so slick. Forget about going through a big README file, just copy and paste a single command and you're done.

Matt

[Ross Singer]

On Tue, Sep 8, 2009 at 10:44 AM, Matt Mitchell<good...@gmail.com> wrote:

> But I know we've already had this discussion and clearly we all agreed to
> vendorize this stuff.

I understand the need to vendorize -- I don't really see the need to
store them in SCM, though.

What's the rationale for that?

-Ross.

[Matt Mitchell]

Not sure actually, and I agree with you. Also there are C based gems there, which really should be built on the system they're running on right?

Matt

[Vernon Chapman (g8tor)]

Ross,

I think that it's simply the way it has been done in the past and it works.
I mean think about it we are a little less than 24 hours away from
attempting to move
and we are just talking about it now. I think the other reason is that
it lowers the entry
level bar, that way people don't have to go searching for this gem that
gem and the
other gem (specific versions etc).

I think Matt's idea on the rails template is a good idea ( I had a
project a while back on github that explored this option
but I borked it) I think someone actually cloned it though.

Matt do you already have something that would work or were you just
throwing
that idea out there for consideration?

Vernon

[Matt Mitchell]

I don't have anything for Blacklight, but I do have a working "Consuminator" template:

http://github.com/mwmitchell/consuminator/tree/master

and this is the source:

http://github.com/mwmitchell/consuminator/raw/master/template.rb

Where was your template?

Matt

[Ross Singer]

Vernon, my point wasn't to gripe about it -- I mean, I understand the
historical reason for vendorizing and storing them with the plugin.
It's just, as you point out, sort of gotten out of control. The
plugin takes FOREVER to download or move (which is especially painful
given that I already have many of the gems locally at the same
version).

Now that we've got the chance to improve it, why not?

The Rails templates look really cool -- this is exactly the sort of
thing I was referring to when I mentioned the rake task.

I say we go with whatever we can churn out the fastest.

-Ross.

[Jamie Orchard-Hays]

yup

[Jamie Orchard-Hays]

Using a Rails template might simplify the whole installation process. It'd be worth looking into. 

You'd need one for the plugin and one for the demo, right?

[Jonathan Rochkind]

My impression is that the idea of actually packaging gems with Rails
source pre-dates the easy way to build gem dependencies in Rails 2.x.
I think now that there's an easy built-in way to have gem dependencies
with one-line installation of dependencies (including native C gems,
which you couldn't package the built gem with the Rails app anyway),
there's probably nothing to recommend actually packaging the gems with
the Rails app/plugin.

Jonathan

[Matt Mitchell]

Yeah that makes sense. One for both.

Matt

p.s. if anyone tried to install "Consuminator" you would have ended up with an error. That's fixed now - so try out the template if you'd like.

[Vernon Chapman (g8tor)]

Ross,

Sorry I didn't think you were complaining at all. I wasn't bashing you
in fact, I agree with you 100%
(Sorry if it didn't come through that way damn emails). Seriously I
think this is an awesome idea.

I also agree that now that we have the opportunity to right the wrongs
so to speak we should.

Vernon

[Vernon Chapman (g8tor)]

Matt,

I think it was at http://github.com/chapman/blacklight-template but I
deleted it a while back

[Bess Sadler]

It seems to me we might want to postpone the switch to git until next
Wednesday. That will give us more time to work on this, and I haven't
had a chance to re-write any of the docs yet.

Vernon, you're the driving force in this. How would postponing the
move for a week be for you?

Bess

[Vernon Chapman (g8tor)]

Bess,

I'm up for whatever.
I just want to get it done right.

I'm not too sure, next week is the week. My Boss will be out of town and
I will be acting in her place. Meetings etc so I'm not sure it would be
good,
the week after no problem.

In light of the lively discussion this has generated, I vote we postpone
it till then.
However, in the mean time can we come to some kind of consensus on what we
are going to do?

I say bring over the 2.X releases (as is) as git tags along with the
trunk as master,
add a doc folder (along side jetty & rails in the demo and the top
level of the plugin)
with a README explaong the log situation. Finally remove the vendorized gems
to save space on the git account and replace them with Matt's template
(http://gist.github.com/183080).
(I'm probably missing somethings but that is the gist of it )

What say you blacklight?

Vernon

[Ross Singer]

On Tue, Sep 8, 2009 at 1:46 PM, Vernon Chapman (g8tor)
> What say you blacklight?

+1

It doesn't seem like time is so of the essence that it cannot wait 2 weeks.

-Ross.

[Matt Mitchell]

+1 on wait and do it right

[Bess Sadler]

Vernon just pinged me on chat and he needs a decision, and it sounds to me like the motion has been carried (four in favor, none opposed). Our new target switch date is September 23. Sound right? 

Bess

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Blacklight Development" group.
To post to this group, send email to blacklight-...@googlegroups.com
To unsubscribe from this group, send email to blacklight-develo...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/blacklight-development?hl=en
-~----------~----~----~----~------~----~------~--~---

[Naomi Dushay]

On Sep 8, 2009, at 10:46 AM, Vernon Chapman (g8tor) wrote:

>
> Bess,
>
> I'm up for whatever.
> I just want to get it done right.

+1 on wait and do it right.

>
> I'm not too sure, next week is the week. My Boss will be out of
> town and
> I will be acting in her place. Meetings etc so I'm not sure it would
> be
> good,
> the week after no problem.
>
> In light of the lively discussion this has generated, I vote we
> postpone
> it till then.
> However, in the mean time can we come to some kind of consensus on
> what we
> are going to do?
>
> I say bring over the 2.X releases (as is) as git tags along with the
> trunk as master,
> add a doc folder (along side jetty & rails in the demo and the top
> level of the plugin)
> with a README explaong the log situation.

sure. The lack of old releases, and old log messages, will become
less and less important as time goes by. We are an active project
doing lots o' commits.

> Finally remove the vendorized gems
> to save space on the git account and replace them with Matt's template
> (http://gist.github.com/183080).
> (I'm probably missing somethings but that is the gist of it )

+1
we do want to make sure it works, and we also want to make sure the
instructions are simple and clear so the barrier to get the blacklight
demo and/or the blacklight plugin up and running is as low as possible.

- Naomi

[Jason Ronallo]

Yeah, I've had a change of heart about Blacklight vendorizing all of
its gems. I'm increasingly of the opinion that config.gem pegged to a
version is easier to deal with and faster to deploy. I end up having
to rebuild the gems on the remote system in any case. I would prefer
that we not specify gem versions using greater-than but be specific
about a version that is known to work.

Another project I'm working on I didn't specify the version of rsolr
and will_paginate and the app broke with the most recent updates to
those two libraries.

So +1 for ripping out the vendorized gems and having the benefit of a
more lightweight repo.

Jason

[Vernon Chapman (g8tor)]

September 23rd sounds good to me.

[Jamie Orchard-Hays]

+1 from over here on the sidelines.

[Matt Mitchell]

yep sounds good!

[Vernon Chapman (g8tor)]

Hello All,

Thanks to some awesome work on the install template by Matt ,
moving over to Git is becoming a reality. In recent days, the need
for the demo application has been called in to question. This has
raised the
question of whether the old organization of the code base (separate
demo and plugin)
has as much relevance in our new Git world.

During today's commiters call, we agreed that we do not see a need to
bring the older releases of blacklight over to the new git repository.
We feel that making a clean break from subversion will make the
transition easier.

To be clear, the current site (blacklight.rubyforge.org) would still be
available, for those needing
older versions of the demo application as well as the plugin.

With that said, we would like to open the floor to anyone that might
have an
objection to such a move. It is imperative that if you have an objection
to us leaving the
older versions of blacklight behind that you voice you opinion so that
your objections
can considered before a decision in made.

Thanks

g8tor
**

[Jamie Orchard-Hays]

I think I have a historian in me because I like have all that history.
Having said that, if we don't need it, chuck it! It's all there on
Rubyforge anyway.

Jamie