Moving the Joomla CMS to Git

288 views
Skip to first unread message

Joe LeBlanc

unread,
Jun 22, 2011, 7:24:40 PM6/22/11
to Joomla! CMS Development
I just listened to the latest episode of JoomStew. Andrew was talking
about the future of both the Joomla Platform and the CMS. Part of that
future is moving the CMS to Git on Github. I'd like to see how
comfortable the community is with Git and get an action plan together
for doing the migration.

Would anyone mind if I posted a quick survey here?

Andrew Eddie

unread,
Jun 22, 2011, 7:29:49 PM6/22/11
to joomla-...@googlegroups.com
I don't think there's any question now we'll be moving the CMS to git.
For now, I think we'd have to maintain both and then the question
becomes when do we retire subversion (and which is the point of truth
- git would be preferable).

Whatever the case, if someone wants to put their hand up to maintain
the git repo, I've already got stuff I want to fork :)

Regards,
Andrew Eddie
http://learn.theartofjoomla.com - training videos for Joomla 1.6 developers

> --
> You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
> To post to this group, send an email to joomla-...@googlegroups.com.
> To unsubscribe from this group, send email to joomla-dev-cm...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/joomla-dev-cms?hl=en-GB.
>
>

Joseph LeBlanc

unread,
Jun 22, 2011, 7:51:16 PM6/22/11
to joomla-...@googlegroups.com
Yes, the question is definitely not "if?" but "when?" By "comfortable," I was implying more "proficient."

Maintaining both simultaneously would be tricky though. I think this is the kind of thing where you have to say "we're doing it on date x" and then just do it. No matter when you end the Subversion support, you're always going to have someone out there who isn't ready yet.

I think a lot of people have moved to using Git for their personal projects already, but there are still people who haven't yet gotten practice with it. So I'd like to do a survey to essentially answer this question: if we moved to Git tomorrow, how many people would have trouble contributing?

-Joe

Michael Babker

unread,
Jun 22, 2011, 8:00:36 PM6/22/11
to joomla-...@googlegroups.com
Moved my code from SVN to Git over the last few months, and honestly, I
didn't find the learning curve to be too steep. GitHub's help is actually
really useful, along with all the internet resources. One repo has been
straight me working out of and the other until today had been a mirror
essentially of the SVN repo, kept in sync through git-svn rebase (not
difficult to set up either).

My two cents: use the time between the 1.7 and "next version" release to
transition to Git and once "next version" is released, the SVN is done.
From right now, that's about 7 months for devs to get comfortable with the
differences, new info to be added to the docs site, and complete the
transition.

Daniel Holmes

unread,
Jun 22, 2011, 8:04:28 PM6/22/11
to joomla-...@googlegroups.com

I am new to Joomla but I've been looking into sorce control for a while now and still don't fully understand the concept although i understand the reason for it in terms of Joomla.

I find that git seems better documented but more complicated and subversion is quite easy but the documentation lacking. For a new developer how do they come to understand the concept?

Joseph LeBlanc

unread,
Jun 22, 2011, 8:16:49 PM6/22/11
to joomla-...@googlegroups.com
My fear is that while you can set up SVN as a remote in Git, a bridge like that will get to be cumbersome when you have commits coming in from multiple people to both the SVN and Git repositories...

However, now that you mention it, we might be able to get away with leaving SVN as a "read only" repository. In this scenario, the only commits going back to SVN would be ones merged in over at GitHub. That way, if testers are used to reading from the SVN repo, they can still access that for a while, but all of the patches get written over at GitHub.

And yes, I think that schedule (start with Git after 1.7, phase out SVN after "1.8") sounds reasonable.

-Joe

elin

unread,
Jun 23, 2011, 1:13:17 AM6/23/11
to joomla-...@googlegroups.com
I'm sure it's a "when" but I think there are a lot of details to think through including the future of CMS issue management (and also how to take issue reports for the platform from people who aren't solving, since those are being managed currently in the CMS tracker). We are still going to need to work with patches and talking to users and we need to make sure to be able to link issue reports to commits. Then we also have to understand that in terms of number of people putting in code JBS is the largest group and they will be managing the 6 weeks after 1.7 according the the documents I've read. I would not want to change JBS procedures in the few weeks following a release. Then I would think at the point of alpha for the next release which will require merging in the then current version of the platform  and  which is when we will go into merging in features that would be a point to go over to github. Looking at the CMS features coming in for 1.7 some things I've noticed are that a lot of them are small and  make sense to manage as patches and the other is that there has been a ton of work in libraries, which I think we can hope will all be happening in the platform repository rather than the CMS.

Elin

Andrew Eddie

unread,
Jun 23, 2011, 1:22:11 AM6/23/11
to joomla-...@googlegroups.com
Well, I think it definitely is "when", not "if", but the process could
take 12 months or more (or less) depending on lots of other factors,
not least of which is the effect of any retraining burden on
volunteers.

I like Joe's suggestion about a first stage being to make SVN read
only - I think that's something to consider. But Elin's also right in
that shifting the code repo is just one aspect of a larger workflow
that needs to be considered. Issue management also has to be part of
the strategy.

Regards,
Andrew Eddie
http://learn.theartofjoomla.com - training videos for Joomla 1.6 developers

> --
> You received this message because you are subscribed to the Google Groups
> "Joomla! CMS Development" group.

> To view this discussion on the web, visit
> https://groups.google.com/d/msg/joomla-dev-cms/-/vCrPc-oz8vEJ.

Rouven Weßling

unread,
Jun 23, 2011, 6:30:58 AM6/23/11
to Joomla! CMS Development
I beg to differ, I think the first step is maintaining a sync of
current SVN on GitHub. I've played around with git-svn and I wasn't
able to get our svn repro into git while maintaining history. But I've
learned two things trough trying that would be great if they could be
taken into account by whoever will have the task to create the git
repro.

1. Maintain the info who committed things
We'll have to manually match svn usernames to email addresses used by
git/github. I know it's extra work, but it would be a shame to loose
that information like we did on the platform (for example
https://github.com/joomla/joomla-platform/commits/master?page=15).

2. Decide what parts of the svn repro will be taken over to github
I don't think the current structure of trunk/branches/releases can be
fitted into the git system. My suggestion would be to turn the
releases into git branches. We don't really need the normal branches
since that stuff can happen trough forks.

In theory once that repro is created in should be relatively
straightforward to sync it trough git svn if no one ever commits into
the git repro.

For step 2, making svn read only there's probably an even easier
solution you may or may not be aware of. For about a year github has
allowed access to the repros trough svn, not only read but also write
access however from my understanding write access is "unsafe". So once
those who have commit access feel comfortable to use git we could
switch there without too much disruption for the bug squad. They could
still use subversion to access the repro.

The only problem that could occur is different patch formats. While it
should be no problem to apply a svn formatted patch to a git repro, it
is, as as far as I know much harder the other way around. So once
people like me would like to start contributing in git patches we may
run into some trouble.

Best regards
Rouven

On Jun 23, 7:22 am, Andrew Eddie <mambob...@gmail.com> wrote:
> Well, I think it definitely is "when", not "if", but the process could
> take 12 months or more (or less) depending on lots of other factors,
> not least of which is the effect of any retraining burden on
> volunteers.
>
> I like Joe's suggestion about a first stage being to make SVN read
> only - I think that's something to consider.  But Elin's also right in
> that shifting the code repo is just one aspect of a larger workflow
> that needs to be considered.  Issue management also has to be part of
> the strategy.
>
> Regards,
> Andrew Eddiehttp://learn.theartofjoomla.com- training videos for Joomla 1.6 developers
>

Matt Thomas

unread,
Jun 23, 2011, 8:13:10 AM6/23/11
to joomla-...@googlegroups.com
Hi Joe,

To answer your original question, I have have completely dropped SVN in favor of Git. I do feel like Git is better supported (Github support is great!) and Git seems to be better documented, not to mention handles conflicts and merges better. I like the idea of clean break away from SVN, with keeping the SVN repo read-only for testers.

Best,

Matt Thomas
betweenbrain | Construct Unified Template Framework for Joomla! 1.5, 1.6, Molajo and Nooku Server



--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.

Joseph LeBlanc

unread,
Jun 23, 2011, 11:22:41 AM6/23/11
to joomla-...@googlegroups.com
Matt,

Agreed (with some of the caveats with patch testing mentioned by Elin and Rouven).


Daniel,

Welcome to Joomla, we're glad to have you here! I don't think Git is all that much more complicated than SVN, except when it comes to the concept of a commit. Commits in SVN are numbered sequentially and occur one right after the other. In order to make a commit in SVN, you *must* have access to the central repository, which keeps all of those commits in sequential order.

With Git, you have a full copy of the repository on your computer and you can commit any time you like. However, the commits stay on your computer until you decide to share them with other people. When you do a push to another repository (such as one on GitHub), it will attempt to merge your commits into the history. If there are no conflicts, your commits become a part of the history. Otherwise, you fix the conflicts, make another commit, and push again. The commits are not sequentially ordered, but it does keep a timestamp of when the commit was made so that the history makes sense.


Rouven,

I'd completely forgotten about the SVN access from GitHub; that does indeed make it one step simpler.

When converting the Joomla SVN into a full Git repository, I have run into the issue where I can only go back so far in history before it times out. This is largely because I have to do everything over the network and do not have a full copy of the SVN repository locally. If I did have a local copy of the SVN repo, it would convert over much faster with the full history.

1. The username/email address matching is indeed an issue. We should be able to extract that information from Joomlacode.org IF people have used the same email address for both Joomlacode.org and GitHub. Otherwise, we may need to do a "send us the address you want to use if you want credit" announcement.

2. Yeah, the 'releases' folder as branches would probably be best. And yes, the current 'branches' folder would work better as forks.

I don't think there's a point in creating a Git mirror of the repo if nobody is allowed to commit to it. How would anyone be able to effectively use it?

As far as the patch formats go, yes this is true. However, SVN users could still do a full checkout of your Git fork that includes the patch. Now yes, it would take longer to do the checkout than to apply the patch. However, I think we're ultimately looking at a situation where people are going to be pulling code from fork branches rather than applying patches from text files posted to trackers. For instance, I was able to apply some of your Joomla Platform patches just now by doing this:



So instead of posting a patch to a tracker issue, you would post a URL to a branch where the commits can be pulled. 

The SVN equivalent would be to check out your repo like this:


Unfortunately, it can only check out your master branch, so your patch would have to go into there.


Elin,

Given that scenario, yes, I agree that the merge point with the Joomla Platform after 1.7 release would be a better time to switch the CMS over to GitHub. Six weeks after the scheduled 1.7 release puts us two months away from a potential GitHub switchover. The Joomla Platform merge after that could be another potential one.

The remaining issue then is getting a new tracker workflow (and possibly new tracker entirely) into place.


Everyone,

I think given the interest in this, I will proceed with doing a quick Google Docs survey on people's familiarity with Git and GitHub. Would it be appropriate to cross-post it to the JBS list?

After the survey, we can then start evaluating workflows.


-Joe

Oliver Ratzesberger

unread,
Jun 24, 2011, 10:52:53 AM6/24/11
to joomla-...@googlegroups.com
Joe,

Thanks a lot for posting this up!

As Joe knows but for the rest of the mailing list: Kunena is also in the process of moving to github from the joomlacode svn. Any learnings we can share or contribute we'd be happy to do so. I understand that the risk goes up significantly with the size of a project. Kunena is small compared to Joomla itself, but the move is still significant. 

Thanks for all the contributions so far!

Oliver

On Jun 24, 2011, at 7:41 AM, Joseph LeBlanc wrote:

Joseph LeBlanc

unread,
Jun 24, 2011, 10:41:44 AM6/24/11
to joomla-...@googlegroups.com
-Joe

On Jun 23, 2011, at 11:22 AM, Joseph LeBlanc wrote:

Rouven Weßling

unread,
Jun 25, 2011, 7:24:44 AM6/25/11
to joomla-...@googlegroups.com
On 23.06.2011, at 17:22, Joseph LeBlanc wrote:

I don't think there's a point in creating a Git mirror of the repo if nobody is allowed to commit to it. How would anyone be able to effectively use it?

I'd certainly use it. It's a great way to work on some larger features. If the git repro stays in sync with trunk I could just run a diff between the two and submit that as a patch against trunk. (in theory). Also it's a step that would need to be done anyways, so why not doing it sooner rather than later. It would get people a chance to toy around with git.

As far as the patch formats go, yes this is true. However, SVN users could still do a full checkout of your Git fork that includes the patch. Now yes, it would take longer to do the checkout than to apply the patch. However, I think we're ultimately looking at a situation where people are going to be pulling code from fork branches rather than applying patches from text files posted to trackers. For instance, I was able to apply some of your Joomla Platform patches just now by doing this:



So instead of posting a patch to a tracker issue, you would post a URL to a branch where the commits can be pulled. 

The SVN equivalent would be to check out your repo like this:


In a previous discussion we had in the JBS Skype channel some people (including me) thought it would be less disruptive to the bug squad process if we keep doing patches for the beginning. Git has a lot more complexity and we have quite a few members who are completely new to SCM or don't use it for anything outside Joomla

Unfortunately, it can only check out your master branch, so your patch would have to go into there.

And that's the problem. The way I started to work with github on the joomla platform is to do everything in branches. I don't think there's any way around that because I'll usually have several pull requests or things I'm working on at the same time.

Best regards
Rouven

Joseph LeBlanc

unread,
Jun 28, 2011, 6:09:29 PM6/28/11
to joomla-...@googlegroups.com
Quick thanks to everyone who's participated in the survey so far! I've been on the road for the last couple of days and just now getting caught up on things.

Final call for anyone who hasn't yet participated: I'm going to give it another 24 hours, then tally up and publish the results. If you haven't yet participated, it will only take a minute: https://spreadsheets.google.com/spreadsheet/viewform?formkey=dHRkWE1vZ1lMbHVOVFlzZkQ3VkdtMXc6MQ

-Joe

Joseph LeBlanc

unread,
Jun 28, 2011, 6:06:28 PM6/28/11
to joomla-...@googlegroups.com
On Jun 25, 2011, at 6:24 AM, Rouven Weßling wrote:


On 23.06.2011, at 17:22, Joseph LeBlanc wrote:

I don't think there's a point in creating a Git mirror of the repo if nobody is allowed to commit to it. How would anyone be able to effectively use it?

I'd certainly use it. It's a great way to work on some larger features. If the git repro stays in sync with trunk I could just run a diff between the two and submit that as a patch against trunk. (in theory). Also it's a step that would need to be done anyways, so why not doing it sooner rather than later. It would get people a chance to toy around with git.


It sounds like you could do the same by using the native Git/SVN bridge. And I would you would have to convert it to Git again when you're ready for the cutoff. However, having an official GitHub would draw some interest and we could use it for training early on. So doing a provisional repo might end up being beneficial.


As far as the patch formats go, yes this is true. However, SVN users could still do a full checkout of your Git fork that includes the patch. Now yes, it would take longer to do the checkout than to apply the patch. However, I think we're ultimately looking at a situation where people are going to be pulling code from fork branches rather than applying patches from text files posted to trackers. For instance, I was able to apply some of your Joomla Platform patches just now by doing this:



So instead of posting a patch to a tracker issue, you would post a URL to a branch where the commits can be pulled. 

The SVN equivalent would be to check out your repo like this:


In a previous discussion we had in the JBS Skype channel some people (including me) thought it would be less disruptive to the bug squad process if we keep doing patches for the beginning. Git has a lot more complexity and we have quite a few members who are completely new to SCM or don't use it for anything outside Joomla

Fair enough.

-Joe

elin

unread,
Jun 29, 2011, 8:05:57 AM6/29/11
to joomla-...@googlegroups.com
The idea of JBS is that it should have extremely low barriers to entry. People can start out just confirming issues and don't need to know how to make or apply a patch at all. A lot of beginners may be working on a specific issue like untranslated strings and making diffs using an editor. We also have a lot of well thought out documentation about how to work with SVN.  And last but not least the fact that we encourage people to use Eclipse and eGit has less than ideal integration with github when working from a fork of a remote repo (as opposed to one you start locally) creates some challenges. 

Elin

Joseph LeBlanc

unread,
Jul 2, 2011, 5:11:53 PM7/2/11
to joomla-...@googlegroups.com
Hey everyone,

I've compiled the results of the Git survey. I'd like to discuss some specifics a little later, but have a pool party to run off to ;) The majority of the people who participated are actively using Git and supportive of the move. Most of the concerns are about the Bug Squad workflow.

After keeping the survey open for almost a week, 86 people responded. The complete results can be viewed here: https://spreadsheets.google.com/spreadsheet/ccc?key=0AsADis6j-2VldHRkWE1vZ1lMbHVOVFlzZkQ3VkdtMXc#gid=0

Bear in mind that this is by no means a scientific survey (nor was it intended to be one). This is to say: everyone who participated in the survey did so voluntarily, rather than be selected by random sampling. The intent was to get a rough picture of the awareness and comfort level with Git in the Joomla.

Almost half (41) of the responses came from people who write patches. The same number of people are also members of the Joomla Bug Squad (although not necessarily the same 41). Slightly over a third (30) only report issues on the tracker and do not test patches.

While the majority actively use Git in at least one project (53), there's still a significant number of people who've either tried it a couple of times, or not at all. The vast majority (69) have been to GitHub.com, and 57 have downloaded or otherwise worked with code there.

Nearly half (41) would be immediately functional if Joomla switched to Git today, with the majority (59) at least being able to figure it out quickly. A small portion (4) would find a switch difficult.

Almost every response was from either a programmer or someone who does a little bit of everything.

The comments revealed a lot of enthusiasm over moving to Git as soon as possible, along with some concerns and questions over what the new workflow would entail. Comments in favor of a speedy move cited the following points:

* Ease of contributions under Git
* The need to have the Joomla Platform and CMS maintained consistently
* Ease of branching
* Would encourage more participation from Git users
* Joomlacode.org is dated
* Windows clients are now plentiful

There were also some concerns about the move:

* Desire to keep using Joomlacode for 3PD extensions
* The need for updated tutorials
* Whether or not GitHub has sufficient issue tracking/sorting for a project of Joomla's size
* The ability to follow repository histories after merges
* Numerous GitHub "forks" might make merges more complicated
* The need to rewrite SVN maintenance code

Several responses came from people who were unfamiliar with Git, or were unsure of what a move to Git would entail.

Reply all
Reply to author
Forward
0 new messages