cluster? Multiple Mirrors?

[timon]

I have a few distributed sites and we are using gitorious as a content
repository in each sites.

Initially we had a central gitorious host and things were working
quite well, however recently we brought up so remote locations and
with these remote locations came increased latency for git pulls.
Admittedly we are probably not using git entirely correctly. a few of
the repositories we have a large (in and around the GB mark).

What I envision, but do not know if it is possible is a "cluster" or
"mesh" or what ever term fits. Let me try to describe how I would
like the functionality.

I would like to have 3-4 peer gitorious nodes in synch with each
other. i.e. if I clone a repo out to a machine, make changes to the
content of the repo and commit and push back to the gitorious node, I
would like that change automagically synchronized to the other 2-3
nodes.

I have been googling for a while and haven't really found any strong
leads on how to do this or whether it is possible.

two ideas that I have had, are either post commit hooks, or perhaps an
activeMQ mesh underlying the gitorious nodes.

am I on the right track with either of these options?

[Marius Mårnes Mathiesen]

On Thu, Sep 23, 2010 at 6:23 PM, timon <john.a...@gmail.com> wrote:

I would like to have 3-4 peer gitorious nodes in synch with each
other.  i.e. if I clone a repo out to a machine, make changes to the
content of the repo and commit and push back to the gitorious node, I
would like that change automagically synchronized to the other 2-3
nodes.

Interesting! We just discussed something similar at the office today; I'm not convinced the centralized model is the only way to share work in a decentralized version control system. When challenged by my colleague, however, I couldn't really come up with any really good use cases :-/

Having the different "remotes" know of each other could have some interesting uses: for instance if the installation I'm currently working with knows of a few other Gitorious installations and someone searches for eg "ruby fuse", it could be interesting if the server would tell me that "I don't have any such projects, but I know of this other Gitorious installation which has something you may be interested in..."

The Gittorrent project came up with a MirrorSync concept: http://code.google.com/p/gittorrent/wiki/MirrorSync

One thing to keep in mind is that the top priority feature we're working on right now is to add support for several git backend servers. The way Gitorious works today, the web app needs direct (disk) access to the Git repositories - which means that there's a very real limit to how much load a single Gitorious installation can handle. Gitorious.org runs on some pretty decent hardware, but we'll be getting to a point soon where we'll need to distribute the workload on more boxen. Now, this feature is probably not of interest to most Gitorious installations, but we will be introducing some kind of networked protocol to replace Grit's direct disk access. 

The SSH and Git traffic is a lot easier than the web stuff, since Git can easily be proxied (Gitorious.org uses proxymachine for this, which means the backend could be on another host) and the SSH traffic could just pass through the frontend server. But once this is done, the repositories could reside on several physical servers as long as these speak the protocol Gitorious will use. Would this solve your issues?

Cheers,

- Marius

[timon]

On Sep 23, 12:59 pm, Marius Mårnes Mathiesen
<marius.mathie...@gmail.com> wrote:

I don't know if this will solve my issue :) since I am not sure I
fully understood it. I am going to have a look at gittorrent that you
referenced above and proxymachine.

I am going to try and rephrase my original question with the hopes
that as I try to re-explain what I think would be a solution to my
problem I will better understand it and you would be able to determine
if what you propose is in fact a solution.

I have a gitorious server in North America, it has a few git
repositories on it. some of which are large. no if I have a group of
machines in say, the UK somewhere and I need each of these machines to
pull down local replicas of these repositories. in the interest of
decreased latency and fewer pull across the pond. I would like to
have a replica/mirror on the local lan in the UK that these machines
could pull from. However any changes that get made on any of these
given machines, I would like the push to be to the gitorious server in
NA and then have that delta trickle to the replica in the UK.

so in short, pulls are local to which ever region you are in, and
commits always get pushed to a 'master' and replicate to the other
mirrors.

I guess a push could go to the local server as long as the change gets
synchronized to the other servers.

--
thx

JT

[Alberto Negri]

I guess those are different tasks that have to be accomplished by different softwares.
i.e. git to store source code and rsync to sync mirrors...

my 2 cents
Alberto

--
To post to this group, send email to gito...@googlegroups.com
To unsubscribe from this group, send email to
gitorious+...@googlegroups.com

--
http://negri.web.cs.unibo.it/