Replication problems
Fredrik Luthander
Some background on a problem we're seeing:
When replicating from our master gerrit server to the slave in a LAN
location, it seems open sshd gets flooded or something, because it
stops working after a while and it generally requires you to restart
the entire machine once it happens. It's only valid for server in the
same geographical office as the master, other server where never
affected by this.
We were never really able to troubleshoot this properly, and instead
we implemented a work around for this:
We sync over git: to our local slave on the master site. We have a git
daemon open for traffic from our master host only, and it replicates
new changes over this protocol to the slave.
Generic look of our replication.config:
[remote "mirror"]
url = git://host.intra.net/${name}.git
This way of replicating works fine, and with plenty of speed, but
introduces some new problems as well, however.
Because suddenly create-project doesn't work very well, the git's
don't appear on this particular slave any more, we have to initialize
them manually over an ssh link.
We've so far solved this by replacing the create-project with a
scripted command that creates the project for us at all hosts,
registering it in the database. It also optionally creates the project
with an initial commit as repo can't handle a project in the manifest
that doesn't already have at least one initial commit (thus making it
clonable).
So, in the situation we are right now, my initial thought was to
suggest to split the URL-value under the remote section in two parts,
one syncurl (which could be any valid protocol, git: for instance) and
one administrative URL, which would then have to be an URL offering a
shell or corresponding possibilities to execute commands on the remote
host, thereby enabling a way to initialize gits and perform any other
administrative tasks for the future.
However, on second thought I seem to remember us in the past
discussing whether the slave itself should initialize new projects,
perhaps that would be a better way to handle this. It is clear that
sync over git: isn't really an option today unless you have supportive
scripts on the side.
Any comments on this? I'll be happy to file new issues, but I'd rather
file them on the right thing. Do we have anything in the pipe on this
that I'm unaware of?
BR,
Fredrik
Shawn Pearce
<fredrik....@sonyericsson.com> wrote:
> When replicating from our master gerrit server to the slave in a LAN
> location, it seems open sshd gets flooded or something, because it
> stops working after a while and it generally requires you to restart
> the entire machine once it happens. It's only valid for server in the
> same geographical office as the master, other server where never
> affected by this.
Yikes. That sounds bad.
> We sync over git: to our local slave on the master site. We have a git
> daemon open for traffic from our master host only, and it replicates
> new changes over this protocol to the slave.
>
> Generic look of our replication.config:
> [remote "mirror"]
> url = git://host.intra.net/${name}.git
>
> This way of replicating works fine, and with plenty of speed, but
> introduces some new problems as well, however.
> Because suddenly create-project doesn't work very well, the git's
> don't appear on this particular slave any more, we have to initialize
> them manually over an ssh link.
Yea, that is a downside to using the anonymous git protocol.
> However, on second thought I seem to remember us in the past
> discussing whether the slave itself should initialize new projects,
> perhaps that would be a better way to handle this. It is clear that
> sync over git: isn't really an option today unless you have supportive
> scripts on the side.
I think we would prefer to do this. We should be able to use the
master's host key(s) to authenticate to the slave as the magic user
"Gerrit Code Review", assuming the slave had the master's host key
listed in its $site_path/etc/peer_keys file.
A minor code change to CreateProject could introduce a new --slave
flag that doesn't update the database, just initializes the new
project repository locally.
Then we just need to put the slave's URL into a new key in the remote
block in replication.config, and if present use `gerrit create-project
--slave` to create the new project. It doesn't sound that hard
actually. Most difficult part might be setting up the JSch session
with the master's host keys.