Issue when mirroring large git repositories to Gerrit

[Ahmed Hosni]

Hi everybody,

I still discovering gerrit, i tried to mirror some of our git repositories (around 30 repository) to Gerrit but i faced some issues with repositories having high ref count.

I made the choice of a server-client-server mirroring rather than server to server mirroring.

As you can see in the attached error_log, I got an issue with the plugin replication, 

A second issue with the VIBE git repository which contains 8019 tags and 8115 branches and is the origin of this unpack error for a given tag:

[2013-11-07 11:33:34,853] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user codex_support account 2) during git-receive-pack gerrit-dry-run/VIBE

com.google.gerrit.sshd.BaseCommand$Failure: fatal: Unpack error, check server log

at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:157)

at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:106)

at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:34)

at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:72)

at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:430)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

at java.util.concurrent.FutureTask.run(FutureTask.java:166)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

at java.lang.Thread.run(Thread.java:679)

Caused by: java.io.IOException: Unpack error on project "gerrit-dry-run/VIBE":

  AdvertiseRefsHook: com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook@68cd3558class com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook

at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:156)

... 13 more

Caused by: org.eclipse.jgit.errors.UnpackException: Exception while parsing pack stream

at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:202)

at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:142)

at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:98)

... 13 more

Caused by: java.io.IOException: Invalid tag 68dd0ff5540aefe8e667544a9a93301f29d6071b:invalid tagger

at org.eclipse.jgit.transport.PackParser.verifySafeObject(PackParser.java:1014)

at org.eclipse.jgit.transport.PackParser.whole(PackParser.java:995)

at org.eclipse.jgit.transport.PackParser.indexOneObject(PackParser.java:912)

at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:486)

at org.eclipse.jgit.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:179)

at org.eclipse.jgit.transport.BaseReceivePack.receivePack(BaseReceivePack.java:915)

at org.eclipse.jgit.transport.BaseReceivePack.receivePackAndCheckConnectivity(BaseReceivePack.java:754)

at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:167)

... 15 more

And a third issue that occured 15 time and comes with an ORM stack trace that i'm not able to interprete ..

Could someone help me on how to investigate those issues?

Is the server to server mirroring via rsync a better choice for what i'm trying to do?

We are running Gerrit 2.5.2 with mysql 5.5.29 and git 1.7.1 on a RHEL 6.3 server.

Kind regards,

Ahmed

[Shawn Pearce]

On Fri, Nov 8, 2013 at 12:49 AM, Ahmed Hosni
<ahmedhosn...@gmail.com> wrote:
> Caused by: java.io.IOException: Invalid tag
> 68dd0ff5540aefe8e667544a9a93301f29d6071b:invalid tagger
> at
> org.eclipse.jgit.transport.PackParser.verifySafeObject(PackParser.java:1014)

This tag appears to be corrupt.

Look at your source repository and run:

git cat-file tag 68dd0ff5540aefe8e667544a9a93301f29d6071b

JGit does not like this object, as it believes the object is corrupted
or malformed, so Gerrit is rejecting the data.

[Ahmed Hosni]

Hi Shawn,

This tag is one among a bunch of dangling tags, i should expect that JGit reject it while performing fsck, and then abort the mirroring of all my tags...

   hosniah@tunlx107{VIBE.git}162 :git cat-file tag 68dd0ff5540aefe8e667544a9a93301f29d6071b

   object 1f6357bf01e14124ef6ec27d303b0c19693cd693

   type commit

   tag FIRST

   tagger Stuart xxxxx <stuart.xxxxx @st.com>

When i try:

 git push --tags ssh://myu...@mygerrit.st.com:29418/gerrit-dry-run/VIBE.git

JGit is raising an unpack error due to an "invalid tagger" for many dangling tags that matches legacy stuff  (from 2004)within our company:

hosniah@tunlx107{VIBE.git}169 :git show 68dd0ff5540aefe8e667544a9a93301f29d6071b

tag FIRST

Tagger: Stuart xxxxx <stuart.xxxxx @st.com>

Date:   Thu Jan 1 00:00:00 1970 +0000

commit 1f6357bf01e14124ef6ec27d303b0c19693cd693

Author: Stuart xxxxx <stuart.xxxxx @st.com>

Date:   Thu Feb 19 18:41:13 2004 +0000

....

This isue souds too close to the one within this thread https://groups.google.com/forum/#!topic/repo-discuss/rfIdvXb35ag, i'm looking for a workaround  that avoid that the JGit housekeeping reject the whole upload when i'm mirroring such tags.

Would the server-to-server mirrroring of those tags be relevant (if i run a rsync + gerrit flush-caches)?

Could i bypass this JGit housekeeping stuff when i directly fetch those tag as gerritadm from my gerrit repository?

Best regards,

Ahmed

[Shawn Pearce]

Yes a direct fetch as Gerrit admin into the Gerrit managed repository will bypass the check. So would an rsync and a cache flush as you suggested.

I think you can also grant yourself forge committer identity on refs/tags/* to import those by push.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
 
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ravi Subramanian]

I hit this same issue attempting to clone the codeaurora forums' Android fork into gerrit.

For anyone wondering exactly how to get around this in case you have a local mirrored repository, here's what worked for me:

1. I impersonated the Gerrit Linux user (this is the user that gerrit itself runs as):
   sudo su gerrit
   
   
2. I pushed the entire repository (the one with the bad tag) directly to gerrit's own bare git like so:
   cd local/path/to/project.git
   git push /home/gerrit/review_site/git/path/to/project.git +refs/heads/* +refs/tags/*;

This seemed to do the trick.