Gerrit -> Github replication: Push updates all refs (and leads to a blocked Github repository)

[Andy G.]

Hey,

we (TYPO3) replicating the "Packages/TYPO3.CMS" project from our Gerrit System [1] to Github "typo3-ci/TYPO3.CMS-pre-merge-tests" [2].

For this we are using the replication plugin by Gerrit with the configuration:

[remote "github/TYPO3-ci/TYPO3.CMS-pre-merge-tests"]
  projects = Packages/TYPO3.CMS
  url = https://github.com/TYPO3-ci/TYPO3.CMS-pre-merge-tests.git
  push = +refs/changes/*:refs/heads/changes/*
  push = +refs/heads/*:refs/heads/*
  push = +refs/tags/*:refs/tags/*
  authGroup = Git Mirror
  mirror = true
  timeout = 120

Three days ago we recieved an email by Github:

Hello,

Since about 6:00am PST today Friday November 20, 2015, we have noticed repeated failed pushes to the typo3-ci/TYPO3.CMS-pre-merge-tests repository.  It looks like each push is attempting to update roughly 51 thousands refs.

We suspect that this may be due to an errant script or automation and would like to request that the source of the pushes be identified, and for the number of refs pushed to be reduced to a much smaller number, and the operations batched.

As you see from the configuration we replicating every change.

We do this to enable Pre Merge Testing with TravisCI and gotrap [3].

• Do you have any idea, why every ref is updated even if no change was done in this ref?
• Any idea how to improve the sync process to be able to access every change as branch in Github without over comitting resources by Github?
• Anyone got similar issues?

Thanks for answers,

Andy

[1] https://review.typo3.org/

[2] https://github.com/typo3-ci/TYPO3.CMS-pre-merge-tests

[3] https://github.com/andygrunwald/gotrap

[Bassem Rabil]

Was this working for you before Nov 20th ?

Basically what replication plugin does in case of any change is: It checks the refs on the source and the destination and push replication for the delta between the source and replication destination. If everytime it tries to push probably all your refs ~ 50k refs this indicates that replication plugin sees no refs on github destination repository, and thus tries to push all refs from your Gerrit instance to github. The only scenarios I can think of is either a corruption on the repository on github that prevents listing the refs from the replication plugin, or permission issues to access the refs of this repo on github from the replication plugin.

I hope this helps.

Regards

Bassem

[Luca Milanesio]

I see as well that the Typo3 guys are pushing all the changes / patch-sets: this would easily explain the 50K refs each patch-set is a different ref and Typo3 has over 45K changes.

One solution could be disabling the refs/changes/* in the ref-spec as GitHub does not really manage effectively refs outside the canonical refs/tags and refs/heads.

What is the value of pushing refs/changes/* if then GitHub does not know what to do with them?

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steffen Gebert]

Dear all,

we (at TYPO3) are pushing refs/changes/* to this very repo to run TravisCI for every patch set.
Andy described this here: https://github.com/andygrunwald/gotrap

Pushing all these refs definitely worked before (we unfortunately also killed TravisCI once, because when we switched it on (for the 2nd or 3rd time), the were overwhelmed by all the build jobs :-()

When I look back in our logs, I only see the following mentioning the pre-merge-tests repo:

> [2015-10-01 07:00:12,170] ERROR com.googlesource.gerrit.plugins.replication.ReplicationQueue : Cannot replicate to https://github.com/TYPO3-ci/TYPO3.CMS-pre-merge-tests.git
> org.eclipse.jgit.errors.TransportException: Read timed out after 120,000 ms
> at org.eclipse.jgit.transport.BasePackConnection.readAdvertisedRefs(BasePackConnection.java:182)
> at org.eclipse.jgit.transport.TransportHttp$SmartHttpFetchConnection.<init>(TransportHttp.java:770)
> at org.eclipse.jgit.transport.TransportHttp.openFetch(TransportHttp.java:314)


In earlier logs, I see the following instead:
> [2015-09-02 08:50:15,255] INFO com.googlesource.gerrit.plugins.replication.ReplicationQueue : Push to https://github.com/TYPO3-ci/TYPO3.CMS-pre-merge-tests.git references: [RemoteRefUpdate[remoteName=refs/heads/changes/77/42977/1, NOT_ATTEMPTED, (null)...4de8cfafa23f464734dd3ac31c7a688a776d4011, srcRef=refs/changes/77/42977/1, forceUpdate, message=null]]


So do you think it takes more than 120sec to read all refs from the repo? Or can you imagine how to fix this and keep Github happy as well as we would be (the pre-merge tests help a lot)?

Thanks
Steffen

[luca.mi...@gmail.com]

I think GitHub is just throttling your Git requests because of the high rate of operations on refs/changes :-(

Why don't you do pre-commit validation from Gerrit directly? We can host your jobs on Gerrit-ci.gerritforge.com.

Luca

Sent from my iPhone

[Luca Milanesio]

Out of curiosity: do you have any evidence on your gerrit replication_log of the ~50K pushes? These could only be generated if for any reason GitHub (or Gerrit) think that all the existing refs/changes/* have gone on GitHub and need to be pushed again.

Luca.