Migration to NoteDb: blocking issue. After migration, replication to GitHub causes "Internal Server Error"

[lucamilanesio]

Hi all,

I know that you are going to say "hey this problem should not be posted to THIS mailing list" ... but I am experiencing a blocking issue, at least from a GerritHub.io perspective.

After having migrated my repositories to NoteDb (in the failover site, this is a pre-rollout test), the sync to GitHub is not working anymore.

I thought it was a replication plugin or JGit issue ... and then I tried with standalone C-Git and I got:

Counting objects: 598000, done.

Delta compression using up to 12 threads.

Compressing objects: 100% (290775/290775), done.

Writing objects: 100% (598000/598000), 129.41 MiB | 18.62 MiB/s, done.

Total 598000 (delta 240221), reused 576502 (delta 218724)

remote: Resolving deltas: 100% (240221/240221), done.

remote: Checking connectivity: 598000, done.

remote: Internal Server Error

Everything up-to-date

Does anyone know some engineer in GitHub that can give some lights on this problem?

(Jeff King perhaps? Not sure if he is monitoring this mailing list though)

P.S. I know, the repo is big, but definitely not the largest one that a Gerrit Server has seen.

Luca.

[Jonathan Nieder]

I think you should contact GitHub support (github.com/support).

Thanks,

Jonathan

ср, 25 апр. 2018 г. в 14:30, lucamilanesio <luca.mi...@gmail.com>:

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luca Milanesio]

Done, but I am not sure if they do support pushing on "non-standard refs" ... and definitely they won't officially support NoteDb :-(

But it is worth trying.

I was just wondering if anyone else (or any GitHub maintainer) has ever tried to push a large Gerrit Repo with NoteDb reviews to GitHub.

Luca.

[Jonathan Nieder]

> I am not sure if they do support pushing on "non-standard refs"

Which refs are we using that are non-standard? If they push back, let me know and *then* I can try internal channels.

> definitely they won't officially support NoteDb

Why should they care what format the content in your repository has? That would be like saying "We don't support PHP projects, only C++".

Thanks,

Jonathan

ср, 25 апр. 2018 г. в 14:47, Luca Milanesio <luca.mi...@gmail.com>:

[Luca Milanesio]

On 25 Apr 2018, at 22:51, Jonathan Nieder <j...@google.com> wrote:

> I am not sure if they do support pushing on "non-standard refs"

Which refs are we using that are non-standard? If they push back, let me know and *then* I can try internal channels.

refs/changes are "non-standard" refs for them. You need to "hack a bit" the UX to discover them.

Let's see how they react :-)

It could well be that they consider a repo migrated to NoteDb "too big" for a single push operation.

I know they try hard to rate limit the clients to prevent DDoS

> definitely they won't officially support NoteDb

Why should they care what format the content in your repository has? That would be like saying "We don't support PHP projects, only C++".

True, I am just wondering why has always worked (GerritHub was launched over 4 years ago) and just now with the conversion to NoteDb returns 500 on push :-(

[Dave Borowitz]

Should we make it easier somehow to avoid replicating NoteDb refs?

[Luca Milanesio]

I was thinking about the same thing: possibly a configuration setting on the replication plugin.

However, I tried by splitting the push into 100 parts (using the NN/ prefix) and it succeeded.

I truly believe it is more about a sort of "size-limit" of the push on their side.

When migrating a big repo to NoteDb, we generate a *LOT MORE* objects and refs, and the first replication after migration can be too big for GitHub :-(

Luca.

[Matthias Sohn]

Does it work with smaller repositories ?

Can you measure size of the requests/packs being transferred in order to get numbers.

I'd expect that if you can clone a repository of size x you should be also able to push data

of similar size.

Luca.

Thanks,

Jonathan

To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Luca Milanesio]

On 25 Apr 2018, at 23:23, Matthias Sohn <matthi...@gmail.com> wrote:

Does it work with smaller repositories ?

Going to do some more testing tomorrow: the migration on our German DC works well, but we definitely need  to sort out the replication post-NoteDb migration before the rollout to the main Canadian DC.

Can you measure size of the requests/packs being transferred in order to get numbers.

Sure.

I'd expect that if you can clone a repository of size x you should be also able to push data

of similar size.

From a protocol perspective, yes. But they could consider "normal" a clone of a large repo and "not-normal" the push of multiple objects at once.

[Jonathan Nieder]

https://github.com/github/git-sizer may also be useful for understanding what kind of repositories they consider pathological.

Thanks again,

Jonathan

ср, 25 апр. 2018 г. в 15:27, Luca Milanesio <luca.mi...@gmail.com>:

Luca.

Thanks,

Jonathan

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[Luca Milanesio]

Yes, good suggestion. Let me try to get the output of the "problematic big ones" tomorrow.

I know that Martin, Hugo and Saša have big ones as well, I wonder if the experience made on GerritHub.io could help them planning their future migrations to 2.15/NoteDb :-)

Luca.

[Luca Milanesio]

On 25 Apr 2018, at 23:34, Luca Milanesio <luca.mi...@gmail.com> wrote:

Yes, good suggestion. Let me try to get the output of the "problematic big ones" tomorrow.

https://github.com/spdk/spdk (failed to push NoteDb refs)

Processing blobs: 44276                        
Processing trees: 109291                        
Processing commits: 130401                        
Matching commits to trees: 130401                        
Processing annotated tags: 0                        
Processing references: 26524                        
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Overall repository size      |           |                                |
| * References                 |           |                                |
|   * Count                    |  26.5 k   | *                              |

Smaller ones are working, see:

https://github.com/att-comdev/shipyard (succeeded to push NoteDb refs)

Processing blobs: 4790                        
Processing trees: 7366                        
Processing commits: 14417                        
Matching commits to trees: 14417                        
Processing annotated tags: 0                        
Processing references: 1925                        
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Biggest checkouts            |           |                                |
| * Maximum path depth     [1] |    11     | *                              |

[1]  4a4b8ec10133b5a7e2cb08ccda5da43d25a90a62 (refs/changes/13/408813/11^{tree})

[Matthew Webber]

For anyone who stumbles across this thread, it's worth mentioning that you can configure the Gerrit Replication Plugin so that it only replicates certain refs.

Example: Our replication.config has

[remote "GitHub-xxxx"]
    ...
    push = +refs/heads/*:refs/heads/*
    push = +refs/tags/*:refs/tags/*

So that it doesn't replicate refs/changes/* etc.

[Luca Milanesio]

Yes, however, you cannot push changes *without* pushing NoteDb as well :-(

[Matthew Webber]

On Thursday, 26 April 2018 12:10:40 UTC+1, lucamilanesio wrote:

Yes, however, you cannot push changes *without* pushing NoteDb as well :-(

I know that you know how it works, Luca  ;-)

[Luca Milanesio]

Update from GitHub: they have acknowledged the problem, but they are not able to provide an ETA for the solution.

The new NoteDb refs are getting pushed, but the existing ones are not.

It seems that GitHub is unable to process a push to a repo with a large number of refs, which is a bit surprising for me because I thought they had spent a lot of effort of making it enterprise-ready :-O

Let's see how it goes in the next few days.

Luca.

[Luca Milanesio]

Two weeks, no news, and still can't push NoteDb to GitHub :-(

Does anybody knows some GitHub-er that can help sorting this out?

Once NoteDb will start getting popular, more and more people will have this problem, including the OpenStack project !

Luca.

[Stefan Beller]

Hi Luca,

On Tue, May 1, 2018 at 2:57 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:
> Update from GitHub: they have acknowledged the problem, but they are not
> able to provide an ETA for the solution.
> The new NoteDb refs are getting pushed, but the existing ones are not.
>
> It seems that GitHub is unable to process a push to a repo with a large
> number of refs, which is a bit surprising for me because I thought they had
> spent a lot of effort of making it enterprise-ready :-O

Can you push one ref at a time as a workaround?
(push doesn't suffer from the big ls-remote output,
so pushing just one non-standard ref ought to be no problem,
and then you do many of them, which may run into throttling issues)

[Jonathan Nieder]

Luca Milanesio wrote:

I was thinking about the same thing: possibly a configuration setting on the replication plugin.

However, I tried by splitting the push into 100 parts (using the NN/ prefix) and it succeeded.

I truly believe it is more about a sort of "size-limit" of the push on their side.

That makes sense to me. It breaks atomicity, but if the server being pushed to doesn't accept pushes with large numbers of refs, what can you do?

Are you imagining an option passed to the replication plugin's "start" command (https://gerrit.googlesource.com/plugins/replication/+/master/src/main/resources/Documentation/cmd-start.md), a setting that applies to all its pushes (https://gerrit.googlesource.com/plugins/replication/+/master/src/main/resources/Documentation/config.md), or some other interface?

Thanks,

Jonathan

[Luca Milanesio]

Yes, we should define a limit to the number of refs pushed at once.

However, I have warned the GitHub guys that splitting the pushes in this way, I will generate potentially hundreds of thousands of Git pushes and, because of the heavy traffic, they could be tempted to blacklist the GerritHub.io which would result in a complete outage of my service, which isn't nice at all :-(

I've pinged them again, and see how it goes.

P.S. I did not consider a push of 1000 refs a *big push* ... I can understand if it was 100k or 1M refs ... but 1000 refs isn't big at all !!!! They must have some serious scalability issues out there :-(

Luca.

[thomasmu...@yahoo.com]

Does this mean that replication won't work without refs/meta?

With phabricator we had to push refs in batches for mediawiki/core. The mirror is still on for that repo so we have refs/changes/ replicated to github. Maybe that will help luca? But i think replication does need to be fixed to support not pushing refs/meta/ then

https://phabricator.wikimedia.org/source/mediawiki/uri/view/4/

[Luca Milanesio]

On 15 May 2018, at 23:05, thomasmulhall410 via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

Does this mean that replication won't work without refs/meta?

I mean that *IF* the replication action includes a lot of refs pushed all at once to GitHub, then the replication action fails.

E.g. A single change won't file, but a batch of 500 changes with their respective meta refs will fail.

With phabricator we had to push refs in batches for mediawiki/core. The mirror is still on for that repo so we have refs/changes/ replicated to github. Maybe that will help luca?

You guys do not have 40k repos to replication I presume :-(

But i think replication does need to be fixed to support not pushing refs/meta/ then

Nothing to do with the refs/meta, it is all about how many refs are pushed at once.

It is a very specific GitHub issue though: pushing to our failover site in Germany (plain Git daemon using plain Git protocol) we did not have a single issue.

https://phabricator.wikimedia.org/source/mediawiki/uri/view/4/

On Tuesday, May 15, 2018 at 10:39:45 PM UTC+1, lucamilanesio wrote:

On 15 May 2018, at 22:34, Jonathan Nieder <j...@google.com> wrote:

Luca Milanesio wrote:

I was thinking about the same thing: possibly a configuration setting on the replication plugin.

However, I tried by splitting the push into 100 parts (using the NN/ prefix) and it succeeded.

I truly believe it is more about a sort of "size-limit" of the push on their side.

That makes sense to me. It breaks atomicity, but if the server being pushed to doesn't accept pushes with large numbers of refs, what can you do?

Are you imagining an option passed to the replication plugin's "start" command (https://gerrit.googlesource.com/plugins/replication/+/master/src/main/resources/Documentation/cmd-start.md), a setting that applies to all its pushes (https://gerrit.googlesource.com/plugins/replication/+/master/src/main/resources/Documentation/config.md), or some other interface?

Yes, we should define a limit to the number of refs pushed at once.

However, I have warned the GitHub guys that splitting the pushes in this way, I will generate potentially hundreds of thousands of Git pushes and, because of the heavy traffic, they could be tempted to blacklist the GerritHub.io which would result in a complete outage of my service, which isn't nice at all :-(

I've pinged them again, and see how it goes.

P.S. I did not consider a push of 1000 refs a *big push* ... I can understand if it was 100k or 1M refs ... but 1000 refs isn't big at all !!!! They must have some serious scalability issues out there :-(

Luca.

[thomasmu...@yahoo.com]

We could not push all those refs/changes/ to github. We had to manually do it in batches to work, same with operations/puppet.

To unsubscribe, email repo-discuss+unsub...@googlegroups.com

[thomasmu...@yahoo.com]

We have over 80,000 changes https://github.com/wikimedia/mediawiki

[Luca Milanesio]

On 15 May 2018, at 23:26, thomasmulhall410 via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

We have over 80,000 changes https://github.com/wikimedia/mediawiki

Yes, but they were created over time so you have never experienced this issue.

When you guys will migrate to NoteDb in v2.15, you'll hit this problem as well, unless the GitHub guys will manage to sort this out before then.

80k changes will generate a "mega-push" of 80k refs all at one which would cause GitHub to crash with 500 during push.

If we introduce a "max refs" option during push to break down pushes to 100 refs at a time, 80k refs will be partitioned in 800 pushes.

You will still need to "advise" GitHub beforehand, because if they receive 800 pushes in rapid sequence, they could blacklist your IP and you'll be offline.

In our case, it is a lot worse because we have 500k changes on 40k repos, and we would generate millions of pushes :-(

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

[David Ostrovsky]

On Thursday, April 26, 2018 at 12:09:11 AM UTC+2, Dave Borowitz wrote:

Should we make it easier somehow to avoid replicating NoteDb refs?

+1. We could even add such an option depending on the remote.

We define GH broken to accept NoteDb refs (for now), and teach

replication plugin to skip the NoteDb refs for broken remotes.

[Luca Milanesio]

Just to remind again: it was not a problem of NoteDb but rather a scalability issue with pushes with 100+ refs.

If you clone a repo with 1k refs and push it again to GitHub, it would just fail with an HTTP error 500.

A ticket has been raised 2 weeks ago, they acknowledged the problem but haven't fixed yet.

Luca.

[David Ostrovsky]

I understand all that. GH service for open source projects

is best effort support thing without any SLA.

Yet the non-working replication is severe outage for 40k projects, that rely

on it on GerritHub.io. That why I think you cannot endlessly wait and hope

for the immediate help from GH .

I see two options for now:

1. Revert migration to NoteDb on GerritHub.io until GH replication problem is resolved

2. Teach replication plugin to skip NoteDb refs for now and revisit it later

[Luca Milanesio]

On 16 May 2018, at 07:53, David Ostrovsky <david.o...@gmail.com> wrote:

On Wednesday, May 16, 2018 at 8:42:10 AM UTC+2, lucamilanesio wrote:

On 16 May 2018, at 00:11, David Ostrovsky <david.o...@gmail.com> wrote:

On Thursday, April 26, 2018 at 12:09:11 AM UTC+2, Dave Borowitz wrote:

Should we make it easier somehow to avoid replicating NoteDb refs?

+1. We could even add such an option depending on the remote.

We define GH broken to accept NoteDb refs (for now), and teach

replication plugin to skip the NoteDb refs for broken remotes.

Just to remind again: it was not a problem of NoteDb but rather a scalability issue with pushes with 100+ refs.

If you clone a repo with 1k refs and push it again to GitHub, it would just fail with an HTTP error 500.

A ticket has been raised 2 weeks ago, they acknowledged the problem but haven't fixed yet.

I understand all that. GH service for open source projects

is best effort support thing without any SLA.

Nope, I pay as a Company and have lots of private repos there. 

FREE service doesn't mean no SLA anyway, we have 99.99% on GerritHub.io.

Yet the non-working replication is severe outage for 40k projects, that rely

on it on GerritHub.io. That why I think you cannot endlessly wait and hope

for the immediate help from GH .

I see two options for now:

1. Revert migration to NoteDb on GerritHub.io until GH replication problem is resolved

Not possible I'm afraid: there are now thousands of changes in NoteDb, ReviewDb is largely obsolete.

2. Teach replication plugin to skip NoteDb refs for now and revisit it later

Yes, a workaround I would say :-)

GitHub should fix their service though, I can't image we are the only ones with this serious problem.

Possibly we are the only ones with so many repos on GitHub, however, the JenkinsCI organisation has several thousands, but none of them with so many refs.

Luca.

[thomasmu...@yahoo.com]

You could do

push = +refs/heads/*:refs/heads/*
push = +refs/tags/*:refs/tags/*

?

https://gerrit.googlesource.com/plugins/replication/+doc/master/src/main/resources/Documentation/config.md

[Luca Milanesio]

> On 16 May 2018, at 08:46, thomasmulhall410 via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
>
> You could do
>
> push = +refs/heads/*:refs/heads/*
> push = +refs/tags/*:refs/tags/*

Nope, that would cut all the changes, not only NoteDb.

[thomasmu...@yahoo.com]

Oh so you use github as a repository viewer, so is you view refs/changes/ there?

Though you could do that as a work around or fix replication to not replicate notedb.

The work around would at least replicate changes merged in the branches.

[Luca Milanesio]

> On 16 May 2018, at 08:51, thomasmulhall410 via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
>
> Oh so you use github as a repository viewer, so is you view refs/changes/ there?

Yes, GitHub is basically the "backend Git store" of Gerrit Code Review: everything happens in GitHub, apart from the review workflow which is in Gerrit.

>
> Though you could do that as a work around or fix replication to not replicate notedb.

This is what we are discussing here.
Of course, as this is a major bug in GitHub, they should fix it :-)

I replicated the problem with:

a) Clone a repo MyBigRepo with --mirror
b) Create a new repo MyBigRepoReplica
c) Push to MyBigRepoReplica
d) ... and BOOM

As easy as that, they managed to replicate the problem, just needs to be fixed on their end.

A "plain vanilla git-daemon" works like a charm ... GitHub crashes :-(

[thomasmu...@yahoo.com]

Should an issue be filled at https://bugs.chromium.org/p/gerrit/issues/list for this?

As we really do need a work around as GitHub could take a while to fix the issue.

[Luca Milanesio]

> On 16 May 2018, at 10:53, thomasmulhall410 via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
>
> Should an issue be filled at https://bugs.chromium.org/p/gerrit/issues/list for this?

Are we tracking GitHub issues now?

>
> As we really do need a work around as GitHub could take a while to fix the issue.

I really hope not :-)
This is a *major* failure, they have to fix it !!!!

Luca.

[thomasmu...@yahoo.com]

We track gerrit issues, but as you said we may get into this problem too so not only a gerrithub problem :).

But github has many servers I presume (high availability) so that could be the issue with them?

[Luca Milanesio]

> On 16 May 2018, at 11:03, thomasmulhall410 via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
>
> We track gerrit issues, but as you said we may get into this problem too so not only a gerrithub problem :).

Yes, and that is still a GitHub issue.

>
> But github has many servers I presume (high availability) so that could be the issue with them?

Software issue IMHO, having HA doesn't help then :-(

Luca.

[lucamilanesio]

I’ve chased them again today: the problem has nothing to do with NoteDB.

Can be easily reproduced by cloning a GitHub repo and pushing it back to GitHub ... boom :-(

Very P1 IMHO

Luca

[David Ostrovsky]

One small step forward here, is to increase the transparency, visibility
and awareness of that problem to broader GH community, as this is in
no way Gerrit/NoteDb specific issue (repo-discuss is not the right palce).

We (you?) could file an issue here: [1]. And you could then track the
progress and communication with GH support team, instead of just
bumping this thread. I assume the communication is not related
to: "if it is not a confidential matter like a security disclosure"? As
mentioned in the project's [1] README.

[1] https://github.com/isaacs/github

[Luca Milanesio]

Yes, good idea. I initially thought that was NoteDb causing it, but now it is clear that can be reproduced outside Gerrit and not relevant or associated to NoteDb in any way.

Will keep the conversation going there, good point.

[Luca Milanesio]

On 24 May 2018, at 11:47, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 24 May 2018, at 11:28, David Ostrovsky <david.o...@gmail.com> wrote:

Am Dienstag, 22. Mai 2018 13:45:27 UTC+2 schrieb lucamilanesio:

I’ve chased them again today: the problem has nothing to do with NoteDB. 

Can be easily reproduced by cloning a GitHub repo and pushing it back to GitHub ... boom :-( 

Very P1 IMHO 

One small step forward here, is to increase the transparency, visibility
and awareness of that problem to broader GH community, as this is in
no way Gerrit/NoteDb specific issue (repo-discuss is not the right palce).

We (you?) could file an issue here: [1]. And you could then track the
progress and communication with GH support team, instead of just
bumping this thread. I assume the communication is not related
to: "if it is not a confidential matter like a security disclosure"? As
mentioned in the project's [1] README.

[1] https://github.com/isaacs/github

Yes, good idea. I initially thought that was NoteDb causing it, but now it is clear that can be reproduced outside Gerrit and not relevant or associated to NoteDb in any way.

Will keep the conversation going there, good point.

Done => https://github.com/isaacs/github/issues/1253

Luca.

[David Ostrovsky]

Am Donnerstag, 24. Mai 2018 12:53:24 UTC+2 schrieb lucamilanesio:

On 24 May 2018, at 11:47, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 24 May 2018, at 11:28, David Ostrovsky <david.o...@gmail.com> wrote:

Am Dienstag, 22. Mai 2018 13:45:27 UTC+2 schrieb lucamilanesio:

I’ve chased them again today: the problem has nothing to do with NoteDB. 

Can be easily reproduced by cloning a GitHub repo and pushing it back to GitHub ... boom :-( 

Very P1 IMHO 

One small step forward here, is to increase the transparency, visibility
and awareness of that problem to broader GH community, as this is in
no way Gerrit/NoteDb specific issue (repo-discuss is not the right palce).

We (you?) could file an issue here: [1]. And you could then track the
progress and communication with GH support team, instead of just
bumping this thread. I assume the communication is not related
to: "if it is not a confidential matter like a security disclosure"? As
mentioned in the project's [1] README.

[1] https://github.com/isaacs/github

Yes, good idea. I initially thought that was NoteDb causing it, but now it is clear that can be reproduced outside Gerrit and not relevant or associated to NoteDb in any way.

Will keep the conversation going there, good point.

Done => https://github.com/isaacs/github/issues/1253

Thanks! I filed this prio 1 issue: [1] on our side, as replication
plugin is affected and there is currently no known work around.

[1] https://bugs.chromium.org/p/gerrit/issues/detail?id=9080

[thomasmu...@yahoo.com]

I cannot immagine how many notedb refs you had pushing to github.

using phabricator we were cloning over 8 million+ refs which was causing the db to have high writes + high load on phabricator's server it's self.

[luca.mi...@gmail.com]

Spdk failed to push, 40k refs.

I did lots of tests and found that the limit was around 1000+ refs, not exaggerated for a medium sized project.

I believe Martin has repos with many more refs ;-)

Luca

Sent from my iPhone

--

[lucamilanesio]

I am pleased to announce that GitHub has fixed the issue and the replication, including NoteDb, is back on track. Everything is working fine.

No need to workaround the problem on our side anymore.

Thanks again to GitHub for fixing the issue !!!!

Luca.

To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.