Getting a 500 Server error on some repos.

[jme...@sybernet.ie]

Hi, 

I am getting a 500 Server errors when trying to interact with some repos.

It is only happening on some of my repos, others are fine. Those repos that appear to be affected have not been accessed on Gerrithub in a while so I don't know if that is the issue. 

I tried to clone the repo on Github Desktop and I get the following error: 

Cloning into 'C:\temp\<the repo>'... error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500 fatal: expected 'packfile'

If I try to access the repo via Eclipse which is my normal environment it also shows a 500 Error. 

Grateful for any suggestions. 

Best Regards

John

[Luca Milanesio]

On 25 Nov 2024, at 17:01, jme...@sybernet.ie <jme...@sybernet.ie> wrote:

Hi, 

I am getting a 500 Server errors when trying to interact with some repos.

It is only happening on some of my repos, others are fine. Those repos that appear to be affected have not been accessed on Gerrithub in a while so I don't know if that is the issue. 

I tried to clone the repo on Github Desktop and I get the following error: 

Cloning into 'C:\temp\<the repo>'... error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500 fatal: expected 'packfile'

I believe this has been caused by the recent upgrade to Gerrit master, the clones are the bump to f94208d7ec are now failing with:

[2024-11-25T18:07:13.713Z] [HTTP POST /<repo name>/git-upload-pack (<<>>)] ERROR com.google.gerrit.httpd.GitOverHttpServlet.GerritUploadPackErrorHandler : Internal error during upload-pack from Repository[/var/gerrit/git/<repo name>] [CONTEXT project="<repo name>" request="GIT_UPLOAD" ]

com.google.gerrit.server.notedb.InvalidServerIdException: invalid server id, expected d5d70762-12d0-45a1-890d-524b12d3f735: actual: f20de2ab-2874-4956-8ac9-502896a5ce11

        at com.google.gerrit.server.notedb.ChangeNotes.onLoad(ChangeNotes.java:678)

        at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:170)

        at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:149)

        at com.google.gerrit.server.notedb.ChangeNotes$Factory.createChecked(ChangeNotes.java:155)

Can you try to repeat the upload on review-2.gerrithub.io and see if the clone succeeds?

Luca.

If I try to access the repo via Eclipse which is my normal environment it also shows a 500 Error. 

Grateful for any suggestions. 

Best Regards

John

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/1056805e-81c7-4c2e-b5d7-a3c7bf624dd5n%40googlegroups.com.

[jme...@sybernet.ie]

Luca, 

The behaviour seems to be intermittent. I used review-2.gerrithub.io and tried to clone the repo and it seemed to get further as it presented the branches that were available. It did not do that before, however it failed to clone the repo - it seemed to timeout or something.

I have just tried it again but it behaves the same way as with review.gerrithub.io now.

It seems to have something to do with what IPs review.gerrithub.io resolves to. 

192.99.233.76 - seems to fail 

95.216.78.153 - seems to work 

I will keep trying to see if I can see a pattern. 

Best Regards

John

[Luca Milanesio]

> On 25 Nov 2024, at 20:02, jme...@sybernet.ie <jme...@sybernet.ie> wrote:
>
> Luca,
>
> The behaviour seems to be intermittent. I used review-2.gerrithub.io and tried to clone the repo and it seemed to get further as it presented the branches that were available. It did not do that before, however it failed to clone the repo - it seemed to timeout or something.

I saw it failing on both with the same exception.

What is happening is that you have some change refs associated with the serverId = f20de2ab-2874-4956-8ac9-502896a5ce11 which isn’t GerritHub.io.
Did you have those repositories hosted somewhere else?

I also see that the /meta refs for some of them were created by an accountId = 1, which is clearly NOT GerritHub.io because our account ids start with 1000000.

commit c56a459a8d33f86f3fae1a7bafdffe77671dceb5
Author: Gerrit User 1 <1@f20de2ab-2874-4956-8ac9-502896a5ce11>
Date: Thu Jun 15 12:12:25 2023 +0100

Create change

Uploaded patch set 1.

Patch-set: 1
Change-id: Ie93bf6be9ebafa7c606c7f125d187ffd73df231e


What puzzles me is that this server-id mismatch *should* not block you from cloning the repository.

Luca.

> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/0f9e3add-679c-43ba-803e-0ba82ec8f390n%40googlegroups.com.

[jme...@sybernet.ie]

Luca 

Yes the repositories were originally hosted on an internal Git server. 

I did the following to get them into Gerrithub/Github. 

git push --mirror https://github.com/<org>/<repo>.git

git push https://github.com/SyberNetLtd/<repo>.git --delete meta/config ( I did this because Gerrithub complained about this file when importing)

Imported the repository into Gerrithub.io from Github

Cloned the repository from Gerrithub into Eclipse environment. 

These projects all worked at the time and without any issues. Also I have followed this procedure for several other projects without a problem. 

I cannot clone the repository using Eclipse "Clone Repository" or I cannot "Synchronise Workspace" - both actions fail. 

I also cannot fetch -  "Synchronise Workspace" does a fetch at the beginning so that is the same thing. 

[Luca Milanesio]

Hi John,

> On 25 Nov 2024, at 21:18, jme...@sybernet.ie <jme...@sybernet.ie> wrote:
>
> Luca
>
> Yes the repositories were originally hosted on an internal Git server.
> I did the following to get them into Gerrithub/Github.
>
> git push --mirror https://github.com/<org>/<repo>.git
> git push https://github.com/SyberNetLtd/<repo>.git --delete meta/config ( I did this because Gerrithub complained about this file when importing)

Sure, that was needed because the ‘refs/meta/config’ was referring to group-ids that do not make sense on GerritHub and you would have been potentially unable to manage them.

> Imported the repository into Gerrithub.io from Github
> Cloned the repository from Gerrithub into Eclipse environment.
> These projects all worked at the time and without any issues. Also I have followed this procedure for several other projects without a problem.
>
> I cannot clone the repository using Eclipse "Clone Repository" or I cannot "Synchronise Workspace" - both actions fail.
> I also cannot fetch - "Synchronise Workspace" does a fetch at the beginning so that is the same thing.

Gerrit is complaining when trying to check if you have permissions or not to access the changes, because of the serverId mismatch.
However, changes with a mismatched serverId should be just ignored and not generate 500s.

I need to bisect and understand when this regression was introduced in Gerrit and if that was intentional or not.

Luca.

> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/2faacbdb-0a98-4a99-b913-99f51e950c7an%40googlegroups.com.

[Luca Milanesio]

> On 25 Nov 2024, at 21:36, Luca Milanesio <luca.mi...@gmail.com> wrote:
>
> Hi John,
>
>> On 25 Nov 2024, at 21:18, jme...@sybernet.ie <jme...@sybernet.ie> wrote:
>>
>> Luca
>>
>> Yes the repositories were originally hosted on an internal Git server.
>> I did the following to get them into Gerrithub/Github.
>>
>> git push --mirror https://github.com/<org>/<repo>.git
>> git push https://github.com/SyberNetLtd/<repo>.git --delete meta/config ( I did this because Gerrithub complained about this file when importing)
>
> Sure, that was needed because the ‘refs/meta/config’ was referring to group-ids that do not make sense on GerritHub and you would have been potentially unable to manage them.
>
>> Imported the repository into Gerrithub.io from Github
>> Cloned the repository from Gerrithub into Eclipse environment.
>> These projects all worked at the time and without any issues. Also I have followed this procedure for several other projects without a problem.
>>
>> I cannot clone the repository using Eclipse "Clone Repository" or I cannot "Synchronise Workspace" - both actions fail.
>> I also cannot fetch - "Synchronise Workspace" does a fetch at the beginning so that is the same thing.
>
> Gerrit is complaining when trying to check if you have permissions or not to access the changes, because of the serverId mismatch.
> However, changes with a mismatched serverId should be just ignored and not generate 500s.

I reproduced the issue locally with a plain Gerrit v3.10, and it is intermittent indeed :-O

1. Clone the repository => all good
2. Clone the same repository again => internal server error

rm -Rf load-test && git clone ssh://developer@localhost:29418/load-test

Cloning into 'load-test'...
Warning: Permanently added '[127.0.0.1]:29418' (ED25519) to the list of known hosts.
remote: Counting objects: 197, done
remote: Finding sources: 100% (196/196)
remote: Total 9227 (delta 46), reused 9181 (delta 46)
Receiving objects: 100% (9227/9227), 167.74 MiB | 79.05 MiB/s, done.
Resolving deltas: 100% (3812/3812), done.

$ rm -Rf load-test && git clone ssh://developer@localhost:29418/load-test

Cloning into 'load-test'...
Warning: Permanently added '[127.0.0.1]:29418' (ED25519) to the list of known hosts.
fatal: remote error: internal server error
fatal: internal server error

The exception is exactly the same one similar to your case on GerritHub.io:

Caused by: com.google.gerrit.server.notedb.InvalidServerIdException: invalid server id, expected f4ab4960-add9-482c-835f-d03f3feb62b7: actual: f4ab4960-add9-482c-835f-d03f3feb62b6
at com.google.gerrit.server.notedb.ChangeNotes.onLoad(ChangeNotes.java:671)

at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:170)
at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:149)

at com.google.gerrit.server.notedb.ChangeNotes$Factory.createChecked(ChangeNotes.java:152)
at com.google.gerrit.server.query.change.ChangeData.loadChange(ChangeData.java:784)
at com.google.gerrit.server.query.change.ChangeData.change(ChangeData.java:766)
at com.google.gerrit.server.query.change.ChangeData.branchOrThrow(ChangeData.java:699)
at com.google.gerrit.server.permissions.PermissionBackend$ForProject.change(PermissionBackend.java:277)
at com.google.gerrit.server.permissions.ProjectControl$ForProjectImpl.change(ProjectControl.java:370)
at com.google.gerrit.server.permissions.GitVisibleChangeFilter.lambda$getVisibleChanges$1(GitVisibleChangeFilter.java:81)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at com.google.gerrit.server.permissions.GitVisibleChangeFilter.getVisibleChanges(GitVisibleChangeFilter.java:90)
at com.google.gerrit.server.permissions.DefaultRefFilter.lambda$filter$1(DefaultRefFilter.java:147)
at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:181)
at com.google.gerrit.server.permissions.DefaultRefFilter.filterRefs(DefaultRefFilter.java:268)
at com.google.gerrit.server.permissions.DefaultRefFilter.filter(DefaultRefFilter.java:154)
at com.google.gerrit.server.permissions.ProjectControl$ForProjectImpl.filter(ProjectControl.java:427)
at com.google.gerrit.server.git.PermissionAwareReadOnlyRefDatabase.getRefsByPrefix(PermissionAwareReadOnlyRefDatabase.java:139)
at com.google.gerrit.server.git.PermissionAwareReadOnlyRefDatabase.getRefs(PermissionAwareReadOnlyRefDatabase.java:192)

It must be some caching thing; let me investigate further.

Luca.

[Luca Milanesio]

It is indeed a caching thing: I believe it was caused by Change 376094, which was introduced around 1 year ago [1].
Let me see if this cache can be disabled. It benefits only on Gerrit replicas but does not improve the performance on Gerrit primaries.

Also, I should raise an issue and add a warning to the documentation on the side effects.

Luca.

[1] https://gerrit-review.googlesource.com/c/gerrit/+/376094

[Luca Milanesio]

Confirmed: we can fix the internal server error by simply clearing out the ‘changes_by_project’ cache that was introduced in [1].

See here below my tests:

1) Start Gerrit (caches are empty)

2) Clone the repository => OK

$ rm -Rf load-test && git clone ssh://developer@localhost:29418/load-test

Cloning into 'load-test'...
Warning: Permanently added '[127.0.0.1]:29418' (ED25519) to the list of known hosts.

remote: Counting objects: 197, done
remote: Finding sources: 100% (196/196)
remote: Total 9227 (delta 46), reused 9181 (delta 46)

Receiving objects: 100% (9227/9227), 167.74 MiB | 74.46 MiB/s, done.

Resolving deltas: 100% (3812/3812), done.

3) Clone the repository again => Internal Server Error

$ rm -Rf load-test && git clone ssh://developer@localhost:29418/load-test

Cloning into 'load-test'...
Warning: Permanently added '[127.0.0.1]:29418' (ED25519) to the list of known hosts.
fatal: remote error: internal server error
fatal: internal server error

4) Flush the changes_by_project

$ ssh -p 29418 admin@localhost gerrit flush-caches --cache changes_by_project

5) Clone the repository again => OK

$ rm -Rf load-test && git clone ssh://developer@localhost:29418/load-test

Cloning into 'load-test'...
Warning: Permanently added '[127.0.0.1]:29418' (ED25519) to the list of known hosts.

remote: Counting objects: 197, done
remote: Finding sources: 100% (196/196)
remote: Total 9227 (delta 46), reused 9181 (delta 46)

Receiving objects: 100% (9227/9227), 167.74 MiB | 80.30 MiB/s, done.

Resolving deltas: 100% (3812/3812), done.

Let me create the issue, update the release notes, and disable that cache on GerritHub.io.

Luca.

[Luca Milanesio]

Issue 380898784 was raised [1], release notes were updated [2], and the cache was disabled on GerritHub.io.

@John, thanks for reporting it. Can you check again if the issue is resolved?

[1] https://issues.gerritcodereview.com/issues/380898784
[2] https://gerrit-review.googlesource.com/c/homepage/+/444141

Luca.

[jme...@sybernet.ie]

Luca, 

Thanks very much for your help, my issue is sorted now. 

Best Regards

John

[Luca Milanesio]

On 26 Nov 2024, at 16:06, jme...@sybernet.ie <jme...@sybernet.ie> wrote:

Luca, 

Thanks very much for your help, my issue is sorted now. 

Glad to hear that  \o/

Luca.

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/2c369444-92dd-49a5-a5e4-683f3308641en%40googlegroups.com.