Gerrit projects are broken after gc ?

[Sehen]

Hello,

We are running Gerrit 2.16.17 and facing issue, Some project's objects are deleted maybe.

(There were no directories in object/, except info, pack/)

1. Cannot find that projects on Gerrit web UI.

2. Cannot clone that projects.

3. Error logs are below.

[SSH git-upload-pack bbb-project (ccc)] ERROR org.eclipse.jgit.internal.storage.file.ObjectDirectory : Failed to access pack file aaa.pack, caught 1 consecutive errors while trying to access this pack.

[SSH git-upload-pack bbb-project (ccc)] ERROR org.eclipse.jgit.util.FS : Mount point not found

....

[SSH git-upload-pack bbb-project (ccc)] WARN  com.google.gerrit.server.args4j.ProjectHandler : Cannot read project bbb-project

We are guessing that 'Gerrit gc' were occurring this error.

But It is not sure. 

Gerrit gc was running daily.(gerrit gc --all --show-progress)

After it is stopped, This issue is not occurred.

But I know gc is necessary job, Can you help resolve this issue?

[Matthias Sohn]

On Mon, Nov 1, 2021 at 10:49 AM Sehen <ohse...@gmail.com> wrote:

Hello,

We are running Gerrit 2.16.17 and facing issue, Some project's objects are deleted maybe.

(There were no directories in object/, except info, pack/)

This is not necessarily an invalid state. After gc all objects might be packed so that no loose objects are left.

A packed bare git repository minimally contains the following files:

HEAD

config (repo configuration)

packed-refs (all refs in packed form)

objects/pack/<sha1>.pack (all objects in one pack file)

objects/pack/<sha1>.idx (index of the pack file)

1. Cannot find that projects on Gerrit web UI.

2. Cannot clone that projects.

3. Error logs are below.

[SSH git-upload-pack bbb-project (ccc)] ERROR org.eclipse.jgit.internal.storage.file.ObjectDirectory : Failed to access pack file aaa.pack, caught 1 consecutive errors while trying to access this pack.

Did you check if these pack files are present on the gerrit server under $GERRIT_SITE/git/<repo path>/objects/pack/ ?
Run git fsck --full in the affected repositories on the server. Does that report errors ?

If not, does git log work and show the expected history ?

[SSH git-upload-pack bbb-project (ccc)] ERROR org.eclipse.jgit.util.FS : Mount point not found

Maybe some file system volume was detached or is corrupt ?

Check the content of the gerrit site. 

....

[SSH git-upload-pack bbb-project (ccc)] WARN  com.google.gerrit.server.args4j.ProjectHandler : Cannot read project bbb-project

We are guessing that 'Gerrit gc' were occurring this error.

I don't think that gc removed a mount point 

But It is not sure. 

Gerrit gc was running daily.(gerrit gc --all --show-progress)

After it is stopped, This issue is not occurred.

But I know gc is necessary job, Can you help resolve this issue?

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/c4a56216-a23e-44c3-af9b-aec17bae4bb9n%40googlegroups.com.

[Sehen]

Thank you for your help, Matthias Sohn.

This is for your answers.

  1. Gerrit gc result is not an invalid state. -  I see.

  2. About "Mount point  not found", There was no file system volume error.

  3. In Issued git project, git log is fine now.

But I'll show you more detail things of errror log, This case is so confused.

1. From 10/22, There are two kind of many "Too many open files" in error_log

    - [HTTP-344449] WARN org.eclipse.jetty.io.ManagedSelector : Accept failed for channel null

        java.io.IOException: Too many open files

    - org.apache.sshd.common.io.nio2.Nio2Acceptor : Caught IOException while accepting incoming connection from /0:0:0:0:0:0:0:0:xxxx: Too many open files

2. In error_log (From 10/26 19:27)

    - FileNotFoundException: .../objects/pack/pack-abcde.pack (Too many open files) - "But, this pack file is in .../objects/pack/ already (10/26 04:46) !!!!!"

    - org.eclipse.jgit.util.FS : Mount point not found

    - Cannot read project bbb-project

3. Project's name was changed(old_), daily Gerrit GC job is stopped(10/27), after that,  

    - No more project Issues now

    - Issued project(old_) clone is fine now 

    - 'git log' command is fine in Issued project(old_)

    - Not listed in Gerrit New UI project list(But It is on Old UI)

    - There are no "Too many open files" error log more..

Can you help me again?

Thank you.

2021년 11월 1일 월요일 오후 9시 54분 51초 UTC+9에 Matthias Sohn님이 작성: