Replication error | java.io.EOFException: Short read of block.

125 views
Skip to first unread message

tech....@gmail.com

unread,
Sep 27, 2023, 12:24:26 AM9/27/23
to Repo and Gerrit Discussion
Hello Team,

We have Master , mirror architecture. Currently we are fasing issue with one of the repo not being replicated due to some curropted blob may be. 

What is the safest way to remediate it, without losing data.

[2023-09-27 04:18:33,331] Cannot replicate to ger...@hibllxgrt.com:/data/gerrit/git/vendor/kernel-modules.git [CONTEXT pushOneId="51ae40a2" ]
org.eclipse.jgit.errors.TransportException: ger...@hibllxgrtmir.com:/data/gerrit/git/vendor/kernel-modules.git: reading objects from local repository failed: Short read of block.
        at org.eclipse.jgit.transport.PushProcess.isFastForward(PushProcess.java:290)
        at org.eclipse.jgit.transport.PushProcess.prepareRemoteUpdates(PushProcess.java:244)
        at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:150)
        at org.eclipse.jgit.transport.Transport.push(Transport.java:1537)
        at org.eclipse.jgit.transport.Transport.push(Transport.java:1583)
        at com.googlesource.gerrit.plugins.replication.PushOne.pushInBatches(PushOne.java:591)
        at com.googlesource.gerrit.plugins.replication.PushOne.pushVia(PushOne.java:584)
        at com.googlesource.gerrit.plugins.replication.PushOne.runImpl(PushOne.java:555)
        at com.googlesource.gerrit.plugins.replication.PushOne.doRunPushOperation(PushOne.java:437)
        at com.googlesource.gerrit.plugins.replication.PushOne.runPushOperation(PushOne.java:405)
        at com.googlesource.gerrit.plugins.replication.PushOne.lambda$run$2(PushOne.java:391)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:186)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:174)
        at com.google.gerrit.server.git.PerThreadRequestScope$Propagator.lambda$scope$0(PerThreadRequestScope.java:70)
        at com.googlesource.gerrit.plugins.replication.PushOne.run(PushOne.java:394)
        at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:95)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.EOFException: Short read of block.
        at org.eclipse.jgit.util.IO.readFully(IO.java:203)
        at org.eclipse.jgit.internal.storage.file.UnpackedObject.open(UnpackedObject.java:72)
        at org.eclipse.jgit.internal.storage.file.LooseObjects.getObjectLoaderWithoutRefresh(LooseObjects.java:227)
        at org.eclipse.jgit.internal.storage.file.LooseObjects.getObjectLoader(LooseObjects.java:210)
        at org.eclipse.jgit.internal.storage.file.LooseObjects.open(LooseObjects.java:172)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openLooseObject(ObjectDirectory.java:426)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:370)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:360)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:140)
        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:216)
        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:1110)

Regards,

Shad

Matthias Sohn

unread,
Sep 27, 2023, 9:26:06 AM9/27/23
to tech....@gmail.com, Repo and Gerrit Discussion
On Wed, Sep 27, 2023 at 6:24 AM tech....@gmail.com <tech....@gmail.com> wrote:
Hello Team,

We have Master , mirror architecture. Currently we are fasing issue with one of the repo not being replicated due to some curropted blob may be. 

What is the safest way to remediate it, without losing data.

Run `git fsck --full` on the bare repo on the gerrit server and check if it reports any errors.
 
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/ce9bb2a7-b5b0-425a-8c5e-245275e1e929n%40googlegroups.com.

Shad Perwez

unread,
Sep 27, 2023, 3:18:21 PM9/27/23
to Matthias Sohn, Repo and Gerrit Discussion


On Wed, 27 Sept, 2023, 6:56 pm Matthias Sohn, <matthi...@gmail.com> wrote:


On Wed, Sep 27, 2023 at 6:24 AM tech....@gmail.com <tech....@gmail.com> wrote:
Hello Team,

We have Master , mirror architecture. Currently we are fasing issue with one of the repo not being replicated due to some curropted blob may be. 

What is the safest way to remediate it, without losing data.

Run `git fsck --full` on the bare repo on the gerrit server and check if it reports any errors.
Yes it does throw error with one or two files corrupt

What is the best way to remove/delete it. 
Do we need reindex after removing it from FS since it is a shared drive and we are on HA

Martin Fick

unread,
Sep 27, 2023, 3:51:18 PM9/27/23
to Shad Perwez, Matthias Sohn, Repo and Gerrit Discussion
On Wed, 27 Sept 2023 at 13:18, Shad Perwez <tech....@gmail.com> wrote:
On Wed, 27 Sept, 2023, 6:56 pm Matthias Sohn, <matthi...@gmail.com> wrote:
Run `git fsck --full` on the bare repo on the gerrit server and check if it reports any errors.
Yes it does throw error with one or two files corrupt

What is the best way to remove/delete it.

We need more details about the files that have errors, and the errors. Removing a file may not be a valid option. What are the exact errors?
What sort of files are problematic? Are they object files, or pack files, index files (git storage question).  Or are they logical blobs/trees/commits/tags with issues? 

Generally deleting logical objects from a version control system is not good, so your best bet tends to be to find a non corrupted copy of those objects. Since git is a distributed VCS, there usually are copies on someone else's machine, and simply dropping the storage files containing them in the right location in your repo can resolve the issue if you then also remove the corrupted version, but care needs to be taken if it is a pack file to ensure you replace everything in it or you could lose a lot more data,

-Martin

tech....@gmail.com

unread,
Oct 3, 2023, 11:01:04 AM10/3/23
to Repo and Gerrit Discussion
On Thursday, 28 September 2023 at 01:21:18 UTC+5:30 Martin Fick wrote:
On Wed, 27 Sept 2023 at 13:18, Shad Perwez <tech....@gmail.com> wrote:
On Wed, 27 Sept, 2023, 6:56 pm Matthias Sohn, <matthi...@gmail.com> wrote:
Run `git fsck --full` on the bare repo on the gerrit server and check if it reports any errors.
Yes it does throw error with one or two files corrupt

What is the best way to remove/delete it.

We need more details about the files that have errors, and the errors. Removing a file may not be a valid option. What are the exact errors?
What sort of files are problematic? Are they object files, or pack files, index files (git storage question).  Or are they logical blobs/trees/commits/tags with issues? 

Here is the output inside bare repository
git fsck --ful
error: object file ./objects/6d/f50114369a5a087871081c5e22c48082587b37 is empty
error: unable to mmap ./objects/6d/f50114369a5a087871081c5e22c48082587b37: No such file or directory
error: 6df50114369a5a087871081c5e22c48082587b37: object corrupt or missing: ./objects/6d/f50114369a5a087871081c5e22c48082587b37
error: object file ./objects/db/81a767cc15678216366a1e8ed00c9a29030e5d is empty
error: unable to mmap ./objects/db/81a767cc15678216366a1e8ed00c9a29030e5d: No such file or directory
error: db81a767cc15678216366a1e8ed00c9a29030e5d: object corrupt or missing: ./objects/db/81a767cc15678216366a1e8ed00c9a29030e5d
Checking object directories: 100% (256/256), done.
Checking objects: 100% (86592/86592), done.
error: object file ./objects/db/81a767cc15678216366a1e8ed00c9a29030e5d is empty
error: object file ./objects/db/81a767cc15678216366a1e8ed00c9a29030e5d is empty
fatal: loose object db81a767cc15678216366a1e8ed00c9a29030e5d (stored in ./objects/db/81a767cc15678216366a1e8ed00c9a29030e5d) is corrupt 

Matthias Sohn

unread,
Oct 3, 2023, 11:50:40 AM10/3/23
to tech....@gmail.com, Repo and Gerrit Discussion
On Tue, Oct 3, 2023 at 5:01 PM tech....@gmail.com <tech....@gmail.com> wrote:


On Thursday, 28 September 2023 at 01:21:18 UTC+5:30 Martin Fick wrote:
On Wed, 27 Sept 2023 at 13:18, Shad Perwez <tech....@gmail.com> wrote:
On Wed, 27 Sept, 2023, 6:56 pm Matthias Sohn, <matthi...@gmail.com> wrote:
Run `git fsck --full` on the bare repo on the gerrit server and check if it reports any errors.
Yes it does throw error with one or two files corrupt

What is the best way to remove/delete it.

We need more details about the files that have errors, and the errors. Removing a file may not be a valid option. What are the exact errors?
What sort of files are problematic? Are they object files, or pack files, index files (git storage question).  Or are they logical blobs/trees/commits/tags with issues? 

Here is the output inside bare repository
git fsck --ful
error: object file ./objects/6d/f50114369a5a087871081c5e22c48082587b37 is empty
error: unable to mmap ./objects/6d/f50114369a5a087871081c5e22c48082587b37: No such file or directory
error: 6df50114369a5a087871081c5e22c48082587b37: object corrupt or missing: ./objects/6d/f50114369a5a087871081c5e22c48082587b37
error: object file ./objects/db/81a767cc15678216366a1e8ed00c9a29030e5d is empty
error: unable to mmap ./objects/db/81a767cc15678216366a1e8ed00c9a29030e5d: No such file or directory
error: db81a767cc15678216366a1e8ed00c9a29030e5d: object corrupt or missing: ./objects/db/81a767cc15678216366a1e8ed00c9a29030e5d
Checking object directories: 100% (256/256), done.
Checking objects: 100% (86592/86592), done.
error: object file ./objects/db/81a767cc15678216366a1e8ed00c9a29030e5d is empty
error: object file ./objects/db/81a767cc15678216366a1e8ed00c9a29030e5d is empty
fatal: loose object db81a767cc15678216366a1e8ed00c9a29030e5d (stored in ./objects/db/81a767cc15678216366a1e8ed00c9a29030e5d) is corrupt 

Search your backups for the objects which git fsck found missing or corrupted.
Hopefully you take regular backups ? See [1].
You can also search clones of the corrupt repo for the objects needed to repair.
See [2] and [3] for ways to repair corruption. 

Important: first create a backup of the repo and try any repair surgery on a copy.
You can try this
  1. Create a clone of a known good repository.
  2. Add the clone as an object alternate [4] to the broken repository.
  3. Do a git-repack -a -d to lift the external objects into repo-local packs [5].
  4. Remove the clone


Generally deleting logical objects from a version control system is not good, so your best bet tends to be to find a non corrupted copy of those objects. Since git is a distributed VCS, there usually are copies on someone else's machine, and simply dropping the storage files containing them in the right location in your repo can resolve the issue if you then also remove the corrupted version, but care needs to be taken if it is a pack file to ensure you replace everything in it or you could lose a lot more data,

-Martin

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages