BFG git repo modifications gerrit compatibility

[dbiFP]

Hello,

I have been trying for a while now to modify a gerrit hosted project using the BFG tool. Basically to remove big binary files that should have never made it to the project...

This does not seem to be supported by gerrit. Here is the detail description of the problem:

Let an existing git repository in gerrit project called 'project1'. This project as big files I want to get rid of.

To do so I first 'mirror' clone to a separate git project 'project2' using bare git mirror clone command.

This project2 is not under gerrit git repositories.

In gerrit I create a brand new project 'project2' without initial commit and then using git command line I push 'mirror' the previously mirrored git repo 'project2'.

At this stage I have what seems to be a working gerrit project name 'project2', I can push reviews to it and all seems fine.

However, whenever I try to perform a gerrit reindex from this state, it fails miserably. My guess is that gerrit stores its metadata in the repository in a manner that if the git history is modified by BFG, gerrit will not be able to recover from this new state.

Does anyone have any idea how this could be worked out? Is there some way to cleanup a git repository from all the gerrit metadata without breaking the repo?

Thanks in advance!

[Nasser Grainawi]

Hey there,

I’m not super familiar with what all BFG can do, but there definitely could be issues if it tries to remove data Gerrit expects to be present.

What version of Gerrit are you running?

My best guess would be that you’re running a 3.* version of Gerrit and you haven’t removed project1. In that case, I would expect Gerrit to find all the changes in project1 and duplicate changes in project2. Do you have error messages that go with “fails miserably”?

Nasser

Thanks in advance!

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/6304f4c6-7de6-4d6e-8610-c5d08198f101n%40googlegroups.com.

[dbiFP]

Hello Nasser,

Thanks for your answer, the bfg repo-cleaner tool is an alternative to regular git-filter-branch that allows to remove some files in the git repository history.

It basically rewrites the git history and removes unwanted files from the git history. I am using this to cleanup big files from our repo.

Our gerrit version is the v3.3.1. The project1 is indeed not removed.

During the reindex a huge number of error like this one occur:

[2021-05-05 12:13:57,730] [Index-Batch-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@34207b95[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@4b83dd53[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@55394332]]]] WARN  com.google.gerrit.server.index.change.AllChangesIndexer : Failed to read change 60821 for indexing
com.google.gerrit.exceptions.StorageException: java.io.IOException: Error loading refs/changes/21/60821/meta in project2.git_ at a37e046175c6bc6fea5829253172067f643f7112
    at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:152)
    at com.google.gerrit.server.notedb.ChangeNotes$Factory.toResult(ChangeNotes.java:258)
    at com.google.gerrit.server.notedb.ChangeNotes$Factory.scanOneChange(ChangeNotes.java:246)
    at com.google.gerrit.server.notedb.ChangeNotes$Factory.lambda$scan$2(ChangeNotes.java:231)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
    at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
    at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
    at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectIndexer.call(AllChangesIndexer.java:287)
    at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectIndexer.call(AllChangesIndexer.java:254)
    at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
    at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
    at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
    at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:103)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Error loading refs/changes/21/60821/meta in project2.git_ at a37e046175c6bc6fea5829253172067f643f7112
    at com.google.gerrit.server.notedb.ChangeNotesCache.get(ChangeNotesCache.java:384)
    at com.google.gerrit.server.notedb.ChangeNotes.onLoad(ChangeNotes.java:554)
    at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:149)
    ... 26 more
Caused by: java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.ConfigInvalidException: Change 60821: missing or multiple Patch-set: []
    at com.github.benmanes.caffeine.guava.CaffeinatedGuavaCache.get(CaffeinatedGuavaCache.java:88)
    at com.google.gerrit.server.notedb.ChangeNotesCache.get(ChangeNotesCache.java:380)
    ... 28 more
Caused by: org.eclipse.jgit.errors.ConfigInvalidException: Change 60821: missing or multiple Patch-set: []
    at com.google.gerrit.server.notedb.ChangeNotes.parseException(ChangeNotes.java:91)
    at com.google.gerrit.server.notedb.ChangeNotesParser.parseException(ChangeNotesParser.java:1132)
    at com.google.gerrit.server.notedb.ChangeNotesParser.expectedOneFooter(ChangeNotesParser.java:1103)
    at com.google.gerrit.server.notedb.ChangeNotesParser.parseExactlyOneFooter(ChangeNotesParser.java:484)
    at com.google.gerrit.server.notedb.ChangeNotesParser.parsePatchSetId(ChangeNotesParser.java:651)
    at com.google.gerrit.server.notedb.ChangeNotesParser.parse(ChangeNotesParser.java:330)
    at com.google.gerrit.server.notedb.ChangeNotesParser.parseAll(ChangeNotesParser.java:198)
    at com.google.gerrit.server.notedb.ChangeNotesCache$Loader.call(ChangeNotesCache.java:354)
    at com.google.gerrit.server.notedb.ChangeNotesCache$Loader.call(ChangeNotesCache.java:336)
    at com.github.benmanes.caffeine.guava.CaffeinatedGuavaCache.lambda$get$0(CaffeinatedGuavaCache.java:69)
    at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2344)
    at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1947)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2342)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2325)
    at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
    at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
    at com.github.benmanes.caffeine.guava.CaffeinatedGuavaCache.get(CaffeinatedGuavaCache.java:67)
    ... 29 more

Thanks for you support !