MergeException during reindex after Gerrit upgrade to 2.11
409 views
Skip to first unread message
Khai Do
Nov 3, 2015, 2:04:22 AM11/3/15
to Repo and Gerrit Discussion
I am attempting to upgrade from Gerrit 2.8.4 to Gerrit 2.11.4. I do not get any failures when upgrading the database however when I go to reindex I get a bunch of com.google.gerrit.server.git.MergeException due to org.eclipse.jgit.errors.MissingObjectException and org.eclipse.jgit.errors.IncorrectObjectTypeException. I was wondering if anybody else has run into this? Is there a way to tell Gerrit to resolve these? or is this something that's benign and we should just leave it alone?
Full log is here: http://paste.openstack.org/show/477827
---- log snippet -----
Reindexing changes: projects: 66% (522/783), 61% (144712/234672) (\)[2015-11-03 05:25:50,163] ERROR com.google.gerrit.server.change.MergeabilityCacheImpl : Error checking mergeability of b540ea78e95fe1fe917f4adfc471bcc411a4f185 into 06eba86fce5d1c3683eb86273e63065106d95196 (MERGE_IF_NECESSARY) com.google.gerrit.server.git.MergeException: Cannot merge b540ea78e95fe1fe917f4adfc471bcc411a4f185 at com.google.gerrit.server.git.MergeUtil.canMerge(MergeUtil.java:403) at com.google.gerrit.server.git.strategy.MergeIfNecessary.dryRun(MergeIfNecessary.java:69) at com.google.gerrit.server.change.MergeabilityCacheImpl$Loader.load(MergeabilityCacheImpl.java:239) at com.google.gerrit.server.change.MergeabilityCacheImpl$Loader.load(MergeabilityCacheImpl.java:210) at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.load(H2CacheImpl.java:229) at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.load(H2CacheImpl.java:209) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) at com.google.common.cache.LocalCache.get(LocalCache.java:3937) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) at com.google.gerrit.server.cache.h2.H2CacheImpl.get(H2CacheImpl.java:113) at com.google.gerrit.server.change.MergeabilityCacheImpl.get(MergeabilityCacheImpl.java:297) at com.google.gerrit.server.query.change.ChangeData.isMergeable(ChangeData.java:635) at com.google.gerrit.server.index.ChangeField$21.get(ChangeField.java:452) at com.google.gerrit.server.index.ChangeField$21.get(ChangeField.java:448) at com.google.gerrit.server.index.Schema$1.apply(Schema.java:100) at com.google.gerrit.server.index.Schema$1.apply(Schema.java:95) at com.google.common.collect.Iterators$8.transform(Iterators.java:799) at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:651) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at com.google.gerrit.lucene.LuceneChangeIndex.toDocument(LuceneChangeIndex.java:499) at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:306) at com.google.gerrit.server.index.ChangeIndexer.index(ChangeIndexer.java:153) at com.google.gerrit.server.index.SiteIndexer$ProjectIndexer.getPathsAndIndex(SiteIndexer.java:352) at com.google.gerrit.server.index.SiteIndexer$ProjectIndexer.call(SiteIndexer.java:328) at com.google.gerrit.server.index.SiteIndexer$3.call(SiteIndexer.java:250) at com.google.gerrit.server.index.SiteIndexer$3.call(SiteIndexer.java:234) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob bdcf7470de2547638989cf3a75681f7d8cfc4da6 at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:147) at org.eclipse.jgit.merge.ResolveMerger.getRawText(ResolveMerger.java:853) at org.eclipse.jgit.merge.ResolveMerger.contentMerge(ResolveMerger.java:664) at org.eclipse.jgit.merge.ResolveMerger.processEntry(ResolveMerger.java:642) at org.eclipse.jgit.merge.ResolveMerger.mergeTreeWalk(ResolveMerger.java:1072) at org.eclipse.jgit.merge.ResolveMerger.mergeTrees(ResolveMerger.java:1022) at org.eclipse.jgit.merge.ResolveMerger.mergeImpl(ResolveMerger.java:304) at org.eclipse.jgit.merge.Merger.merge(Merger.java:202) at org.eclipse.jgit.merge.Merger.merge(Merger.java:155) at org.eclipse.jgit.merge.ThreeWayMerger.merge(ThreeWayMerger.java:114) at com.google.gerrit.server.git.MergeUtil.canMerge(MergeUtil.java:396) ... 40 more Reindexing changes: projects: 75% (589/783), 70% (164543/234672) (/)[2015-11-03 05:26:53,682] ERROR com.google.gerrit.server.change.MergeabilityCacheImpl : Error checking mergeability of f75eee844123719bd7612ee1e3cf3bf3c6fc792d into 0d9423ebc3b81fe3824c4c0ab3d5fd7a789c96dc (MERGE_IF_NECESSARY) com.google.gerrit.server.git.MergeException: Cannot merge f75eee844123719bd7612ee1e3cf3bf3c6fc792d at com.google.gerrit.server.git.MergeUtil.canMerge(MergeUtil.java:403) at com.google.gerrit.server.git.strategy.MergeIfNecessary.dryRun(MergeIfNecessary.java:69) at com.google.gerrit.server.change.MergeabilityCacheImpl$Loader.load(MergeabilityCacheImpl.java:239) at com.google.gerrit.server.change.MergeabilityCacheImpl$Loader.load(MergeabilityCacheImpl.java:210) at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.load(H2CacheImpl.java:229) at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.load(H2CacheImpl.java:209) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) at com.google.common.cache.LocalCache.get(LocalCache.java:3937) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) at com.google.gerrit.server.cache.h2.H2CacheImpl.get(H2CacheImpl.java:113) at com.google.gerrit.server.change.MergeabilityCacheImpl.get(MergeabilityCacheImpl.java:297) at com.google.gerrit.server.query.change.ChangeData.isMergeable(ChangeData.java:635) at com.google.gerrit.server.index.ChangeField$21.get(ChangeField.java:452) at com.google.gerrit.server.index.ChangeField$21.get(ChangeField.java:448) at com.google.gerrit.server.index.Schema$1.apply(Schema.java:100) at com.google.gerrit.server.index.Schema$1.apply(Schema.java:95) at com.google.common.collect.Iterators$8.transform(Iterators.java:799) at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:651) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at com.google.gerrit.lucene.LuceneChangeIndex.toDocument(LuceneChangeIndex.java:499) at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:306) at com.google.gerrit.server.index.ChangeIndexer.index(ChangeIndexer.java:153) at com.google.gerrit.server.index.SiteIndexer$ProjectIndexer.getPathsAndIndex(SiteIndexer.java:352) at com.google.gerrit.server.index.SiteIndexer$ProjectIndexer.call(SiteIndexer.java:328) at com.google.gerrit.server.index.SiteIndexer$3.call(SiteIndexer.java:250) at com.google.gerrit.server.index.SiteIndexer$3.call(SiteIndexer.java:234) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.eclipse.jgit.errors.IncorrectObjectTypeException: Object 8e06772faf74df5a1f59cba44cd564aa674e16ef is not a blob. at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:150) at org.eclipse.jgit.merge.ResolveMerger.getRawText(ResolveMerger.java:853) at org.eclipse.jgit.merge.ResolveMerger.contentMerge(ResolveMerger.java:664) at org.eclipse.jgit.merge.ResolveMerger.processEntry(ResolveMerger.java:611) at org.eclipse.jgit.merge.ResolveMerger.mergeTreeWalk(ResolveMerger.java:1072) at org.eclipse.jgit.merge.ResolveMerger.mergeTrees(ResolveMerger.java:1022) at org.eclipse.jgit.merge.ResolveMerger.mergeImpl(ResolveMerger.java:304) at org.eclipse.jgit.merge.Merger.merge(Merger.java:202) at org.eclipse.jgit.merge.Merger.merge(Merger.java:155) at org.eclipse.jgit.merge.ThreeWayMerger.merge(ThreeWayMerger.java:114) at com.google.gerrit.server.git.MergeUtil.canMerge(MergeUtil.java:396) ... 40 more
Matthias Sohn
Nov 3, 2015, 4:51:32 AM11/3/15
to Khai Do, Repo and Gerrit Discussion
did you try to run git fsck to detect corrupt or missing objects in the affected repositories ?
-Matthias
Khai Do
Nov 4, 2015, 1:55:13 AM11/4/15
to Repo and Gerrit Discussion, zaro...@gmail.com
Here's some additional info:
- The MergeExceptions do not happen when reindexing with our current Gerrit 2.8. It only occurs when we reindex after upgrading to Gerrit 2.11.
- All of the changes flagged as "Cannot merge" are in the abandoned state.
- The SHAs referenced in the MergeExceptions happen in 3 of our repos: openstack/packstack, openstack/refstack-client, openstack/openstack-manuals
When I run 'git fsck' on those repos I don't see any problems:
~/temp/packstack$ git fsck --dangling --unreachable --strict --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (13775/13775), done.
~/temp/refstack-client$ git fsck --dangling --unreachable --strict --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (390/390), done.
~/temp/openstack-manuals$ git fsck --dangling --unreachable --strict --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (114311/114311), done.
Also I was able to start Gerrit 2.11 after the upgrade without error. I can even view the changes referenced by the MergeException without problems.
Ian Kumlien
Nov 4, 2015, 2:49:17 AM11/4/15
to Khai Do, Repo and Gerrit Discussion
I'm pretty sure that this is part of the "can be merged" status message - it could be that you have listed changes somewhere in the database that is no longer valid (for various reasons) and that is what you see failing...
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
David Pursehouse
Nov 4, 2015, 3:10:27 AM11/4/15
to Khai Do, Repo and Gerrit Discussion
On 11/04/2015 03:55 PM, Khai Do wrote:
> Here's some additional info:
>
> - The MergeExceptions do not happen when reindexing with our current
> Gerrit 2.8. It only occurs when we reindex after upgrading to Gerrit 2.11.
The mergeability of a change is only stored in the secondary index since
> Here's some additional info:
>
> - The MergeExceptions do not happen when reindexing with our current
> Gerrit 2.8. It only occurs when we reindex after upgrading to Gerrit 2.11.
2.11 [1], so that's why you don't see this error with 2.8.
> - All of the changes flagged as "Cannot merge" are in the abandoned state.
> - The SHAs referenced in the MergeExceptions happen in 3 of our repos:
> openstack/packstack, openstack/refstack-client, openstack/openstack-manuals
>
> When I run 'git fsck' on those repos I don't see any problems:
>
Also if you can identify exactly which changes it's failing on, you
could try to run the consistency checker [2] on them.
> ~/temp/packstack$ git fsck --dangling --unreachable --strict --full
> Checking object directories: 100% (256/256), done.
> Checking objects: 100% (13775/13775), done.
>
> ~/temp/refstack-client$ git fsck --dangling --unreachable --strict --full
> Checking object directories: 100% (256/256), done.
> Checking objects: 100% (390/390), done.
>
> ~/temp/openstack-manuals$ git fsck --dangling --unreachable --strict
> --full
> Checking object directories: 100% (256/256), done.
> Checking objects: 100% (114311/114311), done.
>
> Also I was able to start Gerrit 2.11 after the upgrade without error. I
> can even view the changes referenced by the MergeException without
> problems.
>
verbose log message [3]. After the log it returns false, and at the
call site the "mergeable" field is set to false in the index [4], and
indexing continues.
So it looks like the index operation has succeeded, but there might
still be some other underlying consistency problems with those
projects/changes where the logs were emitted.
[1] https://gerrit-review.googlesource.com/#/c/60937/
[2]
https://gerrit-documentation.storage.googleapis.com/Documentation/2.11/rest-api-changes.html#check-change
[3]
https://gerrit.googlesource.com/gerrit/+/stable-2.11/gerrit-server/src/main/java/com/google/gerrit/server/change/MergeabilityCacheImpl.java#299
[4]
https://gerrit.googlesource.com/gerrit/+/stable-2.11/gerrit-server/src/main/java/com/google/gerrit/server/query/change/ChangeData.java#635
Khai Do
Dec 9, 2015, 1:50:29 PM12/9/15
to Repo and Gerrit Discussion, zaro...@gmail.com
I finally got around to answering your questions..
On 11/04/2015 03:55 PM, Khai Do wrote:
> Here's some additional info:
>
> - The MergeExceptions do not happen when reindexing with our current
> Gerrit 2.8. It only occurs when we reindex after upgrading to Gerrit 2.11.
The mergeability of a change is only stored in the secondary index since
2.11 [1], so that's why you don't see this error with 2.8.
> - All of the changes flagged as "Cannot merge" are in the abandoned state.
> - The SHAs referenced in the MergeExceptions happen in 3 of our repos:
> openstack/packstack, openstack/refstack-client, openstack/openstack-manuals
>
> When I run 'git fsck' on those repos I don't see any problems:
>
What do you see with 'git show' on the sha1s mentioned in the error logs?
git show reports bad object:
~/temp/zz/refstack-client$ git show f75eee844123719bd7612ee1e3cf3bf3c6fc792d
fatal: bad object f75eee844123719bd7612ee1e3cf3bf3c6fc792d
Also if you can identify exactly which changes it's failing on, you
could try to run the consistency checker [2] on them.
Consistency checker reports no problems therefore can't fix:
~/temp/zz/refstack-client$ curl --digest --user $MYUSER:$MYPASS http://$MYHOST:8080/a/changes/openstack%2Frefstack-client~master~Iab14982ac2ee954ce198604f7215a1ebae6a7a56/check
)]}'
{
"id": "openstack%2Frefstack-client~master~Iab14982ac2ee954ce198604f7215a1ebae6a7a56",
"project": "openstack/refstack-client",
"branch": "master",
"topic": "unit_tests_for_refstack_client",
"hashtags": [],
"change_id": "Iab14982ac2ee954ce198604f7215a1ebae6a7a56",
"subject": "Add unit tests for refstack-client",
"status": "ABANDONED",
"created": "2014-08-16 20:58:18.000000000",
"updated": "2014-09-22 19:14:12.000000000",
"mergeable": false,
"insertions": 167,
"deletions": 21,
"_number": 114791,
"owner": {
"_account_id": 9377
},
"problems": []
}
David Pursehouse
Dec 9, 2015, 7:21:14 PM12/9/15
to Khai Do, Repo and Gerrit Discussion
On Thu, Dec 10, 2015 at 3:50 AM Khai Do <zaro...@gmail.com> wrote:
I finally got around to answering your questions..On Wednesday, November 4, 2015 at 12:10:27 AM UTC-8, David Pursehouse wrote:On 11/04/2015 03:55 PM, Khai Do wrote:
> Here's some additional info:
>
> - The MergeExceptions do not happen when reindexing with our current
> Gerrit 2.8. It only occurs when we reindex after upgrading to Gerrit 2.11.
The mergeability of a change is only stored in the secondary index since
2.11 [1], so that's why you don't see this error with 2.8.
> - All of the changes flagged as "Cannot merge" are in the abandoned state.
> - The SHAs referenced in the MergeExceptions happen in 3 of our repos:
> openstack/packstack, openstack/refstack-client, openstack/openstack-manuals
>
> When I run 'git fsck' on those repos I don't see any problems:
>
What do you see with 'git show' on the sha1s mentioned in the error logs?
git show reports bad object:~/temp/zz/refstack-client$ git show f75eee844123719bd7612ee1e3cf3bf3c6fc792dfatal: bad object f75eee844123719bd7612ee1e3cf3bf3c6fc792d
The sha1 is visible on the change in Gerrit [1] and in gitweb [2] so it should be there. Where are you running that 'git show' command? If it's local, you need to have explicitly fetched the patch set; it won't be included in a fresh clone/fetch.
Khai Do
Dec 15, 2015, 4:57:37 PM12/15/15
to Repo and Gerrit Discussion, zaro...@gmail.com
git show reports bad object:~/temp/zz/refstack-client$ git show f75eee844123719bd7612ee1e3cf3bf3c6fc792dfatal: bad object f75eee844123719bd7612ee1e3cf3bf3c6fc792dThe sha1 is visible on the change in Gerrit [1] and in gitweb [2] so it should be there. Where are you running that 'git show' command? If it's local, you need to have explicitly fetched the patch set; it won't be included in a fresh clone/fetch.
I am testing on my own test gerrit server with replicated data. I've fetched [1] the patch set and was able to see changes with git show now. The consistency checker still reports no problems therefore I think you are correct that the Error message is a result of just overly verbose logging. I don't see any problems with the patch set. Thanks David.
[1] git fetch http://$MY_HOST:8080/openstack/refstack-client refs/changes/91/114791/7 && git checkout FETCH_HEAD
Reply all
Reply to author
Forward
0 new messages