StackOverflowError when trying to access a change

[Bassem Rabil]

Hi 

We are running a Gerrit instance using  stable-2.6 branch (2.6.1 and several bug fixes targeted for 2.6.2). When we try to access a specific change, we get the following error, and eventually ends up with Jetty crash:

 ERROR org.apache.wicket.Session : Exception when detaching/serializing page

java.lang.StackOverflowError

        at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1779)

        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1492)

        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1429)

        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1175)

        at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1375)

        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1171)

For this change the following entries in the DB are there:

-In patch_sets table the following entries are there for this change

<Commit-ID1> | <User-ID2>             | 2013-09-09 05:55:37.796-04 | <Change-ID>   | 1            | N

<Commit-ID2> | <User-ID2>             | 2013-09-10 03:51:43.22-04   | <Change-ID>   | 2            | N

<Commit-ID3> | <User-ID2>             | 2013-09-10 04:57:15.022-04 | <Change-ID>   | 3            | N

<Commit-ID4> | <User-ID1>             | 2013-09-10 07:39:47.512-04 | <Change-ID>   | 4            | N

-And on the patch_set_approvals, the following entries are there:

0     | 2013-09-10 04:46:00.232-04 | Y           | <Change-sort-key> | <Change-ID>    | 1            | <User-ID1>    | Verified

0     | 2013-09-10 04:46:00.486-04 | Y           | <Change-sort-key> | <Change-ID>    | 2            | <User-ID1>    | Code-Review

1     | 2013-09-09 07:07:48.242-04 | Y           | <Change-sort-key> | <Change-ID>    | 1            | <User-ID2>    | Verified

1     | 2013-09-09 07:07:48.242-04 | Y           | <Change-sort-key> | <Change-ID>    | 1            | <User-ID2>   | Code-Review

-I am attaching the jstack for this particular query for this change.

Any hint on how to fix this issue will be very much appreciated.

Thanks

Bassem

[Bassem Rabil]

When accessing this change using the Web-UI, the browser is not displaying the change and you get Proxy error 502, and using command line the gerrit query for this change is hanging in the process queue. Eventually this cause a crash on Jetty because of this change specifically.

[Bassem Rabil]

Debugging this issue showed that it gets stuck on the method DiffFormatter.toFileHeader from JGit Library at the Gerrit Code at: PatchListLoader.readPatchList(PatchListKey, Repository)

A small detail which may help is that this change involved deleting a file of 6 MB size.

Thread [SSH gerrit query --format=JSON --patch-sets --current-patch-set <Change-ID> (<User-ID>)] (Suspended)

owns: ZStreamRef  (id=90)

owns: BufferedInputStream  (id=91)

owns: BufferedInputStream  (id=92)

owns: LocalCache$StrongAccessEntry<K,V>  (id=93)

owns: BaseCommand$TaskThunk  (id=94)

Inflater.inflateBytes(long, byte[], int, int) line: not available [native method]

Inflater.inflate(byte[], int, int) line: 256

InflaterInputStream.read(byte[], int, int) line: 152

InflaterInputStream.skip(long) line: 208

BufferedInputStream.skip(long) line: 366

ObjectStream$Filter.skip(long) line: 199

IO.skipFully(InputStream, long) line: 330

LargePackedDeltaObject$2(DeltaStream).seekBase() line: 333

LargePackedDeltaObject$2(DeltaStream).read(byte[], int, int) line: 213

BufferedInputStream.fill() line: 235

BufferedInputStream.read1(byte[], int, int) line: 275

BufferedInputStream.read(byte[], int, int) line: 334

BufferedInputStream(FilterInputStream).read(byte[]) line: 107

TeeInputStream.close() line: 121

LargePackedDeltaObject$1(ObjectStream$Filter).close() line: 229

LargePackedDeltaObject$1.close() line: 198

LargePackedDeltaObject(ObjectLoader).getCachedBytes(int) line: 206

LargePackedDeltaObject(ObjectLoader).getBytes(int) line: 130

DiffFormatter.open(DiffEntry$Side, DiffEntry) line: 993

DiffFormatter.createFormatResult(DiffEntry) line: 918

DiffFormatter.toFileHeader(DiffEntry) line: 889

PatchListLoader.readPatchList(PatchListKey, Repository) line: 157

PatchListLoader.load(PatchListKey) line: 81

PatchListLoader.load(Object) line: 67

H2CacheImpl$Loader<K,V>.load(K) line: 247

H2CacheImpl$Loader<K,V>.load(Object) line: 227

LocalCache$LoadingValueReference<K,V>.loadFuture(K, CacheLoader<? super K,V>) line: 3599

LocalCache$Segment<K,V>.loadSync(K, int, LoadingValueReference<K,V>, CacheLoader<? super K,V>) line: 2379

LocalCache$Segment<K,V>.lockedGetOrLoad(K, int, CacheLoader<? super K,V>) line: 2342

LocalCache$Segment<K,V>.get(K, int, CacheLoader<? super K,V>) line: 2257

LocalCache<K,V>.get(K, CacheLoader<? super K,V>) line: 4000

LocalCache<K,V>.getOrLoad(K) line: 4004

LocalCache$LocalLoadingCache<K,V>.get(K) line: 4874

H2CacheImpl<K,V>.get(K) line: 109

PatchListCacheImpl.get(PatchListKey) line: 81

PatchListCacheImpl.get(Change, PatchSet) line: 95

EventFactory.asPatchSetAttribute(PatchSet) line: 373

EventFactory.addPatchSets(ChangeAttribute, Collection<PatchSet>, Map<Id,Collection<PatchSetApproval>>, boolean, Change, LabelTypes) line: 274

EventFactory.addPatchSets(ChangeAttribute, Collection<PatchSet>, Map<Id,Collection<PatchSetApproval>>, LabelTypes) line: 265

QueryProcessor.query(String) line: 300

Query.run() line: 85

SshCommand$1.run() line: 35

BaseCommand$TaskThunk.run() line: 429

Executors$RunnableAdapter<T>.call() line: 471

FutureTask$Sync.innerRun() line: 334

ScheduledThreadPoolExecutor$ScheduledFutureTask<V>(FutureTask<V>).run() line: 166

ScheduledThreadPoolExecutor$ScheduledFutureTask<V>.access$201(ScheduledThreadPoolExecutor$ScheduledFutureTask) line: 178

ScheduledThreadPoolExecutor$ScheduledFutureTask<V>.run() line: 292

WorkQueue$ProjectTask<V>(WorkQueue$Task<V>).run() line: 337

WorkQueue$Executor(ThreadPoolExecutor).runWorker(ThreadPoolExecutor$Worker) line: 1145

ThreadPoolExecutor$Worker.run() line: 615

Thread.run() line: 724

[Bassem Rabil]

This turned out to be caused by corrupted repository, when we ran gerrit gc on this repository we got the following error in our logs:

[TIME-STAMP] ERROR com.google.gerrit.server.events.EventFactory : Cannot get size information for <Change-ID>,<patchset-id>.
com.google.gerrit.server.patch.PatchListNotAvailableException: java.io.FileNotFoundException: <repository-path>/objects/pack/pack-<SHA1>.pack (No such file or directory)

We checked the logs and found the relevant StackOverFlowError has started to appear after we upgraded to Gerrit 2.6 which introduced the new jgit library.

Does anyone have similar issues of corrupted repositories ? Will running gerrit gc be enough to eliminate such inconsistencies for repositories ?

How frequent you normally run gerrit gc ? Does this garbage collection affect pulling/pushing to the repositories ?

Thanks

Bassem