Gerrit 2.12.2, 500 internal server error caused by Lucene Exception 'ClosedChannelException' related to ssh git-receive-pack killed

wenliang yang

unread,

Mar 24, 2016, 9:51:42 AM3/24/16

to Repo and Gerrit Discussion

can anybody help on this issue? How to fix it ? Thanks a lot!

[2016-03-24 14:42:56,514] [SSH git-receive-pack '/dongms/shuqireaderios' (zhangchao4)] ERROR com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook : Cannot list open changes of dongms/shuqireaderios

com.google.gwtorm.server.OrmException: java.io.IOException: null: NIOFSIndexInput(path="/home/gerrit/review_site/index/changes_0025/open/_9y_Lucene50_0.tim")

at com.google.gerrit.lucene.LuceneChangeIndex$QuerySource.read(LuceneChangeIndex.java:465)

at com.google.gerrit.server.index.IndexedChangeQuery.read(IndexedChangeQuery.java:106)

at com.google.gerrit.server.query.change.AndSource.readImpl(AndSource.java:115)

at com.google.gerrit.server.query.change.AndSource.read(AndSource.java:99)

at com.google.gerrit.server.query.change.QueryProcessor.queryChanges(QueryProcessor.java:162)

at com.google.gerrit.server.query.change.QueryProcessor.queryChanges(QueryProcessor.java:103)

at com.google.gerrit.server.query.change.QueryProcessor.queryChanges(QueryProcessor.java:87)

at com.google.gerrit.server.query.change.InternalChangeQuery.query(InternalChangeQuery.java:245)

at com.google.gerrit.server.query.change.InternalChangeQuery.byProjectOpen(InternalChangeQuery.java:207)

at com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook.advertiseOpenChanges(ReceiveCommitsAdvertiseRefsHook.java:101)

at com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook.advertiseRefs(ReceiveCommitsAdvertiseRefsHook.java:90)

at org.eclipse.jgit.transport.AdvertiseRefsHookChain.advertiseRefs(AdvertiseRefsHookChain.java:85)

at org.eclipse.jgit.transport.BaseReceivePack.sendAdvertisedRefs(BaseReceivePack.java:1042)

at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:179)

at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:161)

at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:115)

at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)

at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)

at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)

at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:437)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:377)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException: null: NIOFSIndexInput(path="/home/gerrit/review_site/index/changes_0025/open/_9y_Lucene50_0.tim")

at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:190)

at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)

at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)

at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)

at org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:221)

at org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:157)

at org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:507)

at org.apache.lucene.index.TermContext.build(TermContext.java:94)

at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:192)

at org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:855)

at org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:56)

at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:203)

at org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:855)

at org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:838)

at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:486)

at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:695)

at org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:680)

at org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:622)

at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:528)

at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:578)

at com.google.gerrit.lucene.LuceneChangeIndex$QuerySource.read(LuceneChangeIndex.java:435)

... 27 more

Caused by: java.nio.channels.ClosedByInterruptException

at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)

at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:705)

at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)

at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:180)

... 47 more

From sshd_log, "git-receive-pack .. 0ms 4866ms killed" is always seen when it happened.

[2016-03-24 14:42:56,351 +0800] 28ed1681 zhangchao4 a/15 git-receive-pack./dongms/shuqireaderios 0ms 4866ms killed

Saša Živkov

unread,

Mar 30, 2016, 11:23:23 AM3/30/16

to wenliang yang, Repo and Gerrit Discussion

On Thu, Mar 24, 2016 at 8:11 AM, wenliang yang <xiaoxi...@gmail.com> wrote:

can anybody help on this issue? How to fix it ? Thanks a lot!

[2016-03-24 14:42:56,514] [SSH git-receive-pack '/dongms/shuqireaderios' (zhangchao4)] ERROR com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook : Cannot list open changes of dongms/shuqireaderios
com.google.gwtorm.server.OrmException: java.io.IOException: null: NIOFSIndexInput(path="/home/gerrit/review_site/index/changes_0025/open/_9y_Lucene50_0.tim")

Do you also see something like: "MultiProgressMonitor worker killed after ..." in the error_log?

What is your setting for the receive.timeout [1] and transfer.timeout [2] parameters?

[1] https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#receive.timeout

[2] https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#transfer.timeout

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

wenliang yang

unread,

Apr 5, 2016, 10:18:41 PM4/5/16

to Saša Živkov, Repo and Gerrit Discussion

hello Saša,

Thanks for your reply! See my answer in red color.

The problem is that every time the git-receive-pack ssh session got killed for some reason (maybe network issue), the ClosedByInterruptException will occur at the same time, and I have to restart gerrit to recovery.

On Wed, Mar 30, 2016 at 11:22 PM, Saša Živkov <ziv...@gmail.com> wrote:

On Thu, Mar 24, 2016 at 8:11 AM, wenliang yang <xiaoxi...@gmail.com> wrote:
can anybody help on this issue? How to fix it ? Thanks a lot!

[2016-03-24 14:42:56,514] [SSH git-receive-pack '/dongms/shuqireaderios' (zhangchao4)] ERROR com.google.gerrit.server.git.ReceiveCommitsAdvertiseRefsHook : Cannot list open changes of dongms/shuqireaderios
com.google.gwtorm.server.OrmException: java.io.IOException: null: NIOFSIndexInput(path="/home/gerrit/review_site/index/changes_0025/open/_9y_Lucene50_0.tim")

Do you also see something like: "MultiProgressMonitor worker killed after ..." in the error_log?

No, I haven't seen it in error_log.

1) Have tried several receive.timeout settings, from 2 min to 6 min, did't solve the problem. Now remove the setting of section receive meaning using the default timeout.

2) I did't do any change on transfer.timeout part.

Reply all

Reply to author

Forward