getting lots of "java.lang.IllegalStateException: Timer already terminated" after upgrade to 3.2.7

rastafaray1100

unread,

Apr 12, 2021, 4:02:31 AM4/12/21

to Repo and Gerrit Discussion

Hi,

we upgraded our gerrit from 2.15.3 to gerrit 3.27 ( inclduing 2.16 and migration to NoteDB)

everything went well, including re-indexing.

but, since then, we get a lot of erorrs, especially when working with Jenins:

v at java.lang.Thread.run(Thread.java:748)

[2021-04-12T10:44:10.796+0300] [SSH git-upload-pack /projects/project1 (jenkins)] ERROR com.googl

e.gerrit.sshd.BaseCommand : Internal server error (user jenkins account 1000280) during git-upload-pa

ck '/projects/project1'

of course the jenking test is failing at this point.

the server is not loaded and it has a lot of CPU'S and RAM.

also, checked the Average pulls/clones per day and it is about 26000 a day.

some repos, are small. 2 or 3 are about 3.5G in size.

searched the Internet for such error without any luck.

Thanks in Advance.

Avishay

Matthias Sohn

unread,

Apr 12, 2021, 4:29:41 AM4/12/21

to rastafaray1100, Repo and Gerrit Discussion

On Mon, Apr 12, 2021 at 10:02 AM rastafaray1100 <avisha...@gmail.com> wrote:

Hi,
we upgraded our gerrit from 2.15.3 to gerrit 3.27 ( inclduing 2.16 and migration to NoteDB)
everything went well, including re-indexing.

I guess you mean 3.2.7

but, since then, we get a lot of erorrs, especially when working with Jenins:
v at java.lang.Thread.run(Thread.java:748)
[2021-04-12T10:44:10.796+0300] [SSH git-upload-pack /projects/project1 (jenkins)] ERROR com.googl
e.gerrit.sshd.BaseCommand : Internal server error (user jenkins account 1000280) during git-upload-pa
ck '/projects/project1'

Can you provide the full stacktrace ?

Are all these errors of the same type ?

Do they happen all on the same repository ?

of course the jenking test is failing at this point.
the server is not loaded and it has a lot of CPU'S and RAM.
also, checked the Average pulls/clones per day and it is about 26000 a day.
some repos, are small. 2 or 3 are about 3.5G in size.

searched the Internet for such error without any luck.

Thanks in Advance.
Avishay

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/116cdc48-e1e7-4e90-834f-029154ff908cn%40googlegroups.com.

rastafaray1100

unread,

Apr 12, 2021, 6:20:28 AM4/12/21

to Repo and Gerrit Discussion

Hi,

1. All errors are of the same type.

2. different repos, different jenkins users, even sometimes regular users.

3. about 200 such errors a day, intermittent.

4. checked queue,connections - not busy. to say the least

Please see below stack trace:

[2021-04-12T11:12:44.600+0300] [SSH git-upload-pack /project/project1/project2 (jenkins1)] ERROR com.google

.gerrit.sshd.BaseCommand : Internal server error (user jenkins1 account 1000253) during git-upload-pa

ck '/project/project1/project2'

java.lang.IllegalStateException: Timer already terminated

at org.eclipse.jgit.util.io.InterruptTimer$AlarmState.begin(InterruptTimer.java:175)

at org.eclipse.jgit.util.io.InterruptTimer.begin(InterruptTimer.java:92)

at org.eclipse.jgit.util.io.TimeoutOutputStream.beginWrite(TimeoutOutputStream.java:127)

at org.eclipse.jgit.util.io.TimeoutOutputStream.write(TimeoutOutputStream.java:91)

at org.eclipse.jgit.transport.UploadPack$ResponseBufferedOutputStream.write(UploadPack.java:237

1)

at org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:141)

at org.eclipse.jgit.transport.SideBandOutputStream.flushBuffer(SideBandOutputStream.java:94)

at org.eclipse.jgit.transport.SideBandOutputStream.flush(SideBandOutputStream.java:100)

at org.eclipse.jgit.transport.UploadPack$SideBandErrorWriter.writeError(UploadPack.java:2404)

at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:766)

at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:101)

at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:108)

at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)

at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:73)

at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:488)

at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable

.java:110)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThr

eadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:610)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

Matthias Sohn

unread,

Apr 13, 2021, 2:01:52 PM4/13/21

to rastafaray1100, Repo and Gerrit Discussion

this is a timeout after transfer.timeout expired, see

https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#transfer

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/3a0f8e21-1be8-4e1b-9dfb-3953e7ff558an%40googlegroups.com.

rastafaray1100

unread,

Apr 13, 2021, 3:12:37 PM4/13/21

to Matthias Sohn, Repo and Gerrit Discussion

Hi,

I have the transfer timeout set to 600s in my config which seems to be enough.

I tested the biggest repo(3.5G) and it took about 2.5 Minutes to clone.

I will try to increase the timeout to 900s and see what happens

is there a way to dump actual server configuration to see the real servers configuration?

Thanks

Avishay

[transfer]
timeout = 600s

Matthias Sohn

unread,

Apr 13, 2021, 3:59:53 PM4/13/21

to rastafaray1100, Repo and Gerrit Discussion

On Tue, Apr 13, 2021 at 9:12 PM rastafaray1100 <avisha...@gmail.com> wrote:

Hi,

I have the transfer timeout set to 600s in my config which seems to be enough.
I tested the biggest repo(3.5G) and it took about 2.5 Minutes to clone.
I will try to increase the timeout to 900s and see what happens
is there a way to dump actual server configuration to see the real servers configuration?

I recently implemented this

https://gerrit-review.googlesource.com/c/gerrit/+/299702

https://gerrit-review.googlesource.com/c/plugins/metrics-reporter-prometheus/+/299703

but in the review discussion we found that exposing the current configuration via metrics is not

the best approach since it may expose sensitive configuration data and monitoring systems