javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

1,539 views
Skip to first unread message

Sasa Zivkov

unread,
May 9, 2011, 11:47:10 AM5/9/11
to repo-d...@googlegroups.com
We are running gerrit with embedded Jetty and over https.
Since yesterday we are getting millions of log entries like:

...
3166773 [2011-05-09 14:22:52,556] WARN  org.eclipse.jetty.util.log : javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
3166774 [2011-05-09 14:22:52,556] WARN  org.eclipse.jetty.util.log : javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
3166775 [2011-05-09 14:22:52,556] WARN  org.eclipse.jetty.util.log : javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
3166776 [2011-05-09 14:22:52,556] WARN  org.eclipse.jetty.util.log : javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
3166777 [2011-05-09 14:22:52,556] WARN  org.eclipse.jetty.util.log : javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
...

the first column is the line number from my text editor.

From timestamps we see that log entries are generated in chunks... i.e. within a few minutes hundreds of thousands of them are written, then there is "silence"
for some period of time and then again the next chunk comes.

At the same time the number of entries in httpd_log doesn't increase. Does it mean that these warnings are not caused by an incoming https request?

We recently (few days ago) upgraded to the latest RC (2.1.7-rc0). I think we haven't seen this issue before the upgrade but can't confirm 100%.
And, I just upgraded java to 1.6.0_25 but this didn't help.

Is there a way to get more info logged for these warnings (full stack trace, IP Address of the incoming request, etc...)?

Sasa Zivkov

Shawn Pearce

unread,
May 9, 2011, 1:21:58 PM5/9/11
to Sasa Zivkov, repo-d...@googlegroups.com
On Mon, May 9, 2011 at 08:47, Sasa Zivkov <ziv...@gmail.com> wrote:
> We are running gerrit with embedded Jetty and over https.
> Since yesterday we are getting millions of log entries like:
>
> ...
> 3166773 [2011-05-09 14:22:52,556] WARN  org.eclipse.jetty.util.log :
> javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
> 3166774 [2011-05-09 14:22:52,556] WARN  org.eclipse.jetty.util.log :
> javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
> 3166775 [2011-05-09 14:22:52,556] WARN  org.eclipse.jetty.util.log :
> javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
> 3166776 [2011-05-09 14:22:52,556] WARN  org.eclipse.jetty.util.log :
> javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
> 3166777 [2011-05-09 14:22:52,556] WARN  org.eclipse.jetty.util.log :
> javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
> ...
>
> the first column is the line number from my text editor.
>
> From timestamps we see that log entries are generated in chunks... i.e.
> within a few minutes hundreds of thousands of them are written, then there
> is "silence"
> for some period of time and then again the next chunk comes.
>
> At the same time the number of entries in httpd_log doesn't increase. Does
> it mean that these warnings are not caused by an incoming https request?

Sounds like its Jetty's selector bug triggering again, but for SSL
instead of plain text HTTP.

> We recently (few days ago) upgraded to the latest RC (2.1.7-rc0). I think we
> haven't seen this issue before the upgrade but can't confirm 100%.
> And, I just upgraded java to 1.6.0_25 but this didn't help.

We did upgrade Jetty. Its possible there is a new Jetty bug.

> Is there a way to get more info logged for these warnings (full stack trace,
> IP Address of the incoming request, etc...)?

You would need to hand-edit the log4j.properties file in the Gerrit
WAR file (which is probably itself buried within a JAR file somewhere,
I forget where). We don't support changing the logging except by
editing the source and rebuilding.

Sasa Zivkov

unread,
May 10, 2011, 9:01:08 AM5/10/11
to Shawn Pearce, repo-d...@googlegroups.com
As a test I downgraded gerrit version back to 7.2.1.v20101111 and since then the issue
is (almost) solved. Actually, it is still there but instead of millions we see only about 10 log
entries and there is no high CPU usage symptom any more. This issue was probably always
there but seems like Jetty 7.2.1... was better in detecting and stopping it.

I posted a question on jetty-users mailing list about that.


Shall we downgrade gerrit version for our v2.1.7-rcX ?

Shawn Pearce

unread,
May 10, 2011, 10:15:17 AM5/10/11
to Sasa Zivkov, repo-d...@googlegroups.com
On Tue, May 10, 2011 at 06:01, Sasa Zivkov <ziv...@gmail.com> wrote:
> As a test I downgraded gerrit version back to 7.2.1.v20101111 and since then
> the issue
> is (almost) solved. Actually, it is still there but instead of millions we
> see only about 10 log
> entries and there is no high CPU usage symptom any more. This issue was
> probably always
> there but seems like Jetty 7.2.1... was better in detecting and stopping it.
>
> I posted a question on jetty-users mailing list about that.
>
> Shall we downgrade gerrit version for our v2.1.7-rcX ?

Yes. I'll revert that upgrade. We will use the same Jetty we have in 2.1.6.1.

Reply all
Reply to author
Forward
0 new messages