[JIRA] (JENKINS-60857) jetty no longer accepts the keystore

48 views
Skip to first unread message

christian.keck@macio.de (JIRA)

unread,
Jan 24, 2020, 5:13:03 AM1/24/20
to jenkinsc...@googlegroups.com
Christian Keck created an issue
 
Jenkins / Bug JENKINS-60857
jetty no longer accepts the keystore
Issue Type: Bug Bug
Assignee: Unassigned
Components: core
Created: 2020-01-24 10:12
Environment: CentOS 6.10
Jenkins 2.217
Wildcard-SSL-Certificate in Java-Keystore in PKCS12 format
Labels: jenkins SSL jetty
Priority: Critical Critical
Reporter: Christian Keck

With 2.217 Jenkins no longer accepts the supplied keystore which worked flawlessly with all former versions.

It complains about "multiple certificates" even if there is only one stored in the keystore.

Re-creating the keystore doesn't change a thing.

Here's the log output:

2020-01-24 09:59:56.255+0000 [id=1]     SEVERE  winstone.Logger#logInternal: Container startup failed
java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)
        at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1275)
        at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1256)
        at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374)
        at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:92)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:320)
        at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
        at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.server.Server.doStart(Server.java:385)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at winstone.Launcher.<init>(Launcher.java:188)
Caused: java.io.IOException: Failed to start Jetty
        at winstone.Launcher.<init>(Launcher.java:190)
        at winstone.Launcher.main(Launcher.java:359)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at Main._main(Main.java:375)
        at Main.main(Main.java:151)
2020-01-24 09:59:56.256+0000 [id=22]    WARNING o.j.h.a.Index$2$1#fetch: Failed to load hudson.model.Queue
java.lang.ClassNotFoundException: hudson.model.queue.QueueSorter
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:543)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
Caused: java.lang.NoClassDefFoundError: hudson/model/queue/QueueSorter
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
        at java.lang.Class.getDeclaredMethods(Class.java:1975)
        at org.jvnet.hudson.annotation_indexer.Index$2$1.fetch(Index.java:103)
        at org.jvnet.hudson.annotation_indexer.Index$2$1.hasNext(Index.java:73)
        at org.jvnet.hudson.annotation_indexer.SubtypeIterator.fetch(SubtypeIterator.java:18)
        at org.jvnet.hudson.annotation_indexer.SubtypeIterator.hasNext(SubtypeIterator.java:28)
        at hudson.init.TaskMethodFinder.discoverTasks(TaskMethodFinder.java:56)
        at hudson.init.InitializerFinder.discoverTasks(InitializerFinder.java:33)
        at hudson.init.TaskMethodFinder.discoverTasks(TaskMethodFinder.java:32)
        at org.jvnet.hudson.reactor.TaskBuilder$2.discoverTasks(TaskBuilder.java:61)
        at org.jvnet.hudson.reactor.Reactor.<init>(Reactor.java:151)
        at org.jvnet.hudson.reactor.Reactor.<init>(Reactor.java:156)
        at jenkins.model.Jenkins$5.<init>(Jenkins.java:1127)
        at jenkins.model.Jenkins.executeReactor(Jenkins.java:1127)
        at jenkins.model.Jenkins.<init>(Jenkins.java:966)
        at hudson.model.Hudson.<init>(Hudson.java:85)
        at hudson.model.Hudson.<init>(Hudson.java:81)
        at hudson.WebAppMain$3.run(WebAppMain.java:233)
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

hfeldker@web.de (JIRA)

unread,
Jan 24, 2020, 5:30:02 AM1/24/20
to jenkinsc...@googlegroups.com
H. Feldker commented on Bug JENKINS-60857
 
Re: jetty no longer accepts the keystore

Hello,

we encountered a similar issue (same exception), but our keystore is in JKS format and we do not have a wildcard certificate.

My guess is that this issue is connected with the jetty update 9.4.23:

releasenotes entry: 4325 Deprecate SniX509ExtendedKeyManager constructor without SslContextFactory$Server)

 

Best regards,

Heiko

christian.keck@macio.de (JIRA)

unread,
Jan 24, 2020, 5:35:03 AM1/24/20
to jenkinsc...@googlegroups.com

I agree with Heiko that it must be related with a change in the API of Jetty or at least a change in the acceptance of missing parameters in some methods.

However, the noted deprecation seems to lead to a different error message as found here for example: https://github.com/eclipse/jetty.project/issues/4425

christian.keck@macio.de (JIRA)

unread,
Jan 24, 2020, 5:54:02 AM1/24/20
to jenkinsc...@googlegroups.com
Christian Keck edited a comment on Bug JENKINS-60857
I agree with Heiko that it must be related with a change in the API of Jetty or at least a change in the acceptance of missing parameters in some methods.

However, the noted deprecation seems to lead to a different error message as found here for example: [https://github.com/eclipse/jetty.project/issues/4425]

EDIT: Nevermind - the message had been updated in 4.25...

christian.keck@macio.de (JIRA)

unread,
Jan 24, 2020, 6:01:02 AM1/24/20
to jenkinsc...@googlegroups.com
Christian Keck updated an issue
 
Change By: Christian Keck
Labels: SSL jenkins jetty keystore

francois.isabelle75@gmail.com (JIRA)

unread,
Jan 24, 2020, 7:40:03 AM1/24/20
to jenkinsc...@googlegroups.com
François Isabelle commented on Bug JENKINS-60857
 
Re: jetty no longer accepts the keystore

I got the same issue after upgrading to 2.217.

alexander.gaengel@1und1.de (JIRA)

unread,
Jan 24, 2020, 8:09:02 AM1/24/20
to jenkinsc...@googlegroups.com

Same for me

christian.keck@macio.de (JIRA)

unread,
Jan 24, 2020, 8:44:04 AM1/24/20
to jenkinsc...@googlegroups.com
Christian Keck updated an issue
Change By: Christian Keck
Priority: Critical Blocker

jglick@cloudbees.com (JIRA)

unread,
Jan 24, 2020, 10:01:05 AM1/24/20
to jenkinsc...@googlegroups.com
Jesse Glick commented on Bug JENKINS-60857
 
Re: jetty no longer accepts the keystore

I will check if I can reproduce the issue and look for a fix.

jglick@cloudbees.com (JIRA)

unread,
Jan 24, 2020, 10:01:06 AM1/24/20
to jenkinsc...@googlegroups.com
Jesse Glick assigned an issue to Jesse Glick
 
Change By: Jesse Glick
Assignee: Jesse Glick

jglick@cloudbees.com (JIRA)

unread,
Jan 24, 2020, 10:01:07 AM1/24/20
to jenkinsc...@googlegroups.com
Jesse Glick started work on Bug JENKINS-60857
 
Change By: Jesse Glick
Status: Open In Progress

shawnbat@sbcglobal.net (JIRA)

unread,
Jan 24, 2020, 10:14:02 AM1/24/20
to jenkinsc...@googlegroups.com
Shawn Bates commented on Bug JENKINS-60857
 
Re: jetty no longer accepts the keystore

Same issue for me too.  Is there a way to revert back to previous version?

christian.keck@macio.de (JIRA)

unread,
Jan 24, 2020, 10:17:03 AM1/24/20
to jenkinsc...@googlegroups.com

Shawn Bates: Yes, you can savely downgrade to a former version.

shawnbat@sbcglobal.net (JIRA)

unread,
Jan 24, 2020, 10:17:03 AM1/24/20
to jenkinsc...@googlegroups.com
Shawn Bates edited a comment on Bug JENKINS-60857
Same issue for me too.  Is there a way to revert back to previous version using Linux ?

shawnbat@sbcglobal.net (JIRA)

unread,
Jan 24, 2020, 10:19:07 AM1/24/20
to jenkinsc...@googlegroups.com

Would you have steps or commands for downgrading on Linux?

christian.keck@macio.de (JIRA)

unread,
Jan 24, 2020, 10:21:03 AM1/24/20
to jenkinsc...@googlegroups.com

Redhat/Centos: "yum downgrade jenkins"

Debian/Ubuntu: "apt-get install jenkins=<version>"

You may need to find the exakt <version> sting by invoking "apt-cache madison jenkins" before.

HTH

shawnbat@sbcglobal.net (JIRA)

unread,
Jan 24, 2020, 10:27:03 AM1/24/20
to jenkinsc...@googlegroups.com

That worked.  Thank you Sir!

jglick@cloudbees.com (JIRA)

unread,
Jan 24, 2020, 12:07:03 PM1/24/20
to jenkinsc...@googlegroups.com

I am not able to reproduce this error. First of all, HttpsConnectorFactoryTest passes in Winstone sources. Second, I followed these instructions (amended only in trivial ways) and was able to run a development build of Jenkins without any issues, including using curl -ik to access the index page after it started up. Is there some crucial factor I am missing? I can offer a pull request for the deprecation warning but it is senseless to ship a purported fix without being able to verify the effect.

jglick@cloudbees.com (JIRA)

unread,
Jan 24, 2020, 12:26:03 PM1/24/20
to jenkinsc...@googlegroups.com

Seems to only affect keystores using SNI: wildcards, multiple hosts, multiple aliases. I am looking for a sample to test against.

jglick@cloudbees.com (JIRA)

unread,
Jan 24, 2020, 12:57:03 PM1/24/20
to jenkinsc...@googlegroups.com

Tried to pick up a test keystore from Jetty tests, but so far I have been unable to get Winstone/Jenkins to open it. Not a topic I am at all familiar with.

francois.isabelle75@gmail.com (JIRA)

unread,
Jan 24, 2020, 2:10:02 PM1/24/20
to jenkinsc...@googlegroups.com

Instructions to generate the keystore (as Ansible snippets)

With some variables defined


jenkins_ssl_cert: "\{{ jenkins_ssl_dest }}/your_domain_example_org.crt"
jenkins_ssl_key: "\{{ jenkins_ssl_dest }}/your_domain_example_org.key"
jenkins_ssl_jks: "\{{ jenkins_ssl_dest }}/jenkins.jks"
jenkins_ssl_chain: "\{{ jenkins_ssl_dest }}/DigiCertCA.crt"

 \{code}

The keystores are created in 2 steps.
  • name: Create PKCS
    command: "openssl pkcs12 -inkey {{ jenkins_ssl_key }} -in {{ jenkins_ssl_cert }} -export -out {{ jenkins_ssl_dest }}/keys.pkcs12 -passout pass:$SECRET”
    become: true
    become_user: jenkins
    register: create_pkcs_result
  • name: Create JKS
    command: " keytool -importkeystore -srckeystore {{ jenkins_ssl_dest }}/keys.pkcs12 -srcstoretype pkcs12 -destkeystore {{ jenkins_ssl_jks }} -srcstorepass jenkins -deststorepass $SECRET -noprompt"
    become: true
    become_user: jenkins
    register: create_jks_result
 
                                                            

                                                        
 
Maybe that can be useful.
 		 
                                                
 
                                            
                                        
 		 
                                
francois.isabelle75@gmail.com (JIRA)
unread,
Jan 24, 2020, 2:10:03 PM1/24/20
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
        
 
             
            
                
 
                     
                        
 
                            
                                
 
                                       
                                     François Isabelle edited a comment on Bug JENKINS-60857  
                                
 
                            
                        
 		 
                
 
                
 
                     
                        
 
                            
 Instructions to generate the keystore (as Ansible snippets)

With some variables defined
 
 
{code :java }
 
 jenkins_ssl_cert: "\{{ jenkins_ssl_dest }}/your_domain_example_org.crt"
jenkins_ssl_key: "\{{ jenkins_ssl_dest }}/your_domain_example_org.key"
jenkins_ssl_jks: "\{{ jenkins_ssl_dest }}/jenkins.jks"
jenkins_ssl_chain: "\{{ jenkins_ssl_dest }}/DigiCertCA.crt"

  \ 
{code}


The keystores are created in 2 steps.

{code}
 
 - name: Create PKCS

command: "openssl pkcs12 -inkey \{{ jenkins_ssl_key }} -in \{{ jenkins_ssl_cert }} -export -out \{{ jenkins_ssl_dest }}/keys.pkcs12 -passout pass:$SECRET”
become: true
become_user: jenkins
register: create_pkcs_result

- name: Create JKS

command: " keytool -importkeystore -srckeystore \{{ jenkins_ssl_dest }}/keys.pkcs12 -srcstoretype pkcs12 -destkeystore \{{ jenkins_ssl_jks }} -srcstorepass jenkins -deststorepass $SECRET -noprompt"
become: true
become_user: jenkins
register: create_jks_result

{code :java }
  {code} 
Maybe that can be useful.
francois.isabelle75@gmail.com (JIRA)
unread,
Jan 24, 2020, 2:11:03 PM1/24/20
to jenkinsc...@googlegroups.com

                
 
                     
                        
 
                            
                                
 
                                       
                                     François Isabelle edited a comment on Bug JENKINS-60857  
                                
 
                            
                        
 		 
                
 
                
 Instructions to generate the keystore (as Ansible snippets)

With some variables defined
{code:java}
jenkins_ssl_cert: "\{{ jenkins_ssl_dest }}/your_domain_example_org.crt"
jenkins_ssl_key: "\{{ jenkins_ssl_dest }}/your_domain_example_org.key"
jenkins_ssl_jks: "\{{ jenkins_ssl_dest }}/jenkins.jks"
jenkins_ssl_chain: "\{{ jenkins_ssl_dest }}/DigiCertCA.crt"

{code}
 
 The keystores are created in 2 steps.
 
 
{code :java }

 - name: Create PKCS
 command: "openssl pkcs12 -inkey \{{ jenkins_ssl_key }} -in \{{ jenkins_ssl_cert }} -export -out \{{ jenkins_ssl_dest }}/keys.pkcs12 -passout pass:$SECRET”
 become: true
 become_user: jenkins
 register: create_pkcs_result

 - name: Create JKS
 command: " keytool -importkeystore -srckeystore \{{ jenkins_ssl_dest }}/keys.pkcs12 -srcstoretype pkcs12 -destkeystore \{{ jenkins_ssl_jks }} -srcstorepass jenkins -deststorepass $SECRET -noprompt"
 become: true
 become_user: jenkins
 register: create_jks_result
 
{code:java}
  {code}
Maybe that can be useful. 
francois.isabelle75@gmail.com (JIRA)
unread,
Jan 24, 2020, 2:13:04 PM1/24/20
to jenkinsc...@googlegroups.com

                
 
                     
                        
 
                            
                                
 
                                       
                                     François Isabelle edited a comment on Bug JENKINS-60857  
                                
 
                            
                        
 		 
                
 
                
 
                                     
                                        
 
                                            
                                                
 
                                                     Confirmed, I am using wildcard.

 
Instructions to generate the keystore (as Ansible snippets)

With some variables defined
{code:java}
jenkins_ssl_cert: "\{{ jenkins_ssl_dest }}/your_domain_example_org.crt"
jenkins_ssl_key: "\{{ jenkins_ssl_dest }}/your_domain_example_org.key"
jenkins_ssl_jks: "\{{ jenkins_ssl_dest }}/jenkins.jks"
jenkins_ssl_chain: "\{{ jenkins_ssl_dest }}/DigiCertCA.crt"

{code}
The keystores are created in 2 steps.
{code:java}
 - name: Create PKCS
 command: "openssl pkcs12 -inkey \{{ jenkins_ssl_key }} -in \{{ jenkins_ssl_cert }} -export -out \{{ jenkins_ssl_dest }}/keys.pkcs12 -passout pass:$SECRET”
 become: true
 become_user: jenkins
 register: create_pkcs_result

 - name: Create JKS
 command: " keytool -importkeystore -srckeystore \{{ jenkins_ssl_dest }}/keys.pkcs12 -srcstoretype pkcs12 -destkeystore \{{ jenkins_ssl_jks }} -srcstorepass jenkins -deststorepass $SECRET -noprompt"
 become: true
 become_user: jenkins
 register: create_jks_result
 {code}
Maybe that can be useful.
christian.keck@macio.de (JIRA)
unread,
Jan 24, 2020, 2:18:03 PM1/24/20
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
                                     
                                        
 
                                            
                                                
 
                                                     
Jesse Glick, I don't think it's related to SNI or multi-hosts only. Maybe it has something in common with the way the keystore was/is created.
 
I our case we created the keystore file like described here: https://coderwall.com/p/3t4xka/import-private-key-and-certificate-into-java-keystore
 
Maybe we are missing something here? AFAIK all environment-variables that Jenkins need to operate with a keystore are
 
JENKINS_HTTPS_KEYSTORE and JENKINS_HTTPS_KEYSTORE_PASSWORD - is that still valid?
 		 
                                                
 
                                            
                                        
 		 
                                
jglick@cloudbees.com (JIRA)
unread,
Jan 24, 2020, 2:34:02 PM1/24/20
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
                                     
                                        
 
                                            
                                                
 
                                                     
                                                        

                                                            
Instructions to generate the keystore (as Ansible snippets)

                                                        
 
Sorry, I do not do Ansible, and anyway if I understand those instructions correctly they presume that you have a certificate & key as input. For steps to reproduce, I need something I can run from scratch. A shell command to create a new keystore (self-signed is fine I suppose), then the jenkins.war arguments to run with it which worked in older releases and fails now.
 
                                                        

                                                            
I don't think it's related to SNI or multi-hosts only.

                                                        
 
I am just going by what I read here. I do not claim to know exactly what I am looking at.
 
                                                        

                                                            
all environment-variables that Jenkins need to operate with a keystore are JENKINS_HTTPS_KEYSTORE and JENKINS_HTTPS_KEYSTORE_PASSWORD - is that still valid?

                                                        
 
I am not aware of such environment variables (perhaps you are using some Docker image?). The Winstone arguments that I know of are named --httpsKeyStore and --httpsKeyStorePassword.
 		 
                                                
 
                                            
                                        
 		 
                                
jglick@cloudbees.com (JIRA)
unread,
Jan 24, 2020, 2:46:02 PM1/24/20
to jenkinsc...@googlegroups.com
 
                                     
                                        
 
                                            
                                                
 
                                                     
Ah, I think Christian Keck was referring to an RPM option.
 		 
                                                
 
                                            
                                        
 		 
                                
jglick@cloudbees.com (JIRA)
unread,
Jan 24, 2020, 3:53:03 PM1/24/20
to jenkinsc...@googlegroups.com
 
                                     
                                        
 
                                            
                                                
 
                                                     
I think I finally managed to reproduce the issue and will work on codifying this into an automated test.
 		 
                                                
 
                                            
                                        
 		 
                                
jglick@cloudbees.com (JIRA)
unread,
Jan 24, 2020, 4:14:04 PM1/24/20
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    

                                        
 		 
                                
 
                                
 
                                     
                                        
 
                                            
                                                
 
                                                    Change By: 
                                                     Jesse Glick  
                                                
 
                                                
 
                                                    Status: 
                                                     In  Progress  Review  
                                                
 
                                            
                                        
 		 
                                
Reply all
Reply to author
Forward
0 new messages