[JIRA] [active-directory-plugin] (JENKINS-29860) SSH Auth Breaks with "No Password" error from ActiveDirectoryUnixAuthenticationProvider

10 views
Skip to first unread message

dev@baltrinic.com (JIRA)

unread,
Aug 7, 2015, 2:28:01 PM8/7/15
to jenkinsc...@googlegroups.com
Kenneth Baltrinic created an issue
 
Jenkins / Bug JENKINS-29860
SSH Auth Breaks with "No Password" error from ActiveDirectoryUnixAuthenticationProvider
Issue Type: Bug Bug
Assignee: Unassigned
Components: active-directory-plugin, core
Created: 07/Aug/15 6:27 PM
Environment: Jenkins 1.609.2 with the following plugins:

active-directory 1.41
analysis-core 1.72
ansicolor 0.4.1
ant 1.2
antisamy-markup-formatter 1.1
audit-trail 2.1
build-token-root 1.1
build-user-vars-plugin 1.4
credentials 1.18
credentials-binding 1.5
cvs 2.11
email-ext 2.40.5
envinject 1.91.4
extensible-choice-parameter 1.3.1
external-monitor-job 1.4
gatling 1.1.1
git 2.4.0
git-client 1.15.0
groovy 1.27
javadoc 1.1
jobConfigHistory 2.12
jquery 1.11.2-0
junit 1.2-beta-4
ldap 1.11
log-parser 1.0.8
mailer 1.11
matrix-auth 1.1
matrix-project 1.4.1
maven-plugin 2.7.1
metrics 3.0.11
nodelabelparameter 1.5.1
ownership 0.7
pam-auth 1.1
plain-credentials 1.0
postbuildscript 0.17
role-strategy 2.2.0
scm-api 0.2
scm-sync-configuration 0.0.8
script-security 1.13
slack 1.8
ssh-credentials 1.10
ssh-slaves 1.9
subversion 1.54
support-core 2.27
token-macro 1.10
translation 1.10
view-job-filters 1.26
warnings 4.48
windows-slaves 1.0
workflow-step-api 1.5
Labels: authentication client ssh
Priority: Critical Critical
Reporter: Kenneth Baltrinic

Our Jenkins instance is chef managed, meaning a process runs on the box every 30 minutes that enforces certain configuration options in Jenkins. It does this by executing jenkins-cli commands. The Jenkins cli authenticates using SSH keys. The ssh key in question is associated with a user that is a full admin on the master. This user is also in our AD since we use the active-directory plugin for authentication. This has been working fine for a long time. (I think we stood this box up in January 2015).

As of 1500 EDT on Aug 3, we started seeing the below error in our Jenkins log, from the time the error occur until we restart Jenkins, all attempts to use the jenkins-cli as described fail, effectively disabling our chef management. Initially this was happening about once a day, it is now typically occurring within one to two hours after a the last corrective restart.

The only notable changes made in this timeframe were the addition of the extensible-choice-parameter and nodelabelparameter plugins which were added on the 4th and 6th of August respectively. 

Aug 07, 2015 4:56:25 PM hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider retrieveUser
WARNING: Credential exception trying to authenticate against **REDACTED** domain
org.acegisecurity.BadCredentialsException: Empty password
  at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:254)
  at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:219)
  at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:163)
  at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:122)
  at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:200)
  at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:47)
  at jenkins.security.BasicHeaderRealPasswordAuthenticator.authenticate(BasicHeaderRealPasswordAuthenticator.java:55)
  at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:79)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
  at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
  at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
  at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
  at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
  at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
  at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
  at org.eclipse.jetty.server.Server.handle(Server.java:370)
  at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
  at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
  at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
  at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
  at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
  at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
  at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
  at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
  at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

dev@baltrinic.com (JIRA)

unread,
Aug 7, 2015, 2:46:02 PM8/7/15
to jenkinsc...@googlegroups.com
Kenneth Baltrinic updated an issue
Change By: Kenneth Baltrinic
Our Jenkins instance is chef managed, meaning a process runs on the box every 30 minutes that enforces certain configuration options in Jenkins.  It does this by executing jenkins-cli commands.  The Jenkins cli authenticates using SSH keys.  The ssh key in question is associated with a user that is a full admin on the master.  This user is also in our AD since we use the active-directory plugin for authentication.  This has been working fine for a long time. (I think we stood this box up in January 2015).  

As of 1500 EDT on Aug 3, we started seeing the below error in our Jenkins  log, from  logs.  From  the time the error occur until we restart Jenkins, all attempts to use the jenkins-cli as described fail, effectively disabling our chef management.   

   Initially this was happening about once a day, it is now typically occurring within one to two hours after a the last corrective restart.   However this may simply be due to our more aggressive restarting as soon as we see the error.

The  initial error  only  appears once in the log, after that each subsequent authentication attempt results in a different error, also shown below.  The jenkins-cli, for its part, returns the cryptic error reported in JENKINS-22346, which we have come to understand as an authentication failure.

The only  notable changes made in this timeframe were the addition of the extensible-choice-parameter and nodelabelparameter plugins which were added on the 4th and 6th of August respectively. 

 h3. Initial Error that Initiates the Problem
{code:java}
{code}
h3. Error seen on Subsequent cli authentication attempts
{code:java}
Aug 07, 2015 5:01:36 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted connection #9 from /127.0.0.1:48680
Exception in thread "Thread-1234" org.acegisecurity.userdetails.UsernameNotFoundException: Authentication was successful but cannot locate the user information for 
  at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:295)

  at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:219)
  at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:163)
  at hudson.plugins.active_directory.AbstractActiveDirectoryAuthenticationProvider.loadUserByUsername(AbstractActiveDirectoryAuthenticationProvider.java:53)
  at jenkins.security.ImpersonatingUserDetailsService.loadUserByUsername(ImpersonatingUserDetailsService.java:32)
  at hudson.model.User.impersonate(User.java:309)
  at org.jenkinsci.main.modules.cli.auth.ssh.SshCliAuthenticator.authenticate(SshCliAuthenticator.java:44)
  at hudson.cli.CliManagerImpl$2.run(CliManagerImpl.java:109)
{code}

dev@baltrinic.com (JIRA)

unread,
Aug 10, 2015, 7:55:01 AM8/10/15
to jenkinsc...@googlegroups.com
Kenneth Baltrinic commented on Bug JENKINS-29860
 
Re: SSH Auth Breaks with "No Password" error from ActiveDirectoryUnixAuthenticationProvider

A little more information on this issue:

  • Apparently when this issue occurs it only affects the given user. Our tests show that other users having configured ssh keys are able to continue to use the CLI with ssh auth.
  • This is only affecting SSH Auth, the affected user is able to use the CLI if we switch to username/password auth.
  • We have tried various things short of starting the service in order to resolve this issue (removing and re-assigning the publish SSH key for the user in Jenkins, Disabling and re-enabling AD auth in Jenkins, etc.) None of that seems to work. A 'soft' restart (a restart initiated from the plugin-update page) does work.

dev@baltrinic.com (JIRA)

unread,
Aug 10, 2015, 7:56:01 AM8/10/15
to jenkinsc...@googlegroups.com
Kenneth Baltrinic edited a comment on Bug JENKINS-29860
A little more information on this issue:  
* Apparently when this issue occurs it only affects the given user.  Our tests show that other users having configured ssh keys are able to continue to use the CLI with ssh auth.
* This is only affecting SSH Auth, the affected user is able to use the CLI if we switch to username/password auth.
* We have tried various things short of  starting  restarting  the service in order to resolve this issue (removing and re-assigning the publish SSH key for the user in Jenkins, Disabling and re-enabling AD auth in Jenkins, etc.)  None of that seems to work.  A 'soft' restart (a restart initiated from the plugin-update page) does work.

dev@baltrinic.com (JIRA)

unread,
Aug 10, 2015, 7:57:01 AM8/10/15
to jenkinsc...@googlegroups.com
Kenneth Baltrinic edited a comment on Bug JENKINS-29860
A little more information on this issue:  
* Apparently when this issue occurs it only affects the given user.  Our tests show that other users having configured ssh keys are able to continue to use the CLI with ssh auth.
* This is only affecting SSH Auth, the affected user is able to use the CLI if we switch to username/password auth.
* We have tried various things short of restarting the service in order to resolve this issue (removing and re-assigning the  publish  public  SSH key for the user in Jenkins,  Disabling  disabling  and re-enabling AD auth in Jenkins, etc.)  None of that seems to work.  A 'soft' restart (a restart initiated from the plugin-update page) does work.

dev@baltrinic.com (JIRA)

unread,
Aug 12, 2015, 7:42:05 AM8/12/15
to jenkinsc...@googlegroups.com
Kenneth Baltrinic updated an issue
Change By: Kenneth Baltrinic
Our Jenkins instance is chef managed, meaning a process runs on the box every 30 minutes that enforces certain configuration options in Jenkins.  It does this by executing jenkins-cli commands.  The Jenkins cli authenticates using SSH keys.  The ssh key in question is associated with a user that is a full admin on the master.  This user is also in our AD since we use the active-directory plugin for authentication.  This has been working fine for a long time. (I think we stood this box up in January 2015).  

As of 1500 EDT on Aug 3, we started seeing the below error in our Jenkins logs.  From the time the error occur until we restart Jenkins, all attempts to use the jenkins-cli  as described  by this user  fail, effectively disabling our chef management.  

dev@baltrinic.com (JIRA)

unread,
Aug 12, 2015, 7:44:43 AM8/12/15
to jenkinsc...@googlegroups.com
Kenneth Baltrinic assigned an issue to Jesse Glick
Change By: Kenneth Baltrinic
Assignee: Jesse Glick

jglick@cloudbees.com (JIRA)

unread,
Aug 12, 2015, 4:11:01 PM8/12/15
to jenkinsc...@googlegroups.com
Jesse Glick assigned an issue to Unassigned
 

Not sure why this was assigned to me.
Change By: Jesse Glick
Component/s: core
Assignee: Jesse Glick

dev@baltrinic.com (JIRA)

unread,
Aug 13, 2015, 9:39:02 AM8/13/15
to jenkinsc...@googlegroups.com
Kenneth Baltrinic commented on Bug JENKINS-29860
 
Re: SSH Auth Breaks with "No Password" error from ActiveDirectoryUnixAuthenticationProvider

Jesse Glick Perhaps assigning to you is not the best approach but it is unassigned and I don't know who it should best be assigned to. However this is becoming a significant operation issue for us and not being assigned to anyone, no one seems to be looking at it. I was hoping you would at least be able to assign it to the right person.

jglick@cloudbees.com (JIRA)

unread,
Aug 13, 2015, 3:39:02 PM8/13/15
to jenkinsc...@googlegroups.com

You can consider contracting with a commercial support provider, such as my employer.

fbelzunc@gmail.com (JIRA)

unread,
Jan 20, 2016, 4:01:03 AM1/20/16
to jenkinsc...@googlegroups.com

I believe this issue might happen because somehow there is a defined user internally with a/several space/s as name. Something like:

http://<JENKINS_URL>/user/%20/configure

This user might have the same SSH Key than the Chef user you are using. I am able to re-produce the issue you are facing when this happen.

You can check if there are several users with the same SSH Key with the Groovy script below executing in http://<JENKINS_URL>/script


import hudson.model.User
import hudson.model.UserProperty
import org.jenkinsci.main.modules.cli.auth.ssh.*

User rootUser = User.get("<CHEF_USER>");
UserPropertyImpl userRootProperty = rootUser.getProperty(UserPropertyImpl.class);
  
allUsers = User.getAll();
            allUsers.each { user ->
                UserPropertyImpl userProperty = user.getProperty(UserPropertyImpl.class);
              	if (userProperty!=null && userProperty.authorizedKeys==userRootProperty.authorizedKeys) {
                	println "User $rootUser has the same SSH key than $user"
              	}
            }

fbelzunc@gmail.com (JIRA)

unread,
Jan 21, 2016, 2:50:01 AM1/21/16
to jenkinsc...@googlegroups.com

Kenneth Baltrinic confirmed me privately that the issue here was a phantom user. The script above can help other users to find the phantom user. I created https://issues.jenkins-ci.org/browse/JENKINS-32534 and did the pull request below to fix the issue on the corresponded jenkins module.

https://github.com/jenkinsci/ssh-cli-auth-module/pull/3

fbelzunc@gmail.com (JIRA)

unread,
Jan 21, 2016, 2:51:03 AM1/21/16
to jenkinsc...@googlegroups.com
Change By: Félix Belzunce Arcos
Assignee: Félix  Belzunce Arcos

fbelzunc@gmail.com (JIRA)

unread,
Jan 21, 2016, 2:51:04 AM1/21/16
to jenkinsc...@googlegroups.com
Félix Belzunce Arcos closed an issue as Fixed
Status: Open Closed
Resolution: Fixed
Reply all
Reply to author
Forward
0 new messages