With
- Jenkins: 2.7.1
- subversion-plugin: 2.6.0
- Master: Windows Server
- Slave: Windows 7
the problem is still there, but the workaround mentioned in https://issues.jenkins-ci.org/browse/JENKINS-21785 (additional credentials with auth realm) seems to be working. From our findings, there seem to be two problems: 1. The subversion-plugin seems to be forgetting to send the authentication credentials at some svn:external call. 2. After the failed call due to unauthorized access (because of missing credentials) there seems to be a second call to the same svn:external with credentials, which then will be answered correctly, but the exception is still thrown with the authentication error. These assumptions are based on the following SVN server access logs:
<IP> - <username> [12/Jul/2016:17:27:14 +0200] "REPORT /repos/<project a>/!svn/vcc/default HTTP/1.1" 200 31394267 "-" "SVN/1.8.1 SVNKit/1.8.11 (http://svnkit.com/) r10483_v20150925_0010"
<IP> - - [12/Jul/2016:17:27:17 +0200] "OPTIONS /repos/<project a>/libs/<folder a>/tags/<tag a>/out HTTP/1.1" 401 381 "-" "SVN/1.8.1 SVNKit/1.8.11 (http://svnkit.com/) r10483_v20150925_0010"
<IP> - <username> [12/Jul/2016:17:27:17 +0200] "OPTIONS /repos/<project a>/libs/<folder a>/tags/<tag a>/out HTTP/1.1" 200 196 "-" "SVN/1.8.1 SVNKit/1.8.11 (http://svnkit.com/) r10483_v20150925_0010"
<IP> - <username> [12/Jul/2016:17:27:18 +0200] "PROPFIND /repos/<project a>/libs/<folder a>/tags/<tag a>/out HTTP/1.1" 207 766 "-" "SVN/1.8.1 SVNKit/1.8.11 (http://svnkit.com/) r10483_v20150925_0010"
I hope this helps finding the underlying issue. We also found out, the load on the server seems to be relevant. We measured the rate of failure by having four jobs checking out a fresh copy of the same repository (with externals) every ten minutes. Normally this checkout is about 3.5 minutes. The checkouts tend to get longer and longer up to 5 minutes. Then there is a job failure. The next 3-4 checkouts are back to 3.5minutes. Then there is a job failure. And then its back to normal for an unknown time longer than 2-3hrs. On weekends (nobody working) there are no failures at all. And both of the job failures are at about the same time in all four jobs. So they fail in the same five minutes range, one after the other. Stack trace:
14:55:55 hudson.util.IOException2: revision check failed on https://<domain>/repos/<project b>/<folder b>/trunk/scripts
14:55:55 at hudson.scm.SubversionChangeLogBuilder.buildModule(SubversionChangeLogBuilder.java:208)
14:55:55 at hudson.scm.SubversionChangeLogBuilder.run(SubversionChangeLogBuilder.java:138)
14:55:55 at hudson.scm.SubversionSCM.calcChangeLog(SubversionSCM.java:725)
14:55:55 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:860)
14:55:55 at hudson.scm.SCM.checkout(SCM.java:485)
14:55:55 at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
14:55:55 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
14:55:55 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
14:55:55 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
14:55:55 at hudson.model.Run.execute(Run.java:1741)
14:55:55 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
14:55:55 at hudson.model.ResourceController.execute(ResourceController.java:98)
14:55:55 at hudson.model.Executor.run(Executor.java:410)
14:55:55 Caused by: org.tmatesoft.svn.core.SVNCancelException: svn: E200015: E200015: ISVNAuthentication provider did not provide credentials; HTTP authorization cancelled.
14:55:55 svn: E200015: ISVNAuthentication provider did not provide credentials; HTTP authorization cancelled.
14:55:55 at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:60)
14:55:55 at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
14:55:55 at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:793)
14:55:55 at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:398)
14:55:55 at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:386)
14:55:55 at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:863)
14:55:55 at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:699)
14:55:55 at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:118)
14:55:55 at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1049)
14:55:55 at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getLatestRevision(DAVRepository.java:189)
14:55:55 at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.getRevisionNumber(SvnNgRepositoryAccess.java:118)
14:55:55 at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.getLocations(SvnRepositoryAccess.java:184)
14:55:55 at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepositoryFor(SvnNgRepositoryAccess.java:45)
14:55:55 at org.tmatesoft.svn.core.internal.wc2.remote.SvnRemoteLog.run(SvnRemoteLog.java:160)
14:55:55 at org.tmatesoft.svn.core.internal.wc2.remote.SvnRemoteLog.run(SvnRemoteLog.java:35)
14:55:55 at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
14:55:55 at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1235)
14:55:55 at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
14:55:55 at org.tmatesoft.svn.core.wc.SVNLogClient.doLog(SVNLogClient.java:968)
14:55:55 at org.tmatesoft.svn.core.wc.SVNLogClient.doLog(SVNLogClient.java:873)
14:55:55 at hudson.scm.SubversionChangeLogBuilder.buildModule(SubversionChangeLogBuilder.java:194)
14:55:55 ... 12 more
14:55:55 Caused by: org.tmatesoft.svn.core.SVNCancelException: svn: E200015: ISVNAuthentication provider did not provide credentials; HTTP authorization cancelled.
14:55:55 at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:723)
14:55:55 ... 30 more
14:55:55 Started calculate disk usage of build
Which fits to: https://github.com/jenkinsci/subversion-plugin/blob/2.6.0/src/main/java/hudson/scm/SubversionSCM.java#L860 and https://github.com/jenkinsci/subversion-plugin/blob/2.6.0/src/main/java/hudson/scm/SubversionChangeLogBuilder.java#L208 |