[JIRA] (JENKINS-60280) Jobs failing sporadically with javax.crypto.BadPaddingException: Invalid TLS padding data

7 views
Skip to first unread message

stephan.vogt@elektrobit.com (JIRA)

unread,
Nov 26, 2019, 2:13:03 AM11/26/19
to jenkinsc...@googlegroups.com
Stephan Vogt created an issue
 
Jenkins / Bug JENKINS-60280
Jobs failing sporadically with javax.crypto.BadPaddingException: Invalid TLS padding data
Issue Type: Bug Bug
Assignee: Unassigned
Components: core, swarm-plugin
Created: 2019-11-26 07:12
Environment: Jenkins Master (v 2.164.1) running directly (no container) on a Windows 7 (64bit) host. Several Jenkins clients running in Windows 10 (64bit) VMs on different machines. Clients connect via swarm client (v3.17). Both master and clients are started as batch files, and are running in a console window (no Windows service).
Issue seen both with Oracle Java 8u141 and OpenJDK 8u232.
Priority: Major Major
Reporter: Stephan Vogt

We are sporadically seeing jobs failing with this callstack shown in the log:

javax.crypto.BadPaddingException: Invalid TLS padding data
	at sun.security.ssl.CipherBox.removePadding(CipherBox.java:802)
	at sun.security.ssl.CipherBox.decrypt(CipherBox.java:574)
	at sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:200)
	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:963)
Caused: javax.net.ssl.SSLException: Invalid TLS padding data
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709)
	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:970)
	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896)
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766)
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:347)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668)
	at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:160)
	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from <server name>/<server ip>:54103 failed. The channel is closing down or has closed down
	at hudson.remoting.Channel.call(Channel.java:950)
	at hudson.FilePath.act(FilePath.java:1068)
	at hudson.FilePath.act(FilePath.java:1057)
	at hudson.FilePath.mkdirs(FilePath.java:1242)
	at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:181)
	at org.jenkinsci.plugins.durabletask.WindowsBatchScript$BatchController.<init>(WindowsBatchScript.java:93)
	at org.jenkinsci.plugins.durabletask.WindowsBatchScript$BatchController.<init>(WindowsBatchScript.java:91)
	at org.jenkinsci.plugins.durabletask.WindowsBatchScript.doLaunch(WindowsBatchScript.java:60)
	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launchWithCookie(FileMonitoringTask.java:106)
	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:99)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:305)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:268)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:176)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
	at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1278)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1172)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:810)
	at groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:46)
	at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1278)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1172)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
	at buildscript.call(C:\Jenkins_master\workspace\jobs\NightlyBuild_SDK_QNX\builds\365\libs\pipeline-common\vars\buildscript.groovy:62)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor291.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45)
	at sun.reflect.GeneratedMethodAccessor374.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor291.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor291.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

There are days when this is seen several times a day, and others when it is not seen at all. It happens on different Windows10 clients, but so far I have not seen it on any of our Linux clients. When restarting the failing job (with the same parameters) it usually runs fine.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

me@basilcrow.com (JIRA)

unread,
Feb 16, 2020, 9:24:02 PM2/16/20
to jenkinsc...@googlegroups.com
Basil Crow commented on Bug JENKINS-60280
 
Re: Jobs failing sporadically with javax.crypto.BadPaddingException: Invalid TLS padding data

Hey Stephan Vogt, I'm not sure what the root cause of this problem is. May I suggest a few things:

  • The stack trace you posted above is from the Jenkins master (server) side. It would be interesting to see if there are any errors on the Swarm (client) side. To do this, try starting the Swarm client with more verbose logging. This page provides an example of a verbose logging.properties file that logs as much as possible. Perhaps the additional logs will shed more light into what the Swarm client was doing at the time of the failure.
  • Are you running the Swarm clients with the same Java version as the master? The Java ecosystem these days is growing quite diverse, and even minor versions (e.g. 8u151 vs 8u202) can have subtle differences leading to serialization incompatibilities. Another potential source of incompatibility is using e.g. the Oracle JDK vs OpenJDK/AdoptOpenJDK or other open source Java runtimes.

stephan.vogt@elektrobit.com (JIRA)

unread,
Feb 17, 2020, 8:00:02 AM2/17/20
to jenkinsc...@googlegroups.com

Hi, thanks for the suggestions. Especially the one about activating additional logging might come in handy.
As for the Java versions: Both the master and the client were using the exact same Java version - every time I tried a new one I installed it on all machines at the same time.

In the meantime however I have upgraded our environment to Windows 10, OpenJDK 11.0.6_10, and Jenkins master 2.204.1. So far I have not seen this issue again in several weeks of usage. I am not sure what part exactly fixed this issue for me, but with the new configuration it is no longer seen, so I think we can close this ticket.

me@basilcrow.com (JIRA)

unread,
Feb 17, 2020, 10:40:03 AM2/17/20
to jenkinsc...@googlegroups.com
Basil Crow closed an issue as Cannot Reproduce
 

I am not sure what part exactly fixed this issue for me, but with the new configuration it is no longer seen, so I think we can close this ticket.

Done.
Change By: Basil Crow
Status: Open Closed
Resolution: Cannot Reproduce
Reply all
Reply to author
Forward
0 new messages