Kubernets Plugin Slaves Are Terminating Mysteriously

120 views
Skip to first unread message

ape...@shoprunner.com

unread,
Jul 11, 2018, 7:40:42 PM7/11/18
to Jenkins Users
We are experiencing an issue with the kubernetes plugin where certain builds are failing because the slave has died/terminated unexpectedly. This is happening to us on a variety of builds and its occurring seemingly at random to different builds we have so it's hard to pin down a root cause.  The errors I've pasted below haven't really given us any clear direction or shown a pattern.  I was hoping someone has run into this issue before and fixed it and might have some advice.

Thanks in advance
Asaf

In the build we see the following error...

java.nio.channels.ClosedChannelException
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179)
at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:789)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from 100.96.4.163/100.96.4.163:45342 failed. The channel is closing down or has closed down
at hudson.remoting.Channel.call(Channel.java:948)
at hudson.FilePath.act(FilePath.java:1036)
at hudson.FilePath.act(FilePath.java:1025)
at hudson.FilePath.mkdirs(FilePath.java:1213)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:115)
at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:198)
at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:190)
at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:111)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:66)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:176)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
at sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:133)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
Caused: java.io.IOException: remote file operation failed: /home/jenkins/workspace/... at hudson.remoting.Channel@2ea8c3b3:JNLP4-connect connection from 100.96.4.163/100.96.4.163:45342
at hudson.FilePath.act(FilePath.java:1043)
at hudson.FilePath.act(FilePath.java:1025)
at hudson.FilePath.mkdirs(FilePath.java:1213)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:115)
at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:198)
at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:190)
at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:111)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:66)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:176)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
at sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:133)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
at WorkflowScript.run(WorkflowScript:15)
at ___cps.transform___(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
at sun.reflect.GeneratedMethodAccessor207.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Looking at the logs of the slave I've found the following two errors so far...
jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
WARNING: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/100.96.29.249:50000 remote=100.96.4.163/100.96.4.163:45342]] / Computer.threadPoolForRemoting [#2489] for jenkins-slave-wvlhz-w475h terminated\njava.nio.channels.ClosedChannelException
        at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
        at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179)
        at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:789)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)\n"


"Jul 11, 2018 8:08:56 PM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
WARNING: Computer.threadPoolForRemoting [#3281] for jenkins-slave-b1lxq-btww9 terminated
java.nio.channels.ClosedChannelException
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209)
at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:314)
at hudson.remoting.Channel.close(Channel.java:1450)
at hudson.remoting.Channel.close(Channel.java:1403)
at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:746)
at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:99)
at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:664)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Carlos Sanchez

unread,
Jul 12, 2018, 2:52:20 AM7/12/18
to Jenkins Users
does the pod die or it just disconnects ? when you describe the pod does it show OOM killed flag?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ebc46d67-41db-4edb-b062-42da895e6957%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ape...@shoprunner.com

unread,
Jul 15, 2018, 4:29:54 PM7/15/18
to Jenkins Users
From what we see in a timeline, the slave connects successfully (see this in the slave logs) & then the next message we see associated with the slave is from the master saying that its terminating it (log from master).  A short time later (1 or 2 minutes) we see in the slave logs that it's is terminating.

We have not see any memory issues thus far with these slaves.  Here are all the events I see when I grep all events for a slave that died during a build.

>kubectl get events --all-namespaces=true --sort-by=.metadata.creationTimestamp -o wide --show-all=true | grep jenkins-slave-l168c-c844q
jenkins       56m         56m          1         jenkins-slave-l168c-c844q.1540acac602a022b              Pod                                                     Normal    SuccessfulMountVolume   kubelet, ip-.ec2.internal    MountVolume.SetUp succeeded for volume "workspace-volume"
jenkins       56m         56m          1         jenkins-slave-l168c-c844q.1540acac5292e6da              Pod                                                     Normal    Scheduled               default-scheduler                         Successfully assigned jenkins-slave-l168c-c844q to ip-172-20-86-205.ec2.internal
jenkins       56m         56m          1         jenkins-slave-l168c-c844q.1540acac715794bc              Pod       spec.containers{awscli-terraform}             Normal    Pulled                  kubelet, ip-1.ec2.internal    Container image "dtr.s-9.us/awscli1.14.32-terraform0.11.3" already present on machine
jenkins       56m         56m          1         jenkins-slave-l168c-c844q.1540acac73906d60              Pod       spec.containers{awscli-terraform}             Normal    Created                 kubelet, ip-5.ec2.internal    Created container
jenkins       56m         56m          1         jenkins-slave-l168c-c844q.1540acac6032e3d7              Pod                                                     Normal    SuccessfulMountVolume   kubelet, ip-172-5.ec2.internal    MountVolume.SetUp succeeded for volume "volume-0"
jenkins       56m         56m          1         jenkins-slave-l168c-c844q.1540acac60774d48              Pod                                                     Normal    SuccessfulMountVolume   kubelet, ip-172-5.ec2.internal    MountVolume.SetUp succeeded for volume "default-token-8rhb5"
jenkins       56m         56m          1         jenkins-slave-l168c-c844q.1540acac782b66c1              Pod       spec.containers{docker}                       Normal    Pulled                  kubelet, ip-1725.ec2.internal    Container image "docker:17.09.0" already present on machine
jenkins       56m         56m          1         jenkins-slave-l168c-c844q.1540acac78180463              Pod       spec.containers{awscli-terraform}             Normal    Started                 kubelet, ip-172-5.ec2.internal    Started container
jenkins       56m         56m          1         jenkins-slave-l168c-c844q.1540acac7e6e7aaf              Pod       spec.containers{docker}                       Normal    Started                 kubelet, ip-172--.ec2.internal    Started container
jenkins       56m         56m          1         jenkins-slave-l168c-c844q.1540acac7eb02361              Pod       spec.containers{jnlp}                         Normal    Pulled                  kubelet, ip-172-5.ec2.internal    Container image "jenkins/jnlp-slave:alpine" already present on machine
jenkins       56m         56m          1         jenkins-slave-l168c-c844q.1540acac816804cd              Pod       spec.containers{jnlp}                         Normal    Created                 kubelet, ip-1725.ec2.internal    Created container
jenkins       56m         56m          1         jenkins-slave-l168c-c844q.1540acac8526b787              Pod       spec.containers{jnlp}                         Normal    Started                 kubelet, ip-1725.ec2.internal    Started container
jenkins       56m         56m          1         jenkins-slave-l168c-c844q.1540acac7aca6582              Pod       spec.containers{docker}                       Normal    Created                 kubelet, ip-1205.ec2.internal    Created container
jenkins       51m         51m          1         jenkins-slave-l168c-c844q.1540acf6d1e5b777              Pod       spec.containers{jnlp}                         Normal    Killing                 kubelet, ip-172-2.ec2.internal    Killing container with id docker://jnlp:Need to kill Pod
jenkins       51m         51m          1         jenkins-slave-l168c-c844q.1540acfdbadbcb4e              Pod       spec.containers{awscli-terraform}             Normal    Killing                 kubelet, ip-1205.ec2.internal    Killing container with id docker://awscli-terraform:Need to kill Pod
jenkins       51m         51m          1         jenkins-slave-l168c-c844q.1540acfdbba4cbeb              Pod       spec.containers{docker}                       Normal    Killing                 kubelet, ip-17205.ec2.internal    Killing container with id docker://docker:Need to kill Pod

YAN-HONG WANG

unread,
Feb 22, 2019, 8:06:42 AM2/22/19
to Jenkins Users
Hello @Carlos Sanchez,

I have similar warning message in Jenkins Master log. These warning message doesn't influence result of Jenkins Job execution.
And in my case, I didn't see 'OOM killed' flag in period of Pod.

Jenkins version: 2.150.3 alpine
kubernetes plugin version: 1.14.5

But, is that possible to avoid these warning message?

/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
WARNING: Connection #10 failed
java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:197)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:244)

/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
WARNING: Computer.threadPoolForRemoting [#286] for jenkins-slave-z23g0-9z5q0 terminated

java.nio.channels.ClosedChannelException
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)

    at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:784)

    at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:314)
    at hudson.remoting.Channel.close(Channel.java:1450)
    at hudson.remoting.Channel.close(Channel.java:1403)
    at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:821)
    at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:105)
    at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:737)

    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

Thanks very much.

BR
Hong
Reply all
Reply to author
Forward
0 new messages