Not Provisioning 'xxxxxx'; Server 'Docker Build Cloud' full with '10' container(s)

[Gang cheng]

In short, I think the container pool is full which cause this, but I don't know how to resolve this. (I tried to restart Jenkins server, it doesn't help...)

Any advices are appreciated! 

Please see details below.

============================================

I have a jenkins server setup to run build jobs on docker. Everything works fine until today I see a new job is pending for next available executor.

When I looked into the Jenkins log, I see below logs:

INFO: Will provision 'xxxxxx', for label: 'xxxxxx', in cloud: 'Docker Build Cloud'

Mar 29, 2016 4:29:59 PM com.nirima.jenkins.plugins.docker.DockerCloud addProvisionedSlave

INFO: Not Provisioning 'xxxxxx'; Server 'Docker Build Cloud' full with '10' container(s)

Mar 29, 2016 4:30:09 PM com.nirima.jenkins.plugins.docker.DockerCloud provision

INFO: Asked to provision 1 slave(s) for: xxxxxx

......

Also in Manage Jenkins->Dockers->Docker Build Cloud, when I tried to stop one of the docker container, I got an error like below:

javax.servlet.ServletException: javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:134)
	at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:131)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:125)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:370)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out
	at com.github.dockerjava.jaxrs.connector.ApacheConnector.apply(ApacheConnector.java:490)
	at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:246)
	at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:667)
	at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:664)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:424)
	at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:664)
	at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:424)
	at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:333)
	at com.github.dockerjava.jaxrs.StopContainerCmdExec.execute(StopContainerCmdExec.java:26)
	at com.github.dockerjava.jaxrs.StopContainerCmdExec.execute(StopContainerCmdExec.java:11)
	at com.github.dockerjava.jaxrs.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:24)
	at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:33)
	at com.github.dockerjava.core.command.StopContainerCmdImpl.exec(StopContainerCmdImpl.java:67)
	at com.nirima.jenkins.plugins.docker.DockerManagementServer.doControlSubmit(DockerManagementServer.java:65)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	... 53 more
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at shaded.org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
	at shaded.org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152)
	at shaded.org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270)
	at shaded.org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at shaded.org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at shaded.org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at shaded.org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)
	at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at shaded.org.apache.http.impl.conn.CPoolProxy.invoke(CPoolProxy.java:138)
	at com.sun.proxy.$Proxy55.receiveResponseHeader(Unknown Source)
	at shaded.org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at shaded.org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:253)
	at shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
	at shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
	at shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
	at shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
	at shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
	at com.github.dockerjava.jaxrs.connector.ApacheConnector.apply(ApacheConnector.java:443)
	... 79 more

[Gang cheng]

reboot the server (that host Jenkins), also docker rm exited containers, resolve the issue.