| Due to firewall (mis)configuration dc.services.visualstudio.com became partially accessible - the TCP connection to it succeeded but no data could be sent. As a result of this Jenkins instances using the azure-commons plugin were unable to start as on startup the plugin was trying to send data but it couldn't so it got blocked, and this completely blocked Jenkins itself as well (at the "Jenkins is starting" phase). It could be nice if it tried to access dc.services either asynchronously or timing out after a few seconds. Also it could possibly write something meaningful into the logs because finding the evidence in the Java thread dump is not the easiest. The thread dump obtained while Jenkins was blocked during startup:
"AppInsightsPluginLoadListener.onPluginsLoaded" #29 daemon prio=5 os_prio=0 tid=0x00007fea7cfa9000 nid=0x15636 runnable [0x00007fea29688000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
- locked <0x0000000086c82868> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
- locked <0x0000000086c82928> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:404)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:364)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at com.microsoft.jenkins.azurecommons.telemetry.JenkinsTelemetryClient.send(JenkinsTelemetryClient.java:53)
at com.microsoft.jenkins.azurecommons.telemetry.AppInsightsClient.sendEvent(AppInsightsClient.java:85)
at com.microsoft.jenkins.azurecommons.telemetry.AppInsightsPluginLoadListener.traceAzurePlugins(AppInsightsPluginLoadListener.java:44)
at com.microsoft.jenkins.azurecommons.telemetry.AppInsightsPluginLoadListener.onPluginsLoaded(AppInsightsPluginLoadListener.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1091)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thanks, Istvan |
