SSH slave connection issues after Jenkins update

[Pasi]

Hi,

We just updated Jenkins from 1.451 to 1.499 and now all our SSH slaves stay offline, our master runs on Ubuntu 10.10. Anybody got a clue why is this?

Here's a couple of stacktraces, might or might not be relevant:

1. java.io.IOException: Cannot run program "arp": java.io.IOException: error=2, No such file or directory
2.         at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
3.         at java.lang.Runtime.exec(Runtime.java:593)
4.         at java.lang.Runtime.exec(Runtime.java:431)
5.         at java.lang.Runtime.exec(Runtime.java:328)
6.         at controller.WakeUpServlet.getmac(WakeUpServlet.java:62)
7.         at controller.Controller.getLog(Controller.java:200)
8.         at controller.Controller.doRefresh(Controller.java:130)
9.         at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
10.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
11.         at java.lang.reflect.Method.invoke(Method.java:597)
12.         at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288)
13.         at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151)
14.         at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90)
15.         at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
16.         at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
17.         at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573)
18.         at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658)
19.         at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384)
20.         at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573)
21.         at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658)
22.         at org.kohsuke.stapler.Stapler.invoke(Stapler.java:487)
23.         at org.kohsuke.stapler.Stapler.service(Stapler.java:164)
24.         at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
25.         at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
26.         at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
27.         at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
28.         at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
29.         at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
30.         at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
31.         at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
32.         at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
33.         at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
34.         at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
35.         at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
36.         at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
37.         at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
38.         at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
39.         at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
40.         at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
41.         at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
42.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
43.         at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
44.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
45.         at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
46.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
47.         at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
48.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
49.         at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
50.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
51.         at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
52.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
53.         at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
54.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
55.         at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
56.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
57.         at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
58.         at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
59.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
60.         at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
61.         at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
62.         at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
63.         at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
64.         at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
65.         at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
66.         at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
67.         at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
68.         at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
69.         at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
70.         at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
71.         at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
72.         at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
73.         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
74.         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
75.         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
76.         at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
77.         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
78.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
79.         at java.lang.Thread.run(Thread.java:662)
80. Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
81.         at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
82.         at java.lang.ProcessImpl.start(ProcessImpl.java:65)
83.         at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
84.         ... 77 more
    
    
    The next one shows up after one slave in the new Smart Jenkins Controller view:
    1. Size: 0
    2. Stacktrace:
    3. javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index: 36
    4. Size: 0 at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:615) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658) at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:487) at org.kohsuke.stapler.Stapler.service(Stapler.java:164) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.IndexOutOfBoundsException: Index: 36
       
       
       Can anybody help me?
       
       
       Thanks,
       -Pasi
    
    

[Merrow, Frank]

Yeah, google it . . . tons of information on this:

 

In summary, 1.498 fixed a security bug, BUT in the process foobar’ed getting the slave code from the master (this might be a windows only issue . . . not sure).

 

Solutions are:

 

1.       Turn off Jenkins security so anonymous can get the slave jar

2.       Put the updated slave jar directly onto the slave hard disk and then change xml config file to use local file system instead of http’ing in the latest version.

 

The problem with solution #2 of course is that on your NEXT update . . . the jar won’t automatically update . . . so you’ll need to remember you did that.

 

This is the page that got me started down the solution:

 

https://issues.jenkins-ci.org/browse/JENKINS-16273

 

Your mileage may vary.

 

Frank

 

From: jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com] On Behalf Of Pasi
Sent: Friday, January 25, 2013 2:33 AM
To: jenkins...@googlegroups.com
Subject: SSH slave connection issues after Jenkins update

 

Hi,

 

We just updated Jenkins from 1.451 to 1.499 and now all our SSH slaves stay offline, our master runs on Ubuntu 10.10. Anybody got a clue why is this?

 

Here's a couple of stacktraces, might or might not be relevant:

1.  java.io.IOException: Cannot run program "arp": java.io.IOException: error=2, No such file or directory

2.          at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)

3.          at java.lang.Runtime.exec(Runtime.java:593)

4.          at java.lang.Runtime.exec(Runtime.java:431)

5.          at java.lang.Runtime.exec(Runtime.java:328)

6.          at controller.WakeUpServlet.getmac(WakeUpServlet.java:62)

7.          at controller.Controller.getLog(Controller.java:200)

8.          at controller.Controller.doRefresh(Controller.java:130)

9.          at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)

10.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

11.         at java.lang.reflect.Method.invoke(Method.java:597)

12.         at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288)

13.         at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151)

14.         at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90)

15.         at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)

16.         at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)

17.         at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573)

18.         at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658)

19.         at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384)

20.         at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573)

21.         at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658)

22.         at org.kohsuke.stapler.Stapler.invoke(Stapler.java:487)

23.         at org.kohsuke.stapler.Stapler.service(Stapler.java:164)

24.         at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)

25.         at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)

26.         at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)

27.         at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)

28.         at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)

29.         at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)

30.         at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)

31.         at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)

32.         at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)

33.         at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)

34.         at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)

35.         at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)

36.         at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)

37.         at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)

38.         at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

39.         at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)

40.         at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)

41.         at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

42.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)

43.         at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)

44.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

45.         at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)

46.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

47.         at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)

48.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

49.         at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)

50.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

51.         at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)

52.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

53.         at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)

54.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

55.         at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)

56.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

57.         at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)

58.         at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)

59.         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

60.         at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)

61.         at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)

62.         at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)

63.         at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

64.         at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)

65.         at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)

66.         at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

67.         at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)

68.         at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)

69.         at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

70.         at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)

71.         at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)

72.         at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)

73.         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

74.         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

75.         at java.util.concurrent.FutureTask.run(FutureTask.java:138)

76.         at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)

77.         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

78.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

79.         at java.lang.Thread.run(Thread.java:662)

80. Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory

81.         at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)

82.         at java.lang.ProcessImpl.start(ProcessImpl.java:65)

83.         at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)

84.         ... 77 more

 

The next one shows up after one slave in the new Smart Jenkins Controller view:

1.  Size: 0

2.  Stacktrace:

3.  javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index: 36

4.  Size: 0 at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:615) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658) at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:487) at org.kohsuke.stapler.Stapler.service(Stapler.java:164) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.IndexOutOfBoundsException: Index: 36

[Pasi]

I think this is not the same issue, I have no trouble connecting Windows and Linux slaves via JNLP. SSH slave connection was not the only issue with the new Jenkins version, it also didn't start any jobs even though there were slaves available.

I created a blocker ticket for my issue:

https://issues.jenkins-ci.org/browse/JENKINS-16482

-Pasi

[Pasi]

Nobody has any ideas about this? What logs to check, increase debug prints somehow?

Yesterday I once more did Jenkins update from 1.451 to 1.466.2 and SSH slaves are still not working. There is nothing related to SSH connections in any logs I checked, and monitoring IP packets with tcpdump doesn't show any SSH action between the master and the SSH slave.

I'm getting quite frustrated as SSH slave handling was working really well before we did the catastrophic update from 1.451 to 1.499, and it was much easier to do than with JNLP :(

-Pasi

[Pasi]

I finally got this fixed by removing ruby-runtime plugin. I think that got installed during the update and was left in the system even after the downgrade. I found out the issue by copying all the configs and plugins to a test Jenkins instance, and one by one disabling plugins until the SSH connections started even printing something in Jenkins logs.

After the real culprit was found, I asked for maintenance break in our production Jenkins and removed the plugin - SSH started working there too!

-Pasi