[JIRA] (JENKINS-53924) Jenkins 2.136+ is not restartable with GLIBC 2.5

15 views
Skip to first unread message

raspy27@gmail.com (JIRA)

unread,
Oct 5, 2018, 11:21:03 AM10/5/18
to jenkinsc...@googlegroups.com
Krzysztof Malinowski created an issue
 
Jenkins / Bug JENKINS-53924
Jenkins 2.136+ is not restartable with GLIBC 2.5
Issue Type: Bug Bug
Assignee: Unassigned
Components: core
Created: 2018-10-05 15:20
Environment: Red Hat Enterprise Linux Server release 5.11 (Tikanga)
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
Priority: Minor Minor
Reporter: Krzysztof Malinowski

When Jenkins 2.136 (or newer) starts up, the log lists problem with loading libc library for com.sun.akuma.CLibrary:

Oct 05, 2018 4:08:34 PM jenkins.diagnosis.HsErrPidList <init>
WARNING: Failed to list up hs_err_pid files
java.lang.UnsatisfiedLinkError: /tmp/jna-578977101/jna607378441395487448.tmp: /lib64/libc.so.6: version `GLIBC_2.7' not found (required by /tmp/jna-578977101/jna607378441395487448.tmp)
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
        at java.lang.Runtime.load0(Runtime.java:809)
        at java.lang.System.load(System.java:1086)
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947)
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922)
        at com.sun.jna.Native.<clinit>(Native.java:190)
        at com.sun.akuma.CLibrary.<clinit>(CLibrary.java:89)
        at com.sun.akuma.JavaVMArguments.resolvePID(JavaVMArguments.java:128)
        at com.sun.akuma.JavaVMArguments.ofLinux(JavaVMArguments.java:116)
        at com.sun.akuma.JavaVMArguments.of(JavaVMArguments.java:104)
        at com.sun.akuma.JavaVMArguments.current(JavaVMArguments.java:92)
...

When visiting /safeRestart URL, there is no option to restart and instead there is a message saying: "Jenkins cannot restart itself as currently configured."

Attempt to restart from command line fails with:

$ java -jar jenkins-cli.jar -s <jenkins_url> -ssh -user <user> safe-restart

ERROR: Unexpected exception occurred while performing safe-restart command.
java.lang.NoClassDefFoundError: Could not initialize class com.sun.akuma.CLibrary
        at com.sun.akuma.JavaVMArguments.resolvePID(JavaVMArguments.java:128)
        at com.sun.akuma.JavaVMArguments.ofLinux(JavaVMArguments.java:116)
        at com.sun.akuma.JavaVMArguments.of(JavaVMArguments.java:104)
        at com.sun.akuma.JavaVMArguments.current(JavaVMArguments.java:92)
        at hudson.lifecycle.UnixLifecycle.<init>(UnixLifecycle.java:55)
        at hudson.lifecycle.Lifecycle.get(Lifecycle.java:94)
        at jenkins.model.Jenkins.getLifecycle(Jenkins.java:2538)
        at org.jenkins.ci.plugins.saferestart.JenkinsUtils.canRestart(JenkinsUtils.java:37)
        at org.jenkins.ci.plugins.saferestart.SafeRestartRootAction.getIconFileName(SafeRestartRootAction.java:47)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
        at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
        at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
        at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
        at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
        at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
        at org.apache.commons.jexl.parser.ASTExpression.value(ASTExpression.java:54)
        at org.apache.commons.jexl.parser.ASTTernaryNode.value(ASTTernaryNode.java:41)
        at org.apache.commons.jexl.parser.ASTExpression.value(ASTExpression.java:54)
        at org.apache.commons.jexl.parser.ASTExpressionExpression.value(ASTExpressionExpression.java:56)
        at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
        at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
        at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:121)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
        at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
        at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
        at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
        at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
        at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
        at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
        at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
        at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:140)
        at org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:30)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:801)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
        at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)
        at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)
        at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)
        at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
        at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
        at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
        at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
        at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
        at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at winstone.BoundedExecutorService.lambda$scheduleNext$0(BoundedExecutorService.java:80)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Caused: hudson.lifecycle.RestartNotSupportedException: Failed to obtain the command line arguments of the process
        at hudson.lifecycle.UnixLifecycle.verifyRestartable(UnixLifecycle.java:104)
        at jenkins.model.Jenkins.safeRestart(Jenkins.java:4216)
        at jenkins.model.Jenkins.doSafeRestart(Jenkins.java:4176)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at hudson.cli.declarative.MethodBinder.call(MethodBinder.java:114)
        at hudson.cli.declarative.CLIRegisterer$1.main(CLIRegisterer.java:225)
        at org.jenkinsci.main.modules.sshd.CLICommandAdapter$1.run(CLICommandAdapter.java:37)
        at org.jenkinsci.main.modules.sshd.AsynchronousCommand$1.run(AsynchronousCommand.java:112)
        at java.lang.Thread.run(Thread.java:748)

This is a regression introduced in 2.136. JENKINS-52771 introduced new version of JNA library, which requires GLIBC 2.7 to start. I saw a similar issue JENKINS-10354 was fixed some time ago with recompilation of JNA library against older glibc fixed the issue. Would it be possible to fix it the same way now? RHEL5 extended support is up to 2020.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

o.v.nenashev@gmail.com (JIRA)

unread,
Oct 6, 2018, 3:40:02 AM10/6/18
to jenkinsc...@googlegroups.com

o.v.nenashev@gmail.com (JIRA)

unread,
Oct 6, 2018, 7:22:02 AM10/6/18
to jenkinsc...@googlegroups.com

gordon@idisagree.org (JIRA)

unread,
Nov 5, 2019, 6:32:02 PM11/5/19
to jenkinsc...@googlegroups.com

As info, I had this same issue and found that the cause was that Linux's 'noexec' setting was enabled on the directory that was being used as my JVM's JNI tmpdir. The fix was to use "-Djna.tmpdir=<some path that has execute capability>" in the java command line used to start Jenkins.

 

Specifically, I added this to my start script:

  1. Java's JNA tmpdir cannot be 'noexec' or Jenkins' restart functionality won't work so we're relocating it.
    mkdir -p ${JENKINS_HOME}/tmp
    TMPDIR_ARG=-Djna.tmpdir=${JENKINS_HOME}/tmp

nohup $JAVA_HOME/bin/java $TMPDIR_ARG -jar jenkins.war <various other arguments omitted as those will differ for you> &

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

gordon@idisagree.org (JIRA)

unread,
Nov 5, 2019, 6:33:03 PM11/5/19
to jenkinsc...@googlegroups.com
Gordon Daugherty edited a comment on Bug JENKINS-53924
As info, I had this same issue and found that the cause was that Linux's 'noexec' setting was enabled on the directory that was being used as my JVM's JNI tmpdir. The fix was to use "-Djna.tmpdir=<some path that has execute capability>" in the java command line used to start Jenkins.

 

Specifically, I added this to my start script:

{quote} \ # Java's JNA tmpdir cannot be 'noexec' or Jenkins' restart functionality won't work so we're relocating it.

mkdir -p ${JENKINS_HOME}/tmp
TMPDIR_ARG=-Djna.tmpdir=${JENKINS_HOME}/tmp

nohup $JAVA_HOME/bin/java $TMPDIR_ARG -jar jenkins.war <various other arguments omitted as those will differ for you> &

{quote}
Reply all
Reply to author
Forward
0 new messages