Scriptler "suddenly" broken - not sure how to investigate further

[Itamar Ostricher]

Hi,

We've been using the Scriptler plugin with Jenkins for a couple of months now, with no problems so far.

Earlier today, I restarted the Jenkins master, and it appears that suddenly Scriptler stopped functioning correctly. Restarting again didn't help.

For example, when accessing the "spark_status.groovy" page at "http://our.jenkins.master/scriptler/run/spark_status.groovy?build_id=686", which would normally return a short status JSON, I now get the stack trace below.

I have no idea what's going on. We haven't changed anything in the Jenkins environment itself (Jenkins / Scriptler versions installed) or in the groovy scripts. Also, the fact that the stack trace is truncated doesn't really help in understanding what's going on.

Any advice would be very appreciated, as we're relying on Scriptler to drive our Spark-based production pipelines, which means they're down until we resolve this...

Stack trace

javax.servlet.ServletException: java.lang.NullPointerException
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:411)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:123)
	at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120)
	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:46)
	at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103)
	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:42)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:114)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:370)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
	at org.jenkinsci.plugins.scriptler.ScriptlerManagement.prepareParameters(ScriptlerManagement.java:545)
	at org.jenkinsci.plugins.scriptler.ScriptlerManagement.doRun(ScriptlerManagement.java:527)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	... 53 more

Thanks,

- Itamar.

[Victor Martinez]

Have you tried to debug those logs?

https://wiki.jenkins-ci.org/display/JENKINS/Logging

As far as I see:
- greenballs plugin
- scm syn configuration
- privileges

You can also double check whether those privileges are granted properly for running scripts.

I hope it helps

[Itamar Ostricher]

Hi Victor,

Thanks.

I don't quite follow what it means to "debug those logs".

I created a new logger, following the instructions in the link you mentioned, and added a couple of log handlers to it (tried level "fine" or "all"). I get very few log lines on that logger, so I probably don't know what handlers I should write for debugging this properly.

What privileges are you referring to? Do you mean filesystem level ACL's? Or something else?

Sorry, I'm not a Jenkins developer, or even a Java developer. Just a simple Jenkins user :-)

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/8ASFRjj3z8A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/76e137e0-cfb9-46ef-b0fe-f8d2819618e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Slide]

I had to uninstall greenballs a while back, it caused all sorts of weird issues in different places.

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAMD8Vh3E3-5GvTZ2t--Z6NFa-4rye7U5TY_jUZaguf_so%3DKRVQ%40mail.gmail.com.

[Itamar Ostricher]

Uninstalled greenballs too. No change in the errors.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPiUgVdS9FVjpgS5fT9Qq9ADU11d25d4J6Txmou0%2BUMsuyVdpg%40mail.gmail.com.

[Victor Martinez]

If you use scritler plugin you can add the below logger:
- org.jenkinsci.plugins.scriptler.*

As you can see In the below source code:
- https://github.com/jenkinsci/scriptler-plugin/blob/master/src/main/java/org/jenkinsci/plugins/scriptler/ScriptlerPluginImpl.java

I meant, to add some finest levels in the logger

Security Jenkins layout is based on different types:
- https://wiki.jenkins-ci.org/display/JENKINS/Standard+Security+Setup

If you configure another jenkins instance with similar plugin setup, does it work?

You can use a Jenkins docker instance easily, use the below page:
- http://jenkins-ci.org/content/official-jenkins-lts-docker-image

Besides of that, if you uninstalled the green plugin did you restart your Jenkins service?

Cheera

[Itamar Ostricher]

Thanks again Victor!

I added the wildcard logger, but still don't see much in that custom logger. I'd like to see the full stack trace - what am I doing wrong?

I disabled security completely for sake of debugging this.

This master is already running in a docker container.

Another instance does not present the same symptoms.

But Jenkins is not a "clean" containerized application - it has a lot of state stored outside and mounted from the host, so I guess something went wrong there...

I did restart the service after uninstalling green balls plugin.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/8ASFRjj3z8A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/4cd12b3d-b7d7-47e6-9511-53de497115f7%40googlegroups.com.

[Victor Martinez]

Uhm, I'm afraid I've not clue, if you say you can use a similar jenkins instance somewhere else and cannot see those errors, it might be related to some "corruption"... it's weird though. Maybe it's a presentation layout issue rather than backend one so in that case the logger won't provide enough information AFAIK.

Have you tried to downgrade that scriptler version? 

Can yo uplease raise a ticket in https://issues.jenkins-ci.org/secure/Dashboard.jspa? maybe someone else already reported a similar error

[Itamar Ostricher]

I ended up setting up a new jenkins instance and copying the state from the broken one, and it seemed to be working fine.

I don't really know what happened, so I don't quite have what to report in a ticket. I don't think a partial stack trace is very useful...

Thanks anyway :-)

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/88a98a6d-f754-4ba2-9089-9775085abfe3%40googlegroups.com.

[gsim...@netflix.com]

I was able to fix this issue by removing all files from the scriptler git repo.  In the Jenkins home directory:

cd scriptler/scripts
rm./*
git add *
git commit -am "Remove corrupted scripts"
 

In my case, the scripts were on disk from a failed scriptler install and they were added automatically by scriptler (per the git log).

The files will still appear in the gui until you restart Jenkins, reload config from disk or my favorite, delete them from the scriptler page.
Deleting them from the front end without removing them from the git repo did not solve the problem.