Troubles starting simian army war with jetty runner

[Alexey Gravanov]

Hi guys,

We are trying to run a simian army, but not exactly using the way described in the quick start guide (./gradlew jettyRun). We're trying to build simian army as single artefact (simianarmy-x.y.z.war) and then deploy it as a binary artefact to an instance and run it with jetty with our own configs afterwards.

Took latest jetty-runner-9.3.7.v20160115.jar, tried to start it with our own configs located in ./config folder:

java \
 -Dchaos.properties=/as24_chaos.properties \
 -Dclient.properties=/as24_client.properties \
 -Dconformity.properties=/as24_conformity.properties \
 -Djanitor.properties=/as24_janitor.properties \
 -Dlog4j.properties=/as24_log4j.properties \
 -Dsimianarmy.properties=/as24_simianarmy.properties \
 -DvolumeTagging.properties=/as24_volumeTagging.properties \
 -jar jetty-runner-9.3.7.v20160115.jar \
 --classes config \
 simianarmy-2.5.0.war

It starts, loads our configs (we can see our settings as simian army prints it), start doing something, like adding monkeys, lists our ASGs, but afterwards throws an exception, I assume it happened while jetty is trying to provide simian army over HTTP:

2016-03-10 12:33:34.780:INFO:oejs.ServerConnector:main: Started ServerConnector@1f892cc2{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
Exception in thread "main" java.lang.IllegalAccessError: tried to access method com.sun.jersey.core.reflection.ReflectionHelper.getContextClassLoader()Ljava/lang/ClassLoader; from class com.sun.jersey.spi.scanning.AnnotationScannerListener
    at com.sun.jersey.spi.scanning.AnnotationScannerListener.<init>(AnnotationScannerListener.java:89)
    at com.sun.jersey.spi.scanning.PathProviderScannerListener.<init>(PathProviderScannerListener.java:59)
    at com.sun.jersey.api.core.ScanningResourceConfig.init(ScanningResourceConfig.java:79)
    at com.sun.jersey.api.core.PackagesResourceConfig.init(PackagesResourceConfig.java:104)
    at com.sun.jersey.api.core.PackagesResourceConfig.<init>(PackagesResourceConfig.java:78)
    at com.sun.jersey.api.core.PackagesResourceConfig.<init>(PackagesResourceConfig.java:89)
    at com.sun.jersey.spi.container.servlet.WebComponent.createResourceConfig(WebComponent.java:700)
    at com.sun.jersey.spi.container.servlet.WebComponent.createResourceConfig(WebComponent.java:678)
    at com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:203)
    at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:373)
    at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:556)
    at javax.servlet.GenericServlet.init(GenericServlet.java:244)
    at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:640)
    at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:419)
    at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:350)
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1379)
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1341)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
    at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:263)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:517)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.server.Server.start(Server.java:405)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.Server.doStart(Server.java:372)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.runner.Runner.run(Runner.java:502)
    at org.eclipse.jetty.runner.Runner.main(Runner.java:547)

Ok, first idea was we got wrong version of jetty and something went wrong there, so we got exactly same version which is used by gradle jetty plugin, the very old 6.1.25, but still got same stuff:

2016-03-10 14:53:33.621:WARN::failed jersey-servlet: java.lang.IllegalAccessError: tried to access method com.sun.jersey.core.reflection.ReflectionHelper.getContextClassLoader()Ljava/lang/ClassLoader; from class com.sun.jersey.spi.scanning.AnnotationScannerListener
2016-03-10 14:53:33.621:WARN::failed org.mortbay.jetty.webapp.WebAppContext@1a9c38eb{/,simianarmy-2.5.0.war}: java.lang.IllegalAccessError: tried to access method com.sun.jersey.core.reflection.ReflectionHelper.getContextClassLoader()Ljava/lang/ClassLoader; from class com.sun.jersey.spi.scanning.AnnotationScannerListener
2016-03-10 14:53:33.621:WARN::failed ContextHandlerCollection@319bc845: java.lang.IllegalAccessError: tried to access method com.sun.jersey.core.reflection.ReflectionHelper.getContextClassLoader()Ljava/lang/ClassLoader; from class com.sun.jersey.spi.scanning.AnnotationScannerListener
2016-03-10 14:53:33.621:WARN::Error starting handlers
java.lang.IllegalAccessError: tried to access method com.sun.jersey.core.reflection.ReflectionHelper.getContextClassLoader()Ljava/lang/ClassLoader; from class com.sun.jersey.spi.scanning.AnnotationScannerListener
 at com.sun.jersey.spi.scanning.AnnotationScannerListener.<init>(AnnotationScannerListener.java:89)
 at com.sun.jersey.spi.scanning.PathProviderScannerListener.<init>(PathProviderScannerListener.java:59)
 at com.sun.jersey.api.core.ScanningResourceConfig.init(ScanningResourceConfig.java:79)
 at com.sun.jersey.api.core.PackagesResourceConfig.init(PackagesResourceConfig.java:104)
 at com.sun.jersey.api.core.PackagesResourceConfig.<init>(PackagesResourceConfig.java:78)
 at com.sun.jersey.api.core.PackagesResourceConfig.<init>(PackagesResourceConfig.java:89)
 at com.sun.jersey.spi.container.servlet.WebComponent.createResourceConfig(WebComponent.java:700)
 at com.sun.jersey.spi.container.servlet.WebComponent.createResourceConfig(WebComponent.java:678)
 at com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:203)
 at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:373)
 at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:556)
 at javax.servlet.GenericServlet.init(GenericServlet.java:244)
 at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
 at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685)
 at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
 at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1272)
 at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
 at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
 at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
 at org.mortbay.jetty.Server.doStart(Server.java:224)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at org.mortbay.jetty.Main.main(Main.java:89)

Ok, tried to start "./gradlew jettyRunWar" and got exactly same problem.

Does anyone has an idea what can go wrong here?

Thanks in advance!

Regards

Alexey

[Alexey Gravanov]

Ok, took latest Tomcat, put .war there, started, simian army was started, even killed one of my instanced, but when I opened .../v1/chaos, I got same exception like with jetty:

java.lang.IllegalAccessError: tried to access method com.sun.jersey.core.reflection.ReflectionHelper.getContextClassLoader()Ljava/lang/ClassLoader; from class com.sun.jersey.spi.scanning.AnnotationScannerListener

        com.sun.jersey.spi.scanning.AnnotationScannerListener.<init>(AnnotationScannerListener.java:89)
        com.sun.jersey.spi.scanning.PathProviderScannerListener.<init>(PathProviderScannerListener.java:59)
        com.sun.jersey.api.core.ScanningResourceConfig.init(ScanningResourceConfig.java:79)
        com.sun.jersey.api.core.PackagesResourceConfig.init(PackagesResourceConfig.java:104)
        com.sun.jersey.api.core.PackagesResourceConfig.<init>(PackagesResourceConfig.java:78)
        com.sun.jersey.api.core.PackagesResourceConfig.<init>(PackagesResourceConfig.java:89)
        com.sun.jersey.spi.container.servlet.WebComponent.createResourceConfig(WebComponent.java:700)
        com.sun.jersey.spi.container.servlet.WebComponent.createResourceConfig(WebComponent.java:678)
        com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:203)
        com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:373)
        com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:556)
        javax.servlet.GenericServlet.init(GenericServlet.java:158)
        org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
        org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
        org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
        org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522)
        org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095)
        org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672)
        org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
        org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        java.lang.Thread.run(Thread.java:745)

Does anyone has any suggestions what it could be? I'm running out of ideas at the moment...

Best regards

Alexey

[Alexey Gravanov]

I found what is missing, jersey-bundle is excluded from dependencies:

https://github.com/Netflix/SimianArmy/blob/master/build.gradle#L42

After adding jersey-bundle-1.19.jar to classpath everything working like a charm. Change was introduced by this commit:

https://github.com/Netflix/SimianArmy/commit/2dc17b1658867d66d98d9e9581f125f11124320c

Any idea why / what for it was done? How you guys run Simian Army so that you do not have any problems with it?

[Ed Bukoski]

Alexey -- Thanks for tracking this down. If you create a PR I can merge it in.

--
You received this message because you are subscribed to the Google Groups "Simian Army Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to simianarmy-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexey Gravanov]

Hi Ed,

With second try I managed to trace down and fix the problem, which was also created on github for the sake of traceability: https://github.com/Netflix/SimianArmy/issues/246

https://github.com/Netflix/SimianArmy/pull/247 was fail, so I closed it on my own.

I believe https://github.com/Netflix/SimianArmy/pull/248 is the right solution.

It would be great if you can merge it. Thanks!

Regards

Alexey