Quasar Core - Java Agent Loading Problem with Wildfly

306 views

Skip to first unread message

Pratik Parikh

unread,

Jul 12, 2015, 6:10:33 PM7/12/15

to quasar-pu...@googlegroups.com

Dear Authors,

I am trying to configure quasar core java agent with wildfly and am running into following problems.

1) I have update the standalone.conf under wildfly/bin/statndalone.config with following:

if [ "x$JBOSS_MODULES_SYSTEM_PKGS" = "x" ]; then

JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,co.paralleluniverse"

JAVA_OPTS="$JAVA_OPTS -javaagent:/opt/assets/liveoak/javaagent/quasar/quasar-core.jar"

2) started wildfly with following command ./bin/standalone.sh -b 0.0.0.0

When i hit the particular jboss modular service, i get the following because the quasar-core does not have all it's dependencies (Quasar depends on few 3rd party dependencies like Guava in this case but many others and thus I think there needs to be fat jar with all its dependency to deal with this problem but if you have other alternatives that work then please let me know and i will give it a try).

at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:32) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:299) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_25]

Caused by: java.lang.NoClassDefFoundError: com/google/common/collect/MapMaker

at co.paralleluniverse.fibers.FiberScheduler.<init>(FiberScheduler.java:36) [quasar-core-0.7.1-SNAPSHOT-jdk8.jar:0.7.1-SNAPSHOT]

at co.paralleluniverse.fibers.FiberForkJoinScheduler.<init>(FiberForkJoinScheduler.java:59) [quasar-core-0.7.1-SNAPSHOT-jdk8.jar:0.7.1-SNAPSHOT]

at co.paralleluniverse.fibers.FiberForkJoinScheduler.<init>(FiberForkJoinScheduler.java:73) [quasar-core-0.7.1-SNAPSHOT-jdk8.jar:0.7.1-SNAPSHOT]

Caused by: java.lang.ClassNotFoundException: com.google.common.collect.MapMaker

at java.net.URLClassLoader$1.run(URLClassLoader.java:372) [rt.jar:1.8.0_25]

at java.net.URLClassLoader$1.run(URLClassLoader.java:361) [rt.jar:1.8.0_25]

at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_25]

at java.net.URLClassLoader.findClass(URLClassLoader.java:360) [rt.jar:1.8.0_25]

at java.lang.ClassLoader.loadClass(ClassLoader.java:424) [rt.jar:1.8.0_25]

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) [rt.jar:1.8.0_25]

at java.lang.ClassLoader.loadClass(ClassLoader.java:357) [rt.jar:1.8.0_25]

... 61 more

Regards,

Pratik Parikh

fa...@paralleluniverse.co

unread,

Jul 13, 2015, 5:50:17 AM7/13/15

to quasar-pu...@googlegroups.com

Hi Pratik,

the agent currently works only for embedded servers. When running a servlet container in standalone mode you need to use an instrumenting classloader instead, which Comsat provides for Tomcat and Jetty but not yet for Wildfly.

I'm not very knowledgeable in Widlfly but standalone servlet containers usually have (at least) a "root" classloader loading classes common to all webapps (that's where Quasar instrumentation parts usually live) and then as many children classloaders as there are deployed webapps. This implements webapp classes segregation, which allows running several self-contained webapps together in the same JVM even if they have conflicting classes (1).

Usually, though, standalone servlet containers use this classloading delegation model: each child class loader will first try to load each class itself and only delegate to the root one when it cannot find it, unless the packages are system ones.

So, as an alternative to the "fat jar" (or copying to Wildfly all dependencies jars as well), I suggest you make sure your webapp contains the Quasar jar and all of its dependencies and then that you try to remove "co.paralleluniverse" from the system packages list. This should allow your application classes to load Quasar runtime classes from their own classloader first, where Quasar runtime dependencies (like Guava) can be found when they attempt to load the classes they depend on. Let me know if this works.

NOTES:

- Comsat's Tomcat and Jetty loaders are indeed fat jars including all dependencies: have a look at their build file to see how this can be done with Gradle (in fact it's quite simple).

- Quasar 0.7.2 is out so you don't need to use 0.7.1-SNAPSHOT anymore.

-- Fabio

(1) As an aside, I think this approach can get very complicated because it requires a lot of very tightly coupled work with IT/DevOps (just think about how difficult it can be for them to figure out if a common lib dependency upgrade creates an incompatibility with some code from Dev, and even for Dev actually, especially if they don't know of the upgrade) and it also complicates tuning each webapp individually and optimally (because the OS process and JVM settings will be shared between all webapps).

I think it's just much simpler and better to run each webapp in its own JVM through embedded servers. For example, Capsule can help a lot in making this straightforward.

Of course I understand from my own experience that it's not always cost-effective to quickly revolutionize established IT/DevOps processes and practices.

Pratik Parikh

unread,

Jul 13, 2015, 9:30:17 AM7/13/15

to fa...@paralleluniverse.co, quasar-pu...@googlegroups.com

Hi Fabio,

The problem for me is that Wildfly is a module based webcontainer and i have a micro service that is a module thus most of the suggestion about root classloader does not work for me. I tried to do that but did not find a way. Could we add a fat jar option in the gradle build it is been while i have updated gradle build so was wonder if someone from your team can help me with that?

One option that i could take is to building a capsule as you suggested but this will have some wrap up time for me which i kinda don't have.

Regards,

Pratik Parikh

--
You received this message because you are subscribed to a topic in the Google Groups "quasar-pulsar-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/quasar-pulsar-user/U8Kx8MWQ7lM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to quasar-pulsar-u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Pratik Parikh
- Mantra - Keep It Simple and Straightforward

fa...@paralleluniverse.co

unread,

Jul 14, 2015, 5:53:30 AM7/14/15

to quasar-pu...@googlegroups.com, fa...@paralleluniverse.co

Hi Pratik, I hope this can be helpful: https://github.com/circlespainter/quasar-fatjar

-- Fabio

Pratik Parikh

unread,

Jul 14, 2015, 11:47:14 AM7/14/15

to fa...@paralleluniverse.co, quasar-pu...@googlegroups.com

Fabio,

I am confused. i cloned the git repo locally and did the build and i don't find any quasar dependency aggregated in the jar. When i talk about fatjar i mean the following the output should have a single jar with all the dependency included. for example.

co.parallelunivers.*

com.google.common.*

com.esotericsoftware.*

org.hdrhistogram.*

org.latencyutils.*

de.javakaffee.*

com.lmax.*

This way when i add the jar as a javaagent in WildFly then it will be self contained. Below is the discussion of such other group that ran into this, thank very fun for helping me with this.
https://issues.jboss.org/browse/HWKBTM-24

Regards,

Pratik Parikh

--

You received this message because you are subscribed to a topic in the Google Groups "quasar-pulsar-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/quasar-pulsar-user/U8Kx8MWQ7lM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to quasar-pulsar-u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

fa...@paralleluniverse.co

unread,

Jul 14, 2015, 12:18:57 PM7/14/15

to quasar-pu...@googlegroups.com, fa...@paralleluniverse.co

Sorry, my fault: a last-sec simplification had broken the fatjar build but now it should be ok.

Just "git pull", "gradle clean" it and then run "gradle --info", which will perform "gradle test --info" which in turn depends on "gradle assemble". The latter will create "build/libs/

quasar-fatjar-0.7.3-SNAPSHOT.jar" (should be something like 5066993 bytes) and "quasar-fatjar-0.7.3-SNAPSHOT-jdk8.jar" (should be something like 4816125 bytes) embedding all of the dependencies you mentioned and including the Quasar instrumentor manifest.

You can use this Gradle-based Quasar fatjar example as a starting point.

-- Fabio

Pratik Parikh

unread,

Jul 16, 2015, 7:33:30 PM7/16/15

to fa...@paralleluniverse.co, quasar-pu...@googlegroups.com

Hi Fabio,

The fatjar worked and i was able to use it as base to add libraries as needed. I am running into a problem wanted your opinion to see if this was because of quasar or something that i am not performing correctly?

2015-07-16 23:22:11,818 DEBUG [io.liveoak.places.nearby] (nioEventLoopGroup-2-2) running provider nokia

2015-07-16 23:22:11,867 DEBUG [io.liveoak.places.nearby] (ForkJoinPool.commonPool-worker-1) running provider googleplaces

2015-07-16 23:22:11,899 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Exception in Fiber "fiber-placesProviderService-10000004" java.lang.VerifyError: Bad type on operand stack

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Exception Details:

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Location:

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) org/scribe/model/Request.<init>(Lorg/scribe/model/Verb;Ljava/lang/String;Ljava/lang/Boolean;)V @94: invokespecial

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Reason:

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Type 'co/paralleluniverse/fibers/okhttp/FiberOkHttpClient' (current frame, stack[3]) is not assignable to 'com/squareup/okhttp/OkHttpClient'

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Current Frame:

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) bci: @94

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) flags: { }

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) locals: { 'org/scribe/model/Request', 'org/scribe/model/Verb', 'java/lang/String', 'java/lang/Boolean' }

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) stack: { 'org/scribe/model/Request', uninitialized 83, uninitialized 83, 'co/paralleluniverse/fibers/okhttp/FiberOkHttpClient' }

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Bytecode:

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000000: 2ab7 0001 2a01 b500 022a 01b5 0003 2a03

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000010: b500 042a 04b5 0005 2a01 b500 062a 01b5

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000020: 0007 2a01 b500 082a 2bb5 0009 2a2c b500

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000030: 0a2a bb00 0b59 b700 0cb5 000d 2abb 000b

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000040: 59b7 000c b500 0e2a bb00 0f59 b700 10b5

2015-07-16 23:22:11,900 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000050: 0011 2abb 0012 59bb 0013 59b7 0014 b700

2015-07-16 23:22:11,901 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000060: 15b5 0016 2a2d b500 17b1

2015-07-16 23:22:11,901 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1)

2015-07-16 23:22:11,901 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at io.liveoak.places.service.NearbyPlacesService.run(NearbyPlacesService.java:151)

2015-07-16 23:22:11,901 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at io.liveoak.places.service.NearbyPlacesService.run(NearbyPlacesService.java:38)

2015-07-16 23:22:11,901 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at co.paralleluniverse.fibers.Fiber.run(Fiber.java:1019)

2015-07-16 23:22:11,947 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Exception in Fiber "fiber-placesProviderService-10000005" java.lang.VerifyError: Bad type on operand stack

2015-07-16 23:22:11,947 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Exception Details:

2015-07-16 23:22:11,947 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Location:

2015-07-16 23:22:11,947 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) org/scribe/model/Request.<init>(Lorg/scribe/model/Verb;Ljava/lang/String;Ljava/lang/Boolean;)V @94: invokespecial

2015-07-16 23:22:11,947 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Reason:

2015-07-16 23:22:11,947 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Type 'co/paralleluniverse/fibers/okhttp/FiberOkHttpClient' (current frame, stack[3]) is not assignable to 'com/squareup/okhttp/OkHttpClient'

2015-07-16 23:22:11,947 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Current Frame:

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) bci: @94

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) flags: { }

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) locals: { 'org/scribe/model/Request', 'org/scribe/model/Verb', 'java/lang/String', 'java/lang/Boolean' }

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) stack: { 'org/scribe/model/Request', uninitialized 83, uninitialized 83, 'co/paralleluniverse/fibers/okhttp/FiberOkHttpClient' }

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Bytecode:

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000000: 2ab7 0001 2a01 b500 022a 01b5 0003 2a03

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000010: b500 042a 04b5 0005 2a01 b500 062a 01b5

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000020: 0007 2a01 b500 082a 2bb5 0009 2a2c b500

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000030: 0a2a bb00 0b59 b700 0cb5 000d 2abb 000b

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000040: 59b7 000c b500 0e2a bb00 0f59 b700 10b5

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000050: 0011 2abb 0012 59bb 0013 59b7 0014 b700

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000060: 15b5 0016 2a2d b500 17b1

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1)

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at io.liveoak.places.service.NearbyPlacesService.run(NearbyPlacesService.java:151)

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at io.liveoak.places.service.NearbyPlacesService.run(NearbyPlacesService.java:38)

2015-07-16 23:22:11,948 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at co.paralleluniverse.fibers.Fiber.run(Fiber.java:1019)

2015-07-16 23:22:11,952 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) WARNING: Uninstrumented methods (marked '**') or call-sites (marked '!!') detected on the call stack:

2015-07-16 23:22:11,952 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at io.liveoak.places.service.NearbyPlacesService.run (NearbyPlacesService.java:151 bci: 350) **

2015-07-16 23:22:11,952 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at io.liveoak.places.service.NearbyPlacesService.run (NearbyPlacesService.java:38 bci: 1) **

2015-07-16 23:22:11,952 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at co.paralleluniverse.fibers.Fiber.run (Fiber.java:1019 bci: 11)

2015-07-16 23:22:11,952 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at co.paralleluniverse.fibers.Fiber.run1 (Fiber.java:1014 bci: 1)

2015-07-16 23:22:11,952 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Exception in Fiber "fiber-placesProviderService-10000004" java.lang.VerifyError: Bad type on operand stack

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Exception Details:

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Location:

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) org/scribe/model/Request.<init>(Lorg/scribe/model/Verb;Ljava/lang/String;Ljava/lang/Boolean;)V @94: invokespecial

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Reason:

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Type 'co/paralleluniverse/fibers/okhttp/FiberOkHttpClient' (current frame, stack[3]) is not assignable to 'com/squareup/okhttp/OkHttpClient'

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Current Frame:

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) bci: @94

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) flags: { }

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) locals: { 'org/scribe/model/Request', 'org/scribe/model/Verb', 'java/lang/String', 'java/lang/Boolean' }

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) stack: { 'org/scribe/model/Request', uninitialized 83, uninitialized 83, 'co/paralleluniverse/fibers/okhttp/FiberOkHttpClient' }

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) Bytecode:

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000000: 2ab7 0001 2a01 b500 022a 01b5 0003 2a03

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000010: b500 042a 04b5 0005 2a01 b500 062a 01b5

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000020: 0007 2a01 b500 082a 2bb5 0009 2a2c b500

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000030: 0a2a bb00 0b59 b700 0cb5 000d 2abb 000b

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000040: 59b7 000c b500 0e2a bb00 0f59 b700 10b5

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000050: 0011 2abb 0012 59bb 0013 59b7 0014 b700

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) 0x0000060: 15b5 0016 2a2d b500 17b1

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1)

2015-07-16 23:22:11,953 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at io.liveoak.places.service.NearbyPlacesService.run(NearbyPlacesService.java:151)

2015-07-16 23:22:11,954 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at io.liveoak.places.service.NearbyPlacesService.run(NearbyPlacesService.java:38)

2015-07-16 23:22:11,954 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at co.paralleluniverse.fibers.Fiber.run(Fiber.java:1019)

2015-07-16 23:22:11,954 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) WARNING: Uninstrumented methods (marked '**') or call-sites (marked '!!') detected on the call stack:

2015-07-16 23:22:11,954 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at io.liveoak.places.service.NearbyPlacesService.run (NearbyPlacesService.java:151 bci: 350) **

2015-07-16 23:22:11,954 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at io.liveoak.places.service.NearbyPlacesService.run (NearbyPlacesService.java:38 bci: 1) **

2015-07-16 23:22:11,954 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at co.paralleluniverse.fibers.Fiber.run (Fiber.java:1019 bci: 11)

2015-07-16 23:22:11,954 ERROR [stderr] (ForkJoinPool-placesProviderService-worker-1) at co.paralleluniverse.fibers.Fiber.run1 (Fiber.java:1014 bci: 1)

2015-07-16 23:22:11,955 DEBUG [io.liveoak.places.nearby] (nioEventLoopGroup-2-2) provide response as an error: java.util.concurrent.ExecutionException: java.lang.VerifyError: Bad type on operand stack

Exception Details:

Location:

org/scribe/model/Request.<init>(Lorg/scribe/model/Verb;Ljava/lang/String;Ljava/lang/Boolean;)V @94: invokespecial

Reason:

Type 'co/paralleluniverse/fibers/okhttp/FiberOkHttpClient' (current frame, stack[3]) is not assignable to 'com/squareup/okhttp/OkHttpClient'

Current Frame:

bci: @94

flags: { }

locals: { 'org/scribe/model/Request', 'org/scribe/model/Verb', 'java/lang/String', 'java/lang/Boolean' }

stack: { 'org/scribe/model/Request', uninitialized 83, uninitialized 83, 'co/paralleluniverse/fibers/okhttp/FiberOkHttpClient' }

Bytecode:

0x0000000: 2ab7 0001 2a01 b500 022a 01b5 0003 2a03

0x0000010: b500 042a 04b5 0005 2a01 b500 062a 01b5

0x0000020: 0007 2a01 b500 082a 2bb5 0009 2a2c b500

0x0000030: 0a2a bb00 0b59 b700 0cb5 000d 2abb 000b

0x0000040: 59b7 000c b500 0e2a bb00 0f59 b700 10b5

0x0000050: 0011 2abb 0012 59bb 0013 59b7 0014 b700

0x0000060: 15b5 0016 2a2d b500 17b1

at co.paralleluniverse.fibers.Fiber.get(Fiber.java:1304) [quasar-fatjar-0.7.3-SNAPSHOT-jdk8.jar:0.7.3-SNAPSHOT]

at io.liveoak.places.resources.NearbyResource.lambda$members$17(NearbyResource.java:197)

at io.liveoak.places.resources.NearbyResource$$Lambda$36/1270605911.accept(Unknown Source)

at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) [rt.jar:1.8.0_25]

at java.util.TreeMap$ValueSpliterator.forEachRemaining(TreeMap.java:2893) [rt.jar:1.8.0_25]

at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512) [rt.jar:1.8.0_25]

at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) [rt.jar:1.8.0_25]

at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) [rt.jar:1.8.0_25]

at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [rt.jar:1.8.0_25]

at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:400) [rt.jar:1.8.0_25]

at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:728) [rt.jar:1.8.0_25]

at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) [rt.jar:1.8.0_25]

at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) [rt.jar:1.8.0_25]

at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) [rt.jar:1.8.0_25]

at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) [rt.jar:1.8.0_25]

at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583) [rt.jar:1.8.0_25]

at io.liveoak.places.resources.NearbyResource.members(NearbyResource.java:182)

at io.liveoak.spi.resource.SynchronousResource.readMembers(SynchronousResource.java:47) [liveoak-spi.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.common.codec.driver.MembersEncodingDriver.encode(MembersEncodingDriver.java:32) [liveoak-common.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.common.codec.driver.AbstractEncodingDriver.encodeNext(AbstractEncodingDriver.java:80) [liveoak-common.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.common.codec.driver.PropertiesEncodingDriver.close(PropertiesEncodingDriver.java:38) [liveoak-common.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.common.codec.driver.AbstractEncodingDriver.encodeNext(AbstractEncodingDriver.java:77) [liveoak-common.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.common.codec.driver.PropertiesEncodingDriver$MyPropertySink.complete(PropertiesEncodingDriver.java:96) [liveoak-common.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.spi.resource.SynchronousResource.readProperties(SynchronousResource.java:81) [liveoak-spi.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.common.codec.driver.PropertiesEncodingDriver.encode(PropertiesEncodingDriver.java:30) [liveoak-common.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.common.codec.driver.AbstractEncodingDriver.encodeNext(AbstractEncodingDriver.java:80) [liveoak-common.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.common.codec.driver.ResourceEncodingDriver.encode(ResourceEncodingDriver.java:35) [liveoak-common.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.container.ResourceStateHandler.encode(ResourceStateHandler.java:84) [liveoak-container.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.container.ResourceStateHandler.lambda$write$27(ResourceStateHandler.java:46) [liveoak-container.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.container.ResourceStateHandler$$Lambda$30/1460046456.run(Unknown Source)

at io.liveoak.container.ResourceStateHandler.write(ResourceStateHandler.java:52) [liveoak-container.jar:1.0.0.Final-SNAPSHOT]

at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:691) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:681) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:716) [netty-all-4.0.25.Final.jar:4.0.25.Final]

at io.liveoak.container.traversal.BaseResponder.resourceRead(BaseResponder.java:45) [liveoak-container.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.container.traversal.TraversingResponder.resourceRead(TraversingResponder.java:35) [liveoak-container.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.spi.resource.SynchronousResource.readMember(SynchronousResource.java:68) [liveoak-spi.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.container.traversal.ReadStep.execute(ReadStep.java:21) [liveoak-container.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.container.traversal.TraversingResponder.lambda$doStep$2(TraversingResponder.java:86) [liveoak-container.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.container.traversal.TraversingResponder$$Lambda$29/198839400.run(Unknown Source)

at io.liveoak.container.traversal.TraversingResponder.doStep(TraversingResponder.java:94) [liveoak-container.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.container.traversal.TraversingResponder.doNextStep(TraversingResponder.java:45) [liveoak-container.jar:1.0.0.Final-SNAPSHOT]

at io.liveoak.container.traversal.TraversingResponder.resourceRead(TraversingResponder.java:39) [liveoak-container.jar:1.0.0.Final-SNAPSHOT]