Newrelic with Spray

[Jonathan Knight]

I am trying to use Kamon to get my spray app (on heroku) to send data to newrelic. I have followed the Kamon instructions, and it is registering the spray-can classes, but when it gets to the newrelic part it fails with some type errors. Have I missed an import somewhere ?

2014-02-24T15:19:03.201454+00:00 app[web.1]: [AppClassLoader@41dee0d7] error can't determine implemented interfaces of missing type com.newrelic.agent.bridge.Config

2014-02-24T15:19:03.201454+00:00 app[web.1]: when processing declare parents com.newrelic.agent.config.AgentConfigImpl

2014-02-24T15:19:03.201454+00:00 app[web.1]: when processing type mungers 

2014-02-24T15:19:03.201454+00:00 app[web.1]: when weaving 

2014-02-24T15:19:03.201454+00:00 app[web.1]:  [Xlint:cantFindType]

2014-02-24T15:19:03.272104+00:00 app[web.1]: [AppClassLoader@41dee0d7] error can't determine implemented interfaces of missing type com.newrelic.agent.bridge.Config

2014-02-24T15:19:03.272104+00:00 app[web.1]: when processing declare parents com.newrelic.agent.config.AgentConfig

2014-02-24T15:19:03.272104+00:00 app[web.1]: when processing type mungers 

2014-02-24T15:19:03.272104+00:00 app[web.1]: when weaving 

2014-02-24T15:19:03.272104+00:00 app[web.1]:  [Xlint:cantFindType]

2014-02-24T15:19:03.632211+00:00 app[web.1]: Feb 24, 2014 15:19:03 +0000 [2 1] com.newrelic INFO: Agent Host: bf45f57a-5eab-4647-b847-c328f142a175 IP: 172.19.36.186

2014-02-24T15:19:03.652335+00:00 app[web.1]: [AppClassLoader@41dee0d7] error can't determine implemented interfaces of missing type com.newrelic.agent.bridge.PublicApi

2014-02-24T15:19:03.652335+00:00 app[web.1]: when processing declare parents com.newrelic.api.agent.NewRelicApiImplementation

2014-02-24T15:19:03.652335+00:00 app[web.1]: when processing type mungers 

2014-02-24T15:19:03.652335+00:00 app[web.1]: when weaving 

2014-02-24T15:19:03.652335+00:00 app[web.1]:  [Xlint:cantFindType]

2014-02-24T15:19:03.676602+00:00 app[web.1]: [AppClassLoader@41dee0d7] error can't determine implemented interfaces of missing type com.newrelic.agent.bridge.Transaction

2014-02-24T15:19:03.676602+00:00 app[web.1]: when processing declare parents com.newrelic.agent.Transaction

2014-02-24T15:19:03.676602+00:00 app[web.1]: when processing type mungers 

2014-02-24T15:19:03.676602+00:00 app[web.1]: when weaving 

2014-02-24T15:19:03.676602+00:00 app[web.1]:  [Xlint:cantFindType]

2014-02-24T15:19:03.707803+00:00 app[web.1]: [AppClassLoader@41dee0d7] error can't determine implemented interfaces of missing type com.newrelic.agent.bridge.PrivateApi

2014-02-24T15:19:03.707803+00:00 app[web.1]: when processing declare parents com.newrelic.agent.PrivateApiImpl

2014-02-24T15:19:03.707803+00:00 app[web.1]: when processing type mungers 

2014-02-24T15:19:03.707803+00:00 app[web.1]: when weaving 

2014-02-24T15:19:03.707803+00:00 app[web.1]:  [Xlint:cantFindType]

2014-02-24T15:19:03.716141+00:00 app[web.1]: [AppClassLoader@41dee0d7] error can't determine implemented interfaces of missing type com.newrelic.agent.bridge.Instrumentation

2014-02-24T15:19:03.716141+00:00 app[web.1]: when processing declare parents com.newrelic.agent.InstrumentationImpl

2014-02-24T15:19:03.716141+00:00 app[web.1]: when processing type mungers 

2014-02-24T15:19:03.716141+00:00 app[web.1]: when weaving 

2014-02-24T15:19:03.716141+00:00 app[web.1]:  [Xlint:cantFindType]

2014-02-24T15:19:03.938976+00:00 app[web.1]: [AppClassLoader@41dee0d7] error can't determine implemented interfaces of missing type com.newrelic.agent.bridge.AsyncApi

2014-02-24T15:19:03.938976+00:00 app[web.1]: when processing declare parents com.newrelic.agent.AsyncApiImpl

2014-02-24T15:19:03.938976+00:00 app[web.1]: when processing type mungers 

2014-02-24T15:19:03.938976+00:00 app[web.1]: when weaving 

2014-02-24T15:19:03.938976+00:00 app[web.1]:  [Xlint:cantFindType]

2014-02-24T15:19:04.267525+00:00 app[web.1]: [AppClassLoader@41dee0d7] error can't determine implemented interfaces of missing type com.newrelic.agent.bridge.ExitTracer

2014-02-24T15:19:04.267525+00:00 app[web.1]: when processing declare parents com.newrelic.agent.tracers.Tracer

2014-02-24T15:19:04.267525+00:00 app[web.1]: when processing type mungers 

2014-02-24T15:19:04.267525+00:00 app[web.1]: when weaving 

2014-02-24T15:19:04.267525+00:00 app[web.1]:  [Xlint:cantFindType]

2014-02-24T15:19:09.289894+00:00 app[web.1]: [AppClassLoader@41dee0d7] error can't determine implemented interfaces of missing type com.newrelic.agent.bridge.ObjectFieldManager

2014-02-24T15:19:09.289894+00:00 app[web.1]: when processing declare parents com.newrelic.agent.instrumentation.weaver.ObjectFieldManagerImpl

2014-02-24T15:19:09.289894+00:00 app[web.1]: when processing type mungers 

2014-02-24T15:19:09.289894+00:00 app[web.1]: when weaving 

2014-02-24T15:19:09.289894+00:00 app[web.1]:  [Xlint:cantFindType]

[Diego Parra]

Hi Jonathan,

you can give us more detail about the configuration? 

for example the version of NewRelic Agent and if you are running the application with -javaagent:/...aspectjweaver.jar -javaagent:/.../newrelic-agent-3.4.2.jar -Dnewrelic.environment=production -Dnewrelic.config.file=/.../newrelic.yml

Cheers,

Diego

[Jonathan Knight]

I am running with a Heroku Procfile which includes

 

 -J-javaagent:target/universal/stage/lib/org.aspectj.aspectjweaver-1.7.2.jar -J-javaagent:target/universal/stage/lib/com.newrelic.agent.java.newrelic-agent-3.4.2.jar

I don't specify production or the newrelic.yml file explicitly.

Thanks

Jonathan

[Ivan Topolnjak]

Jonathan, which Kamon version are you using? I remember seeing some messages like that before 0.0.13 due to incorrect filtering in our aop.xml files but that is fixed now. Also remember to add the kamon.newrelic sections to your configuration file [1] to make sure the data is being pushed correctly. Let us know how it goes, best regards!

[1] https://github.com/kamon-io/Kamon/blob/master/kamon-newrelic/src/main/resources/reference.conf

--
You received this message because you are subscribed to the Google Groups "kamon-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kamon-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Jonathan Knight]

I am using v 0.0.11 of Kamon-core, spray and newrelic as per your site. I will upgrade to 0.0.13 and see what happens.

[Jonathan Knight]

I upgrade to V 0.0.13, and I don't get errors on startup any more - but still no newrelic stats. 

I have attached the startup logs. My configs are: 

AKKA_OPTS:             -Dconfig.file=target/universal/stage/conf/application.conf

JAVA_OPTS:             -Xmx384m -Xss512k -XX:+UseCompressedOops

KAMON_OPTS:            -J-javaagent:target/universal/stage/lib/org.aspectj.aspectjweaver-1.7.2.jar

NEWRELIC_OPTS:         -J-javaagent:target/universal/stage/lib/com.newrelic.agent.java.newrelic-agent-3.4.2.jar

Newrelic says that the app is idle in the dashboard so nothing is being sent.

I have checked the akka conf file and the newrelic.yml file so they are OK. 

By the way I thought I would give your example newrelic app a go to see if I could get it reporting, what command line do you use for starting the example app, is it an sbt command ?

[Diego Parra]

Hi,

yes it´s a sbt application, but we recomend run the example inside of your favorite IDE, because sbt introduces some issues between the aspectj classloader weaver and its classloader. 

with the  -javaagent:/...aspectjweaver.jar -javaagent:/.../newrelic-agent-3.4.2.jar should be enough.

please let us know how it goes, best regards!

[Ivan Topolnjak]

Jonathan,

how are you? I looked at your log and there is something missing there, you should see something like this:

19:50:33.247 INFO  [undefined][akka://test/user/kamon-newrelic/agent] [test-akka.actor.default-dispatcher-9] kamon.newrelic.Agent - Agent initialized with runID: [609272566] and collector: [collector-0.newrelic.com]

after the application initialization ends... could you please turn on DEBUG logging to see if there is any problem when establishing the session between Kamon and NR collectors?

[Jonathan Knight]

Thanks for the help - I've got it all working now. I can see response times and which endpoint is being hit. It would be good to see if it is possible to broaden out the metrics over time, but this will mean we can run the app and have a chance to see what is going on.

[Ivan Topolnjak]

Cool! Please note that the default name for Spray traces in Kamon has the format "METHOD: PATH", and that seems like a reasonable default but if you hit different paths that logically represent the same operation, as would be to send a GET request to "/users/ivan" and then to "/users/diego", that will generate different trace names, or web transactions in NR terminology.. you can group them by using the `traceName` directive [1], might look like this:

get {

  path("users") {

    traceName("GetUsers") {
      // ...
    }
  }
}

the name can be any string you want, and that same name will overwrite the default so your web transactions will start showing with this name in NR! There will be a lot more information in NR soon, I'm on vacations until next week so after that you will see some improvements there, thanks a lot for trying Kamon and please feel free to ask anything you need, give us some feedback and suggestion to make Kamon better :).

Finally, wanted to point out that Kamon doesn't need you to use the newrelic agent. Currently we suggest using both agents because the NR agent pushes all the JVM metrics and Kamon only pushes the web transaction metrics. Depending on your license and requirements you might drop the NR agent if you want, since it introduces a startup + memory + cpu overhead. That is up to you, best regards!

[1] https://github.com/kamon-io/Kamon/blob/master/kamon-spray/src/main/scala/kamon/spray/KamonTraceDirectives.scala#L28

On Wed, Feb 26, 2014 at 10:44 PM, Jonathan Knight <jrlk...@googlemail.com> wrote:

Thanks for the help - I've got it all working now. I can see response times and which endpoint is being hit. It would be good to see if it is possible to broaden out the metrics over time, but this will mean we can run the app and have a chance to see what is going on.

--

[Jonathan Knight]

We've been monitoring our app for a couple of days and all looks fine. We are running on Heroku. 

A couple of comments from what we have seen so far: 

Is there a way to change the logging level as the app starts up ? There is so much that it is hard to pick out the important info - as I proved when trying work out why things weren't set up right. I have set the Akka logging to WARN to try to reduce the amount but then we lose some important output.

Secondly, the startup time is a problem for us, it is taking nearly a minute for the app to start. If it does not respond in 60 seconds Heroku assumes there is  problem and will restart the app again. Also it makes scaling/redeploying quite difficult if it takes so long for the app to start. Heroku can restart our app at any time, and does so at least once a day. Would it be better to compile in the AspectJ weaving rather than at startup ? As we are on Heroku we use sbt-native-packager to build the app (the heroku scala buildpack does a 'sbt clean stage' to build the app) so we would have to hook into the build process to make this work.

Very important to us is distribution of response times, because this will pick up problems early.

We will put our app into production next week for a small part of our traffic so I will have a better chance to see it working with real data.

[Diego Parra]

Hi,

in order to avoid the aspectj weaver logging you can add these system properties

-Dorg.aspectj.weaver.showWeaveInfo=false

-Daj.weaving.verbose=false

you could also try run the application with:

-Daj.weaving.cache.enabled=true

-Daj.weaving.cache.dir=/tmp/aspectj-cache

that enables a bytecode cache for loadtime weaving and basically it caches woven bytecode on first start of a system using LTW and then reuses that woven bytecode on subsequent starts - this saves weaving time and also memory consumption.

 best regards!

[Diego Parra]

Hi,

I was playing with some AspectJ Weaver settings and found this with good results:

-Daj.weaving.cache.enabled=true

-Daj.weaving.cache.dir=/tmp/aspectj-cache

-Daj.weaving.cache.impl=shared

 Let us know how it goes, best regards!

[Jonathan Knight]

Thanks Diego - but unfortunately with Heroku we can't do this kind of thing as we don't have access to the server. All we can do is send our repo up to Heroku for the build and control the start parameters.

[Ivan Topolnjak]

Jonathan,

I totally understand your concerns and I'm sad to say that there is not much we can do about it..  NewRelic is an awesome tool, but it's agent indeed introduces significant startup overhead as well as memory and cpu usage overhead. We try to lower the overhead of using Kamon to the minimum, and using the aspectj weaver is something we can't get rid of without making it more complicated for the users to get Kamon working, and we also found that the combination of both agents produce way higher startup times.. That's why I always end up talking about the fact that Kamon doesn't need the NR agent to work and in a regular notebook with Linux you can get a simple Spray application up and running only with Kamon in less than 15 seconds.

We don't want people to think that they *must* stop using any cool NR feature they might have available (profiling, jvm metrics and so on) to use Kamon and be successful with Akka and Spray, but make them aware of that they can choose.

Finally, the NR team suggested us to turn off the akka instrumentation in one of our applications by adding the following section to the newrelic.yml file:

  class_transformer:
    akka_instrumentation:
      enabled: false

please note that the spacing is necessary. Maybe that can lower the startup time a little more, but that is just a guess. Hope you find this information helpful, best regards!

[Jonathan Knight]

Just wanted to let you know we have been running Kamon in production for a couple of weeks now and it is working well.

[Ivan Topolnjak]

Jonathan,

Wonderful news! We are really glad to know Kamon is helping you monitor your reactive app :), any additional feedback is always welcome, best regards!

On Mon, Mar 31, 2014 at 10:41 AM, Jonathan Knight <jrlk...@googlemail.com> wrote:

Just wanted to let you know we have been running Kamon in production for a couple of weeks now and it is working well.

--

You received this message because you are subscribed to the Google Groups "kamon-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kamon-user+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.