Getting started with StatsD module

[Richard Rodseth]

After getting by with the log reporter for a while, I'm taking another stab at Statsd. 

I've used KiteMatic to launch the Docker image, and the settings tab tells me the access URL (numbers changed) is

123.456.99.100:32769 for Docker port 8125.

So I have the following in my Kamon statsd config for the app

    hostname = "123.456.99.100"

    port = 32769

But the dashboard shows $app and $hostname, and there are no datapoints in the graph.

Any idea what I'm missing? Thanks.

[Richard Rodseth]

Should have mentioned we're still on 0.3.5. 

[Richard Rodseth]

I started tackling the 0.3.5 to 0.4 or 0.5 upgrade in the hopes that I might have better luck setting up this StatsD demo using the Docker image, but that's not going well, and I would probably stick with 0.3.5 if I could just get the StatsD proof of concept working.

Any suggestions on this? If I want to ssh into the Docker image to run netcat to see if the UDP message is arriving, what's the password?

Thanks.

[Ivan Topolnjak]

Hey Richard!

Well, the docker image doesn't have any sshd running so you wont be able to ssh to it, but if you are in the same host you can attach to it and get a terminal to play with! Now, the category names for akka-related metrics changed in 0.4.0 and the Docker Image currently labeled as "latest" is matching the naming conventions on 0.4.x/0.5.x, I would suggest getting the upgrade to 0.5.0 ready first and then jump into setting things up for StatsD.

Since you are coming from 0.3.5, did you check the little migration guide in our docs? [1], specially the configuration changes and the fact that you now need to include your actor system name in the filters for akka-related stuff. Maybe things are in place but no filters are being matched!

[1] http://kamon.io/introduction/project-info/migrating-from-0.3.x-and-0.2.x/

--
You received this message because you are subscribed to the Google Groups "kamon-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kamon-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Richard Rodseth]

I had not paid enough attention to the migration docs, though I had changed the metrics config.

However, now I have now replaced core with scala and akka and get the error below. Why is there a core artifact if it's deprecated? Also, core is still mentioned here:

http://kamon.io/core/modules/using-modules/

In any case, I added this to my POM

<dependency>

<groupId>io.kamon</groupId>

<artifactId>kamon-scala_2.10</artifactId>

<version>0.4.0</version>

</dependency>

<dependency>

<groupId>io.kamon</groupId>

<artifactId>kamon-akka_2.10</artifactId>

<version>0.4.0</version>

</dependency>

and got this

Exception in thread "main" java.lang.NoSuchMethodError: kamon.scala.instrumentation.FutureInstrumentation.aspectOf()Lkamon/scala/instrumentation/FutureInstrumentation;

at scala.concurrent.impl.CallbackRunnable.<init>(Promise.scala:28)

at scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:269)

at akka.actor.ActorSystemImpl.terminationCallbacks$lzycompute(ActorSystem.scala:639)

[Richard Rodseth]

It's so confusing when the docs are for 0.5 and it's not released. I wish you could have different versions of the docs, the way Akka does.

[Ivan Topolnjak]

But, 0.5.0 was released last week! [1], what is about to come out is 0.5.1.. Also, the kamon-core module has not been deprecated by any means, we just took out all instrumentation to other separate modules in order to keep the core of metrics and tracing APIs clean.. not all of our users are in the Scala/Akka world.

Now, that NoSuchMethodError is really weird.. how are you setting up the weaver in your runtime?

[1] http://kamon.io/teamblog/2015/08/17/kamon-0.5.0-is-out/

[Richard Rodseth]

Sorry. I'll take a deep breath.

I misinterpreted that sentence of the migration guide. I now have core + akka (left out scala, not sure what it's even for).

You are correct, there are maven artifacts for 0.5, but I ran into that unsupported class version issue (not yet on 8) and so decided to go to 0.4 first.

As of now, I am successfully filtering my kamon logreporter logs into a separate file, but am only getting system metrics. Presumably I have something wrong with the actor filter, but I did add the akka prefix, and my alarm sytem name. Stay tuned. 

[Richard Rodseth]

Here's how I launch on my laptop (don't be confused by the directory name osgi - this is running standalone)

java -cp "/srv/osgi" -Dconfig.file=/srv/osgi/xyz-service-app.conf -javaagent:/srv/osgi/aspectjweaver-1.8.4.jar -jar target/xyz-service-app-1.0-SNAPSHOT.jar 

[Diego Parra]

Richard,

Btw you can take a look at our sbt-aspectj-runner plugin : https://github.com/kamon-io/sbt-aspectj-runner

Cheers,

Diego

[Richard Rodseth]

Anything else I can try? Still no actor metrics.

Starting Kamon like this:

  val customConfig  = ConfigFactory.load("xyz-service-app")

  Kamon.start(customConfig.withFallback(ConfigFactory.load()))

  implicit lazy val system = ActorSystem("xyz")

Config as shown below, and I get no actor metrics or dispatcher metrics in the log file, but do get system metrics (even though log-reporter.report-system-metrics is false, oddly enough)

kamon {

  internal-config {

  akka.loggers = ["akka.event.slf4j.Slf4jLogger"]

   }

  metric {

    # tick-interval = 60 seconds

    tick-interval = 10 seconds

    filters {

akka-actor {

      includes = [ "xyz/user/**" ]

      excludes = []

    } 

    akka-dispatcher {

    includes = [ "alarms/akka.actor.default-dispatcher" ]

  }

    trace {

      includes = [ "*" ]

      excludes = []

    }

    }

  }

  log-reporter {

    # Enable system metrics

    # In order to not get a ClassNotFoundException, we must register the kamon-sytem-metrics module

    report-system-metrics = false

  }

  statsd {

    # Hostname and port in which your StatsD is running. Remember that StatsD packets are sent using UDP and

    # setting unreachable hosts and/or not open ports wont be warned by the Kamon, your data wont go anywhere.

    hostname = "123.456.99.100"

    port = 32769

    # Interval between metrics data flushes to StatsD. It's value must be equal or greater than the

    # kamon.metrics.tick-interval setting.

    flush-interval = 60 seconds

    # Max packet size for UDP metrics data sent to StatsD.

    max-packet-size = 1024 bytes

    # Subscription patterns used to select which metrics will be pushed to StatsD. Note that first, metrics

    # collection for your desired entities must be activated under the kamon.metrics.filters settings.

    includes {

      actor       = [ "*" ]

      trace       = [ "*" ]

      dispatcher  = [ "*" ]

    }

    simple-metric-key-generator {

      # Application prefix for all metrics pushed to StatsD. The default namespacing scheme for metrics follows

      # this pattern:

      #    application.host.entity.entity-name.metric-name

      application = "xyz"

    }

  }

[Richard Rodseth]

Update. It's looking like the custom config is not getting picked up. 

Our xyz-service-app.conf is generated, and has this structure:

include "application"

<generated stuff>

include "xyz-service-app-overrides"

When I try something like the code below, the "kamon" config object printed out just contains the default values.

  val customConfig  = ConfigFactory.load("xyz-service-app")

  val configWithFallBack = customConfig.withFallback(ConfigFactory.load("application")).withFallback(ConfigFactory.defaultReference)

  val k = configWithFallBack.getConfig("kamon")

  println(s"k == $k")

  Kamon.start(configWithFallBack)