don't get mailbox metrics from kamon

[alon...@gmail.com]

hi,

trying to collect actors' mailbox metrics, with no lack so far.
timer_data_type, timer_data, counter types arrives, but not mailbox's metrics. or it's there and i simply don't know how to read?
can the problem be spotted by looking at my config?

thanks!

# Default Config
service {
port = 18080
externalAddress = "http://localhost"
}
db {
orient.connection.url = "remote:is-performance-db/inventory"
}
akka {
loglevel = "INFO"
extensions = ["kamon.metric.Metrics", "kamon.statsd.StatsD"]
}
kamon {
metric {
tick-interval = 10 seconds
default-collection-context-buffer-size = 33792
disable-aspectj-weaver-missing-error = false
track-unmatched-entities = yes
filters {
trace {
includes = [ "**" ]
excludes = [ ]
}
}
default-instrument-settings {
histogram {
precision = normal
lowest-discernible-value = 1
highest-trackable-value = 3600000000000
}
min-max-counter {
precision = normal
lowest-discernible-value = 1
highest-trackable-value = 999999999
refresh-interval = 100 milliseconds
}
gauge {
precision = normal
lowest-discernible-value = 1
highest-trackable-value = 3600000000000
refresh-interval = 100 milliseconds
}
}
instrument-settings {
}
}
trace {
level-of-detail = metrics-only
sampling = random
random-sampler {
chance = 10
}
ordered-sampler {
sample-interval = 8
}
threshold-sampler {
minimum-elapsed-time = 1 second
}
incubator {
min-incubation-time = 5 seconds
check-interval = 1 second
max-incubation-time = 20 seconds
}
}
internal-config {
akka.actor.default-dispatcher {
fork-join-executor {
parallelism-min = 2
parallelism-factor = 2.0
parallelism-max = 10
}
}
spray {
}
}
show-aspectj-missing-warning = yes
modules {
}
statsd {
hostname = "elasticsearch"
port = 8125
flush-interval = 1 second
max-packet-size = 1024 bytes
includes {
actor = ["*"]
trace = ["*"]
dispatcher = ["*"]
}
simple-metric-key-generator {
application = "kamon"
}
}
}

[Ivan Topolnjak]

Hello Alon, welcome to our community!

May I ask what Kamon version are you using? the configuration settings that you have there do not look like those of the 0.3.5/0.2.5 release which was the latest public release, but do look like the settings we have in our master branch. 

If you are using 0.3.5/0.2.5, which seems to be the most likely scenario, please make the configuration match the expected structure, most important being these two items:

  - the earlier Kamon release uses `kamon.metrics` rather than `kamon.metric`.

  - you have no actor filters included there, I'm not sure if your actors are actually being tracked or not. 

Please, take a look at the reference.conf file for the 0.3.5 release [1] and ensure your settings are in the right places. Also, you don't need to provide all the settings again in your application.conf file, only those that change from the default values in our reference.conf file.

When you pass that you should start seeing the actor metrics in your StatsD instance. Please note that the mailbox size is measured using a MinMaxCounter [2] and as such, it's snapshot is essentialy a Histogram. When using statsd, that data will be sent as a timer and you should look up the mailbox size metrics with a pattern like this one:

stats.timers.[[application]].[[host]].actor.[[actor-name]].mailbox-size

Please, let us know if you need any further help, regards!

[1] https://github.com/kamon-io/Kamon/blob/v0.3.5/kamon-core/src/main/resources/reference.conf#L48

[2] http://kamon.io/core/metrics/instruments/

[alon...@gmail.com]

Hello Ivan,

Thanks!

version 0.3.5
i have deployed the right config, with the only change
actor { includes = ["*"]

Statsd log shows mailbox size is there, but it is not been flushed to es or i don't know how to read it. am i missing something in the statsd's config?

18 Feb 13:37:22 - DEBUG: kamon.is-performance-srv.router.user_healthCheckService.errors:0|c
{"counters":{"statsd.bad_lines_seen":0,"statsd.packets_received":50,"statsd.metrics_received":610,"kamon.is-performance-srv.router.user_dataSetSer
vice.errors":0,"kamon.is-performance-srv.actor.user_healthCheckService_c.errors":0,"kamon.is-performance-srv.actor.user_schemaService_b.errors":0,
"kamon.is-performance-srv.actor.user_healthCheckService_a.errors":0,"kamon.is-performance-srv.actor.user_healthCheckService.errors":0,"kamon.is-pe
rformance-srv.router.user_healthCheckService.errors":0,"kamon.is-performance-srv.actor..errors":0,"kamon.is-performance-srv.actor.user_etlService_
a.errors":0,"kamon.is-performance-srv.actor.user_statsd-metrics-sender.errors":0,"kamon.is-performance-srv.actor.user_schemaService.errors":0,"kamon.is-performance-srv.router.user_schemaService.errors":0,"kamon.is-performance-srv.actor.user_dataSetService_b.errors":0,"kamon.is-performance-srv.actor.user_a.errors":0,"kamon.is-performance-srv.actor.user_dataCollectionService.errors":0,"kamon.is-performance-srv.router.user_dataCollectionService.errors":0,"kamon.is-performance-srv.actor.user_dataSetService_a.errors":0,"kamon.is-performance-srv.actor.user_schemaService_c.errors":0,"kamon.is-performance-srv.actor.user_dataCollectionService_a.errors":0,"kamon.is-performance-srv.actor.system.errors":0,"kamon.is-performance-srv.actor.user_etlService.errors":0,"kamon.is-performance-srv.router.user_etlService.errors":0,"kamon.is-performance-srv.actor.user_dataCollectionService_b.errors":0,"kamon.is-performance-srv.actor.user_kamon-metrics-subscriptions.errors":0,"kamon.is-performance-srv.actor.user_healthCheckService_b.errors":0,"kamon.is-performance-srv.actor.user_dataSetService_c.errors":0,"kamon.is-performance-srv.actor.user.errors":0,"kamon.is-performance-srv.actor.user_schemaService_a.errors":0,"kamon.is-performance-srv.actor.user_etlService_c.errors":0,"kamon.is-performance-srv.actor.user_dataCollectionService_c.errors":0,"kamon.is-performance-srv.actor.user_etlService_b.errors":0,"kamon.is-performance-srv.actor.user_dataSetService.errors":0},"gauges":{"statsd.timestamp_lag":0},"timers":{"kamon.is-performance-srv.actor.user_healthCheckService_c.mailbox-size":[0,0,0,0,0,0,0,0,0,0],"kamon.is-performance-srv.actor.user_schemaService_b.mailbox-size":[0,0,0,0,0,0,0,0,0,0],"kamon.is-performance-srv.actor.user_healthCheckService_a.mailbox-size":[0,0,0,0,0,0,0,0,0,0],"kamon.is-performance-srv.actor.user_healthCheckService.mailbox-size":[0,0,0,0,0,0,0,0,0,0],"kamon.is-performance-srv.actor..mailbox-size":[0,0,0,0,0,0,0,0,0,0],"kamon.is-performance-srv.actor.user_etlService_a.mailbox-size":[0,0,0,0,0,0,0,0,0,0],"kamon.is-performance-srv.actor.user_statsd-metrics-sender.processing-time":[1089536,1089536,1105920,1171456,1236992,1245184,1286144,1400832,4325376,4521984]

[alon...@gmail.com]

Ivan,

i see it goes to "Counters", is that correct or should it be under "timer" or "timer_data"?

[dgru...@gmail.com]

Hi, I am Working with Alon on this problem

We have two questions:

1. We have 5 name part for the data with mailbox size and I think the mapping there fail as it expect 4 parts , Is it true

2. In debug for StatsD we see the messages that succeed separated into fields like ns, grp, etc. But for the mailbox it just putting the whole message as one string in debug. I assume its a failure in mapping!

How can we control the parts of the data and make it four instead of five Or is there other mapping we can use

Thanks
Danny

[Ivan Topolnjak]

Hello guys,

There are a couple things I don't understand yet, please see me questions bellow:

Statsd log shows mailbox size is there, but it is not been flushed to es or i don't know how to read it. am i missing something in the statsd's config?

What is es? are you talking about some StatsD backend here?, it would be nice if you could give us a quick description of how your infrastructure looks like and what you want to achieve with Kamon so we can better direct our replies :).

1. We have 5 name part for the data with mailbox size and I think the mapping there fail as it expect 4 parts , Is it true

I don't get this question, could you please be a bit more specific? Currently the metric key generator for metrics in our StatsD module [1] transforms every metric in a 5-part metric with format 'application.host.entity.entity-name.metric-name' so, all metrics should already have 5 parts. In case you want to change the way in which metric keys are generated when sent to StatsD, then you can create your own implementation of `kamon.statsd.MetricKeyGenerator` and replace the default one in the configuration file [2].

2. In debug for StatsD we see the messages that succeed separated into fields like ns, grp, etc. But for the mailbox it just putting the whole message as one string in debug. I assume its a failure in mapping!

I'm not sure what you are referring to here :(... could you please explain in more detail?

Sorry that I can't be of much help to this moment, hope to get some more feedback from you so we can move forward, regards!

[1] https://github.com/kamon-io/Kamon/blob/v0.3.5/kamon-statsd/src/main/scala/kamon/statsd/SimpleMetricKeyGenerator.scala#L13

[2] https://github.com/kamon-io/Kamon/blob/v0.3.5/kamon-statsd/src/main/resources/reference.conf#L35

[alon...@gmail.com]

Hello Ivan,

es=elastic search, yes statsd backend. we are implementing some services with spray and want Kamon to collect actor related metrics. metrics sent to statsd and then we want a dashboard in Kibana to show it nicely.

i think metrics format is ok as per your explanation 'application.host.entity.entity-name.metric-name' here (we use 'kamon' application name just for test): kamon.is-performance-srv.actor.user_etlService_a.mailbox-size":[0,0,0,0,0,0,0,0,0,0]

just to clarify the question, in the statsd log i see entries as the following one:
"timers":{"kamon.is-performance-srv.actor..mailbox-size":[0,0,0,0,0,0,0,0,0,0],"kamon.is-performance-srv.actor.user_statsd-metrics-sender.processing-time":[892928,974848,1097728,1130496,1163264,1269760,1441792,1458176,1548288,1630208],"kamon.is-performance-srv.actor.user_statsd-metrics-sender.mailbox-size":[0,0,0,0,0,0,0,0,0,0],"kamon.is-performance-srv.actor.user_statsd-metrics-sender.time-in-mailbox":[24576,25600,25728,30208,30976,35840,39936,40448,52480,52736]
here kamon.is-performance-srv.actor..mailbox-size entity-name is missing. is that ok?
Any 'mailbox' related metrics never got into elastic.

i hope i'm giving here enough info, please let know otherwise.

Thanks a lot

[Ivan Topolnjak]

Alon,

thanks for the feedback! You found a small oddity/bug there: Kamon is incorrectly always registering the root guardian [1]! That actor has an empty name and that's why the `kamon.is-performance-srv.actor..mailbox-size` looks like it is. Sorry about that, you can now workaround that by adding a "" exclude in your config file and we will take care of that for the next release [2].

Getting past that issue, there is very little I can do to troubleshoot why the metrics are not getting into elasticsearch because we have never experimented with a elasticsearch backend for StatsD :(... if you are in the mood for experimenting you can take a look at our StatsD/Grafana/Graphite docker image [3] which we know much better and can probably be of more help if you have any issue. 

I think that answers your question on the wrong metric name, please let us know if you need any further assistance, regards!

[1] http://doc.akka.io/docs/akka/2.3.9/general/supervision.html#The_Top-Level_Supervisors

[2] https://github.com/kamon-io/Kamon/issues/157

[3] https://github.com/kamon-io/docker-grafana-graphite

--
You received this message because you are subscribed to the Google Groups "kamon-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kamon-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[alon...@gmail.com]

> To unsubscribe from this group and stop receiving emails from it, send an email to kamon-user+...@googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.

Ivan,

"" exclude did the magic, so i don't get the empty name any more.

I'm going to explore Grafana option.

Thank you

[alon...@gmail.com]

Hello Ivan,

switched to https://github.com/kamon-io/docker-grafana-graphite, but i only see test graph and not Actor or Trace Metrics. Is there is something i missed in configuration?

Thanks,
Alon

[alon...@gmail.com]

And under Dashboard Settings/Metrics i see
Load metrics keys into elastic search
Work in progress...
Which never complete

[Ivan Topolnjak]

Hello Alon,

I'm sorry, there was a problem with the image available in the docker registry and a few minutes ago I was finally able to push the latest version we have. Please, do a `docker pull kamon/grafana_graphite` to get the latest version and launch the container again. Currently, there is a 20 seconds delay after the container is started for the dashboards to be loaded so please, launch the container, give at least 20 seconds and then go to your browser and it should take you to the actor/traces dashboard as shown in the StatsD section on our site. 

Please let us know how it goes for you, regards!

[alon...@gmail.com]

I'm seeing default Grafana dashboard but not the Kamon dashboard
Used docker on CentOS 6.6
What could be wrong?

[Ivan Topolnjak]

Did you just refresh your browser or went to http://localhost/ and let it redirect you to the right place?, also.. if you run `docker images` are you seeing this same output for the grafana_graphite image?

kamon/grafana_graphite   latest              9fed1a92d4a2        5 hours ago         917.8 MB

[alon...@gmail.com]

missed that one. will test and let know

Thanks!

[alon...@gmail.com]

Hi Ivan,

Kamon dashboard is there and it looks great!
Thank you!

Alon