Standard Micrometer Metrics cause eventbus to stall

[Yanick Salzmann]

Hi all

When using Micrometer Metrics I can realize that my event bus is starting to have troubles delivering events. I normally send thousands of events a second over the bus, but when I enable metrics i think it starts to deadlock (sometimes taking over 30 seconds to reply to a request - i guess more, it just timeouts).

So, how do I know it is related to Micrometer Metrics? The first thing is that simply changing this to false fixes it:

@Bean
MetricsOptions metricsOptions() {
   return new MicrometerMetricsOptions()
         .setEnabled(true);
}

The second clue is that the thread health check realizes that the tread is blocked and dumps a stack trace like this:

io.vertx.core.VertxException: Thread blocked

at io.micrometer.core.instrument.MeterRegistry.getOrCreateMeter(MeterRegistry.java:557) ~[micrometer-core-1.0.0.jar:1.0.0]

at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:537) ~[micrometer-core-1.0.0.jar:1.0.0]

at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:512) ~[micrometer-core-1.0.0.jar:1.0.0]

at io.micrometer.core.instrument.MeterRegistry.gauge(MeterRegistry.java:242) ~[micrometer-core-1.0.0.jar:1.0.0]

at io.micrometer.core.instrument.Gauge$Builder.register(Gauge.java:128) ~[micrometer-core-1.0.0.jar:1.0.0]

at io.vertx.micrometer.impl.meters.Gauges.get(Gauges.java:61) ~[vertx-micrometer-metrics-3.5.4.jar:3.5.4]

at io.vertx.micrometer.impl.VertxEventBusMetrics.handlerRegistered(VertxEventBusMetrics.java:62) ~[vertx-micrometer-metrics-3.5.4.jar:3.5.4]

at io.vertx.micrometer.impl.VertxEventBusMetrics.handlerRegistered(VertxEventBusMetrics.java:32) ~[vertx-micrometer-metrics-3.5.4.jar:3.5.4]

at io.vertx.core.eventbus.impl.HandlerRegistration.setResult(HandlerRegistration.java:173) ~[vertx-core-3.5.4.jar:3.5.4]

The location where it is hanging is:

private Meter getOrCreateMeter(@Nullable DistributionStatisticConfig config,
                               BiFunction<Id, /*Nullable Generic*/ DistributionStatisticConfig, Meter> builder,
                               Id mappedId, Function<Meter.Id, ? extends Meter> noopBuilder) {
    Meter m = meterMap.get(mappedId);

    if (m == null) {
        if (isClosed()) {
            return noopBuilder.apply(mappedId);
        }

     --->>   synchronized (meterMapLock) {

From what I understand it keeps creating the same gauge over and over again which is strange, because I would have expected that it gets created once it went over the function. One other explanation would be that creating the meter later fails so it wont be added, however I have created a conditional breakpoint and the meter is never null and successfully created.

Does anyone know what causes this issue?

BR

Yanick

[jtak...@redhat.com]

Hi Yanick,

It could be interesting to know what the "mappedId" passed to "getOrCreateMeter" looks like. Maybe it's different for each call, which would explain why it creates new gauge every time.

Especially, in the current release of the vertx micrometer module (3.5.x), controlling the labels cardinality is entirely up to the user. So if the event bus address changes a lot, it's going to affect metrics labels and generate new "ids".

See here about how to control it: https://vertx.io/docs/vertx-micrometer-metrics/java/#_label_matchers

Example, to ignore the "address" label, metrics options should be configured as such:

new MicrometerMetricsOptions()
         .addLabelMatch(new Match()
      .setLabel("address")
      .setType(MatchType.REGEX)
      .setValue(".*")
      .setAlias("_"))
         .setEnabled(true);

Note that, this will change in vertx 3.6, as labels with potentially high values cardinality will be ignored by default, hence including eventbus "address".

Please let me know if it fixes the problem

Joel