Monitoring Druid's performance (metrics) with Kafka?

[Avinoam Ben Nachum]

Hello dear Druid Users.

We have recently decided to upgrade our cluster performance monitoring from Tranquility to Kafka, but with no success. I personally feel like I am missing some basic knowledge regarding how the overall setup should look with Kafka.

Our old setup was a Tranquility server, started up with a spec (.json) for the metrics dataSource.

the Tranquility server would then send realtime tasks to our middleManagers and the Metrics dataSource would be filled with data, which we would then visualize with superset.

(this is already half magical to me...)

Now, I am trying to replace the Tranquility server with a Kafka server, but can't find what I am literally suppose to do. whats the steps?

I followed the docs at here, got a supervisor going (presented on the overlord GUI) and lots of "index_kafka_metrics" tasks. no idea what actuly happened in the background, and the Metrics dataSource did not update.

I tried the emitter option here, I've set it on a single broker in the cluster. all I got was this in the logs:
```

Skipping druid.emitter.kafka.metric.topic property: one of it's prefixes is also used as a property key. Prefix: druid
Skipping druid.emitter.kafka.alert.topic property: one of it's prefixes is also used as a property key. Prefix: druid
Skipping druid.emitter.kafka.bootstrap.servers property: one of it's prefixes is also used as a property key. Prefix: druid
Skipping druid.emitter.kafka.producer.config property: one of it's prefixes is also used as a property key. Prefix: druid
```

I dont know if the docs are lacking the pure basics of what needs to be done to get the Metrics dataSource up and filling, or if I'm clouded by spending to much time on this without really understanding it.

A helping hand will be super appreciated!

let me know what information should I share.

best regards,

Avinoam

[Peter Marshall]

Looks like it's the same as this?

https://github.com/apache/druid/issues/5876

I'm not sure Avinoam but maybe you're using "druid" as your target topic name and this may be reserved perhaps???

Let me know as I will ask around if that's not the case :)

[Avinoam Ben Nachum]

Hi Peter,

Thank you for your reply!

My druid.emitter.kafka.alert.topic was set to "metrics", it looked like this under a broker's runtime.properties:

```

druid.emitter.kafka.bootstrap.servers=my-kafka-server:custom_port

druid.emitter.kafka.metric.topic=metrics

druid.emitter.kafka.alert.topic=druid-alert

druid.emitter.kafka.producer.config={"max.block.ms":10000}

```

I have stumbled on the link you shared before opening up this ticket here, as the only answer there is very un-helpful.

and the log is unclear...

But is my understanding of this is correct?

i need to enable metrics on my brokers, enable emitters, set emitters to be sent to the kafka server, the kafka server will  ingest data into the metrics dataSource?

Thank you kindly!

Avinoam

[Peter Marshall]

My understanding also - though I've never set that up myself...

Did what you did work?

[Avinoam Ben Nachum]

Unfortunately no, had to roll back to Tranquility.

I am not sure what am I missing... if i am missing key background knowledge in something, or reading the docs wrong.

[Peter Marshall]

So I belieeeeeve the flow is:

Druid Metrics Emitter --> Kafka --> Druid Ingestion Supervisor

If you have the emitter configured, sending to a Kafka topic, check with the console consumer that there are messages in there.

Then if there are, maybe just use the Console "Load Data" page to connect to the same topic from inside Druid using the wizard.

These two should at least isolate if it's a Druid-to-Kafka issue or a Druid Emitter-to-Kafka issue.

--
You received this message because you are subscribed to a topic in the Google Groups "Druid User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-user/SJpkZ6L60nA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/8dad486e-f9b6-402b-a500-0b1da992db4cn%40googlegroups.com.

--

Peter Marshall

Apache Druid® Community Technology Evangelist

+447813618895

peter.m...@imply.io

https://www.imply.io