Handle Unable to register metrics as an old set with the same name exists

[Bhupendra Baraiya]

We are running Debezium with Kafka in kafka distributed mode.

We were able to extract Debezium metrics to Prometheus when we were using only single worker.

Since Kafka is distributed mode allows to run multiple workers, we started pulling multiple tables in parallel using Debezium and it worked fine but we started getting warnings (refer subject) followed by error (after completing 12 attempts) when it comes to capturing Debezium metrics.

To our surprise, JMX only capture metrics of first worker and not for other workers (Jconsole shows only the first kafka distributed worker execution).

This will prevent us to optimally use kafka because Debezium metrics are must for us.

We are running below command in our connect-distrubuted.sh file to trigger jmx

exec $(dirname $0)/kafka-run-class.sh $EXTRA_ARGS -javaagent:/opt/jmx/jmx-exporter.jar=7071:/opt/jmx/config.yml org.apache.kafka.connect.cli.ConnectDistributed "$@"

Request you to kindly guide us on how can we trigger JMX exporter for each kafka worker execution so that we can capture metrics for each worker

[Chris Cranford]

Hi Bhupendra -

This sounds like you've defined the same value for the "database.server.name" in each of your connector deployments.  This connector property must be unique across all connectors and if it isn't, then this explains why you're only seeing metrics for the first connector that is deployed. 

Chris

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/f18d06d6-1af3-4519-b7ba-73fcdb9b9409n%40googlegroups.com.

[Bhupendra Baraiya]

Hi Chris,

Database server name will remain same right because I am pulling data from the same server and database 

[Chris Cranford]

Hi Bhupendra

No, unfortunately that's inaccurate.  The documentation is clear that the "database.server.name" is a logical name assignment and that it must be unique among all connectors.  Please see the connector properties for your connector [1] for information on the restrictions and implications for needing to change this after the fact.

Chris

[1]: https://debezium.io/documentation/reference/stable/connectors/index.html

[Bhupendra Baraiya]

Hi Chris,

Thanks a lot for your help, it worked fine.

But I didn't understand how  "database.server.name" property can cause Mbean error.

Till now I was under impression that it's actual database server name

[Chris Cranford]

Hi Bhupendra

The "database.server.name" is a logical designation and it is part of the object-name when the MBean is created.  This means that if you deploy multiple connectors for the same connector type, i.e. mysql, then the names of MBeans will be the same and this creates the uniqueness problem you faced.  This configuration is also used to prefix topics in Kafka as well.

Hope that makes it clear.

CC

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/34f0930f-e8df-4044-8f41-927493d8f56en%40googlegroups.com.

[Bhupendra Baraiya]

Yes thank u again 

[Shreeram Goyal]

Hi Chris,
I had a doubt regarding this. I tried to capture multiple tables with a single connector and on grafana, all the table metrics had same display name. So should we always have different connectors for different tables for getting seperate metrics for them?

[Chris Cranford]

Hi Shreeram,

The Debezium connectors do not provide per table metrics, but instead they provide per connector metrics.  So if a single connector captures multiple tables from the source database, then the metrics you capture from that connector will be cumulative across all tables captured.  If you need metrics per table, then yes you would need to deploy a single connector per table, but this can have some performance costs on the source database depending on what the source is and how many tables we're talking about.

So perhaps it's worthwhile for us to have a discussion specifically on what you're trying to track per-table here in Grafana.  Can you share some detail on this.

Thanks,
Chris

TIMES PRIME Member’s FEST is LIVE!

15 Days 15 Gifts. Year-long OTTs. New Brand Launches & Exclusive events. Claim your complimentary gifts everyday from 23rd Jan - 6th Feb here!

"The best way to find yourself is to lose yourself in the service of others." - Mahatma Gandhi

__________________________________________________________________________

VCO Inspiration

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/8360ea9f-42e9-47ac-a64b-86f578c88a0fn%40googlegroups.com.

[Shreeram Goyal]

Hi Chris,
Thanks for the information. I am trying to have multiple connectors specifically for capturing table level metrics. Our source db is MariaDb. I had another issue other than this which I found while capturing metrics. I saw that the metric 'TotalNumberOfEventsSeen' increases with a number greater than actual change. For example, if I run a single update query on a table which is being captured by  4-5 connectors, I see random increase in 'TotalNumberOfEventsSeen' and not 1. Sometimes, it's 5 or sometime it is 7. Can you please clarify on this if I am missing something or this is an incorrect behaviour.

Thanks,

Shreeram

[Shreeram Goyal]

Infact the TotalNumberOfEventsSeen keep increasing after sometime even when there is no query being run on the table. That is quite strange. 

[Chris Cranford]

Hi Shreeram -

This metric can be incremented not only for INSERT, UPDATE, or DELETE events, but also for DDL changes as well as in the case where an event could not be properly deserialized.  I would suggest enabling DEBUG logging and look for "Received query command: ..." log entries.  Chances are you have some schema changes happening that are attributing to the metric's increase.

Hope that helps.
Chris

"The most wasted of all days is one without laughter." - Nicolas Chamfort

__________________________________________________________________________

VCO Inspiration

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/9d65ae00-6313-4304-abb4-3c04be348dc7n%40googlegroups.com.

[Ilona]

I have a similar problem, I deploy a cluster on AWS  MSK Kafka and when I create a connector based on debezium-debezium-connector-mysql-2.2.1 - I get this error, but the biggest problem is that I have only one connector with one "database. server.name" parameter and there is no such thing anywhere else:

ERROR [MySQL-Connector-QA|task-0] Failed to register metrics MBean, metrics will not be available (io.debezium.metrics.Metrics:99)

среда, 8 февраля 2023 г. в 17:22:26 UTC+2, Chris Cranford:

[jiri.p...@gmail.com]

Please check JMX first before you deploy the connector just in case

J.

[Siyuan Wang]

Hi, is there any way to disable the JMX/MBean stuffs? I don't want this feature and just want to hardcode the "database.server.name" to a constant value across multiple connectors.

[jiri.p...@gmail.com]

Hi,

unfortunately this is not possible. You could configure router SMT for all your connectors to route to the same topic namespace.

Jiri