Handle Unable to register metrics as an old set with the same name exists

4,512 views
Skip to first unread message

Bhupendra Baraiya

unread,
Jul 22, 2022, 2:29:53 AM7/22/22
to debezium
We are running Debezium with Kafka in kafka distributed mode.
We were able to extract Debezium metrics to Prometheus when we were using only single worker.

Since Kafka is distributed mode allows to run multiple workers, we started pulling multiple tables in parallel using Debezium and it worked fine but we started getting warnings (refer subject) followed by error (after completing 12 attempts) when it comes to capturing Debezium metrics.

To our surprise, JMX only capture metrics of first worker and not for other workers (Jconsole shows only the first kafka distributed worker execution).
This will prevent us to optimally use kafka because Debezium metrics are must for us.

We are running below command in our connect-distrubuted.sh file to trigger jmx

exec $(dirname $0)/kafka-run-class.sh $EXTRA_ARGS -javaagent:/opt/jmx/jmx-exporter.jar=7071:/opt/jmx/config.yml org.apache.kafka.connect.cli.ConnectDistributed "$@"

Request you to kindly guide us on how can we trigger JMX exporter for each kafka worker execution so that we can capture metrics for each worker

Chris Cranford

unread,
Jul 22, 2022, 8:48:12 AM7/22/22
to debe...@googlegroups.com, Bhupendra Baraiya
Hi Bhupendra -

This sounds like you've defined the same value for the "database.server.name" in each of your connector deployments.  This connector property must be unique across all connectors and if it isn't, then this explains why you're only seeing metrics for the first connector that is deployed. 

Chris
--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/f18d06d6-1af3-4519-b7ba-73fcdb9b9409n%40googlegroups.com.

Bhupendra Baraiya

unread,
Jul 22, 2022, 8:58:34 AM7/22/22
to Chris Cranford, debe...@googlegroups.com
Hi Chris,

Database server name will remain same right because I am pulling data from the same server and database 

Chris Cranford

unread,
Jul 22, 2022, 9:17:18 AM7/22/22
to Bhupendra Baraiya, debe...@googlegroups.com
Hi Bhupendra

No, unfortunately that's inaccurate.  The documentation is clear that the "database.server.name" is a logical name assignment and that it must be unique among all connectors.  Please see the connector properties for your connector [1] for information on the restrictions and implications for needing to change this after the fact.

Chris

[1]: https://debezium.io/documentation/reference/stable/connectors/index.html

Bhupendra Baraiya

unread,
Jul 22, 2022, 11:18:19 AM7/22/22
to debezium
Hi Chris,

Thanks a lot for your help, it worked fine.
But I didn't understand how  "database.server.name" property can cause Mbean error.
Till now I was under impression that it's actual database server name

Chris Cranford

unread,
Jul 22, 2022, 12:21:36 PM7/22/22
to debe...@googlegroups.com, Bhupendra Baraiya
Hi Bhupendra

The "database.server.name" is a logical designation and it is part of the object-name when the MBean is created.  This means that if you deploy multiple connectors for the same connector type, i.e. mysql, then the names of MBeans will be the same and this creates the uniqueness problem you faced.  This configuration is also used to prefix topics in Kafka as well.

Hope that makes it clear.

CC

Bhupendra Baraiya

unread,
Jul 22, 2022, 2:01:43 PM7/22/22
to Chris Cranford, debe...@googlegroups.com
Yes thank u again 

Shreeram Goyal

unread,
Jan 27, 2023, 3:55:37 AM1/27/23
to debezium
Hi Chris,
I had a doubt regarding this. I tried to capture multiple tables with a single connector and on grafana, all the table metrics had same display name. So should we always have different connectors for different tables for getting seperate metrics for them?

Chris Cranford

unread,
Jan 30, 2023, 8:13:00 AM1/30/23
to debe...@googlegroups.com
Hi Shreeram,

The Debezium connectors do not provide per table metrics, but instead they provide per connector metrics.  So if a single connector captures multiple tables from the source database, then the metrics you capture from that connector will be cumulative across all tables captured.  If you need metrics per table, then yes you would need to deploy a single connector per table, but this can have some performance costs on the source database depending on what the source is and how many tables we're talking about.

So perhaps it's worthwhile for us to have a discussion specifically on what you're trying to track per-table here in Grafana.  Can you share some detail on this.

Thanks,
Chris

TIMES PRIME Member’s FEST is LIVE!

15 Days 15 Gifts. Year-long OTTs. New Brand Launches & Exclusive events. Claim your complimentary gifts everyday from 23rd Jan - 6th Feb here!

"The best way to find yourself is to lose yourself in the service of others." - Mahatma Gandhi

__________________________________________________________________________

VCO Inspiration

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.

Shreeram Goyal

unread,
Feb 8, 2023, 2:58:22 AM2/8/23
to debezium
Hi Chris,
Thanks for the information. I am trying to have multiple connectors specifically for capturing table level metrics. Our source db is MariaDb. I had another issue other than this which I found while capturing metrics. I saw that the metric 'TotalNumberOfEventsSeen' increases with a number greater than actual change. For example, if I run a single update query on a table which is being captured by  4-5 connectors, I see random increase in 'TotalNumberOfEventsSeen' and not 1. Sometimes, it's 5 or sometime it is 7. Can you please clarify on this if I am missing something or this is an incorrect behaviour.

Thanks,
Shreeram

Shreeram Goyal

unread,
Feb 8, 2023, 3:06:28 AM2/8/23
to debezium
Infact the TotalNumberOfEventsSeen keep increasing after sometime even when there is no query being run on the table. That is quite strange. 

Chris Cranford

unread,
Feb 8, 2023, 10:22:26 AM2/8/23
to debe...@googlegroups.com
Hi Shreeram -

This metric can be incremented not only for INSERT, UPDATE, or DELETE events, but also for DDL changes as well as in the case where an event could not be properly deserialized.  I would suggest enabling DEBUG logging and look for "Received query command: ..." log entries.  Chances are you have some schema changes happening that are attributing to the metric's increase.

Hope that helps.
Chris

"The most wasted of all days is one without laughter." - Nicolas Chamfort

__________________________________________________________________________

VCO Inspiration

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.

Ilona

unread,
Sep 6, 2023, 4:29:01 AM9/6/23
to debezium
I have a similar problem, I deploy a cluster on AWS  MSK Kafka and when I create a connector based on debezium-debezium-connector-mysql-2.2.1 - I get this error, but the biggest problem is that I have only one connector with one "database. server.name" parameter and there is no such thing anywhere else:
ERROR [MySQL-Connector-QA|task-0] Failed to register metrics MBean, metrics will not be available (io.debezium.metrics.Metrics:99)

среда, 8 февраля 2023 г. в 17:22:26 UTC+2, Chris Cranford:

jiri.p...@gmail.com

unread,
Sep 6, 2023, 6:45:04 AM9/6/23
to debezium
Please check JMX first before you deploy the connector just in case

J.

Siyuan Wang

unread,
Nov 15, 2023, 5:41:04 AM11/15/23
to debezium
Hi, is there any way to disable the JMX/MBean stuffs? I don't want this feature and just want to hardcode the "database.server.name" to a constant value across multiple connectors.

jiri.p...@gmail.com

unread,
Nov 15, 2023, 6:42:52 AM11/15/23
to debezium
Hi,

unfortunately this is not possible. You could configure router SMT for all your connectors to route to the same topic namespace.

Jiri

Reply all
Reply to author
Forward
0 new messages