Prometheus mysqld_exporter failing with errors.

23 views
Skip to first unread message

Sai Krishna

unread,
Jan 13, 2022, 3:18:06 PMJan 13
to Prometheus Users
Hello everyone, 

Would appreciate any quick assistance in below error. I have upgraded my exporter from 0.12.0 to 0.13.0 but this did not help. Did someone fixed this before? 

6 error(s) occurred:
* [from Gatherer #2] collected metric "mysql_global_variables_validate_password_check_user_name" { gauge:<value:1 > } was collected before with the same name and label values


- Sai

Brian Candler

unread,
Jan 13, 2022, 3:36:25 PMJan 13
to Prometheus Users
It says that your scrape job is seeing the same metric multiple times with the same labels.

Scrape your exporter directly with curl, and I'm guessing you'll see

mysql_global_variables_validate_password_check_user_name 1
...
mysql_global_variables_validate_password_check_user_name 1

or

mysql_global_variables_validate_password_check_user_name{foo="abc"} 1
...
mysql_global_variables_validate_password_check_user_name{foo="abc"} 1

Either that, or your scrape job is scraping the same exporter multiple times.

Matthias Rampke

unread,
Jan 13, 2022, 3:52:04 PMJan 13
to Brian Candler, Prometheus Users
This may be caused by the mysqld exporter invalidly conflating multiple variables.

Can you please share what your MySQL version is, and what you get from

SHOW GLOBAL VARIABLES LIKE 'validate_password%';

From a quick read, it seems that there is both a plugin, using configuration variables like validate_password_check_user_name, and (as of MySQL 8), a component, using variables like validate_password.check_user_name. The exporter might normalize both to the same metric name, causing the issue.

If that is the case, the SHOW GLOBAL VARIABLES will reveal it. Remove the duplicate (plugin) configuration as per the documentation [1]

/MR


--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/dc4c8d69-bc6e-4a3f-95bc-0b0d950048bcn%40googlegroups.com.

Sai Krishna

unread,
Jan 14, 2022, 2:35:42 AMJan 14
to Prometheus Users

Thanks Matt and Brain,

I'm seeing this in MySQL 8.0.21 and 8.0.25. Out of 100 servers, only 4 servers giving this error. 

An error has occurred while serving metrics: 6 error(s) occurred: * [from Gatherer #2] collected metric "mysql_global_variables_validate_password_check_user_name" { gauge:<value:1 > } was collected before with the same name and label values * [from Gatherer #2] collected metric "mysql_global_variables_validate_password_length" { gauge:<value:8 > } was collected before with the same name and label values * [from Gatherer #2] collected metric "mysql_global_variables_validate_password_mixed_case_count" { gauge:<value:1 > } was collected before with the same name and label values * [from Gatherer #2] collected metric "mysql_global_variables_validate_password_number_count" { gauge:<value:1 > } was collected before with the same name and label values * [from Gatherer #2] collected metric "mysql_global_variables_validate_password_special_char_count" { gauge:<value:1 > } was collected before with the same name and label values * [from Gatherer #2] collected metric "mysql_global_status_validate_password_dictionary_file_words_count" { untyped:<value:0 > } was collected before with the same name and label values

Sai Krishna

unread,
Jan 14, 2022, 4:59:06 AMJan 14
to Prometheus Users

SHOW GLOBAL VARIABLES LIKE 'validate_password%';
+--------------------------------------+--------+
| Variable_name                        | Value  |
+--------------------------------------+--------+
| validate_password.check_user_name    | ON     |
| validate_password.dictionary_file    |        |
| validate_password.length             | 8      |
| validate_password.mixed_case_count   | 1      |
| validate_password.number_count       | 1      |
| validate_password.policy             | MEDIUM |
| validate_password.special_char_count | 1      |
| validate_password_check_user_name    | ON     |
| validate_password_dictionary_file    |        |
| validate_password_length             | 8      |
| validate_password_mixed_case_count   | 1      |
| validate_password_number_count       | 1      |
| validate_password_policy             | MEDIUM |
| validate_password_special_char_count | 1      |
+--------------------------------------+--------+

Sai Krishna

unread,
Jan 14, 2022, 5:00:18 AMJan 14
to Prometheus Users
Its weird that all other exporters with same versions working fine. 

Brian Candler

unread,
Jan 14, 2022, 8:26:45 AMJan 14
to Prometheus Users
You can see you have both "validate_password.check_user_name" and "validate_password_check_user_name".  That's the problem.

Metric names cannot have dots, so "validate_password.check_user_name"  is exposed as "validate_password_check_user_name", and duplicates the real "validate_password_check_user_name"

Sai Krishna

unread,
Jan 14, 2022, 8:31:33 AMJan 14
to Brian Candler, Prometheus Users
Is it possible to ignore from Prometheus exporter config.

Reply all
Reply to author
Forward
0 new messages