only 4 metrics show in Metrics explorer

836 views
Skip to first unread message

Salam Elias

unread,
Jan 20, 2022, 6:44:06 PM1/20/22
to Prometheus Users
I have just successful setup and run mysql exporter, when I switch to Prometheus UI, try to search for sql metrics, I see only 4 items

mysql_exporter_last_scrape_error
mysql_exporter_scrapes_total
mysql_up
mysqld_exporter_build_info

in spite of the fact that I created a service file on Fedora 35 with all following flags
********************************************
[client]
user=mysqld_exporter
password=myuser

[Unit]
Description=Prometheus MySQL Exporter
After=network.target
User=prometheus
Group=prometheus

[Service]
Type=simple
Restart=always
ExecStart=/usr/local/bin/mysqld_exporter \
--config.my-cnf /etc/.mysqld_exporter.cnf \
--collect.global_status \
--collect.info_schema.innodb_metrics \
--collect.auto_increment.columns \
--collect.info_schema.processlist \
--collect.binlog_size \
--collect.info_schema.tables.databases \
--collect.info_schema.tablestats \
--collect.global_variables \
--collect.info_schema.query_response_time \
--collect.info_schema.userstats \
--collect.info_schema.tables \
--collect.perf_schema.tablelocks \
--collect.perf_schema.file_events \
--collect.perf_schema.eventswaits \
--collect.perf_schema.indexiowaits \
--collect.perf_schema.tableiowaits \
--collect.perf_schema.memory_events \
--collect.slave_status \
--web.listen-address=192.168.1.48:9104

[Install]
WantedBy=multi-user.target

***********************************************

It starts succesfuly by running the service status in green and no errors. Thanks for your help

Brian Candler

unread,
Jan 21, 2022, 3:04:36 AM1/21/22
to Prometheus Users
What do you get from:


?

Salam Elias

unread,
Jan 21, 2022, 4:20:35 AM1/21/22
to Prometheus Users
Hi, here is what I get

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.9799e-05
go_gc_duration_seconds{quantile="0.25"} 0.000104399
go_gc_duration_seconds{quantile="0.5"} 0.000155999
go_gc_duration_seconds{quantile="0.75"} 0.000194999
go_gc_duration_seconds{quantile="1"} 0.39236267
go_gc_duration_seconds_sum 0.435731728
go_gc_duration_seconds_count 213
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 9
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.16.4"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.477192e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 3.92554464e+08
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.478224e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 1.887392e+06
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 2.0797628174909467e-05
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 5.082024e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.477192e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.28736e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 3.743744e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 4029
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 6.2078976e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.6617344e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.6427552726178036e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 1.891421e+06
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 2400
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 52224
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 65536
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.574928e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 518672
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 491520
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 491520
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.4269704e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 6
# HELP mysql_exporter_last_scrape_error Whether the last scrape of metrics from MySQL resulted in an error (1 for error, 0 for success).
# TYPE mysql_exporter_last_scrape_error gauge
mysql_exporter_last_scrape_error 1
# HELP mysql_exporter_scrapes_total Total number of times MySQL was scraped for metrics.
# TYPE mysql_exporter_scrapes_total counter
mysql_exporter_scrapes_total 1766
# HELP mysql_up Whether the MySQL server is up.
# TYPE mysql_up gauge
mysql_up 0
# HELP mysqld_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which mysqld_exporter was built.
# TYPE mysqld_exporter_build_info gauge
mysqld_exporter_build_info{branch="HEAD",goversion="go1.16.4",revision="ad2847c7fa67b9debafccd5a08bacb12fc9031f1",version="0.13.0"} 1
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 7.76
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 10
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.4118912e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.64272887993e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.3238528e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 1765
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

Brian Candler

unread,
Jan 21, 2022, 4:40:52 AM1/21/22
to Prometheus Users
On Friday, 21 January 2022 at 09:20:35 UTC sa...@salam.hd.free.fr wrote:
# HELP mysql_up Whether the MySQL server is up.
# TYPE mysql_up gauge
mysql_up 0

There's your answer.  The exporter is failing to talk to mysql.  It could be that the configuration in /etc/.mysqld_exporter.cnf is wrong, or mysql isn't accepting connections from the exporter's IP address.

Try looking at exporter logs.  Since you're running it under systemd then use "journalctl -eu mysql_exporter" or similar, depending on what you named the service file.

Salam Elias

unread,
Jan 21, 2022, 5:38:20 AM1/21/22
to Prometheus Users
You are correct, there is an issu coonnecting to SQL, I am not sure why it is trying to use [::1]. Anyway, I granted the user on this ipv6 but still not able to connect
CREATE USER 'mysqld_exporter'@'::1' IDENTIFIED BY 'mysqld_exporter' WITH MAX_USER_CONNECTIONS 2;

restarted the services but still getting

Jan 21 11:21:47 puppetmaster29 mysqld_exporter[3863944]: level=error ts=2022-01-21T10:21:47.619Z caller=exporter.go:149 msg="Error pinging mysqld" err="dial tcp [::1]:3306: c>
Jan 21 11:22:02 puppetmaster29 mysqld_exporter[3863944]: level=error ts=2022-01-21T10:22:02.619Z caller=exporter.go:149 msg="Error pinging mysqld" err="dial tcp [::1]:3306: c>
Jan 21 11:22:17 puppetmaster29 mysqld_exporter[3863944]: level=error ts=2022-01-21T10:22:17.619Z caller=exporter.go:149 msg="Error pinging mysqld" err="dial tcp [::1]:3306: c>

On Friday, January 21, 2022 at 9:04:36 AM UTC+1 Brian Candler wrote:

Brian Candler

unread,
Jan 21, 2022, 6:42:51 AM1/21/22
to Prometheus Users
What do these show?

telnet ::1 3306
telnet localhost 3306

The requirements for enabling mysql to accept IPv6 connections are here:

Have you set bind_address=:: ?

As for why it's using IPv6, my guess is that your /etc/hosts has something like this:

127.0.0.1 localhost
::1 localhost

One workaround is to change your exporter config to point to "127.0.0.1" instead of "localhost".  Another workaround is to change /etc/hosts to

127.0.0.1 localhost
::1 localhost6

Salam Elias

unread,
Jan 21, 2022, 8:47:01 AM1/21/22
to Prometheus Users
I gave all IPS and disabled localhost in hosts by commetning ::1 but still getting the same error


User        Hosts        Certificate        Permissions


mysqld_exporter        localhost        None        Select table data | Manage processes | Client replication
mysqld_exporter        ::1        None        Select table data | Manage processes | Client replication
mysqld_exporter        192.168.3.100        None        Select table data | Manage processes | Client replication
mysqld_exporter        puppetmaster29        None        Select table data | Manage processes | Client replication
mysqld_exporter        Any        None        Select table data | Manage processes | Client replication
mysqld_exporter        127.0.0.1        None        Select table data | Manage processes | Client replication

Salam Elias

unread,
Jan 21, 2022, 8:47:01 AM1/21/22
to Prometheus Users
No the bind-address is set to my IPv4 for MariaDB

Salam Elias

unread,
Jan 21, 2022, 8:47:01 AM1/21/22
to Prometheus Users
Ok, you say, " change your exporter config to point to "127.0.0.1" instead of "localhost". ", where in which file? and what exactly should be written? I visited the github nothing mentioned about pointing exporter to a specific IP

Salam Elias

unread,
Jan 21, 2022, 8:47:01 AM1/21/22
to Prometheus Users

I did
grant super on *.* to exporter@localhost;
restarted exporter and hop it workd. So why it needs super privileges which indicated nowhere!!!!

Salam Elias

unread,
Jan 21, 2022, 8:47:01 AM1/21/22
to Prometheus Users
I changed BindAdree to 0.0.0.0, restarted exporter, started getting

Jan 21 13:20:47 puppetmaster29 mysqld_exporter[120483]: level=error ts=2022-01-21T12:20:47.712Z caller=exporter.go:174 msg="Error from scraper" scraper=slave_status err="Error 1227: Access denied; you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation"
Jan 21 13:21:02 puppetmaster29 mysqld_exporter[120483]: level=error ts=2022-01-21T12:21:02.688Z caller=exporter.go:174 msg="Error from scraper" scraper=slave_status err="Error 1227: Access denied; you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation"
Jan 21 13:21:17 puppetmaster29 mysqld_exporter[120483]: level=error ts=2022-01-21T12:21:17.684Z caller=exporter.go:174 msg="Error from scraper" scraper=slave_status err="Error 1227: Access denied; you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation"
Jan 21 13:21:32 puppetmaster29 mysqld_exporter[120483]: level=error ts=2022-01-21T12:21:32.679Z caller=exporter.go:174 msg="Error from scraper" scraper=slave_status err="Error 1227: Access denied; you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation"
Jan 21 13:21:47 puppetmaster29 mysqld_exporter[120483]: level=error ts=2022-01-21T12:21:47.684Z caller=exporter.go:174 msg="Error from scraper" scraper=slave_status err="Error 1227: Access denied; you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation"
Jan 21 13:22:02 puppetmaster29 mysqld_exporter[120483]: level=error ts=2022-01-21T12:22:02.687Z caller=exporter.go:174 msg="Error from scraper" scraper=slave_status err="Error 1227: Access denied; you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation"

Brian Candler

unread,
Jan 21, 2022, 10:58:21 AM1/21/22
to Prometheus Users
I would guess it's doing something like "show slave status" and needs appropriate permissions.

The required grants are documented here:

Salam Elias

unread,
Jan 21, 2022, 12:21:01 PM1/21/22
to Prometheus Users
Yes, I have granted what is stated on the url you mentioned but it did not work until I granted SUPER Status

Brian Candler

unread,
Jan 21, 2022, 12:36:22 PM1/21/22
to Prometheus Users
Try adding "SLAVE MONITOR" instead of "SUPER", as the error message says.

Salam Elias

unread,
Jan 21, 2022, 2:11:16 PM1/21/22
to Prometheus Users

I granted REPLICATION SLAVE
GRANT PROCESS, REPLICATION CLIENT, REPLICATION SLAVE, SELECT ON *.* TO 'mysqld_exporter'@'localhost' WITH MAX_USER_CONNECTIONS 2;

as "Slave Monitor" privilege does not exist, and yes exporter worked correctly and started sending info to prometheus
Reply all
Reply to author
Forward
0 new messages