Disk performance not viewable in PMM

[mysqlfish]

Hi,

I configured PMM using docker.

All the parameters, I can see as a graph in PMM page.

I added

1. mysql:queries
   
2. mysql:metrics
   
3. linux:metrics
   

When I see the disk performance in OS tab, couldn’t find anything.

Any help would be appreciated to fix this.

-------------- --------------------------------- ----------- -------- ---------------------------------- ------------------------------------------
SERVICE TYPE   NAME                              LOCAL PORT  RUNNING  DATA SOURCE                        OPTIONS         
-------------- --------------------------------- ----------- -------- ---------------------------------- ------------------------------------------
mysql:queries  mar1. . . . .net  -           YES      pmm_mon:***@unix(/tmp/mysql.sock)  query_source=slowlog, query_examples=true
linux:metrics  mar1. . . . .net  42000       YES      -                                                  
mysql:metrics  mar1. . . . .net  42002       YES      pmm_mon:***@unix(/tmp/mysql.sock)

​

[michael...@percona.com]

hi mysqlfish,

I am assuming you are looking at the Disk Performance dashboard for a specific server but you have empty graphs for this host.  This is what a functional Disk Performance dashboard should look like:

https://pmmdemo.percona.com/graph/dashboard/db/disk-performance?var-host=ps57&var-device=sda

Are any of the graphs displaying metrics, or is it just the Disk Performance dashboard?  can you please provide the exact commands you ran:

1. docker run
2. pmm-admin config
3. pmm-admin add
4. pmm-admin check-network

Thank you!

[mysqlfish]

Hi Micheal,

There is only a issue with disk performance dashboard. Below are the commands I ran.

[root@mar01 ~]# pmm-admin config --server 10.16.230.157
OK, PMM server is alive.

PMM Server      | 10.16.230.157
Client Name     | mar01.ana.uat.vaste. .net
Client Address  | 172.31.216.41
[root@mar01 ~]#

[root@mar01 ~]# pmm-admin add mysql:queries --user="pmm_mon" --host="localhost" --password=' '

[root@mar01 ~]# pmm-admin check-network
PMM Network Status

Server Address | 10.16.230.157
Client Address | 172.31.216.41

* System Time
NTP Server (0.pool.ntp.org)         | 2017-11-10 05:23:28 -0500 EST
PMM Server                          | 2017-11-10 10:23:28 +0000 GMT
PMM Client                          | 2017-11-10 05:23:28 -0500 EST
PMM Server Time Drift               | OK
PMM Client Time Drift               | OK
PMM Client to PMM Server Time Drift | OK

* Connection: Client --> Server
-------------------- -------
SERVER SERVICE       STATUS
-------------------- -------
Consul API           OK
Prometheus API       OK
Query Analytics API  OK

Connection duration | 213.875628ms
Request duration    | 224.714114ms
Full round trip     | 438.589742ms

* Connection: Client <-- Server
-------------- --------------------------------- -------------------- ------- ---------- ---------
SERVICE TYPE   NAME                              REMOTE ENDPOINT      STATUS  HTTPS/TLS  PASSWORD
-------------- --------------------------------- -------------------- ------- ---------- ---------
linux:metrics  mar01.ana.uat.vaste. .net  172.31.216.41:42000  DOWN    YES        -
mysql:metrics  mar01.ana.uat.vaste. .net  172.31.216.41:42002  DOWN    YES        -

When an endpoint is down it may indicate that the corresponding service is stopped (run 'pmm-admin list' to verify).
If it's running, check out the logs /var/log/pmm-*.log

When all endpoints are down but 'pmm-admin list' shows they are up and no errors in the logs,
check the firewall settings whether this system allows incoming connections from server to address:port in question.

Also you can check the endpoint status by the URL: http://10.16.230.157/prometheus/targets

[root@mar01 ~]#

[root@mar01 ~]# pmm-admin add mysql --user="pmm_mon" --host="localhost" --password=' '
[linux:metrics] OK, already monitoring this system.
[mysql:metrics] OK, already monitoring MySQL metrics.
[mysql:queries] OK, already monitoring MySQL queries.
[root@mar01 ~]#

[mysqlfish]

Adding the below comamnd for reference

```

[root@mar01 ~]# pmm-admin list

pmm-admin 1.4.1

PMM Server      | 10.16.230.157

Client Name     | mar01.ana.uat.vaste. .net

Client Address  | 172.31.216.41

Service Manager | unix-systemv

-------------- --------------------------------- ----------- -------- ---------------------------------- ------------------------------------------

SERVICE TYPE   NAME                              LOCAL PORT  RUNNING  DATA SOURCE                        OPTIONS         

-------------- --------------------------------- ----------- -------- ---------------------------------- ------------------------------------------

mysql:queries  mar01.ana.uat.vaste. .net  -           YES      pmm_mon:***@unix(/tmp/mysql.sock)  query_source=slowlog, query_examples=true

linux:metrics  mar01.ana.uat.vaste. .net  42000       YES      -                                                  

mysql:metrics  mar01.ana.uat.vaste. .net  42002       YES      pmm_mon:***@unix(/tmp/mysql.sock)                  

```

On Thursday, 9 November 2017 16:59:52 UTC+5:30, mysqlfish wrote:

[mysqlfish]

Hi Micheal,

When I check the command pmm-admin check-network, I could see the status of mysql:metrics and linux-metrics are down.

When I check the logs, below are the stuffs I can see…. don’t have a clue whats this actually. Looks like

1. port issue
2. firewall issue

[root@mar01 log]# tail pmm-mysql-metrics-42002.log
2017/11/10 03:52:15 http: TLS handshake error from 10.16.230.157:40844: EOF
2017/11/10 03:53:04 http: TLS handshake error from 10.16.230.157:41258: EOF
2017/11/10 03:53:29 http: TLS handshake error from 10.16.230.157:41470: EOF
2017/11/10 03:53:41 http: TLS handshake error from 10.16.230.157:41572: EOF
time="2017-11-10T05:22:30-05:00" level=info msg="Starting mysqld_exporter (version=1.4.1, branch=master, revision=c5b2f15a2b2b46eb53192c6aded039c90f406733)" source="mysqld_exporter.go:798"
time="2017-11-10T05:22:30-05:00" level=info msg="Build context (go=go1.8, user=, date=)" source="mysqld_exporter.go:799"
time="2017-11-10T05:22:30-05:00" level=info msg="HTTPS/TLS is enabled" source="mysqld_exporter.go:843"
time="2017-11-10T05:22:30-05:00" level=info msg="Listening on 172.31.216.41:42002" source="mysqld_exporter.go:846"
2017/11/10 05:23:29 http: TLS handshake error from 172.31.216.41:52956: tls: first record does not look like a TLS handshake
2017/11/10 05:25:38 http: TLS handshake error from 172.31.216.41:52980: tls: first record does not look like a TLS handshake

[root@mar01 log]# tail pmm-linux-metrics-42000.log
time="2017-11-10T05:22:29-05:00" level=info msg=" - netstat" source="node_exporter.go:164"
time="2017-11-10T05:22:29-05:00" level=info msg=" - time" source="node_exporter.go:164"
time="2017-11-10T05:22:29-05:00" level=info msg=" - uname" source="node_exporter.go:164"
time="2017-11-10T05:22:29-05:00" level=info msg=" - vmstat" source="node_exporter.go:164"
time="2017-11-10T05:22:29-05:00" level=info msg=" - diskstats" source="node_exporter.go:164"
time="2017-11-10T05:22:29-05:00" level=info msg=" - filefd" source="node_exporter.go:164"
time="2017-11-10T05:22:29-05:00" level=info msg=" - filesystem" source="node_exporter.go:164"
time="2017-11-10T05:22:29-05:00" level=info msg="Starting HTTPS server of 172.31.216.41:42000 ..." source="server.go:106"
2017/11/10 05:23:29 http: TLS handshake error from 172.31.216.41:53824: tls: first record does not look like a TLS handshake
2017/11/10 05:25:38 http: TLS handshake error from 172.31.216.41:53848: tls: first record does not look like a TLS handshake
[root@mar01 log]#

Help me to fix this, if possible.

[michael...@percona.com]

Hi mysqlfish, interesting - list is showing services listening but check-network shows the service is down.  This is generally indicative of  a network or firewall blocking access from PMM Server to the target exporter.  From the host where you installed PMM Server via docker, please execute:

curl -s -k http://mar01.ana.uat.vaste.ABCXYZ.net:42000/metrics

This should output a long list of key-value pairs, that looks something like this:

[root@pmmdemo ~]# curl -s -k https://163.172.28.90:42000/metrics | head -5

# HELP go_gc_duration_seconds A summary of the GC invocation durations.

# TYPE go_gc_duration_seconds summary

go_gc_duration_seconds{quantile="0"} 5.0972e-05

go_gc_duration_seconds{quantile="0.25"} 8.6293e-05

go_gc_duration_seconds{quantile="0.5"} 0.000113265

If nothing is returned, please check to see if you are permitting TCP access from PMM Server -> client on port 42000

Let us know how you progress, thank you,