NFS IO Stats.

[yagyans...@gmail.com]

Hi. I have enabled the mounstats, nfs and nfsd collectors for the NFS side metrics but there is a plethora of metrics without any proper documentation. Can somebody help with which metrics would give me the reads and writes completed on a particular mount? Like we have node_disk_reads_completed_total for SSD and HDDs, is there any exact alternative for NFS to get the IO stats?

Thanks in advance!

[Chris Siebenmann]

If you care about NFS client statistics, there is no exact analog
because NFS mount metrics are fundamentally at a different level than
disk IO statistics. Disk IO statistics are obviously at the level of
data (blocks) transfered, regardless of why they are transfered and
how much of the data will be used. NFS client mount metrics are at the
level of filesystem operations, and filesystem operations can have
unpredictable disk IO impacts. All NFS operations send data to the NFS
server and get data back from it, but some only (potentially) create
read disk IO on the server while others cause write IO.

To put together NFS client IO statistics that are useful to you, you
will have to figure out what you care about in all of this. Some of this
will be workload dependent; for example, if your workload mostly reads
and writes a small number of big files, the filesystem level read and
write IO (which you can get numbers on) is highly predictive of disk IO
on the server and probably of performance. If your workload spends a lot
of time creating and deleting small files, looking only at the bytes
read and written from those files is probably missing a lot of server
disk IO, especially write IO.

As for what the metrics mean, that is hard to summarize well here.
The 'mountstat' collector is the most detailed and more or less what
it is collecting is written up in:

https://utcc.utoronto.ca/~cks/space/blog/linux/NFSMountstatsIndex

This is from 2013 and for NFS v3, not NFS v4, but I believe that not
much has changed in this area of Linux since then and NFS v4 is pretty
similar to NFS v3.

(Locally I have a program that significantly aggregates this information
on a per-mount basis, because we have a *lot* of NFS mounts and the raw
mountstats data for them all adds up to too many metrics for us.)

- cks

[yagyans...@gmail.com]

Thanks a lot for the detailed explanation,

> Hi. I have enabled the mounstats, nfs and nfsd collectors for the NFS
> side metrics but there is a plethora of metrics without any proper
> documentation. Can somebody help with which metrics would give me
> the reads and writes completed on a particular mount? Like we have
> node_disk_reads_completed_total for SSD and HDDs, is there any exact
> alternative for NFS to get the IO stats?

If you care about NFS client statistics, there is no exact analog
because NFS mount metrics are fundamentally at a different level than
disk IO statistics. Disk IO statistics are obviously at the level of
data (blocks) transfered, regardless of why they are transfered and
how much of the data will be used. NFS client mount metrics are at the
level of filesystem operations, and filesystem operations can have
unpredictable disk IO impacts. All NFS operations send data to the NFS
server and get data back from it, but some only (potentially) create
read disk IO on the server while others cause write IO.

To put together NFS client IO statistics that are useful to you, you
will have to figure out what you care about in all of this. Some of this
will be workload dependent; for example, if your workload mostly reads
and writes a small number of big files, the filesystem level read and
write IO (which you can get numbers on) is highly predictive of disk IO
on the server and probably of performance. If your workload spends a lot
of time creating and deleting small files, looking only at the bytes
read and written from those files is probably missing a lot of server
disk IO, especially write IO.

 >> Yes, I need these IOs exactly that are caused by the NFS operations. Can you give me an idea on which Prometheus metrics can give me this?

 

As for what the metrics mean, that is hard to summarize well here.
The 'mountstat' collector is the most detailed and more or less what
it is collecting is written up in:

https://utcc.utoronto.ca/~cks/space/blog/linux/NFSMountstatsIndex

>> This is quite useful. Thanks!