Monitoring bgfs via collectd+influxdb+grafana

[Eric]

Dear all,

Everything is in the title, I'm wondering how to monitor beegfs using the collectd+influxdb+grafana combination.
Do you guys developed a  collectd plugin or  is what collectd provides enough?
Nothing wrong with admon  but since I use grafana to monitor clusters that'd me great if I could include information for beegfs as well.
Thank you.

[Michael Ruepp]

Hi Eric, 

i can connect you to some guys from the Vienna Biocenter who are doing this with pcp.

They think about switching to CrateDB however, it has a SQL Layer and Grafana Plugin. 

Send me a Email if you are interested.

(I work for Crate.io)

Mit freundlichen Grüssen/Best regards,

Michael Ruepp, BA

_________________

mic...@ruepp.at

fon +43 676 911 40 90

skype michaelruepp

CONFIDENTIALITY NOTICE 

This message (including any attachments transmitted with it) contains confidential information and is intended only for the individual named herein. If you are not the herein named addressee you should not disseminate, distribute, copy or otherwise make use of this message. Please notify the sender immediately by e-mail if you have received this message by mistake, and delete it from your systems.

--
You received this message because you are subscribed to the Google Groups "beegfs-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jérémie Sebban]

Hello Eric,

I just finished setting up an influxdb+collectd+grafana yesterday for my new storage cluster. There is no Beegfs plugin yet but it's pretty simple to make it work with the collectd exec plugin.

Here is the shell script I use to gather read and write stats and send them via collectd:

#!/bin/sh
HOSTNAME="${COLLECTD_HOSTNAME:-`hostname -f`}"
INTERVAL="1"
while sleep "$INTERVAL"
do
        FULLSTAT=$(/usr/bin/beegfs-ctl --serverstats --nodetype=storage --history=1 -cfgFile=/etc/beegfs/beegfs-client-iostat.conf | tail -2 |head -1)

        RX=$(echo $FULLSTAT | awk '{print $3}')
        TX=$(echo $FULLSTAT | awk '{print $2}')
        echo "PUTVAL $HOSTNAME/BGiostat/disk_octets interval=$INTERVAL N:$RX:$TX"
done

I plan to add other stats later on. Also be aware that this script can not be run as root and it will trigger errors if you are using rdma in your cluster. The workaround is to use an alternate BeeGFS client conf file where connUseRDMA is set to false.

It's really convenient to get all the stats on the same place. I'm able to check each target throughput, iops and the behaviour of my nvme caches in one place. Snaphots are also really useful to "replay" workloads.

Regards,

Jeremie

[Nathan R.M. Crawford]

Hi Eric,

  We have been transitioning from collectd/influx/grafana to telegraf/influx/grafana, and I just converted the beegfs script yesterday (see below). The main difference is that it is called once per (10s) cycle by telegraf rather than as a collectd exec script that has a 10s internal loop. The guts of the script are the same. 

  There are times when the beegfs-ctl command stalls for a while, which makes me wish this info was just presented as user-readable files in /proc/fs/beegfs.

-Nate

#!/bin/bash

#This will report some custom BeeGFS info to telegraf/influx

HOSTNAME="$(hostname -s)"

mapfile -s 2 -t lines < <(beegfs-ctl --serverstats --nodetype=storage --history=10)

kwrite=0

kread=0

reqs=0

qlen=0

bsy=0 

for (( line=0; line<10; line++ )); do

  items=( ${lines[$line]} )

  kwrite=$(( kwrite + items[1] ))

  kread=$(( kread + items[2] ))

  reqs=$(( reqs + items[3] ))

  qlen=$(( qlen + items[4] ))

  bsy=$(( bsy + items[5] ))

done

bread=$((kread * 1024 / 10))

bwrite=$((kwrite * 1024 / 10))

dreqs=$((reqs / 10))

dqlen=$((qlen / 10))

dbsy=$((bsy / 10))

mapfile -s 2 -t lines < <(beegfs-ctl --serverstats --nodetype=metadata --history=10)

reqs=0

qlen=0

bsy=0 

for (( line=0; line<10; line++ )); do

  items=( ${lines[$line]} )

  reqs=$(( reqs + items[1] ))

  qlen=$(( qlen + items[2] ))

  bsy=$(( bsy + items[3] ))

done

mreqs=$((reqs / 10))

mqlen=$((qlen / 10))

mbsy=$((bsy / 10))

echo "beegfs,host=${HOSTNAME} bytes_read=$bread,bytes_written=$bwrite,data_reqs=$dreqs,data_queue=$dqlen,data_busy=$dbsy,meta_reqs=$mreqs,meta_queue=$mqlen,meta_busy=$mbsy"

################

To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Dr. Nathan Crawford              
Modeling Facility Director
Department of Chemistry
1102 Natural Sciences II         Office: 2101 Natural Sciences II
University of California, Irvine  Phone: 949-824-4508
Irvine, CA 92697-2025, USA

[Eric]

Thanks  Michael,

I think that's too far from what I'm trying to do.
Thank again,

            Éric.

[Eric]

Thank you Jeremie,

I'll give it a try.

       Éric.

[Eric]

Thanks Nathan,

I'll give it a try and let you know.

              Éric.

Dr. Nathan Crawford              
Modeling Facility Director
Department of Chemistry
1102 Natural Sciences II         Office: 2101 Natural Sciences II
University of California, Irvine  Phone: 949-824-4508
Irvine, CA 92697-2025, USA

[Sreenath T M]

Hi Eric,

         Did any of these work for you?