Meaning of measurements in beegfs-mon

[Steve Eppert]

Hi

I'm currently playing around with beegfs-mon and I'm not quite sure if I understand these values correctly.

As of the stats I have a user that has these "meta operation list" values (10m aggregated)

close: 1902470

createLI: 18122

sum_getXA: 52338605

open: 1903698

revealLI: 8791415

stat: 42243163

statLI: 835815

And these storage operation list values:

B-rd: 2.43 TB

ops-rd: 99 M

And now I want to get some useful information out of this data.

Using the reference manual I can see that most (all) of the metadata operations are  just some internal messages that "can be related to corresponding system calls".

If this counter only counts "internal messages" - in what way would this be useful? Is there any way I can get information about the usage pattern of the users (like # of files) from this data?

The storage operation data seems useful - the user read around 2.43 TB of data.

Best

Steve

[Waltar]

Never used beegfs-mon as you even cannot change anythink on measured values which happen in the past nor you know what happen in future for. I find this most useful to see what happen actual and if deployed hw is enough for load:

alias beeperf="watch -n 1 'beegfs-ctl --nodetype=meta --serverstats --history=3;beegfs-ctl --nodetype=storage --serverstats --history=3'"
beeperf 

Total results for 2 nodes:
 time_index   reqs   qlen bsy
 1760975812   4718      1   0
 1760975813   5028      1   1
 1760975814   5049      0   1

Total results for 2 nodes:
 time_index write_KiB  read_KiB   reqs   qlen bsy
 1760976788         2    658383   1319      0   1
 1760976789         0    522280    837      0   0
 1760976790         0    431936    458      0   1

[Waltar]

That's an ongoing performance monitoring until you do Ctrl-c

[Steve Eppert]

The problem with this solution is that you cannot break down to individual users. So if you see an overload you have to watch and cannot do anything while with the beegfs-mon stats you at least (if these values somehow show something useful of course) can identify the problem (and hopefully fix it)

In general I think BeeGFS lacks some debugging tools and I thought I could use beegfs-mon for this

[Joshua Baker-LePain]

For real-time debugging of performance issues, 'beegfs-ctl' is my tool
of choice. You *can* query specific servers and/or users. For
'beegfs-ctl --serverstats', I generally use the '--perserver' flag to
break out the stats for each server. If you want to look at a single
server only, just append the nodeID of the server you want to look at,
and you'll get stats on that server only. To see stats for users, use
'beegfs-ctl --userstats'. You can also look at user stats on a single
server node only by again appending the nodeID to the 'beegfs-ctl'
command. And there's a '--filter' option to userstats if you want to
look at a specific user (see 'beegfs-ctl --userstats --help').

So my basic flow is to watch the output of 'beegfs-ctl --serverstats
--perserver' for both storage and meta. If there are any servers with
big queues (high value of 'qlen'), look at userstats for that server
(or clientstats to see if there's a particularly troublesome client
node) to see who is hammering on it. And then you look at the
users/clients in question and see what they're doing. beegfs-mon can
be useful in certain situations to somewhat characterize users' file
access patterns. But it definitely has limitations.

You also mentioned # of files per user -- that sort of info is (sort
of) available via 'beegfs-ctl --getquota' (if you have quotas turned
on, of course).

Caveat: All of the above is based on 7.x. Haven't moved up to 8.x yet.

> --
> You received this message because you are subscribed to the Google Groups "beegfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/fhgfs-user/5e4dbe11-8097-4a27-8ae9-bfe13f78e687n%40googlegroups.com.



--
Joshua Baker-LePain

[Steve Eppert]

OK, I never noticed that I can dig this deep using beegfs-ctl. Thanks, this looks awesome.