MDT operations in database

[Frederik Ferner]

All,

we've recently started to look more seriously at LMT (3.1.3) to monitor
performance four our Lustre file systems. Getting live data for OSTs and
MDT operations works nicely, as does getting historical/aggregated data
for each OST. Now we are also interested in historical/aggregated
metadata operations and as far as I can see LMT doesn't provide this by
default.

It looks like the raw data is available in the mysql DB in the table
MDS_OPS_DATA but there is no script to aggregate this. Writing one
should not be that hard, but before we start on this, I wanted to ask if
anyone has already done this and is willing to share the script. (There
is a script to aggregate data from the table MDS_DATA, this contains
throughput data but not metadata operations.)

I'd also be interested if anyone has any tools they are willing to share
that extract the historical data from the database and graphs them. I'm
aware of lwatch and again this works ok as it is, especially for OST
data, but I'm not sure if it currently provides functionality to display
aggregated MDT operation data. If I've missed it, I'd welcome any pointers.

Kind regards,
Frederik
--
Frederik Ferner
Computer Systems Administrator phone: +44 1235 77 8624
Diamond Light Source Ltd. mob: +44 7917 08 5110
(Apologies in advance for the lines below. Some bits are a legal
requirement and I have no control over them.)

--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

[Andrew Uselton]

It's been a long standing todo for me to push my scripts back out to the community. "Real Soon Now". I do not have anything that does metadata operations, but I have a lot of  DB query to graph operations and statistical operations on historical data. The sort of thing I produce can be viewed at:

http://portal.nersc.gov/project/pma/

The Franklin stuff is pretty complete. Hopper LMT is in flux - soon.
What you see there is ad hoc Perl scripts building on the LMT.pl in the distribution. I also have a large collection of Python based NumPy analysis scripts. In the absence of pushing them back up stream I'm happy to consult one-on-one.
Cheers,
Andrew

--
You received this message because you are subscribed to the Google Groups "lmt-discuss" group.
To post to this group, send email to lmt-d...@googlegroups.com.
To unsubscribe from this group, send email to lmt-discuss+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/lmt-discuss?hl=en.

[Frederik Ferner]

On 02/09/11 02:44, Andrew Uselton wrote:
> It's been a long standing todo for me to push my scripts back out to the
> community. "Real Soon Now". I do not have anything that does metadata
> operations, but I have a lot of DB query to graph operations and
> statistical operations on historical data. The sort of thing I produce can
> be viewed at:
>
> http://portal.nersc.gov/project/pma/

This looks interesting. Is all of that based on the data that LMT
collects by default or did you add any other data into the mix/change
any LMT settings?

> The Franklin stuff is pretty complete. Hopper LMT is in flux - soon.
> What you see there is ad hoc Perl scripts building on the LMT.pl in the
> distribution. I also have a large collection of Python based NumPy analysis
> scripts. In the absence of pushing them back up stream I'm happy to consult
> one-on-one.

I'd certainly be interested in this. I've had a brief look at LMT.pl and
it seem usable (not knowing that much perl), but it would be nice, if I
didn't have to start from scratch.

[Andrew Uselton]

All of what I have on the web site is from the LMT table OST_DATA fields READ_BYTES and WRITE_BYTES. In order to do the queries you also need to access OST_INFO and TIMESTAMP_INFO.
Given a start time stamp and end time stamp, image the data is a 2d array of num_osts by num_time steps values. The delta from one time step to the next for a given ost divided by the delta from one time step to the next (usually 5 seconds) gives you a rate for that interval on that ost. All the rest is a variety of aggregations and transformations on that data set.

-A