some questions

[lisa]

Hi all,
I just installed cerebro on my 5 oss's and on my mgs and mdt(note
these are on two separate machines) and I have
a few questions.
the first thing I notice is that when I run cerebro-stat -m lmt_ost I
only see info for 1 ost on each of the oss's.
I have 6 ost's on each oss so this is odd.

Second - I don't see a metric for the mgs. Only lmt_mds. is there a
metric for the mgs?

Third I am not sure where the lmt server needs to be installed. I
tried to load it on a machine
that was already running mysql but this machine is not a lustre client
or server. once cerebro is
installed it does not see any of the lmt lustre metrics. Does this
machine have to at least be a
lustre client? Is it advisable to run it on the mgs or mds?


thanks for any insight you can provide.

I am using cerebro version cerebro-1.10-1 with lmt 1.8.3 and lustre
server version 1.8.2.

lisa

[Andrew Uselton]

Howdy Lisa,

  These are all good questions, and most new users run up against them.

- only see info for 1 ost on each of the oss's.

    when a packet arrives at the monitor (where the DB is) the monitor immediately stuffs that data into the DB. However, the cerebro utility does not look at what's in the DB. It queries the daemon directly, askin "What do you have for service X from host Y?" In that way you will only ever see the latest update from a given OSS, which will be some one randomly chosen value for the OST metric. This is normal. It soulds like you are a long way towards getting things running.

- I don't see a metric for the mgs

     There is no Cerebro_metric_lmt_mgs.so. When Cerebro starts and attempts to load the cerebro_metric_lmt_mds.so (or any other) it will call the metric's initialization routine, which looks for the relevant entries in /proc/fs/lustre/... If it does not find what it wants it exits. I am not sure what /proc entries go with the mgs services versus mds. A look at the metric's source code should allow you to decide what and where.

 

- I am not sure where the lmt server needs to be installed.

      I am reading between the lines here a little. I think you want to know where to run the cerebro_monitor_lmt.so. If you run it on the same server as where you MySQL DB resides then, modulo configuration details, cerebro will receive the data sent from the metrics and stuff them into your DB. That host does not need to mount any Lustre services.

Cheers,

Andrew

--
You received this message because you are subscribed to the Google Groups "lmt-discuss" group.
To post to this group, send email to lmt-d...@googlegroups.com.
To unsubscribe from this group, send email to lmt-discuss...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/lmt-discuss?hl=en.

[Jim Garlick]

On Tue, Aug 31, 2010 at 12:18:25PM -0700, lisa wrote:
> Hi all,
> I just installed cerebro on my 5 oss's and on my mgs and mdt(note
> these are on two separate machines) and I have
> a few questions.
> the first thing I notice is that when I run cerebro-stat -m lmt_ost I
> only see info for 1 ost on each of the oss's.
> I have 6 ost's on each oss so this is odd.

Hi Lisa, that's a known issue that really only affects cerebro-stat.
The data is hashed by hostname so the different ost's overwrite each other.
However the cerebro monitor module gets a callback on arrival of each message
(before the hash) so it doesn't affect lmt.

> Second - I don't see a metric for the mgs. Only lmt_mds. is there a
> metric for the mgs?

Right, there isn't one.

> Third I am not sure where the lmt server needs to be installed. I
> tried to load it on a machine
> that was already running mysql but this machine is not a lustre client
> or server. once cerebro is
> installed it does not see any of the lmt lustre metrics. Does this
> machine have to at least be a
> lustre client? Is it advisable to run it on the mgs or mds?

We typically run it on a management node that is not running any lustre
services. You will need cerebro configured to listen on the network the
others are talking on.

> thanks for any insight you can provide.
>
> I am using cerebro version cerebro-1.10-1 with lmt 1.8.3 and lustre
> server version 1.8.2.

I think you have an lmt 2.x version there, based on your description.

Jim

[lisa]

Thanks for the quick responses.

Regarding the absence of an MGS metric:
do I not need to care about what the MGS is doing or not doing as the
case may be?

Regarding the LMT server configuration:
Is there any documentation on how to configure cerebro to listen on
the
network the others are talking on? I did not have to do anything on
the lustre servers
to get this to happen. And I see some minimal documentation about an /
etc/cerebro.conf file but none
of the machines actually has one of these.

lisa

[Jim Garlick]

On Tue, Aug 31, 2010 at 02:11:19PM -0700, lisa wrote:
> Thanks for the quick responses.
>
> Regarding the absence of an MGS metric:
> do I not need to care about what the MGS is doing or not doing as the
> case may be?

Generally the MGS has not been a problem for us so we felt no strong need
to develop software to monitor it.

> Regarding the LMT server configuration:
> Is there any documentation on how to configure cerebro to listen on
> the
> network the others are talking on? I did not have to do anything on
> the lustre servers
> to get this to happen. And I see some minimal documentation about an /
> etc/cerebro.conf file but none
> of the machines actually has one of these.

There is a man page cerebro.conf - I think this is the relevant section:

cerebrod_listen ( on | off )
Specify if the cerebrod daemon should receive packets or not.
Generally speaking, this option should be turned off for all but
just a few machines in a cluster. Just a few of the machines
should be responsible for collecting the Cerebro data so that
computational power of the cluster as a whole isn’t lost. The
default configuration turns this option on.

(Although I admit I'm not the expert on how to do this).

Jim

[Andrew Uselton]

yup. start there. post further questions :)

[Al Chu]

On Tue, 2010-08-31 at 14:26 -0700, Jim Garlick wrote:
> On Tue, Aug 31, 2010 at 02:11:19PM -0700, lisa wrote:
> > Thanks for the quick responses.
> >
> > Regarding the absence of an MGS metric:
> > do I not need to care about what the MGS is doing or not doing as the
> > case may be?
>
> Generally the MGS has not been a problem for us so we felt no strong need
> to develop software to monitor it.
>
> > Regarding the LMT server configuration:
> > Is there any documentation on how to configure cerebro to listen on
> > the
> > network the others are talking on? I did not have to do anything on
> > the lustre servers
> > to get this to happen. And I see some minimal documentation about an /
> > etc/cerebro.conf file but none
> > of the machines actually has one of these.

FYI the newest cerebro comes with a template /etc/cerebro.conf. But to
answer your question, cerebro may have chosen a different network by
default to speak/listen on. You can tell cerebro to use a different
interface via:

cerebrod_listen_message_config ip [port network_interface]

the manpage cerebro.conf will have more config details.

Al

> > > > For more options, visit this group athttp://*groups.google.com/group/lmt-discuss?hl=en.

> >
> > --
> > You received this message because you are subscribed to the Google Groups "lmt-discuss" group.
> > To post to this group, send email to lmt-d...@googlegroups.com.
> > To unsubscribe from this group, send email to lmt-discuss...@googlegroups.com.

> > For more options, visit this group at http://*groups.google.com/group/lmt-discuss?hl=en.
> >
>
--
Albert Chu
ch...@llnl.gov
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory

[lisa]

thanks for all your input everyone.
I think I have made some progress....

I got the cerebro.conf file template from the newer version (thanks
Al)
and now I have another question.
I have edited that file on my lmt-server and listed all my lustre
servers for
cerebro-metric-server and cerebro-event-server. now when I do cerebro-
stat -l
I finally see the metric options for lustre. I am not sure I need to
list them all for
both metric and event. Whats the difference between those two anyway?


Now I guess I need to try to add a table to my already existing mysql
database
for the lmt data.

Thanks for the help!

lisa

[lisa]

Me again....
;-)

I am trying to figure out what I need in my configuration file for the
create_lmtconfig command.
I am completely confused by the ost section.
I have 5 oss's called cmsls1-cmsls5. each has 6 ost's. ost with uuid
0-5 is on cmsls1 and uuid 6-11 is on cmsls2, etc.
I *think* the name section is for the name of the OSS and that I need
5 ost sections, one for each OSS.
Then I am not sure what I put in for the "skip" parameter. the ost's
are only seen on the one OSS.

Any help would be appreciated.

lisa

[Andrew Uselton]

Howdy Lisa,

  The create_lmtconfig utility is, ahem, idiosyncratic. It is trying to summarize your file system in a consice way, but the naming conventions and name expansion process are very LLNL-centric. If you have some Perl skills look through the code and see what it is trying to do - it's not complicated. If you have some SQL skills you can try running it an look at the resulting output. Unless you will be doing this a lot, or for a very large and complicated file system, it might be easiest to just hand craft the SQL. Take careful notes so you don't have to figure it out again later. Then write Jim a polite note about improving that utility :)

Cheers,

Andrew

PS if you get stuck drop me another note. It isn't that bad once you get the idea what it's trying to do.

-A

For more options, visit this group at http://groups.google.com/group/lmt-discuss?hl=en.

[Jim Garlick]

Something like this but put in your mds hostname, your file system
name (subst for "test")?

# test.cf
mds {
name cmsls0
uuid test-MDT0000
}

ost {
name cmsls[1-5]
numdevs 6
uuid foo-OST{HEXINDEX}
skip 1
}

create_lmtconfig -d test -f test.cf >test.sql

Sorry this utility is a pain. There is some (possibly deficient)
discussion of how this all works in the create_lmtconfig man page.

Jim

[lisa]

Thanks jim and everone else for the info on the create_lmtconfig
config file.
I was able to get that part done and use the file to create tables in
the mysql database.
Now the problem is that there is no other data in those tables except
for what I put in
from the output of create_lmtconfig.
I have set up the aggregation cron job and since it has no data to
work on, it fails.

Am I missing some other step that tells the lmt server to gather the
data from the lustre servers?
Do I need something else on the lustre servers to tell them to talk to
my lmt server?
Note that when I do a cerebro-stat -m lmt_ost on my lmt server I do
get data for my osts.

lisa

On Sep 8, 9:31 pm, Jim Garlick <garlick....@gmail.com> wrote:
> Something like this but put in your mds hostname, your file system
> name (subst for "test")?
>
> # test.cf
> mds {
>    name cmsls0
>    uuid test-MDT0000
>
> }
>
> ost {
>    name cmsls[1-5]
>    numdevs 6
>    uuid foo-OST{HEXINDEX}
>    skip 1
>
> }
>
> create_lmtconfig  -d test -f test.cf >test.sql
>
> Sorry this utility is a pain.  There is some (possibly deficient)
> discussion of how this all works in the create_lmtconfig man page.
>
> Jim
>

> On Wed, Sep 8, 2010 at 2:07 PM, Andrew Uselton <andrew.usel...@gmail.com> wrote:
> > Howdy Lisa,
> >   The create_lmtconfig utility is, ahem, idiosyncratic. It is trying to
> > summarize your file system in a consice way, but the naming conventions and
> > name expansion process are very LLNL-centric. If you have some Perl skills
> > look through the code and see what it is trying to do - it's not
> > complicated. If you have some SQL skills you can try running it an look at
> > the resulting output. Unless you will be doing this a lot, or for a very
> > large and complicated file system, it might be easiest to just hand craft
> > the SQL. Take careful notes so you don't have to figure it out again later.
> > Then write Jim a polite note about improving that utility :)
> > Cheers,
> > Andrew
> > PS if you get stuck drop me another note. It isn't that bad once you get the
> > idea what it's trying to do.
> > -A
>

[Jim Garlick]

If lmt-server is installed on your server system, and cerebro is
getting the data, then the cerebro "monitor" module should be getting
the cerebro "metrics" and attempting to insert them into the database.
Some things to check:

1. Does monitor module have insert permission? It reads its database
user/password from /usr/share/lmt/cron/lmtrc.

2. The database is named filesystem_fsname. The devices (e.g.
fsname-OST0000) have to match the database name.

3. Is there a mismatch between hostnames used in metrics vs. hostnames
in the _INFO tables? I.e. does one use the fully qualified domain
name and one not?

4. Check syslog - there may be some errors from the monitor module there.

Jim

[lisa]

In #1, whats the name of the monitor module user? I have only added an
entry for lwatchadmin to the
/usr/share/lmt/cron/lmtrc file.

the other 3 items have all been checked and are ok.

lisa

On Sep 9, 11:33 am, Jim Garlick <garlick....@gmail.com> wrote:
> If lmt-server is installed on your server system, and cerebro is
> getting the data, then the cerebro "monitor" module should be getting
> the cerebro "metrics" and attempting to insert them into the database.
>    Some things to check:
>
> 1. Does monitor module have insert permission?  It reads its database
> user/password from /usr/share/lmt/cron/lmtrc.
>
> 2. The database is named filesystem_fsname.  The devices (e.g.
> fsname-OST0000) have to match the database name.
>
> 3. Is there a mismatch between hostnames used in metrics vs. hostnames
> in the _INFO tables?  I.e. does one use the fully qualified domain
> name and one not?
>
> 4. Check syslog - there may be some errors from the monitor module there.
>
> Jim
>

> ...
>
> read more »

[Andrew Uselton]

lwatchadmin should do the trick. For some other activities you may also want an "lwatchclient". The daemon is going to try to attach to the DB on "localhost" (127.0.0.1) using the port specified in the lmtrc. You should be able to do that on the command line as well, as a sanity check.
-Andrew

For more options, visit this group at http://groups.google.com/group/lmt-discuss?hl=en.

[lisa]

I used the port specified in the sampe lmtrc file on the twiki.
its 3306. Is that the default. I have not changed any ports that are
used.

Using mysql directly I can get connected to the filesystem_test6tb db
(thats the name of my DB)
as user lwatchadmin on host localhost.
Thats how I knew that the DB was not getting updated.

Here is my full lmtrc file:
filesys.1.name=test6tb
filesys.1.mountname=/lustre
filesys.1.dbhost=127.0.0.1
filesys.1.dbport=3306
filesys.1.dbuser=lwatchadmin
filesys.1.dbauth=
filesys.1.dbname=filesystem_test6tb


One other thing to note is that the machine I have this lmt server on
has NO lustre modules loaded.
Its not even a client and does not have the lustre file system mounted
in any way.
From previous discussions I thought this was OK.

Lisa

On Sep 9, 11:53 am, Andrew Uselton <andrew.usel...@gmail.com> wrote:
> lwatchadmin should do the trick. For some other activities you may also want
> an "lwatchclient". The daemon is going to try to attach to the DB on
> "localhost" (127.0.0.1) using the port specified in the lmtrc. You should be
> able to do that on the command line as well, as a sanity check.
> -Andrew
>

> ...
>
> read more »

[Andrew Uselton]

Yes, you do not need any lustre modules on your DB server. 3306 is the default port for mysql so that should be fine. If you want to be paranoid you can verify it. lsof -i | grep mysql should show either "3306" or it may just say "mysql" if you have an entry in /etc/services (grep mysql /etc/services).

You can also specify the port explicitly on the mysql command line. 

I'm guessing that all is set correctly, so now it may be time to go throu the logs and see if you can find any relevant messages.

-A

> ...
>
> read more »

[lisa]

Hi all,

I confirmed we are using the standard port for mysql.
So the logs in /var/log show have no entries for cerebro or lmt
today.
there are some cerebro errors from the 7rh and then late in august
but
thats it.

I changed the cerebro.conf file on my lmt_server to turn debugging on
for
cerebro_listen_debug
cerebrod_metric_server_debug
cerebrod_event_server_debug

and I still see nothing.

are I had also turned on the option to keep logging for the agg cron
job.
Its running at 20 minutes past the hour and is mainly complaining
about not
having any data to aggregate:
# OST - filesystem_test6tb
#####################

Updating hourly ost agg table for filesystem_test6tb
Updating OST_AGGREGATE_HOUR for filesystem_test6tb...
Determining starting and ending points in various tables...
Final timestamp from OST_AGGREGATE_HOUR: 2006-01-01 00:00:00, id=0
Final timestamp from OST_DATA (raw): , id=
First timestamp from OST_DATA: No matching timestamp in raw
data
Updating other ost agg tables for filesystem_test6tb
Updating aggregate tables from OST_AGGREGATE_HOUR for
filesystem_test6tb...
First timestamp to use from hourly data: ()
Final timestamp to use from hourly data: ()
Updating OST_AGGREGATE_DAY
bad date '' at /usr/lib/perl5/vendor_perl/5.8.8/Date/Manip.pm line
4436.
Updating filesys-level ost tables for filesystem_test6tb
Updating FILESYSTEM_AGGREGATE from OST_AGGREGATE_HOUR for
filesystem_test6tb...
Updating FILESYSTEM_AGGREGATE from OST_AGGREGATE_DAY for
filesystem_test6tb...
Updating FILESYSTEM_AGGREGATE from OST_AGGREGATE_WEEK for
filesystem_test6tb...
Updating FILESYSTEM_AGGREGATE from OST_AGGREGATE_MONTH for
filesystem_test6tb


any ideas?

lisa

On Sep 9, 12:26 pm, Andrew Uselton <andrew.usel...@gmail.com> wrote:
> Yes, you do not need any lustre modules on your DB server. 3306 is the
> default port for mysql so that should be fine. If you want to be paranoid
> you can verify it. lsof -i | grep mysql should show either "3306" or it may
> just say "mysql" if you have an entry in /etc/services (grep mysql
> /etc/services).
>
> You can also specify the port explicitly on the mysql command line.
>
> I'm guessing that all is set correctly, so now it may be time to go throu
> the logs and see if you can find any relevant messages.
> -A
>

> ...
>
> read more »

[Andrew Uselton]

Others should correct me if I misremember, but I think you need to recompile Cerebro with debugging on before the config file debug settings will work. If my memory is correct it might be worth a try. The debugging output can be quite illuminating. 
-Andrew

> ...
>
> read more »

[lisa]

I am not sure when it happened but now I discover that cerebrod on my
lmt-server machine will not stay running.
I was trying to recompile things with debugging and got the newest
version of cerebro, 1.11-1.
If I start cerebrod from the command like with the --debug option I
see:

[root@cmssrv84 log]# /usr/sbin/cerebrod --debug
**************************************
* Cerebrod Configuration:
* -------------------------------
* Command Line Options
* -------------------------------
* debug: 1
* config_file: "/etc/cerebro.conf"
* -------------------------------
* Configuration File Options
* -------------------------------
* heartbeat_frequency_min: 10
* heartbeat_frequency_max: 20
* speak: 1
* speak[0]: ip: "239.2.11.72"
* speak[0]: destination_port: 8851
* speak[0]: source_port: 8850
* speak[0]: network_interface: "(null)"
* speak[0]: ip_is_multicast: 1
* speak[0]: ip_in_addr: 239.2.11.72
* speak[0]: network_interface_in_addr: 131.225.206.95
* speak[0]: network_interface_index: 2
* speak_message_ttl: 1
* listen: 1
* listen[0]: ip: "239.2.11.72"
* listen[0]: port: 8851
* listen[0]: network_interface: "(null)"
* listen[0]: ip_is_multicast: 1
* listen[0]: ip_in_addr: 239.2.11.72
* listen[0]: network_interfaces_in_addr: 131.225.206.95
* listen[0]: network_interfaces_index: 2
* listen_threads: 2
* metric_controller: 1
* metric_server: 1
* metric_server_port: 8852
* event_server: 1
* event_server_port: 8853
* forward_message_ttl: 1
* forward_host_accept:
* speak_debug: 0
* listen_debug: 0
* metric_controller_debug: 0
* metric_server_debug: 0
* event_server_debug: 0
* alternate_hostname: (null)
* -------------------------------
* Calculated Configuration
* -------------------------------
* heartbeat_frequency_is_ranged: 1
**************************************
cerebrod: (cerebro_clusterlist_hostsfile.c,
hostsfile_clusterlist_setup, 251): hostsfile '/etc/hostsfile' cannot
be opened: No such file or directory
cerebrod: (cerebrod_listener.c, _cerebrod_listener_initialize, 379):
clusterlist_module_setup

----------------------
Not sure what this /etc/hostsfile is supposed to be but I created it
and just stuffed some of the node names for the machines in the
cluster in it.
Then I try to start again and I get:

all that other stuff seen above + this error at the bottom:
/usr/sbin/cerebrod: symbol lookup error: /usr/lib64/cerebro/
cerebro_monitor_lmt.so: undefined symbol: mysql_init



mysql is running. and I had loaded it with the database info for this
previously.

any ideas?

lisa

On Sep 9, 4:05 pm, Andrew Uselton <andrew.usel...@gmail.com> wrote:
> Others should correct me if I misremember, but I think you need to recompile
> Cerebro with debugging on before the config file debug settings will work.
> If my memory is correct it might be worth a try. The debugging output can be
> quite illuminating.
> -Andrew
>

> ...
>
> read more »

[lisa]

any ideas?

lisa

On Sep 9, 4:05 pm, Andrew Uselton <andrew.usel...@gmail.com> wrote:
> Others should correct me if I misremember, but I think you need to recompile
> Cerebro with debugging on before the config file debug settings will work.
> If my memory is correct it might be worth a try. The debugging output can be
> quite illuminating.
> -Andrew
>

> ...
>
> read more »

[lisa]

any ideas?

lisa

On Sep 9, 4:05 pm, Andrew Uselton <andrew.usel...@gmail.com> wrote:
> Others should correct me if I misremember, but I think you need to recompile
> Cerebro with debugging on before the config file debug settings will work.
> If my memory is correct it might be worth a try. The debugging output can be
> quite illuminating.
> -Andrew
>

> ...
>
> read more »

[Albert Chu]

> all that other stuff seen above + this error at the bottom:
> /usr/sbin/cerebrod: symbol lookup error: /usr/lib64/cerebro/
> cerebro_monitor_lmt.so: undefined symbol: mysql_init

This suggests that the plugin isn't linked correctly. Do libmysql show
up if you run

ldd /usr/lib64/cerebro/cerebro_monitor_lmt.so

Al

> --
> You received this message because you are subscribed to the Google Groups "lmt-discuss" group.
> To post to this group, send email to lmt-d...@googlegroups.com.
> To unsubscribe from this group, send email to lmt-discuss...@googlegroups.com.

> For more options, visit this group at http://BLOCKEDgroups.google.com/group/lmt-discuss?hl=en.

[lisa]

no libmysql does not show up on the ldd.
I just rebuilt the lmt-server package (rpmbuild -ba lmt.spec), then
removed current lmt-server
rpm and installed the newly built one.
Still the ldd does not show any libmysql


Can someone also explain what the /etc/hostsfile is for, what its
supposed to have in it and
what the syntax is?
lisa

> ...
>
> read more »

[Albert Chu]

On Fri, 2010-09-17 at 06:33 -0700, lisa wrote:
> no libmysql does not show up on the ldd.
> I just rebuilt the lmt-server package (rpmbuild -ba lmt.spec), then
> removed current lmt-server
> rpm and installed the newly built one.
> Still the ldd does not show any libmysql
>
>
> Can someone also explain what the /etc/hostsfile is for, what its
> supposed to have in it and
> what the syntax is?

All you do is list each host in the cluster, one host per line.
However, if you don't have to have it. Cerebro is probably demanding
you have it because you installed the hostsfile plugin
(cerebro-clusterlist-hostsfile or something like that).

Al

[lisa]

Still not sure how the hostsfile inclusion was introduced but I guess
it does not really matter.
Now that I know how to work around that I guess its not that big of a
deal.
Thanks for the definition of what needs to be in the file.

But, I am still unsure how to get the mysql libs built into the
cerebro_monitor_lmt.so file.
The only thing I did on the cerebro build was to add a line to the
cerebro.spec to enable
debug.
I didn't do anything at all to change the config of the lmt-server
rpm.

Any advice on this?


lisa

> ...
>
> read more »