Re: Setting up LMT

[Andrew Uselton]

Howdy FJ,

  Cerebro has a core functionality even without the cerebro_monitor_... and cerebro_metric_... libraries. Have you verified that it will start and give you eg updown_state? Working from memory I think the command:

/usr/sbin/cerebro-stat -l 

will list what metrics it thinks are available, and 

/usr/sbin/cerebro-stat -m <metric>

will give you some feedback on whether cerebrod is doing anything at all. The /etc/cerebrod.conf has a short list of debug options at the end, but doesn't say much about them. If you are trying to start with cerebro_monitor_lmt.so then you will probably want:

cerebrod_listen_debug       on

To use any of the debug modes you have to build cerebrod with debugging enabled:

./configure ...other options...  --enable-debug

If cerebrod is exitting at startup this may give you some indication of why. Let me know if you have success.

Cheers,

Andrew

On Mon, Jun 14, 2010 at 9:42 AM, FJ Kum <fongje...@hotmail.com> wrote:

Hi,
 
I am in the situation where I can't seem to start up my cerebrod due to cerebro_monitor_lmt.so. Can you tell me how can I debug what is the problem? Issuing 'service cerebrod restart' just keep failing on me.
 
I can't find any documentation elsewhere on the internet on how to turn on debug mode to diagnose the problem here.
 
Hope you can show me some light here.
 
Thanks.
 
FJ

 

---

The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail. Get busy.

[mike]

Hi Andrew,

I added in the cerebrod_debug_mode on in /etc/cerebro.conf as
suggested by you and I am getting more info. In the /var/log/message,
when I issue the 'service cerebrod restart', I am seeing this in the
messages log:

it is saying "Starting cerebrod: cerebrod: load config: error loading
cerebro config file configuration"
*"startproc: exit status of parent of /usr/sbin/cerebrod: 1"

> > ------------------------------

> > The New Busy is not the too busy. Combine all your e-mail accounts with

> > Hotmail. Get busy.<http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid...>- Hide quoted text -
>
> - Show quoted text -

[Andrew Uselton]

Oh wait. This may be misleading. I vaguely recollect that with debug mode on you may be drawing from a different config file than in normal mode. 

Do you feel like walking the source code a little? 

In your source tree at this point:

src/libs/util/config_util.c: line 1047

There is this code stanza:

#if CEREBRO_DEBUG

  if (!config_debug_config_file)

    config_file = CEREBRO_CONFIG_FILE_DEFAULT;

  else

    config_file = config_debug_config_file;

#else  /* !NDEBUG */

  config_file = CEREBRO_CONFIG_FILE_DEFAULT;

#endif /* !NDEBUG */

See if you can convince yourself it's picking up the right config file. The error you're getting comes from line 1073:

      if (errnum)

        *errnum = CEREBRO_ERR_CONFIG_FILE;

      goto cleanup;

But make sure it isn't just telling you you need special handling in debug mode for the config file.

If you can't sort it out let me know, I may be able to look a little more this evening.

Cheers,

Andrew

--
You received this message because you are subscribed to the Google Groups "lmt-discuss" group.
To post to this group, send email to lmt-d...@googlegroups.com.
To unsubscribe from this group, send email to lmt-discuss...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/lmt-discuss?hl=en.

[mike]

Hi Andrew,

Yes, I did just that by looking through the source code and the exact
line you pointed out. However, I don't see anything
that could potentially causing the problem of loading the /etc/
cerebro.conf.

JFYI, below is the content of my /etc/cerebro.conf:

cerebrod_listen on
cerebrod_listen_message_config 192.168.0.1 0 ib0
cerebrod_speak_message_config 192.168.0.1 0 ib0
cerebrod_listen_debug on

> > lmt-discuss...@googlegroups.com<lmt-discuss%2Bunsubscribe@googlegr­oups.com>
> > .

> > For more options, visit this group at

> >http://groups.google.com/group/lmt-discuss?hl=en.- Hide quoted text -

[Al Chu]

The cerebro manpage shows:

cerebrod_speak_message_config ip [destination_port source_port
network_interface]

You have only 3 inputs to 'cerebrod_speak_message_config'

Al

> > >http://*www.*windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid...>-

> > > Hide quoted text -
> >
> > > > - Show quoted text -
> >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "lmt-discuss" group.
> > > To post to this group, send email to lmt-d...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > lmt-discuss...@googlegroups.com<lmt-discuss%2Bunsubscribe@googlegr­oups.com>
> > > .
> > > For more options, visit this group at

> > >http://*groups.google.com/group/lmt-discuss?hl=en.- Hide quoted text -

> >
> > - Show quoted text -
>
--

Albert Chu
ch...@llnl.gov
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory

[mike]

Sorry, I was typing the content of the /etc/cerebro.conf without
referring to it.
So yes, i have all 4 inputs in there.

> Lawrence Livermore National Laboratory- Hide quoted text -

[Al Chu]

It's always possible there's a bug in the cerebro code I don't know
about. If you compile with debugging (--enable-debug in ./configure)
you can run cerebrod w/ the -d option, which will run cerebrod in the
foreground and give you more debug info.

Al

> busy.<http://*www.*windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid...>- Hide quoted text -

> >
> > - Show quoted text -
>
> --
> You received this message because you are subscribed to the
> Google Groups "lmt-discuss" group.
> To post to this group, send email to
> lmt-d...@googlegroups.com.
> To unsubscribe from this group, send email to lmt-discuss

> +unsub...@googlegroups.com.

> For more options, visit this group at

> http://*groups.google.com/group/lmt-discuss?hl=en.

>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "lmt-discuss" group.
> To post to this group, send email to lmt-d...@googlegroups.com.
> To unsubscribe from this group, send email to lmt-discuss

> +unsub...@googlegroups.com.

> For more options, visit this group at

> http://*groups.google.com/group/lmt-discuss?hl=en.

[mike]

Okay, just did the compilation with --enable-debug and now I have -d
option.
Below is the snippet of the debug message returned that looks
suspicious:

**************************************
* No Monitor Modules Found
**************************************
cerebrod: (cerebro_metric_boottime.c, boottime_metric_setup, 99):
strstr: boottime can't be found
cerebrod: (cerebrod_speaker_data.c, _setup_metric_modules, 226):
metric_module_setup: boottime
**************************************
* Received Message
* -----------------------
**************************************
* Cerebrod Message:
* -------------------
* version: 2
* nodename: "mds"
* metrics_len: 17
* memfree: metric type = 2, len = 4 value = 9938636
* packetsin: metric type = 2, len = 4 value = 11279689
* loadavg15: metric type = 3, len = 4 value = 0.030000
* bytesin: metric type = 7, len = 8 value = 1720817027
* memtotal: metric type = 2, len = 4 value = 12298496
* rxerrs: metric type = 2, len = 4 value = 0
* swaptotal: metric type = 2, len = 4 value = 2104472
* slurm_state: metric type = 2, len = 4 value = 0
* packetsout: metric type = 2, len = 4 value = 2020618
* memused: metric type = 2, len = 4 value = 2359860
* loadavg1: metric type = 3, len = 4 value = 0.280000
* bytesout: metric type = 7, len = 8 value = 462515208
* swapused: metric type = 2, len = 4 value = 0
* swapfree: metric type = 2, len = 4 value = 2104472
* txerrs: metric type = 2, len = 4 value = 0
* shutdown: metric type = 2, len = 4 value = 0
* loadavg5: metric type = 3, len = 4 value = 0.120000
**************************************
**************************************
**************************************

What is the meaning of the 'No Monitor Modules Found' up there? Is it
referring to cerebro_monitor_lmt.so?

> >         busy.<http://*www.*windowslive.com/campaign/thenewbusy?tile=multiaccount&oc­id...>- Hide quoted text -

>
> >         > - Show quoted text -
>
> >         --
> >         You received this message because you are subscribed to the
> >         Google Groups "lmt-discuss" group.
> >         To post to this group, send email to
> >         lmt-d...@googlegroups.com.
> >         To unsubscribe from this group, send email to lmt-discuss
> >         +unsub...@googlegroups.com.
> >         For more options, visit this group at
> >         http://*groups.google.com/group/lmt-discuss?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google
> > Groups "lmt-discuss" group.
> > To post to this group, send email to lmt-d...@googlegroups.com.
> > To unsubscribe from this group, send email to lmt-discuss
> > +unsub...@googlegroups.com.
> > For more options, visit this group at
> > http://*groups.google.com/group/lmt-discuss?hl=en.
>
> --
> Albert Chu
> ch...@llnl.gov
> Computer Scientist
> High Performance Systems Division

> Lawrence Livermore National Laboratory- Hide quoted text -

[Al Chu]

Yup,

> **************************************
> * No Monitor Modules Found
> **************************************

means it couldn't find 'cerebro_monitor_lmt.so'. Is it installed in the
right place? Should be /usr/lib{64}/cerebro/ I think.

Al

> > > busy.<http://**www.**windowslive.com/campaign/thenewbusy?tile=multiaccount&oc­id...>- Hide quoted text -

> >
> > > > - Show quoted text -
> >
> > > --
> > > You received this message because you are subscribed to the
> > > Google Groups "lmt-discuss" group.
> > > To post to this group, send email to
> > > lmt-d...@googlegroups.com.
> > > To unsubscribe from this group, send email to lmt-discuss
> > > +unsub...@googlegroups.com.
> > > For more options, visit this group at

> > > http://**groups.google.com/group/lmt-discuss?hl=en.

> >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "lmt-discuss" group.
> > > To post to this group, send email to lmt-d...@googlegroups.com.
> > > To unsubscribe from this group, send email to lmt-discuss
> > > +unsub...@googlegroups.com.
> > > For more options, visit this group at

> > > http://**groups.google.com/group/lmt-discuss?hl=en.

[mike]

Yes, it's in /usr/lib64/cerebro folder.

> > > >         busy.<http://**www.**windowslive.com/campaign/thenewbusy?tile=multiaccount&­oc­id...>- Hide quoted text -

[mike]

Okay, I was wrong with the path. Because I was compiling the source
using ./configure --enable-debug, now the default path to look up the
modules is in /usr/local/lib/cerebro instead of /usr/lib64/cerebro.I
believe the rpmbuild -b cerebro.spec put them in /usr/lib64/cerebro
and ./configure puts them in /usr/local/lib/cerebro by default.

So now I basically copy the /usr/lib64/cerebro/cerebro_monitor_lmt.so
into /usr/local/lib/cerebro and issue 'service cerebrod restart' and
now I am getting different error message:

**************************************
* Cerebro Clusterlist
* -----------------------
* Using Clusterlist: genders
**************************************
**************************************
* Setup Monitor Module: mon_lmt
**************************************
/usr/sbin/cerebrod: symbol lookup error: /usr/local/lib/cerebro/
cerebro_monitor_lmt.so: undefined symbol: mysql_init

mysql_init seems like not in the module?
What am I missing here?

> > - Show quoted text -- Hide quoted text -

[mike]

I'm trying to understand why mysql_init was not found.

mds1:/usr/src/packages/SOURCES/lmt-2.6.4/server/src # objdump -R /usr/
lib64/libmysqlclient.so | grep mysql_init
00000000003da878 R_X86_64_JUMP_SLOT mysql_init
00000000003dc6d0 R_X86_64_JUMP_SLOT mysql_init_character_set
mds1:/usr/src/packages/SOURCES/lmt-2.6.4/server/src # ldd /usr/local/
lib/cerebro/cerebro_monitor_lmt.so
linux-vdso.so.1 => (0x00007fff04586000)
libc.so.6 => /lib64/libc.so.6 (0x00007f66741f9000)
/lib64/ld-linux-x86-64.so.2 (0x00007f667477b000)
mds1:/usr/src/packages/SOURCES/lmt-2.6.4/server/src # strings /usr/
local/lib/cerebro/cerebro_monitor_lmt.so | grep -e mysql_init -e
mysqlclient
mysql_init

CLearly cerebro_monitor_lmt.so does not express dependency on
libmysqlclient.so but is trying to use its function.
Is there a compile/build option(s) for lmt-server in order to make
this dependency explicit?

> ...
>
> read more »- Hide quoted text -

[Andrew Uselton]

I have an old note to myself regarding compiling LMT modules. It was not finding mysql includes, so I did this. Not sure if that's what you need, but I thought I'd toss it out there.

Cheers,

Andrew

------old note to self--------------------

...

When you run configure you need to have told it where any special

includes reside, such as the Cerebro includes mentioned above.  Also

required are the MySQL includes.  

export CPPFLAGS="-I /usr/common/osp/cerebro/include -I /usr/common/osp/mysql/5.0.51b/include"

...

-----------------------------------------------

Of course, your includes will be in a different place...

To unsubscribe from this group, send email to lmt-discuss...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/lmt-discuss?hl=en.

[mike]

HI Andrew,

Thanks for the excellent info on MySQL includes for compiling lmt-
server but I found a workaround to statically link the
libmysqlclient.so to cerebrod.