MCollective discovery - we did not discover any nodes

[Jens Braeuer]

Hi everyone,

I run MCollective 1.2.1 together with ActiveMQ 5.5 under Scientific Linux 6.1 on Amazon EC2. Overall it works like a
charm, but sometimes (eg. 1/30) discovery fails. Still the exit-code of mco will be 0, which is a problem for me as I
use MCollective e.g. to trigger deployments from Jenkins.

I would like to ask for some feedback on the following ideas, that could fix this problem.

a) Increase discovery timeout
mco offers a option to tweak the discovery timeout. What is your experience with increasing this value? When running
"mco ping", I see ping times of 130ms, so 2 seconds (the default should be enough), or?
Is there a way to configure is global?

b) Mco should exit != 0 when no nodes are found
I would like to see a "--batch" or "--non-interactive" mode, where mco has a exit code different from 0, when no nodes
are found.

c) Add "expected count" to mco command
I thing there are some situation, where one knows the number of MCollective nodes. So what about adding a options
"--expect-number" option to mco, where I can either give a count or range of expected nodes.

d) Is this normal at all?
I have no experience with MCollective in a datacenter, so: Is this problem cloud/EC2 related or does it happen in
non-cloud setups too? How could I debug what makes the discovery fail?

I really appreciate your feedback. You can find me a jbraeuer on #puppet.
Cheers,
Jens

[R.I.Pienaar]

hello,

----- Original Message -----
> Hi everyone,
>
> I run MCollective 1.2.1 together with ActiveMQ 5.5 under Scientific
> Linux 6.1 on Amazon EC2. Overall it works like a
> charm, but sometimes (eg. 1/30) discovery fails. Still the exit-code
> of mco will be 0, which is a problem for me as I
> use MCollective e.g. to trigger deployments from Jenkins.

does mco ping exhibit the same behavior if you run it often? Does it
tend to happen more after a period of the collective being idle or just
really randomly?

>
> I would like to ask for some feedback on the following ideas, that
> could fix this problem.
>
> a) Increase discovery timeout
> mco offers a option to tweak the discovery timeout. What is your
> experience with increasing this value? When running
> "mco ping", I see ping times of 130ms, so 2 seconds (the default
> should be enough), or?
> Is there a way to configure is global?

it's not global - its a client setting but with those ping times it
should be sufficient. Discovery does exactly what mco ping does so its
a good way to diagnose

Might be worth enabling verbose gc logging on your activemq its possible
that during these times it just did a big full garbage collection which
would block it and that might indicate some tuning is needed

>
> b) Mco should exit != 0 when no nodes are found
> I would like to see a "--batch" or "--non-interactive" mode, where
> mco has a exit code different from 0, when no nodes
> are found.

ok, you can file tickets for this

> c) Add "expected count" to mco command
> I thing there are some situation, where one knows the number of
> MCollective nodes. So what about adding a options
> "--expect-number" option to mco, where I can either give a count or
> range of expected nodes.

mcollective 1.3.x which will soon become 2.0 have a new mode of communications
where you can provide it a host list etc and it will bypass discovery, this
is ment to be used for things like deployers where you know what machines you
wish to affect and it will probably help

>
> d) Is this normal at all?
> I have no experience with MCollective in a datacenter, so: Is this
> problem cloud/EC2 related or does it happen in
> non-cloud setups too? How could I debug what makes the discovery
> fail?

it shouldn't happen, I've seen it happen:

- activemq doing long full garbage collections
- network is interrupted after long periods of idle time
- activemq is idle for a long time and was swapped etc
- you have very busy machines that do not respond at all - unlikely in your case

there are probably other reasons too but these are the rough likely causes.
Amazon has a pretty aggressive idle connection timeout though so you might
enable registration just to keep the stomp connections from being idle too long

[Jens Braeuer]

Hi everyone,

thank R.I.P. for the pointers. I already planed to add more monitoring to ActiveMQ, so I'll take this as a chance. I
dont think Amazon network settings are the root cause in my case, as I have registration agent enabled on all machines.
This should keep the connection busy.

I filed a bug regarding the exit-code. Is there any chance the fix will make it in 1.2.2?

Thanks,
Jens

--
Jens Br�uer
Senior Systems Engineer
Dipl. Inf.
NumberFour AG
Sch�nhauser Allee 8
10119 Berlin
Germany
Mobile: +49 175 221 88 34
Phone: +49 30 40505411
Fax: +49 30 40505410
je...@numberfour.eu

numberfour.eu
facebook.com/NumberFour
twitter.com/numberfourag

[R.I.Pienaar]

----- Original Message -----
> Hi everyone,
>
> thank R.I.P. for the pointers. I already planed to add more
> monitoring to ActiveMQ, so I'll take this as a chance. I
> dont think Amazon network settings are the root cause in my case, as
> I have registration agent enabled on all machines.
> This should keep the connection busy.
>
> I filed a bug regarding the exit-code. Is there any chance the fix
> will make it in 1.2.2?

No, behavior changes can't go into the current prod branch so will go
into 1.3.x which should become 2.0.0 very soon - like in January maybe

[Yaakov Nemoy]

If you're looking for monitoring tips, one thing we do is run an 'mco
find' on our entire network. A nagios page goes out if the count ('wc
-l') drops below a threshold.

-Yaakov

On Thu, Dec 8, 2011 at 06:50, Jens Braeuer <jens.b...@numberfour.eu> wrote:
> Hi everyone,
>
> thank R.I.P. for the pointers. I already planed to add more monitoring to ActiveMQ, so I'll take this as a chance. I
> dont think Amazon network settings are the root cause in my case, as I have registration agent enabled on all machines.
> This should keep the connection busy.
>
> I filed a bug regarding the exit-code. Is there any chance the fix will make it in 1.2.2?
>
> Thanks,
> Jens
>
> --

> Jens Bräuer

> Senior Systems Engineer
> Dipl. Inf.
> NumberFour AG

> Schönhauser Allee 8

> 10119 Berlin
> Germany
> Mobile: +49 175 221 88 34
> Phone: +49 30 40505411
> Fax: +49 30 40505410
> je...@numberfour.eu
>
> numberfour.eu
> facebook.com/NumberFour
> twitter.com/numberfourag
>

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>

[Jens Braeuer]

Hi Yaakov,

would you be so kind and share your Nagios script with us? Would be cool :-)

Jens

Am 08.12.11 23:22, schrieb Yaakov Nemoy:

> If you're looking for monitoring tips, one thing we do is run an 'mco
> find' on our entire network. A nagios page goes out if the count ('wc
> -l') drops below a threshold.
>
> -Yaakov
>
> On Thu, Dec 8, 2011 at 06:50, Jens Braeuer <jens.b...@numberfour.eu> wrote:
>> Hi everyone,
>>
>> thank R.I.P. for the pointers. I already planed to add more monitoring to ActiveMQ, so I'll take this as a chance. I
>> dont think Amazon network settings are the root cause in my case, as I have registration agent enabled on all machines.
>> This should keep the connection busy.
>>
>> I filed a bug regarding the exit-code. Is there any chance the fix will make it in 1.2.2?
>>
>> Thanks,
>> Jens
>>
>> --

>> Jens Br�uer

>> Senior Systems Engineer
>> Dipl. Inf.
>> NumberFour AG

>> Sch�nhauser Allee 8

>> 10119 Berlin
>> Germany
>> Mobile: +49 175 221 88 34
>> Phone: +49 30 40505411
>> Fax: +49 30 40505410
>> je...@numberfour.eu
>>
>> numberfour.eu
>> facebook.com/NumberFour
>> twitter.com/numberfourag
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
>> To post to this group, send email to puppet...@googlegroups.com.
>> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
>> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>>

--
Jens Br�uer

Senior Systems Engineer
Dipl. Inf.
NumberFour AG

Sch�nhauser Allee 8

[R.I.Pienaar]

----- Original Message -----
> Hi Yaakov,
>
> would you be so kind and share your Nagios script with us? Would be
> cool :-)

I have some activemq monitor stuff @ https://github.com/ripienaar/monitoring-scripts/tree/master/activemq
not quite what you're after but will help.

You can also use the registration system and the file based monitor to
notify you of machines thats not functional

http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AgentRegistrationMonitor

and indeed, the check Yaakov propose would also be useful

[Yaakov Nemoy]

Absolutely. Attached.

On Fri, Dec 9, 2011 at 01:08, Jens Braeuer <jens.b...@numberfour.eu> wrote:
> Hi Yaakov,
>
> would you be so kind and share your Nagios script with us? Would be cool :-)
>
> Jens
>
> Am 08.12.11 23:22, schrieb Yaakov Nemoy:
>> If you're looking for monitoring tips, one thing we do is run an 'mco
>> find' on our entire network. A nagios page goes out if the count ('wc
>> -l') drops below a threshold.
>>
>> -Yaakov
>>
>> On Thu, Dec 8, 2011 at 06:50, Jens Braeuer <jens.b...@numberfour.eu> wrote:
>>> Hi everyone,
>>>
>>> thank R.I.P. for the pointers. I already planed to add more monitoring to ActiveMQ, so I'll take this as a chance. I
>>> dont think Amazon network settings are the root cause in my case, as I have registration agent enabled on all machines.
>>> This should keep the connection busy.
>>>
>>> I filed a bug regarding the exit-code. Is there any chance the fix will make it in 1.2.2?
>>>
>>> Thanks,
>>> Jens
>>>
>>> --

>>> Jens Bräuer

>>> Senior Systems Engineer
>>> Dipl. Inf.
>>> NumberFour AG

>>> Schönhauser Allee 8

>>> 10119 Berlin
>>> Germany
>>> Mobile: +49 175 221 88 34
>>> Phone: +49 30 40505411
>>> Fax: +49 30 40505410
>>> je...@numberfour.eu
>>>
>>> numberfour.eu
>>> facebook.com/NumberFour
>>> twitter.com/numberfourag
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
>>> To post to this group, send email to puppet...@googlegroups.com.
>>> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
>>> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>>>
>
>
> --

> Jens Bräuer

> Senior Systems Engineer
> Dipl. Inf.
> NumberFour AG

> Schönhauser Allee 8

[Jayapandian Ponraj]

 Overall it works like a

> charm, but sometimes (eg. 1/30) discovery fails. 

does mco ping exhibit the same behavior if you run it often?  Does it

tend to happen more after a period of the collective being idle or just
really randomly?

Am facing the same problem where the discovery fails (1/4). I have a setup with 

a network of 2 activemq brokers and the client and nodes fails over from one to other.

The mco ping also exhibits the same problem, mostly after beeing idle and when one of the broker 

boxes is heavily loaded. I have nodes with 4000, 5000 ms ping and with 60, 70 ms ping 

but when i run mco ping with default i get no nodes discovered. Atleast it should discover the

low ping boxes right? 

I have configured fail over as in the documentaion but yet i find these discovery problems especially when

one of the brokers is loaded. Can mcollective to tuned to use the less loaded broker ?

 

>
> I would like to ask for some feedback on the following ideas, that
> could fix this problem.
>
> a) Increase discovery timeout
> mco offers a option to tweak the discovery timeout. What is your
> experience with increasing this value? When running
> "mco ping", I see ping times of 130ms, so 2 seconds (the default
> should be enough), or?
> Is there a way to configure is global?

it's not global - its a client setting but with those ping times it
should be sufficient.  Discovery does exactly what mco ping does so its
a good way to diagnose

by this documentation it shows its a server config  

http://docs.puppetlabs.com/mcollective/reference/basic/configuration.html

when i add the command line option --dt 5 it works fine. I dono why the low ping nodes

are not discovered in the default discovery timeout. I find this option 

export MCOLLECTIVE_EXTRA_OPTS="--dt 5 --timeout 3 --config /home/you/mcollective.cfg"

Is there a way to configure default discovery timeout for the client in the client.cfg?

Can anyone clarify me on this whole discovery timeout thing?

 

 

Might be worth enabling verbose gc logging on your activemq its possible
that during these times it just did a big full garbage collection which
would block it and that might indicate some tuning is needed

>
> b) Mco should exit != 0 when no nodes are found
> I would like to see a "--batch" or "--non-interactive" mode, where
> mco has a exit code different from 0, when no nodes
> are found.

ok, you can file tickets for this

I haven't setup any mcollective monitoring, Is checking if mco ping works fine 

enough ?

 

> c) Add "expected count" to mco command
> I thing there are some situation, where one knows the number of
> MCollective nodes. So what about adding a options
> "--expect-number" option to mco, where I can either give a count or
> range of expected nodes.

mcollective 1.3.x which will soon become 2.0 have a new mode of communications
where you can provide it a host list etc and it will bypass discovery, this
is ment to be used for things like deployers where you know what machines you
wish to affect and it will probably help

Am using the direct addressing stuff and i works just fine

 - activemq doing long full garbage collections
 - network is interrupted after long periods of idle time
 - activemq is idle for a long time and was swapped etc
 - you have very busy machines that do not respond at all - unlikely in your case

there are probably other reasons too but these are the rough likely causes.
Amazon has a pretty aggressive idle connection timeout though so you might
enable registration just to keep the stomp connections from being idle too long

Mine is a data centre setup, this is my first experience with message brokers so can 

i get some ideas in fine tuning activemq or ll using rabbitmq solve any of the problems? 

 

 

[Jayapandian Ponraj]

Any reply to my questions is hugely appreciated..