nagios not checking anything
Cory Coager
and the worker node is talking to the server, I can see this with
queue_top.pl. Upon starting nagios the server goes through all the
host and service checks once and never again! Then I see entries in
the nagios log about host and service checks being stale so the server
is forcing the check:
[1294407569] Warning: The check of host 'example01' looks like it was
orphaned (results never came back). I'm scheduling an immediate check
of the host...
Why is this happening?
Sven Nierlein
queue_top.pl is the old one. Use gearman_top instead. Do you see any worker connected to your gearmand?
Are the queues filling up?
Cory Coager
> Are the queues filling up?
and 2 for check_results. If I kill the worker process the jobs
waiting fills up from 0 to 318 for both service and host queues. If I
restart the worker the jobs waiting goes to 0.
Sven Nierlein
2 for check_results is 1 too much. Nagios usually only starts a single worker for the results queue.
Seems like there is a results worker which takes away all the results, which then leads to the
"results never came back" never came back messages.
Is there an orphaned nagios process running or something like that?
Cory Coager
> Seems like there is a results worker which takes away all the results, which then leads to the
> "results never came back" never came back messages.
> Is there an orphaned nagios process running or something like that?
check_results queue as well. Sorry this is my first experience with
gearman and I couldn't find a lot of information on how to set this up.
Cory Coager
service. There are 0 jobs waiting and 0 jobs running in the queue for
each. However, I am still getting orphaned messages in the nagios
logs.
Sven Nierlein
Could you try to stop Nagios and move the retention.dat away so you have a clean start.
check_gearman is able to monitor the check counter for the worker. Another idea is to
increase the loglevel from the worker. You should then see the jobs coming in.
Regarding the documentation: Help is always welcome :-)
Its not easy for me to guess what a new user expects or what pitfalls they can avoid.
Sven
Cory Coager
> check_gearman is able to monitor the check counter for the worker. Another idea is to
> increase the loglevel from the worker. You should then see the jobs coming in.
and service checks say pending so there is definitely a problem. One
more thing to mention, if I run the gearman client in the foreground
it spits out complete garbage when it receives jobs. Does this mean
anything to you? If not, whats next? Should I turn on the logging
for the gearman server?
Sven Nierlein
> more thing to mention, if I run the gearman client in the foreground
> it spits out complete garbage when it receives jobs. Does this mean
> anything to you? If not, whats next? Should I turn on the logging
> for the gearman server?
This usually means a problem with encryption. There should be messages
like "discarding invalid job" or something like that. Make sure your encryption
settings from the worker match the ones from the server. Password, Encryption enabled etc...
Cory Coager
> like "discarding invalid job" or something like that. Make sure your encryption
> settings from the worker match the ones from the server. Password, Encryption enabled etc...
client spits out garbage and no checks are processed in nagios.
Cory Coager
I don't see any logs being generated.
Sven Nierlein
> I set debug=3 in mod_gearman.conf and restarted nagios. Unfortunately
> I don't see any logs being generated.
Have you set the logfile option?
Cory Coager
to the directory.
Sven Nierlein
Ah i mixed that up. mod_gearman logs to the nagios.log. But don't do that (at least not in production), because Nagios is not thread safe and will segfault with a version <= 3.2.3. Hopefully the next nagios is better. They are working on that problem. The logfile option is only for the worker.
Cory Coager
[1294427513] mod_gearman: received job for queue service: example01 -
Check Ping
Cory Coager
Sven Nierlein
>>> Ah i mixed that up. mod_gearman logs to the nagios.log. But don't do that (at least not in production), because Nagios is not thread safe and will segfault with a version <= 3.2.3. Hopefully the next nagios is better. They are working on that problem. The logfile option is only for the worker.
>
> I don't see any logging options for the client/worker.
How do you start your worker? Its in the same config file as the encryption settings for your worker.
Cory Coager
Sven Nierlein
>> How do you start your worker? Its in the same config file as the encryption settings for your worker.
>
> I'm using the gearman client.
The gearman client provided by the gearman package itself? Thats probably just a sample client. You have to start the mod_gearman_worker provided by mod_gearman. This would explain everything...
Cory Coager
that. Silly me...
Sven Nierlein
I think a small step by step instruction would be really helpful sometimes...