thruk page loading very slow

[Enrico Magrin]

Hi, I have recently installed Naemon (Core 1.4.1) and Thruk (3.04) on an Ubuntu 22.04 VM. I'm monitoring not so many hosts (~100, with mod_gearman) and the server load is very low, but every Thruk page I try to load it takes about 30 seconds or more, is there any known reason for this problem? How can I debug this situation?

Thanks

PS: this is an excerpt from the thruk log:

[2023/02/24 15:29:50][naemon][WARN] ***************************
[2023/02/24 15:29:50][naemon][WARN] slow_page_log_threshold (15s) hit, page took 43.4s to load.
[2023/02/24 15:29:50][naemon][WARN] page:    http://xxx.xxx.xxx.xxx/thruk/cgi-bin/tac.cgi?scrollTo=0_0___0&_=1677248946667
[2023/02/24 15:29:50][naemon][WARN] params:  {'_' => '1677248946667','scrollTo' => '0_0___0'}
[2023/02/24 15:29:50][naemon][WARN] user:    thrukadmin
[2023/02/24 15:29:50][naemon][WARN] address: xxx.xxx.xxx.xxx
[2023/02/24 15:29:50][naemon][WARN] +----------------------------------------------------------------------------------+-------------+
[2023/02/24 15:29:50][naemon][WARN] | total time                                                                       |   43.40743s |
[2023/02/24 15:29:50][naemon][WARN] +----------------------------------------------------------------------------------+-------------+
[2023/02/24 15:29:50][naemon][WARN] | _dispatcher: http://xxx.xxx.xxx.xxx/thruk/cgi-bin/tac.cgi?scrollTo=0_0___0&_=16   |   43.40742s |
[2023/02/24 15:29:50][naemon][WARN] |   time: Fri Feb 24 15:29:06 2023 - host: naemon - pid: 1853767 - req: 7          |             |
[2023/02/24 15:29:50][naemon][WARN] |   Root begin                                                                     |    0.00056s |
[2023/02/24 15:29:50][naemon][WARN] |     User::set_dynamic_attributes                                                 |    0.00006s |
[2023/02/24 15:29:50][naemon][WARN] |   Thruk::Controller::tac::index                                                  |   43.40286s |
[2023/02/24 15:29:50][naemon][WARN] |     AddDefaults::add_defaults(CACHED_DEFAULT)                                    |    3.35634s |
[2023/02/24 15:29:50][naemon][WARN] |       AddDefaults::get_proc_info                                                 |    3.35555s |
[2023/02/24 15:29:50][naemon][WARN] |         AddDefaults::set_processinfo(CACHED_DEFAULT)                             |    3.35551s |
[2023/02/24 15:29:50][naemon][WARN] |           AddDefaults::set_processinfo fetch                                     |    3.35546s |
[2023/02/24 15:29:50][naemon][WARN] |             _do_on_peers(get_processinfo)                                        |    3.35508s |
[2023/02/24 15:29:50][naemon][WARN] |               _get_result_serial(get_processinfo)                                |    3.35503s |
[2023/02/24 15:29:50][naemon][WARN] |               _merge_answer()                                                    |    0.00001s |
[2023/02/24 15:29:50][naemon][WARN] |       Utils::Menu::read_navigation()                                             |    0.00057s |
[2023/02/24 15:29:50][naemon][WARN] |     _do_on_peers(get_performance_stats)                                          |   30.03507s |
[2023/02/24 15:29:50][naemon][WARN] |       _get_result_serial(get_performance_stats)                                  |   30.03406s |
[2023/02/24 15:29:50][naemon][WARN] |       _merge_stats_answer()                                                      |    0.00096s |
[2023/02/24 15:29:50][naemon][WARN] |     _do_on_peers(get_host_stats)                                                 |    5.00504s |
[2023/02/24 15:29:50][naemon][WARN] |       _get_result_serial(get_host_stats)                                         |    5.00497s |
[2023/02/24 15:29:50][naemon][WARN] |     _do_on_peers(get_service_stats)                                              |    5.00605s |
[2023/02/24 15:29:50][naemon][WARN] |       _get_result_serial(get_service_stats)                                      |    5.00596s |
[2023/02/24 15:29:50][naemon][WARN] |   Root end                                                                       |    0.00008s |
[2023/02/24 15:29:50][naemon][WARN] |   render_tt: tac.tt                                                              |    0.00385s |
[2023/02/24 15:29:50][naemon][WARN] |     render: tac.tt                                                               |    0.00348s |
[2023/02/24 15:29:50][naemon][WARN] | finalize_request                                                                 |    0.00001s |
[2023/02/24 15:29:50][naemon][WARN] +----------------------------------------------------------------------------------+-------------+
[2023/02/24 15:29:50][naemon][WARN] | total time waited on backends                                                    |   43.39992s |
[2023/02/24 15:29:50][naemon][WARN] | total time waited on rendering                                                   |    0.00386s |
[2023/02/24 15:29:50][naemon][WARN] | total time                                                                       |   43.40743s |
[2023/02/24 15:29:50][naemon][WARN] +----------------------------------------------------------------------------------+-------------+

[Sven Nierlein]

Hi,

this is indeed way too slow. Is this all running on the same host (to rule out network issues)?
I just tried a ubuntu22-04 test vm with that setup and the tactical overview rendered in 50ms.

You could try to enable debug logging for livestatus in /etc/naemon/module-conf.d/livestatus.cfg.
Change the broker to:
broker_module=/usr/lib/naemon/naemon-livestatus/livestatus.so debug=3 /var/cache/naemon/live

After reloading naemon you should see the queries in /var/log/naemon/livestatus.log

You could also try a test query like this:

%> time echo "GET hosts" | unixcat /var/cache/naemon/live
...
real 0m0.004s
user 0m0.001s
sys 0m0.004s

Is there anything unusual?

Cheers,
Sven

> --
> You received this message because you are subscribed to the Google Groups "Thruk" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to thruk+un...@googlegroups.com <mailto:thruk+un...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/thruk/654d7d83-f4f5-4249-afca-faa216c9e8c0n%40googlegroups.com <https://groups.google.com/d/msgid/thruk/654d7d83-f4f5-4249-afca-faa216c9e8c0n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Enrico Magrin]

Hi,
I'm sorry for the very late reply but I fixed by myself and forgot to write it here. The problem was that I (for no reason) installed also the mod gearman worker on the server that runs naemon and the mod gearman server, and of course it wasn't configured properly so it slowed everything down. Now with the worker installed on the other server only (I need it for networking reasons) it works fine.
Thank you