Backend not available when generating reports

Marco Musso

unread,

May 3, 2017, 4:22:08 AM5/3/17

to Thruk

Hi Sven,

I'm getting this strange error (Backend not available) when generating reports (web interface as well cli) for a BP.

Setup is Naemon 1.0.6 + livestatus 1.0.6 with Thruk 2.14-1 talking with lmd 1.0.3 (started externally and not with use_lmd_core).

-sh-4.2$ systemctl status lmd

● lmd.service - Livestatus Multitool Daemon

Loaded: loaded (/etc/systemd/system/lmd.service; enabled; vendor preset: disabled)

Active: active (running) since Wed 2017-04-26 22:18:15 UTC; 6 days ago

Main PID: 1600 (lmd)

CGroup: /system.slice/lmd.service

└─1600 /usr/local/bin/lmd -v --config=/etc/lmd/lmd.ini -pidfile /run/lmd/lmd.pid

This is the verbose output (ran as apache):

-sh-4.2$ THRUK_VERBOSE=4 thruk -A thrukadmin -a 'url=avail.cgi?host=UAM&service=UAM&timeperiod=last7days&csvoutput=1'

reading config file: /etc/thruk/thruk.conf

reading config file: /etc/thruk/thruk_local.d/naemon.conf

reading config file: /etc/thruk/thruk_local.conf

reading config file: /etc/thruk/cgi.cfg

[07:46:51,973][DEBUG][Thruk] logging initialized

[07:46:51,987][DEBUG][Thruk] http://localhost/thruk/cgi-bin/remote.cgi

[07:46:51,988][DEBUG][Thruk] $VAR1 = {};

[07:46:51,988][DEBUG][Thruk] checking auth

[07:46:51,988][DEBUG][Thruk] remote.cgi does not use authentication

TT passthrough.tt:

cnt percent clk user sys cuser csys template

1 0 % 0 0.00 0.00 0.00 0.00 passthrough.tt

[07:46:51,991][INFO][Thruk] 20035 Req: 001 mem: 33.59 MB 0.00 MB dur: 0.004s ---- size: 0.003 kb stat: 200 url: remote.cgi

[07:46:51,991][DEBUG][Thruk] Profile:

+----------------------------------------------------------------------------------+-------------+

| _dispatcher: http://localhost/thruk/cgi-bin/remote.cgi | 0.00326s |

| Root begin | 0.00088s |

| Utils::read_cgi_cfg() | 0.00002s |

| Root end | 0.00004s |

| render_tt: passthrough.tt | 0.00097s |

| render: passthrough.tt | 0.00092s |

| _after_dispatch | 0.00010s |

+----------------------------------------------------------------------------------+-------------+

[07:46:51,992][DEBUG][Thruk] can_submit_commands: 1

[07:46:51,992][DEBUG][Thruk] _set_enabled_backends() using defaults

[07:46:51,992][DEBUG][Thruk] disabled_backends: $VAR1 = {};

[07:46:51,992][DEBUG][Thruk] livestatus: get_contactgroups_by_contact: b888b

[07:46:51,993][DEBUG][Thruk] selectall_arrayref("GET contactgroups\nColumns: name\nFilter: members >= thrukadmin", {'addpeer' => 1,'slice' => {}}, 0)

[07:46:51,993][DEBUG][Thruk] > $VAR1 = 'GET contactgroups

Columns: name

Filter: members >= thrukadmin

OutputFormat: json

ResponseHeader: fixed16

';

[07:46:51,993][DEBUG][Thruk] using new connection

[07:46:51,994][DEBUG][Thruk] header: 200 3

[07:46:51,994][DEBUG][Thruk] query status 200

[07:46:51,994][DEBUG][Thruk] status: $VAR1 = 200;

[07:46:51,994][DEBUG][Thruk] msg: $VAR1 = 'OK. Reponse contains the queried data.';

[07:46:51,995][DEBUG][Thruk] < $VAR1 = [];

[07:46:51,996][DEBUG][Thruk] creating new user cache for thrukadmin

[07:46:51,996][DEBUG][Thruk] livestatus: get_can_submit_commands: b888b

[07:46:51,996][DEBUG][Thruk] selectall_arrayref("GET contacts\nColumns: can_submit_commands alias\nFilter: name = thrukadmin", {'addpeer' => 1,'slice' => {}}, 0)

[07:46:51,997][DEBUG][Thruk] > $VAR1 = 'GET contacts

Columns: can_submit_commands alias

Filter: name = thrukadmin

OutputFormat: json

ResponseHeader: fixed16

';

[07:46:51,997][DEBUG][Thruk] using new connection

[07:46:51,997][DEBUG][Thruk] header: 200 3

[07:46:51,997][DEBUG][Thruk] query status 200

[07:46:51,998][DEBUG][Thruk] status: $VAR1 = 200;

[07:46:51,998][DEBUG][Thruk] msg: $VAR1 = 'OK. Reponse contains the queried data.';

[07:46:51,998][DEBUG][Thruk] < $VAR1 = [];

[07:46:51,998][DEBUG][Thruk] can_submit_commands: 1

[07:46:52,003][DEBUG][Thruk] logging initialized

[07:46:52,005][DEBUG][Thruk] http://localhost/thruk/cgi-bin/avail.cgi?host=UAM&service=UAM&timeperiod=last7days&csvoutput=1

[07:46:52,005][DEBUG][Thruk] $VAR1 = {

'csvoutput' => '1',

'host' => 'UAM',

'service' => 'UAM',

'timeperiod' => 'last7days'

};

[07:46:52,005][DEBUG][Thruk] checking auth

[07:46:52,005][DEBUG][Thruk] user not authenticated yet

[07:46:52,006][DEBUG][Thruk] user authenticated as: thrukadmin

[07:46:52,008][DEBUG][Thruk] _set_enabled_backends() using defaults

[07:46:52,008][DEBUG][Thruk] disabled_backends: $VAR1 = {};

[07:46:52,008][DEBUG][Thruk] livestatus: get_processinfo: b888b

[07:46:52,008][DEBUG][Thruk] selectall_arrayref("GET status\nColumns: accept_passive_host_checks accept_passive_service_checks check_external_commands check_host_freshness check_service_freshness enable_event_handlers enable_flap_detection enable_notifications execute_host_checks execute_service_checks last_command_check last_log_rotation livestatus_version nagios_pid obsess_over_hosts obsess_over_services process_performance_data program_start program_version interval_length", {'addpeer' => 1,'rename' => {'livestatus_version' => 'data_source_version'},'slice' => {},'wrapped_json' => 0}, 0)

[07:46:52,008][DEBUG][Thruk] > $VAR1 = 'GET status

Columns: accept_passive_host_checks accept_passive_service_checks check_external_commands check_host_freshness check_service_freshness enable_event_handlers enable_flap_detection enable_notifications execute_host_checks execute_service_checks last_command_check last_log_rotation livestatus_version nagios_pid obsess_over_hosts obsess_over_services process_performance_data program_start program_version interval_length

OutputFormat: json

ResponseHeader: fixed16

';

[07:46:52,009][DEBUG][Thruk] using new connection

[07:46:52,009][DEBUG][Thruk] header: 200 78

[07:46:52,009][DEBUG][Thruk] query status 200

[07:46:52,009][DEBUG][Thruk] status: $VAR1 = 200;

[07:46:52,009][DEBUG][Thruk] msg: $VAR1 = 'OK. Reponse contains the queried data.';

[07:46:52,010][DEBUG][Thruk] < $VAR1 = [

[

1,

0,

1,

0,

'1.0.6',

365,

0,

1,

'1493793955',

'1.0.6-pkg',

60

]

];

[07:46:52,010][DEBUG][Thruk] livestatus: get_contactgroups_by_contact: b888b

[07:46:52,011][DEBUG][Thruk] selectall_arrayref("GET contactgroups\nColumns: name\nFilter: members >= thrukadmin", {'addpeer' => 1,'slice' => {}}, 0)

[07:46:52,011][DEBUG][Thruk] > $VAR1 = 'GET contactgroups

Columns: name

Filter: members >= thrukadmin

OutputFormat: json

ResponseHeader: fixed16

';

[07:46:52,011][DEBUG][Thruk] using new connection

[07:46:52,011][DEBUG][Thruk] header: 200 3

[07:46:52,012][DEBUG][Thruk] query status 200

[07:46:52,012][DEBUG][Thruk] status: $VAR1 = 200;

[07:46:52,012][DEBUG][Thruk] msg: $VAR1 = 'OK. Reponse contains the queried data.';

[07:46:52,012][DEBUG][Thruk] < $VAR1 = [];

[07:46:52,012][DEBUG][Thruk] creating new user cache for thrukadmin

[07:46:52,013][DEBUG][Thruk] livestatus: get_can_submit_commands: b888b

[07:46:52,013][DEBUG][Thruk] selectall_arrayref("GET contacts\nColumns: can_submit_commands alias\nFilter: name = thrukadmin", {'addpeer' => 1,'slice' => {}}, 0)

[07:46:52,013][DEBUG][Thruk] > $VAR1 = 'GET contacts

Columns: can_submit_commands alias

Filter: name = thrukadmin

OutputFormat: json

ResponseHeader: fixed16

';

[07:46:52,013][DEBUG][Thruk] using new connection

[07:46:52,014][DEBUG][Thruk] header: 200 3

[07:46:52,014][DEBUG][Thruk] query status 200

[07:46:52,014][DEBUG][Thruk] status: $VAR1 = 200;

[07:46:52,014][DEBUG][Thruk] msg: $VAR1 = 'OK. Reponse contains the queried data.';

[07:46:52,014][DEBUG][Thruk] < $VAR1 = [];

[07:46:52,015][DEBUG][Thruk] can_submit_commands: 1

[07:46:52,030][DEBUG][Thruk] start: 1493192812 - Wed Apr 26 07:46:52 2017

[07:46:52,030][DEBUG][Thruk] end : 1493797612 - Wed May 3 07:46:52 2017

[07:46:52,030][DEBUG][Thruk] logstart: 1492847212 - Sat Apr 22 07:46:52 2017

[07:46:52,030][DEBUG][Thruk] livestatus: get_services: b888b

[07:46:52,031][DEBUG][Thruk] selectall_arrayref("GET services\nColumns: accept_passive_checks acknowledged action_url action_url_expanded active_checks_enabled check_command check_interval check_options check_period check_type checks_enabled comments current_attempt current_notification_number description event_handler event_handler_enabled custom_variable_names custom_variable_values execution_time first_notification_delay flap_detection_enabled groups has_been_checked high_flap_threshold host_acknowledged host_action_url_expanded host_active_checks_enabled host_address host_alias host_checks_enabled host_check_type host_latency host_plugin_output host_perf_data host_current_attempt host_check_command host_comments host_groups host_has_been_checked host_icon_image_expanded host_icon_image_alt host_is_executing host_is_flapping host_name host_notes_url_expanded host_notifications_enabled host_scheduled_downtime_depth host_state host_accept_passive_checks host_last_state_change icon_image icon_image_alt icon_image_expanded is_executing is_flapping last_check last_notification last_state_change latency low_flap_threshold max_check_attempts next_check notes notes_expanded notes_url notes_url_expanded notification_interval notification_period notifications_enabled obsess_over_service percent_state_change perf_data plugin_output process_performance_data retry_interval scheduled_downtime_depth state state_type modified_attributes_list last_time_critical last_time_ok last_time_unknown last_time_warning display_name host_display_name host_custom_variable_names host_custom_variable_values in_check_period in_notification_period host_parents\nFilter: host_name = UAM\nFilter: description = UAM\nAnd: 2", {'addpeer' => 0,'callbacks' => {'last_state_change_order' => sub { "DUMMY" },'state_order' => sub { "DUMMY" }},'slice' => {}}, 0)

[07:46:52,031][DEBUG][Thruk] > $VAR1 = 'GET services

Columns: accept_passive_checks acknowledged action_url action_url_expanded active_checks_enabled check_command check_interval check_options check_period check_type checks_enabled comments current_attempt current_notification_number description event_handler event_handler_enabled custom_variable_names custom_variable_values execution_time first_notification_delay flap_detection_enabled groups has_been_checked high_flap_threshold host_acknowledged host_action_url_expanded host_active_checks_enabled host_address host_alias host_checks_enabled host_check_type host_latency host_plugin_output host_perf_data host_current_attempt host_check_command host_comments host_groups host_has_been_checked host_icon_image_expanded host_icon_image_alt host_is_executing host_is_flapping host_name host_notes_url_expanded host_notifications_enabled host_scheduled_downtime_depth host_state host_accept_passive_checks host_last_state_change icon_image icon_image_alt icon_image_expanded is_executing is_flapping last_check last_notification last_state_change latency low_flap_threshold max_check_attempts next_check notes notes_expanded notes_url notes_url_expanded notification_interval notification_period notifications_enabled obsess_over_service percent_state_change perf_data plugin_output process_performance_data retry_interval scheduled_downtime_depth state state_type modified_attributes_list last_time_critical last_time_ok last_time_unknown last_time_warning display_name host_display_name host_custom_variable_names host_custom_variable_values in_check_period in_notification_period host_parents

Filter: host_name = UAM

Filter: description = UAM

And: 2

OutputFormat: json

ResponseHeader: fixed16

';

[07:46:52,032][DEBUG][Thruk] using new connection

[07:46:52,058][DEBUG][Thruk] header: 200 915

[07:46:52,059][DEBUG][Thruk] query status 200

[07:46:52,059][DEBUG][Thruk] status: $VAR1 = 200;

[07:46:52,059][DEBUG][Thruk] msg: $VAR1 = 'OK. Reponse contains the queried data.';

[07:46:52,059][DEBUG][Thruk] < $VAR1 = [

[

1,

0,

'',

0,

'check:passive',

5,

0,

'24x7',

1,

0,

[],

1,

0,

'UAM',

'',

0,

[

'THRUK_NODE_ID',

'THRUK_BP_ID'

],

[

'node1',

'109'

],

0,

1,

[

'tag:noc',

'tag:business_process'

],

1,

0,

'',

0,

'UAM',

'Business Process: UAM',

0,

1,

0,

'OK - business process calculation of 22 nodes complete in 0.766s',

'runtime=0.766s',

1,

'check:passive',

[],

[

'tag:noc',

'tag:business_process',

'all_hosts'

],

1,

'',

0,

'UAM',

'',

0,

1,

'1492695942',

'',

0,

'1493797599',

0,

'1493757760',

0,

3,

'1493797709',

'This object is autogenerated by Business Process.',

'http://wiki.example.com/display/OPS/Monitoring/businessprocess/$SERVICEDESC$',

'http://wiki.example.com/display/OPS/Monitoring/businessprocess/UAM',

1440,

'24x7',

1,

0,

'',

'OK - everything is fine',

1,

2,

0,

1,

[],

'1493245186',

'1493797599',

0,

'1493757760',

'UAM',

[

'THRUK_NODE_ID',

'THRUK_BP_ID'

],

[

'node1',

'109'

],

1,

[]

]

];

[07:46:52,061][DEBUG][Thruk] livestatus: get_logs: b888b

[07:46:52,062][DEBUG][Thruk] selectall_arrayref("GET log\nColumns: time type message\nFilter: time >= 1492847212\nFilter: time <= 1493797612\nAnd: 2\nFilter: host_name = UAM\nFilter: type = HOST DOWNTIME ALERT\nAnd: 2\nFilter: host_name = UAM\nFilter: type = SERVICE ALERT\nFilter: state_type = HARD\nFilter: state = 0\nOr: 2\nFilter: type = INITIAL SERVICE STATE\nFilter: state_type = HARD\nFilter: state = 0\nOr: 2\nFilter: type = CURRENT SERVICE STATE\nFilter: state_type = HARD\nFilter: state = 0\nOr: 2\nFilter: type = SERVICE DOWNTIME ALERT\nOr: 7\nAnd: 2\nOr: 2", {'addpeer' => 0,'slice' => {}}, 0)

[07:46:52,062][DEBUG][Thruk] > $VAR1 = 'GET log

Columns: time type message

Filter: time >= 1492847212

Filter: time <= 1493797612

And: 2

Filter: host_name = UAM

Filter: type = HOST DOWNTIME ALERT

And: 2

Filter: host_name = UAM

Filter: type = SERVICE ALERT

Filter: state_type = HARD

Filter: state = 0

Or: 2

Filter: type = INITIAL SERVICE STATE

Filter: state_type = HARD

Filter: state = 0

Or: 2

Filter: type = CURRENT SERVICE STATE

Filter: state_type = HARD

Filter: state = 0

Or: 2

Filter: type = SERVICE DOWNTIME ALERT

Or: 7

And: 2

Or: 2

OutputFormat: json

ResponseHeader: fixed16

';

[07:46:52,062][DEBUG][Thruk] using new connection

[07:47:22,063][DEBUG][Thruk] header: 400 49

[07:47:22,064][DEBUG][Thruk] query status 400

[07:47:22,064][ERROR][Thruk] Error in: /thruk/cgi-bin/avail.cgi

[07:47:22,065][ERROR][Thruk] The request contains an invalid header. at /usr/share/thruk/lib/Monitoring/Livestatus/Class/Lite.pm line 378.

at /usr/lib64/thruk/perl5/Plack/Test/MockHTTP.pm line 24.

Plack::Test::MockHTTP::try {...} () called at /usr/lib64/thruk/perl5/Try/Tiny.pm line 98

eval {...} called at /usr/lib64/thruk/perl5/Try/Tiny.pm line 94

Try::Tiny::try('CODE(0x2f77cd0)', 'Try::Tiny::Catch=REF(0x307d138)') called at /usr/lib64/thruk/perl5/Plack/Test/MockHTTP.pm line 27

Plack::Test::MockHTTP::request('Plack::Test::MockHTTP=HASH(0x2d89f20)', 'HTTP::Request=HASH(0x2d7dfd8)') called at /usr/share/thruk/lib/Thruk/Utils/CLI.pm line 451

Thruk::Utils::CLI::_dummy_c(undef, '/thruk/cgi-bin/avail.cgi?host=UAM&service=UAM&timeperiod=last...') called at /usr/share/thruk/lib/Thruk/Utils/CLI.pm line 216

Thruk::Utils::CLI::request_url('Thruk::Context=HASH(0x307d3c0)', '/thruk/cgi-bin/avail.cgi?host=UAM&service=UAM&timeperiod=last...') called at /usr/share/thruk/lib/Thruk/Utils/CLI.pm line 1241

Thruk::Utils::CLI::_cmd_url('Thruk::Context=HASH(0x307d3c0)', undef, 'HASH(0x12bb000)') called at /usr/share/thruk/lib/Thruk/Utils/CLI.pm line 595

Thruk::Utils::CLI::_run_command_action('Thruk::Context=HASH(0x307d3c0)', 'HASH(0x12bb000)', 'local', 'url=avail.cgi?host=UAM&service=UAM&timeperiod=last7days&csvou...') called at /usr/share/thruk/lib/Thruk/Utils/CLI.pm line 545

Thruk::Utils::CLI::_run_commands('Thruk::Context=HASH(0x307d3c0)', 'HASH(0x12bb000)', 'local') called at /usr/share/thruk/lib/Thruk/Utils/CLI.pm line 465

Thruk::Utils::CLI::_from_local('Thruk::Utils::CLI=HASH(0x12bafd0)', 'Thruk::Context=HASH(0x307d3c0)', 'HASH(0x12bb000)') called at /usr/share/thruk/lib/Thruk/Utils/CLI.pm line 362

Thruk::Utils::CLI::_run('Thruk::Utils::CLI=HASH(0x12bafd0)') called at /bin/thruk line 84

[07:47:22,065][ERROR][Thruk] No Backend available

[07:47:22,065][ERROR][Thruk] on page: http://localhost/thruk/cgi-bin/avail.cgi?host=UAM&service=UAM&timeperiod=last7days&csvoutput=1

[07:47:22,065][ERROR][Thruk] Local Core: ERROR: The request contains an invalid header.. (/var/cache/naemon/lmd.sock)

[Wed May 3 07:47:22 2017][ERROR] No Backend available

[Wed May 3 07:47:22 2017][ERROR] None of the configured Backends could be reached, please have a look at the logfile for detailed information and make sure the core is up and running.

[Wed May 3 07:47:22 2017][ERROR] Local Core: ERROR: The request contains an invalid header.. (/var/cache/naemon/lmd.sock)

No Backend available at /usr/share/thruk/lib/Thruk/Controller/error.pm line 274.

Thruk::Controller::error::index('Thruk::Context=HASH(0x2d899e0)', 13) called at /usr/share/thruk/lib/Thruk.pm line 305

Thruk::_dispatcher('HASH(0x2d89a40)') called at /usr/lib64/thruk/perl5/Plack/Test/MockHTTP.pm line 24