Anomaly detection

[filip faredge]

Hello,

Did anyone encountered this issue when starting an anomaly detector ?

This is my first anomaly detector.
There are no errors thrown in the GUI  (stuck in initializing)

But I see this 

 cat /var/log/elasticsearch/elasticsearch.log  | grep -i -E "error|warn"

[2021-08-27T01:48:32,639][ERROR][c.a.o.a.u.AlertingException] [node-1] Alerting error: [.opendistro-alerting-config] IndexNotFoundException[no such index [.opendistro-alerting-config]]

[2021-08-27T01:49:51,597][WARN ][c.a.o.a.t.RCFResultTransportAction] [node-1] Anomaly Detector ZG9BhXsBKYXT_LNNvoLf com.amazon.opendistroforelasticsearch.ad.common.exception.ResourceNotFoundException: No checkpoints found for model id ZG9BhXsBKYXT_LNNvoLf_model_rcf_0

[2021-08-27T01:49:51,597][ERROR][c.a.o.a.t.AnomalyResultTransportAction] [node-1] Received an error from node g0-4WdW7TdmSzDGMHurViw while doing model inference for ZG9BhXsBKYXT_LNNvoLf

[2021-08-27T01:49:51,598][ERROR][c.a.o.a.AnomalyDetectorJobRunner] [node-1] InternalFailure happened when executing anomaly result action for ZG9BhXsBKYXT_LNNvoLf

        at org.elasticsearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:56) [elasticsearch-7.10.2.jar:7.10.2]

[2021-08-27T01:49:56,325][ERROR][c.a.o.a.u.AlertingException] [node-1] Alerting error: [.opendistro-alerting-config] IndexNotFoundException[no such index [.opendistro-alerting-config]]

My system 

Wazuh manager: 4.1.5  ( upgraded from 4.1.0 on Aug 10 2021 )

Open Distro for Elasticsearch:    opendistroforelasticsearch-1.13.2

Filebeat-OSS:    filebeat-7.10.2

Kibana: 7.10.2

Regards

Filip

[mayte...@wazuh.com]

Hi Filip,

 

Sorry for the late response.

 

I have replicated your issue. After creating a detector using the provided sample detectors, I got the same error logs:

 

[2021-09-06T09:49:54,517][ERROR][c.a.o.a.u.AlertingException] [node-1] Alerting error: [.opendistro-alerting-config] IndexNotFoundException[no such index [.opendistro-alerting-config]]

[2021-09-06T09:50:12,308][ERROR][c.a.o.a.u.AlertingException] [node-1] Alerting error: [.opendistro-alerting-config] IndexNotFoundException[no such index [.opendistro-alerting-config]]

[2021-09-06T09:59:50,404][WARN ][c.a.o.a.t.RCFResultTransportAction] [node-1] [.opendistro-anomaly-checkpoints] IndexNotFoundException[no such index [.opendistro-anomaly-checkpoints]]

[2021-09-06T09:59:50,405][ERROR][c.a.o.a.t.AnomalyResultTransportAction] [node-1] Received an error from node nVXSHhp3RWKH4L2iPSqPAA while doing model inference for 8YODunsBPd6kP9slkhR7

[2021-09-06T09:59:50,415][ERROR][c.a.o.a.AnomalyDetectorJobRunner] [node-1] InternalFailure happened when executing anomaly result action for 8YODunsBPd6kP9slkhR7

        at org.elasticsearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:56) [elasticsearch-7.10.2.jar:7.10.2]

[2021-09-06T10:15:14,437][ERROR][c.a.o.a.u.AlertingException] [node-1] Alerting error: [.opendistro-alerting-config] IndexNotFoundException[no such index [.opendistro-alerting-config]]

 

It seems there is a bug in Opendistro as it does not create the .opendistro-alerting-config index even though it is necessary.

 

As a workaround, go to the Alerting module and create a monitor to trigger the index creation (once the index is created, the monitor can be deleted).

 

# curl -k -u <user>:<pass> "https://localhost:9200/_cat/indices/.opendistro-alerting*?expand_wildcards=all&s=index"

green open .opendistro-alerting-alert-history-2021.09.06-1 bpOciJwRTGmZzeZR3teQ9w 1 0 0 0   208b   208b

green open .opendistro-alerting-alerts                     ErpXXMewQemU6-3GxUEGYg 1 0 0 0   208b   208b

green open .opendistro-alerting-config                     mrP36m-kRpCqnzHMiooX0w 1 0 0 1 12.9kb 12.9kb

 

I hope it helps.

 

Please keep us updated!

 

Best regards,

Mayte Ariza