Elasticsearch and syscheck.diff errors

200 views

Skip to first unread message

Carlos Lopez

unread,

Nov 6, 2018, 9:36:59 AM11/6/18

to wa...@googlegroups.com

Hi all,

I am seeing three or four times a day, errors like these in my Elasticsearch cluster (Wazuh 3.6.1 release):

[2018-11-06T12:39:59,395][DEBUG][o.e.a.b.TransportShardBulkAction] [wazuh-alerts-3.x-2018.11.06][3] failed to execute bulk item (index) BulkShardRequest [[wazuh-alerts-3.x-2018.11.06][3]] containing [index {[wazuh-alerts-3.x-2018.11.06][wazuh][-KYJ6WYBZVcEt1C-ZqOc], source[n/a, actual length: [119.8kb], max length: 2kb]}]
java.lang.IllegalArgumentException: Document contains at least one immense term in field="syscheck.diff" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[49, 49, 54, 49, 56, 99, 49, 49, 54, 49, 56, 44, 50, 52, 50, 56, 57, 10, 60, 32, 97, 108, 101, 114, 116, 32, 116, 99, 112, 32]...', original message: bytes can be at most 32766 in length; got 59141
at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:798) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45]
at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:428) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45]
at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:392) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45]
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45]
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45]
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1602) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45]
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1221) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45]
at org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1053) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:999) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:847) ~[elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:749) ~[elasticsearch-6.4.2.jar:6.4.2]

Does this error appear because of the length of the message sent by the agent?. It would be strange because the process that generates the alert is syscheck.diff, but on the other side I see the following entry in ossec.log's agent file:

2018/11/06 12:41:57 ossec-syscheckd: ERROR: Unable to run `diff "/var/ossec/queue/diff/local/etc/cron.daily/.ids_upd_rules_www.swp/state.1541508114" "/var/ossec/queue/diff/local/etc/cron.daily/.ids_upd_rules_www.swp/last-entry" > "/var/ossec/queue/diff/local/etc/cron.daily/.ids_upd_rules_www.swp/diff.1541508117" 2> /dev/null`

I guess the mistake is due to this. Am I correct?
Regards,
C. L. Martinez

jesus.g...@wazuh.com

unread,

Nov 14, 2018, 5:43:21 AM11/14/18

to Wazuh mailing list

Hi Carlos,

Elasticsearch has reached its own limit for a single term. Take a look:

whose UTF8 encoding is longer than the max length 32766
...
bytes can be at most 32766 in length; got 59141

Regarding syscheckd the problem was similar, a so much large diff that we couldn’t manage properly.

Finally, I want to say that the Wazuh manager is who generates the alert, then Logstash/Filebeat read it then they send the alert to Elasticsearch.

We are studying this specific problem. A solution may be ignoring that field before indexing the alert, this way you have a copy in your rotated logs but you are not trying
to index that field in Elasticsearch. If this works for you, let me know.

Take a look at your mutate > remove_field section from your Logstash configuration file and just add syscheck.diff (not tested, it should work), then restart Logstash:

    mutate {
        remove_field => [ "syscheck.diff", "timestamp", "beat", "input_type", "tags", "count", "@version", "log", "offset", "type", "@src_ip", "host"]
    }