Character Encoding errors in logstash

[Erik Vetters]

Hi,

I do not exaclty since when, but in the last days I receive quite often the following error message.

[2018-08-10T08:38:46,665][WARN ][logstash.codecs.json     ] Received an event that has a different character encoding than you configured. **** deleteed :expected_charset=>"UTF-8"}
[2018-08-10T08:38:46,665][ERROR][logstash.codecs.json     ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('\' (code 92)): was expecting double-quote to start field name

How to get rid  of this. There is no configured character encoding in the logstash config from wazuh.

Many Greetings

Erik

[Juanjo Jiménez]

Hello Erik,

By default, the charset for the JSON codec is UTF-8, that setting is correct.

May I ask you, did you added custom rules to the Wazuh manager, or edited the existing ones? The error means that, when trying to parse and process the event (an alert), Logstash encountered a problem on the JSON (a missing character).

Please let me know about it so we can try to narrow the cause of this error message.

Regards,

Juanjo

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/2e83188d-a6f5-4643-a81a-3054447185b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Erik Vetters]

Hi  Juanjo

sorry for the late response. Was busy with other stuff.

I'm still having this errors in both logstash.log and syslog (but that is another thread I think.). Yes I have custom rules and no I do not alter existing ones.

I have some rules and 2 decoders ... maybe that is the reason ?

I have the feeling that this 2 lines (the warn and the error) belongs togehter. (The input or the alert is the same). It looks like an agent is sending a different   character encoding which some server sending (?) logsthash could not parse it. Or the logfiles in windows are not utf-8

Maybe ?  How to solve this error. Especially that this lines are also in syslog and keeps generating alerts.

Many Greetings and appreciated for any help.

Erik

[Juanjo Jiménez]

Hello Erik,

One of the possible reasons is your custom rules and decoders; it could be the rule/decoder format or could be the logs or events that they’re parsing.

We can do the following. Check your custom rules and decoders, ensure that the format is appropriate and they are not missing any important character. If you want, paste here your custom rules and decoders, and we could take them a look to see if there’s something strange on them.

Another thing that could be useful is to paste here some of your alerts (from the alerts.json file) related to your custom rules/decoders, so we can see the JSON output to search possible missing characters or bad formatting.

Thanks for your patience.

Regards,
Juanjo

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/2e83188d-a6f5-4643-a81a-3054447185b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f099338a-1684-4e27-9eb8-677f1b85975a%40googlegroups.com.

[Erik Vetters]

Hi,

sorry for the late reply .. Here are the erros from logstash

[2018-12-18T13:02:46,717][WARN ][logstash.codecs.json     ] Received an event that has a different character encoding than you configured. {:text=>"{\\\"timestamp\\\":\\\"2018-12-18T13:02:45.498+0100\\\",\\\"rule\\\":{\\\"level\\\":5,\\\"description\\\":\\\"MS SQL Server Logon Failure.\\\",\\\"id\\\":\\\"18180\\\",\\\"firedtimes\\\":22,\\\"mail\\\":false,\\\"groups\\\":[\\\"windows\\\",\\\"win_authentication_failed\\\"],\\\"pci_dss\\\":[\\\"10.2.4\\\",\\\"10.2.5\\\"],\\\"gpg13\\\":[\\\"7.1\\\"],\\\"gdpr\\\":[\\\"IV_35.7.d\\\",\\\"IV_32.2\\\"]},\\\"agent\\\":{\\\"id\\\":\\\"050\\\",\\\"name\\\":\\\"SERVERNAME\\\"},\\\"manager\\\":{\\\"name\\\":\\\"FGSV2050\\\"},\\\"id\\\":\\\"1545134565.6901501610\\\",\\\"full_log\\\":\\\"2018 Dec 18 13:02:40 WinEvtLog: Application: AUDIT_FAILURE(18456): MSSQLSERVER: svc_sp_appl_fsg: FCE: SERVERNAME.DOMAIN.COM: Fehler bei der Anmeldung f\\xFCr den Benutzer 'DOM\\\\\\\\svc_sp_appl_fsg'.Ursache: Fehler beim \\xD6ffnen der explizit angegebenen Datenbank 'Suchdienstanwendung_DB_56a80e0ce8944ed4a9b498114343128a'. [CLIENT: <local machine>]  \\\",\\\"predecoder\\\":{\\\"program_name\\\":\\\"WinEvtLog\\\",\\\"timestamp\\\":\\\"2018 Dec 18 13:02:40\\\"},\\\"decoder\\\":{\\\"parent\\\":\\\"windows\\\",\\\"name\\\":\\\"windows\\\"},\\\"data\\\":{\\\"dstuser\\\":\\\"svc_sp_appl_fsg\\\",\\\"id\\\":\\\"18456\\\",\\\"status\\\":\\\"AUDIT_FAILURE\\\",\\\"data\\\":\\\"MSSQLSERVER\\\",\\\"system_name\\\":\\\"SERVERNAME.DOMAIN.COM\\\",\\\"type\\\":\\\"Application\\\"},\\\"location\\\":\\\"WinEvtLog\\\"}", :expected_charset=>"UTF-8"}
[2018-12-18T13:02:46,718][ERROR][logstash.codecs.json     ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('\' (code 92)): was expecting double-quote to start field name at [Source: (String)"{\"timestamp\":\"2018-12-18T13:02:45.498+0100\",\"rule\":{\"level\":5,\"description\":\"MS SQL Server Logon Failure.\",\"id\":\"18180\",\"firedtimes\":22,\"mail\":false,\"groups\":[\"windows\",\"win_authentication_failed\"],\"pci_dss\":[\"10.2.4\",\"10.2.5\"],\"gpg13\":[\"7.1\"],\"gdpr\":[\"IV_35.7.d\",\"IV_32.2\"]},\"agent\":{\"id\":\"050\",\"name\":\"SERVERNAME\"},\"manager\":{\"name\":\"FGSV2050\"},\"id\":\"1545134565.6901501610\",\"full_log\":\"2018 Dec 18 13:02:40 WinEvtLog: Application: AUDIT"[truncated 641 chars]; line: 1, column: 3]>, :data=>"{\\\"timestamp\\\":\\\"2018-12-18T13:02:45.498+0100\\\",\\\"rule\\\":{\\\"level\\\":5,\\\"description\\\":\\\"MS SQL Server Logon Failure.\\\",\\\"id\\\":\\\"18180\\\",\\\"firedtimes\\\":22,\\\"mail\\\":false,\\\"groups\\\":[\\\"windows\\\",\\\"win_authentication_failed\\\"],\\\"pci_dss\\\":[\\\"10.2.4\\\",\\\"10.2.5\\\"],\\\"gpg13\\\":[\\\"7.1\\\"],\\\"gdpr\\\":[\\\"IV_35.7.d\\\",\\\"IV_32.2\\\"]},\\\"agent\\\":{\\\"id\\\":\\\"050\\\",\\\"name\\\":\\\"SERVERNAME\\\"},\\\"manager\\\":{\\\"name\\\":\\\"FGSV2050\\\"},\\\"id\\\":\\\"1545134565.6901501610\\\",\\\"full_log\\\":\\\"2018 Dec 18 13:02:40 WinEvtLog: Application: AUDIT_FAILURE(18456): MSSQLSERVER: svc_sp_appl_fsg: FCE: SERVERNAME.DOMAIN.COM: Fehler bei der Anmeldung f\\xFCr den Benutzer 'DOM\\\\\\\\svc_sp_appl_fsg'.Ursache: Fehler beim \\xD6ffnen der explizit angegebenen Datenbank 'Suchdienstanwendung_DB_56a80e0ce8944ed4a9b498114343128a'. [CLIENT: <local machine>]  \\\",\\\"predecoder\\\":{\\\"program_name\\\":\\\"WinEvtLog\\\",\\\"timestamp\\\":\\\"2018 Dec 18 13:02:40\\\"},\\\"decoder\\\":{\\\"parent\\\":\\\"windows\\\",\\\"name\\\":\\\"windows\\\"},\\\"data\\\":{\\\"dstuser\\\":\\\"svc_sp_appl_fsg\\\",\\\"id\\\":\\\"18456\\\",\\\"status\\\":\\\"AUDIT_FAILURE\\\",\\\"data\\\":\\\"MSSQLSERVER\\\",\\\"system_name\\\":\\\"SERVERNAME.DOMAIN.COM\\\",\\\"type\\\":\\\"Application\\\"},\\\"location\\\":\\\"WinEvtLog\\\"}"}

And the alerts.json logs like this

{"timestamp":"2018-12-18T13:02:47.257+0100","rule":{"level":2,"description":"Unknown problem somewhere in the system.","id":"1002","firedtimes":94,"mail":false,"groups":["syslog","errors"],"gpg13":["4.3"]},"agent":{"id":"000","name":"MANAGERNAME"},"manager":{"name":"MANAGERNAME"},"id":"1545134567.6901628784","full_log":"Dec 18 13:02:46 MANAGERNAME logstash[25495]: [2018-12-18T13:02:46,717][WARN ][logstash.codecs.json     ] Received an event that has a different character encoding than you configured. {:text=>\"{\\\\\\\"timestamp\\\\\\\":\\\\\\\"2018-12-18T13:02:45.498+0100\\\\\\\",\\\\\\\"rule\\\\\\\":{\\\\\\\"level\\\\\\\":5,\\\\\\\"description\\\\\\\":\\\\\\\"MS SQL Server Logon Failure.\\\\\\\",\\\\\\\"id\\\\\\\":\\\\\\\"18180\\\\\\\",\\\\\\\"firedtimes\\\\\\\":22,\\\\\\\"mail\\\\\\\":false,\\\\\\\"groups\\\\\\\":[\\\\\\\"windows\\\\\\\",\\\\\\\"win_authentication_failed\\\\\\\"],\\\\\\\"pci_dss\\\\\\\":[\\\\\\\"10.2.4\\\\\\\",\\\\\\\"10.2.5\\\\\\\"],\\\\\\\"gpg13\\\\\\\":[\\\\\\\"7.1\\\\\\\"],\\\\\\\"gdpr\\\\\\\":[\\\\\\\"IV_35.7.d\\\\\\\",\\\\\\\"IV_32.2\\\\\\\"]},\\\\\\\"agent\\\\\\\":{\\\\\\\"id\\\\\\\":\\\\\\\"050\\\\\\\",\\\\\\\"name\\\\\\\":\\\\\\\"FCSV074\\\\\\\"},\\\\\\\"manager\\\\\\\":{\\\\\\\"name\\\\\\\":\\\\\\\"MANAGERNAME\\\\\\\"},\\\\\\\"id\\\\\\\":\\\\\\\"1545134565.6901501610\\\\\\\",\\\\\\\"full_log\\\\\\\":\\\\\\\"2018 Dec 18 13:02:40 WinEvtLog: Application: AUDIT_FAILURE(18456): MSSQLSERVER: svc_sp_appl_fsg: DOM: SERVERNAME.DOM.COM: Fehler bei der Anmeldung f\\\\xFCr den Benutzer 'DOM\\\\\\\\\\\\\\\\svc_sp_appl_fsg'.Ursache: Fehler beim \\\\xD6ffnen der explizit angegebenen Datenbank 'Suchdienstanwendung_DB_56a80e0ce8944ed4a9b498114343128a'. [CLIENT: <local machine>]  \\\\\\\",\\\\\\\"predecoder\\\\\\\":{\\\\\\\"program_name\\\\\\\":\\\\\\\"WinEvtLog\\\\\\\",\\\\\\\"timestamp\\\\\\\":\\\\\\\"2018 Dec 18 13:02:40\\\\\\\"},\\\\\\\"decoder\\\\\\\":{\\\\\\\"parent\\\\\\\":\\\\\\\"windows\\\\\\\",\\\\\\\"name\\\\\\\":\\\\\\\"windows\\\\\\\"},\\\\\\\"data\\\\\\\":{\\\\\\\"dstuser\\\\\\\":\\\\\\\"svc_sp_appl_fsg\\\\\\\",\\\\\\\"id\\\\\\\":\\\\\\\"18456\\\\\\\",\\\\\\\"status\\\\\\\":\\\\\\\"AUDIT_FAILURE\\\\\\\",\\\\\\\"data\\\\\\\":\\\\\\\"MSSQLSERVER\\\\\\\",\\\\\\\"system_name\\\\\\\":\\\\\\\"SERVERNAME.DOM.COM\\\\\\\",\\\\\\\"type\\\\\\\":\\\\\\\"Application\\\\\\\"},\\\\\\\"location\\\\\\\":\\\\\\\"WinEvtLog\\\\\\\"}\", :expected_charset=>\"UTF-8\"}","predecoder":{"program_name":"logstash","timestamp":"Dec 18 13:02:46","hostname":"MANAGERNAME"},"decoder":{},"location":"/var/log/syslog"}
{"timestamp":"2018-12-18T13:02:47.257+0100","rule":{"level":2,"description":"Unknown problem somewhere in the system.","id":"1002","firedtimes":96,"mail":false,"groups":["syslog","errors"],"gpg13":["4.3"]},"agent":{"id":"000","name":"MANAGERNAME"},"manager":{"name":"MANAGERNAME"},"id":"1545134567.6901631012","full_log":"Dec 18 13:02:46 MANAGERNAME logstash[25495]:  at [Source: (String)\"{\\\"timestamp\\\":\\\"2018-12-18T13:02:45.498+0100\\\",\\\"rule\\\":{\\\"level\\\":5,\\\"description\\\":\\\"MS SQL Server Logon Failure.\\\",\\\"id\\\":\\\"18180\\\",\\\"firedtimes\\\":22,\\\"mail\\\":false,\\\"groups\\\":[\\\"windows\\\",\\\"win_authentication_failed\\\"],\\\"pci_dss\\\":[\\\"10.2.4\\\",\\\"10.2.5\\\"],\\\"gpg13\\\":[\\\"7.1\\\"],\\\"gdpr\\\":[\\\"IV_35.7.d\\\",\\\"IV_32.2\\\"]},\\\"agent\\\":{\\\"id\\\":\\\"050\\\",\\\"name\\\":\\\"FCSV074\\\"},\\\"manager\\\":{\\\"name\\\":\\\"MANAGERNAME\\\"},\\\"id\\\":\\\"1545134565.6901501610\\\",\\\"full_log\\\":\\\"2018 Dec 18 13:02:40 WinEvtLog: Application: AUDIT\"[truncated 641 chars]; line: 1, column: 3]>, :data=>\"{\\\\\\\"timestamp\\\\\\\":\\\\\\\"2018-12-18T13:02:45.498+0100\\\\\\\",\\\\\\\"rule\\\\\\\":{\\\\\\\"level\\\\\\\":5,\\\\\\\"description\\\\\\\":\\\\\\\"MS SQL Server Logon Failure.\\\\\\\",\\\\\\\"id\\\\\\\":\\\\\\\"18180\\\\\\\",\\\\\\\"firedtimes\\\\\\\":22,\\\\\\\"mail\\\\\\\":false,\\\\\\\"groups\\\\\\\":[\\\\\\\"windows\\\\\\\",\\\\\\\"win_authentication_failed\\\\\\\"],\\\\\\\"pci_dss\\\\\\\":[\\\\\\\"10.2.4\\\\\\\",\\\\\\\"10.2.5\\\\\\\"],\\\\\\\"gpg13\\\\\\\":[\\\\\\\"7.1\\\\\\\"],\\\\\\\"gdpr\\\\\\\":[\\\\\\\"IV_35.7.d\\\\\\\",\\\\\\\"IV_32.2\\\\\\\"]},\\\\\\\"agent\\\\\\\":{\\\\\\\"id\\\\\\\":\\\\\\\"050\\\\\\\",\\\\\\\"name\\\\\\\":\\\\\\\"FCSV074\\\\\\\"},\\\\\\\"manager\\\\\\\":{\\\\\\\"name\\\\\\\":\\\\\\\"MANAGERNAME\\\\\\\"},\\\\\\\"id\\\\\\\":\\\\\\\"1545134565.6901501610\\\\\\\",\\\\\\\"full_log\\\\\\\":\\\\\\\"2018 Dec 18 13:02:40 WinEvtLog: Application: AUDIT_FAILURE(18456): MSSQLSERVER: svc_sp_appl_fsg: DOM: fcsv074.DOM.com: Fehler bei der Anmeldung f\\\\xFCr den Benutzer 'DOM\\\\\\\\\\\\\\\\svc_sp_appl_fsg'.Ursache: Fehler beim \\\\xD6ffnen der explizit angegebenen Datenbank 'Suchdienstanwendung_DB_56a80e0ce8944ed4a9b498114343128a'. [CLIENT: <local machine>]  \\\\\\\",\\\\\\\"predecoder\\\\\\\":{\\\\\\\"program_name\\\\\\\":\\\\\\\"WinEvtLog\\\\\\\",\\\\\\\"timestamp\\\\\\\":\\\\\\\"2018 Dec 18 13:02:40\\\\\\\"},\\\\\\\"decoder\\\\\\\":{\\\\\\\"parent\\\\\\\":\\\\\\\"windows\\\\\\\",\\\\\\\"name\\\\\\\":\\\\\\\"windows\\\\\\\"},\\\\\\\"data\\\\\\\":{\\\\\\\"dstuser\\\\\\\":\\\\\\\"svc_sp_appl_fsg\\\\\\\",\\\\\\\"id\\\\\\\":\\\\\\\"18456\\\\\\\",\\\\\\\"status\\\\\\\":\\\\\\\"AUDIT_FAILURE\\\\\\\",\\\\\\\"data\\\\\\\":\\\\\\\"MSSQLSERVER\\\\\\\",\\\\\\\"system_name\\\\\\\":\\\\\\\"fcsv074.DOM.com\\\\\\\",\\\\\\\"type\\\\\\\":\\\\\\\"Application\\\\\\\"},\\\\\\\"location\\\\\\\":\\\\\\\"WinEvtLog\\\\\\\"}\"}","predecoder":{"program_name":"logstash","timestamp":"Dec 18 13:02:46","hostname":"MANAGERNAME"},"decoder":{},"location":"/var/log/syslog"}

And I have 2 special decoders ... and a lot of rules, I can share them via PM. 

<decoder name="windows_fields">
    <type>windows</type>
    <parent>windows</parent>
    <regex>Creating Scriptblock text \(\.*\)(\.*)\s+ScriptBlock ID:(\.*)\s+Path:(\.*)</regex>
    <order>powershell.data,powershell.id,powershell.path</order>
</decoder>

<decoder name="windows_fields">
    <type>windows</type>
    <parent>windows</parent>
    <regex>User: (\.*)\s*</regex>
    <order>srcuser</order>
</decoder>

Maybe someone can help me. This drives me nuts. I make a rules which blocks this. Keeps spamming the system ;-)

Many Greetings

Erik

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/2e83188d-a6f5-4643-a81a-3054447185b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

[Juanjo Jiménez]

Hello Erik,

It looks like your alerts contain non-UTF-8 characters. By default, Logstash expects UTF-8 characters. As we can see here from your logs, I guess some German characters are not being properly parsed:

“Fehler bei der Anmeldung f\xFCr den Benutzer “ (I suspect it’s ü).

We can try the following. Open your Logstash configuration file (/etc/logstash/conf.d/01-wazuh.conf) and replace the codec on the input/beats section with this:

codec => "json_lines" {
    charset => "ISO-8859-1"
}

Save the file and restart Logstash:

systemctl restart logstash

Let me know if this helps to solve your problem.

Regards,

Juanjo Jiménez Software Engineer  The Open Source Security Platform

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/2e83188d-a6f5-4643-a81a-3054447185b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f099338a-1684-4e27-9eb8-677f1b85975a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/26f5937e-8b0f-4bd1-937f-f2fa05024cf4%40googlegroups.com.