Backslash problem in archives.json and dupe content message

[Fco. Javier C.]

Hi,

I am having a problem both in production (installed following the documentation, Wazuh 3.12.2) and in the laboratory using the Wazuh VM (wazuh-manager-3.10.2-1).


In the configuration file ( /var/ossec/etc/ossec.conf ) I have the following options enabled

   [...]

    <logall>yes</logall>
    <logall_json>yes</logall_json>

   [...]

  
  
The events in /var/ossec/logs/archives/archives.log are logged correctly, regardless of the type of event/source (eventchannel, SCA, Windows, Linux, etc). Also in archives.json and alerts.log / alerts.json

Example #1 archives.log

2020 Apr 27 08:42:44 (WIN00) any->EventChannel {"win":{"system":{"providerName":"Microsoft-Windows-Security-Auditing","providerGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","eventID":"4625","version":"0","level":"0","task":"12544","opcode":"0","keywords":"0x8010000000000000","systemTime":"2020-04-27T09:54:34.483472500Z","eventRecordID":"435211","processID":"460","threadID":"2756","channel":"Security","computer":"WIN00.DOMAIN.LOCAL","severityValue":"AUDIT_FAILURE","message":"Error de una cuenta al iniciar sesión."},"eventdata":{"subjectUserSid":"S-1-0-0","subjectLogonId":"0x0","targetUserSid":"S-1-0-0","targetUserName":"USUARIO_NO_EXISTE","status":"0xc000006d","failureReason":"%%2313","subStatus":"0xc0000064","logonType":"3","logonProcessName":"NtLmSsp","authenticationPackageName":"NTLM","workstationName":"Coco","keyLength":"0","processId":"0x0"}}}

Example #1 archives.json

{"timestamp":"2020-04-27T08:42:44.000+0000","rule":{"level":5,"description":"Logon Failure - Unknown user or bad password","id":"60122","firedtimes":1,"mail":false,"groups":["windows"," windows_security","win_authentication_failed"],"pci_dss":["10.2.4","10.2.5"],"gpg13":["7.1"],"gdpr":["IV_35.7.d","IV_32.2"],"hipaa":["164.312.b"],"nist_800_53":["AU.14","AC.7"]},"agent":{"id":"002","name":"WIN00","ip":"192.168.9.10"},"manager":{"name":"wazuhmanager"},"id":"1587976964.1390624","decoder":{"name":"windows_eventchannel"},"data":{"win":{"system":{"providerName":"Microsoft-Windows-Security-Auditing","providerGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","eventID":"4625","version":"0","level":"0","task":"12544","opcode":"0","keywords":"0x8010000000000000","systemTime":"2020-04-27T09:54:34.483472500Z","eventRecordID":"435211","processID":"460","threadID":"2756","channel":"Security","computer":"WIN00.DOMAIN.LOCAL","severityValue":"AUDIT_FAILURE","message":"Error de una cuenta al iniciar sesión."},"eventdata":{"subjectUserSid":"S-1-0-0","subjectLogonId":"0x0","targetUserSid":"S-1-0-0","targetUserName":"USUARIO_NO_EXISTE","status":"0xc000006d","failureReason":"%%2313","subStatus":"0xc0000064","logonType":"3","logonProcessName":"NtLmSsp","authenticationPackageName":"NTLM","workstationName":"Coco","keyLength":"0","processId":"0x0"}}},"location":"EventChannel"}

Example #1 alerts.json

{"timestamp":"2020-04-27T08:42:44.000+0000","rule":{"level":5,"description":"Logon Failure - Unknown user or bad password","id":"60122","firedtimes":1,"mail":false,"groups":["windows"," windows_security","win_authentication_failed"],"pci_dss":["10.2.4","10.2.5"],"gpg13":["7.1"],"gdpr":["IV_35.7.d","IV_32.2"],"hipaa":["164.312.b"],"nist_800_53":["AU.14","AC.7"]},"agent":{"id":"002","name":"WIN00","ip":"192.168.9.10"},"manager":{"name":"wazuhmanager"},"id":"1587976964.1390624","decoder":{"name":"windows_eventchannel"},"data":{"win":{"system":{"providerName":"Microsoft-Windows-Security-Auditing","providerGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","eventID":"4625","version":"0","level":"0","task":"12544","opcode":"0","keywords":"0x8010000000000000","systemTime":"2020-04-27T09:54:34.483472500Z","eventRecordID":"435211","processID":"460","threadID":"2756","channel":"Security","computer":"WIN00.DOMAIN.LOCAL","severityValue":"AUDIT_FAILURE","message":"Error de una cuenta al iniciar sesión."},"eventdata":{"subjectUserSid":"S-1-0-0","subjectLogonId":"0x0","targetUserSid":"S-1-0-0","targetUserName":"USUARIO_NO_EXISTE","status":"0xc000006d","failureReason":"%%2313","subStatus":"0xc0000064","logonType":"3","logonProcessName":"NtLmSsp","authenticationPackageName":"NTLM","workstationName":"Coco","keyLength":"0","processId":"0x0"}}},"location":"EventChannel"}

Ok, everything is correct, as the documentation indicates (archives.json/alerts.json "Alerts will be duplicated if you use both of these files. Also, note that both files receive fully decoded event data."  -- https://documentation.wazuh.com/3.12/getting-started/architecture.html )


Now, in the event that an event does not match any rule, something else happens to me. For another event that does not match in any rule, the following is logged:

Example #2 archives.log

2020 Apr 27 08:46:16 (WIN00) any->EventChannel {"win":{"system":{"providerName":"Microsoft-Windows-Security-Auditing","providerGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","eventID":"4672","version":"0","level":"0","task":"12548","opcode":"0","keywords":"0x8020000000000000","systemTime":"2020-04-27T09:58:07.358859200Z","eventRecordID":"435242","processID":"460","threadID":"1236","channel":"Security","computer":"WIN00.DOMAIN.LOCAL","severityValue":"AUDIT_SUCCESS","message":"Se asignaron privilegios especiales a un nuevo inicio de sesión."},"eventdata":{"subjectUserSid":"S-1-5-18","subjectUserName":"WIN00$","subjectDomainName":"DOMAIN","subjectLogonId":"0x7fd3f","privilegeList":"SeSecurityPrivilege     SeBackupPrivilege     SeRestorePrivilege     SeTakeOwnershipPrivilege     SeDebugPrivilege     SeSystemEnvironmentPrivilege     SeLoadDriverPrivilege     SeImpersonatePrivilege     SeEnableDelegationPrivilege"}}}

Example #2 archives.json ****  Duplicate content message in "full_log" and "data" ?!?! ( of course, If this  matched any rule it would apply "no_full_log" and the message was not duplicated.

{"timestamp":"2020-04-27T08:46:16.861+0000","agent":{"id":"002","name":"WIN00","ip":"192.168.9.10"},"manager":{"name":"wazuhmanager"},"id":"1587977176.1430203","full_log":"{\"win\":{\"system\":{\"providerName\":\"Microsoft-Windows-Security-Auditing\",\"providerGuid\":\"{54849625-5478-4994-A5BA-3E3B0328C30D}\",\"eventID\":\"4672\",\"version\":\"0\",\"level\":\"0\",\"task\":\"12548\",\"opcode\":\"0\",\"keywords\":\"0x8020000000000000\",\"systemTime\":\"2020-04-27T09:58:07.358859200Z\",\"eventRecordID\":\"435242\",\"processID\":\"460\",\"threadID\":\"1236\",\"channel\":\"Security\",\"computer\":\"WIN00.DOMAIN.LOCAL\",\"severityValue\":\"AUDIT_SUCCESS\",\"message\":\"Se asignaron privilegios especiales a un nuevo inicio de sesión.\"},\"eventdata\":{\"subjectUserSid\":\"S-1-5-18\",\"subjectUserName\":\"WIN00$\",\"subjectDomainName\":\"DOMAIN\",\"subjectLogonId\":\"0x7fd3f\",\"privilegeList\":\"SeSecurityPrivilege     SeBackupPrivilege     SeRestorePrivilege     SeTakeOwnershipPrivilege     SeDebugPrivilege     SeSystemEnvironmentPrivilege     SeLoadDriverPrivilege     SeImpersonatePrivilege     SeEnableDelegationPrivilege\"}}}","decoder":{"name":"windows_eventchannel"},"data":{"win":{"system":{"providerName":"Microsoft-Windows-Security-Auditing","providerGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","eventID":"4672","version":"0","level":"0","task":"12548","opcode":"0","keywords":"0x8020000000000000","systemTime":"2020-04-27T09:58:07.358859200Z","eventRecordID":"435242","processID":"460","threadID":"1236","channel":"Security","computer":"WIN00.DOMAIN.LOCAL","severityValue":"AUDIT_SUCCESS","message":"Se asignaron privilegios especiales a un nuevo inicio de sesión."},"eventdata":{"subjectUserSid":"S-1-5-18","subjectUserName":"WIN00$","subjectDomainName":"DOMAIN","subjectLogonId":"0x7fd3f","privilegeList":"SeSecurityPrivilege     SeBackupPrivilege     SeRestorePrivilege     SeTakeOwnershipPrivilege     SeDebugPrivilege     SeSystemEnvironmentPrivilege     SeLoadDriverPrivilege     SeImpersonatePrivilege     SeEnableDelegationPrivilege"}}},"location":"EventChannel"}

Another example:

Example #3 - archives.log

2020 Apr 27 10:49:18 (WIN00) any->EventChannel {"win":{"system":{"providerName":"Service Control Manager","providerGuid":"{555908d1-a6d7-4695-8e1e-26931d2012f4}","eventSourceName":"Service Control Manager","eventID":"7036","version":"0","level":"4","task":"0","opcode":"0","keywords":"0x8080000000000000","systemTime":"2020-04-27T12:01:09.904990500Z","eventRecordID":"2884","processID":"452","threadID":"32","channel":"System","computer":"WIN00.DOMAIN.LOCAL","severityValue":"INFORMATION","message":"El servicio Servicio de detección automática de proxy web WinHTTP entró en estado \"en ejecución\"."},"eventdata":{"param1":"Servicio de detección automática de proxy web WinHTTP","param2":"en ejecución","binary":"570069006E0048007400740070004100750074006F00500072006F00780079005300760063002F0034000000"}}}

Example #3 -archives.json  **** Duplicate message in "full_log" and "data" ?!?! ( of course, If this  matched any rule it would apply "no_full_log" and the message was not duplicated.

{"timestamp":"2020-04-27T10:49:18.881+0000","agent":{"id":"002","name":"WIN00","ip":"192.168.9.10"},"manager":{"name":"wazuhmanager"},"id":"1587984558.2605386","full_log":"{\"win\":{\"system\":{\"providerName\":\"Service Control Manager\",\"providerGuid\":\"{555908d1-a6d7-4695-8e1e-26931d2012f4}\",\"eventSourceName\":\"Service Control Manager\",\"eventID\":\"7036\",\"version\":\"0\",\"level\":\"4\",\"task\":\"0\",\"opcode\":\"0\",\"keywords\":\"0x8080000000000000\",\"systemTime\":\"2020-04-27T12:01:09.904990500Z\",\"eventRecordID\":\"2884\",\"processID\":\"452\",\"threadID\":\"32\",\"channel\":\"System\",\"computer\":\"WIN00.DOMAIN.LOCAL\",\"severityValue\":\"INFORMATION\",\"message\":\"El servicio Servicio de detección automática de proxy web WinHTTP entró en estado \\\"en ejecución\\\".\"},\"eventdata\":{\"param1\":\"Servicio de detección automática de proxy web WinHTTP\",\"param2\":\"en ejecución\",\"binary\":\"570069006E0048007400740070004100750074006F00500072006F00780079005300760063002F0034000000\"}}}","decoder":{"name":"windows_eventchannel"},"data":{"win":{"system":{"providerName":"Service Control Manager","providerGuid":"{555908d1-a6d7-4695-8e1e-26931d2012f4}","eventSourceName":"Service Control Manager","eventID":"7036","version":"0","level":"4","task":"0","opcode":"0","keywords":"0x8080000000000000","systemTime":"2020-04-27T12:01:09.904990500Z","eventRecordID":"2884","processID":"452","threadID":"32","channel":"System","computer":"WIN00.DOMAIN.LOCAL","severityValue":"INFORMATION","message":"El servicio Servicio de detección automática de proxy web WinHTTP entró en estado \"en ejecución\"."},"eventdata":{"param1":"Servicio de detección automática de proxy web WinHTTP","param2":"en ejecución","binary":"570069006E0048007400740070004100750074006F00500072006F00780079005300760063002F0034000000"}}},"location":"EventChannel"}

Why is it that non-matching events in any rule appear in both "full_log" and "data"  on archives.json and why I have so many backslash in archives.json or malformated archives.json "?

I have checked the configuration several times but I cannot see if it is an error of mine or a possible bug. I have also looked for info in the doc, forum, github, etc and I am aware of the backslash bug when mentioning processes and/or path... Maybe it's that I haven't searched well enough


Regards

[Antonio Manuel Fresneda Rodríguez]

Hi Fco. Javier.

As you mentioned, if you enable the logall_json option from the config file, in the archive.json you will have the information of the alert duplicated.

The reason why you are not seeing the full_log field in some archives is because there was a bug that provoked that when an alert was generated, the same output was written in the archive.json and in the alert.json files.

If you have a look at the rule, you can see that the option no_full_log is enabled. This option removes this field from the alert and due to this bug, also removed the field from the archive.

<rule id="60122" level="5">
    <if_sid>60105</if_sid>
    <field name="win.system.eventID">^529$|^4625$</field>
    <description>Logon Failure - Unknown user or bad password</description>
    <options>no_full_log</options>
    <group>win_authentication_failed,pci_dss_10.2.4,pci_dss_10.2.5,gpg13_7.1,gdpr_IV_35.7.d,gdpr_IV_32.2,hipaa_164.312.b,nist_800_53_AU.14,nist_800_53_AC.7,</group>
 </rule>

This is the cause because the full_log filed wasn' t in the archive.json file.

This bug has been fixed and this is de Pull Request that solved it: https://github.com/wazuh/wazuh/pull/4906.

I'm going to check the backslash problem and the data problem.

I hope that all your doubts have been solved. If you have any other questions don' t hesitate in ask them.

Kind regards,

Antonio

[Antonio Manuel Fresneda Rodríguez]

Hi again Fco. Javier.

The duplicated data (full_log and data fields) are normal. The archive.json file contain ALL the raw data.

Sorry for not mentioned this in the previous e-mail.

Regards.

[Fco. Javier C.]

Hello Antonio;

Don't worry, when I reread your message I had the feeling that you wanted to say that. In the end it happened to me that I did not see the issue/PR on Github :-( ... I will do more tests and see what happens.

What I don't quite understand is that an event that doesn't match any rule is logged in archives.json but:
- With the duplicate event (once in the "full_log" field and again in the "data" field) // That this may be normal, I don't care
- That the different "fields" within "full_log", all come with the backslash. // I don't like this jajaja

Thanks Antonio for your time and for responding.

----------------------------------------------

Hola Antonio;

No te preocupes, al releer tu mensaje tuve la sensación de que querías decir eso. Al final me ha pasado que no ví el issue/PR en Github y mira que llevo días con este tema  :-( ... haré más pruebas y a ver qué pasa.

Lo que me tiene más mosqueado es que un evento que no haga match en ninguna regla se registre en archives.json pero:

- Con el evento duplicado ( una vez en el campo "full_log" y otra vez en el "campo "data" )  // Que esto puede ser lo normal, no me preocupa

- Que los diferentes "campos" dentro de "full_log", venga todo con la barra invertida.   // Que esto no me mola jajaja

Gracias Antonio por tu tiempo y por responder.

[Antonio Manuel Fresneda Rodríguez]

Hi again Fco. Javier.

First of all, sorry for the late reply.

The backslash that you are seeing in the archive.json are normal because the  full_log field in the archive.json is a string (result of adding the alert in JSON format). If the backslashes are omitted, characters like " can interfere with the global JSON.

Those backlash shouldn't appear in the kibana app. 

Could you check that they are not showing in kibana?

Kind regards,

Antonio.

[Fco. Javier C.]

Hi  Antonio;

Sorry about the delay in responding. Thanks for the information, I have been seeing information that has misled me and considering the infrastructure I have, certain information was displaying correctly in Kibana but not in Graylog.

To your question, I confirm that everything is displayed correctly in the Wazuh application for Kibana.

Again, thanks for everything.

Regards