Filter message body in elastic search using Kibana

[riiky devils]

Hi Team,

I'm curious how to filter message body in ELK using kibana because i have too many XML line in JSON message

This is my example JSON alert in kibana UI

"message": "\"The Federation Service failed to issue a valid token. See XML for failure details. \r\n\r\nActivity ID: nbnbnbnb-jkaskjka-iauous \r\n\r\nAdditional Data \r\nXML: <?xml version=\"1.0\" encoding=\"utf-16\"?>\r\n<AuditBase xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:type=\"AppTokenAudit\">\r\n  <AuditType>AppToken</AuditType>\r\n  <AuditResult>Failure</AuditResult>\r\n  <FailureType>IssuanceAuthZError</FailureType>\r\n  <ErrorCode>N/A</ErrorCode>\r\n  <ContextComponents>\r\n    <Component xsi:type=\"ResourceAuditComponent\">\r\n      <RelyingParty>hghghhas-oioiosa</RelyingParty>\r\n      <ClaimsProvider>N/A</ClaimsProvider>\r\n      <UserId>domain\user</UserId>\r\n    </Component>\r\n    <Component xsi:type=\"AuthNAuditComponent\">\r\n      <PrimaryAuth>N/A</PrimaryAuth>\r\n      <DeviceAuth>false</DeviceAuth>\r\n      <DeviceId>N/A</DeviceId>\r\n      <MfaPerformed>false</MfaPerformed>\r\n      <MfaMethod>N/A</MfaMethod>\r\n      <TokenBindingProvidedId>false</TokenBindingProvidedId>\r\n      <TokenBindingReferredId>false</TokenBindingReferredId>\r\n      <SsoBindingValidationLevel>NotSet</SsoBindingValidationLevel>\r\n    </Component>\r\n    <Component xsi:type=\"ProtocolAuditComponent\">\r\n      <OAuthClientId>N/A</OAuthClientId>\r\n      <OAuthGrant>N/A</OAuthGrant>\r\n    </Component>\r\n    <Component xsi:type=\"RequestAuditComponent\">\r\n      <Server>https://axfs.aaaa.net/adfs</Server>\r\n      <AuthProtocol>OAuth</AuthProtocol>\r\n      <NetworkLocation>Extranet</NetworkLocation>\r\n      <IpAddress>10.xx.xx.xx</IpAddress>\r\n      <ProxyServer>10.xx.xx.xx</ProxyServer>\r\n      <UserAgentString>Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36</UserAgentString>\r\n      <Endpoint>/adfs/oauth2/authorize/</Endpoint>\r\n    </Component>\r\n  </ContextComponents>\r\n</AuditBase>\"",

i wanto to extract just specific information from message field like <UserId>, <NetworkLocation> and <IpAddress>

i know windows_eventchannel decoder can't be extended so we can't parse win.eventdata.data to get IP address, Proxy, UserId, etc so i think if we can filter by display 

I found other example for using pipe line processors to extract the specific information from message field (https://stackoverflow.com/questions/63152207/filter-message-body-in-elastic-search-using-kibana)

Is this possible to achieve that from my message field?

Thank You,

[elw...@wazuh.com]

Hello riiky,

You can achieve your requirement using scripted fields (https://www.elastic.co/blog/using-painless-kibana-scripted-fields) in Kibana. I will describe below an example which you can adapt to your needs:

• The windows event has a field Message containing sub-information similar to yours:
  
  { "win": { "system": { "providerName": "Microsoft-Windows-Security-Auditing", "providerGuid": "{54849625-5478-4994-A5BA-3E3B0328C30D}", "eventID": "4656", "version": "1", "level": "0", "task": "12812", "opcode": "0", "keywords": "0x8020000000000000", "systemTime": "2020-03-03T12:41:53.308655200Z", "eventRecordID": "37277348", "processID": "560", "threadID": "568", "channel": "Security", "computer": "testhost.testdomain.Local", "severityValue": "AUDIT_SUCCESS", "message": "\"A handle to an object was requested.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\ttesthost$\r\n\tAccount Domain:\t\ttestdomain\r\n\tLogon ID:\t\t0x3E7\r\n\r\nObject:\r\n\tObject Server:\t\tSecurity\r\n\tObject Type:\t\tFile\r\n\tObject Name:\t\t\\Device\\Floppy0\r\n\tHandle ID:\t\t0x624\r\n\tResource Attributes:\t-\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x570\r\n\tProcess Name:\t\tC:\\Program Files\\VMware\\VMware Tools\\vmtoolsd.exe\r\n\r\nAccess Request Information:\r\n\tTransaction ID:\t\t{00000000-0000-0000-0000-000000000000}\r\n\tAccesses:\t\tREAD_CONTROL\r\n\t\t\t\tSYNCHRONIZE\r\n\t\t\t\tReadData (or ListDirectory)\r\n\t\t\t\tReadEA\r\n\t\t\t\tReadAttributes\r\n\t\t\t\t\r\n\tAccess Reasons:\t\t-\r\n\tAccess Mask:\t\t0x120089\r\n\tPrivileges Used for Access Check:\t-\r\n\tRestricted SID Count:\t0\"" }, "eventdata": { "subjectUserSid": "S-1-5-18", "subjectUserName": "testhost$", "subjectDomainName": "testdomain", "subjectLogonId": "0x3e7", "objectServer": "Security", "objectType": "File", "objectName": "\\\\Device\\\\Floppy0", "handleId": "0x624", "transactionId": "{00000000-0000-0000-0000-000000000000}", "accessList": "%%1538 %%1541 %%4416 %%4419 %%4423", "accessMask": "0x120089", "restrictedSidCount": "0", "processId": "0x570", "processName": "C:\\\\Program Files\\\\VMware\\\\VMware Tools\\\\vmtoolsd.exe" }, "keywords": { "keyword": "Audit Success" } } }
  
  
• my painless script to split the message and extract the first part :
  
  if (doc['data.win.system.message'].size()==0) { return "doc with no win Message " } else { return doc['data.win.system.message'].value.splitOnToken('.')[0]; }
  
  
  
  
• As a result, I will get a new field named WinMessageTitle extracted from the original data.win.system.message:
  
  
  
  
  
• As a result, I will get a new field named WinMessageTitle extracted from the original data.win.system.message:
  
  
  

The same process can be adapted to any other use case.

Ressources for your reference about scripted fields :

• https://www.elastic.co/guide/en/kibana/current/scripted-fields.html

• https://www.elastic.co/blog/using-painless-kibana-scripted-fields

• https://qbox.io/blog/painless-scripting-elasticsearch-kibana-scripted-fields

Hope this helps.

Regards,
Wali

[riiky devils]

Hi Wali,

Thanks for your reference. But i'm still confuse about create this painless script because none of work to get substring between 2 string information in message field.

I think because this field not in keyword type?

Can you give me example to extract those information?

Thank You,

[elw...@wazuh.com]

Hello riiky,

I do not have the exact script for your event but I think that you should use the matcher and return the match. Following are few examples that I have used for different purposes which I hope can be useful :


def m = /^.*\.([a-z]+)$/.matcher(doc['host.keyword'].value); if ( m.matches() ) { return m.group(1) } else { return "no match" }


def ref = doc['http.request.referer'].value;
 ref.contains('www') ? ref.splitOnToken('/')[2].replace('www.', '') : ref.splitOnToken('/')[2];


def codeMap = ['2': 'Informational Response', '2': 'Successful', '3': 'Redirect', '4': 'Client Error', '5': 'Server Error'];
 return codeMap[doc['http.response.status_code'].value.charAt(0).toString()];


def m = /.*(windows|darwin).*/.matcher(doc['url.path'].value);
 return m.matches() ? m.group(1) : 'linux';


It might be challenging to build the exact script for your use case but you can always refer to the provided links/docs to do so.

Note that the best approach would be (if possible) to modify the application log source in your windows server to change the format or the way it dumps the fields.

Hope this helps,

Regards,
Wali