How does regex work with multiline messages?

[Dmitry Mikheev]

I have a cycle of rules:

  <rule id="160001" level="6">

    <if_sid>61070</if_sid>

    <field name="win.system.providerName">^MSSQLSERVER$</field>

    <description>Unclassisised MS SQL server audit event.</description>

    <group>pci_dss_10.2.1,</group>

  </rule>

  <rule id="160010" level="5">

    <if_sid>160001</if_sid>

    <field name="win.system.providerName">^MSSQLSERVER$</field>

    <field name="win.system.eventID">33205</field>

    <description>MS SQL unclassisised operation</descr>

    <group>pci_dss_10.2.1,</group>

  </rule>

  <rule id="160012" level="4">

    <if_sid>160010</if_sid>

    <field name="win.system.providerName">^MSSQLSERVER$</field>

    <field name="win.system.eventID">33205</field>

    <field name="win.system.message">^.*server_principal_name:dbadmin.*$</field>

    <description>MS SQL operation by dbadmin.</description>

    <group>pci_dss_10.2.1,</group>

  </rule>

Rule 160012 does not work:

Trying rule: 61070 - Windows application audit success event.

*Rule 61070 matched

*Trying child rules

Trying rule: 160001 - Unclassisised MS SQL server audit event.

*Rule 160001 matched

*Trying child rules

Trying rule: 160002 - MS SQL database backup.

Trying rule: 160010 - MS SQL unclassisised operation

*Rule 160010 matched

*Trying child rules

Trying rule: 160012 - MS SQL operation by dbadmin.

Trying rule: 160011 - MS SQL operation by app.

**Phase 3: Completed filtering (rules).

id: '160010'

level: '5'

description: 'MS SQL unclassisised operation

groups: '['windows', 'mssql']'

firedtimes: '1'

mail: 'False'

pci_dss: '['10.2.1']'

**Alert to be generated.

I think the difference is in the multi-line query, but https://regex101.com says it's ok.

What could be the reason?

Query text

{"win":{"system":{"providerName":"MSSQLSERVER","eventID":"33205","level":"0","task":"5","keywords":"0xa0000000000000","systemTime":"2024-08-05T14:29:04.269054600Z","eventRecordID":"254474","channel":"Application","computer":"ServerTest","severityValue":"AUDIT_SUCCESS","message":"\"Audit event: audit_schema_version:1\nevent_time:2024-08-05 14:29:03.7964116\nsequence_number:1\naction_id:SL  \nsucceeded:true\nis_column_permission:true\nsession_id:61\nserver_principal_id:268\ndatabase_principal_id:1\ntarget_server_principal_id:0\ntarget_database_principal_id:0\nobject_id:1037246750\nuser_defined_event_id:0\ntransaction_id:994220\nclass_type:U \nduration_milliseconds:0\nresponse_rows:0\naffected_rows:0\nclient_ip:local machine\npermission_bitmask:00000000000000000000000000000001\nsequence_group_id:287B0CF1-76E9-4560-9097-B053BBC07802\nsession_server_principal_name:dbadmin\nserver_principal_name:dbadmin\nserver_principal_sid:e049d381904ab84b8958db0a28df8d18\ndatabase_principal_name:dbo\ntarget_server_principal_name:\ntarget_server_principal_sid:\ntarget_database_principal_name:\nserver_instance_name:SERVERTEST\ndatabase_name:Data\nschema_name:dbo\nobject_name:msgs\nstatement:select * from Data.dbo.msgs\nadditional_information:\nuser_defined_information:\napplication_name:Microsoft SQL Server Management Studio - Query\nconnection_id:603B29EF-9CC4-4407-B062-52F9AA881B24\ndata_sensitivity_information:\nhost_name:SERVERTEST\n.\""},"eventdata":{"data":"audit_schema_version:1 event_time:2024-08-05 14:29:03.7964116 sequence_number:1 action_id:SL   succeeded:true is_column_permission:true session_id:61 server_principal_id:268 database_principal_id:1 target_server_principal_id:0 target_database_principal_id:0 object_id:1037246750 user_defined_event_id:0 transaction_id:994220 class_type:U  duration_milliseconds:0 response_rows:0 affected_rows:0 client_ip:local machine permission_bitmask:00000000000000000000000000000001 sequence_group_id:287B0CF1-76E9-4560-9097-B053BBC07802 session_server_principal_name:dbadmin server_principal_name:dbadmin server_principal_sid:e049d381904ab84b8958db0a28df8d18 database_principal_name:dbo target_server_principal_name: target_server_principal_sid: target_database_principal_name: server_instance_name:SERVERTEST database_name:Data schema_name:dbo object_name:msgs statement:select * from Data.dbo.msgs additional_information: user_defined_information: application_name:Microsoft SQL Server Management Studio - Query connection_id:603B29EF-9CC4-4407-B062-52F9AA881B24 data_sensitivity_information: host_name:SERVERTEST"}}}

[Diego Andrés Cappri]

Hi Dmitry,
let me share the proper documentation to set multi-line logs using a couple of options in ossec.conf:
- Activate log_format to multi-line-regex -> https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/localfile.html#log-format
- Set multiline_regex -> https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/localfile.html#multiline-regex

Hope this help.

Diego.

[Dmitry Mikheev]

Hi Diego, thanks for the hints.

I have already tried to write a decoder to parse multi-line messages.
Unfortunately, we are dealing with windows_eventchannel. I cannot make a child decoder for it. 

The problem is described: https://github.com/wazuh/wazuh/issues/3193  but there is still no solution. :(

That's why I can only do something through rules.
I work with text already reduced to one win.system.message, where the line break is marked as \n

I probably have
- or a problem with regex and I will be grateful if you tell me who else can check it. Because the recommended https://regex101.com or https://regexr.com/ do not find errors
- or I am not taking into account something else in rule 160012

[Diego Andrés Cappri]

Hi Dmitry,
my apologies for my delayed answer, Have you tried adapting your regex patterns to be more lenient or focused on the core data you need, rather than expecting the exact structure, to reduce the dependency on the message format and focuses on the presence of the string you're interested in. For example:
<field name="win.system.message">server_principal_name:dbadmin</field>

Hope this help.

[Dmitry Mikheev]

I found the problem. 

Need to pull everything into one line using the (?s) tag.

<field name="win.system.message">(?s)^.*server_principal_name:dbadmin.*$</field>

Thank you all

[Rajesh Kumar]

The issue you're experiencing with the Wazuh rule 160012 not matching is likely due to the regex pattern you are using to match the multi-line win.system.message field. The pattern you’ve written should technically work according to regex101, but there are a few considerations when working with multi-line logs in Wazuh:

Considerations:

1. Multi-Line Matching:

   • Wazuh processes log entries line by line. For multi-line log entries, especially those with newline characters \n, the regex pattern needs to account for them. Ensure that the configuration and processing in Wazuh handle multi-line messages correctly.

2. Escape Sequences:

   • When writing regex patterns in Wazuh rules, you might need to be cautious about escape sequences. If your regex pattern includes special characters (like \), ensure they are properly escaped in the rule file.

3. Rule Ordering and Inheritance:

   • The rule 160012 inherits from 160010, which itself inherits from 160001. If 160012 is not triggering, it might be due to how the rules are applied or the specificity of the patterns.
   • Consider adjusting the rule hierarchy or making the regex more specific to ensure that it captures the intended events.

Solutions:

1. Test Regex with Single Line:

   • Test the win.system.message regex as a single line without \n to see if that triggers correctly. You might need to simplify the regex to ensure it’s capturing the event you want.

2. Enable Multi-Line Processing:

   • If Wazuh isn't handling multi-line logs as you expect, you might need to enable or adjust multi-line log processing in the Wazuh agent configuration.

3. Check for Log Normalization:

   • Ensure that the log message is being normalized correctly before the rule is applied. Sometimes, the way the log is parsed or normalized can affect how the regex matches.

Alternative Approach:

If you suspect that the multi-line nature of the message is causing issues, try to modify your rule as follows:

<rule id="160012" level="4">
  <if_sid>160010</if_sid>
  <field name="win.system.providerName">^MSSQLSERVER$</field>
  <field name="win.system.eventID">33205</field>

  <pcre>.*server_principal_name:dbadmin.*</pcre>

  <description>MS SQL operation by dbadmin.</description>
  <group>pci_dss_10.2.1,</group>
</rule>

• Explanation: By using the <pcre> tag instead of <field name="win.system.message">, you directly leverage a Perl-compatible regular expression, which might better handle complex or multi-line patterns.

Testing:

1. Simulate the Log Entry:

   • Use the ossec-logtest tool in Wazuh to simulate the log entry and see if the rule matches correctly.
   • Command:

     /var/ossec/bin/ossec-logtest
     

   • Paste your log entry and observe how Wazuh processes it.

2. Verbose Logging:

   • Enable verbose logging in Wazuh to see exactly how the log is being processed and which part of the rule chain is failing.

By carefully testing and adjusting the regex and rule logic, you should be able to resolve the issue with 160012.

--
You received this message because you are subscribed to the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/c2d41ed6-f428-44c6-9052-88ab6b68f4b4n%40googlegroups.com.

[Dmitry Mikheev]

Thanks 

In the previous message I just gave an example of fixing it.

Note that your recommendation wazuh-logtest does not work directly with WindowsEvent. 

The problem is described: https://github.com/wazuh/wazuh/issues/3193

How to bypass it for the duration of the test https://groups.google.com/g/wazuh/c/6tutWb-IkNY/m/Y-v9GF_DBAAJ?utm_medium=email&utm_source=footer