Regex not working on own decoder

[Cristian Radu]

Hello guys,

I have written my own decoder and rule.

This is my decoder:

<decoder name="tmg_decoder">
    <program_name>tmg_decoder_program</program_name>
</decoder>

<decoder name="tmg-exchange">
  <parent>tmg_decoder</parent>
  <regex>^([\d.]+)\t([\w\(\)\\\@\.]+)\t([\w -\/;]+)\t\w\t[\w-]+\t[\w:]+\t\w+\t[\w-]+\t\-\t[\w.]+\t([\d.]+)\t([\d]+)\t\d+\t\d+\t\d+\t\w+\t(\w+)\t\w+\t([\w:\/.\-\?\=\&\%]+)\t[\w\/\.\-\;\ \=]+\t[\w\-]+\t\d+\t[\w]+\t[\w\ \-]+\t[\w\ \:\;\=\,\%]+\t[\w\-]+\t[\w\-\ ]+\t\w+\t(\w+).+</regex>
  <order> srcip,user,system_name,dstip,dstport,protocol,url,action</order>
</decoder>

Testing with wazuh-logtest. 

I created the regex using regex101.com

/var/ossec/bin/wazuh-logtest
Starting wazuh-logtest v4.2.5
Type one log per line

46.97.176.144   (LDAP)m25059    Android-SAMSUNG-SM-A530F/101.9  Y       2022-02-08      13:43:17        W3ReverseProxy  B-TMG1  -       webmail.alro.ro 10.0.3.102      443     685905  398     455     httpsTCP      POST    http://webmail.alro.ro/Microsoft-Server-ActiveSync?Cmd=Ping&User=slatina%5Cm25059&DeviceId=SEC1D47C937AC571&DeviceType=SamsungDevice    application/vnd.ms-sync.wbxml   Inet    200     0x40000008    Slatina ActiveSync      Req ID: 072ee0dc; Compression: client=Yes, server=No, compress rate=0% decompress rate=0% ; FBA cookie: exists=no, valid=no, updated=yes, logged off=no, client type=unknown, user activity=yes       External        Local Host      0x780   Allowed 2022-02-08 13:43:17     dcsl01.slatina.marcogroup.local Allowed -       -       -       -       -       -       -       0       -    0-       -       -       -       -       -       0       0       -       0       -       -       -       None    webmail.alro.ro 60196   -

** Wazuh-logtest error -1:
        ERROR: (1452): Syntax error on regex: '^([\d.]+)\t([\w\(\)\\\@\.]+)\t([\w -\/;]+)\t\w\t[\w-]+\t[\w:]+\t\w+\t[\w-]+\t\-\t[\w.]+\t([\d.]+)\t([\d]+)\t\d+\t\d+\t\d+\t\w+\t(\w+)\t\w+\t([\w:\/.\-\?\=\&\%]+)\t[\w\/\.\-\;\ \=]+\t[\w\-]+\t\d+\t[\w]+\t[\w\ \-]+\t[\w\ \:\;\=\,\%]+\t[\w\-]+\t[\w\-\ ]+\t\w+\t(\w+).+'
        ERROR: (2107): Decoder configuration error: 'tmg-exchange'.
        ERROR: (7311): Failure to initializing session

What am I missing with my regex? I checked what is allowed an not. 

Thanks,

Cristian

[Hanes Nahuel Sciarrone]

Hi Cristian,

I hope you are well, thank you for using Wazuh and sharing your question with the Wazuh mailing list. The first thing that I can see is regex101.com is using pcre2 type and in the regex field it doesn't specify this. I recommend you to rewrite the decoder as follows:

<decoder name="tmg_decoder">
    <program_name>tmg_decoder_program</program_name>
</decoder>

<decoder name="tmg-exchange">
  <parent>tmg_decoder</parent>

  <regex type="pcre2">^([\d.]+)\t([\w\(\)\\\@\.]+)\t([\w -\/;]+)\t\w\t[\w-]+\t[\w:]+\t\w+\t[\w-]+\t\-\t[\w.]+\t([\d.]+)\t([\d]+)\t\d+\t\d+\t\d+\t\w+\t(\w+)\t\w+\t([\w:\/.\-\?\=\&\%]+)\t[\w\/\.\-\;\ \=]+\t[\w\-]+\t\d+\t[\w]+\t[\w\ \-]+\t[\w\ \:\;\=\,\%]+\t[\w\-]+\t[\w\-\ ]+\t\w+\t(\w+).+</regex>
  <order> srcip,user,system_name,dstip,dstport,protocol,url,action</order>
</decoder>

Wazuh by default uses the osregex type for the regular expression. Please try this and let me know if it works or not. I will do some testing in the meantime.

Best regards

Hanes 

[Cristian Radu]

Hi Hanes,

Yes, thanks! You are right, I found that too with pcre2. The issue is that now when I test the log, a default decoder triggers instead of mine.

/var/ossec/bin/wazuh-logtest
Starting wazuh-logtest v4.2.5
Type one log per line

46.97.176.144   (LDAP)m25059    Android-SAMSUNG-SM-A530F/101.9  Y       2022-02-08      13:43:17        W3ReverseProxy  B-TMG1  -       webmail.alro.ro 10.0.3.102      443     685905  398     455     httpsTCP      POST    http://webmail.alro.ro/Microsoft-Server-ActiveSync?Cmd=Ping&User=slatina%5Cm25059&DeviceId=SEC1D47C937AC571&DeviceType=SamsungDevice    application/vnd.ms-sync.wbxml   Inet    200     0x40000008    Slatina ActiveSync      Req ID: 072ee0dc; Compression: client=Yes, server=No, compress rate=0% decompress rate=0% ; FBA cookie: exists=no, valid=no, updated=yes, logged off=no, client type=unknown, user activity=yes       External        Local Host      0x780   Allowed 2022-02-08 13:43:17     dcsl01.slatina.marcogroup.local Allowed -       -       -       -       -       -       -       0       -    0-       -       -       -       -       -       0       0       -       0       -       -       -       None    webmail.alro.ro 60196   -

**Phase 1: Completed pre-decoding.
        full event: '46.97.176.144      (LDAP)m25059    Android-SAMSUNG-SM-A530F/101.9  Y       2022-02-08      13:43:17        W3ReverseProxy  B-TMG1  -       webmail.alro.ro 10.0.3.102      443     685905398     455     https   TCP     POST    http://webmail.alro.ro/Microsoft-Server-ActiveSync?Cmd=Ping&User=slatina%5Cm25059&DeviceId=SEC1D47C937AC571&DeviceType=SamsungDevice    application/vnd.ms-sync.wbxmlInet     200     0x40000008      Slatina ActiveSync      Req ID: 072ee0dc; Compression: client=Yes, server=No, compress rate=0% decompress rate=0% ; FBA cookie: exists=no, valid=no, updated=yes, logged off=no, client type=unknown, user activity=yes       External        Local Host      0x780   Allowed 2022-02-08 13:43:17     dcsl01.slatina.marcogroup.local Allowed -       -       -       -       -       -    -0       -       0       -       -       -       -       -       -       0       0       -       0       -       -       -       None    webmail.alro.ro 60196   -'

**Phase 2: Completed decoding.
        name: 'cylance_threats'
        cylance_threats.av_industry: '455'
        cylance_threats.cylance_score: '685905'
        cylance_threats.file_name: '46.97.176.144       (LDAP)m25059    Android-SAMSUNG-SM-A530F/101.9  Y       2022-02-08      13:43:17        W3ReverseProxy  B-TMG1  -       webmail.alro.ro 10.0.3.102'
        cylance_threats.file_status: '443'
        cylance_threats.global_quarantined: 'https'
        cylance_threats.safelisted: 'TCP'
        cylance_threats.signature_status: '398'
        cylance_threats.signed: 'POST'

**Phase 3: Completed filtering (rules).
        id: '87050'
        level: '0'
        description: 'Cylance threats messages grouped.'
        groups: '['cylance']'
        firedtimes: '1'
        mail: 'False'

This is how my decoder looks like:

<decoder name="tmg_decoder">
    <program_name>tmg_decoder_program</program_name>
</decoder>

<decoder name="tmg-exchange">
  <parent>tmg_decoder</parent>

  <regex type="pcre2">([\d.]+)\t([\w\(\)\\\@\.]+)\t([\w -\/;]+)\t\w\t[\w-]+\t[\w:]+\t\w+\t[\w-]+\t\-\t[\w.]+\t([\d.]+)\t([\d]+)\t\d+\t\d+\t\d+\t\w+\t(\w+)\t\w+\t([\w:\/.\-\?\=\&\%]+)\t[\w\/\.\-\;\ \=]+\t[\w\-]+\t\d+\t[\w]+\t[\w\ \-]+\t[\w\ \:\;\=\,\%]+\t[\w\-]+\t[\w\-\ ]+\t\w+\t(\w+).+</regex>
  <order> srcip,user,system_name,dstip,dstport,protocol,url,action</order>
</decoder>

BR,

Cristian

[Hanes Nahuel Sciarrone]

Hi Cristian

As far as I can see the regex applied in the decoder doesn't work. I use the same tool you shared with me and applied some changes but I don't know what you mean by the action field in the decoder. I leave you a screenshot with the new regex that I made in the tool.

<decoder name="tmg_decoder">
    <program_name>tmg_decoder_program</program_name>
</decoder>

<decoder name="tmg-exchange">
  <parent>tmg_decoder</parent>

  <regex type="pcre2">^([\d.]+)[ ]+([\w\(\)\\\@\.]+)[ ]+([\w\-\/;.]+)[ ]+\w[ ]+[\w\-]+[ ]+[\w:]+[ ]+[\w]+[ ]+[\w-]+[ ]+-[ ]+[\w.]+ ([\d.]+)[ ]+([\d]+)[ ]+\d+[ ]+\d+[ ]+\d+[ ]+\w+[ ]+\w+[ ]+(\w+):([\w:\/.\-\?\=\&\%]+)[ ]+[\w\/\.\-\;=]+[ ]+[\w\-]+[ ]+\d+[ ]+[\w]+[ ]+</regex>
  <order> srcip,user,system_name,dstip,dstport,protocol,url</order>
</decoder>

Note that I do not pick up the field of action in the above decoder. I hope you find the information useful.

Best regards

Hanes.

[Cristian Radu]

Hi Hanes,

One of the issue noticed was that between fields I had <tab> in the original log file, and in yours multiple <spaces>. I changed my regex accordingly and it still does not work. Changed \t with [ ]+.

And my decoder conf file:

<decoder name="tmg_decoder">
    <program_name>tmg_decoder_program</program_name>
</decoder>

<decoder name="tmg-exchange">
  <parent>tmg_decoder</parent>

  <regex type="pcre2">^([\d.]+)[ ]+([\w\(\)\\\@\.]+)[ ]+([\w\-\/;.]+)[ ]+\w[ ]+[\w\-]+[ ]+[\w:]+[ ]+[\w]+[ ]+[\w-]+[ ]+-[ ]+[\w.]+ ([\d.]+)[ ]+([\d]+)[ ]+\d+[ ]+\d+[ ]+\w+[ ]+(\w+)[ ]+\w+[ ]+([\w:\/.\-\?\=\&\%]+)[ ]+[\w\/\.\-]+[ ]+\d+[ ]+\w+[ ]+\w+\s\w+\s+.+(Allowed|Denied).+</regex>
  <order>srcip,user,system_name,dstip,dstport,protocol,url,action</order>
</decoder>

Still no match for my log using the above decoder.

/var/ossec/bin/wazuh-logtest
Starting wazuh-logtest v4.2.5
Type one log per line

46.97.176.144      (LDAP)m25059    Android-SAMSUNG-SM-A530F/101.9  Y       2022-02-08      13:43:17        W3ReverseProxy  B-TMG1  -       webmail.alro.ro 10.0.3.102      443     685905398     455     https   TCP     POST    http://webmail.alro.ro/Microsoft-Server-ActiveSync?Cmd=Ping&User=slatina%5Cm25059&DeviceId=SEC1D47C937AC571&DeviceType=SamsungDevice    application/vnd.ms-sync.wbxmlInet     200     0x40000008      Slatina ActiveSync      Req ID: 072ee0dc; Compression: client=Yes, server=No, compress rate=0% decompress rate=0% ; FBA cookie: exists=no, valid=no, updated=yes, logged off=no, client type=unknown, user activity=yes       External        Local Host      0x780   Allowed 2022-02-08 13:43:17     dcsl01.slatina.marcogroup.local Allowed -       -       -       -       -       -    -0       -       0       -       -       -       -       -       -       0       0       -       0       -       -       -       None    webmail.alro.ro 60196   -

**Phase 1: Completed pre-decoding.
        full event: '46.97.176.144      (LDAP)m25059    Android-SAMSUNG-SM-A530F/101.9  Y       2022-02-08      13:43:17        W3ReverseProxy  B-TMG1  -       webmail.alro.ro 10.0.3.102      443     685905398     455     https   TCP     POST    http://webmail.alro.ro/Microsoft-Server-ActiveSync?Cmd=Ping&User=slatina%5Cm25059&DeviceId=SEC1D47C937AC571&DeviceType=SamsungDevice    application/vnd.ms-sync.wbxmlInet     200     0x40000008      Slatina ActiveSync      Req ID: 072ee0dc; Compression: client=Yes, server=No, compress rate=0% decompress rate=0% ; FBA cookie: exists=no, valid=no, updated=yes, logged off=no, client type=unknown, user activity=yes       External        Local Host      0x780   Allowed 2022-02-08 13:43:17     dcsl01.slatina.marcogroup.local Allowed -       -       -       -       -       -    -0       -       0       -       -       -       -       -       -       0       0       -       0       -       -       -       None    webmail.alro.ro 60196   -'

**Phase 2: Completed decoding.

        No decoder matched.

What am I doing wrong?

BR,

Cristian

[Hanes Nahuel Sciarrone]

HI Cristian,

Sorry for my delay in replying. In the previous rule, I passed you, I use space because the log you sent me didn't have tabs. If the original logs use tabs try replacing the [ ]+ with \t. The only problem I find is if you have any part of the logs with space instead of tab. Please let me know if you can do the rule.

Best regards

Hanes

[Cristian Radu]

Hi Hanes,

No worries. Yes, I noticed that in the original log I had tabs, but when I tested the log with logtest in there were spaces. I used \s+ for either space or tab. 

Thanks for the help anyway!

BR,

Cristian