prematch not working

[Sk Joherul Haque]

Hi All,

Trying to decode log line:

2021-09-26 06:08:33,469 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server

with decoder conf:

```

<decoder name="yarn">
    <prematch>^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d,\d\d\d \w+ </prematch>
</decoder>

<decoder name="yarn">
    <parent>yarn</parent>
    <regex>^(\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d,\d\d\d) (\w+) org.apache.hadoop.yarn.(\w*) \((\w*)\): (\.*)$</regex>
    <order>datetime,flag,component,context,msg</order>
</decoder>

```

logtest:

```

Starting wazuh-logtest v4.2.1
Type one log per line

2021-09-26 06:08:33,469 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server

**Phase 1: Completed pre-decoding.
        full event: '2021-09-26 06:08:33,469 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server'
        timestamp: '2021-09-26 06:08:33,46'

**Phase 2: Completed decoding.
        No decoder matched.

```

Am I doing something wrong? Any help is appreciated ...

Regards

Joher

[elw...@wazuh.com]

Hello Joher,

Based on your regex I am assuming that the logs always start with org.apache.hadoop.yarn, the following decoder/rule should help with your requirement :

Decoder :

<decoder name="yarn">
    <prematch>org.apache.hadoop.yarn</prematch>

</decoder>

<decoder name="yarn">
    <parent>yarn</parent>

    <regex>\((\w*)\): (\.*)$</regex>
    <order>context,msg</order>
</decoder>


Rule:

<group name="yarn">
<rule id="100010" level="3">
  <decoded_as>yarn</decoded_as>
  <description>yarn alerts</description>
</rule>
</group>

Test result:

2021-09-26 06:08:33,469 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server


**Phase 1: Completed pre-decoding.
       full event: '2021-09-26 06:08:33,469 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server'
       timestamp: '2021-09-26 06:08:33,46'

       hostname: 'INFO'
       program_name: '(null)'
       log: 'org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server'

**Phase 2: Completed decoding.
       decoder: 'yarn'
       context: 'main'
       msg: 'Connecting to Application History server'

**Phase 3: Completed filtering (rules).
       Rule id: '100010'
       Level: '3'
       Description: 'yarn alerts'
**Alert to be generated.


You can find more details about creating custom decoders/rules in the following resources:

• https://documentation.wazuh.com/current/user-manual/ruleset/custom.html
  
• https://wazuh.com/blog/creating-decoders-and-rules-from-scratch/
  

Hope this helps.

Regards,
Wali

[Joher]

Thank You very much

But no "INFO" field. this field is very important for us (debug/info/warn/error/fatal)

[elw...@wazuh.com]

Hello Joher,

I believe that the same question has been asked and addressed in both slack and GitHub, where it has been explained that the field "INFO" is part of the pre-decoding phase and currently is not possible to manipulate in the decoding phase. However, if the goal is to trigger alerts based on the type (INFO or DEBUG etc) of logs, you can leverage the use of `hostname` in this specific case in the rules and it can be used as follows :

Rules:

<group name="yarn">
<rule id="100010" level="3">
  <decoded_as>yarn</decoded_as>
  <description>yarn alerts</description>
</rule>

<rule id="100011" level="3">
  <if_sid>100010</if_sid>
  <hostname>INFO</hostname>
  <description>INFO yarn alerts</description>
</rule>

<rule id="100012" level="3">
  <if_sid>100010</if_sid>
  <hostname>DEBUG</hostname>
  <description>DEBUG yarn alerts</description>
</rule>

</group>


Results:

2021-09-26 06:08:33,469 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History serve

**Phase 1: Completed pre-decoding.
       full event: '2021-09-26 06:08:33,469 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History serve'

       timestamp: '2021-09-26 06:08:33,46'
       hostname: 'INFO'
       program_name: '(null)'

       log: 'org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History serve'

**Phase 2: Completed decoding.
       decoder: 'yarn'
       context: 'main'

       msg: 'Connecting to Application History serve'

**Phase 3: Completed filtering (rules).

       Rule id: '100011'
       Level: '3'
       Description: 'INFO yarn alerts'
**Alert to be generated.




2021-09-26 06:08:33,469 DEBUG org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History serve


**Phase 1: Completed pre-decoding.
       full event: '2021-09-26 06:08:33,469 DEBUG org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History serve'

       timestamp: '2021-09-26 06:08:33,46'

       hostname: 'DEBUG'
       program_name: '(null)'
       log: 'org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History serve'

**Phase 2: Completed decoding.
       decoder: 'yarn'
       context: 'main'

       msg: 'Connecting to Application History serve'

**Phase 3: Completed filtering (rules).

       Rule id: '100012'
       Level: '3'
       Description: 'DEBUG yarn alerts'
**Alert to be generated.

Hope this helps.

Regards,
Wali