Log analysis of non syslog style .txt logs

[Sebastian Silva]

Hello dear Wazuh community,

I am trying to analyze a log of one of my own programs. The log is NOT in syslog format and it seems that logtest and my custom decoder do not accept this format.

When I try to enter a example string of my oen format into logtest I get: '**Wazuh-logtest error when connecting with ossec-analysisd' (no entry for this error in ossec.log).

Is it possible that only .txt logs in syslog format are possible for either logtest or decoders in general??

All the best

Sebastian

[Christian Borla]

Hi Sebastian.
I hope you are doing fine!

It's not mandatory process syslog events, it could be a plain text too. Wazuh will try to proccess the event as a syslog event, the pre-decoding (**Phase 1), if doesn't match it leaves the message as it arrives, and custom rules start working in **Phase 2. wazuh-logtest example:

    /var/ossec/bin/wazuh-logtest
    dhcp,debug,packet     Domain-Server = 8.8.8.8

    **Phase 1: Completed pre-decoding.
        full event: 'dhcp,debug,packet     Domain-Server = 8.8.8.8'

    **Phase 2: Completed decoding.
        name: 'MikroTik'
        domain: '8.8.8.8'

The error "**Wazuh-logtest error when connecting with ossec-analysisd"  means analysisd is not running, it appears when wazuh-logtest try to connect to analysisd.
Please check if wazuh-analysisd is running, if the custom rule has some error it will stop analysisd process. it is possible check it with following command:

ps ax | grep analysisd
 453109 ?        Sl     0:00 /var/ossec/bin/wazuh-analysisd

Also you can check wazuh log message in /var/ossec/logs/ossec.log, there should find some ERROR messages.

Could you share your custom decoders and rules, also any of your own program logs? I would like to help you to create a decoder and a rule as example.
Let me know if this information is useful to you.
Regards.

[Sebastian Silva]

Hello Christian,

I just found out that the reason why my custom decoder is not working might be that Integrity_checksum_changed overrides my custom decoder.
The log I'm trying to observe is within a whodata observed folder. Maybe that means that the Integrity_checksum_changed-decoder just matches first and Wazuh does not check any further to match the log with my custom decoder.

Its also not possible to attach my custom decoder as a child to Integrity_checksum_changed.

How could I fix this?

Best regards

Sebastian

[Christian Borla]

Hi Sebastian.
Sorry for the answer delay, I hope you are doing fine!
When you talk about integrity_checksum do you refere to Anomaly and malware detection?
Also it's possible to generate an alert from any default alert, add a new decoder to a default decoder file, or change a defualt decoder. That way after applies the default decoder it will run your custom decoder.

I see it's not available syscheck_integrity_changed decoder as other decoders, but, did you try to include you custom decoder in /var/ossec/etc/decoders/local_decoder.xml including following names?

<decoder name="syscheck_integrity_changed_child">
  <parent>syscheck_integrity_changed</parent>
  ...
</decoder>

If you can share your decoder it will easier to me to test it.
Let me know if it works for you!
Regards.

[Sebastian Silva]

Hi Christian,

first of all:

- N: Number

- L: Letter

- Str: String

- after every ',' there is no space in the real log this is just for READABLITY here.

Now  the log format is:

N-N-NLN:N:N+N:N, N, Str-Str, Str, Str, N.NLN - N, N.N.N.N, N.N, Str, , , , N.N.N, Str

No line-break all in one line each time.

Trying to attach my new decoder to system_integrity_changed as a parent throws an error since system_integrity_changed already is a child decoder.

I did write my decoder in local_decoder. But as far as i understand Wazuh stops checking for other decoders once one matched and system_integrity_changed seems to match first every time.

My decoder has a parent with a prematch:

^\d\d\d\d-\d\d-\d\d\w\d\d:\d\d:\d\d

and then an actual decoder with the <regex> like:

(\d+-\d+-\d+\w\d+:\d+:\d+\p\d+:\d+),(\d+),(\w+-?\w*),(\w+),(\w*),(\w*\.\w*\s-\s\w*),(\d+\.?\d*\.?\d*\.?\d*),(\d+\.?\d*\.?\d*),(\w*),,,,(\d+\.?\d*\.?\d*\.?\d*),(.*)

Thanks a lot

Sebastian

[Sebastian Silva]

Hello Christian,

I now solved the issue by adding <log_format>multi-line:1</log_format> to my localfile in the agent conf.

Now a new problem has emerged. The file I'm trying to run my custom log analysis decoder over is WITHIN a folder I am  monitoring with whodata. Now when I enable whodata, my custom decoder does not get the chance to analyze said log within the folder since all is matched and handled by 'syscheck_integrity_changed'.

Neither ignore nor restrict are helping in excluding said log fromthe general who data monitoring. also it should be possible to monitor whodata and use a custom decoder in the same folder!

Could you tell me how to solve this? Or if not, whta the parent decoder for system_integrity_changed is so that i can append my custom decoders to this parent?

Regards

Sebastian

[Christian Borla]

Hi Sebastian!
I hope you are doing fine!!

Just in case, I would like mention that \d+ or any + after wildcard character are support only by pcre2 regex engine, so if you want to use it, regex section must include a pcre2 label as <regex type="pcre2">.  

I did a test following your custom rule and examples, I created a custom decoder as following:

<decoder name="syscheck_integrity_changed_child">
  <parent>syscheck_integrity_changed</parent>

  <regex type="pcre2">(\d-\d-\w{3}:\d:\d\+\d:\d),(\d),([^-]+-[^,]+),([^,]+),([^,]+),(\d\.\w{3}-\d),(\d\.\d\.\d\.\d,\d\.\d),([^,]+),,,,(\d\.\d\.\d),(\S+)</regex>
  <order>datetime, field2, host, application, field5, field6, srcip, field8, , field9, field10</order>
</decoder>

But it fails,

2022/05/05 15:24:12 wazuh-analysisd[92515] analysisd.c:618 at main(): ERROR: analysisd/decoders/decoders_list.c:184 at OS_AddOSDecoder(): (2101): Parent decoder name invalid: 'syscheck_integrity_changed'.
2022/05/05 15:24:12 wazuh-analysisd[92515] analysisd.c:618 at main(): ERROR: analysisd/decoders/decode-xml.c:815 at ReadDecodeXML(): (2106): Error adding decoder plugin.

Then I realize that syscheck_integrity_changed decoder is a implicit decoder, are part of the code, so it's not possible add child to those decoders, but it's possible add rules, as following.  

  <rule id="550" level="7">
    <category>ossec</category>
    <decoded_as>syscheck_integrity_changed</decoded_as>
    <description>Integrity checksum changed.</description>
    <mitre>
      <id>T1492</id>
    </mitre>
    <group>syscheck,pci_dss_11.5,gpg13_4.11,gdpr_II_5.1.f,hipaa_164.312.c.1,hipaa_164.312.c.2,nist_800_53_SI.7,tsc_PI1.4,tsc_PI1.5,tsc_CC6.1,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>
  </rule>

So the answer it's not possible create a child decoder from syscheck_integrity_changed

In other hand, Syscheck integrity is a functionality to validate the file integrity, not to collect events logs, and all integrity changes will generate an alert. I guess you have in ossec.conf a syscheck section similar than following.  

<!-- File integrity monitoring -->
<syscheck>
 .
 .
 <!-- Files/directories to ignore -->
 <ignore>/etc/mtab</ignore>
 <ignore>/etc/hosts.deny</ignore>
 .
 .
 <!-- File types to ignore -->
 <ignore type="sregex">.log$|.swp$</ignore>
 .
 .

Where is possible to set the ignore file option  

If ignore section doesn't work for this case, you can try creating a rule with level 0, to avoid create alerts from those events, here you have table with possibles fields to use, also some level 0 examples rules.

Let me know if this information is useful.
Regards.

[Sebastian Silva]

Hey Christian!

Thanks a lot for your work so far. As mentioned in my second post from yesterday, I managed to make it all work except for the fact that whodata claims new entries in all logs in its folder with its own system_integrity_changed decoder. Meaning that even if I set my log to ignore, another decoder still can't read it because system_integrity_changed occupies it even if it ignores the new entries themselves and does not generate checksum alarms from them.

I'll try to set the rule to 0 but I think the problem will stay the same. My custom decoder does not get to decode the log because whodata has control of the whole folder and matches it with its own decoder.

Regards

Sebastian

[Sebastian Silva]

Hello Christian,

I now managed to have both log analysis and FIM run at once! I think this issue can be closed now.

Thanks a lot or your tries and explanations.

Have a great weekend!

Regards

Sebastian

[Christian Borla]

Hello Sebastian!
Good news!!
have a good weekend too!
Regards