new logs not shown

[Ale De Nocciola]

Hello everyone,

I am writing this post even though there are other similar posts, but unfortunately, they haven't been helpful to me.

I have created my own Python program that writes custom logs to a file at irregular intervals. Here's an example of a syslog-format log created by the application:

Aug 16 21:37:49 Extension extension_log[99999]: Extension with IP '127.0.0.1' and agent ID '001' attempted to access the host 'site.com'

It's important to specify that these logs, when created, should be visible in Kibana.

Here are the settings I added to the ossec.conf file (for convenience and testing, I created a dummy log file located at /tmp/test/test.log with 2 lines of logs similar to the example mentioned earlier):

[...] <global> <jsonout_output>yes</jsonout_output> <alerts_log>yes</alerts_log> <logall>yes</logall> <logall_json>yes</logall_json> [...] <alerts> <log_alert_level>3</log_alert_level> <email_alert_level>12</email_alert_level> </alerts> [...] <localfile> <log_format>syslog</log_format> <location>/tmp/test/test.log</location> <frequency>60</frequency> </localfile>

This is what I added to the /var/ossec/etc/decoders/local_decoder.xml file: <decoder name="extension_log"> <program_name>^extension_log</program_name> </decoder>

<decoder name="extension_log"> <parent>extension_log</parent> <regex>Extension with IP '(\d+.\d+.\d+.\d+)' and agent ID '(\d\d\d)' attempted to access the host '(\w+)'</regex> <order>srcip, data, user</order> </decoder>

And this is what I added to /var/ossec/etc/rules/local_rules.xml: <group name="extension_log_rule"> <rule id="100010" level="14"> <program_name>extension_url_blocked</program_name> <description>Domain blocked by extension</description> </rule> </group>

When running the program /var/ossec/bin/wazuh-logtest to test the log, this is the output: root @ alex:/var/ossec/etc/rules# /var/ossec/bin/wazuh-logtest Starting wazuh-logtest v4.5.0 Type one log per line

Aug 16 21:37:49 Extension extension_log[99999]: Extension with IP '127.0.0.1' and agent ID '001' attempted to access the host 'site.com'

**Phase 1: Completed pre-decoding. full event: 'Aug 16 21:37:49 Extension extension_log[99999]: Extension with IP '127.0.0.1' and agent ID '001' attempted to access the host 'site.com'' timestamp: 'Aug 16 21:37:49' hostname: 'Extension' program_name: 'extension_log'

**Phase 2: Completed decoding. name: 'extension_log'

**Phase 3: Completed filtering (rules). id: '100010' level: '14' description: 'Domain blocked by extension' groups: '['extension_log_rule']' firedtimes: '1' mail: 'True' **Alert to be generated.

In Kibana, under the "Modules" > "Security Events" section, I don't see any logs with the ID 100010 or that reflect what is written in the log.

I would appreciate your help in understanding why I am not seeing the log and how I could resolve this issue.

Thank you very much!

Best regards, Alex

[Luis Daniel Avendaño Larios]

Hi Alex,

Thank you for reaching out to us with your issue. Based on the information you provided, it seems that the logs generated by your Python program are not being properly processed and indexed in Kibana. To troubleshoot this, we recommend checking the following:

• Double-check the local_decoder.xml file to ensure that the 'extension_log' decoder is properly defined and matches the log format generated by your program. Pay attention to the regular expression used to extract the relevant fields from the log message.
• Review the local_rules.xml file and confirm that the 'extension_log_rule' group and the corresponding rule with ID 100010 are correctly defined. Check if the rule is being triggered based on the log message.
  
• Check the Wazuh manager logs (/var/ossec/logs/ossec.log) for any error messages or warnings related to log processing. This can provide valuable insights into the issue.

If you have verified all of the above and are still experiencing issues, please provide us with the relevant log files and any error messages you encounter. Also, could you please share with me the custom decoder and rules you created for these events? This will help us further investigate the problem and provide you with a more accurate solution.

Best regards,

Luis.

[Ale De Nocciola]

Hello Luis,

I've checked what you wrote to me but I couldn't find any error.
I tried change the name of the rule and of the decoder but nothing happens.

In the ossec.log there are not errors:

2023/08/17 12:52:34 wazuh-remoted: INFO: (1410): Reading authentication keys file.
2023/08/17 12:52:35 wazuh-logcollector: INFO: Monitoring output of command(360): df -P
2023/08/17 12:52:35 wazuh-logcollector: INFO: Monitoring full output of command(360): netstat -tulpn | sed 's/\([[:alnum:]]\+\)\ \+[[:digit:]]\+\ \+[[:digit:]]\+\ \+\(.*\):\([[:digit:]]*\)\ \+\([0-9\.\:\*]\+\).\+\ \([[:digit:]]*\/[[:alnum:]\-]*\).*/\1 \2 == \3 == \4 \5/' | sort -k 4 -g | sed 's/ == \(.*\) ==/:\1/' | sed 1,2d
2023/08/17 12:52:35 wazuh-logcollector: INFO: Monitoring full output of command(360): last -n 20
2023/08/17 12:52:35 wazuh-logcollector: INFO: (1950): Analyzing file: '/tmp/test/test.log'.
2023/08/17 12:52:35 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/ossec/logs/active-responses.log'.
2023/08/17 12:52:35 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/log/auth.log'.
2023/08/17 12:52:35 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/log/syslog'.
2023/08/17 12:52:35 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/log/dpkg.log'.
2023/08/17 12:52:35 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/log/kern.log'.
2023/08/17 12:52:35 wazuh-logcollector: INFO: Started (pid: 2933).
2023/08/17 12:52:36 wazuh-monitord: INFO: Started (pid: 2954).
2023/08/17 12:52:37 wazuh-modulesd: WARNING: 'update_from_year' option cannot be used for 'nvd' provider.
2023/08/17 12:52:37 wazuh-modulesd: INFO: Started (pid: 2976).

also the wazuh-logtest doesn^t give errors (you can see it in mine previous post and you can see the picture below):


In attachement you can find the local decoders and rules that I've created.

Thank you for your help