Syslog Not Generating Alerts
DMFA
- I've got syslog configured on my esxi server pointing to my wazuh manager.
- netstat -lupnd
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
udp 0 0 0.0.0.0:111 0.0.0.0:* 469/rpcbind
udp 0 0 127.0.0.1:323 0.0.0.0:* 417/chronyd
udp 0 0 192.168.1.215:514 0.0.0.0:* 26388/wazuh-remoted
udp 0 0 0.0.0.0:627 0.0.0.0:* 469/rpcbind
udp6 0 0 :::111 :::* 469/rpcbind
udp6 0 0 ::1:323 :::* 417/chronyd
udp6 0 0 :::627 :::* 469/rpcbind
- netstat -lupnd
- I can see the syslog events hitting the server.
- tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:32:48.921270 IP (tos 0x0, ttl 64, id 61437, offset 0, flags [none], proto UDP (17), length 410)
192.168.1.205.12867 > 192.168.1.215.syslog: SYSLOG, length: 382
Facility local4 (20), Severity warning (4)
Msg: 2022-02-01T07:29:27.057Z localhost.localdomain Rhttpproxy: warning rhttpproxy[1051295] [Originator@6876 sub=RhttpProxy] SSL Handshake failed for stream <SSL(<io_obj p:0x0000009e95a079e8, h:16, <TCP '192.168.1.205 : 443'>, <TCP '192.168.1.200 : 27995'>>)>: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown)\0x0a
- tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
- I enabled <logall>yes</logall> and can see the syslogs in archives.log.
- 2022 Feb 01 01:51:01 st.localdomain->192.168.1.205 2022-01-31T19:47:40.010Z localhost.localdomain Hostd: info hostd[1050772] [Originator@6876 sub=Libs] [ConfigStore:1050297:218424395520:] Get invoked for: component esx group graphics key devices instanceId 0000:00:02.0, userdatacontext: 1
- I created a custom decoder in local_decoder.xml and custom rules in local_rules.xml. When I test the logs, with the wazuh-logtest it works fine and says, "**Alert to be generated" and firedtimes: '1' and I changed Level: '10'
- **Phase 1: Completed pre-decoding.
full event: '2022 Jan 31 13:31:47 st.localdomain->192.168.1.205 2022-01-31T07:28:26.595Z localhost.localdomain Hostd: info hostd[1050772] [Originator@6876 sub=Default opID=esxui-85bf-9276] Rejected password for user root from 192.168.1.200'
timestamp: '2022 Jan 31 13:31:47'
**Phase 2: Completed decoding.
name: 'dmfa-esxi'
dstip: '192.168.1.205'
dstuser: 'root'
srcip: '192.168.1.200'
**Phase 3: Completed filtering (rules).
id: '100003'
level: '10'
description: 'Login Failure'
groups: '['syslog', 'authentication_failed']'
firedtimes: '1'
mail: 'False'
**Alert to be generated.
- **Phase 1: Completed pre-decoding.
- However, nothing is showing up in alerts.log and if I click on Discover in Wazuh there are no matching events.
Mariano Koremblum
Mariano Koremblum
Hi again DMFA,
We are sorry for the delay, we were analyzing your case.
So, first of all, I would recommend you always use the logall_json option, when you want to debug the arriving logs, instead of the logall one, as the json configuration provides much more information related to the incoming events.
The second point would be, for you to know, that the actual log is not:
2022 Jan 31 13:31:47 st.localdomain->192.168.1.205 2022-01-31T07:28:26.595Z localhost.localdomain Hostd: info hostd[1050772] [Originator@6876 sub=Default opID=esxui-85bf-9276] Rejected password for user root from 192.168.1.200
but:
2022-01-31T07:28:26.595Z localhost.localdomain Hostd: info hostd[1050772] [Originator@6876 sub=Default opID=esxui-85bf-9276] Rejected password for user root from 192.168.1.200
The first part (2022 Jan 31 13:31:47 st.localdomain->192.168.1.205) is added by Wazuh, so if you are trying to create a custom rule/decoder you should always do it using the raw log (full_log field of the events written on the archives.json file) and not the enriched one.
So, in this case, I have tested the raw log with our wazuh-logtest tool and this is the output:
# /var/ossec/bin/wazuh-logtest
Starting wazuh-logtest v4.2.5
Type one log per line
2022-01-31T07:28:26.595Z localhost.localdomain Hostd: info hostd[1050772] [Originator@6876 sub=Default opID=esxui-85bf-9276] Rejected password for user root from 192.168.1.200
**Phase 1: Completed pre-decoding.
full event: '2022-01-31T07:28:26.595Z localhost.localdomain Hostd: info hostd[1050772] [Originator@6876 sub=Default opID=esxui-85bf-9276] Rejected password for user root from 192.168.1.200'
timestamp: '2022-01-31T07:28:26.595Z localh'
program_name: 'Hostd'
**Phase 2: Completed decoding.
No decoder matched.
This tells us that the predecoder is not working properly, as it cannot correctly strip the timestamp. So, there is a Wazuh limitation here, as the timestamp is wrong from the start and it won’t be indexed in Kibana no matter if your custom rules and decoders are correct or not. You can see that there is an open issue on Github related to this problem: https://github.com/wazuh/wazuh/issues/3525
In order to skip this limitation, are you able to change from the source the timestamp format?
I will be waiting for your reply,
Mariano Koremblum
DMFA
DMFA
Hi @cleberb,
I agree. The pre-decoding stage is currently a static decoder written in C, we can't change or disable it.
It would be better improving the regex engine and allowing the user to define custom pre-decoders.
That's not an easy task but let me mark this issue as enhancement so we will discuss it with the team.
Thanks for your feedback.
Regards.
Mariano Koremblum
I am sorry to tell you that there are no workarounds to directly ingest these kinds of logs.
What you could do is to set up a rsyslog server, on the same machine where your Wazuh manager is installed, to receive these remote logs and store them in a local file. Then, you can monitor this local file with the Wazuh manager as follows.
Set the following configuration block on your manager’s ossec.conf file:
<log_format>syslog</log_format>
<location>/my/custom/path/to/localfile.log</location>
<out_format>$(timestamp) $(hostname) Hostd: $(log)</out_format>
</localfile>
What the out_format will cause is to add information to the raw log ($(log)). So, if you have the following raw log:
2022-01-31T07:28:26.595Z localhost.localdomain Hostd: info hostd[1050772] [Originator@6876 sub=Default opID=esxui-85bf-9276] Rejected password for user root from 192.168.1.200
What you will get to be processed by Wazuh would end up being something like this (timestamp format might be different):
Feb 01 11:45:01 localhost Hostd: 2022-01-31T07:28:26.595Z localhost.localdomain Hostd: info hostd[1050772] [Originator@6876 sub=Default opID=esxui-85bf-9276] Rejected password for user root from 192.168.1.200
The date is added when the log is collected from the localfile, so it will differ from the actual one that was inserted when the log was produced in the source, but it will be a valid timestamp format that the Wazuh preprocessor can digest.
Then you should create your custom rules and decoders taking into account the new format of the log.
In this case, given the decoder that you have shared with us before, you would have something like the following:And the output, which is obtained by inspecting the log (the second one), on wazuh-logtest will be the following:
| # /var/ossec/bin/wazuh-logtest
| Starting wazuh-logtest v4.2.5
| Type one log per line
|
|
| **Phase 1: Completed pre-decoding.
| full event: 'Feb 01 11:45:01 localhost Hostd: 2022-01-31T07:28:26.595Z localhost.localdomain Hostd: info hostd[1050772] [Originator@6876 sub=Default opID=esxui-85bf-9276] Rejected password for user root from 192.168.1.200'
| timestamp: 'Feb 01 11:45:01'
| hostname: 'localhost'
|
| **Phase 2: Completed decoding.
| dstuser: 'root'
| location: 'localhost.localdomain'
| srcip: '192.168.1.200'
| timestamp_esxi: '2022-01-31T07:28:26.595Z'
You should take into account that, in the second decoder, the child one, the regex field only takes effect on the $(log) part of the full log.
Best Regards,
Mariano Koremblum
DMFA
Mariano Koremblum
Best Regards