ESXi Log Decoding

[Matthew M.]

The built-in VMWare decoder does not work with modern ESXi syslogs. I can't tell exactly what changed, but I believe it has something to do with the timestamp.

Because the current decoder and rules DON'T work I have created the following decoder:

<decoder name="VMWareESXI7">
<prematch type="pcre2">\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z \S+ \S+: .+</prematch>
</decoder>

And I'm testing with This string:

2022-12-29T20:28:24.376Z esx02.test.local Rhttpproxy: verbose rhttpproxy[2098976] [Originator@6876 sub=Proxy Req 95800] Resolved endpoint : [N7Vmacore4Http16LocalServiceSpecE:0x0000000dc3115700] _serverNamespace = /vpxa action = Allow _port = 8089

I get the following result when testing:

**Phase 1: Completed pre-decoding.    

full event: '2022-12-29T20:28:24.376Z esx02.test.local Rhttpproxy: verbose rhttpproxy[2098976] [Originator@6876 sub=Proxy Req 95800] Resolved endpoint : [N7Vmacore4Http16LocalServiceSpecE:0x0000000dc3115700] _serverNamespace = /vpxa action = Allow _port = 8089'    

timestamp: '2022-12-29T20:28:24.376Z esx02.'    

program_name: 'Rhttpproxy' 

 **Phase 2: Completed decoding.    No decoder matched.

I have verified the decoder matches in EVERY OTHER context, but will not match in Wazuh (The regex is sound). I assume it's because of the pre-matching incorrectly identifying the timestamp string by also including part of the ESXi hostname.

I'd like some help on this.

[Nicolas Agustin Guevara Pihen]

Hello Matthew, thank you for using Wazuh! 

I'm verifying this situation. Apparently, Wazuh is wrongly pre-decoding the timestamp, and also Rhttpproxy as the program name. As you pointed, the regex is correct and this log should be decoded. I will be helping you to achieve that. Could you kindly tell me the way in which you are sending those logs to Wazuh? 

Kind regards,

[Matthew M.]

Nicolas,

I am shipping the logs over from the ESXi servers to a rsyslog server with the Wazuh agent installed. These are coming over TCP 514 to /var/log/vmware.log. I have configured the Wazuh agent to ingest that log file through the agent configuration. I can verify that the logs are populating into that log file.

Let me know what other information you need.

Thanks,

Matthew

[Nicolas Agustin Guevara Pihen]

HI Matthew, 

There is a workaround for this kind of issue when the logs are being read from a localfile: we can set an out_format. That will add some information at the beginning of the log, and that will prevent the pre-decoder from interfering with the log.

You can add, for example, the agent name before each of those logs, so the log 

2022-12-29T20:28:24.376Z esx02.test.local Rhttpproxy: verbose rhttpproxy[2098976] [Originator@6876 sub=Proxy Req 95800] Resolved endpoint : [N7Vmacore4Http16LocalServiceSpecE:0x0000000dc3115700] _serverNamespace = /vpxa action = Allow _port = 8089

Will be

Agent01: 2022-12-29T20:28:24.376Z esx02.test.local Rhttpproxy: verbose rhttpproxy[2098976] [Originator@6876 sub=Proxy Req 95800] Resolved endpoint : [N7Vmacore4Http16LocalServiceSpecE:0x0000000dc3115700] _serverNamespace = /vpxa action = Allow _port = 8089

And your decoder will be matched, as the pre-decoder will not match anything: 



To configure that, you need to modify the agent configuration and add an <out_format>. For example:

 <localfile>

           <log_format>syslog</log_format>
           <location> /var/log/vmware.log  </location>

           <out_format>$(hostname): $(log)</out_format>
     </localfile>

In this case, I'm using hostname and log, but there are several other options that may be useful. You can check them in this documentation.

I hope you find this information helpful, let me know if you have any questions or doubts.

Kind regards,