Nutanix Log Forwarding

[swaps]

Hello Team,

I just wanted to know if I enable remote log forwarding at Nutanix Virtualization boxes to Wazuh Manager, does Wazuh has capabilities to monitor/alert these syslog events?

Are there any preset decoders/rulesets w.r.t. Nutanix?

Kindly guide me in this.

Thanks,

swapnils

[Jesus Linares]

Hi,

Yes, you can monitor syslog events with Wazuh. There are two ways:

1. Configure the Wazuh manager to listen syslog. Checkout the "remote" setting documentation: https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/remote.html.
2. Configure rsyslog in the same server as the Wazuh manager and read them with Wazuh. This is useful if you want to use rsyslog templates to change the log format. You have two ways:
   1. Forward the events from rsyslog to a file and read the file with Wazuh
   2. Send the events locally from rsyslog to the manager listening syslog (as in the first method). With this method, you don't need to handle the file since the events are sent directly from rsyslog to Wazuh.

Currently, there is no preset of decoders/rules for Nutanix, but we can help you to create them. Also, if the event is in JSON you will not need to create decoders.

I hope it helps.

[swaps]

Thank you very much Jesus!

I have configured `remote` section to receive logs to Port 513 UDP in `ossec.conf`. Now will be forwarding logs to Manager.

I was concerned about the decoders as how the alerts will get triggered without any rules defined.  I shall check type of logs being received to get the more understanding.

Regards,

[Jesus Linares]

Hello,

Well, if there are no rules for the events, any alerts will be triggered. So, if you want to review it, you can enable the "logall" setting. In this way, every event that Wazuh receives will be sent to "/var/ossec/logs/archives/archives.json" regardless if there is a rule for that event. This could fill up your disk quickly, so enable it temporarily.

Use the event and the logtest tool to create your own rules. We can help you if you share the events (do not share confidential data).

I hope it helps.

[swaps]

Hello Jesus,

Apologies for delayed revert! I was little occupied with some other priority work.
I already have logall json enabled to be compliant with security standards. I checked using tcpdump and archives logs that Nutanix devices are sending logs to the manager. But when trying to Discover via Wazuh Dashboard, it isnt showing anything.
archives.json has following lines (some of the data is masked.)

{"timestamp":"2023-01-20T11:39:18.668+0530","agent":{"id":"000","name":"wazuh-mgr-02"},"manager":{"name":"wazuh-mgr-02"},"id":"1674194958.143096904","cluster":{"name":"wazuh","node":"ws-02"},"full_log":"2023-01-20T11:39:18.664244+05:30 NTNX-96SM6G511379-D-CVM audispd[8623]: node=ntnx-96SM6G511379-d-cvm type=SYSCALL msg=audit(1674194958.663:18514813): arch=c000003e syscall=90 success=yes exit=0 a0=5971970 a1=1a0 a2=1 a3=0 items=1 ppid=11549 pid=11573 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=4294967295 comm=\"python2.7\" exe=\"/usr/bin/python2.7\" subj=unconfined_u:unconfined_r:unconfined_t:s0 key=\"nutanix_config_modified\"","predecoder":{"program_name":"audispd","timestamp":"2023-01-20T11:39:18.664244+05:30"},"decoder":{},"location":"172.20.20.2"} {"timestamp":"2023-01-20T11:39:18.669+0530","agent":{"id":"000","name":"wazuh-mgr-02"},"manager":{"name":"wazuh-mgr-02"},"id":"1674194958.143096904","cluster":{"name":"wazuh","node":"ws-02"},"full_log":"2023-01-20T11:39:18.664768+05:30 NTNX-96SM6G511379-D-CVM audispd[8623]: node=ntnx-96SM6G511379-d-cvm type=PATH msg=audit(1674194958.663:18514813): item=0 name=\"/home/nutanix/config/tmpjj5Qwx\" inode=524354 dev=08:03 mode=0100600 ouid=1000 ogid=1000 rdev=00:00 obj=unconfined_u:object_r:user_home_t:s0 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0","predecoder":{"program_name":"audispd","timestamp":"2023-01-20T11:39:18.664768+05:30"},"decoder":{},"location":"172.20.20.2"} {"timestamp":"2023-01-20T11:39:18.670+0530","agent":{"id":"000","name":"wazuh-mgr-02"},"manager":{"name":"wazuh-mgr-02"},"id":"1674194958.143096904","cluster":{"name":"wazuh","node":"ws-02"},"full_log":"2023-01-20T11:39:18.665335+05:30 NTNX-96SM6G511379-D-CVM audispd[8623]: node=ntnx-96SM6G511379-d-cvm type=PROCTITLE msg=audit(1674194958.663:18514813): proctitle=2F7573722F62696E2F707974686F6E322E37002D42002F686F6D652F6E7574616E69782F636C75737465722F62696E2F67656E65736973002D2D666F726567726F756E643D74727565002D2D6C6F675F7468726561645F69643D74727565002D2D666C616766696C653D2F686F6D652F6E7574616E69782F636F6E6669672F67","predecoder":{"program_name":"audispd","timestamp":"2023-01-20T11:39:18.665335+05:30"},"decoder":{},"location":"172.20.20.2"} {"timestamp":"2023-01-20T11:39:26.288+0530","rule":{"level":2,"description":"Unknown problem somewhere in the system.","id":"1002","firedtimes":1633,"mail":true,"groups":["syslog","errors"],"gpg13":["4.3"]},"agent":{"id":"000","name":"wazuh-mgr-02"},"manager":{"name":"wazuh-mgr-02"},"id":"1674194966.143103775","cluster":{"name":"wazuh","node":"ws-02"},"full_log":"2023-01-20T11:39:26.283440+05:30 NTNX-96SM6G511379-D-CVM genesis: 2023-01-20 06:09:18,401Z ERROR 58019824 ipv4config.py:1895 Unable to get the KVM device configuration, ret 1, stdout , stderr br0-backplane: error fetching interface information: Device not found\\n","predecoder":{"program_name":"genesis","timestamp":"2023-01-20T11:39:26.283440+05:30"},"decoder":{},"location":"172.20.20.2"} {"timestamp":"2023-01-20T11:39:26.288+0530","rule":{"level":2,"description":"Unknown problem somewhere in the system.","id":"1002","firedtimes":1634,"mail":true,"groups":["syslog","errors"],"gpg13":["4.3"]},"agent":{"id":"000","name":"wazuh-mgr-02"},"manager":{"name":"wazuh-mgr-02"},"id":"1674194966.143103775","cluster":{"name":"wazuh","node":"ws-02"},"full_log":"2023-01-20T11:39:26.283556+05:30 NTNX-96SM6G511379-D-CVM genesis: 2023-01-20 06:09:19,220Z ERROR 58019824 ipv4config.py:1895 Unable to get the KVM device configuration, ret 1, stdout , stderr br0-backplane: error fetching interface information: Device not found\\n","predecoder":{"program_name":"genesis","timestamp":"2023-01-20T11:39:26.283556+05:30"},"decoder":{},"location":"172.20.20.2"} {"timestamp":"2023-01-20T11:39:45.935+0530","agent":{"id":"000","name":"wazuh-mgr-02"},"manager":{"name":"wazuh-mgr-02"},"id":"1674194985.143118620","cluster":{"name":"wazuh","node":"ws-02"},"full_log":"2023-01-20T11:39:45.929187+05:30 NTNX-96SM6G511379-D-CVM audispd[8623]: node=ntnx-96SM6G511379-d-cvm type=SYSCALL msg=audit(1674194985.926:18514814): arch=c000003e syscall=2 success=yes exit=6 a0=7f82752da32e a1=2 a2=3e8 a3=3 items=1 ppid=13591 pid=13791 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4454865 comm=\"sshd\" exe=\"/usr/sbin/sshd\" subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=\"logins\"","predecoder":{"program_name":"audispd","timestamp":"2023-01-20T11:39:45.929187+05:30"},"decoder":{},"location":"172.20.20.2"}

​

[swaps]

Hi Jesus/Team,

Any thoughts suggestions here? Your guidance is highly appreciated!

[Jesus Linares]

Hello,

Checking the "full_log" field of every line of your archives.json you will get the raw events. In your case:

> 2023-01-20T11:39:18.664244+05:30 NTNX-96SM6G511379-D-CVM audispd[8623]: node=ntnx-96SM6G511379-d-cvm type=SYSCALL msg=audit(1674194958.663:18514813): arch=c000003e syscall=90 success=yes exit=0 a0=5971970 a1=1a0 a2=1 a3=0 items=1 ppid=11549 pid=11573 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=4294967295 comm=\"python2.7\" exe=\"/usr/bin/python2.7\" subj=unconfined_u:unconfined_r:unconfined_t:s0 key=\"nutanix_config_modified\"

> 2023-01-20T11:39:18.664768+05:30 NTNX-96SM6G511379-D-CVM audispd[8623]: node=ntnx-96SM6G511379-d-cvm type=PATH msg=audit(1674194958.663:18514813): item=0 name=\"/home/nutanix/config/tmpjj5Qwx\" inode=524354 dev=08:03 mode=0100600 ouid=1000 ogid=1000 rdev=00:00 obj=unconfined_u:object_r:user_home_t:s0 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0

Now, you can use the /var/ossec/bin/wazuh-logtest binary to see what is missing (decoders, rules, or both):

# /var/ossec/bin/wazuh-logtest
Starting wazuh-logtest v4.3.6
Type one log per line

2023-01-20T11:39:18.664768+05:30 NTNX-96SM6G511379-D-CVM audispd[8623]: node=ntnx-96SM6G511379-d-cvm type=PATH msg=audit(1674194958.663:18514813): item=0 name=\"/home/nutanix/config/tmpjj5Qwx\" inode=524354 dev=08:03 mode=0100600 ouid=1000 ogid=1000 rdev=00:00 obj=unconfined_u:object_r:user_home_t:s0 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0

**Phase 1: Completed pre-decoding.
    full event: '2023-01-20T11:39:18.664768+05:30 NTNX-96SM6G511379-D-CVM audispd[8623]: node=ntnx-96SM6G511379-d-cvm type=PATH msg=audit(1674194958.663:18514813): item=0 name=\"/home/nutanix/config/tmpjj5Qwx\" inode=524354 dev=08:03 mode=0100600 ouid=1000 ogid=1000 rdev=00:00 obj=unconfined_u:object_r:user_home_t:s0 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0'
    timestamp: '2023-01-20T11:39:18.664768+05:30'
    program_name: 'audispd'

**Phase 2: Completed decoding.
    No decoder matched.

As you can see you need both, decoders and rules. For that reason you don't see them in the dashboard: Only alerts with levels higher than 2 are indexed by default.

That said, in your case, you are forwarding auditd events:

2023-01-20T11:39:18.664768+05:30 NTNX-96SM6G511379-D-CVM audispd[8623]: node=ntnx-96SM6G511379-d-cvm type=PATH msg=audit(1674194958.663:18514813): item=0 name=\"/home/nutanix/config/tmpjj5Qwx\" inode=524354 dev=08:03 mode=0100600 ouid=1000 ogid=1000 rdev=00:00 obj=unconfined_u:object_r:user_home_t:s0 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0

If you paste the auditd event (red part) to logtest, you will see that we have decoders and alerts by default:

type=PATH msg=audit(1674194958.663:18514813): item=0 name=\"/home/nutanix/config/tmpjj5Qwx\" inode=524354 dev=08:03 mode=0100600 ouid=1000 ogid=1000 rdev=00:00 obj=unconfined_u:object_r:user_home_t:s0 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0

**Phase 1: Completed pre-decoding.
    full event: 'type=PATH msg=audit(1674194958.663:18514813): item=0 name=\"/home/nutanix/config/tmpjj5Qwx\" inode=524354 dev=08:03 mode=0100600 ouid=1000 ogid=1000 rdev=00:00 obj=unconfined_u:object_r:user_home_t:s0 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0'

**Phase 2: Completed decoding.
    name: 'auditd'
    audit.gid: '1000'
    audit.id: '18514813'
    audit.type: 'PATH'

**Phase 3: Completed filtering (rules).
    id: '80700'
    level: '0'
    description: 'Audit: Messages grouped.'
    groups: '['audit']'
    firedtimes: '1'
    mail: 'False'

So, I think you have two options:

• Create your own decoders/rules
• Modify the event in rsyslog to remove the blue part, so the auditd events will match the current ruleset

Let me know if you need more help.

Regards.

[swaps]

Hello Jesus,

Thank you so much for your comments!

Nutanix is consist of multiple logs; for now we had enabled only few logs. One of which is Audit logs.

Will it be possible for you to share some KB links to create own decoders/rules? (Trust it will be very complicated)

I did not understand 2nd option where you have mentioned 'blue part'. Could you please elaborate a little?

Regards,

[Jesus Linares]

Hello,

Sorry, I formatted the text with different colors (blue and red) but it is not shown...

Here you can find information about the ruleset: https://documentation.wazuh.com/current/user-manual/ruleset/index.html.

The point here is that you create new decoders/rules when there are no decoders/rules for your events. Your case is not exactly that. Let me explain it.

This is your event:

> 2023-01-20T11:39:18.664244+05:30 NTNX-96SM6G511379-D-CVM audispd[8623]: node=ntnx-96SM6G511379-d-cvm type=SYSCALL msg=audit(1674194958.663:18514813): arch=c000003e syscall=90 success=yes exit=0 a0=5971970 a1=1a0 a2=1 a3=0 items=1 ppid=11549 pid=11573 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=4294967295 comm=\"python2.7\" exe=\"/usr/bin/python2.7\" subj=unconfined_u:unconfined_r:unconfined_t:s0 key=\"nutanix_config_modified\"

We could say that the event has this format: <syslog or nutanix headers> <auditd event>

• syslog or nutanix headers: 2023-01-20T11:39:18.664244+05:30 NTNX-96SM6G511379-D-CVM audispd[8623]: node=ntnx-96SM6G511379-d-cvm 
  
• auditd event: type=SYSCALL msg=audit(1674194958.663:18514813): arch=c000003e syscall=90 success=yes exit=0 a0=5971970 a1=1a0 a2=1 a3=0 items=1 ppid=11549 pid=11573 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=4294967295 comm=\"python2.7\" exe=\"/usr/bin/python2.7\" subj=unconfined_u:unconfined_r:unconfined_t:s0 key=\"nutanix_config_modified\"
  

So, we don't have decoders/rules for your event (<syslog or nutanix headers> <auditd event>) but we have for auditd events (<auditd event>). For that reason, I said that you have two options:

• Create decoders/rules for your event. You will have to replicate the current auditd rules. This is a lot of work, but it is an option.
• Send to Wazuh only the "auditd event". I will explain better below.

Send to Wazuh only the "auditd event"

Right now, I understand that you are sending Nutanix log via syslog to the Wazuh manager: 

[Syslog] -> [Wazuh manager - syslog]. 

But, there are two more options:

• [Syslog] -> [rsyslog -> Wazuh manager - syslog]
• [Syslog] -> [rsyslog -> file <- Wazuh manager - logcollector]

What is the advantage? You can use rsyslog templates to convert the event from:

> 2023-01-20T11:39:18.664244+05:30 NTNX-96SM6G511379-D-CVM audispd[8623]: node=ntnx-96SM6G511379-d-cvm type=SYSCALL msg=audit(1674194958.663:18514813): arch=c000003e syscall=90 success=yes exit=0 a0=5971970 a1=1a0 a2=1 a3=0 items=1 ppid=11549 pid=11573 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=4294967295 comm=\"python2.7\" exe=\"/usr/bin/python2.7\" subj=unconfined_u:unconfined_r:unconfined_t:s0 key=\"nutanix_config_modified\"

to

> type=SYSCALL msg=audit(1674194958.663:18514813): arch=c000003e syscall=90 success=yes exit=0 a0=5971970 a1=1a0 a2=1 a3=0 items=1 ppid=11549 pid=11573 auid=4294967295 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=4294967295 comm=\"python2.7\" exe=\"/usr/bin/python2.7\" subj=unconfined_u:unconfined_r:unconfined_t:s0 key=\"nutanix_config_modified\"

In that way, you don't need to create decoders/rules because it is the expected format for auditd events.

That is the idea. Let me know if you need more help ;)

[swaps]

Thank you so much for the detailed explanation, Jesus!

I am trying to relate & absorb things; which might take a while. So from what you explained, there are three ways to send logs to Wazuh -

1st) [Syslog] -> [Wazuh manager - syslog] >> Existing configuration

2nd) [Syslog] -> [rsyslog -> Wazuh manager - syslog]

3rd) [Syslog] -> [rsyslog -> file <- Wazuh manager - logcollector]

Is there any different configuration to be done to adapt the 2nd or 3rd approach? Could you please tell me how it is done?
From the documentation, I understood that there are two ways to enable remote logging (agentless or remote). Is it something different than this? Currently I have used this approach for remote logs.

Also auditd is one of the modules of Nutanix. We may get a request to integrate other modules as well for logging purpose. I am not sure if writing own decoders is I could manage (though I could give it a try).
Please let me know your thoughts.

Regards,
swpanils

​

[Jesus Linares]

Hello,

Sorry for the late reply. Let me explain the three options:

1st) [Syslog] -> [Wazuh manager - syslog]

In this option, you just configure the manager to receive syslog events. Here is the documentation: https://documentation.wazuh.com/current/user-manual/capabilities/log-data-collection/how-it-works.html#receiving-syslog-logs-in-a-custom-port.

There is nothing wrong with this option, it is pretty simple but you can't "modify" the events.

2nd) [Syslog] -> [rsyslog -> Wazuh manager - syslog]

In this option, you configure rsyslog in the same server as the manager. Now, you send the events from your syslog devices to rsyslog. Then, you forward them locally to the manager.

Why do you need to use the extra step of a rsyslog? Because you have more options in your events, for example you can modify them using templates (more information).

3rd) [Syslog] -> [rsyslog -> file <- Wazuh manager - logcollector]

This option is the same as option two, but instead of forwarding directly to Wazuh, you send the events to a file. Then, you read that file with Wazuh. The main advantage is that you can modify the events and create different files per source:

• ntnx-96SM6G511379-auditd.log
  
• ntnx-96SM6G511379-othermodule.log
  

In this way, the field "location" of your alert will say the source of the event. This can be complicated with option 2 (depending on how you modify the event you will have the source or not).

The disadvantage is that you have to rotate the files to prevent disk usage issues.

What option is the best one? It's up to you. If you want to modify the events to match the current rules as I explained in previous posts, you can go with option 2 or 3. If you are OK with creating new decoders/rules, go with option 1 since it is the simplest one.

If you need help with the decoders/rules, we can help you ;)