Inquiry on Manually Adding Json Logs to Wazuh

[Kelvin Mak]

Dear Team,

I am currently working on integrating SentinelOne XDR logs and AWS VPC logs into Wazuh. We intend to use Wazuh purely as a log storage tool, we do not connect agents or tools.  I would like to know if it is possible to manually add these logs in JSON format without directly connecting to the XDR or AWS services.

Currently, I download periodic XDR records to JSON and upload them to the Wazuh server. I have also created a `s1.xml` file with the following content to collect all logs from the JSON file:

```
<group name="local,">
  <rule id="100001" level="3">
    <decoded_as>json</decoded_as>
    <location>/home/wazuh-user/s1.json</location>
  </rule>
</group>
```

Additionally, I have added the following configuration to the `<ossec_config>` block of the `/var/ossec/etc/ossec.conf` local configuration file:

```
<localfile>
  <log_format>json</log_format>
  <location>/home/wazuh-user/s1.json</location>
</localfile>
```
To confirm, our main steps are:

Download periodic XDR records to JSON and upload them to the Wazuh server.
Collect all logs from the JSON file using an XML file.
Add the configuration to the <ossec_config> block of the /var/ossec/etc/ossec.conf local configuration file.
Are there any steps we might have missed? Additionally, if we use this method, will the logs be available for review in the “Threat Hunting” interface?

After our trying, the log shows wazuh analyzing the json file, but I cannot find any logs in the "Threat Hunting".
 
Thank you for your assistance.

[Juan Manuel Segura Duarte]

Hello Kelvin,

One way of achieving the desired functionality is to enable the archives to get all the events even if they are not decoded (although in this case all the JSON logs should be decoded by the json decoder) and adding the archives to the indexer, creating a new index pattern. Here is the guide on how to do so: https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-indices.html#the-wazuharchives-indices

Regards,

Juan Manuel

[Kelvin Mak]

Hi,

Thank you for your reply and the guide.

I have successfully used the wazuh-archives-* indices, but I still do not see the data I need is collected in archives.json. Could you please advise on how to let the archives.json detect my data file and save it in the archives.json? I tried placing a file named s1.json by Filezilla in the /home/wazuh-user folder. Will this be automatically collected into archives.json? (I am using the Wazuh virtual machine OVA.)

Best regards,
Kelvin

Juan Manuel Segura Duarte 在 2024年11月11日 星期一晚上10:06:32 [UTC+8] 的信中寫道：

[Juan Manuel Segura Duarte]

Hello Kelvin,

Sorry for the late answer.

How are logs being written in the `s1.json` file? If the write operations performed change the inode of the file then logs will not be collected, e.g. this happens when using a text editor like `vim`, which actually creates a temporary file to which changes are written and then renames the temporary file to replace the original one, thus modifying the inode of the original file. One way of doing this is by using `cat >> s1.json` or `echo >> s1.json` which keeps the file's inode intact.

Apart from that, are JSON logs multiline? The `log_format` json only allows for single-line JSON files, as stated in the documentation: 

Used for single-line JSON files and allows for customized labels to be added to JSON events.

Here you can find the mentioned documentation: https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/localfile.html#log-format

If logs are multi-line JSON, then either the logs are changed to be single-line (e.g. removing the \n at source if possible, or maybe having a small script that acts as an intermediary that receives the multi-line logs and turns them into single-line logs to later write them to the s1.json file) or by using another log_format in the Wazuh configuration, i.e. using the log format `multi-line-regex` (or `multi-line`, although this one is quite less flexible than its regex based counterpart), and configuring the regular expression to capture each log separately (this is done in the `multiline_regex` field of the `localfile` section). Here you have the documentation about multiline_regex: https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/localfile.html#multiline-regex

We hope you find this information useful.

Regards,
Juan Manuel

[Kelvin Mak]

Hi Juan Manuel,

Thank you for the detailed explanation.

My log file s1.json was downloaded from the SentinelOne system and has not been modified since. I used FileZilla to transfer it to the Wazuh server (VM OVA), placing it in the /home/wazuh-user folder. The transfer was done using the wazuh-user account, while other operations were performed with the root account.

Additionally, my JSON file is in single-line format. Here are some examples:

[
  {
    "event.time": "Nov 11 2024 14:25:16",
    "dataSource.name": "SentinelOne",
    "event.type": "Registry Value Modified"
  },
  {
    "event.time": "Nov 11 2024 14:25:16",
    "dataSource.name": "SentinelOne",
    "event.type": "Logout"
  },
  {
    "event.time": "Nov 11 2024 14:25:16",
    "dataSource.name": "SentinelOne",
    "event.type": "Behavioral Indicators"
  }
]

Best regards,

Kelvin

Juan Manuel Segura Duarte 在 2024年11月12日 星期二下午5:24:18 [UTC+8] 的信中寫道：

[Juan Manuel Segura Duarte]

Hello again Kelvin,

Thanks for clarifying the log obtention method. In that case, I believe the best way for the manager to collect those logs is to create an empty file, restart the manager to make sure that the logcollector is taking that file into account, and then writing the logs in that file with a tool like `cat`, so the steps to follow could be the following:

1. Move the file s1.json to another location, or change its name for logcollector to stop trying to read that file. E.g.: `mv /home/wazuh-user/s1.json /home/wazuh-user/source_s1.json`

2. Create the empty file with the name provided in the configuration. In this case: `touch /home/wazuh-user/s1.json`

3. Restart wazuh-manager to make sure logcollector is now monitoring the newly created file which, at the moment, is empty.

4. After the successful restart of the manager, write the contents of `source_s1.json` into `s1.json`, you can use the command `cat`: `cat /home/wazuh-user/source_s1.json >> /home/wazuh-user/s1.json`

5. Now logcollector should be getting the JSON logs

As to the JSON format, when the documentation talks about "single-line format" it means one JSON object per line. In the example you showed, unless formatted to improve readibility, it seems like one JSON object spans across multiple lines.

And the objects also seem to be wrapped in brackets, but JSON objects should have curly braces at the outermost level. Lists/Arrays are supported as values for a field in JSON.

We hope you find this information useful. Let us know if you need more help.

Regards,
Juan Manuel