JSON Logs for Wazuh

[Utkarsh Bhargava]

Hi Community,

I am running Wazuh 4.3.7 and trying to ingest Oracle Cloud Logs using Wazuh Agent 4.3.7.

Wazuh agent is reading the log file but I am unable to see anything on Kibana.

I have decoder and alert rules for the same still nothing is there.

I have also enabled logall option on Wazuh Manager but still no OCI logs are there.

I am understanding why agent is not forwarding the logs.

For your reference I have attached the agent logs screenshot decoder/alert rule screenshot and OCI sample logs.

Thanks & Regards

Utkarsh

 

[Federico Gustavo Galland]

Hi There!

I see you got the rules part addressed since the logtest is showing a triggered rule for your test event, so it's probably not related to that.

The logall_json variable enables logging of all incoming data to /var/ossec/logs/archives/archives.json regardless of the rule matching and decoding stages.

If the new log lines don't show in your archives file, it means that they are not getting to the manager at all.

In order to make sure you are correctly parsing your logs, I suggest you check that you are properly retrieving the log files (with possible use of a wildcard, since it seems your log names contain a hash).

Secondly I'd try and check that the log files ownership and permissions allow the wazuh user to read them.

In order to config your Agent, it's best to use centralized configuration by creating a group and assigning the Agent to it:

Go to groups:

Create a new group:

Clock on the new group's name from the list and then go to "Manage agents"

Add your desired agent to it:

Apply and get back to the groups list. Then hit the pencil icon on your new group's row.

Add the proper <localfile> block. Note I'm using the wildcard sign to grab logs with any name within /var/log/oracle. Tune to taste.

With this done, I'd go to the agent's terminal and make sure you are using the right permissions for the files.

Usually, log files have 640 octal permissions, which you would change using chmod.

Aside from that you might have to need the wazuh user to the logging group of whatever software is dumping your logs.

You can reach back to us for assistance on the latter if needed.

I hope this helps a bit.

Regards,

Fede

[Federico Gustavo Galland]

A member of my team pointed out the "data" field could be causing issues, so an actual solution to this issue is to use the <out_format> tag to wrap the full json objects between curly brackets.

In this way I'm able to actually ingest your sample log file.

So the actual agent config block looks like:

<localfile>
  <log_format>syslog</log_format>
  <location>/var/log/oracle.log</location>
  <out_format>{"oraclelog":$(log)}</out_format>
</localfile>

And the rule I'm using is:

<group name="oracledb">
  <rule id="100002" level="3">
    <field name="oraclelog.type">com.oraclecloud.vcn.flowlogs.DataEvent</field>
    <description>OracleCloud flowlog parent rule</description>
  </rule>
</group>
Which issued the following kind of alerts in my dashboard:

Hope this helps!

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/XW8qrdpGwZc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/bcdcaf56-02ac-49fe-8fd1-b5cc59c068e9n%40googlegroups.com.

--

Federico Galland Pre-sales Engineer +1 (844) 349 2984 wazuh.com federico...@wazuh.com f-galland

[Utkarsh Bhargava]

Hi Federico,

Thank you for your quick response, I tried the configuration that you suggested still it seems that wazuh agent is not forwarding the logs to wazuh manager.

Is there any way I can force Wazuh Agent to read the logs from a specific file from the beginning of the file ?

regards

On Sep 20 2022, at 4:09 pm, Federico Gustavo Galland <federico...@wazuh.com> wrote:

Hi There!

I see you got the rules part addressed since the logtest is showing a triggered rule for your test event, so it's probably not related to that.

The logall_json variable enables logging of all incoming data to /var/ossec/logs/archives/archives.json regardless of the rule matching and decoding stages.

If the new log lines don't show in your archives file, it means that they are not getting to the manager at all.

In order to make sure you are correctly parsing your logs, I suggest you check that you are properly retrieving the log files (with possible use of a wildcard, since it seems your log names contain a hash).

Secondly I'd try and check that the log files ownership and permissions allow the wazuh user to read them.

In order to config your Agent, it's best to use centralized configuration by creating a group and assigning the Agent to it:

Go to groups:

With this done, I'd go to the agent's terminal and make sure you are using the right permissions for the files.

Usually, log files have 640 octal permissions, which you would change using chmod.

Aside from that you might have to need the wazuh user to the logging group of whatever software is dumping your logs.

You can reach back to us for assistance on the latter if needed.

I hope this helps a bit.

Regards,

Fede

On Tuesday, September 20, 2022 at 5:48:21 AM UTC-3 utk...@null.co.in wrote:

Hi Community,

I am running Wazuh 4.3.7 and trying to ingest Oracle Cloud Logs using Wazuh Agent 4.3.7.

Wazuh agent is reading the log file but I am unable to see anything on Kibana.

I have decoder and alert rules for the same still nothing is there.

I have also enabled logall option on Wazuh Manager but still no OCI logs are there.

I am understanding why agent is not forwarding the logs.

For your reference I have attached the agent logs screenshot decoder/alert rule screenshot and OCI sample logs.

Thanks & Regards

Utkarsh

 

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

[Federico Gustavo Galland]

Hi Utkarsh,

The first time you start the agent with the new <localfile> block it will parse the full output.

Afterwards it will only read new lines.

One technique to check whether some log is being ingested is appending whatever log you want to test into a file you know is being monitored properly, such as /var/log/messages or /var/log/syslog (if you can afford to mess those up a little).

So you can do something like:

cat testlog.log >> /var/log/messages

with your agent running and it should at least show up in the archives.json/archives.log files if you enable logall/logall_json.

Let me know if this helped.

[Utkarsh Bhargava]

Hi Federico,

I tried as you suggested but it didn't work for me.

thank you

On Sep 22 2022, at 9:33 pm, Federico Gustavo Galland <federico...@wazuh.com> wrote:

Hi Utkarsh,

The first time you start the agent with the new <localfile> block it will parse the full output.

Afterwards it will only read new lines.

One technique to check whether some log is being ingested is appending whatever log you want to test into a file you know is being monitored properly, such as /var/log/messages or /var/log/syslog (if you can afford to mess those up a little).

So you can do something like:

cat testlog.log >> /var/log/messages

with your agent running and it should at least show up in the archives.json/archives.log files if you enable logall/logall_json.

Let me know if this helped.

[Federico Gustavo Galland]

Hi Utkasrh,

Can you share some sample log for me to try to replicate on my end?

Regards,

Fede

[Utkarsh Bhargava]

Hi Federico,

Please find the attached sample logs.

thank you

On Sep 29 2022, at 10:18 pm, Federico Gustavo Galland <federico...@wazuh.com> wrote:

Hi Utkasrh,

Can you share some sample log for me to try to replicate on my end?

Regards,

Fede

[F Tux]

Hi There,

I was struggling to get your logs to work as well when I noticed the log lines are all above the maximum log size Wazuh can handle.

I'm actually getting the following kind of errors:

root@manager:~# grep Logs /var/ossec/logs/ossec.log
2022/09/30 12:57:09 wazuh-logcollector: ERROR: Large message size from file '/var/log/utkarsh/Logs_2022-09-11_04_27_to_2022-09-18_04_27_UTC.json' (length = 65279): '{"results":[{"data":{"datetime":1663475261000,"logContent":{"dat'...

Are these logs output in a single line format?

If you could change that, the Wazuh part should work just fine.

Regards,

Fede

[Utkarsh Bhargava]

These logs are from Oracle Cloud Infrastructure and are in JSON format.

I will search for some tool which can convert these logs into single line.

thank you for your help and support.

I will keep you posted about it.

regards

On Sep 30 2022, at 5:32 pm, F Tux <federic...@gmail.com> wrote:

Hi There,

I was struggling to get your logs to work as well when I noticed the log lines are all above the maximum log size Wazuh can handle.

I'm actually getting the following kind of errors:

root@manager:~# grep Logs /var/ossec/logs/ossec.log

2022/09/30 12:57:09 wazuh-logcollector: ERROR: Large message size from file '/var/log/utkarsh/Logs_2022-09-11_04_27_to_2022-09-18_04_27_UTC.json' (length = 65279): '{"results":[{"data":{"datetime":1663475261000,"logContent":{"dat'...

Are these logs output in a single line format?

If you could change that, the Wazuh part should work just fine.

Regards,

Fede

On Friday, September 30, 2022 at 3:13:43 AM UTC-3 utk...@null.co.in wrote:

Hi Federico,

Please find the attached sample logs.

thank you

On Sep 29 2022, at 10:18 pm, Federico Gustavo Galland <federico...@wazuh.com> wrote:

Hi Utkasrh,

Can you share some sample log for me to try to replicate on my end?

Regards,

Fede

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/5bcb6e5c-da67-4f24-aee3-2544eca81ad5n%40googlegroups.com.

[Federico Gustavo Galland]

Utkarsh,

The problem I've found is these logs are already dumped to a single line.

We need to have them split to process them correctly, otherwise the will surpass the 64KB per log limit.

You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/XW8qrdpGwZc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/F051D3BF-3425-4080-BF9A-FC3022A1F5A2%40getmailspring.com.