Wazuh Log Is not parsed

[Suat Toksöz]

Hi,

We are trying to get syslog-514-UDP from 10.10.10.15 IP address, the port is forwarded and I can see the log on the wazuh server with tcpdump, but I can not see log on kibana?

tcpdump -v 10.10.10.15

10.10.10.15.51867 > testwazuh.syslog: SYSLOG, length: 453
Facility local0 (16), Severity info (6)
Msg: 1 2020-03-31T13:14:50+03:00 test.com SecurtySystem: - - - 2020-03-31 13:14:50 - NODE1 - [176.40.171.192] USER01(Users)[...............] - Host Checker policy '.............' failed on host '179.140.172.182' address '34-dd-52-99-80-52'  for user 'USER01' reason 'Anti-Virus software listed in security requirements is not installed.;Rule-DomainMember:Registry value mismatch for configured key, found='.
13:15:03.345184 IP (tos 0x0, ttl 64, id 53158, offset 0, flags [DF], proto UDP (17), length 254)

--

Best regards,

Suat Toksoz

[Francisco Navarro]

Hello Suat,

My recommendation here would be to enable the “logall” parameter in the /var/ossec/etc/ossec.conf file (See https://documentation.wazuh.com/3.12/user-manual/reference/ossec-conf/global.html#logall), then, restart the manager and check the /var/ossec/logs/archives/archives.json to see if it contains any of your desired logs.

If /var/ossec/logs/archives/archives.json contains your logs, disable logall, that means that the manager is analyzing your logs but not generating alerts. Is important to disable logall because it makes your manager to store lot of information and can make it to fill yoru disk quickly.

You won’t have any log in Kibana, Kibana only store alerts generated by logs analyzed by Wazuh. Wazuh need to decode the log with certain decoder and it must match any rule of our ruleset in order to generate alerts. You have more information about how this work here: https://documentation.wazuh.com/3.12/user-manual/capabilities/log-data-collection/how-it-works.html

In order to easily check if your logs would generate alerts, I recommend using the ossec-logtest utility (See https://documentation.wazuh.com/3.11/user-manual/reference/tools/ossec-logtest.html)

/var/ossec/bin/ossec-logtest`

If you paste some logs in the console after executing that command, the logs will be analyzed and you will know if they can be decoded with any available decoder and if they match any rule.

If your log is not being processed by any decoder or doesn’t match any rule, you need to write your own custom rules or decoders for it. Read this for further information: https://documentation.wazuh.com/3.12/user-manual/ruleset/custom.html

​

[Suat Toksöz]

Hi Francisco,thanks

Nothing from the logall=yes optins.

I have tested the log here is the results:

**Phase 1: Completed pre-decoding.
       full event: '14:25:55.020782 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto UDP (17), length 481) 10.10.1015.18516 > .........................................:Registry value mismatch for configured key, found='.'
       timestamp: '(null)'
       hostname: 'testwazuh'
       program_name: '(null)'
       log: '14:25:55.020782 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto UDP (17), length 481) 10.10.10.15.18516 > ................................................ed.;Rule-DomainMember:Registry value mismatch for configured key, found='.'

**Phase 2: Completed decoding.
       No decoder matched.

**Phase 3: Completed filtering (rules).
       Rule id: '1002'
       Level: '2'
       Description: 'Unknown problem somewhere in the system.'
**Alert to be generated.

Even no alerts generated I should at least see the log on the kibana right?

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/9f284df5-da09-4d78-a17a-0697d9d869c1%40googlegroups.com.

[Francisco Navarro]

Hello Suat,

No, Elasticsearch won't store all your logs send to Wazuh, it wouldn't make sense. The point of having a log analysis engine is to store only alerts generated for certain events instead of archive thousands of meaningless logs.

Are you sure that your logs doesn't reach the `archive.json` file after enabling `logall` parameter? If that is true, your manager probably is not configured  your manager to listen to the incoming logs.

Could you share with us your configuration for remote logs or the full manager `ossec.conf` ?

If you have not configured yet your manager to listen to incomming logs, please take a look to this page of our documentation: https://documentation.wazuh.com/3.12/user-manual/capabilities/log-data-collection/how-it-works.html#remote-syslog

Also, which software is generating these logs? I cannot recognize such log format and neither Wazuh, if that software is not a well known one, for example, some software written by you, you will need to create some decoders and rules to generate alerts.

If you want all your logs to generate alerts, regardless of its content, so they reach Kibana, you can always create a simple decoder for all of them and a single generic rule that always trigger an alert of level 4 or higer.

I hope we can solve your problem soon.

Best regards,

[Suat Toksöz]

 Hi Francisco,

Again test the logall , but nothing is coming up on alert.jsonfile.

 manager `ossec.conf

 <remote>
    <connection>syslog</connection>
    <port>514</port>
    <protocol>udp</protocol>
    <allowed-ips>0.0.0.0/0</allowed-ips>
    <local_ip>managerwazuhserverip</local_ip>
  </remote>

the log is PulseSecure sslvpn logs, I just want to make sure that the logs are coming to wazuh.

thanks

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/28f2d877-c853-4c9a-982d-5bcf2500efac%40googlegroups.com.

[Francisco Navarro]

Hello Suat,

Okey, could you share your `ossec.log` file? You should have something like that in there:

```

2020/04/01 09:32:13 ossec-remoted[12897] remoted.c:52 at HandleRemote(): INFO: Remote syslog allowed from: '0.0.0.0/0'
2020/04/01 09:32:13 ossec-remoted[12897] remoted.c:113 at HandleRemote(): INFO: Started (pid: 12901). Listening on port 514/UDP (syslog).
```

Also, check the port is properly open and listening in the desired interface:

netstat -tunap | grep 514

The output should look like this:

udp        0      0 0.0.0.0:514             0.0.0.0:*                           8821/ossec-remoted

If that is configured your manager is properly configured to allow syslog connections from any IP.

Also, I would like to ask you to set the exact ip of the firewall in the `allowed_ips` field just to be sure there are no problem related to allowing all the ips incomings (`0.0.0.0`)

Another usefull thing could be that you share a gist with the content of your archives.logs after enabling again `logall` and restart the manager.

Best regards,

[Suat Toksöz]

Hi Francisco,

ossec log file as the following logs;

:~# cat /var/ossec/logs/ossec.log 

2020/04/01 11:40:54 ossec-remoted: INFO: Remote syslog allowed from: '0.0.0.0/0'
2020/04/01 11:40:54 ossec-remoted: INFO: Started (pid: 7230). Listening on port 514/UDP (syslog).

netstat is looking good as well:

udp        0      0 10.10.10.13:514        0.0.0.0:*                           7230/ossec-remoted 

firewall is set there is not problem on that also, tcpdump shows the logs are coming to wazuh manager.

I am not sure what is your suggestion on "Another usefull thing could be that you share a gist with the content of your archives.logs after enabling again `logall` and restart the manager.

"

Thanks

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/15158195-5a42-4358-b32a-c7c668442228%40googlegroups.com.

[Francisco Navarro]

I mean that everything seems to be correct in your configuration, I would like you, after activating the logall parameter as I mentioned (and having restarted the manager), to wait a few minutes and check the content of the /var/ossec/logs/archives/archives.json file and share it with us to make sure it doesn’t fail.

I would also like you to try to modify the agent configuration so that the allowed_ips field contains exactly the firewall IP, just to check that there are no networking problems related to the use of 0.0.0.0/0.

Best regards.

​

[Suat Toksöz]

Hi Francisco,

Just changed the logall on wazuh manager ossec.conf file, and listen to tail -f /var/ossec/logs/archives/archives.log | grep "10.10.10.20" --color and I am getting the logs.

Q1 - Why I don't have /var/ossec/logs/archives/archives.json ?

Q2 - Why I don't get any alerts-notice-logs on the kibana dashboard for this IP address?

Thanks

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/1499ef97-2f15-46ce-b0fa-556664dd0286%40googlegroups.com.

[Francisco Navarro]

Hello Suat,

My fault, there are two different parameters available in the Wazuh configuration, logall and logall_json, I should have told you to use the second one or check archives.log, I was confused when I told you.

Remember to disable logall to avoid disk usage issues.

Regarding your second question, as I said, Kibana will only store alerts generated by Wazuh. You shared this:

**Phase 2: Completed decoding.
       No decoder matched.

**Phase 3: Completed filtering (rules).
       Rule id: '1002'
       Level: '2'
       Description: 'Unknown problem somewhere in the system.'

That mean, your logs are not being correctly decoded and they match a level 2 rule (only alerts with level 3 or higher will generate alerts)

You have various options here. My recomendation would be to write a custom decorder to match your logs format and a custom rule to generate alerts for them.

Here you have more information about custom rules and decoders: https://documentation.wazuh.com/3.12/user-manual/ruleset/custom.html

Another option would be defining a smaller alert level threshold: https://documentation.wazuh.com/3.12/user-manual/manager/alert-threshold.html for example, set it to 1 would make your current alert ‘unknown problem somewhere in the system’ to reach KIbana, but I don’t recommend this option, I encourage you to make at least a simple decoder for your logs and a simple rule common for all the logs decoded with that decoder. Then you will be able to create child rules for specific logs to generate interesting alerts.

I hope that help you, if you want to create custom rules and decoders and need help Ia encorauge you to open a new different thread in the forum or ask us in our slack channel.

If you have further question related with this topic do not hesitate to keep asking in this thread.

Best regards.

​

[Suat Toksöz]

Hi Francisco,

Thanks for the help.

Here is the another log which is rule level = 2 , this log shows up on kibana dashboard. I can not understand why some of logs alerts(level2) shows up on kibana , while others does not show up on kibana.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/6e500f55-81e3-4e59-821b-97e2c8bbec30%40googlegroups.com.

[Francisco Navarro]

Hello Suat, that is not possible, probably you're missing something.

Can you please share with us an example of an alert in your Kibana environment with level two and another alert in `/var/ossec/logs/alerts/alert.json` file with level 2 and a query in Kibana that proves if that alert is not being shown in the Kibana interface?

An important detail here: even ossec-logtest says that your log would generate an alert of level 2, it also says that your log doesn't match with any decoder, so probably that level 2 alert is not being generated.

I recommend you to write a decoder your logs.

Best regards,

[Suat Toksöz]

Hi Francisco,

I will go with local decoder & local rule set for this log. Here is the log I want to create a decoder & a ruleset:

The sample log :

1 2020-04-03T12:40:17+03:00 sslvpn.test.com PulseSecure: - - - 2020-04-03 12:40:17 - NODE1 - [18.133.320.333] test00001'(Users)[SSLVPNTEST1] - Host Checker policy 'SSLVPMODE1' failed on host '718.273.230.333' address 'w8-w2-w6-wa-wa-w1'  for user 'test00001' reason 'Rule-ANTIVURUS:Anti-Virus software listed in security requirements is not installed.;Rule-DomainMember:Registry value mismatch for configured key, found='

Local Decoder:

<decoder name="example">

<prematch>^1 2020</prematch>

</decoder>

<decoder name="example-fields">

<parent>example</parent>

<prematch>status</prematch>

<regex offset="after_prematch">\w\w\w\w\d\d\d\d\d</regex>

<order>status</order>

</decoder>

Local Rule Set:

<group name="local,example,">

<rule id="100005" level="3">

<decoded_as>example</decoded_as>

<match>Anti-Virus software listed in security requirements is not installed</match>

<description> SSLVPN Antivirus not Installed</description>

</rule>

</group>

Log Test Results:

**Phase 1: Completed pre-decoding.
       full event: '1 2020-04-03T12:40:17+03:00 sslvpn.test.com PulseSecure: - - - 2020-04-03 12:40:17 - NODE1 - [18.133.320.333] test00001(Users)[SSLVPNTEST1] - Host Checker policy 'SSLVPMODE1' failed on host '718.273.230.333' address 'w8-w2-w6-wa-wa-w1'  for user 'test00001' reason 'Rule-ANTIVURUS:Anti-Virus software listed in security requirements is not installed.;Rule-DomainMember:Registry value mismatch for configured key, found='.'
       timestamp: '(null)'
       hostname: 'zkxswazuh'
       program_name: '(null)'
       log: '1 2020-04-03T12:40:17+03:00 sslvpn.test.com PulseSecure: - - - 2020-04-03 12:40:17 - NODE1 - [18.133.320.333] go800711(Users)[SSLVPNTEST1] - Host Checker policy 'SSLVPMODE1' failed on host '718.273.230.333' address 'w8-w2-w6-wa-wa-w1'  for user 'test00001' reason 'Rule-ANTIVURUS:Anti-Virus software listed in security requirements is not installed.;Rule-DomainMember:Registry value mismatch for configured key, found='.'

**Phase 2: Completed decoding.
       decoder: 'squid-accesslog'

**Phase 3: Completed filtering (rules).

       Rule id: '35000'
       Level: '0'
       Description: 'Squid messages grouped.'

What I am missing ? Thanks

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/407d14bb-6320-4387-bcd5-d988fd4923ad%40googlegroups.com.

[Francisco Navarro]

Hello Suat, sorry for the late answer, I thought I had already answered you.

Your problem is that ^ in the first decoder:

<decoder name="example">
    <prematch>^1 2020</prematch>
</decoder>

Just remove it and write it like

<decoder name="example">
    <prematch>^1 2020</prematch>
</decoder>

And it should work!

I think that the problem is that ^ doesn’t work well with numbers… you could use it with string words but not with that 1

Best regargs,

​