pfSense stopping sending logs

[Felipe Kuchnier]

Hi there,

I have a wazuh server with 8 agents, one of them is a pfSense FreeBSD 14.0 + Snort + custom decoders and rules.

Lately I've been having problems where the agent is still active but it doesn't send my IDS logs anymore, it only sends them again when I restart the agent.

Searching through the agent logs I found that these logs may point to the problem, but I couldn't find any solution online:

$ cat /var/ossec/logs/ossec.log

2024/06/13 08:27:45 wazuh-logcollector: ERROR: socketerr (not available).
2024/06/13 08:27:45 wazuh-logcollector: ERROR: Unable to send message to 'queue/sockets/queue' after a successfull reconnection...
2024/06/13 08:27:45 wazuh-logcollector: ERROR: socketerr (not available).
2024/06/13 08:27:45 wazuh-logcollector: ERROR: Unable to send message to 'queue/sockets/queue' (wazuh-agentd might be down). Attempting to reconnect.
2024/06/13 08:27:45 wazuh-logcollector: INFO: Successfully reconnected to 'queue/sockets/queue'

Yesterday Wazuh stopped receiving pfSense logs at 17:11:

-----

Note that even with these various error messages, the logs were being sent without problems, but when the error stops, so do the logs.

I don't think it's something to do with the number of alerts per second, since when I first installed the agent, snort flooded the server with lots of alerts and it worked without any problems, now I'm filtering the false positives directly through snort, so it doesn't generate unnecessary logs for wazuh to process and the problem started to appear.

-----------------------------------------

Here are some command outputs and ossec.conf (attached) to see if anyone can help me:

$ tail /var/ossec/logs/ossec.log
2024/06/13 08:45:45 wazuh-logcollector: ERROR: Unable to send message to 'queue/sockets/queue' (wazuh-agentd might be down). Attempting to reconnect.
2024/06/13 08:45:45 wazuh-logcollector: INFO: Successfully reconnected to 'queue/sockets/queue'
2024/06/13 08:51:45 wazuh-logcollector: ERROR: socketerr (not available).
2024/06/13 08:51:45 wazuh-logcollector: ERROR: Unable to send message to 'queue/sockets/queue' (wazuh-agentd might be down). Attempting to reconnect.
2024/06/13 08:51:45 wazuh-logcollector: INFO: Successfully reconnected to 'queue/sockets/queue'
2024/06/13 08:51:45 wazuh-logcollector: ERROR: socketerr (not available).
2024/06/13 08:51:45 wazuh-logcollector: ERROR: Unable to send message to 'queue/sockets/queue' after a successfull reconnection...
2024/06/13 08:51:45 wazuh-logcollector: ERROR: socketerr (not available).
2024/06/13 08:51:45 wazuh-logcollector: ERROR: Unable to send message to 'queue/sockets/queue' (wazuh-agentd might be down). Attempting to reconnect.
2024/06/13 08:51:45 wazuh-logcollector: INFO: Successfully reconnected to 'queue/sockets/queue'

$ /var/ossec/bin/wazuh-control status
wazuh-modulesd is running...
wazuh-logcollector is running...
wazuh-syscheckd is running...
wazuh-agentd is running...
wazuh-execd is running...

$ ps aux | grep wazuh
root    2771    0.0  0.0   20464    7688  -  S    07:57        0:00.05 /var/ossec/bin/wazuh-execd
wazuh  10590    0.0  0.0   38460   10468  -  S    07:57        0:00.59 /var/ossec/bin/wazuh-agentd
root   19934    0.0  0.0   35504   12688  -  SN   07:57        0:01.91 /var/ossec/bin/wazuh-syscheckd
root   21529    0.0  0.0   53384   10860  -  S    07:57        0:00.39 /var/ossec/bin/wazuh-logcollector
root   29530    0.0  0.0  151808   22556  -  I    07:57        0:00.43 /var/ossec/bin/wazuh-modulesd
root   48619    0.0  0.0   12752    2360  0  S+   08:52        0:00.00 grep wazuh

$ ls -lrt /var/ossec/queue/sockets/queue
srw-rw----  1 wazuh wazuh 0 Jun 13 07:57 /var/ossec/queue/sockets/queue

Thanks in advance.

[Roman Luna]

Hi,

How are you forwarding the logs from the device to the agent? I would recommend installing rsyslog and using a localfile to ingest the files, here is a guide on how to do so: https://documentation.wazuh.com/current/cloud-service/your-environment/send-syslog-data.html.

Also, is the agent installed in Linux? As you mentioned Freebsd pfsense, I would like to get a clarification on that regard just in case. Thre is an option to install the Wazuh Agent on Freebsd too. Additionally, what do you mean by snort? Is it another program that you are using?

Regards.

[Felipe Kuchnier]

Hello, Roman

So, I'm using a firewall called pfSense, which is based on the FreeBSD operating system, I installed the agent on the operating system via its package manager.
Snort is a module for the firewall, it is an IDS, which performs intrusion detection for the network and generates logs for /var/log/snort/.../alerts, which is where wazuh gets the logs.

According to my ossec.conf, this is how I'm getting the logs from the IDS:

<localfile>
    <log_format>syslog</log_format>
    <location>/var/log/snort/snort_bce157424/alert</location>
</localfile>

The other logs have the agent's default configuration.

Until then everything was fine, working as I wanted it to, but then the events started to stop appearing on the wazuh server.

And when I went through the logs I found those logs about socket errors.

Regards.

[Gorka Etxebarria]

I am encountering the same problem but with Suricata. I have conducted all sorts of tests, changed parameters in the logcollector, and even updated the agent to 4.8, but I am still facing the same issue as you and haven't found a way to fix it for several days. 

I've noticed it happens more often when I generate an alert, which I understand is when it writes to Suricata's `eve.json`. It also occurs more frequently with interfaces that have more traffic.

 I have also made changes to the queue, but I haven't managed to stop those error messages from appearing. 

Let's hope someone can help us resolve the problem. 

Best regards.

[Roman Luna]

Hi,

The alert could be generated, but it is not shown in the indexer, meaning it won't be displayed in the dashboard. Do you any other alerts from any agent on the dashboard? This is to check that you still have space left.

Additionally, from the dev tools in the dashboard, you can run the following query. This would display the amount of shards and the cluster state:

   GET _cluster/health

Also, it could be important to check that the logs are reaching the manager. For this, you could grep the alerts or the archives, in your case, as you had previous alerts it should be present.

    grep -i '<pattern-from-pfsense>' /var/ossec/logs/alerts/alerts.json

This will grep the current alerts for the day to check if there are any presents, if there are not, try to reinject an alert into the log file that you are monitoring:

   echo '<log-line>' >> /var/log/<log-file-name>

You also mentioned there could be flooding issues, without having more information, we can change the queue to maximum just in case. You can apply the following in the centralized configuration in a group, instead of doing it in the agent itself. I have already included the agent_config brackets to encapsulate the configuration, you can add multiple of them in the group configuration:

<agent_config>

  <client_buffer>
    <!-- Agent buffer options -->
    <disabled>no</disabled>
    <queue_size>10000</queue_size>
    <events_per_second>1000</events_per_second>
  </client_buffer>

</agent_config>

Documentation:  Centralized configuration (agent.conf) - Reference (wazuh.com)

Let me know if you have questions.

[Felipe Kuchnier]

Hi Gorka

I thought about switching from snort to suricata to see if it would solve the problem, but then it won't.

I think it's a problem with FreeBSD, and as it's not officially supported by Wazuh, we won't get any feedback on this problem.