Wazuh Disk Space = No incoming logs?

[Adam D.G]

Hi,

I am running a test setup. I have a wazuh-manager, with 5 servers that have agents installed which connect to the manager.

Few days ago all logs stopped coming in on my manager, and was just wondering if this is due to

the disk space availability on my manager. I have used up 95% of disk space.

root@wazuh-syslogger:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           395M  1.1M  394M   1% /run
/dev/sda2        20G   18G 1018M  95% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/loop1       90M   90M     0 100% /snap/core/6818
/dev/loop0       87M   87M     0 100% /snap/core/4917
/dev/loop2       90M   90M     0 100% /snap/core/6673
tmpfs           395M     0  395M   0% /run/user/1000

Would this be the reason the logs arent coming in/displaying in kibana. The agents are still connected though.

Kind Regards,

Adam

[Miguel Casares]

Hello Adam,

Yes, you are right. If you fill the disk space where Wazuh is installed, the Manager will stop storing logs since it is not able to store further data in the system. In addition, several processes will stop working and the Manager will show in the logs something like this:

2019/05/09 11:46:14 ossec-analysisd: ERROR: Could not write PID file '/var/run/ossec-analysisd-7399.pid': No space left on device (28)
2019/05/09 11:46:14 ossec-analysisd: CRITICAL: (1212): Unable to create PID file.
2019/05/09 11:46:28 ossec-syscheckd: ERROR: (1210): Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2019/05/09 11:46:28 rootcheck: CRITICAL: (1211): Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up..
2019/05/09 11:46:28 ossec-logcollector: ERROR: (1210): Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2019/05/09 11:46:29 ossec-logcollector: CRITICAL: (1211): Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up..
2019/05/09 11:46:29 wazuh-modulesd: ERROR: (1210): Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2019/05/09 11:46:29 wazuh-modulesd: ERROR: (1210): Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2019/05/09 11:47:17 sca: ERROR: Can't connect to queue.
2019/05/09 11:47:17 wazuh-modulesd: ERROR: (1210): Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2019/05/09 11:47:17 wazuh-modulesd:syscollector: ERROR: Can't connect to queue.
2019/05/09 11:47:21 wazuh-modulesd: ERROR: At wm_sendmsg(): Unable to send message to queue: (No such file or directory)
2019/05/09 11:47:21 sca: ERROR: (1210): Queue '/queue/ossec/queue' not accessible: 'No such file or directory'.

Additionally, the Manager will not show that the Agents are active and reporting. I highly recommend to free up space in the disk. You may remove old logs from the alerts folder, for example. You may check the size of the folders using the following command:

du -bsh /folder

Finally, I would like to add that we are working in a method to improve the log rotation. Thus, you may configure in the future how long you want to store the logs of Wazuh, and it will avoid problems like this one in the future. You can check it out here: https://github.com/wazuh/wazuh/issues/3072

Let me know if you have any further questions. I will be happy to help.

Regards,

Miguel Casares

[Adam D.G]

Thank you . So for now it's cron jobs I take it ? :P

On Thursday, May 9, 2019 at 12:06:52 PM UTC+2, Adam D.G wrote:

[Adam D.G]

Actually,  I just tailed alerts.log. Logs are coming in there, but just are not coming up on Kibana.

What would be the cause of this. I had logs coming up on Kibana until about the 6th of May but seems to have stopped.

Any help or ideas with this would be appreciated.

Kind Regards,

Adam

On Thursday, May 9, 2019 at 12:06:52 PM UTC+2, Adam D.G wrote:

[Miguel Casares]

Hello Adam, 

Could you check if all the processes of Wazuh are up and running?

ps aux | grep ossec

In addition, the problem could be in another component. I will share a guide of troubleshooting that I usually use in cases like this one:

We have to check if Filebeat is reading the alerts.json file:

lsof /var/ossec/logs/alerts/alerts.json
COMMAND    PID  USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
ossec-ana 2346 ossec    9w   REG    8,1   288241 1836217 /var/ossec/logs/alerts/alerts.json
filebeat  2629  root    5r   REG    8,1   288241 1836217 /var/ossec/logs/alerts/alerts.json

We also have to check if Filebeat is sending the logs to Logstash and if there is communication between Filebeat and Logstash:

filebeat test output
logstash: IP:5000...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: IP
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK

In Logstash you can check if it is working by checking the logs:

/var/log/logstash/logstash-plain.log

[2019-04-16T07:54:49,457][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5000"}
[2019-04-16T07:54:49,521][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x785bc2c1 run>"}
[2019-04-16T07:54:49,659][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-04-16T07:54:49,713][INFO ][org.logstash.beats.Server] Starting server on port: 5000
[2019-04-16T07:54:50,213][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

In Elasticsearch you can check if the index was created and if Elasticsearch is working by:

curl IP_Elasticsearch:9200/_cluster/health?pretty
{
  "cluster_name" : "my-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 21,
  "active_shards" : 21,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0

curl IP_Elasticsearch:9200/_cat/indices
green open wazuh-monitoring-3.x-2019.03.22 iIV07lTrR_GaGS0JsILGEA 2 0    6 0    35kb    35kb
green open .wazuh-version                  ZItpZuyaRr-1sW7ZAGL0Xg 1 0    1 0   5.1kb   5.1kb
green open wazuh-monitoring-3.x-2019.04.16 PUWkHBk6Sw2snPk7NL8Iug 2 0   28 0   150kb   150kb
green open .kibana_1                       A4SoA1C2QUWXWm2rBL8TjQ 1 0    2 0  13.1kb  13.1kb
green open palo-alto-2019.03.21            OXeGs99FR3CeTAG1AgEe1g 1 0 5115 0 755.1kb 755.1kb
green open palo-alto-2019.03.22            lWhB_r1YR0CbI5_LQlvDYQ 1 0 9020 0   1.2mb   1.2mb
green open wazuh-monitoring-3.x-2019.03.21 bZ2rqN2FSTOHqtD9YDKklQ 2 0    0 0    522b    522b
green open .kibana_3                       BO664i0LRhuu_bNl0NwyVQ 1 0    6 1    85kb    85kb
green open .kibana_2                       Fs2NPw55QRK5FyxE0QdCfQ 1 0    6 0  55.2kb  55.2kb
green open wazuh-alerts-3.x-2019.04.16     1vxEQpnWT6ePhvXtjgzXRA 1 0  286 0   469kb   469kb
green open wazuh-monitoring-3.x-2019.04.05 ldcOzaS7R0WsDcjvZtWn7A 2 0   14 0  73.8kb  73.8kb
green open wazuh-alerts-3.x-2019.03.22     4Qp4t8vSS9eDTlc_N6zPjQ 1 0 3917 0 940.3kb 940.3kb
green open .tasks                          zm93XMt6ST-h64PH2s_fIQ 1 0    1 0   6.2kb   6.2kb
green open wazuh-alerts-3.x-2019.03.21     daTjtDX4RS2dwYgi2yGeLw 1 0 5571 0     1mb     1mb
green open wazuh-alerts-3.x-2019.04.05     p3HZyYnfQCqTeudOV_gmBA 1 0   79 0 156.1kb 156.1kb
green open .wazuh                          VIhi_jdySrqwXDn42gFobQ 1 0    2 0  21.6kb  21.6kb
green open wazuh-alerts-3.x-2019.04.04     f0UtQ8O0S465U7YUF-HoLA 1 0  508 0 430.7kb 430.7kb

Do not forget to change the <IP> tag for the IP of your server. 

You can also check the logs of Elasticsearch:

/var/log/elasticsearch/

If everything is correct and Kibana is working, Kibana will be able to show the data.

I hope it helps. Let me know if you have any question.

Regards,

Miguel Casares

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/812c41b6-e75d-475d-8189-6715c377632c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Adam D.G]

The filebeat test output was not connecting.

I am running a  single-host setup so based on the documentation I can see that it is not needed for this type of architecture. I just simply removed filebeat and restarted the box.

Can see logs coming through now and will keep an eye on it. Dont know if this was the fix but will keep you posted.

Kind Regards,

Adam

On Thursday, May 9, 2019 at 12:06:52 PM UTC+2, Adam D.G wrote:

[Miguel Casares]

Hello Adam,

Probably was something wrong with the configuration and that impede to the logs come to Logstash. In a single node architecture, Logstash should be the component responsible for reading the alerts file.

If you face this issue again or any other issue, please let me know. Definitely, we can help you out.

Regards,

Miguel Casares

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/9297e61e-531d-4166-874b-61051cea032b%40googlegroups.com.

[Adam D.G]

Thank you, appreciate all the help so much.

Will keep you updated.

Kind Regards,
Adam

On Thursday, May 9, 2019 at 12:06:52 PM UTC+2, Adam D.G wrote:

[Adam D.G]

Hi,

Just checked and still getting the same issue.

If I run root@wazuh-syslogger:/etc/logstash# tail -f /var/log/logstash/logstash-plain.log

I get this output :

[2019-05-13T10:18:39,741][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
[2019-05-13T10:18:39,742][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-05-13T10:18:39,742][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
[2019-05-13T10:18:39,772][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-05-13T10:18:39,772][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}

What do you think this means?

Also if i run that curl command with my box's IP :

root@wazuh-syslogger:/etc/logstash# curl 192.168.70.69:9200/_cluster/health?pretty
curl: (7) Failed to connect to 192.168.70.69 port 9200: Connection refused

Any help would be appreciated.

Will see what I can do in the meantime.

Kind Regards,

Adam

On Thursday, May 9, 2019 at 12:06:52 PM UTC+2, Adam D.G wrote:

[Miguel Casares]

Hello again Adam,

I can help you here.

Elasticsearch has changed your indices settings to read_only. This implies you can't index new data in them.

This is related to low storage in your node. You should either increase your storage capacity or delete old indices to prevent this from happening again.

For the time being, you can disable the read_only setting with the following command in Kibana dev tools:

PUT */_settings
{
 "index.blocks.read_only_allow_delete": null
}

Make sure to restart Kibana:

systemctl restart kibana

Regarding Elasticsearch, could you provide the configuration of Elasticsearch, please? It should be located in the following path:

/etc/elasticsearch/elasticsearch.yml

Hope this will help.

Regards,

Miguel Casares

[Adam D.G]

Thank you.

Well this is only a small lab setup, so no biggy but thank you for the info and workaround. Good to know now it is def space related.

On Thursday, May 9, 2019 at 12:06:52 PM UTC+2, Adam D.G wrote:

[Miguel Casares]

Hello Adam,

My pleasure. Let me know if you have any further questions. We will be more than happy to help.

Regards,

Miguel Casares

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/92f45efe-73c6-4129-8e5b-af25038fbf3d%40googlegroups.com.

[Adam D.G]

Hi,

How do I go about freeing up space on my wazuh manager? Do I have to delete old indices? Logs? Still relatively new to this system.

Kind Regards,
Adam

On Thursday, May 9, 2019 at 12:06:52 PM UTC+2, Adam D.G wrote:

[Miguel Casares]

Hello Adam,

It is only necessary to free up space only if the server runs out of disk space. Otherwise, it is not necessary.

If you want to free up space you may remove old alerts and logs that you do not need anymore.

Sorry for the inconvenience that this may be caused you. We are working in an option to do it automatically, hopefully, we will get it in the next future: https://github.com/wazuh/wazuh/issues/3072

Do not hesitate to seek help if you need it.

Regards,

Miguel Casares

[Adam D.G]

Hi,

Thanks for the info. Well I am testing right now with pretty small VM's (20gb). We are looking at rolling this out on our core. Is it safe to assume that this functionality you are working on to rotate logs will be

available with 3.10.0 ?

Kind Regards,

Adam

[Miguel Casares]

Hello Adam,

Yes, that is a pretty small environment. For a production environment, we usually recommend around 200GB in the Manager, this way and with the current rotation option, it is hard to face that kind of problem.

We are working so hard to achieve this functionality in the next version, so hopefully, you will be able to get it.

Do not hesitate to seek help!

Regards,

Miguel Casares

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/14ad9650-381b-452a-8ba4-49bf178f4ce0%40googlegroups.com.

[Adam D.G]

Thanks Miguel,

Much appreciated.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.