.db files in /var/ossec/queue/db/ disk usage

[Max Kalachov]

Hi guys,

I have around 90k .db files in /var/ossec/queue/db/ with dates from January till now with a total size of 23G.

What those files are? What do I do with them, can I get rid of them?

Wazuh: 4.03

In one of the posts on Github, regarding .db-wal files,  I saw the following "large file size can be a sign of an intensive use being made on your database so there is no opportunity to commit." is it relevant to my situation?  

Thank you

Regards,

Max

[Jose Luis Carreras Marin]

Hello Max Kalachov!

Let's take a good look at your environment. How many agents do you have connected to the manager? All the agents and the manager are Wazuh version 4.03?

If everything is correct, there should be one .db file for each agent connected. 

Regarding the issue you mentioned, it was a bug that was fixed in version 3.12, so it should not affect you. You can check the fix here: https://github.com/wazuh/wazuh/pull/4949
Anyway, do you have any of those *.db-wal files? The agent files should have the form 1234.db, corresponding to agent 1234.

On the other hand, the size of the database depends directly on the configuration that you have in the agents, if this is very generic it will necessarily consume a great amount of space. For example, in Windows, some data related to the monitoring, as for example the permissions, can be very noisy.

Regards

[Max Kalachov]

Hello Jose,

I have 59 agents in total on this server, all agents and the managers are version 4.03.

Currently, I have already over 90k files:

[root#wazuh-02-pr1 ~]# ls -l /var/ossec/queue/db/ | wc -l
94185

All files in the folder are .db and one file .db-journal I don't have any .db-wal files.

I see *.db files that corresponding to an agent number, however, I also have thousands of others that are not corresponding to any existing agent.

As an example I will take agent 10049:

[root#wazuh-02-pr1 db]# /var/ossec/bin/agent_control -l | grep 10049
   ID: 10049, Name: PS-PEER1-06, IP: any, Active

[root#wazuh-02-pr1 db]# ls -l | grep 10049
-rw-r----- 1 ossec ossec   245760 Jul  3 00:14 100495.db
-rw-r----- 1 ossec ossec   249856 Jul  3 00:16 100497.db
-rw-r----- 1 ossec ossec   249856 Jul  3 00:18 100499.db
-rw-r----- 1 ossec ossec  2744320 Aug 26 09:55 10049.db
-rw-r----- 1 ossec ossec   249856 Aug 20 09:10 210049.db

Regards,

Max

[Jose Luis Carreras Marin]

Hello,

I have been investigating possible causes, and I have found a bug that leaves the db files of each agent without deleting, but it only happens in cluster environments. Is this your case?

If so, the fix is included for the future version 4.3, here you can read all the related info:

https://github.com/wazuh/wazuh/issues/8327

Greetings, Jose

[Jose Luis Carreras Marin]

By the way,

if your case is that of the cluster, I can give you a punctual solution to make a "cleaning" of databases. You can use this script (you must execute it in each node of the cluster).

What it does is to eliminate all the databases that do not correspond with any agent (it checks it with the client.keys file), although the problem will continue existing until version 4.3 comes out.

I hope this can help you,

greetings!

[Max Kalachov]

Hello Jose,

Thank you for the investigation and reply. In my case, it's a standalone machine.

Should I still try the script or just try to delete files that not corresponding to the agents' IDs?

Thank you

Regards,

Max

[Jose Luis Carreras Marin]

Hello Max,

Yes, you can use the script to remove all *.db files that do not match with an agent. But this does not fix the error that caused this situation.
I found this other possible cause, duplicated agents names:
https://github.com/wazuh/wazuh/issues/8940
Can you check your ossec.log file? Some logs like this:

2021/08/31 07:27:49 wazuh-remoted: WARNING: (1408): Invalid ID X for the source ip: ''X.X.X.X' (name 'unknown').

Regards, Jose

[Max Kalachov]

Hi Jose,

Yes, I do have such log records for at least two IPs. Those two servers recently switched their IP between them.

I'm also certain that previously I had at least two servers with the same hostname, so probably the agent name would be also the same.

One more thing, I don't know if it's also might be related, but I had a server with wazuh-agent service flapping, it was going up and down many times a day.

The link you attached at least partially matches the situation.

I ran the script to clean up the *.db files and got the error: /usr/bin/mv: Argument list too long

The error is on line 31: mv -n /var/ossec/queue/db/

So I tried to modify line 31 it to: ls -1 /var/ossec/queue/db/ | xargs -i mv -n /var/ossec/queue/db/{}

Got many errors with for *.db files: mv: missing destination file operand after ‘/var/ossec/queue/db/XXXXXX.db’

So I modified the line 31 to: ls -1 /var/ossec/queue/db/ | xargs -i mv -n /var/ossec/queue/db/{} /tmp/agent-db/

This time the script worked, however, the API crushed and I needed to restart the server.

After the restart I have a resonable number of *.db files and resonable size.

[root#wazuh-02-pr1 ~]# ls -l /var/ossec/queue/db/ | wc -l

63
[root#wazuh-02-pr1 ~]# du -sh /var/ossec/queue/db/
201M    /var/ossec/queue/db/

Thank you

Regards,

Max

[Jose Luis Carreras Marin]

Hello Max

Well, then we have found the reason, although as you can see, the fix is still under development. For the time being, when you need it you can use the script to clean up.

I hope I've been helpful, and if you have any other questions I'll be happy to answer them.

Thanks and best regards!