Errors in module wazuh-db Version: 4.7.3-1

[ShtrudelMan]

Good evening, colleagues!
I faced a problem.
I connected 4 agents on the test bench with version 4.5 each.
Moved them from the "Default" group to their special group using the interface tools on the Wazuh-dashboard website.
Then logged into the server and ran the command "shutdown -r now" on the server running Debian 11.
The Wazuh control panel does not load after a reboot.

These are errors from the log: "tail -f /var/ossec/logs/ossec.conf"

2024/07/22 20:01:58 wazuh-db: ERROR: DB(067) sqlite3_prepare_v2() stmt(88): no such table: sync_info
2024/07/22 20:01:58 wazuh-db: ERROR: DB(067) sqlite3_prepare_v2() stmt(90): no such table: fim_entry
2024/07/22 20:01:58 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/07/22 20:02:01 wazuh-db: ERROR: DB(069) sqlite3_prepare_v2() stmt(88): no such table: sync_info
2024/07/22 20:02:01 wazuh-db: ERROR: DB(069) sqlite3_prepare_v2() stmt(90): no such table: fim_entry
2024/07/22 20:02:01 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/07/22 20:02:06 wazuh-db: ERROR: DB(070) sqlite3_prepare_v2() stmt(88): no such table: sync_info
2024/07/22 20:02:06 wazuh-db: ERROR: DB(070) sqlite3_prepare_v2() stmt(86): no such table: sync_info

admin@Wazuh-Server:~$ sudo systemctl status wazuh-dashboard.service
● wazuh-dashboard.service - wazuh-dashboard
     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-07-22 19:56:22 MSK; 19min ago
   Main PID: 7424 (node)
      Tasks: 11 (limit: 14305)
     Memory: 150.4M
        CPU: 15.331s
     CGroup: /system.slice/wazuh-dashboard.service
             └─7424 /usr/share/wazuh-dashboard/node/bin/node --no-warnings --max-http-header-size=65536 --unhandled-rejections=warn /usr/share/wazuh-dashboard/src/cli/dist -c /etc/wazuh-dashboard/opensearch_dashboards.yml

июл 22 20:05:11 Wazuh-Server opensearch-dashboards[7424]: {"type":"log","@timestamp":"2024-07-22T17:05:11Z","tags":["error","opensearch","data"],"pid":7424,"message":"[ResponseError]: Response Error"}
июл 22 20:05:13 Wazuh-Server opensearch-dashboards[7424]: {"type":"log","@timestamp":"2024-07-22T17:05:13Z","tags":["error","opensearch","data"],"pid":7424,"message":"[ResponseError]: Response Error"}
июл 22 20:05:13 Wazuh-Server opensearch-dashboards[7424]: {"type":"log","@timestamp":"2024-07-22T17:05:13Z","tags":["error","opensearch","data"],"pid":7424,"message":"[ResponseError]: Response Error"}
июл 22 20:05:14 Wazuh-Server opensearch-dashboards[7424]: {"type":"log","@timestamp":"2024-07-22T17:05:14Z","tags":["error","opensearch","data"],"pid":7424,"message":"[ResponseError]: Response Error"}
июл 22 20:05:16 Wazuh-Server opensearch-dashboards[7424]: {"type":"log","@timestamp":"2024-07-22T17:05:16Z","tags":["error","opensearch","data"],"pid":7424,"message":"[ResponseError]: Response Error"}
июл 22 20:05:16 Wazuh-Server opensearch-dashboards[7424]: {"type":"log","@timestamp":"2024-07-22T17:05:16Z","tags":["error","opensearch","data"],"pid":7424,"message":"[ResponseError]: Response Error"}
июл 22 20:05:16 Wazuh-Server opensearch-dashboards[7424]: {"type":"log","@timestamp":"2024-07-22T17:05:16Z","tags":["error","opensearch","data"],"pid":7424,"message":"[ResponseError]: Response Error"}
июл 22 20:05:19 Wazuh-Server opensearch-dashboards[7424]: {"type":"log","@timestamp":"2024-07-22T17:05:19Z","tags":["error","opensearch","data"],"pid":7424,"message":"[ResponseError]: Response Error"}
июл 22 20:05:19 Wazuh-Server opensearch-dashboards[7424]: {"type":"log","@timestamp":"2024-07-22T17:05:19Z","tags":["error","opensearch","data"],"pid":7424,"message":"[ResponseError]: Response Error"}
июл 22 20:05:19 Wazuh-Server opensearch-dashboards[7424]: {"type":"log","@timestamp":"2024-07-22T17:05:19Z","tags":["error","opensearch","data"],"pid":7424,"message":"[ResponseError]: Response Error"}
admin@Wazuh-Server:~$ 

admin@Wazuh-Server:~$ sudo systemctl status wazuh-manager.service
● wazuh-manager.service - Wazuh manager
     Loaded: loaded (/lib/systemd/system/wazuh-manager.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-07-22 19:53:04 MSK; 21min ago
    Process: 5211 ExecStart=/usr/bin/env /var/ossec/bin/wazuh-control start (code=exited, status=0/SUCCESS)
      Tasks: 144 (limit: 14305)
     Memory: 1.0G
        CPU: 3min 23.810s
     CGroup: /system.slice/wazuh-manager.service
             ├─5280 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py
             ├─5281 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py
             ├─5284 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py
             ├─5287 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py
             ├─5310 /var/ossec/bin/wazuh-integratord
             ├─5331 /var/ossec/bin/wazuh-authd
             ├─5347 /var/ossec/bin/wazuh-db
             ├─5371 /var/ossec/bin/wazuh-execd
             ├─5385 /var/ossec/bin/wazuh-analysisd
             ├─5402 /var/ossec/bin/wazuh-syscheckd
             ├─5470 /var/ossec/bin/wazuh-remoted
             ├─5502 /var/ossec/bin/wazuh-logcollector
             ├─5521 /var/ossec/bin/wazuh-monitord
             └─5533 /var/ossec/bin/wazuh-modulesd

июл 22 19:52:55 Wazuh-Server env[5211]: Started wazuh-db...
июл 22 19:52:56 Wazuh-Server env[5211]: Started wazuh-execd...
июл 22 19:52:57 Wazuh-Server env[5211]: Started wazuh-analysisd...
июл 22 19:52:58 Wazuh-Server env[5211]: Started wazuh-syscheckd...
июл 22 19:52:59 Wazuh-Server env[5211]: Started wazuh-remoted...
июл 22 19:53:00 Wazuh-Server env[5211]: Started wazuh-logcollector...
июл 22 19:53:01 Wazuh-Server env[5211]: Started wazuh-monitord...
июл 22 19:53:02 Wazuh-Server env[5211]: Started wazuh-modulesd...
июл 22 19:53:04 Wazuh-Server env[5211]: Completed.
июл 22 19:53:04 Wazuh-Server systemd[1]: Started Wazuh manager.
lines 10-33/33 (END)

admin@Wazuh-Server:~$ sudo systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-07-22 20:05:06 MSK; 10min ago
       Docs: https://documentation.wazuh.com
   Main PID: 8131 (java)
      Tasks: 97 (limit: 14305)
     Memory: 1.4G
        CPU: 3min 27.314s
     CGroup: /system.slice/wazuh-indexer.service
             └─8131 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 ->
июл 22 20:04:41 Wazuh-Server systemd[1]: Starting Wazuh-indexer...
июл 22 20:04:44 Wazuh-Server systemd-entrypoint[8131]: WARNING: A terminally deprecated method in java.lang.Syste>июл 22 20:04:44 Wazuh-Server systemd-entrypoint[8131]: WARNING: System::setSecurityManager has been called by org>июл 22 20:04:44 Wazuh-Server systemd-entrypoint[8131]: WARNING: Please consider reporting this to the maintainers>июл 22 20:04:44 Wazuh-Server systemd-entrypoint[8131]: WARNING: System::setSecurityManager will be removed in a f>июл 22 20:04:45 Wazuh-Server systemd-entrypoint[8131]: WARNING: A terminally deprecated method in java.lang.Syste>июл 22 20:04:45 Wazuh-Server systemd-entrypoint[8131]: WARNING: System::setSecurityManager has been called by org>июл 22 20:04:45 Wazuh-Server systemd-entrypoint[8131]: WARNING: Please consider reporting this to the maintainers>июл 22 20:04:45 Wazuh-Server systemd-entrypoint[8131]: WARNING: System::setSecurityManager will be removed in a f>июл 22 20:05:06 Wazuh-Server systemd[1]: Started Wazuh-indexer.
lines 1-21/21 (END)

What can I do or how can I fix this problem?

[Gastón Palomeque]

Hello ShtrudelMan,

The problem is caused by a race condition. An observed case is when trying to write a base and the `commit_old` process, which runs every one second, closes the data base.

This issue was detected on v4.7.3 and fixed in v4.7.5 (PR). If you would like to know more about it, here is the full investigation our team carried out: https://github.com/wazuh/wazuh/issues/22847.

To solve it, you would have to upgrade your Wazuh manager to v4.7.5 or above (v4.8.0 contains breaking changes so staying in v4.7.x is probably the best option).

Regards,

Gastón Palomeque

[ShtrudelMan]

Hello, Gaston Palomeque!
Colleagues!
I have upgraded Wazuh SIEM to version 4.7.5-1 according to the instructions on the Wazuh Documentation website.
I attach a file with the current logs from the system.

понедельник, 22 июля 2024 г. в 20:40:44 UTC+3, Gastón Palomeque:

[Gastón Palomeque]

Hello ShtrudelMan,

The logs show another issue that is related to the previous one, which is when the database is abruptly closed, it ends up being empty.

This scenario can occur in an environment where the service takes longer than a minute to shut down, as it forcibly terminates wazuh-db with SIGKILL.

Unfortunately, this issue was fixed in v4.8.0, so you would need to upgrade to that version to fix it (v4.8.1 may be a better choice), please beware that v4.8.0 introduces breaking changes to the vulnerability detection module configuration (documentation).

Regards,

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/wHxXV8dg_ZM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/dfa3b228-0d71-4491-a86c-5a546e67faa1n%40googlegroups.com.

--

Gastón Palomeque Software Engineer

[ShtrudelMan]

Good afternoon😀
Thank you for your time!😀
Good afternoon.
I'm having difficulty understanding a point you made in the previous message. You stated that the bugs I mentioned in the initial post had been fixed in version 4.7.5.
I upgraded the specified central components on the node and received new error messages in the "ossec.log" file.
I then sent you a new message with the attached file from "ossec.log," after which you informed me that upgrading to version 4.8 is necessary to resolve this error.
Please confirm whether the errors in the first and second cases are different from those in "sqlite3."

Or are you simply requesting that I upgrade to the latest version of Wazuh, despite the fact that the same GitHub chat you referenced above indicates that this issue is still present in version 4.8?

четверг, 25 июля 2024 г. в 17:25:57 UTC+3, Gastón Palomeque:

[Gastón Palomeque]

Hello ShtrudelMan,

The errors are related because they are both caused by a race condition in wazuh-db, but they are two separate issues with two different solutions. The first error was fixed in version 4.7.5 and the second one in 4.8.0.

So, in order to fix the issue from the last logs you shared, the Wazuh instance must be upgraded to the latest version (4.8.0 or 4.8.1).

I hope this answers your questions.

Regards,

[ShtrudelMan]

Good afternoon, Gaston.
I'm really grateful to you for helping me to understand the issue.
If it isn't an inconvenience, could you possibly elaborate on how critical these issues are if I remain on version 4.7.5? I'm not quite prepared to transition to version 4.8(x) just yet.
Could you kindly elaborate on how these issues might affect the system's operation?

понедельник, 29 июля 2024 г. в 16:26:41 UTC+3, Gastón Palomeque:

[Gastón Palomeque]

Good afternoon ShtrudelMan,

It is a fault tolerance issue in wazuh-db, which may result in empty database files if abruptly closed. This scenario can occur in an environment where the service takes longer than a minute to shut down, as it forcibly terminates wazuh-db with SIGKILL.

What the fix introduced in 4.8.0 does is to use a temporary file until all modifications are performed and then rename that temporary file to the database. That way, it ensures that if the process is interrupted, the original database is not left empty or corrupted (only the temporary file would be).

The system will operate under normal conditions as long as wazuh-db is not forcibly terminated, but a single termination can leave your instance unavailable.

I would suggest keeping a backup of the database files located under `/var/ossec/queue` and trying to upgrade to v4.8.x as soon as possible to avoid such an issue.

Regards,

--

You received this message because you are subscribed to a topic in the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/wHxXV8dg_ZM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/b8ea7a5d-2c25-4987-a06f-2cdc31e80dd2n%40googlegroups.com.

[ShtrudelMan]

Good afternoon,
I am planning to upgrade from Wazuh v4.7.5 to Wazuh v4.8 as part of a test environment.
Please advise if there are any known issues with upgrading all components of the system.
I am currently using Debian 11 and the Wazuh components are installed on the same server.
There are currently 65 agents connected to Wazuh.
There are several groups of agents. Different "ossec.conf" settings have been created for these groups!

I appreciate your attention to this issue.

вторник, 30 июля 2024 г. в 17:29:04 UTC+3, Gastón Palomeque:

[ShtrudelMan]

Good day!
Yesterday I upgraded Wazuh Central Components and Wazuh Agents to version 4.8.2.
After the upgrade I was happy with the stability of the system.
But today on April 4, 2024 I started to receive again notifications with errors about Wazuh-DB module.
The errors have the following form:

• admin@Wazuh-Server:~$ sudo tail -f /var/ossec/logs/ossec.log
• 2024/09/04 13:27:53 wazuh-db: ERROR: DB(069) sqlite3_prepare_v2() stmt(87): no such table: sync_info
• 2024/09/04 13:27:53 wazuh-db: ERROR: DB(069) sqlite3_prepare_v2() stmt(89): no such table: fim_entry
• 2024/09/04 13:27:53 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
• 2024/09/04 13:27:57 wazuh-db: ERROR: DB(067) sqlite3_prepare_v2() stmt(87): no such table: sync_info
• 2024/09/04 13:27:57 wazuh-db: ERROR: DB(067) sqlite3_prepare_v2() stmt(89): no such table: fim_entry
• 2024/09/04 13:27:57 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
• 2024/09/04 13:27:58 wazuh-db: ERROR: DB(070) sqlite3_prepare_v2() stmt(87): no such table: sync_info
• 2024/09/04 13:27:58 wazuh-db: ERROR: DB(070) sqlite3_prepare_v2() stmt(85): no such table: sync_info
• 2024/09/04 13:29:48 wazuh-modulesd:syscollector: INFO: Starting evaluation.
• 2024/09/04 13:29:56 wazuh-modulesd:syscollector: INFO: Evaluation finished.
• 2024/09/04 13:32:53 wazuh-db: ERROR: DB(069) sqlite3_prepare_v2() stmt(87): no such table: sync_info
• 2024/09/04 13:32:53 wazuh-db: ERROR: DB(069) sqlite3_prepare_v2() stmt(89): no such table: fim_entry
• 2024/09/04 13:32:53 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
• 2024/09/04 13:32:57 wazuh-db: ERROR: DB(067) sqlite3_prepare_v2() stmt(87): no such table: sync_info
• 2024/09/04 13:32:57 wazuh-db: ERROR: DB(067) sqlite3_prepare_v2() stmt(89): no such table: fim_entry
• 2024/09/04 13:32:57 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
• 2024/09/04 13:32:58 wazuh-db: ERROR: DB(070) sqlite3_prepare_v2() stmt(87): no such table: sync_info
• 2024/09/04 13:32:58 wazuh-db: ERROR: DB(070) sqlite3_prepare_v2() stmt(85): no such table: sync_info
• 2024/09/04 13:37:53 wazuh-db: ERROR: DB(069) sqlite3_prepare_v2() stmt(87): no such table: sync_info
• 2024/09/04 13:37:53 wazuh-db: ERROR: DB(069) sqlite3_prepare_v2() stmt(89): no such table: fim_entry
• 2024/09/04 13:37:53 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
• 2024/09/04 13:37:57 wazuh-db: ERROR: DB(067) sqlite3_prepare_v2() stmt(87): no such table: sync_info
• 2024/09/04 13:37:57 wazuh-db: ERROR: DB(067) sqlite3_prepare_v2() stmt(89): no such table: fim_entry
• 2024/09/04 13:37:57 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
• 2024/09/04 13:37:58 wazuh-db: ERROR: DB(070) sqlite3_prepare_v2() stmt(87): no such table: sync_info
• 2024/09/04 13:37:58 wazuh-db: ERROR: DB(070) sqlite3_prepare_v2() stmt(85): no such table: sync_info

The only thing I encountered was that one of my Wazuh Agents from one node started sending a lot more events from the system. This caused data flooding. I'm trying to sort it out now

среда, 7 августа 2024 г. в 16:23:04 UTC+3, ShtrudelMan: