Wazuh 4.9 Internal Server Error and Authentication Problems
143 views
Skip to first unread message
Mohamed Maslouh
Feb 17, 2025, 6:34:02 AM2/17/25
to Wazuh | Mailing List
Hello,
For the past four days, I have been receiving an error message about too many requests. Occasionally, I also encountered a "wrong password" issue during authentication. However, today Wazuh stopped working entirely, and I am now getting an "Internal Server Error."
OS: AlmaLinux 9.4
Wazuh Version: 4.9.0
root@srv-wazuh wazuh]# journalctl -u wazuh-indexer --no-pager | tail -n 50
Feb 17 10:28:04 srv-wazuh systemd[1]: Starting wazuh-indexer...
Feb 17 10:28:09 srv-wazuh systemd-entrypoint[1158]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 17 10:28:09 srv-wazuh systemd-entrypoint[1158]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Feb 17 10:28:09 srv-wazuh systemd-entrypoint[1158]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Feb 17 10:28:09 srv-wazuh systemd-entrypoint[1158]: WARNING: System::setSecurityManager will be removed in a future release
Feb 17 10:28:10 srv-wazuh systemd-entrypoint[1158]:
Feb 17, 2025 10:28:10 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Feb 17 10:28:10 srv-wazuh systemd-entrypoint[1158]: WARNING: COMPAT locale provider will be removed in a future release
Feb 17 10:28:12 srv-wazuh systemd-entrypoint[1158]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 17 10:28:12 srv-wazuh systemd-entrypoint[1158]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Feb 17 10:28:12 srv-wazuh systemd-entrypoint[1158]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Feb 17 10:28:12 srv-wazuh systemd-entrypoint[1158]: WARNING: System::setSecurityManager will be removed in a future release
Feb 17 10:28:45 srv-wazuh systemd[1]: Started wazuh-indexer.
Feb 17 12:19:27 srv-wazuh systemd[1]: Stopping wazuh-indexer...
Feb 17 12:19:30 srv-wazuh systemd[1]: wazuh-indexer.service: Deactivated successfully.
Feb 17 12:19:30 srv-wazuh systemd[1]: Stopped wazuh-indexer.
Feb 17 12:19:30 srv-wazuh systemd[1]: wazuh-indexer.service: Consumed 29min 6.397s CPU time.
Feb 17 12:19:30 srv-wazuh systemd[1]: Starting wazuh-indexer...
Feb 17 12:19:32 srv-wazuh systemd-entrypoint[5305]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 17 12:19:32 srv-wazuh systemd-entrypoint[5305]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Feb 17 12:19:32 srv-wazuh systemd-entrypoint[5305]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Feb 17 12:19:32 srv-wazuh systemd-entrypoint[5305]: WARNING: System::setSecurityManager will be removed in a future release
Feb 17 12:19:32 srv-wazuh systemd-entrypoint[5305]: Feb 17, 2025 12:19:32 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Feb 17 12:19:32 srv-wazuh systemd-entrypoint[5305]: WARNING: COMPAT locale provider will be removed in a future release
Feb 17 12:19:33 srv-wazuh systemd-entrypoint[5305]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 17 12:19:33 srv-wazuh systemd-entrypoint[5305]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Feb 17 12:19:33 srv-wazuh systemd-entrypoint[5305]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Feb 17 12:19:33 srv-wazuh systemd-entrypoint[5305]: WARNING: System::setSecurityManager will be removed in a future release
Feb 17 12:19:44 srv-wazuh systemd[1]: Started wazuh-indexer.
root@srv-wazuh wazuh]#
grep memo /var/log/wazuh-indexer/wazuh-cluster.log
96071792], allocations [41], duration [16]
[2025-02-17T12:15:17,299][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1025037952], after [1022681440], allocations [42], duration [17]
[2025-02-17T12:15:24,774][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1054995768], after [1054743112], allocations [14], duration [9]
[2025-02-17T12:15:36,427][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1022467280], after [974015504], allocations [1], duration [11]
[2025-02-17T12:15:46,307][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1021773608], after [975412816], allocations [4], duration [9]
[2025-02-17T12:15:56,332][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did not bring memory usage down, before [1020966264], after [1022027048], allocations [44], duration [26]
[2025-02-17T12:16:05,241][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did not bring memory usage down, before [1053484320], after [1055245256], allocations [14], duration [11]
[2025-02-17T12:16:32,343][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1020189104], after [1016852520], allocations [45], duration [18]
[2025-02-17T12:16:47,349][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1025346600], after [1020633784], allocations [39], duration [17]
[2025-02-17T12:16:57,198][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did not bring memory usage down, before [1064673976], after [1064768976], allocations [1], duration [12]
[2025-02-17T12:17:06,277][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did not bring memory usage down, before [1052961168], after [1054442216], allocations [12], duration [10]
[2025-02-17T12:17:42,384][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1021780848], after [1019277920], allocations [46], duration [16]
[2025-02-17T12:17:49,003][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1048638048], after [1048208248], allocations [21], duration [10]
[2025-02-17T12:18:07,403][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1020846912], after [1013904704], allocations [42], duration [18]
[2025-02-17T12:18:31,416][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1025613464], after [1021548928], allocations [39], duration [16]
[2025-02-17T12:18:43,413][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1021518856], after [975470264], allocations [1], duration [6]
[2025-02-17T12:19:13,442][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1024879400], after [1019376040], allocations [43], duration [18]
[2025-02-17T12:19:19,271][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did not bring memory usage down, before [1051881896], after [1052141144], allocations [17], duration [10]
[2025-02-17T12:19:40,174][INFO ][o.o.a.b.ADCircuitBreakerService] [node-1] Registered memory breaker.
[2025-02-17T12:19:40,615][INFO ][o.o.m.b.MLCircuitBreakerService] [node-1] Registered ML memory breaker.
[2025-02-17T12:19:40,616][INFO ][o.o.m.b.MLCircuitBreakerService] [node-1] Registered ML native memory breaker.
[2025-02-17T12:21:51,547][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1021897304], after [1019436840], allocations [46], duration [21]
[2025-02-17T12:21:57,833][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1062428456], after [1062401608], allocations [8], duration [14]
[2025-02-17T12:23:20,600][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1019929520], after [974193440], allocations [48], duration [13]
[2025-02-17T12:15:17,299][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1025037952], after [1022681440], allocations [42], duration [17]
[2025-02-17T12:15:24,774][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1054995768], after [1054743112], allocations [14], duration [9]
[2025-02-17T12:15:36,427][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1022467280], after [974015504], allocations [1], duration [11]
[2025-02-17T12:15:46,307][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1021773608], after [975412816], allocations [4], duration [9]
[2025-02-17T12:15:56,332][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did not bring memory usage down, before [1020966264], after [1022027048], allocations [44], duration [26]
[2025-02-17T12:16:05,241][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did not bring memory usage down, before [1053484320], after [1055245256], allocations [14], duration [11]
[2025-02-17T12:16:32,343][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1020189104], after [1016852520], allocations [45], duration [18]
[2025-02-17T12:16:47,349][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1025346600], after [1020633784], allocations [39], duration [17]
[2025-02-17T12:16:57,198][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did not bring memory usage down, before [1064673976], after [1064768976], allocations [1], duration [12]
[2025-02-17T12:17:06,277][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did not bring memory usage down, before [1052961168], after [1054442216], allocations [12], duration [10]
[2025-02-17T12:17:42,384][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1021780848], after [1019277920], allocations [46], duration [16]
[2025-02-17T12:17:49,003][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1048638048], after [1048208248], allocations [21], duration [10]
[2025-02-17T12:18:07,403][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1020846912], after [1013904704], allocations [42], duration [18]
[2025-02-17T12:18:31,416][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1025613464], after [1021548928], allocations [39], duration [16]
[2025-02-17T12:18:43,413][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1021518856], after [975470264], allocations [1], duration [6]
[2025-02-17T12:19:13,442][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1024879400], after [1019376040], allocations [43], duration [18]
[2025-02-17T12:19:19,271][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did not bring memory usage down, before [1051881896], after [1052141144], allocations [17], duration [10]
[2025-02-17T12:19:40,174][INFO ][o.o.a.b.ADCircuitBreakerService] [node-1] Registered memory breaker.
[2025-02-17T12:19:40,615][INFO ][o.o.m.b.MLCircuitBreakerService] [node-1] Registered ML memory breaker.
[2025-02-17T12:19:40,616][INFO ][o.o.m.b.MLCircuitBreakerService] [node-1] Registered ML native memory breaker.
[2025-02-17T12:21:51,547][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1021897304], after [1019436840], allocations [46], duration [21]
[2025-02-17T12:21:57,833][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1062428456], after [1062401608], allocations [8], duration [14]
[2025-02-17T12:23:20,600][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [1019929520], after [974193440], allocations [48], duration [13]
Mohamed Maslouh
Feb 17, 2025, 6:55:52 AM2/17/25
to Wazuh | Mailing List
Hello,
I fixed it with the help of ChatGPT :D
i had to increase The JVM heap -Xms and -Xmx values
[root@srv-wazuh wazuh]# cp /etc/wazuh-indexer/jvm.options /etc/wazuh-indexer/jvm.options.bak
[root@srv-wazuh wazuh]# nono /etc/wazuh-indexer/jvm.options
#-Xms1024m
#-Xmx1024m
-Xms4g
-Xmx4g
#-Xmx1024m
-Xms4g
-Xmx4g
hasitha.u...@wazuh.com
Feb 17, 2025, 6:58:05 AM2/17/25
to Wazuh | Mailing List
Hi
Mohamed,
From the logs it seems that it was a memory-related problem in Wazuh Indexer.
Try to increase RAM and configure Heap Memory for Wazuh Indexer: Modify the heap size settings in the JVM options file (/etc/wazuh-indexer/JVM.options). It's recommended to set both -Xms (minimum) and -Xmx (maximum) heap sizes to the same value to avoid performance degradation due to dynamic resizing.
Set Xms and Xmx to no more than 50% of your total memory. Indexer requires memory for purposes other than the JVM heap. Check the current RAM allocation by running this command free -h
For example, if you have allocated 8 GB you need to increase the heap allocation from ~2GB to 4GB:
# -Xms1931m
# -Xmx1931m
-Xms4g
-Xmx4g
Enable Memory Locking
To prevent excessive memory swapping that can worsen performance issues, enable memory locking by adding the following line to /etc/wazuh-indexer/opensearch.yml:
bootstrap.memory_lock: true
For more details, check the Wazuh documentation on indexer tuning.
After making the changes, reload the system daemon and restart the required services:
systemctl daemon-reload
systemctl restart wazuh-indexer
systemctl restart wazuh-dashboard
You can see this documentation for more information about Wazuh indexer tuning https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html#wazuh-indexer-tuning
Let me know if you need any further assistance!
Regards,
Hasitha Upekshitha
From the logs it seems that it was a memory-related problem in Wazuh Indexer.
Try to increase RAM and configure Heap Memory for Wazuh Indexer: Modify the heap size settings in the JVM options file (/etc/wazuh-indexer/JVM.options). It's recommended to set both -Xms (minimum) and -Xmx (maximum) heap sizes to the same value to avoid performance degradation due to dynamic resizing.
Set Xms and Xmx to no more than 50% of your total memory. Indexer requires memory for purposes other than the JVM heap. Check the current RAM allocation by running this command free -h
For example, if you have allocated 8 GB you need to increase the heap allocation from ~2GB to 4GB:
# -Xms1931m
# -Xmx1931m
-Xms4g
-Xmx4g
Enable Memory Locking
To prevent excessive memory swapping that can worsen performance issues, enable memory locking by adding the following line to /etc/wazuh-indexer/opensearch.yml:
bootstrap.memory_lock: true
For more details, check the Wazuh documentation on indexer tuning.
After making the changes, reload the system daemon and restart the required services:
systemctl daemon-reload
systemctl restart wazuh-indexer
systemctl restart wazuh-dashboard
You can see this documentation for more information about Wazuh indexer tuning https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html#wazuh-indexer-tuning
Let me know if you need any further assistance!
Regards,
Hasitha Upekshitha
hasitha.u...@wazuh.com
Feb 17, 2025, 6:59:00 AM2/17/25
to Wazuh | Mailing List
Hi
Mohamed,
I am glad that your issue has been resolved. Let me know if you need further assistance.
Regards,
Hasitha Upekshitha
I am glad that your issue has been resolved. Let me know if you need further assistance.
Regards,
Hasitha Upekshitha
Reply all
Reply to author
Forward
0 new messages