Issue with Wazuh Login and Dashboard Access

222 views
Skip to first unread message

Gowtham Murugesan

unread,
Nov 7, 2024, 6:03:54 AM11/7/24
to Wazuh | Mailing List

Hi All,

We are encountering an issue with Wazuh, where the login page displays an error stating that the username or password is invalid, despite entering the correct credentials. However, after restarting the Wazuh indexer, we are able to log in successfully with the same credentials. Could you confirm whether restarting the indexer is an appropriate workaround for this issue, or if a more permanent solution is needed?

The issue seems to have started around 5:29 AM daily, when Wazuh stopped receiving logs and the Wazuh dashboard became unresponsive. 

Below we have mentioned the disk storage of the wazuh server,

df -h

Filesystem      Size  Used Avail Use% Mounted on

devtmpfs        7.6G     0  7.6G   0% /dev

tmpfs           7.7G   10M  7.7G   1% /dev/shm

tmpfs           7.7G  904K  7.7G   1% /run

tmpfs           7.7G     0  7.7G   0% /sys/fs/cgroup

/dev/nvme0n1p1  710G  530G  181G  75% /

tmpfs           1.6G     0  1.6G   0% /run/user/1000

tmpfs           1.6G     0  1.6G   0% /run/user/1005

tmpfs           1.6G     0  1.6G   0% /run/user/1006



Below we have mentioned the alert from the wazuh manager,

​Wazuh Notification.

2024 Nov 07 02:28:01

Received From: ->/var/log/messages

Rule: 5108 fired (level 12) -> "System running out of memory. Availability of the system is in risk."

Portion of the log(s):

Nov  7 02:26:48  kernel: Out of memory: Kill process 28393 (java) score 855 or sacrifice child


When we checked in the server,


dmesg | grep -i memory


[    0.000000] Base memory trampoline at [ffff97dcc0099000] 99000 size 24576

[    0.000000] Reserving 161MB of memory at 624MB for crashkernel (System RAM: 16083MB)

[    0.000000] Early memory node ranges

[    0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]

[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]

[    0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]

[    0.000000] PM: Registered nosave memory: [mem 0xbffea000-0xbfffffff]

[    0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xdfffffff]

[    0.000000] PM: Registered nosave memory: [mem 0xe0000000-0xe03fffff]

[    0.000000] PM: Registered nosave memory: [mem 0xe0400000-0xfffbffff]

[    0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff]

[    0.000000] Memory: 4957732k/17518592k available (7988k kernel code, 1049056k absent, 563692k reserved, 5756k data, 2176k init)

[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups

[    1.669383] Initializing cgroup subsys memory

[    2.182151] x86/mm: Memory block size: 128MB

[    3.344296] Freeing initrd memory: 54152k freed

[    3.639163] Non-volatile memory driver v1.3

[    3.649361] crash memory driver: version 1.1

[    3.920748] Freeing unused kernel memory: 2176k freed

[    3.932741] Freeing unused kernel memory: 192k freed

[    3.939081] Freeing unused kernel memory: 524k freed

[696820.646831]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[696821.418546] Out of memory: Kill process 9200 (java) score 855 or sacrifice child

[696821.572192]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[696822.455198] Out of memory: Kill process 14732 (opensearch[node) score 855 or sacrifice child

[5030730.238331]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[5030731.030888] Out of memory: Kill process 27194 (java) score 862 or sacrifice child

[8081987.086267]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[8081987.883686] Out of memory: Kill process 20082 (java) score 863 or sacrifice child

[12301967.812453]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[12301968.879640] Out of memory: Kill process 26591 (java) score 861 or sacrifice child

[12447661.096623]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[12447662.446401] Out of memory: Kill process 28012 (java) score 857 or sacrifice child

[12447662.533622]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[12447663.959253] Out of memory: Kill process 28189 (G1 Service) score 857 or sacrifice child

[12532670.705526]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[12532671.974071] Out of memory: Kill process 18168 (java) score 854 or sacrifice child

[12705131.398436]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[12705132.659528] Out of memory: Kill process 3437 (java) score 853 or sacrifice child

[12732255.406694]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[12732256.698956] Out of memory: Kill process 23342 (java) score 854 or sacrifice child

[12853029.056416]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[12853030.283234] Out of memory: Kill process 26090 (java) score 855 or sacrifice child

[12909619.387101]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[12909620.621051] Out of memory: Kill process 3645 (java) score 853 or sacrifice child

[12909620.730593]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[12909622.007609] Out of memory: Kill process 3866 (G1 Conc#0) score 853 or sacrifice child

[13048866.547948]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[13048867.774849] Out of memory: Kill process 31436 (java) score 855 or sacrifice child

[13307862.701595]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[13307864.031638] Out of memory: Kill process 19126 (java) score 857 or sacrifice child

[13349866.705292]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[13349867.898324] Out of memory: Kill process 1451 (java) score 856 or sacrifice child

[13483662.777413]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[13483664.003764] Out of memory: Kill process 20259 (java) score 856 or sacrifice child

[13946108.212126]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[13946109.438313] Out of memory: Kill process 16281 (java) score 854 or sacrifice child

[14020729.344993]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[14020730.559406] Out of memory: Kill process 9259 (java) score 854 or sacrifice child

[14153514.403766]  [<ffffffff997cdaca>] out_of_memory+0x31a/0x500

[14153515.575141] Out of memory: Kill process 28393 (java) score 855 or sacrifice child



free -m

              total        used        free      shared  buff/cache   available

Mem:    15588       14860         168           9         560         399



Note: 

  • To manage storage, we cleared the Wazuh indices for the year 2022 and 2023 logs. 

  • In the global configuration, we have changed yes to no in the <logall>no</logall>.

Here's some additional information that might be helpful:

  • Wazuh version (manager and agent): Manager (4.7.3) and Agent (4.7.1).

Please let us know if you require any further details or have any suggestions to resolve this issue. We look forward to your prompt response.

We would greatly appreciate your assistance in troubleshooting this issue.




hasitha.u...@wazuh.com

unread,
Nov 7, 2024, 6:24:24 AM11/7/24
to Wazuh | Mailing List
Hi Gowtham,

I believe issue might be with RAM, You Ram is almost full. Free memory is 168M.

I suggest you to increase the RAM and configure the heap memory.
Here are some key points to keep in mind to configure heap memory:
Use no more than 50% of your available RAM.
Don’t set the heap size over 32 GB.
Start by checking your memory with:
free -h

Then, update the heap size in the /etc/wazuh-indexer/jvm.options file. For example, if your server has 12 GB of RAM, set the heap size to 6 GB as shown below:
-Xms6g
-Xmx6g

After making these changes, restart the Wazuh indexer for them to take effect:
systemctl restart wazuh-indexer

You can refer to this link for more details:
Ref: https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html#memory-locking

Let me know if the issue resolve.

Regards,
Hasitha Upekshitha
Reply all
Reply to author
Forward
0 new messages