Dashbord not showing new events

277 views
Skip to first unread message

Marcin N.

unread,
Oct 5, 2023, 1:32:43 AM10/5/23
to Wazuh | Mailing List
Hello, have problem. Yesterday wazuh stoped to showing new security events in dashboard so i cleared logs from this day (dev tools  DELETE /wazuh-alerts-4.x-2023.09.03  ) and restarted wazuh. Everything was working fine for some hours but than aroung 2:00 AM wazuh stopped to showing new events again.

I have new alerts in /var/ossec/logs/alerts/alerts.json

Last good log:
{"timestamp":"2023-10-05T01:59:36.640+0200","rule":{"level":3,"description":"Windows logon success.","id":"60106","mitre":{"id":["T1078"],"tactic":["Defense Evasion","Persistence","Privilege Escalation","Initial Access"],"technique":["Valid Accounts"]},"firedtimes":117,"mail":false,"groups":["windows","windows_security","authentication_success"],"gdpr":["IV_32.2"],"gpg13":["7.1","7.2"],"hipaa":["164.312.b"],"nist_800_53":["AC.7","AU.14"],"pci_dss":["10.2.5"],"tsc":["CC6.8","CC7.2","CC7.3"]},"agent":{"id":"011","name":"LAP998","ip":"192.168.99.99"},"manager":{"name":"wazuh.local"},"id":"1696463976.4108307","decoder":{"name":"windows_eventchannel"},"data":{"win":{"system":{"providerName":"Microsoft-Windows-Security-Auditing","providerGuid":"{54849625-5478-4994-a5ba-3e3b0328c30d}","eventID":"4624","version":"2","level":"0","task":"12544","opcode":"0","keywords":"0x8020000000000000","systemTime":"2023-10-04T23:59:35.6351287Z","eventRecordID":"18548349","processID":"800","threadID":"21624","channel":"Security","computer":"LAP998.local","severityValue":"AUDIT_SUCCESS","message":"\"Logowanie do konta zakoÅ<84>czyÅ<82>o siÄ<99> pomyÅ<9b>lnie.\r\n\r\nPodmiot:\r\n\tIdentyfikator zabezpieczeÅ<84>:\t\tS-1-5-18\r\n\tNazwa konta:\t\tLAP998$\r\n\tDomena konta:\t\tlocal\r\n\tIdentyfikator logowania:\t\t0x3E7\r\n\r\nInformacje o logowaniu:\r\n\tTyp logowania:\t\t5\r\n\tOgraniczony tryb administratora:\t-\r\n\tKonto wirtualne:\t\tNie\r\n\tToken z podniesionymi uprawnieniami:\t\tTak\r\n\r\nPoziom personifikacji:\t\tPersonifikacja\r\n\r\nNowe logowanie:\r\n\tIdentyfikator zabezpieczeÅ<84>:\t\tS-1-5-18\r\n\tNazwa konta:\t\tSYSTEM\r\n\tDomena konta:\t\tZARZÄ<84>DZANIE NT\r\n\tIdentyfikator logowania:\t\t0x3E7\r\n\tPoÅ<82>Ä<85>czony identyfikator logowania:\t\t0x0\r\n\tNazwa konta sieciowego:\t-\r\n\tDomena konta sieciowego:\t-\r\n\tIdentyfikator GUID logowania:\t\t{00000000-0000-0000-0000-000000000000}\r\n\r\nInformacje o procesie:\r\n\tIdentyfikator procesu:\t\t0x318\r\n\tNazwa procesu:\t\tC:\\Windows\\System32\\services.exe\r\n\r\nInformacje o sieci:\r\n\tNazwa stacji roboczej:\t-\r\n\tŹródÅ<82>owy adres sieciowy:\t-\r\n\tPort źródÅ<82>owy:\t\t-\r\n\r\nSzczegóÅ<82>owe informacje o logowaniu:\r\n\tProces logowania:\t\tAdvapi  \r\n\tPakiet uwierzytelniania:\tNegotiate\r\n\tUsÅ<82>ugi przejÅ<9b>ciowe:\t-\r\n\tNazwa pakietu (tylko protokóÅ<82> NTLM):\t-\r\n\tDÅ<82>ugoÅ<9b>Ä<87> klucza:\t\t0\r\n\r\nTo zdarzenie jest generowane w przypadku utworzenia sesji logowania. Jest ono generowane na komputerze, do którego uzyskano dostÄ<99>p.\r\n\r\nPola podmiotu wskazujÄ<85> konto w systemie lokalnym, z poziomu którego zażÄ<85>dano logowania. NajczÄ<99>Å<9b>ciej jest to usÅ<82>uga, taka jak usÅ<82>uga serwera, lub proces lokalny, taki jak Winlogon.exe lub Services.exe.\r\n\r\nPole typu logowania wskazuje rodzaj żÄ<85>danego logowania. NajczÄ<99>Å<9b>ciej używane typy to 2 (interaktywne) i 3 (sieciowe).\r\n\r\nPola nowego logowania wskazujÄ<85> konto, dla którego utworzono nowe dane logowania, czyli konto, do którego nastÄ<85>piÅ<82>o zalogowanie.\r\n\r\nPola sieci wskazujÄ<85> lokalizacjÄ<99>, z której pochodziÅ<82>o zdalne żÄ<85>danie logowania. Nazwa stacji roboczej nie jest zawsze dostÄ<99>pna i w niektórych przypadkach jej pole może pozostaÄ<87> puste.\r\n\r\nPoziom personifikacji wskazuje dostÄ<99>pny zakres personifikacji w procesie sesji logowania.\r\n\r\nPola informacji o uwierzytelnianiu zawierajÄ<85> szczegóÅ<82>owe informacje o danym żÄ<85>daniu logowania.\r\n\t- Identyfikator GUID logowania to unikatowy identyfikator, który może byÄ<87> używany do skorelowania tego zdarzenia ze zdarzeniem centrum dystrybucji kluczy.\r\n\t- UsÅ<82>ugi przejÅ<9b>ciowe wskazujÄ<85>, które usÅ<82>ugi poÅ<9b>rednie uczestniczyÅ<82>y w tym żÄ<85>daniu logowania.\r\n\t- Nazwa pakietu wskazuje, który protokóÅ<82> podrzÄ<99>dny spoÅ<9b>ród protokoÅ<82>ów NTLM zostaÅ<82> użyty.\r\n\t- DÅ<82>ugoÅ<9b>Ä<87> klucza wskazuje dÅ<82>ugoÅ<9b>Ä<87> wygenerowanego klucza sesji. JeÅ<9b>li nie zażÄ<85>dano klucza sesji, bÄ<99>dzie to wartoÅ<9b>Ä<87> 0.\""},"eventdata":{"subjectUserSid":"S-1-5-18","subjectUserName":"LAP998$","subjectDomainName":"local","subjectLogonId":"0x3e7","targetUserSid":"S-1-5-18","targetUserName":"SYSTEM","targetDomainName":"ZARZÄ<84>DZANIE NT","targetLogonId":"0x3e7","logonType":"5","logonProcessName":"Advapi","authenticationPackageName":"Negotiate","logonGuid":"{00000000-0000-0000-0000-000000000000}","keyLength":"0","processId":"0x318","processName":"C:\\\\Windows\\\\System32\\\\services.exe","impersonationLevel":"%%1833","virtualAccount":"%%1843","targetLinkedLogonId":"0x0","elevatedToken":"%%1842"}}},"location":"EventChannel"}

Not showed:
{"timestamp":"2023-10-05T02:02:10.465+0200","rule":{"level":3,"description":"Windows logon success.","id":"60106","mitre":{"id":["T1078"],"tactic":["Defense Evasion","Persistence","Privilege Escalation","Initial Access"],"technique":["Valid Accounts"]},"firedtimes":1,"mail":false,"groups":["windows","windows_security","authentication_success"],"gdpr":["IV_32.2"],"gpg13":["7.1","7.2"],"hipaa":["164.312.b"],"nist_800_53":["AC.7","AU.14"],"pci_dss":["10.2.5"],"tsc":["CC6.8","CC7.2","CC7.3"]},"agent":{"id":"012","name":"LAP999","ip":"192.168.99.99"},"manager":{"name":"wazuh.local"},"id":"1696464130.4116427","decoder":{"name":"windows_eventchannel"},"data":{"win":{"system":{"providerName":"Microsoft-Windows-Security-Auditing","providerGuid":"{54849625-5478-4994-a5ba-3e3b0328c30d}","eventID":"4624","version":"2","level":"0","task":"12544","opcode":"0","keywords":"0x8020000000000000","systemTime":"2023-10-05T00:02:09.4533698Z","eventRecordID":"15664285","processID":"128","threadID":"1900","channel":"Security","computer":"LAP999.local","severityValue":"AUDIT_SUCCESS","message":"\"Logowanie do konta zakoÅ<84>czyÅ<82>o siÄ<99> pomyÅ<9b>lnie.\r\n\r\nPodmiot:\r\n\tIdentyfikator zabezpieczeÅ<84>:\t\tS-1-5-18\r\n\tNazwa konta:\t\tLAP999$\r\n\tDomena konta:\t\tlocal\r\n\tIdentyfikator logowania:\t\t0x3E7\r\n\r\nInformacje o logowaniu:\r\n\tTyp logowania:\t\t5\r\n\tOgraniczony tryb administratora:\t-\r\n\tKonto wirtualne:\t\tNie\r\n\tToken z podniesionymi uprawnieniami:\t\tTak\r\n\r\nPoziom personifikacji:\t\tPersonifikacja\r\n\r\nNowe logowanie:\r\n\tIdentyfikator zabezpieczeÅ<84>:\t\tS-1-5-18\r\n\tNazwa konta:\t\tSYSTEM\r\n\tDomena konta:\t\tZARZÄ<84>DZANIE NT\r\n\tIdentyfikator logowania:\t\t0x3E7\r\n\tPoÅ<82>Ä<85>czony identyfikator logowania:\t\t0x0\r\n\tNazwa konta sieciowego:\t-\r\n\tDomena konta sieciowego:\t-\r\n\tIdentyfikator GUID logowania:\t\t{00000000-0000-0000-0000-000000000000}\r\n\r\nInformacje o procesie:\r\n\tIdentyfikator procesu:\t\t0x3e4\r\n\tNazwa procesu:\t\tC:\\Windows\\System32\\services.exe\r\n\r\nInformacje o sieci:\r\n\tNazwa stacji roboczej:\t-\r\n\tŹródÅ<82>owy adres sieciowy:\t-\r\n\tPort źródÅ<82>owy:\t\t-\r\n\r\nSzczegóÅ<82>owe informacje o logowaniu:\r\n\tProces logowania:\t\tAdvapi  \r\n\tPakiet uwierzytelniania:\tNegotiate\r\n\tUsÅ<82>ugi przejÅ<9b>ciowe:\t-\r\n\tNazwa pakietu (tylko protokóÅ<82> NTLM):\t-\r\n\tDÅ<82>ugoÅ<9b>Ä<87> klucza:\t\t0\r\n\r\nTo zdarzenie jest generowane w przypadku utworzenia sesji logowania. Jest ono generowane na komputerze, do którego uzyskano dostÄ<99>p.\r\n\r\nPola podmiotu wskazujÄ<85> konto w systemie lokalnym, z poziomu którego zażÄ<85>dano logowania. NajczÄ<99>Å<9b>ciej jest to usÅ<82>uga, taka jak usÅ<82>uga serwera, lub proces lokalny, taki jak Winlogon.exe lub Services.exe.\r\n\r\nPole typu logowania wskazuje rodzaj żÄ<85>danego logowania. NajczÄ<99>Å<9b>ciej używane typy to 2 (interaktywne) i 3 (sieciowe).\r\n\r\nPola nowego logowania wskazujÄ<85> konto, dla którego utworzono nowe dane logowania, czyli konto, do którego nastÄ<85>piÅ<82>o zalogowanie.\r\n\r\nPola sieci wskazujÄ<85> lokalizacjÄ<99>, z której pochodziÅ<82>o zdalne żÄ<85>danie logowania. Nazwa stacji roboczej nie jest zawsze dostÄ<99>pna i w niektórych przypadkach jej pole może pozostaÄ<87> puste.\r\n\r\nPoziom personifikacji wskazuje dostÄ<99>pny zakres personifikacji w procesie sesji logowania.\r\n\r\nPola informacji o uwierzytelnianiu zawierajÄ<85> szczegóÅ<82>owe informacje o danym żÄ<85>daniu logowania.\r\n\t- Identyfikator GUID logowania to unikatowy identyfikator, który może byÄ<87> używany do skorelowania tego zdarzenia ze zdarzeniem centrum dystrybucji kluczy.\r\n\t- UsÅ<82>ugi przejÅ<9b>ciowe wskazujÄ<85>, które usÅ<82>ugi poÅ<9b>rednie uczestniczyÅ<82>y w tym żÄ<85>daniu logowania.\r\n\t- Nazwa pakietu wskazuje, który protokóÅ<82> podrzÄ<99>dny spoÅ<9b>ród protokoÅ<82>ów NTLM zostaÅ<82> użyty.\r\n\t- DÅ<82>ugoÅ<9b>Ä<87> klucza wskazuje dÅ<82>ugoÅ<9b>Ä<87> wygenerowanego klucza sesji. JeÅ<9b>li nie zażÄ<85>dano klucza sesji, bÄ<99>dzie to wartoÅ<9b>Ä<87> 0.\""},"eventdata":{"subjectUserSid":"S-1-5-18","subjectUserName":"LAP999$","subjectDomainName":"local","subjectLogonId":"0x3e7","targetUserSid":"S-1-5-18","targetUserName":"SYSTEM","targetDomainName":"ZARZÄ<84>DZANIE NT","targetLogonId":"0x3e7","logonType":"5","logonProcessName":"Advapi","authenticationPackageName":"Negotiate","logonGuid":"{00000000-0000-0000-0000-000000000000}","keyLength":"0","processId":"0x3e4","processName":"C:\\\\Windows\\\\System32\\\\services.exe","impersonationLevel":"%%1833","virtualAccount":"%%1843","targetLinkedLogonId":"0x0","elevatedToken":"%%1842"}}},"location":"EventChannel"}

 filebeat test output
elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... OK
  version: 7.10.2

Its all in one installation on Centos8, newest version available in repository.
 Everything was restarted. 

How to debug and fix this? 
Zrzut ekranu 2023-10-05 071432.png

Md. Nazmur Sakib

unread,
Oct 5, 2023, 1:57:24 AM10/5/23
to Wazuh | Mailing List

Hi Marcin N.


Hope you are doing well. Thank you for using Wazuh.


Can you check if there is any error log in ossec.conf 


cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"


tail -n 50 /var/ossec/logs/ossec.log


check status of your wazuh-indexer and wazuh-manger


systemctl status wazuh-indexer

systemctl status wazuh-manager


Please share this information so that I can have a better understanding of your issue and guide you accordingly.


Regards

Md. Nazmur Sakib

Marcin N.

unread,
Oct 5, 2023, 2:02:51 AM10/5/23
to Wazuh | Mailing List
cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"
2023/10/05 00:00:10 wazuh-monitord: ERROR: date or location not NULL or p is NULL
2023/10/05 07:05:34 wazuh-modulesd: WARNING: 'update_from_year' option cannot be used for 'nvd' provider.
2023/10/05 07:05:38 wazuh-modulesd: WARNING: 'update_from_year' option cannot be used for 'nvd' provider.

systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-10-05 07:05:51 CEST; 55min ago
     Docs: https://documentation.wazuh.com
 Main PID: 95805 (java)
    Tasks: 195 (limit: 48936)
   Memory: 5.4G
   CGroup: /system.slice/wazuh-indexer.service
           └─95805 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMe>

Oct 05 07:05:40 SV-Wazuh.centrala.bsr.krakow.pl systemd[1]: Starting Wazuh-indexer...
Oct 05 07:05:42 SV-Wazuh.centrala.bsr.krakow.pl systemd-entrypoint[95805]: WARNING: A terminally deprecated method in java.lang.System has been called
Oct 05 07:05:42 SV-Wazuh.centrala.bsr.krakow.pl systemd-entrypoint[95805]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.6.0.jar)
Oct 05 07:05:42 SV-Wazuh.centrala.bsr.krakow.pl systemd-entrypoint[95805]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Oct 05 07:05:42 SV-Wazuh.centrala.bsr.krakow.pl systemd-entrypoint[95805]: WARNING: System::setSecurityManager will be removed in a future release
Oct 05 07:05:42 SV-Wazuh.centrala.bsr.krakow.pl systemd-entrypoint[95805]: WARNING: A terminally deprecated method in java.lang.System has been called
Oct 05 07:05:42 SV-Wazuh.centrala.bsr.krakow.pl systemd-entrypoint[95805]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.6.0.jar)
Oct 05 07:05:42 SV-Wazuh.centrala.bsr.krakow.pl systemd-entrypoint[95805]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Oct 05 07:05:42 SV-Wazuh.centrala.bsr.krakow.pl systemd-entrypoint[95805]: WARNING: System::setSecurityManager will be removed in a future release
Oct 05 07:05:51 SV-Wazuh.centrala.bsr.krakow.pl systemd[1]: Started Wazuh-indexer.


systemctl status wazuh-manager
● wazuh-manager.service - Wazuh manager
   Loaded: loaded (/usr/lib/systemd/system/wazuh-manager.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-10-05 07:05:40 CEST; 55min ago
    Tasks: 179 (limit: 48936)
   Memory: 716.4M
   CGroup: /system.slice/wazuh-manager.service
           ├─95373 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py
           ├─95413 /var/ossec/bin/wazuh-authd
           ├─95427 /var/ossec/bin/wazuh-db
           ├─95452 /var/ossec/bin/wazuh-execd
           ├─95464 /var/ossec/bin/wazuh-maild
           ├─95471 /var/ossec/bin/wazuh-analysisd
           ├─95481 /var/ossec/bin/wazuh-syscheckd
           ├─95486 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py
           ├─95489 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py
           ├─95588 /var/ossec/bin/wazuh-remoted
           ├─95598 /var/ossec/bin/wazuh-logcollector
           ├─95627 /var/ossec/bin/wazuh-monitord
           └─95639 /var/ossec/bin/wazuh-modulesd

Oct 05 07:05:37 SV-Wazuh.centrala.bsr.krakow.pl env[95314]: Started wazuh-analysisd...
Oct 05 07:05:38 SV-Wazuh.centrala.bsr.krakow.pl env[95314]: Started wazuh-syscheckd...
Oct 05 07:05:38 SV-Wazuh.centrala.bsr.krakow.pl env[95314]: Started wazuh-remoted...
Oct 05 07:05:38 SV-Wazuh.centrala.bsr.krakow.pl env[95314]: Started wazuh-logcollector...
Oct 05 07:05:38 SV-Wazuh.centrala.bsr.krakow.pl env[95314]: Started wazuh-monitord...
Oct 05 07:05:38 SV-Wazuh.centrala.bsr.krakow.pl env[95637]: 2023/10/05 07:05:38 wazuh-modulesd: WARNING: 'update_from_year' option cannot be used for 'nvd' provider.
Oct 05 07:05:38 SV-Wazuh.centrala.bsr.krakow.pl env[95314]: Started wazuh-modulesd...
Oct 05 07:05:38 SV-Wazuh.centrala.bsr.krakow.pl crontab[95737]: (root) LIST (root)
Oct 05 07:05:40 SV-Wazuh.centrala.bsr.krakow.pl env[95314]: Completed.
Oct 05 07:05:40 SV-Wazuh.centrala.bsr.krakow.pl systemd[1]: Started Wazuh manager.



tail -n 50 /var/ossec/logs/ossec.log
2023/10/05 00:00:10 wazuh-monitord: INFO: Starting new log after rotation.
2023/10/05 00:00:10 wazuh-monitord: INFO: Starting daily reporting for 'Daily report: Alerts with level higher than 8'
2023/10/05 00:00:10 wazuh-monitord: ERROR: date or location not NULL or p is NULL
2023/10/05 00:00:15 wazuh-monitord: INFO: Report 'Daily report: Alerts with level higher than 8' completed and zero alerts post-filter.
2023/10/05 00:00:15 wazuh-monitord: INFO: Report 'Daily report: Alerts with level higher than 8' empty.
2023/10/05 00:47:07 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/10/05 00:47:16 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/10/05 01:47:17 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/10/05 01:47:26 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/10/05 02:47:27 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/10/05 02:47:34 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/10/05 03:47:35 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/10/05 03:47:44 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/10/05 04:47:45 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/10/05 04:47:54 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/10/05 05:47:55 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/10/05 05:48:03 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/10/05 06:48:04 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/10/05 06:48:12 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/10/05 07:02:18 wazuh-modulesd:syscollector: INFO: Stop received for Syscollector.
2023/10/05 07:02:18 wazuh-modulesd:syscollector: INFO: Module finished.
2023/10/05 07:02:18 wazuh-monitord: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2023/10/05 07:02:18 wazuh-logcollector: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2023/10/05 07:02:18 wazuh-remoted: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2023/10/05 07:02:18 wazuh-syscheckd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2023/10/05 07:02:18 wazuh-analysisd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2023/10/05 07:02:18 wazuh-maild: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2023/10/05 07:02:19 wazuh-execd: INFO: (1314): Shutdown received. Deleting responses.
2023/10/05 07:02:19 wazuh-execd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2023/10/05 07:02:19 wazuh-db: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2023/10/05 07:02:20 wazuh-authd: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2023/10/05 07:02:21 wazuh-authd: INFO: Exiting...
2023/10/05 07:05:34 wazuh-modulesd: WARNING: 'update_from_year' option cannot be used for 'nvd' provider.
2023/10/05 07:05:36 wazuh-csyslogd: INFO: Remote syslog server not configured. Clean exit.
2023/10/05 07:05:36 wazuh-dbd: INFO: Database not configured. Clean exit.
2023/10/05 07:05:36 wazuh-integratord: INFO: Remote integrations not configured. Clean exit.
2023/10/05 07:05:36 wazuh-agentlessd: INFO: Not configured. Exiting.
2023/10/05 07:05:36 wazuh-authd: INFO: Started (pid: 95413).
2023/10/05 07:05:36 wazuh-authd: INFO: Accepting connections on port 1515. No password required.
2023/10/05 07:05:36 wazuh-authd: INFO: Setting network timeout to 1.000000 sec.
2023/10/05 07:05:36 wazuh-db: INFO: Started (pid: 95427).
2023/10/05 07:05:37 wazuh-execd: INFO: Started (pid: 95452).
2023/10/05 07:05:37 wazuh-maild: INFO: Started (pid: 95464).
2023/10/05 07:05:37 wazuh-maild: INFO: Getting alerts in log format.
2023/10/05 07:05:37 wazuh-syscheckd: INFO: Started (pid: 95481).
2023/10/05 07:05:37 wazuh-syscheckd: INFO: (6003): Monitoring path: '/bin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'.
2023/10/05 07:05:37 wazuh-syscheckd: INFO: (6003): Monitoring path: '/boot', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'.
2023/10/05 07:05:37 wazuh-syscheckd: INFO: (6003): Monitoring path: '/etc', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'.
2023/10/05 07:05:37 wazuh-syscheckd: INFO: (6003): Monitoring path: '/sbin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'.
2023/10/05 07:05:37 wazuh-syscheckd: INFO: (6003): Monitoring path: '/usr/bin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'.
2023/10/05 07:05:37 wazuh-syscheckd: INFO: (6003): Monitoring path: '/usr/sbin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | schedule

Md. Nazmur Sakib

unread,
Oct 5, 2023, 2:10:15 AM10/5/23
to Wazuh | Mailing List
Hi Marcin N. 
Can you also share the output of this command
cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"

Regards

Marcin N.

unread,
Oct 5, 2023, 2:14:29 AM10/5/23
to Wazuh | Mailing List
cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"
2023/10/05 00:00:10 wazuh-monitord: ERROR: date or location not NULL or p is NULL
2023/10/05 07:05:34 wazuh-modulesd: WARNING: 'update_from_year' option cannot be used for 'nvd' provider.
2023/10/05 07:05:38 wazuh-modulesd: WARNING: 'update_from_year' option cannot be used for 'nvd' provider.


two days ago my dashboard stopped showing allerts on same time. Last showed log was: 
Oct 3, 2023 @ 01:59:58.146

Md. Nazmur Sakib

unread,
Oct 5, 2023, 3:32:44 AM10/5/23
to Wazuh | Mailing List

Hi  Marcin N.


I cannot see any relevant log related to your issue. 


Your indexer and filebeat are also working fine.

Can you check if you have any relevant error logs inside Custer log

cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

 As per my understanding. you do not have any alerts in your dashboard currently.

Can you check if you have alerts inside your alert.log file?


tail -n 50 /var/ossec/logs/alerts/alerts.log


Also, check the current status of your storage.

df -h


Regards

Md. Nazmur Sakib

Marcin N.

unread,
Oct 5, 2023, 3:43:35 AM10/5/23
to Wazuh | Mailing List
No alerts in dashboard from 01:59:00
email alerst working fine atm
new allerts presents  /var/ossec/logs/alerts/alerts.json correctly


cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"
[2023-10-05T07:05:43,002][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms3880m, -Xmx3880m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-739452747579112662, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=2034237440, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2023-10-05T07:05:43,761][WARN ][stderr                   ] [node-1] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2023-10-05T07:05:43,762][WARN ][stderr                   ] [node-1] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2023-10-05T07:05:43,762][WARN ][stderr                   ] [node-1] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2023-10-05T07:05:48,131][WARN ][o.o.s.c.Salt             ] [node-1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2023-10-05T07:05:48,159][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2023-10-05T07:05:48,160][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No default storage available, audit log may not work properly. Please check configuration.
[2023-10-05T07:05:49,761][WARN ][o.o.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-10-05T07:05:51,580][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [node-1] Config override setting update called with empty string. Ignoring.
[2023-10-05T07:05:51,604][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-10-05T07:05:51,976][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] get managed-index failed: [.opendistro-ism-config] IndexNotFoundException[no such index [.opendistro-ism-config]]
[2023-10-05T07:05:51,981][WARN ][o.o.o.i.ObservabilityIndex] [node-1] message: index [.opensearch-observability/VEV1xC3zSPWidODlSnMtZg] already exists
[2023-10-05T07:05:57,716][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2023-10-05T07:05:57,760][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2023-10-05T07:05:57,771][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2023-10-05T07:05:57,773][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2023-10-05T07:05:57,775][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2023-10-05T07:06:00,245][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2023-10-05T07:06:00,251][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2023-10-05T07:06:00,253][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2023-10-05T07:06:00,255][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2023-10-05T07:06:02,746][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2023-10-05T07:06:02,753][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2023-10-05T07:06:02,760][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2023-10-05T07:06:02,763][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2023-10-05T07:06:13,728][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696482369118, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T07:06:13,733][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696482369118, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T07:06:13,728][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696482369118, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T07:06:13,745][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696482369118, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T07:06:13,875][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696482369118, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T07:06:13,876][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: java.net.SocketException: Connection reset
[2023-10-05T07:06:35,462][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696482369118, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T07:06:35,463][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696482369118, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T07:06:47,117][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696482369118, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T07:08:26,742][ERROR][o.o.a.u.AlertingException] [node-1] Alerting error: [.opendistro-alerting-config] IndexNotFoundException[no such index [.opendistro-alerting-config]]
[2023-10-05T07:08:34,025][WARN ][o.o.n.a.PluginBaseAction ] [node-1] notifications:IllegalArgumentException:
[2023-10-05T07:36:21,538][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696484168170, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T07:36:21,538][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696484168170, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T07:47:48,380][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: java.net.SocketException: Connection reset
[2023-10-05T07:47:48,760][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: java.net.SocketException: Connection reset
[2023-10-05T07:47:48,761][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: java.net.SocketException: Connection reset
[2023-10-05T08:04:26,914][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: java.net.SocketException: Connection reset
[2023-10-05T08:04:43,169][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696485862268, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T08:04:59,775][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696485862268, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T08:07:54,459][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696486046557, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T08:10:36,573][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696486188646, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T08:10:42,125][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696486188646, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T08:10:42,133][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696486188646, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T08:12:09,953][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696486188646, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T08:12:09,974][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696486188646, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T08:12:09,976][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1696486188646, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2023-10-05T09:00:42,845][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: java.net.SocketException: Connection reset






Wazuh->/var/log/secure
Rule: 5715 (level 3) -> 'sshd: authentication success.'
Src IP: 192.168.99.99
Src Port: 43892
User: wazuh-user
Oct  5 09:36:08 Wazuh sshd[113681]: Accepted password for wazuh-siem from 192.168.99.99 port 43892 ssh2

** Alert 1696491369.34023620: - pam,syslog,authentication_success,pci_dss_10.2.5,gpg13_7.8,gpg13_7.9,gdpr_IV_32.2,hipaa_164.312.b,nist_800_53_AU.14,nist_800_53_AC.7,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,
2023 Oct 05 09:36:09 Wazuh->/var/log/secure
Rule: 5501 (level 3) -> 'PAM: Login session opened.'
User: wazuh-user
Oct  5 09:36:08 Wazuh systemd[113688]: pam_unix(systemd-user:session): session opened for user wazuh-siemby (uid=0)
uid: 0

** Alert 1696491369.34024061: - pam,syslog,authentication_success,pci_dss_10.2.5,gpg13_7.8,gpg13_7.9,gdpr_IV_32.2,hipaa_164.312.b,nist_800_53_AU.14,nist_800_53_AC.7,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,
2023 Oct 05 09:36:09 Wazuh->/var/log/secure
Rule: 5501 (level 3) -> 'PAM: Login session opened.'
User: wazuh-siem
Oct  5 09:36:08 Wazuh sshd[113681]: pam_unix(sshd:session): session opened for user wazuh-siem by (uid=0)
uid: 0

** Alert 1696491377.34024491: - pam,syslog,authentication_success,pci_dss_10.2.5,gpg13_7.8,gpg13_7.9,gdpr_IV_32.2,hipaa_164.312.b,nist_800_53_AU.14,nist_800_53_AC.7,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,
2023 Oct 05 09:36:17 Wazuh->/var/log/secure
Rule: 5501 (level 3) -> 'PAM: Login session opened.'
User: root99
Oct  5 09:36:16 Wazuh su[113738]: pam_unix(su-l:session): session opened for user root99 by wazuh-siem(uid=1000)
uid: 1000

** Alert 1696491429.34024923: - syslog,cisco_ios,gpg13_4.3,
2023 Oct 05 09:37:09 192.168.99.193->/var/log/remote/cisco/99.193.log
Rule: 4713 (level 4) -> 'Cisco IOS error message - UPDOWN'
Oct  5 09:37:08 192.168.99.193 1422: Oct  5 09:37:07.192 CEST: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/21, changed state to down
cisco.facility: LINK
cisco.severity: 3
cisco.mnemonic: UPDOWN

** Alert 1696491431.34025312: - syslog,cisco_ios,gpg13_4.3,
2023 Oct 05 09:37:11 192.168.99.193->/var/log/remote/cisco/99.193.log
Rule: 4713 (level 4) -> 'Cisco IOS error message - UPDOWN'
Oct  5 09:37:10 192.168.99.193 1423: Oct  5 09:37:09.813 CEST: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/21, changed state to up
cisco.facility: LINK
cisco.severity: 3
cisco.mnemonic: UPDOWN

** Alert 1696491448.34025699: - local,systemd,gpg13_4.3,gdpr_IV_35.7.d,
2023 Oct 05 09:37:28 (Rocket) any->/var/log/messages
Rule: 40704 (level 5) -> 'Systemd: Service exited due to a failure.'
Oct  5 09:37:27 Rocket systemd[1]: dnf-makecache.service: Main process exited, code=exited, status=1/FAILURE





df -h
Filesystem          Size  Used Avail Use% Mounted on
devtmpfs            3.8G     0  3.8G   0% /dev
tmpfs               3.8G   60K  3.8G   1% /dev/shm
tmpfs               3.8G   25M  3.8G   1% /run
tmpfs               3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/sda2            30G  4.5G   24G  16% /
/dev/mapper/cs-var   98G   14G   80G  15% /var
/dev/sda1           500M  7.3M  493M   2% /boot/efi
tmpfs               769M     0  769M   0% /run/user/1000
 

Marcin N.

unread,
Oct 5, 2023, 4:22:20 AM10/5/23
to Wazuh | Mailing List


hour of last log was taken from my browaser locale 01:59 but logs showed timestamp was: 23:59:x

Found this log - not sure if this is related or nope: 

Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: 2023-10-05 00:00:00,723 opensearch[node-1][clusterManagerService#updateTask][T#1] ERROR Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation") java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessUserInformation")
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/java.security.AccessController.checkPermission(AccessController.java:1068)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:416)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.checkWriteExtended(UnixFileAttributeViews.java:195)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:264)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:299)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.util.FileUtils.defineFilePosixAttributeView(FileUtils.java:177)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.appender.FileManager.defineAttributeView(FileManager.java:215)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.appender.FileManager.createOutputStream(FileManager.java:202)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:419)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.rollover(RollingFileManager.java:396)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.checkRollover(RollingFileManager.java:308)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:311)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:542)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:500)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:483)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.Logger.log(Logger.java:161)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2205)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2159)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2142)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2040)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1907)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.spi.AbstractLogger.info(AbstractLogger.java:1449)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.metadata.MetadataUpdateSettingsService$1.execute(MetadataUpdateSettingsService.java:247)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:874)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:190)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:228)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/java.lang.Thread.run(Thread.java:833)



Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: 2023-10-05 00:00:00,734 opensearch[node-1][clusterManagerService#updateTask][T#1] ERROR Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster.log" got access denied ("java.lang.RuntimePermission" "accessUserInformation") java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessUserInformation")
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/java.security.AccessController.checkPermission(AccessController.java:1068)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:416)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.checkWriteExtended(UnixFileAttributeViews.java:195)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:264)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:299)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.util.FileUtils.defineFilePosixAttributeView(FileUtils.java:177)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.appender.FileManager.defineAttributeView(FileManager.java:215)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.appender.FileManager.createOutputStream(FileManager.java:202)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:419)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.rollover(RollingFileManager.java:396)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.checkRollover(RollingFileManager.java:308)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:311)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:542)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:500)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:483)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.core.Logger.log(Logger.java:161)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2205)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2159)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2142)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2040)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1907)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.apache.logging.log4j.spi.AbstractLogger.info(AbstractLogger.java:1449)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.metadata.MetadataUpdateSettingsService$1.execute(MetadataUpdateSettingsService.java:247)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:874)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:190)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:228)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Oct  5 00:00:00 Wazuh systemd-entrypoint[1774]: #011at java.base/java.lang.Thread.run(Thread.java:833)

Seems that problem is related with logrotate: 
Not sure why this started to showed 2 days ago. Soft wasnt updated or maintenanced in this time. 

Md. Nazmur Sakib

unread,
Oct 11, 2023, 9:24:24 AM10/11/23
to Wazuh | Mailing List

Hi Marcin N.

Hope you are doing well. Sorry for the late response.

Your storage status looks fine.

Is cleaning logs for recent indices  (dev tools  DELETE /wazuh-alerts-4.x-2023.10.*  ) working for you now?

As I can see this might be the Certificate issue with the dashboard, So please check out here for more info and also check whether the Certificates are deployed correctly or not: 

https://documentation.wazuh.com/current/installation-guide/wazuh-dashboard/step-by-step.html#deploying-certificates

Let me know the update on the issue.


Regards
Md. Nazmur Sakib
Reply all
Reply to author
Forward
0 new messages