Internal Server Error

598 views
Skip to first unread message

Avinash Mohod

unread,
Apr 30, 2024, 11:03:08 AM4/30/24
to Wazuh | Mailing List
Hi,

We are facing error with wazuh getting error "An internal server error occurred".

Below error details 


Please advice how to fix this error.

Regards
Avinash

Delfina Lizarralde Bressan

unread,
Apr 30, 2024, 11:21:39 AM4/30/24
to Wazuh | Mailing List
Hi Avinash,
Thanks for using wazuh!

To get more information about this error, kindly assist with the output of the below commands:
  • Wazuh indexer:
  • cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"
  • Wazuh manager:
  • cat /var/log/filebeat/filebeat | grep -i -E "error|warn"
  • cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"
  • Wazuh dashboard:
  • journalctl -u wazuh-dashboard
  • cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"


Looking forward to your response.

Avinash Mohod

unread,
Apr 30, 2024, 2:35:25 PM4/30/24
to Wazuh | Mailing List
Here are the logs

  • cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

[2024-04-30T09:03:35,695][INFO ][o.o.j.s.JobSweeper       ] [node-1] Error while sweeping shard [.opendistro-ism-config][0], error message: all shards failed
[2024-04-30T09:41:37,232][ERROR][o.o.j.s.u.LockService    ] [node-1] Exception occurred finding lock
[2024-04-30T09:41:37,245][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [DefaultDispatcher-worker-5]
[2024-04-30T09:42:20,041][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-3123864651402167666, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2024-04-30T09:42:31,136][WARN ][o.o.s.c.Salt             ] [node-1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2024-04-30T09:42:31,208][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2024-04-30T09:42:31,211][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No default storage available, audit log may not work properly. Please check configuration.
[2024-04-30T09:42:34,217][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-04-30T09:42:34,221][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-04-30T09:42:34,222][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [node-1] Fail to read queue capacity via reflection
[2024-04-30T09:42:34,382][WARN ][o.o.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-04-30T09:42:36,472][ERROR][o.o.s.t.SecurityRequestHandler] [node-1] OpenSearchException[Transport client authentication no longer supported.]
[2024-04-30T09:42:36,477][ERROR][o.o.s.t.SecurityRequestHandler] [node-1] OpenSearchException[Transport client authentication no longer supported.]
[2024-04-30T09:42:36,482][WARN ][o.o.d.HandshakingTransportAddressConnector] [node-1] handshake failed for [connectToRemoteMasterNode[[::1]:9300]]
[2024-04-30T09:42:36,482][WARN ][o.o.d.HandshakingTransportAddressConnector] [node-1] handshake failed for [connectToRemoteMasterNode[127.0.0.1:9300]]
[2024-04-30T09:42:36,770][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [node-1] Config override setting update called with empty string. Ignoring.
[2024-04-30T09:42:37,315][WARN ][o.o.o.i.ObservabilityIndex] [node-1] message: index [.opensearch-observability/1KESkF77QFS8RLXPzEsF3g] already exists
[2024-04-30T09:42:37,315][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] get managed-index failed: NoShardAvailableActionException[No shard available for [org.opensearch.action.get.MultiGetShardRequest@532d9667]]
[2024-04-30T09:42:37,393][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to get ISM policies with templates: Failed to execute phase [query], all shards failed
[2024-04-30T09:42:37,742][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-04-30T09:42:37,774][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-04-30T09:42:37,780][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-04-30T09:42:37,786][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-04-30T09:42:37,823][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@23748ac6] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-04-30T09:42:37,823][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@23748ac6] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)

  • cat /var/log/filebeat/filebeat | grep -i -E "error|warn"

No logs

  • cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"

2024/04/30 00:03:48 wazuh-modulesd:vulnerability-detector: WARNING: (5575): Unavailable vulnerability data for the agent '000' OS. Skipping it.
2024/04/30 00:08:48 wazuh-modulesd:vulnerability-detector: WARNING: (5575): Unavailable vulnerability data for the agent '000' OS. Skipping it.
2024/04/30 00:13:48 wazuh-modulesd:vulnerability-detector: WARNING: (5575): Unavailable vulnerability data for the agent '000' OS. Skipping it.
2024/04/30 00:18:48 wazuh-modulesd:vulnerability-detector: WARNING: (5575): Unavailable vulnerability data for the agent '000' OS. Skipping it.
2024/04/30 09:42:16 wazuh-authd: WARNING: The <force_insert> tag is deprecated. Use <force> instead.
2024/04/30 09:42:16 wazuh-authd: WARNING: The <force_time> tag is deprecated. Use <force> instead.
2024/04/30 09:42:18 wazuh-modulesd: WARNING: The <ignore_time> tag at module 'vulnerability-detector' is deprecated for version newer than 4.3.
2024/04/30 09:42:18 wazuh-modulesd: WARNING: 'update_from_year' option cannot be used for 'nvd' provider.
2024/04/30 09:42:22 wazuh-authd: WARNING: The <force_insert> tag is deprecated. Use <force> instead.
2024/04/30 09:42:22 wazuh-authd: WARNING: The <force_time> tag is deprecated. Use <force> instead.
2024/04/30 09:42:27 wazuh-modulesd: WARNING: The <ignore_time> tag at module 'vulnerability-detector' is deprecated for version newer than 4.3.
2024/04/30 09:42:27 wazuh-modulesd: WARNING: 'update_from_year' option cannot be used for 'nvd' provider.

  • journalctl -u wazuh-dashboard

Apr 30 09:42:05 wazuh-centos opensearch-dashboards[657]: v16.20.0
Apr 30 09:42:11 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:11Z","tags":["info","plugins-service"],"pid":657,"message":"Plugin \"dataSourceManagement\" has been disable
Apr 30 09:42:11 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:11Z","tags":["info","plugins-service"],"pid":657,"message":"Plugin \"dataSource\" is disabled."}
Apr 30 09:42:11 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:11Z","tags":["info","plugins-service"],"pid":657,"message":"Plugin \"visTypeXy\" is disabled."}
Apr 30 09:42:11 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:11Z","tags":["info","plugins-service"],"pid":657,"message":"Plugin \"mlCommonsDashboards\" is disabled."}
Apr 30 09:42:12 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:12Z","tags":["info","plugins-system"],"pid":657,"message":"Setting up [44] plugins: [alertingDashboards,usag
Apr 30 09:42:12 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:12Z","tags":["info","savedobjects-service"],"pid":657,"message":"Waiting until all OpenSearch nodes are comp
Apr 30 09:42:12 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:12Z","tags":["error","opensearch","data"],"pid":657,"message":"[ConnectionError]: connect ECONNREFUSED 127.0
Apr 30 09:42:12 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:12Z","tags":["error","savedobjects-service"],"pid":657,"message":"Unable to retrieve version information fro
Apr 30 09:42:15 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:15Z","tags":["error","opensearch","data"],"pid":657,"message":"[ConnectionError]: connect ECONNREFUSED 127.0
Apr 30 09:42:17 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:17Z","tags":["error","opensearch","data"],"pid":657,"message":"[ConnectionError]: connect ECONNREFUSED 127.0
Apr 30 09:42:20 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:20Z","tags":["error","opensearch","data"],"pid":657,"message":"[ConnectionError]: connect ECONNREFUSED 127.0
Apr 30 09:42:22 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:22Z","tags":["error","opensearch","data"],"pid":657,"message":"[ConnectionError]: connect ECONNREFUSED 127.0
Apr 30 09:42:25 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:25Z","tags":["error","opensearch","data"],"pid":657,"message":"[ConnectionError]: connect ECONNREFUSED 127.0
Apr 30 09:42:27 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:27Z","tags":["error","opensearch","data"],"pid":657,"message":"[ConnectionError]: connect ECONNREFUSED 127.0
Apr 30 09:42:30 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:30Z","tags":["error","opensearch","data"],"pid":657,"message":"[ConnectionError]: connect ECONNREFUSED 127.0
Apr 30 09:42:32 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:32Z","tags":["error","opensearch","data"],"pid":657,"message":"[ConnectionError]: connect ECONNREFUSED 127.0
Apr 30 09:42:35 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:35Z","tags":["error","opensearch","data"],"pid":657,"message":"[ConnectionError]: connect ECONNREFUSED 127.0
Apr 30 09:42:37 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:37Z","tags":["error","opensearch","data"],"pid":657,"message":"[ResponseError]: Response Error"}
Apr 30 09:42:40 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:40Z","tags":["error","opensearch","data"],"pid":657,"message":"[ResponseError]: Response Error"}
Apr 30 09:42:53 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:53Z","tags":["info","plugins-system"],"pid":657,"message":"Starting [44] plugins: [alertingDashboards,usageC
Apr 30 09:42:54 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:54Z","tags":["listening","info"],"pid":657,"message":"Server running at https://0.0.0.0:443"}
Apr 30 09:42:54 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:42:54Z","tags":["info","http","server","OpenSearchDashboards"],"pid":657,"message":"http server running at http
Apr 30 09:43:11 wazuh-centos opensearch-dashboards[657]: {"type":"error","@timestamp":"2024-04-30T14:43:11Z","tags":["connection","client","error"],"pid":657,"level":"error","error":{"message":"139767376021440:
Apr 30 09:43:11 wazuh-centos opensearch-dashboards[657]: {"type":"error","@timestamp":"2024-04-30T14:43:11Z","tags":["connection","client","error"],"pid":657,"level":"error","error":{"message":"139767376021440:
Apr 30 09:43:12 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:12Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:43:12 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:12Z","tags":["error","savedobjects-service"],"pid":657,"message":"Unable to retrieve version information fro
Apr 30 09:43:14 wazuh-centos opensearch-dashboards[657]: {"type":"error","@timestamp":"2024-04-30T14:43:14Z","tags":["connection","client","error"],"pid":657,"level":"error","error":{"message":"139767376021440:
Apr 30 09:43:14 wazuh-centos opensearch-dashboards[657]: {"type":"error","@timestamp":"2024-04-30T14:43:14Z","tags":["connection","client","error"],"pid":657,"level":"error","error":{"message":"139767376021440:
Apr 30 09:43:14 wazuh-centos opensearch-dashboards[657]: {"type":"error","@timestamp":"2024-04-30T14:43:14Z","tags":["connection","client","error"],"pid":657,"level":"error","error":{"message":"139767376021440:
Apr 30 09:43:15 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:15Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:43:15 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:15Z","tags":["error","http","server","OpenSearchDashboards"],"pid":657,"message":"Error: [circuit_breaking_e
Apr 30 09:43:15 wazuh-centos opensearch-dashboards[657]: {"type":"error","@timestamp":"2024-04-30T14:43:15Z","tags":[],"pid":657,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack"
Apr 30 09:43:15 wazuh-centos opensearch-dashboards[657]: {"type":"response","@timestamp":"2024-04-30T14:43:15Z","tags":[],"pid":657,"method":"get","statusCode":500,"req":{"url":"/app/wazuh","method":"get","head
Apr 30 09:43:17 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:17Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:43:20 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:20Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:43:22 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:22Z","tags":["error","http","server","OpenSearchDashboards"],"pid":657,"message":"Error: [circuit_breaking_e
Apr 30 09:43:22 wazuh-centos opensearch-dashboards[657]: {"type":"error","@timestamp":"2024-04-30T14:43:22Z","tags":[],"pid":657,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack"
Apr 30 09:43:22 wazuh-centos opensearch-dashboards[657]: {"type":"response","@timestamp":"2024-04-30T14:43:22Z","tags":[],"pid":657,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":{"ho
Apr 30 09:43:22 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:22Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:43:30 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:30Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:43:30 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:30Z","tags":["error","savedobjects-service"],"pid":657,"message":"Unable to retrieve version information fro
Apr 30 09:43:35 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:35Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:43:35 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:35Z","tags":["error","savedobjects-service"],"pid":657,"message":"Unable to retrieve version information fro
Apr 30 09:43:40 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:40Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:43:40 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:40Z","tags":["error","savedobjects-service"],"pid":657,"message":"Unable to retrieve version information fro
Apr 30 09:43:40 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:40Z","tags":["error","http","server","OpenSearchDashboards"],"pid":657,"message":"Error: [circuit_breaking_e
Apr 30 09:43:40 wazuh-centos opensearch-dashboards[657]: {"type":"error","@timestamp":"2024-04-30T14:43:40Z","tags":[],"pid":657,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack"
Apr 30 09:43:40 wazuh-centos opensearch-dashboards[657]: {"type":"response","@timestamp":"2024-04-30T14:43:40Z","tags":[],"pid":657,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":{"ho
Apr 30 09:43:42 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:42Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:43:45 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:45Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:43:47 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:47Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:43:50 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:50Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:43:52 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:43:52Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:44:00 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:44:00Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:44:00 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:44:00Z","tags":["error","savedobjects-service"],"pid":657,"message":"Unable to retrieve version information fro
Apr 30 09:44:05 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:44:05Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:44:05 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:44:05Z","tags":["error","savedobjects-service"],"pid":657,"message":"Unable to retrieve version information fro
Apr 30 09:44:07 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:44:07Z","tags":["error","opensearch","data"],"pid":657,"message":"[circuit_breaking_exception]: [parent] Data t
Apr 30 09:44:08 wazuh-centos opensearch-dashboards[657]: {"type":"log","@timestamp":"2024-04-30T14:44:08Z","tags":["error","http","server","OpenSearchDashboards"],"pid":657,"message":"Error: [circuit_breaking_e
Apr 30 09:44:08 wazuh-centos opensearch-dashboards[657]: {"type":"error","@timestamp":"2024-04-30T14:44:08Z","tags":[],"pid":657,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack"
Apr 30 09:44:08 wazuh-centos opensearch-dashboards[657]: {"type":"response","@timestamp":"2024-04-30T14:44:08Z","tags":[],"pid":657,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":{"ho

  • cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

{"data":{"message":"Could not check if the index wazuh-statistics-2024.18w exists due to no permissions for create, delete or check","stack":"ResponseError: Could not check if the index wazuh-statistics-2024.18w exists due to no permissions for create, delete or check\n    at onBody (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:374:23)\n    at IncomingMessage.onEnd (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:293:11)\n    at IncomingMessage.emit (node:events:525:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1358:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)"},"date":"2024-04-30T18:00:01.503Z","level":"info","location":"Cron-scheduler"}
{"data":{"message":"circuit_breaking_exception: [circuit_breaking_exception] Reason: [parent] Data too large, data for [<http_request>] would be [1020800996/973.5mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1020800376/973.5mb], new bytes reserved: [620/620b], usages [request=0/0b, fielddata=8124/7.9kb, in_flight_requests=620/620b]","stack":"ResponseError: circuit_breaking_exception: [circuit_breaking_exception] Reason: [parent] Data too large, data for [<http_request>] would be [1020800996/973.5mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1020800376/973.5mb], new bytes reserved: [620/620b], usages [request=0/0b, fielddata=8124/7.9kb, in_flight_requests=620/620b]\n    at onBody (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:374:23)\n    at IncomingMessage.onEnd (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:293:11)\n    at IncomingMessage.emit (node:events:525:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1358:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)"},"date":"2024-04-30T18:05:00.431Z","level":"info","location":"Cron-scheduler"}
{"data":{"message":"circuit_breaking_exception: [circuit_breaking_exception] Reason: [parent] Data too large, data for [<http_request>] would be [1020802846/973.5mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1020800376/973.5mb], new bytes reserved: [2470/2.4kb], usages [request=0/0b, fielddata=8124/7.9kb, in_flight_requests=2470/2.4kb]","stack":"ResponseError: circuit_breaking_exception: [circuit_breaking_exception] Reason: [parent] Data too large, data for [<http_request>] would be [1020802846/973.5mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1020800376/973.5mb], new bytes reserved: [2470/2.4kb], usages [request=0/0b, fielddata=8124/7.9kb, in_flight_requests=2470/2.4kb]\n    at onBody (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:374:23)\n    at IncomingMessage.onEnd (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:293:11)\n    at IncomingMessage.emit (node:events:525:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1358:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)"},"date":"2024-04-30T18:05:00.433Z","level":"info","location":"Cron-scheduler"}
{"data":{"message":"Could not check if the index wazuh-statistics-2024.18w exists due to no permissions for create, delete or check","stack":"ResponseError: Could not check if the index wazuh-statistics-2024.18w exists due to no permissions for create, delete or check\n    at onBody (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:374:23)\n    at IncomingMessage.onEnd (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:293:11)\n    at IncomingMessage.emit (node:events:525:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1358:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)"},"date":"2024-04-30T18:20:01.371Z","level":"info","location":"Cron-scheduler"}
{"data":{"message":"Could not check if the index wazuh-statistics-2024.18w exists due to no permissions for create, delete or check","stack":"ResponseError: Could not check if the index wazuh-statistics-2024.18w exists due to no permissions for create, delete or check\n    at onBody (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:374:23)\n    at IncomingMessage.onEnd (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:293:11)\n    at IncomingMessage.emit (node:events:525:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1358:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)"},"date":"2024-04-30T18:20:01.385Z","level":"info","location":"Cron-scheduler"}


Regards
Avinash

Jean-Olivier Abrial

unread,
May 2, 2024, 6:21:45 AM5/2/24
to Wazuh | Mailing List
Hi,
I have exactly the same problem. I'm running docker single-node

wazuh-error.png

Message has been deleted

Delfina Lizarralde Bressan

unread,
May 2, 2024, 9:01:34 AM5/2/24
to Wazuh | Mailing List
Hi Avinash!

it seems that you have  circuit_breaking_exception  which indicates RAM resource problems.
The  circuit_breaking_exception  is a mechanism used to prevent operations from causing an  OutOfMemoryError . It seems like Wazuh indexer was using most of the JVM heap configured, and the total memory required for all operations was higher to the memory available, so the operation you requested was aborted.

To fix this problem, increase the heap size, taking in account:
  • Use no more than 50% of available RAM.
  • Use no more than 32 GB.
Note that the JVM heap min and max The values min (Xms) and max (Xmx) sizes must be the same to prevent JVM heap resizing at runtime as this is a very costly process.

in the  java.options  file of the plugin platform:
  • Wazuh indexer (package):  /etc/wazuh-indexer/jvm.options 
  • Wazuh indexer (Docker):  /usr/share/wazuh-indexer/config/jvm.options 
Example:
If you want to increase to 6GB:
-Xms6g
-Xmx6g

Let me know if this works or if you have any other questions please.

Avinash Mohod

unread,
May 3, 2024, 5:46:33 AM5/3/24
to Delfina Lizarralde Bressan, Wazuh | Mailing List
Thank you, it worked.

Regards
Avinash

On Thu, May 2, 2024 at 9:00 AM 'Delfina Lizarralde Bressan' via Wazuh | Mailing List <wa...@googlegroups.com> wrote:
Hi Avinash!

it seems that you have  circuit_breaking_exception  which indicates RAM resource problems.
The  circuit_breaking_exception  is a mechanism used to prevent operations from causing an  OutOfMemoryError . It seems like Wazuh indexer was using most of the JVM heap configured, and the total memory required for all operations was higher to the memory available, so the operation you requested was aborted.

To fix this problem, increase the heap size, taking in account:
  • Use no more than 50% of available RAM.
  • Use no more than 32 GB.
Note that the JVM heap min and max The values min (Xms) and max (Xmx) sizes must be the same to prevent JVM heap resizing at runtime as this is a very costly process.

in the  java.options  file of the plugin platform:
  • Wazuh indexer (package): `/etc/wazuh-indexer/jvm.options`
  • Wazuh indexer (Docker): `/usr/share/wazuh-indexer/config/jvm.options`
Example:
If you want to increase to 6GB:
-Xms6g
-Xmx6g

Let me know if this works or if you have any other questions please.

On Thursday, May 2, 2024 at 7:21:45 AM UTC-3 Jean-Olivier Abrial wrote:

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/eJ3QSsgCK1Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/39a86f9a-8842-480b-a5fc-acb1ac5edc7an%40googlegroups.com.


--
Regards
Avinash
Reply all
Reply to author
Forward
Message has been deleted
0 new messages