null_pointer_exception- Internal Error

[Operation Consultant]

Hi Team,

Getting  -  null_pointer_exception

Cannot invoke "org.elasticsearch.search.aggregations.InternalAggregations.getSerializedSize()" because "reducePhase.aggregations" is null

Wrapper@https://10.X.X.X/36136/bundles/core/core.entry.js:6:4249
_createSuperInternal@https://10.X.X.X/36136/bundles/core/core.entry.js:6:3388
HttpFetchError@https://10.X.X.X/36136/bundles/core/core.entry.js:6:6016
_callee3$@https://10.X.X.X/36136/bundles/core/core.entry.js:6:59535
tryCatch@https://10.X.X.X/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:32004
invoke@https://10.X.X.X/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:35976
defineIteratorMethods/</prototype[method]@https://10.X.X.X/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:33129
fetch_asyncGeneratorStep@https://10.X.X.X/36136/bundles/core/core.entry.js:6:52652
_next@https://10.X.X.X/36136/bundles/core/core.entry.js:6:52992


##

root@siem01:/home/unix-ops# /var/ossec/bin/wazuh-control info | grep WAZUH_VERSION                                      
 WAZUH_VERSION="v4.2.5"


root@siem01:/home/unix-ops## systemctl status kibana
● kibana.service - Kibana
     Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-05-18 09:30:54 IST; 5h 9min ago
   Main PID: 10362 (node)
      Tasks: 11 (limit: 38361)
     Memory: 169.4M
     CGroup: /system.slice/kibana.service
             └─10362 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist -c /etc/kibana/kibana.yml

May 18 14:31:48 siem01 kibana[10362]: {"type":"response","@timestamp":"2022-05-18T09:01:47Z","tags":[],"pid":10362,"method":">
May 18 14:32:00 siem01 kibana[10362]: {"type":"log","@timestamp":"2022-05-18T09:02:00Z","tags":["error","elasticsearch","data>
May 18 14:32:00 siem01 kibana[10362]: {"type":"log","@timestamp":"2022-05-18T09:02:00Z","tags":["error","plugins","wazuh","cr>
May 18 14:32:00 siem01 kibana[10362]: {"type":"log","@timestamp":"2022-05-18T09:02:00Z","tags":["error","elasticsearch","data>
May 18 14:32:00 siem01 kibana[10362]: {"type":"log","@timestamp":"2022-05-18T09:02:00Z","tags":["error","plugins","wazuh","cr>
May 18 14:32:00 siem01 kibana[10362]: {"type":"log","@timestamp":"2022-05-18T09:02:00Z","tags":["error","elasticsearch","data>
May 18 14:37:00 siem01 kibana[10362]: {"type":"log","@timestamp":"2022-05-18T09:07:00Z","tags":["error","elasticsearch","data>
May 18 14:37:00 siem01 kibana[10362]: {"type":"log","@timestamp":"2022-05-18T09:07:00Z","tags":["error","elasticsearch","data>
May 18 14:37:00 siem01 kibana[10362]: {"type":"log","@timestamp":"2022-05-18T09:07:00Z","tags":["error","plugins","wazuh","cr>
May 18 14:37:00 siem01 kibana[10362]: {"type":"log","@timestamp":"2022-05-18T09:07:00Z","tags":["error","plugins","wazuh","cr>

[Alexander Bohorquez]

Hello,

Thank you for using Wazuh!

It would be useful if we can obtain more details about the issue in order to help in the best way possible.

The following questions are useful:

Did you make a change in the configuration of the Elasticsearch components that may lead to this situation? which one?

Is the server running out of space where the Elasticsearch components are working?

Please check the Elasticsearch and Filebeat log files and share with us the output:

cat /var/log/elasticsearch/<elasticsearch-cluster-name>.log | grep -i -E "error|warn"
cat /var/log/messages | grep -i filebeat

I look forward to your comments!

[Operation Consultant]

Hi All, 

PFB - 

root@siem01:/var/log/elasticsearch# cat elasticsearch.log | grep -i -E "error|warn"
Counters=TotalError=12,NetworkCollectionError=12
Counters=TotalError=12,NetworkCollectionError=12
Counters=TotalError=12,NetworkCollectionError=12
Counters=TotalError=12,NetworkCollectionError=12
Counters=TotalError=12,NetworkCollectionError=12
Counters=TotalError=12,NetworkCollectionError=12
[2022-06-06T03:35:33,843][WARN ][c.a.o.a.i.AnomalyDetectionIndices] [node-1] .opendistro-anomaly-results not rolled over. Conditions were: {[max_docs: 250000000]=false}
Counters=TotalError=12,NetworkCollectionError=12
[2022-06-06T03:36:48,330][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx8g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-10819868029660441438, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/elasticsearch/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=4294967296, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2022-06-06T03:36:51,916][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] Directory /etc/elasticsearch/certs has insecure file permissions (should be 0700)
[2022-06-06T03:36:51,916][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/kibana.pem has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,917][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/admin.key has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,917][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/kibana_http.pem has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,920][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/admin.pem has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,920][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/kibana.key has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,921][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/root-ca.key has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,921][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/kibana_http.key has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,921][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/client-certificates.readme has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,922][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] Directory /etc/elasticsearch/certs/old-certs has insecure file permissions (should be 0700)
[2022-06-06T03:36:51,922][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/old-certs/kibana.pem has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,923][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/old-certs/admin.key has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,923][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/old-certs/admin.pem has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,924][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/old-certs/kibana.key has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,924][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/old-certs/root-ca.key has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,924][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/old-certs/elasticsearch_http.pem has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,925][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/old-certs/elasticsearch.pem has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,925][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/old-certs/elasticsearch.key has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,926][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/old-certs/root-ca.pem has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,926][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/old-certs/elasticsearch_http.key has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,927][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/elasticsearch_http.pem has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,927][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/elasticsearch.pem has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,927][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/kibana_elasticsearch_config_snippet.yml has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,928][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/elasticsearch.key has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,928][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/root-ca.pem has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,928][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/elasticsearch_elasticsearch_config_snippet.yml has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,928][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/certs/elasticsearch_http.key has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,929][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/.elasticsearch.keystore.initial_md5sum has insecure file permissions (should be 0600)
[2022-06-06T03:36:51,929][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [node-1] File /etc/elasticsearch/elasticsearch.yml has insecure file permissions (should be 0600)
[2022-06-06T03:36:54,116][WARN ][c.a.o.r.s.PluginSettings ] [node-1] reports:Failed to load /etc/elasticsearch/opendistro-reports-scheduler/reports-scheduler.yml
[2022-06-06T03:36:58,043][WARN ][c.a.o.s.c.Salt           ] [node-1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2022-06-06T03:36:59,532][WARN ][o.e.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-06-06T03:37:02,521][WARN ][o.e.b.BootstrapChecks    ] [node-1] initial heap size [1073741824] not equal to maximum heap size [8589934592]; this can cause resize pauses
[2022-06-06T03:37:03,543][WARN ][c.a.o.e.p.c.s.h.ConfigOverridesClusterSettingHandler] [node-1] Config override setting update called with empty string. Ignoring.
[2022-06-06T03:37:05,156][ERROR][c.a.o.s.a.BackendRegistry] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-06-06T03:37:05,927][ERROR][c.a.o.i.i.ManagedIndexCoordinator] [node-1] get managed-index failed: [.opendistro-ism-config] IndexNotFoundException[no such index [.opendistro-ism-config]]
[2022-06-06T03:37:06,587][ERROR][c.a.o.s.a.BackendRegistry] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-06-06T03:37:08,572][WARN ][c.a.o.s.a.r.AuditMessageRouter] [node-1] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2022-06-06T03:37:30,070][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][29] overhead, spent [1.5s] collecting in the last [1.9s]
Counters=TotalError=13,NetworkCollectionError=13,MasterNodeNotUp=5
[2022-06-06T03:38:40,907][ERROR][c.a.o.s.a.s.InternalESSink] [node-1] Unable to index audit log {"audit_cluster_name":"elasticsearch","audit_transport_headers":{"_system_index_access_allowed":"false"},"audit_node_name":"node-1","audit_trace_task_id":"7VVUB_m2S1i9sItvNtun7w:4340","audit_transport_request_type":"CreateIndexRequest","audit_category":"INDEX_EVENT","audit_request_origin":"REST","audit_request_body":"{}","audit_node_id":"7VVUB_m2S1i9sItvNtun7w","audit_request_layer":"TRANSPORT","@timestamp":"2022-06-05T22:07:10.198+00:00","audit_format_version":4,"audit_request_remote_address":"127.0.0.1","audit_request_privilege":"indices:admin/auto_create","audit_node_host_address":"127.0.0.1","audit_request_effective_user":"admin","audit_trace_indices":["<wazuh-alerts-4.x-{2022.06.05||/d{yyyy.MM.dd|UTC}}>"],"audit_node_host_name":"127.0.0.1"} due to UnavailableShardsException[[security-auditlog-2022.06.05][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[security-auditlog-2022.06.05][0]] containing [index {[security-auditlog-2022.06.05][_doc][OB3oNYEBu5xRTVKW16tz], source[{"audit_cluster_name":"elasticsearch","audit_transport_headers":{"_system_index_access_allowed":"false"},"audit_node_name":"node-1","audit_trace_task_id":"7VVUB_m2S1i9sItvNtun7w:4340","audit_transport_request_type":"CreateIndexRequest","audit_category":"INDEX_EVENT","audit_request_origin":"REST","audit_request_body":"{}","audit_node_id":"7VVUB_m2S1i9sItvNtun7w","audit_request_layer":"TRANSPORT","@timestamp":"2022-06-05T22:07:10.198+00:00","audit_format_version":4,"audit_request_remote_address":"127.0.0.1","audit_request_privilege":"indices:admin/auto_create","audit_node_host_address":"127.0.0.1","audit_request_effective_user":"admin","audit_trace_indices":["<wazuh-alerts-4.x-{2022.06.05||/d{yyyy.MM.dd|UTC}}>"],"audit_node_host_name":"127.0.0.1"}]}] and a refresh]]
Counters=TotalError=12,NetworkCollectionError=12
Counters=TotalError=13,NetworkCollectionError=13
Counters=TotalError=11,NetworkCollectionError=11
Counters=TotalError=12,NetworkCollectionError=12
Counters=TotalError=12,NetworkCollectionError=12
Counters=TotalError=13,NetworkCollectionError=13
Counters=TotalError=11,NetworkCollectionError=11
Counters=TotalError=13,NetworkCollectionError=13
Counters=TotalError=12,NetworkCollectionError=12
Counters=TotalError=11,NetworkCollectionError=11
Counters=TotalError=12,NetworkCollectionError=12
Counters=TotalError=12,NetworkCollectionError=12
Counters=TotalError=13,NetworkCollectionError=13
[2022-06-06T10:21:12,700][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {size=1, index=wazuh-alerts-*}
[2022-06-06T10:21:13,015][WARN ][r.suppressed             ] [node-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1654490563603, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
Counters=TotalError=12,NetworkCollectionError=12
Counters=TotalError=11,NetworkCollectionError=11
root@siem01:/var/log/elasticsearch#
root@siem01:/var/log/elasticsearch#
root@siem01:/var/log/elasticsearch#

[Operation Consultant]

Hi All ,

kindly help me out. 

Regards,

[Alexander Bohorquez]

Hello,

I apologize for the delay with this,

Based on the logs you have shared:

I've seen these errors:

[2022-06-06T03:37:05,156][ERROR][c.a.o.s.a.BackendRegistry] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-06-06T03:37:05,927][ERROR][c.a.o.i.i.ManagedIndexCoordinator] [node-1] get managed-index failed: [.opendistro-ism-config] IndexNotFoundException[no such index [.opendistro-ism-config]]
[2022-06-06T03:37:06,587][ERROR][c.a.o.s.a.BackendRegistry] [node-1] Not yet initialized (you may need to run securityadmin)

This tells us that the cluster doesn't seem to be initialized correctly and that we need to run the securityadmin script first:

If you are using Opendistro with version 4.2.5 you probably have version 1.13.2 of Opendistro with Elasticsearch 7.10.2, so I leave you the reference of how to run the securityadmin script:

export JAVA_HOME=/usr/share/elasticsearch/jdk/ && /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/ -nhnv -cacert /etc/elasticsearch/certs/root-ca.pem -cert /etc/elasticsearch/certs/admin.pem -key /etc/elasticsearch/certs/admin-key.pem

Reference: https://opendistro.github.io/for-elasticsearch-docs/docs/security/configuration/security-admin/

If you are using custom certificates you can change the names on the line above.

NOTE: in case you have made changes in the Opendistro security configuration, you must execute this in the node that has the changes or that you have made previously. Typically this runs on the main or Elasticsearch Master node.

After starting the cluster, have you verified what was mentioned in my last answer?

• Is the server running out of space where the Elasticsearch components are working?

• How many Elasticsearch nodes do you have?

I hope this information helps. Please let me know how it goes and if you have any questions!

Regards.