I appreciate any help you can provide - I've read through a number of support sites and I'm still at an impasse. After a reboot of our Ubuntu 20.04 server to restart all services, when I attempted to log into the Wazuh portal I got the dreaded Kibana Server is Not Ready message. I waited some time in case I had been too hasty attempting to login, but hours later (and now a day later) the message remains.
My apologies if the solution is right in front of me - I'm just getting up to speed on Linux and Wazuh. Below are the .yml.files for Elasticsearch, filebeat and kibana, along with a variety of logs, but, I'm happy to provide additional information if needed.
The issue appears to be an authentication issue, but, I'm a little at a loss as I've rebooted the server before after applying patches and did not receive the error. I didn't patch anything yesterday, but, I do tend to reboot boxes only I use on Monday mornings to free up resources, etc. (Guess that is the Windows guy in me).
Elasticsearch.yml
network.host: xxx.xxx.32.33
cluster.initial_master_nodes: node-1
opendistro_security.ssl.transport.pemcert_filepath: /etc/elasticsearch/certs/elasticsearch.pem
opendistro_security.ssl.transport.pemkey_filepath: /etc/elasticsearch/certs/elasticsearch.key
opendistro_security.ssl.transport.pemtrustedcas_filepath: /etc/elasticsearch/certs/root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.transport.resolve_hostname: false
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: /etc/elasticsearch/certs/elasticsearch_http.pem
opendistro_security.ssl.http.pemkey_filepath: /etc/elasticsearch/certs/elasticsearch_http.key
opendistro_security.ssl.http.pemtrustedcas_filepath: /etc/elasticsearch/certs/root-ca.pem
opendistro_security.nodes_dn:
- CN=node-1,OU=Docu,O=Wazuh,L=California,C=US
opendistro_security.authcz.admin_dn:
- CN=admin,OU=Docu,O=Wazuh,L=California,C=US
opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
Filebeat.yml
# Wazuh - Filebeat configuration file
output.elasticsearch:
hosts: ["xxx.xxx.32.33:9200"]
username: "admin"
password: "xxxxxxxxx"
ssl.verification_mode: none
protocol: https
#username: "Internal Filebeat User"
#password: "xxxxxxxxxxxxxxxxxxxxxxxx"
ssl.certificate_authorities:
- /etc/filebeat/certs/root-ca.pem
ssl.certificate: "/etc/filebeat/certs/filebeat.pem"
ssl.key: "/etc/filebeat/certs/filebeat.key"
setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.ilm.overwrite: true
setup.ilm.enabled: false
filebeat.modules:
- module: wazuh
alerts:
enabled: true
archives:
enabled: false
Kibana.yml
server.host: xxx.xxx.32.33
server.port: 443
elasticsearch.ssl.verificationMode: certificate
elasticsearch.username: kibanaserver
elasticsearch.password: kibanaserver
elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization"]
opendistro_security.multitenancy.enabled: true
opendistro_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: true
server.ssl.key: "/etc/kibana/certs/kibana.key"
server.ssl.certificate: "/etc/kibana/certs/kibana.pem"
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/root-ca.pem"]
server.defaultRoute: /app/wazuh
systemctl status kibana
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-03-29 16:18:11 CDT; 14h ago
Main PID: 1195 (node)
Tasks: 11 (limit: 43202)
Memory: 168.9M
CGroup: /system.slice/kibana.service
└─1195 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist -c /etc/kibana/kibana.yml
Mar 30 06:47:16 secon kibana[1195]: {"type":"log","@timestamp":"2021-03-30T11:47:16Z","tags":["error","elasticsearch","data"],"pid":1195,"message":"[ResponseError]: Response Error"}
Mar 30 06:47:19 secon kibana[1195]: {"type":"log","@timestamp":"2021-03-30T11:47:19Z","tags":["error","elasticsearch","data"],"pid":1195,"message":"[ResponseError]: Response Error"}
Mar 30 06:47:21 secon kibana[1195]: {"type":"log","@timestamp":"2021-03-30T11:47:21Z","tags":["error","elasticsearch","data"],"pid":1195,"message":"[ResponseError]: Response Error"}
Mar 30 06:47:24 secon kibana[1195]: {"type":"log","@timestamp":"2021-03-30T11:47:24Z","tags":["error","elasticsearch","data"],"pid":1195,"message":"[ResponseError]: Response Error"}
Mar 30 06:47:26 secon kibana[1195]: {"type":"log","@timestamp":"2021-03-30T11:47:26Z","tags":["error","elasticsearch","data"],"pid":1195,"message":"[ResponseError]: Response Error"}
Mar 30 06:47:29 secon kibana[1195]: {"type":"log","@timestamp":"2021-03-30T11:47:29Z","tags":["error","elasticsearch","data"],"pid":1195,"message":"[ResponseError]: Response Error"}
Mar 30 06:47:31 secon kibana[1195]: {"type":"log","@timestamp":"2021-03-30T11:47:31Z","tags":["error","elasticsearch","data"],"pid":1195,"message":"[ResponseError]: Response Error"}
Mar 30 06:47:34 secon kibana[1195]: {"type":"log","@timestamp":"2021-03-30T11:47:34Z","tags":["error","elasticsearch","data"],"pid":1195,"message":"[ResponseError]: Response Error"}
Mar 30 06:47:36 secon kibana[1195]: {"type":"log","@timestamp":"2021-03-30T11:47:36Z","tags":["error","elasticsearch","data"],"pid":1195,"message":"[ResponseError]: Response Error"}
Mar 30 06:47:39 secon kibana[1195]: {"type":"log","@timestamp":"2021-03-30T11:47:39Z","tags":["error","elasticsearch","data"],"pid":1195,"message":"[ResponseError]: Response Error"}
Mar 30 06:47:41 secon kibana[1195]: {"type":"log","@timestamp":"2021-03-30T11:47:41Z","tags":["error","elasticsearch","data"],"pid":1195,"message":"[ResponseError]: Response Error"}
Mar 30 06:47:44 secon kibana[1195]: {"type":"log","@timestamp":"2021-03-30T11:47:44Z","tags":["error","elasticsearch","data"],"pid":1195,"message":"[ResponseError]: Response Er
root@secon:/# filebeat test output
elasticsearch: https://xxx.xxx.32.33:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 150.201.32.33
dial up... OK
TLS...
security... WARN server's certificate chain verification is disabled
handshake... OK
TLS version: TLSv1.3
dial up... OK
talk to server... OK
version: 7.10.0
root@secon:/#
root@secon:/etc/kibana# systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-03-29 16:18:33 CDT; 14h ago
Main PID: 1701 (filebeat)
Tasks: 19 (limit: 43202)
Memory: 76.3M
CGroup: /system.slice/filebeat.service
└─1701 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat
Mar 30 06:44:10 secon filebeat[1701]: 2021-03-30T06:44:10.209-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":163201919}},"memory":{"mem":{"usage":{"bytes":8192}}}},"cpu":{"system":{"tick>
Mar 30 06:44:40 secon filebeat[1701]: 2021-03-30T06:44:40.210-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":191861224}},"memory":{"mem":{"usage":{"bytes":12288}}}},"cpu":{"system":{"tic>
Mar 30 06:45:10 secon filebeat[1701]: 2021-03-30T06:45:10.209-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":204897306}},"memory":{"mem":{"usage":{"bytes":8192}}}},"cpu":{"system":{"tick>
Mar 30 06:45:40 secon filebeat[1701]: 2021-03-30T06:45:40.210-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":177108874}},"memory":{"mem":{"usage":{"bytes":8192}}}},"cpu":{"system":{"tick>
Mar 30 06:46:10 secon filebeat[1701]: 2021-03-30T06:46:10.209-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":180737381}},"memory":{"mem":{"usage":{"bytes":139264}}}},"cpu":{"system":{"ti>
Mar 30 06:46:40 secon filebeat[1701]: 2021-03-30T06:46:40.209-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":229196684}},"memory":{"mem":{"usage":{"bytes":-114688}}}},"cpu":{"system":{"t>
Mar 30 06:47:10 secon filebeat[1701]: 2021-03-30T06:47:10.210-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":255212518}},"memory":{"mem":{"usage":{"bytes":8192}}}},"cpu":{"system":{"tick>
Mar 30 06:47:40 secon filebeat[1701]: 2021-03-30T06:47:40.209-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":176423137}},"memory":{"mem":{"usage":{"bytes":8192}}}},"cpu":{"system":{"tick>
Mar 30 06:48:10 secon filebeat[1701]: 2021-03-30T06:48:10.209-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":144247987}},"memory":{"mem":{"usage":{"bytes":8192}}}},"cpu":{"system":{"tick>
Mar 30 06:48:40 secon filebeat[1701]: 2021-03-30T06:48:40.210-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":166625309}},"memory":{"mem":{"usage":{"bytes":8192}}}},"cpu":{"system":{"tick>
lines 1-20/20 (END)
root@secon:/etc/kibana# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-03-29 16:19:16 CDT; 14h ago
Main PID: 1700 (java)
Tasks: 171 (limit: 43202)
Memory: 21.1G
CGroup: /system.slice/elasticsearch.service
└─1700 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.net>
Mar 29 16:18:33 secon systemd[1]: Starting Elasticsearch...
Mar 29 16:19:15 secon systemd-entrypoint[1700]: WARNING: An illegal reflective access operation has occurred
Mar 29 16:19:15 secon systemd-entrypoint[1700]: WARNING: Illegal reflective access by com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.MasterServiceEventMetrics (file:/usr/share/elasticsearch/plugins/opendistro_performance_analyzer/opendistro_performance_analyzer-1.12.0.0.jar) to field java.util.c>
Mar 29 16:19:15 secon systemd-entrypoint[1700]: WARNING: Please consider reporting this to the maintainers of com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.MasterServiceEventMetrics
Mar 29 16:19:15 secon systemd-entrypoint[1700]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Mar 29 16:19:15 secon systemd-entrypoint[1700]: WARNING: All illegal access operations will be denied in a future release
Mar 29 16:19:16 secon systemd[1]: Started Elasticsearch.
lines 1-17/17 (END)
root@secon:/var/ossec/logs# tail -f ossec.log
2021/03/30 04:19:44 ossec-syscheckd: INFO: (6008): File integrity monitoring scan started.
2021/03/30 04:19:52 ossec-syscheckd: INFO: (6009): File integrity monitoring scan ended.
2021/03/30 04:21:59 rootcheck: INFO: Starting rootcheck scan.
2021/03/30 04:23:08 rootcheck: INFO: Ending rootcheck scan.
2021/03/30 05:13:09 ossec-analysisd: ERROR: The new permissions could not be added to the JSON alert.
2021/03/30 05:13:09 ossec-analysisd: ERROR: The new permissions could not be added to the JSON alert.
2021/03/30 05:18:52 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2021/03/30 05:19:10 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2021/03/30 06:18:52 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2021/03/30 06:19:11 wazuh-modulesd:syscollector: INFO: Evaluation finished.
root@secon:/var/log/elasticsearch# tail -f elasticsearch.log
EOE
[2021-03-30T06:53:49,314][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58472
[2021-03-30T06:53:51,816][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58474
[2021-03-30T06:53:54,319][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58472
[2021-03-30T06:53:56,820][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58474
[2021-03-30T06:53:59,322][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58472
[2021-03-30T06:54:01,827][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58474
[2021-03-30T06:54:04,330][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58472
[2021-03-30T06:54:06,828][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58474
[2021-03-30T06:54:09,328][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58472
[2021-03-30T06:54:11,831][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58474
[2021-03-30T06:54:14,332][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58472
[2021-03-30T06:54:15,049][INFO ][c.a.o.j.s.JobSweeper ] [node-1] Running full sweep
[2021-03-30T06:54:16,836][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58474
[2021-03-30T06:54:19,338][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58472
[2021-03-30T06:54:21,839][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58474
[2021-03-30T06:54:24,341][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58472
[2021-03-30T06:54:26,843][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58474
[2021-03-30T06:54:29,363][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58472
[2021-03-30T06:54:31,845][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58474
[2021-03-30T06:54:34,350][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58472
[2021-03-30T06:54:36,850][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58474
[2021-03-30T06:54:39,351][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58472
[2021-03-30T06:54:41,853][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58474
[2021-03-30T06:54:44,356][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58472
[2021-03-30T06:54:46,856][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58474
[2021-03-30T06:54:47,722][INFO ][stats_log ] [node-1] ------------------------------------------------------------------------
Program=PerformanceAnalyzerPlugin
rca-version=0.0.1
StartTime=1617105227.714
EndTime=Tue, 30 Mar 2021 06:54:47 CDT
Time=60008 msecs
Timing=total-time:60008.0/1
Counters=TotalError=0
EOE
[2021-03-30T06:54:47,722][INFO ][stats_log ] [node-1] ------------------------------------------------------------------------
Program=PerformanceAnalyzerPlugin
rca-version=0.0.1
Metrics=
StartTime=0.000
EndTime=Tue, 30 Mar 2021 06:54:47 CDT
Time=1617105287722 msecs
Timing=total-time:1.617105287722E12/1
Counters=
EOE
[2021-03-30T06:54:47,722][INFO ][stats_log ] [node-1] ------------------------------------------------------------------------
Program=PerformanceAnalyzerPlugin
rca-version=0.0.1
Metrics=
StartTime=0.000
EndTime=Tue, 30 Mar 2021 06:54:47 CDT
Time=1617105287722 msecs
Timing=total-time:1.617105287722E12/1
Counters=
EOE
[2021-03-30T06:54:47,722][INFO ][stats_log ] [node-1] ------------------------------------------------------------------------
Program=PerformanceAnalyzerPlugin
rca-version=0.0.1
Metrics=
StartTime=0.000
EndTime=Tue, 30 Mar 2021 06:54:47 CDT
Time=1617105287722 msecs
Timing=total-time:1.617105287722E12/1
Counters=
EOE
[2021-03-30T06:54:47,722][INFO ][stats_log ] [node-1] ------------------------------------------------------------------------
Program=PerformanceAnalyzerPlugin
rca-version=0.0.1
Metrics=
StartTime=0.000
EndTime=Tue, 30 Mar 2021 06:54:47 CDT
Time=1617105287722 msecs
Timing=total-time:1.617105287722E12/1
Counters=
EOE
[2021-03-30T06:54:47,722][INFO ][stats_log ] [node-1] ------------------------------------------------------------------------
Program=PerformanceAnalyzerPlugin
rca-version=0.0.1
Metrics=
StartTime=0.000
EndTime=Tue, 30 Mar 2021 06:54:47 CDT
Time=1617105287722 msecs
Timing=total-time:1.617105287722E12/1
Counters=
EOE
[2021-03-30T06:54:47,722][INFO ][stats_log ] [node-1] ------------------------------------------------------------------------
Program=PerformanceAnalyzerPlugin
rca-version=0.0.1
Metrics=
StartTime=0.000
EndTime=Tue, 30 Mar 2021 06:54:47 CDT
Time=1617105287722 msecs
Timing=total-time:1.617105287722E12/1
Counters=
EOE
[2021-03-30T06:54:47,723][INFO ][stats_log ] [node-1] ------------------------------------------------------------------------
Program=PerformanceAnalyzerPlugin
rca-version=0.0.1
Metrics=
StartTime=0.000
EndTime=Tue, 30 Mar 2021 06:54:47 CDT
Time=1617105287722 msecs
Timing=total-time:1.617105287722E12/1
Counters=
EOE
[2021-03-30T06:54:49,360][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58472
[2021-03-30T06:54:51,861][WARN ][c.a.o.s.a.BackendRegistry] [node-1] Authentication finally failed for kibanaserver from xxx.xxx.32.33:58474
[
root@secon:/var/log/filebeat# tail -f filebeat
2021-03-29T09:17:31.654-0500 INFO instance/beat.go:645 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2021-03-29T09:17:31.654-0500 INFO instance/beat.go:653 Beat ID: 76cfb8ed-490c-4047-9a2d-e769f1acb967
2021-03-29T09:17:31.654-0500 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.10.0' as ILM is enabled.
2021-03-29T09:17:31.655-0500 WARN [tls] tlscommon/tls_config.go:93 SSL/TLS verifications disabled.
2021-03-29T09:17:31.656-0500 WARN [tls] tlscommon/tls_config.go:93 SSL/TLS verifications disabled.
2021-03-29T09:17:31.683-0500 WARN [tls] tlscommon/tls_config.go:93 SSL/TLS verifications disabled.
2021-03-29T09:17:31.714-0500 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.0
root@secon:/usr/share/kibana/data/wazuh/logs# tail -f wazuhapp.log
{"date":"2021-03-29T13:30:01.259Z","level":"info","location":"Cron-scheduler","message":{"name":"ResponseError","meta":{"body":"Unauthorized","statusCode":401,"headers":{"www-authenticate":"Basic realm=\"Open Distro Security\"","content-type":"text/plain; charset=UTF-8","content-length":"12"},"meta":{"context":null,"request":{"params":{"method":"POST","path":"/wazuh-statistics-2021.13w/_doc/_bulk","bulkBody":[{"index":{"_index":"wazuh-statistics-2021.13w"}},{"analysisd":[{"total_events_decoded":131,"syscheck_events_decoded":0,"syscheck_edps":0,"syscollector_events_decoded":0,"syscollector_edps":0,"rootcheck_events_decoded":0,"rootcheck_edps":0,"sca_events_decoded":0,"sca_edps":0,"hostinfo_events_decoded":0,"hostinfo_edps":0,"winevt_events_decoded":97,"winevt_edps":19,"dbsync_messages_dispatched":0,"dbsync_mdps":0,"other_events_decoded":34,"other_events_edps":6,"events_processed":131,"events_edps":26,"events_received":131,"events_dropped":0,"alerts_written":86,"firewall_written":0,"fts_written":0,"syscheck_queue_usage":0,"syscheck_queue_size":16384,"syscollector_queue_usage":0,"syscollector_queue_size":16384,"rootcheck_queue_usage":0,"rootcheck_queue_size":16384,"sca_queue_usage":0,"sca_queue_size":16384,"hostinfo_queue_usage":0,"hostinfo_queue_size":16384,"winevt_queue_usage":0,"winevt_queue_size":16384,"dbsync_queue_usage":0,"dbsync_queue_size":16384,"upgrade_queue_usage":0,"upgrade_queue_size":16384,"event_queue_usage":0,"event_queue_size":16384,"rule_matching_queue_usage":0,"rule_matching_queue_size":16384,"alerts_queue_usage":0,"alerts_queue_size":16384,"firewall_queue_usage":0,"firewall_queue_size":16384,"statistical_queue_usage":0,"statistical_queue_size":16384,"archives_queue_usage":0,"archives_queue_size":16384}],"apiName":"production","cluster":"false","timestamp":"2021-03-29T13:30:00.896Z"}],"querystring":"","body":"{\"index\":{\"_index\":\"wazuh-statistics-2021.13w\"}}\n{\"analysisd\":[{\"total_events_decoded\":131,\"syscheck_events_decoded\":0,\"syscheck_edps\":0,\"syscollector_events_decoded\":0,\"syscollector_edps\":0,\"rootcheck_events_decoded\":0,\"rootcheck_edps\":0,\"sca_events_decoded\":0,\"sca_edps\":0,\"hostinfo_events_decoded\":0,\"hostinfo_edps\":0,\"winevt_events_decoded\":97,\"winevt_edps\":19,\"dbsync_messages_dispatched\":0,\"dbsync_mdps\":0,\"other_events_decoded\":34,\"other_events_edps\":6,\"events_processed\":131,\"events_edps\":26,\"events_received\":131,\"events_dropped\":0,\"alerts_written\":86,\"firewall_written\":0,\"fts_written\":0,\"syscheck_queue_usage\":0,\"syscheck_queue_size\":16384,\"syscollector_queue_usage\":0,\"syscollector_queue_size\":16384,\"rootcheck_queue_usage\":0,\"rootcheck_queue_size\":16384,\"sca_queue_usage\":0,\"sca_queue_size\":16384,\"hostinfo_queue_usage\":0,\"hostinfo_queue_size\":16384,\"winevt_queue_usage\":0,\"winevt_queue_size\":16384,\"dbsync_queue_usage\":0,\"dbsync_queue_size\":16384,\"upgrade_queue_usage\":0,\"upgrade_queue_size\":16384,\"event_queue_usage\":0,\"event_queue_size\":16384,\"rule_matching_queue_usage\":0,\"rule_matching_queue_size\":16384,\"alerts_queue_usage\":0,\"alerts_queue_size\":16384,\"firewall_queue_usage\":0,\"firewall_queue_size\":16384,\"statistical_queue_usage\":0,\"statistical_queue_size\":16384,\"archives_queue_usage\":0,\"archives_queue_size\":16384}],\"apiName\":\"production\",\"cluster\":\"false\",\"timestamp\":\"2021-03-29T13:30:00.896Z\"}\n","headers":{"user-agent":"elasticsearch-js/7.10.0-rc.1 (linux 5.4.0-66-generic-x64; Node.js v10.22.1)","x-elastic-product-origin":"kibana","content-type":"application/x-ndjson","content-length":"1419"},"timeout":30000},"options":{},"id":399540},"name":"elasticsearch-js","connection":{"url":"
https://xxx.xxx.32.33:9200/","id":"
https://xxx.xxx.32.33:9200/","headers":{},"deadCount":0,"resurrectTimeout":0,"_openRequests":1,"status":"alive","roles":{"master":true,"data":true,"ingest":true,"ml":false}},"attempts":0,"aborted":false}}}}
{"date":"2021-03-29T13:30:01.604Z","level":"info","location":"Cron-scheduler","message":{"name":"ResponseError","meta":{"body":"Unauthorized","statusCode":401,"headers":{"www-authenticate":"Basic realm=\"Open Distro Security\"","content-type":"text/plain; charset=UTF-8","content-length":"12"},"meta":{"context":null,"request":{"params":{"method":"POST","path":"/wazuh-statistics-2021.13w/_doc/_bulk","bulkBody":[{"index":{"_index":"wazuh-statistics-2021.13w"}},{"remoted":[{"queue_size":0,"total_queue_size":131072,"tcp_sessions":64,"evt_count":10908168,"ctrl_msg_count":2167212,"discarded_count":0,"msg_sent":2176172,"recv_bytes":9488408187,"dequeued_after_close":0}],"apiName":"production","cluster":"false","timestamp":"2021-03-29T13:30:01.257Z"}],"querystring":"","body":"{\"index\":{\"_index\":\"wazuh-statistics-2021.13w\"}}\n{\"remoted\":[{\"queue_size\":0,\"total_queue_size\":131072,\"tcp_sessions\":64,\"evt_count\":10908168,\"ctrl_msg_count\":2167212,\"discarded_count\":0,\"msg_sent\":2176172,\"recv_bytes\":9488408187,\"dequeued_after_close\":0}],\"apiName\":\"production\",\"cluster\":\"false\",\"timestamp\":\"2021-03-29T13:30:01.257Z\"}\n","headers":{"user-agent":"elasticsearch-js/7.10.0-rc.1 (linux 5.4.0-66-generic-x64; Node.js v10.22.1)","x-elastic-product-origin":"kibana","content-type":"application/x-ndjson","content-length":"338"},"timeout":30000},"options":{},"id":399541},"name":"elasticsearch-js","connection":{"url":"
https://xxx.xxx.32.33:9200/","id":"
https://xxx.xxx.32.33:9200/","headers":{},"deadCount":0,"resurrectTimeout":0,"_openRequests":0,"status":"alive","roles":{"master":true,"data":true,"ingest":true,"ml":false}},"attempts":0,"aborted":false}}}}
{"date":"2021-03-29T13:35:01.029Z","level":"error","location":"cron-scheduler|SaveDocument","message":"Error searching or creating 'wazuh-statistics-2021.13w' due to 'Response Error'"}
{"date":"2021-03-29T13:35:01.031Z","level":"error","location":"cron-scheduler|SaveDocument","message":"Error searching or creating 'wazuh-statistics-2021.13w' due to 'Response Error'"}
{"date":"2021-03-29T13:35:01.377Z","level":"info","location":"Cron-scheduler","message":{"name":"ResponseError","meta":{"body":"Unauthorized","statusCode":401,"headers":{"www-authenticate":"Basic realm=\"Open Distro Security\"","content-type":"text/plain; charset=UTF-8","content-length":"12"},"meta":{"context":null,"request":{"params":{"method":"POST","path":"/wazuh-statistics-2021.13w/_doc/_bulk","bulkBody":[{"index":{"_index":"wazuh-statistics-2021.13w"}},{"analysisd":[{"total_events_decoded":148,"syscheck_events_decoded":0,"syscheck_edps":0,"syscollector_events_decoded":0,"syscollector_edps":0,"rootcheck_events_decoded":0,"rootcheck_edps":0,"sca_events_decoded":0,"sca_edps":0,"hostinfo_events_decoded":0,"hostinfo_edps":0,"winevt_events_decoded":115,"winevt_edps":23,"dbsync_messages_dispatched":0,"dbsync_mdps":0,"other_events_decoded":33,"other_events_edps":6,"events_processed":148,"events_edps":29,"events_received":148,"events_dropped":0,"alerts_written":90,"firewall_written":0,"fts_written":0,"syscheck_queue_usage":0,"syscheck_queue_size":16384,"syscollector_queue_usage":0,"syscollector_queue_size":16384,"rootcheck_queue_usage":0,"rootcheck_queue_size":16384,"sca_queue_usage":0,"sca_queue_size":16384,"hostinfo_queue_usage":0,"hostinfo_queue_size":16384,"winevt_queue_usage":0,"winevt_queue_size":16384,"dbsync_queue_usage":0,"dbsync_queue_size":16384,"upgrade_queue_usage":0,"upgrade_queue_size":16384,"event_queue_usage":0,"event_queue_size":16384,"rule_matching_queue_usage":0,"rule_matching_queue_size":16384,"alerts_queue_usage":0,"alerts_queue_size":16384,"firewall_queue_usage":0,"firewall_queue_size":16384,"statistical_queue_usage":0,"statistical_queue_size":16384,"archives_queue_usage":0,"archives_queue_size":16384}],"apiName":"production","cluster":"false","timestamp":"2021-03-29T13:35:01.030Z"}],"querystring":"","body":"{\"index\":{\"_index\":\"wazuh-statistics-2021.13w\"}}\n{\"analysisd\":[{\"total_events_decoded\":148,\"syscheck_events_decoded\":0,\"syscheck_edps\":0,\"syscollector_events_decoded\":0,\"syscollector_edps\":0,\"rootcheck_events_decoded\":0,\"rootcheck_edps\":0,\"sca_events_decoded\":0,\"sca_edps\":0,\"hostinfo_events_decoded\":0,\"hostinfo_edps\":0,\"winevt_events_decoded\":115,\"winevt_edps\":23,\"dbsync_messages_dispatched\":0,\"dbsync_mdps\":0,\"other_events_decoded\":33,\"other_events_edps\":6,\"events_processed\":148,\"events_edps\":29,\"events_received\":148,\"events_dropped\":0,\"alerts_written\":90,\"firewall_written\":0,\"fts_written\":0,\"syscheck_queue_usage\":0,\"syscheck_queue_size\":16384,\"syscollector_queue_usage\":0,\"syscollector_queue_size\":16384,\"rootcheck_queue_usage\":0,\"rootcheck_queue_size\":16384,\"sca_queue_usage\":0,\"sca_queue_size\":16384,\"hostinfo_queue_usage\":0,\"hostinfo_queue_size\":16384,\"winevt_queue_usage\":0,\"winevt_queue_size\":16384,\"dbsync_queue_usage\":0,\"dbsync_queue_size\":16384,\"upgrade_queue_usage\":0,\"upgrade_queue_size\":16384,\"event_queue_usage\":0,\"event_queue_size\":16384,\"rule_matching_queue_usage\":0,\"rule_matching_queue_size\":16384,\"alerts_queue_usage\":0,\"alerts_queue_size\":16384,\"firewall_queue_usage\":0,\"firewall_queue_size\":16384,\"statistical_queue_usage\":0,\"statistical_queue_size\":16384,\"archives_queue_usage\":0,\"archives_queue_size\":16384}],\"apiName\":\"production\",\"cluster\":\"false\",\"timestamp\":\"2021-03-29T13:35:01.030Z\"}\n","headers":{"user-agent":"elasticsearch-js/7.10.0-rc.1 (linux 5.4.0-66-generic-x64; Node.js v10.22.1)","x-elastic-product-origin":"kibana","content-type":"application/x-ndjson","content-length":"1420"},"timeout":30000},"options":{},"id":399667},"name":"elasticsearch-js","connection":{"url":"
https://xxx.xxx.32.33:9200/","id":"
https://xxx.xxx.32.33:9200/","headers":{},"deadCount":0,"resurrectTimeout":0,"_openRequests":1,"status":"alive","roles":{"master":true,"data":true,"ingest":true,"ml":false}},"attempts":0,"aborted":false}}}}
{"date":"2021-03-29T13:35:01.382Z","level":"info","location":"Cron-scheduler","message":{"name":"ResponseError","meta":{"body":"Unauthorized","statusCode":401,"headers":{"www-authenticate":"Basic realm=\"Open Distro Security\"","content-type":"text/plain; charset=UTF-8","content-length":"12"},"meta":{"context":null,"request":{"params":{"method":"POST","path":"/wazuh-statistics-2021.13w/_doc/_bulk","bulkBody":[{"index":{"_index":"wazuh-statistics-2021.13w"}},{"remoted":[{"queue_size":0,"total_queue_size":131072,"tcp_sessions":64,"evt_count":10915051,"ctrl_msg_count":2169131,"discarded_count":0,"msg_sent":2178106,"recv_bytes":9494792871,"dequeued_after_close":0}],"apiName":"production","cluster":"false","timestamp":"2021-03-29T13:35:01.031Z"}],"querystring":"","body":"{\"index\":{\"_index\":\"wazuh-statistics-2021.13w\"}}\n{\"remoted\":[{\"queue_size\":0,\"total_queue_size\":131072,\"tcp_sessions\":64,\"evt_count\":10915051,\"ctrl_msg_count\":2169131,\"discarded_count\":0,\"msg_sent\":2178106,\"recv_bytes\":9494792871,\"dequeued_after_close\":0}],\"apiName\":\"production\",\"cluster\":\"false\",\"timestamp\":\"2021-03-29T13:35:01.031Z\"}\n","headers":{"user-agent":"elasticsearch-js/7.10.0-rc.1 (linux 5.4.0-66-generic-x64; Node.js v10.22.1)","x-elastic-product-origin":"kibana","content-type":"application/x-ndjson","content-length":"338"},"timeout":30000},"options":{},"id":399668},"name":"elasticsearch-js","connection":{"url":"
https://xxx.xxx.32.33:9200/","id":"
https://xxx.xxx.32.33:9200/","headers":{},"deadCount":0,"resurrectTimeout":0,"_openRequests":0,"status":"alive","roles":{"master":true,"data":true,"ingest":true,"ml":false}},"attempts":0,"aborted":false}}}}
{"date":"2021-03-29T13:40:01.469Z","level":"error","location":"cron-scheduler|SaveDocument","message":"Error searching or creating 'wazuh-statistics-2021.13w' due to 'Response Error'"}
{"date":"2021-03-29T13:40:01.470Z","level":"error","location":"cron-scheduler|SaveDocument","message":"Error searching or creating 'wazuh-statistics-2021.13w' due to 'Response Error'"}
{"date":"2021-03-29T13:40:01.816Z","level":"info","location":"Cron-scheduler","message":{"name":"ResponseError","meta":{"body":"Unauthorized","statusCode":401,"headers":{"www-authenticate":"Basic realm=\"Open Distro Security\"","content-type":"text/plain; charset=UTF-8","content-length":"12"},"meta":{"context":null,"request":{"params":{"method":"POST","path":"/wazuh-statistics-2021.13w/_doc/_bulk","bulkBody":[{"index":{"_index":"wazuh-statistics-2021.13w"}},{"remoted":[{"queue_size":0,"total_queue_size":131072,"tcp_sessions":64,"evt_count":10922173,"ctrl_msg_count":2171048,"discarded_count":0,"msg_sent":2180023,"recv_bytes":9501273747,"dequeued_after_close":0}],"apiName":"production","cluster":"false","timestamp":"2021-03-29T13:40:01.469Z"}],"querystring":"","body":"{\"index\":{\"_index\":\"wazuh-statistics-2021.13w\"}}\n{\"remoted\":[{\"queue_size\":0,\"total_queue_size\":131072,\"tcp_sessions\":64,\"evt_count\":10922173,\"ctrl_msg_count\":2171048,\"discarded_count\":0,\"msg_sent\":2180023,\"recv_bytes\":9501273747,\"dequeued_after_close\":0}],\"apiName\":\"production\",\"cluster\":\"false\",\"timestamp\":\"2021-03-29T13:40:01.469Z\"}\n","headers":{"user-agent":"elasticsearch-js/7.10.0-rc.1 (linux 5.4.0-66-generic-x64; Node.js v10.22.1)","x-elastic-product-origin":"kibana","content-type":"application/x-ndjson","content-length":"338"},"timeout":30000},"options":{},"id":399796},"name":"elasticsearch-js","connection":{"url":"
https://xxx.xxx.32.33:9200/","id":"
https://xxx.xxx.32.33:9200/","headers":{},"deadCount":0,"resurrectTimeout":0,"_openRequests":1,"status":"alive","roles":{"master":true,"data":true,"ingest":true,"ml":false}},"attempts":0,"aborted":false}}}}
{"date":"2021-03-29T13:40:01.820Z","level":"info","location":"Cron-scheduler","message":{"name":"ResponseError","meta":{"body":"Unauthorized","statusCode":401,"headers":{"www-authenticate":"Basic realm=\"Open Distro Security\"","content-type":"text/plain; charset=UTF-8","content-length":"12"},"meta":{"context":null,"request":{"params":{"method":"POST","path":"/wazuh-statistics-2021.13w/_doc/_bulk","bulkBody":[{"index":{"_index":"wazuh-statistics-2021.13w"}},{"analysisd":[{"total_events_decoded":114,"syscheck_events_decoded":0,"syscheck_edps":0,"syscollector_events_decoded":0,"syscollector_edps":0,"rootcheck_events_decoded":0,"rootcheck_edps":0,"sca_events_decoded":0,"sca_edps":0,"hostinfo_events_decoded":0,"hostinfo_edps":0,"winevt_events_decoded":72,"winevt_edps":14,"dbsync_messages_dispatched":0,"dbsync_mdps":0,"other_events_decoded":42,"other_events_edps":8,"events_processed":114,"events_edps":22,"events_received":114,"events_dropped":0,"alerts_written":56,"firewall_written":0,"fts_written":0,"syscheck_queue_usage":0,"syscheck_queue_size":16384,"syscollector_queue_usage":0,"syscollector_queue_size":16384,"rootcheck_queue_usage":0,"rootcheck_queue_size":16384,"sca_queue_usage":0,"sca_queue_size":16384,"hostinfo_queue_usage":0,"hostinfo_queue_size":16384,"winevt_queue_usage":0,"winevt_queue_size":16384,"dbsync_queue_usage":0,"dbsync_queue_size":16384,"upgrade_queue_usage":0,"upgrade_queue_size":16384,"event_queue_usage":0,"event_queue_size":16384,"rule_matching_queue_usage":0,"rule_matching_queue_size":16384,"alerts_queue_usage":0,"alerts_queue_size":16384,"firewall_queue_usage":0,"firewall_queue_size":16384,"statistical_queue_usage":0,"statistical_queue_size":16384,"archives_queue_usage":0,"archives_queue_size":16384}],"apiName":"production","cluster":"false","timestamp":"2021-03-29T13:40:01.471Z"}],"querystring":"","body":"{\"index\":{\"_index\":\"wazuh-statistics-2021.13w\"}}\n{\"analysisd\":[{\"total_events_decoded\":114,\"syscheck_events_decoded\":0,\"syscheck_edps\":0,\"syscollector_events_decoded\":0,\"syscollector_edps\":0,\"rootcheck_events_decoded\":0,\"rootcheck_edps\":0,\"sca_events_decoded\":0,\"sca_edps\":0,\"hostinfo_events_decoded\":0,\"hostinfo_edps\":0,\"winevt_events_decoded\":72,\"winevt_edps\":14,\"dbsync_messages_dispatched\":0,\"dbsync_mdps\":0,\"other_events_decoded\":42,\"other_events_edps\":8,\"events_processed\":114,\"events_edps\":22,\"events_received\":114,\"events_dropped\":0,\"alerts_written\":56,\"firewall_written\":0,\"fts_written\":0,\"syscheck_queue_usage\":0,\"syscheck_queue_size\":16384,\"syscollector_queue_usage\":0,\"syscollector_queue_size\":16384,\"rootcheck_queue_usage\":0,\"rootcheck_queue_size\":16384,\"sca_queue_usage\":0,\"sca_queue_size\":16384,\"hostinfo_queue_usage\":0,\"hostinfo_queue_size\":16384,\"winevt_queue_usage\":0,\"winevt_queue_size\":16384,\"dbsync_queue_usage\":0,\"dbsync_queue_size\":16384,\"upgrade_queue_usage\":0,\"upgrade_queue_size\":16384,\"event_queue_usage\":0,\"event_queue_size\":16384,\"rule_matching_queue_usage\":0,\"rule_matching_queue_size\":16384,\"alerts_queue_usage\":0,\"alerts_queue_size\":16384,\"firewall_queue_usage\":0,\"firewall_queue_size\":16384,\"statistical_queue_usage\":0,\"statistical_queue_size\":16384,\"archives_queue_usage\":0,\"archives_queue_size\":16384}],\"apiName\":\"production\",\"cluster\":\"false\",\"timestamp\":\"2021-03-29T13:40:01.471Z\"}\n","headers":{"user-agent":"elasticsearch-js/7.10.0-rc.1 (linux 5.4.0-66-generic-x64; Node.js v10.22.1)","x-elastic-product-origin":"kibana","content-type":"application/x-ndjson","content-length":"1419"},"timeout":30000},"options":{},"id":399797},"name":"elasticsearch-js","connection":{"url":"
https://xxx.xxx.32.33:9200/","id":"
https://xxx.xxx.32.33:9200/","headers":{},"deadCount":0,"resurrectTimeout":0,"_openRequests":0,"status":"alive","roles":{"master":true,"data":true,"ingest":true,"ml":false}},"attempts":0,"aborted":false}}}}