Elasticsearch ALL SHARDS FAILED

10,937 views
Skip to first unread message

Alejandro Martinez

unread,
Mar 16, 2020, 6:29:41 AM3/16/20
to Wazuh mailing list
Hello, I had some indexing problems in kibana, so I aliased them and the next problem appears when I restart kibana, I get this error message:

Unable to connect to Elasticsearch. Error: [search_phase_execution_exception] all shards failed

The elasticsearch log is:

[2020-03-16T10:28:18,970][DEBUG][o.e.a.s.TransportSearchAction] [elastic] All shards failed for phase: [query]
[2020-03-16T10:28:18,971][WARN ][r.suppressed             ] [elastic] path: /.kibana/_count, params: {index=.kibana}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.0.jar:7.6.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.0.jar:7.6.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.0.jar:7.6.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.0.jar:7.6.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.0.jar:7.6.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.0.jar:7.6.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.0.jar:7.6.0]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.0.jar:7.6.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.0.jar:7.6.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.0.jar:7.6.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]

Pablo Torres

unread,
Mar 16, 2020, 7:36:39 AM3/16/20
to Wazuh mailing list
Hi Alejandro,

What indexing problems you had and how did you solve those problems?
Could you please share with us your Elasticsearch and Kibana logs?
cat /var/log/elasticsearch/elasticsearch.log
and
systemctl status kibana -l -n 150 | grep -i -E "(error|warning)"

Let's also check the Elasticsearch cluster health and the allocation status:
Cluster health
curl elastic_ip:9200/_cluster/health?pretty

Cluster allocation
curl elastic_ip:9200/_cluster/allocation/explain?pretty

curl -XGET elastic_ip:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason,node | grep -i unassigned

Please run all these requests (don't forget to replace elastic_ip with the IP address of your Elasticsearch server) and paste the output in your reply.

Best Regards,
Pablo Torres

Alejandro Martinez

unread,
Mar 16, 2020, 8:44:39 AM3/16/20
to Wazuh mailing list
Each file contains the output of the commands.
allocation.log
elastic.log
elasticsearchHealth.log
kibana.log
unallocated.log

Pablo Torres

unread,
Mar 16, 2020, 11:48:29 AM3/16/20
to Wazuh mailing list
Hi Alejandro,

Thanks for the output, I can see different errors in those files:

[2020-03-16T10:33:39,887][WARN ][o.e.i.c.IndicesClusterStateService] [elastic] [.kibana_task_manager_1][0] marking and sending shard failed due to [failed to create shard] java.nio.file.AccessDeniedException: /var/lib/elasticsearch/nodes/0/indices/EyR4qfaATjaQPklgbron1g
It seems that there is a permission issue in that path so Elasticsearch cannot allocate the shards, its expected that the user that runs Elasticsearch can write in that directory.

Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_task_manager_1 and restarting Kibana.
Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting Kibana.
Kibana also mentions that there is another Kibana instance migrating the index, could you delete those 2 indices that are causing troubles and restart Kibana?

curl -XDELETE elastic_ip:9200/.kibana_*
curl -XDELETE elastic_ip:9200/.kibana_task_manager_*
systemctl restart kibana

Best Regards,
Pablo Torres


Alejandro Martinez

unread,
Mar 16, 2020, 12:23:03 PM3/16/20
to Wazuh mailing list
I erased both indexes, and I still get the same error.

● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-03-16 16:16:07 UTC; 1min 17s ago
 Main PID: 16243 (node)
   CGroup: /system.slice/kibana.service
           └─16243 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Mar 16 16:16:18 srvwaz.servidor.com kibana[16243]: {"type":"log","@timestamp":"2020-03-16T16:16:18Z","tags":["info","plugins","bfetch"],"pid":16243,"message":"Setting up plugin"}
Mar 16 16:16:18 srvwaz.servidor.com kibana[16243]: {"type":"log","@timestamp":"2020-03-16T16:16:18Z","tags":["info","savedobjects-service"],"pid":16243,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
Mar 16 16:16:18 srvwaz.servidor.com kibana[16243]: {"type":"log","@timestamp":"2020-03-16T16:16:18Z","tags":["info","savedobjects-service"],"pid":16243,"message":"Starting saved objects migrations"}
Mar 16 16:16:18 srvwaz.servidor.com kibana[16243]: {"type":"log","@timestamp":"2020-03-16T16:16:18Z","tags":["info","savedobjects-service"],"pid":16243,"message":"Creating index .kibana_1."}
Mar 16 16:16:18 srvwaz.servidor.com kibana[16243]: {"type":"log","@timestamp":"2020-03-16T16:16:18Z","tags":["info","savedobjects-service"],"pid":16243,"message":"Creating index .kibana_task_manager_1."}
Mar 16 16:16:48 srvwaz.servidor.com kibana[16243]: {"type":"log","@timestamp":"2020-03-16T16:16:48Z","tags":["warning","savedobjects-service"],"pid":16243,"message":"Unable to connect to Elasticsearch. Error: Request Timeout after 30000ms"}
Mar 16 16:16:51 srvwaz.servidor.com kibana[16243]: {"type":"log","@timestamp":"2020-03-16T16:16:51Z","tags":["warning","savedobjects-service"],"pid":16243,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_1/h3hApuwUTXqu2HTUhAYljQ] already exists, with { index_uuid=\"h3hApuwUTXqu2HTUhAYljQ\" & index=\".kibana_1\" }"}
Mar 16 16:16:51 srvwaz.servidor.com kibana[16243]: {"type":"log","@timestamp":"2020-03-16T16:16:51Z","tags":["warning","savedobjects-service"],"pid":16243,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting Kibana."}
Mar 16 16:16:51 srvwaz.servidor.com kibana[16243]: {"type":"log","@timestamp":"2020-03-16T16:16:51Z","tags":["warning","savedobjects-service"],"pid":16243,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_task_manager_1/BfCmOBOpSXaWKBzmrA2fsw] already exists, with { index_uuid=\"BfCmOBOpSXaWKBzmrA2fsw\" & index=\".kibana_task_manager_1\" }"}
Mar 16 16:16:51 srvwaz.servidor.com kibana[16243]: {"type":"log","@timestamp":"2020-03-16T16:16:51Z","tags":["warning","savedobjects-service"],"pid":16243,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_task_manager_1 and restarting Kibana."}

The ouput of the indices is this one:

curl --user elastic:H4rdPass.1035 --insecure https://localhost:9200/_cat/indices?pretty
green  open wazuh-alerts-3.x-2020.02.22     sZIfukfUSsWujxA8kdNzwQ 3 0 531331 0 235.9mb 235.9mb
green  open wazuh-monitoring-3.x-2020.02.24 buEqxi6wQ-SvtrdIOwzjhg 2 0    576 0 278.4kb 278.4kb
green  open wazuh-monitoring-3.x-2020.02.23 vXhZ5fTiQgaR516psBXQ1Q 2 0    576 0   317kb   317kb
green  open wazuh-alerts-3.x-2020.02.23     lKYigSvLQCiZVofVUCWswA 3 0 496041 0 217.3mb 217.3mb
green  open wazuh-monitoring-3.x-2020.02.22 Ie30N0qQRluZ2t_Zn5sosQ 2 0    570 0 265.1kb 265.1kb
green  open wazuh-alerts-3.x-2020.02.20     CQItb6vqRECKJYhpx_AEJw 3 0 714090 0 334.2mb 334.2mb
green  open wazuh-alerts-3.x-2020.02.21     3C234oncSgu-1mXnAGH2NA 3 0 710759 0 326.4mb 326.4mb
green  open wazuh-monitoring-3.x-2020.02.21 t5J1LptLTIC5kmi-SZ4itA 2 0    576 0 279.9kb 279.9kb
green  open wazuh-monitoring-3.x-2020.02.20 JrFSbjk0S_memRBlCaU1mQ 2 0    576 0 232.1kb 232.1kb
red    open .kibana_task_manager_1          BfCmOBOpSXaWKBzmrA2fsw 1 0
green  open wazuh-alerts-3.x-2020.02.28     AgV4n8i8TZWQUE7VZY144A 3 0 719532 0 373.5mb 373.5mb
green  open wazuh-alerts-3.x-2020.02.29     eFbh-FuaRMKYbggk9E6bdQ 3 0 540160 0 242.4mb 242.4mb
green  open wazuh-alerts-3.x-2020.02.26     AxYofUbrS8uetvPKx9GMpQ 3 0 724494 0 344.3mb 344.3mb
green  open wazuh-monitoring-3.x-2020.02.28 VyNExxZnTOO5DprLzCeMsw 2 0    258 0 269.8kb 269.8kb
green  open wazuh-alerts-3.x-2020.02.27     H_t-IN2pT2ac6qq2d-T2xw 3 0 702191 0 329.8mb 329.8mb
green  open wazuh-monitoring-3.x-2020.02.27 D32dEi68QZOQ1DfUYxAogQ 2 0    576 0 316.9kb 316.9kb
green  open wazuh-monitoring-3.x-2020.02.26 8JJT7SHOTS6rJMZWYQgdZA 2 0    576 0 245.5kb 245.5kb
green  open wazuh-alerts-3.x-2020.02.24     VndIb38kRSqsUJslr1fypw 3 0 705197 0 332.9mb 332.9mb
green  open wazuh-monitoring-3.x-2020.02.25 OdKwrkC0Rx2HQanBP4DalQ 2 0    576 0 234.4kb 234.4kb
green  open wazuh-alerts-3.x-2020.02.25     zQ-s9J0RQo2ZB0YH04GuEA 3 0 731537 0 355.8mb 355.8mb
yellow open .wazuh                          Xrxj32YBS7yHWtVDzOhN3g 1 1      1 0  12.3kb  12.3kb
green  open wazuh-monitoring-3.x-2020.02.13 _T9zU8-iSDWPfhzkB7j2dQ 2 0    552 0   448kb   448kb
green  open wazuh-alerts-3.x-2020.02.11     ENJBVoVNSyqjSvtD6WPxDA 3 0 696565 0 331.1mb 331.1mb
green  open wazuh-monitoring-3.x-2020.02.12 wK3HI2qsQU-Btas8DoUs-Q 2 0    576 0   267kb   267kb
green  open wazuh-alerts-3.x-2020.02.12     56cWR_DaR9Khv09ZPwR_Ew 3 0 753036 0 372.9mb 372.9mb
green  open wazuh-monitoring-3.x-2020.02.11 yU4W-yDMSeubFhEp9wZszA 2 0    576 0 279.9kb 279.9kb
green  open wazuh-monitoring-3.x-2020.02.10 bZc8I4c4QNOHaxRNYCayrQ 2 0    576 0 266.9kb 266.9kb
green  open wazuh-alerts-3.x-2020.02.10     v8trgdCRSDyXRpWX3LXUwA 3 0 732710 0 352.1mb 352.1mb
green  open wazuh-alerts-3.x-2020.02.19     sWxcJ6oDQ2macQFAiZCasA 3 0 707972 0 328.6mb 328.6mb
green  open wazuh-alerts-3.x-2020.02.17     gVfYAAEVTlaYE2L0JwQMrg 3 0 714184 0 339.2mb 339.2mb
green  open wazuh-monitoring-3.x-2020.02.19 yFHUQsHdR1uELDWDYXAl0w 2 0    576 0 266.9kb 266.9kb
green  open wazuh-alerts-3.x-2020.02.18     Oa-BDXKJS2ezBOBGYKhj8Q 3 0 724437 0 334.5mb 334.5mb
green  open wazuh-monitoring-3.x-2020.02.18 GeBsEBA_TC2nVb0dlDC9QA 2 0    576 0 266.7kb 266.7kb
green  open wazuh-alerts-3.x-2020.02.15     2jBk-wNvQ62fFNNMX1-lpQ 3 0 539025 0 248.1mb 248.1mb
green  open wazuh-monitoring-3.x-2020.02.17 Kd5nAnpRQtOxIKEJ9ceFhw 2 0    576 0 267.4kb 267.4kb
green  open wazuh-alerts-3.x-2020.02.16     8K6VENo9TjmvGLnKUkIl9g 3 0 502740 0 236.4mb 236.4mb
green  open wazuh-monitoring-3.x-2020.02.16 MWMmfRRzS5KPT09BpGu5jg 2 0    576 0 366.4kb 366.4kb
green  open wazuh-alerts-3.x-2020.02.13     p2YuURcfTGSgaUJfCOhZYg 3 0 730978 0   354mb   354mb
green  open wazuh-monitoring-3.x-2020.02.15 rZJ7wRhHRdy6e-4rPk2sLw 2 0    576 0 316.1kb 316.1kb
green  open wazuh-alerts-3.x-2020.02.14     M-3ETKc0QEK1Si8B_Wli3w 3 0 667235 0 312.5mb 312.5mb
green  open wazuh-monitoring-3.x-2020.02.14 UETgkJxWQ6WSSA1BVR2mQA 2 0    576 0 266.1kb 266.1kb
green  open wazuh-alerts-3.x-2019.12.29     ghN02L7STQ2QJEldpt9Izw 3 0 197225 0  99.6mb  99.6mb
green  open wazuh-alerts-3.x-2019.12.24     RzhDbCESTfaERs528owBWA 3 0 245104 0 108.3mb 108.3mb
green  open wazuh-alerts-3.x-2019.12.23     R4zrObiWR6Owy3lwzeDxyg 3 0 298095 0 130.9mb 130.9mb
green  open wazuh-alerts-3.x-2019.12.22     JUv8qum3QVGrVs_yunB1jQ 3 0 198805 0  89.1mb  89.1mb
green  open wazuh-alerts-3.x-2019.12.21     lJUHU0H0Q0uolSp9nbj-Ig 3 0 221968 0   101mb   101mb
green  open wazuh-alerts-3.x-2019.12.28     G--qTiSoSq2OCQSC77RPgw 3 0 244039 0 114.3mb 114.3mb
green  open wazuh-alerts-3.x-2019.12.27     35Sg8OZBQzabIjaM4NbajQ 3 0 294281 0 140.9mb 140.9mb
green  open wazuh-alerts-3.x-2019.12.26     tz0vktn4RSW_5RK33rHQzQ 3 0 241130 0 107.6mb 107.6mb
green  open wazuh-alerts-3.x-2019.12.25     2pFjE9DaQRK3A-F1sXb5rA 3 0 189881 0  87.2mb  87.2mb
green  open wazuh-monitoring-3.x-2020.02.02 IUnWsUH3SkqZ96M6zJozyA 2 0    192 0 261.3kb 261.3kb
green  open wazuh-monitoring-3.x-2020.02.01 xGrfSU9wSfyq-qO3a841aw 2 0    190 0 277.5kb 277.5kb
red    open wazuh-alerts-3.x-2020.03.11     Sxk9qtZRQiidGuKQVZ-yuw 3 0
green  open wazuh-monitoring-3.x-2020.01.31 nRt_uTNjTx2-RvmBVG7JUA 2 0    192 0 245.5kb 245.5kb
green  open wazuh-alerts-3.x-2019.12.31     Brtdl8MdQEShD7CNbwUafQ 3 0 275951 0 131.2mb 131.2mb
green  open wazuh-monitoring-3.x-2020.01.30 5SDxXqjbRTSzF5EGtFm2og 2 0    192 0 228.4kb 228.4kb
green  open wazuh-alerts-3.x-2019.12.30     hjMByE2MRKWTedJJB3Tr2w 3 0 284714 0 140.3mb 140.3mb
green  open wazuh-monitoring-3.x-2020.02.09 WToSSiMuSLudYQFYldMbJQ 2 0    576 0 280.4kb 280.4kb
green  open wazuh-monitoring-3.x-2020.02.08 OdP0mtAoRjeLk6W9W5bJoA 2 0    576 0   295kb   295kb
green  open wazuh-monitoring-3.x-2020.02.07 QWQAvHs8Q3iM6fTw200vtw 2 0    576 0 268.2kb 268.2kb
green  open wazuh-monitoring-3.x-2020.02.06 pqg1w402QZys_W7Obk0yEw 2 0    576 0 233.6kb 233.6kb
green  open wazuh-monitoring-3.x-2020.02.05 GZsjkZaFSLypt05oEaX_2g 2 0    510 0 260.7kb 260.7kb
red    open wazuh-alerts-3.x-2020.03.16     cbgneATVSRmd7zvYQYQAkA 3 0
green  open wazuh-monitoring-3.x-2020.02.04 WRdaNy5eRRuCEKE_87AbIg 2 0    436 0 366.6kb 366.6kb
green  open wazuh-monitoring-3.x-2020.02.03 lnz2usXDSy6PB3UEM0ryJQ 2 0    257 0 276.4kb 276.4kb
green  open wazuh-alerts-3.x-2019.12.19     ZwYVahm3QiqJfE7qvq9E5w 3 0 520014 0 223.4mb 223.4mb
green  open wazuh-alerts-3.x-2019.12.18     cX5GMatrRIG37y10htPffQ 3 0 629450 0 267.7mb 267.7mb
green  open wazuh-alerts-3.x-2019.12.13     BClWA6JrSP-aTpCIO67x_g 3 0 621350 0 264.3mb 264.3mb
green  open wazuh-alerts-3.x-2019.12.12     rSJHD-jhQAe4gyVuP9Ew0Q 3 0 645653 0 276.7mb 276.7mb
green  open wazuh-alerts-3.x-2019.12.11     vqTOqzBoTwCqb302YiuuEQ 3 0 232245 0  99.6mb  99.6mb
green  open wazuh-alerts-3.x-2019.12.17     ZUzSMq3dSGKIxzK4TgYQGQ 3 0 636728 0   269mb   269mb
green  open wazuh-alerts-3.x-2019.12.16     iEZBxBJ5RwW0sy2kUY76UA 3 0 630123 0 266.4mb 266.4mb
green  open wazuh-alerts-3.x-2019.12.15     qHCaK2XdT9i5jNsxYJ6F6g 3 0 477823 0 198.1mb 198.1mb
green  open wazuh-alerts-3.x-2019.12.14     S6zPw9JmRUO3arzV2J5eiQ 3 0 518008 0 210.2mb 210.2mb
green  open wazuh-alerts-3.x-2020.03.02     7qlBtQhkRTuONENuNqaF4Q 3 0 817677 0 543.8mb 543.8mb
green  open wazuh-monitoring-3.x-2020.01.23 1BF3mZaiRQy-XhS05pPIyQ 2 0    192 0 260.9kb 260.9kb
green  open wazuh-monitoring-3.x-2020.01.22 daubirD4RfOEwKpDJaYUXQ 2 0    190 0 293.4kb 293.4kb
green  open .security-7                     Wr-IX7WvTJaBMB-8aK_1oQ 1 0     44 0  77.7kb  77.7kb
green  open wazuh-alerts-3.x-2020.03.01     oe_wuQ4jQNujsr3gw2SqaQ 3 0 499189 0 218.3mb 218.3mb
green  open wazuh-monitoring-3.x-2020.01.21 masLXNDLRjOZH3oRRrtXSA 2 0    192 0 276.7kb 276.7kb
green  open wazuh-monitoring-3.x-2020.01.20 AKQKKlDyRoWcqzLP8ZSbBQ 2 0    190 0 164.6kb 164.6kb
green  open wazuh-alerts-3.x-2019.12.20     Cs7LBX4fRsqsVMUKZ89rFw 3 0 295277 0 128.1mb 128.1mb
green  open wazuh-monitoring-3.x-2020.01.29 LBYZ-qeDQHmV9_GaZv6qQg 2 0    192 0 277.1kb 277.1kb
green  open wazuh-monitoring-3.x-2020.01.28 AHo1N5xmQ9mIoO_0SkXCSQ 2 0    192 0 310.6kb 310.6kb
green  open wazuh-monitoring-3.x-2020.01.27 3aqLxZzIQ1uxQ2sH4guZgQ 2 0    192 0 213.5kb 213.5kb
red    open wazuh-alerts-3.x-2020.03.06     TY2eQzE4TcCVvJpCw6q0ZA 3 0
green  open wazuh-monitoring-3.x-2020.01.26 GqUjHoVoTm-quOUhmySIVA 2 0    192 0   213kb   213kb
green  open wazuh-monitoring-3.x-2020.01.25 LA1_uODWR8y6WLTrFSxHyg 2 0    192 0 180.5kb 180.5kb
red    open wazuh-alerts-3.x-2020.03.03     ge8gXgfwRDWss-YDDiZ3Rw 3 0
green  open wazuh-monitoring-3.x-2020.01.24 ZlCs4X-ESNKGOcDvcB2isA 2 0    192 0 342.6kb 342.6kb
green  open wazuh-monitoring-3.x-2019.12.19 z2aSUcE3QteaYGn_pUxBXg 2 0    190 0 293.1kb 293.1kb
green  open wazuh-alerts-3.x-2020.01.09     sdRw8cOGQ1quOLkDwiC5GQ 3 0 661233 0 293.9mb 293.9mb
green  open wazuh-monitoring-3.x-2019.12.15 neHkhCZ5TeGACFLa4-76OQ 2 0    192 0 277.5kb 277.5kb
green  open wazuh-monitoring-3.x-2019.12.16 OE20She0RLWuko401sMVHg 2 0    192 0 342.2kb 342.2kb
green  open wazuh-monitoring-3.x-2019.12.17 o2So7CvMRcqH0TsgsaerRA 2 0    192 0   245kb   245kb
green  open wazuh-monitoring-3.x-2019.12.18 6U-A849mQjGj0_rQzqqvjg 2 0    192 0 228.9kb 228.9kb
green  open wazuh-monitoring-3.x-2019.12.11 VPXNBnzqS8SRMOeZeVmH-Q 2 0     83 0 241.5kb 241.5kb
green  open wazuh-monitoring-3.x-2019.12.12 58AI5vBJQGiONXKu-RPX2A 2 0    194 0 229.9kb 229.9kb
green  open wazuh-monitoring-3.x-2019.12.13 ZzYmnuKqRmWkGD9pyD6RDA 2 0    192 0 325.7kb 325.7kb
green  open wazuh-monitoring-3.x-2019.12.14 1OsR93U-TLy7UodFuADUdw 2 0    192 0 293.9kb 293.9kb
green  open wazuh-alerts-3.x-2020.01.10     zj0Kbl_YQ5KDF6VgqjZdJw 3 0 687003 0   321mb   321mb
green  open wazuh-monitoring-3.x-2020.01.12 VE3STZhSSE2Qna7EGRHJuA 2 0    192 0 164.3kb 164.3kb
green  open wazuh-monitoring-3.x-2020.01.11 R4xxSj8GS2-VtyZVZsaecw 2 0    192 0 293.8kb 293.8kb
green  open wazuh-alerts-3.x-2020.01.11     jg2o3V5cT22MoF-ZEKxq3A 3 0 572889 0 253.1mb 253.1mb
green  open wazuh-monitoring-3.x-2020.01.10 RD3drNmuTKm5zmzrkJXECQ 2 0    192 0 310.1kb 310.1kb
green  open wazuh-alerts-3.x-2020.01.18     PYIWVUTGTdy8x3TdbBpbjw 3 0 557131 0 236.8mb 236.8mb
green  open wazuh-monitoring-3.x-2020.01.19 HRaakvpsSpO9JOmfSYorWQ 2 0    192 0 244.5kb 244.5kb
green  open wazuh-alerts-3.x-2020.01.19     EYBThxjKSA2YGUgX9_C_Ew 3 0 511675 0 216.7mb 216.7mb
green  open wazuh-monitoring-3.x-2020.01.18 mYp0H47gSIyEWFFiyzJwcQ 2 0    190 0 277.1kb 277.1kb
green  open wazuh-alerts-3.x-2020.01.16     xjrkN9q6SumdQlkRx7FjWA 3 0 678409 0 297.5mb 297.5mb
green  open wazuh-monitoring-3.x-2020.01.17 Cd5E5SiQRam1Z9c1BqHXog 2 0    192 0 358.2kb 358.2kb
green  open wazuh-alerts-3.x-2020.01.17     PXc3ca9iTOKfvevQbeioJA 3 0 664762 0 293.8mb 293.8mb
green  open wazuh-alerts-3.x-2020.01.14     Sp-RgH_LTnWL_4XfL_r4gw 3 0 682024 0 312.2mb 312.2mb
green  open wazuh-monitoring-3.x-2020.01.16 mcrHRPDURT6SlTTXartqCA 2 0    192 0 212.9kb 212.9kb
green  open wazuh-alerts-3.x-2020.01.15     dw5nQmUOTxC5nlJC_HD5eA 3 0 704967 0 321.3mb 321.3mb
green  open wazuh-monitoring-3.x-2020.01.15 OsqySLGvQDCBRd6l3bWFDw 2 0    192 0 148.1kb 148.1kb
green  open wazuh-alerts-3.x-2020.01.12     EHXdWPLMT5Gs2UEE6JqYbw 3 0 513907 0 228.5mb 228.5mb
green  open wazuh-monitoring-3.x-2020.01.14 3rPSC_KdT9OQqJA_aPOj_w 2 0    192 0 374.5kb 374.5kb
green  open wazuh-alerts-3.x-2020.01.13     Tvab1eTYSFmDLaJlJyDilA 3 0 675933 0 307.9mb 307.9mb
green  open wazuh-monitoring-3.x-2020.01.13 dczzakg6Ryagvvm-MkNvug 2 0    192 0 358.1kb 358.1kb
green  open .apm-agent-configuration        PZsi9ER0TB6tTpiP8TDfqw 1 0      0 0    283b    283b
green  open wazuh-monitoring-3.x-2020.01.01 50OrRgM-TPC2CR4IMPjnlA 2 0    192 0 131.9kb 131.9kb
green  open wazuh-alerts-3.x-2020.01.07     wWYAa3J8SUeYmTijSRrBoA 3 0 656771 0 292.7mb 292.7mb
green  open wazuh-monitoring-3.x-2020.01.09 cjSdp0jqQCaiJbuiKhfe7g 2 0    192 0 180.6kb 180.6kb
green  open kibana_sample_data_logs         _RWjtKgKQSuMR7RR6HA6Tw 1 0  14074 0  11.4mb  11.4mb
green  open wazuh-monitoring-3.x-2020.01.08 C_EtAh9vSM2JXy9FAncEpw 2 0    192 0 213.5kb 213.5kb
green  open wazuh-alerts-3.x-2020.01.08     2uqypdlZTY60IeICJJ4dPg 3 0 654699 0   299mb   299mb
green  open wazuh-alerts-3.x-2020.01.05     0rIu99OQSL2wpJocn12o-g 3 0 503132 0 214.6mb 214.6mb
green  open wazuh-monitoring-3.x-2020.01.07 fNzhwhgwRSi2BWKs8rEN3g 2 0    194 0 310.4kb 310.4kb
green  open wazuh-monitoring-3.x-2020.01.06 re0guOmPR4WUwl_6VSfoKQ 2 0    190 0   180kb   180kb
green  open wazuh-alerts-3.x-2020.01.06     Bm8lt-A7T76koHAXanXY-A 3 0 533384 0 236.3mb 236.3mb
green  open wazuh-monitoring-3.x-2020.01.05 VXyzQbjpSa-kxQSCiwRBWw 2 0    192 0 244.9kb 244.9kb
green  open wazuh-alerts-3.x-2020.01.03     -86jHxizS9WnxkmgFYcRdA 3 0 632675 0 293.2mb 293.2mb
green  open wazuh-alerts-3.x-2020.01.04     A0TdBkBiR5WKztO40N_UuQ 3 0 559291 0 246.9mb 246.9mb
green  open wazuh-monitoring-3.x-2020.01.04 c0cTMHE7QGamgGIZvCkXTQ 2 0    192 0 229.3kb 229.3kb
green  open wazuh-alerts-3.x-2020.01.01     WEa8As54R0KTckJV3rBR7Q 3 0 232952 0   116mb   116mb
green  open wazuh-monitoring-3.x-2020.01.03 FqEku6fNR4mlgovpxKFLLA 2 0    192 0 341.9kb 341.9kb
green  open wazuh-alerts-3.x-2020.01.02     Oa2YNLq9QIS8chw5Zu2syQ 3 0 529309 0 261.7mb 261.7mb
green  open wazuh-monitoring-3.x-2020.01.02 NNpmFJcAQFKjHE709T-PRA 2 0    192 0 196.3kb 196.3kb
green  open wazuh-monitoring-3.x-2019.12.30 D-WVpgSaQbOWutXDlS8YDw 2 0    192 0 245.3kb 245.3kb
green  open wazuh-alerts-3.x-2020.02.01     k5HhKup6RGeYzNBiylWlpw 3 0 521041 0 230.3mb 230.3mb
green  open wazuh-monitoring-3.x-2019.12.31 OzFgXhwlTOWc1JoPVpiwmA 2 0    192 0 229.2kb 229.2kb
green  open wazuh-alerts-3.x-2020.01.30     _7iWMcVWRNGIyZbMI5yb5g 3 0 690966 0 342.9mb 342.9mb
green  open wazuh-alerts-3.x-2020.01.31     BbLPhbuSSyOXh1YXQb9a9w 3 0 618415 0 281.1mb 281.1mb
green  open wazuh-alerts-3.x-2020.02.08     vRFJ52nbSoWLyrWmaSI87Q 3 0 532265 0 236.4mb 236.4mb
green  open wazuh-alerts-3.x-2020.02.09     bV3YNb7eQgKrul0JVMi6CA 3 0 488669 0 209.7mb 209.7mb
green  open wazuh-alerts-3.x-2020.02.06     u_26zlnETfGMQxMOMoHokw 3 0 733884 0 357.7mb 357.7mb
green  open wazuh-alerts-3.x-2020.02.07     oWsujmSZRomi63bXJJR1EQ 3 0 692606 0 326.3mb 326.3mb
green  open wazuh-alerts-3.x-2020.02.04     vxSJVKIOTZGY0bkELx26jA 3 0 679371 0 321.9mb 321.9mb
green  open wazuh-alerts-3.x-2020.02.05     mt0fa93OShKYWCtPMJcYoQ 3 0 719308 0 339.6mb 339.6mb
green  open wazuh-alerts-3.x-2020.02.02     yJJVXvlIQ36oJjb1zu9vXw 3 0 472069 0   201mb   201mb
green  open wazuh-alerts-3.x-2020.02.03     WU-_RkRWRc-WdpmIh_UB4w 3 0 597495 0 271.5mb 271.5mb
green  open wazuh-monitoring-3.x-2019.12.26 CU3bkiqbQ7GkYpsAJBUE0g 2 0    192 0 229.5kb 229.5kb
green  open wazuh-monitoring-3.x-2019.12.27 iIugUtMPRi-wJS_9M6vpUQ 2 0    192 0 213.4kb 213.4kb
green  open wazuh-monitoring-3.x-2019.12.28 FmCOuV90SUqC-fTGIflCYA 2 0    192 0 357.6kb 357.6kb
green  open wazuh-monitoring-3.x-2019.12.29 4Wr9bLC1QCiPW4n1ql3jzg 2 0    192 0 245.6kb 245.6kb
green  open wazuh-monitoring-3.x-2019.12.22 f9T7c2bxR6-6lrFYPsPWnw 2 0    192 0 196.9kb 196.9kb
green  open wazuh-monitoring-3.x-2019.12.23 P3Vz5iliS2GgQTyFmHy3Sw 2 0    192 0   213kb   213kb
red    open .kibana_1                       h3hApuwUTXqu2HTUhAYljQ 1 0
green  open wazuh-monitoring-3.x-2019.12.24 pEmHHa8wTcyUUuB-UeP-0Q 2 0    192 0 228.9kb 228.9kb
green  open wazuh-monitoring-3.x-2019.12.25 by_yqIFpSjC1J7vG9ci-eQ 2 0    192 0 245.5kb 245.5kb
green  open wazuh-alerts-3.x-2020.01.21     wORt8-4wRnylSUFl4NP8vg 3 0 676444 0 304.8mb 304.8mb
green  open wazuh-alerts-3.x-2020.01.22     ECMnh6TQS8e_ABSbB0YlgA 3 0 668968 0   318mb   318mb
green  open wazuh-monitoring-3.x-2019.12.20 Cu3D1zrtSGCUHiyHFv66AQ 2 0    192 0 309.2kb 309.2kb
green  open wazuh-alerts-3.x-2020.01.20     OLuJpZBuRRWAy-5mh7MNuw 3 0 672577 0   307mb   307mb
green  open wazuh-monitoring-3.x-2019.12.21 6jkpR2eMQHiy6gG1DxW43g 2 0    192 0 325.7kb 325.7kb
green  open wazuh-alerts-3.x-2020.01.29     RbzVJYATTb2xMITQgpMWvQ 3 0 683840 0 357.4mb 357.4mb
green  open wazuh-alerts-3.x-2020.01.27     64wCRmApTrm7sE2cQSD4hQ 3 0 634517 0 292.1mb 292.1mb
green  open wazuh-alerts-3.x-2020.01.28     z4d0YYjlSK6kXAKXl4DvNA 3 0 634512 0 288.6mb 288.6mb
green  open wazuh-alerts-3.x-2020.01.25     YXMJsqJHQAuCOXYax6w-NA 3 0 505733 0 230.2mb 230.2mb
green  open wazuh-alerts-3.x-2020.01.26     OsMGJzqXQ1K0kJx7Z16vcQ 3 0 459442 0 199.7mb 199.7mb
green  open wazuh-alerts-3.x-2020.01.23     JZsJIUuDQni24NswPnwY7A 3 0 648056 0 312.2mb 312.2mb
green  open wazuh-alerts-3.x-2020.01.24     WjBfZTZtSnC3PTakgLWRyg 3 0 642421 0 309.8mb 309.8mb

The user that executes elasticsearch should be able enough to modify those files.

Pablo Torres

unread,
Jul 24, 2020, 5:38:34 AM7/24/20
to Wazuh mailing list
Hi Alex,
Sorry for the late reply, are you still facing this issue?

Can you please share with me the output of the DELETE commands? make sure that both kibana indices are deleted, as they are still being shown in the _cat/indices output.
curl -XDELETE https://localhost:9200/.kibana*
curl -XDELETE https://localhostelastic_ip:9200/.kibana_task_manager*
After running these commands, please restart Kibana service, so Kibana can recreate those indices.

Regards,
Pablo Torres
Reply all
Reply to author
Forward
0 new messages