6.4.0 can't login and kibanaserver user can't access .kibana
benedi...@gmx.de
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
searc...@6.4.0-14
search-guard-6
Jochen Kressin
[2018![]()
-09-12T17:43:42,182][ERROR][c.f.s.f.SearchGuardFilter] Unexpected exception [] IndexNotFoundException[no such index]
org.elasticsearch.index.IndexNotFoundException: no such index
...
at com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:598) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$900(ConfigModel.java:484) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:768) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel.access$1100(ConfigModel.java:47) ~[?:?]
at com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.get(ConfigModel.java:321) ~[?:?]
Which would also explain the 30 seconds timeout you experienced. When you say:
Benedikt Haug
Thank you for having a look!
No, it never worked. I meant that i removed searchguard
completely by removing the plugin and it started to work without
searchguard.
The upgrade was done by installing the new plugin version and
restarting the cluster altogether. After it didn't work I also
tried to remove the old plugin version before installing the new
one.
I did not change the shard allocation.
The searchguard indices content is here:
https://gist.github.com/gna582/da1d2439a2c51ed3e41d96fd2385bdbe
I just tried to remove the searchguard index with -dci and recreated it afterwards but the kibana journal still looks like this:
Sep 17 15:54:08 mes-any-logwfe-dev001 kibana[23739]: {"type":"error","@timestamp":"2018-09-17T13:54:08Z","tags":["warning","stats-collection"],"pid":23739,"level":"error","error":{"message":"[security_exception] Unexpected exception indices:data/read/search","name":"Error","stack":"[security_exception] Unexpected exception indices:data/read/search :: {\"path\":\"/.kibana/_search\",\"query\":{\"ignore_unavailable\":true,\"filter_path\":\"aggregations.types.buckets\"},\"body\":\"{\\\"size\\\":0,\\\"query\\\":{\\\"terms\\\":{\\\"type\\\":[\\\"dashboard\\\",\\\"visualization\\\",\\\"search\\\",\\\"index-pattern\\\",\\\"graph-workspace\\\",\\\"timelion-sheet\\\"]}},\\\"aggs\\\":{\\\"types\\\":{\\\"terms\\\":{\\\"field\\\":\\\"type\\\",\\\"size\\\":6}}}}\",\"statusCode\":500,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"Unexpected exception indices:data/read/search\\\"}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"Unexpected exception indices:data/read/search\\\"},\\\"status\\\":500}\"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:307:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:266:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:111:20)\n at IncomingMessage.emit (events.js:208:7)\n at endReadableNT (_stream_readable.js:1064:12)\n at _combinedTickCallback (internal/process/next_tick.js:138:11)\n at process._tickDomainCallback (internal/process/next_tick.js:218:9)"},"message":"[security_exception] Unexpected exception indices:data/read/search"}
As this is a dev host I could drop the .kibana index but I don't want to do this for live environments.
Thank you for your help!
--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/de3e56d5-8c8e-4de9-9d2e-4e0dfd702cbc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Jochen Kressin
\"statusCode\":500,\"
...
Unexpected exception indices:data/read/search
This means it is an internal Search Guard error. If the .kibana index would not be accerssible, say, because of wrong roles or permissions, you would see a different message.
Refers to the Search Guard index not being accessible. It roots in SG not being able to read the roles from the index:
at com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.get(ConfigModel.java:321) ~[?:?]
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
Benedikt Haug
Sure! Here it is:
https://gist.github.com/gna582/cdc9e628330808a23443fd71c348e250
But I can retrieve the config via sgadmin -r and it looks
identical except quotation and indention.
Thank you
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3c5adfac-ee43-453c-a18a-dd1d501c6ae6%40googlegroups.com.
Jochen Kressin
logger.searchguard.level = debugOn Monday, September 17, 2018 at
Benedikt Haug
- are you saying that you can retrieve the configs via sgadmin -r from the running cluster?
Yes. It seems to be the same except quotation and indentation.
- are you able to access Elasticsearch directly, means without Kibana?
I pulled the data from the search api via curl so yes that is
working. E.g. this would result in a whole lot of output, so i
pasted only a small portion
curl --header "x-forwarded-for: 10.90.30.226" --user ttlko
--header "x-proxy-user: myuser" --header "x-proxy-roles: mygroup"
https://mes-any-logwfe-dev001.qa.server.lan:9200/.monitoring-es-6-2018.09.14/_search?pretty
...}, "segments" : { "count" : 5, "memory_in_bytes" : 18562, "terms_memory_in_bytes" : 14045, "stored_fields_memory_in_bytes" : 1560, "term_vectors_memory_in_bytes" : 0, "norms_memory_in_bytes" : 0, "points_memory_in_bytes" : 689, "doc_values_memory_in_bytes" : 2268, "index_writer_memory_in_bytes" : 0, "version_map_memory_in_bytes" : 0, "fixed_bit_set_memory_in_bytes" : 0 }, "request_cache" : {...
- can you try to apply the configuration again with sgadmin? Any errors in the logs?
Yes. No errors in the logs:
root@mes-any-logwfe-dev001:~#
/usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh
-cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/
-cacert /etc/ssl/mailcore/certs/padminmesanylogqa.pem -cert
/etc/ssl/mailcore/certs/padminmesanylogqa.pem -h
mes-any-logwfe-dev001.qa.server.lan -cn mes_any_log -key
/etc/ssl/mailcore/certs/padminmesanylogqa.pem -nhnv
WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v6
Will connect to mes-any-logwfe-dev001.qa.server.lan:9300 ... done
Elasticsearch Version: 6.4.0
Search Guard Version: 6.4.0-23.0
Connected as UID=XXXXX,CN=padminmesanylogqa,O=ClientCert
Contacting elasticsearch cluster 'mes_any_log' and wait for YELLOW
clusterstate ...
Clustername: mes_any_log
Clusterstate: GREEN
Number of nodes: 3
Number of data nodes: 2
searchguard index already exists, so we do not need to create one.
Populate config from
/usr/share/elasticsearch/plugins/search-guard-6/sgconfig/
Will update 'sg/config' with
/usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_config.yml
SUCC: Configuration for 'config' created or updated
Will update 'sg/roles' with
/usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_roles.yml
SUCC: Configuration for 'roles' created or updated
Will update 'sg/rolesmapping' with
/usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_roles_mapping.yml
SUCC: Configuration for 'rolesmapping' created or updated
Will update 'sg/internalusers' with
/usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_internal_users.yml
SUCC: Configuration for 'internalusers' created or updated
Will update 'sg/actiongroups' with
/usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_action_groups.yml
SUCC: Configuration for 'actiongroups' created or updated
Done with success
- can you start ES with debug logging? We would need the debug outputs from ES and SG while starting the node, and when the error occurs.
To enable debug logging, in your log4j2.properties, add:
logger.searchguard.name = com.floragunn
logger.searchguard.level = debugOn Monday, September 17, 2018 at
And set the ES logger to debug as well:
logger.action.name = org.elasticsearch.actionlogger.action.level = debug
Sure! The gist is here: https://gist.github.com/gna582/354a9205457b89aa4fefa1a617b0569d
Thank you!
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/be72fb2b-22d7-44e3-ad68-455df6f61bd2%40googlegroups.com.
Search Guard
Benedikt Haug
The first SG version was definitely 6.3.0-22.3. The cluster was
deployed with 6.3.0 initially and was upgraded to 6.4.0-23.0.
--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9e32db2b-f3b2-4e6c-95a7-98984138ec4c%40googlegroups.com.
Search Guard
'.kibana':
'*':
- INDICES_ALL
'?kibana':
'*':
- INDICES_ALL
sg_kibana_server:
readonly: true
cluster:
- CLUSTER_MONITOR
- CLUSTER_COMPOSITE_OPS
- cluster:admin/xpack/monitoring/*
- indices:admin/template/*
indices:
'?kibana':
'*':
- INDICES_ALL
'.kibana':
'*':
- INDICES_ALL
# - indices:admin/template/put
'?kibana-6':
'*':
- INDICES_ALL
'?reporting*':
'*':
- INDICES_ALL
'?monitoring*':
'*':
- INDICES_ALL
Benedikt Haug
You are correct! Thank you! This solved the strange .kibana
issues. I have consequently reviewed all diffs to upstream config
so this doesn't happen again. Sry.
Sadly I can still not log in via proxy authentication due to: "no
xff done for class
org.elasticsearch.http.netty4.Netty4HttpRequest". The same config
works fine with 6.3.0-22.3
I think I have found the cause of this to be the new Netty 4.1.25.Final release newly introduced in ES 6.4.0 after I realized there is a trace debug level and had a look at the code.
Would it be possible for you to have a look and check whether proxy authentication is still working as intended with ES 6.4.0? I made a Gist where I have commented what I think goes wrong in the RemoteIpDetector.java:
https://gist.github.com/gna582/cc8f33835054b71cb44d2f6cba4d8765
Thank you for your help!
--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/2c95a7a4-a85f-4599-8501-a66f95fec1dd%40googlegroups.com.
Search Guard
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.