ES 6.4.0 -> 6.5.2: Kibana cannot migrate index: "No cluster-level perm match for User kibanaserver"

[jbec...@ft-services.com]

* ES 6.4.0 upgrading to 6.5.2; SG 23.2

* JVM 1.8 on Windows 7 SP1

Testing on our dev cluster prior to updating QA and Production.

Unpacked latest ES 6.5.2 / Kibana 6.5.2 into parallel directories to our existing 6.4.0; checked they started up OK.

Installed Search Guard.

I've updated our sg_roles.yml for sg_kibana_server according to the upgrade notes. Kibana instances show creation of .kibana_1 / _2 but when they come to the reindex step a failure is thrown:

[2018-12-10T16:53:51,740][INFO ][c.f.s.p.PrivilegesEvaluator] [node1] No cluster-level perm match for User [name=kibanaserver, roles=[], requestedTenant=null] Resolved [aliases=[], indices=[.kibana, .kibana_1], allIndices=[.kibana, .kibana_1], types=[null, *], isAll()=false, isEmpty()=false] [Action [indices:data/write/reindex]] [RolesChecked [sg_kibana_server, sg_own_index]]
[2018-12-10T16:53:51,741][INFO ][c.f.s.p.PrivilegesEvaluator] [node1] No permissions for [indices:data/write/reindex]
[2018-12-10T16:53:51,871][WARN ][o.e.t.LoggingTaskListener] [node1] 90702 failed with exception

Here's a later log with DEBUG, showing a successful CREATE and a failing REINDEX:

[2018-12-10T17:29:26,250][TRACE][o.e.t.T.tracer           ] [node1] [11791][indices:admin/create] received response from [{node2}{xTlrKLONT9iQaZLhPlu2mQ}{eTc-6o33TmmLjXKYhDh02w}{10.32.0.89}{10.32.0.89:9300}{ml.machine_memory=34263273472, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]
[2018-12-10T17:29:26,256][DEBUG][c.f.s.a.BackendRegistry  ] [node1] User 'User [name=kibanaserver, roles=[], requestedTenant=null]' is authenticated
[2018-12-10T17:29:26,256][DEBUG][c.f.s.a.BackendRegistry  ] [node1] sgtenant 'null'
[2018-12-10T17:29:26,257][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] ### evaluate permissions for User [name=kibanaserver, roles=[], requestedTenant=null] on node1
[2018-12-10T17:29:26,257][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] action: indices:admin/create (CreateIndexRequest)
[2018-12-10T17:29:26,258][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolve aliases, indices and types from CreateIndexRequest
[2018-12-10T17:29:26,258][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] No such indices for pattern .kibana_1, use raw value
[2018-12-10T17:29:26,258][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requestedResolved : Resolved [aliases=[], indices=[.kibana_1], allIndices=[.kibana_1], types=[*], isAll()=false, isEmpty()=false]
[2018-12-10T17:29:26,258][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requested [indices:admin/create] from 127.0.0.1:64107
[2018-12-10T17:29:26,258][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requested resolved indextypes: Resolved [aliases=[], indices=[.kibana_1], allIndices=[.kibana_1], types=[*], isAll()=false, isEmpty()=false]
[2018-12-10T17:29:26,259][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] sgr: sg_kibana_server
[2018-12-10T17:29:26,259][DEBUG][c.f.s.c.PrivilegesInterceptorImpl] [node1] raw requestedTenant: 'null'
[2018-12-10T17:29:26,259][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] Result from privileges interceptor: null
[2018-12-10T17:29:26,259][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] .kibana_1 does not exist in cluster metadata
[2018-12-10T17:29:26,260][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] Allowed because we have all indices permissions for indices:admin/create
[2018-12-10T17:29:26,260][DEBUG][c.f.s.f.SearchGuardFilter] [node1] PrivEvalResponse [allowed=true, missingPrivileges=[indices:admin/create], allowedFlsFields=null, maskedFields=null, queries=null]

[2018-12-10T17:29:26,302][DEBUG][o.e.i.IndicesService     ] [node1] creating Index [[.kibana_1/yG378GmbR8CYO9vB7NDSUw]], shards [1]/[1] - reason [create index]

[2018-12-10T17:29:26,955][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] ### evaluate permissions for User [name=kibanaserver, roles=[], requestedTenant=null] on node1
[2018-12-10T17:29:26,955][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] action: indices:data/write/reindex (ReindexRequest)
[2018-12-10T17:29:26,956][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolve aliases, indices and types from ReindexRequest
[2018-12-10T17:29:26,956][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolved pattern [.kibana_1] to [.kibana_1]
[2018-12-10T17:29:26,956][DEBUG][c.f.s.r.IndexResolverReplacer] [node1] Resolved pattern [.kibana] to [.kibana]
[2018-12-10T17:29:26,957][DEBUG][c.f.s.p.PrivilegesEvaluator] [node1] requestedResolved : Resolved [aliases=[], indices=[.kibana, .kibana_1], allIndices=[.kibana, .kibana_1], types=[null, *], isAll()=false, isEmpty()=false]
[2018-12-10T17:29:26,957][INFO ][c.f.s.p.PrivilegesEvaluator] [node1] No cluster-level perm match for User [name=kibanaserver, roles=[], requestedTenant=null] Resolved [aliases=[], indices=[.kibana, .kibana_1], allIndices=[.kibana, .kibana_1], types=[null, *], isAll()=false, isEmpty()=false] [Action [indices:data/write/reindex]] [RolesChecked [sg_kibana_server, sg_own_index]]
[2018-12-10T17:29:26,958][INFO ][c.f.s.p.PrivilegesEvaluator] [node1] No permissions for [indices:data/write/reindex]
[2018-12-10T17:29:26,958][DEBUG][c.f.s.f.SearchGuardFilter] [node1] PrivEvalResponse [allowed=false, missingPrivileges=[indices:data/write/reindex], allowedFlsFields=null, maskedFields=null, queries=null]
[2018-12-10T17:29:26,959][DEBUG][c.f.s.f.SearchGuardFilter] [node1] no permissions for [indices:data/write/reindex]

 I double checked the permissions for kibanaserver using the Search Guard API - these look correct to me:

$ curl -u admin:x http://localhost:9200/_searchguard/api/internalusers/kibanaserver
{"kibanaserver":{"readonly":"true","hash":""}}
 
$ curl -u admin:x http://localhost:9200/_searchguard/api/rolesmapping/sg_kibana_server
{"sg_kibana_server":{"readonly":"true","users":["kibanaserver"]}}
 
$ curl -u admin:x http://localhost:9200/_searchguard/api/roles/sg_kibana_server
{
                "sg_kibana_server": {
                                "cluster": ["CLUSTER_MONITOR", "CLUSTER_COMPOSITE_OPS", "cluster:admin/xpack/monitoring*", "indices:admin/template*", "indices:data/read/scroll*"],
                                "indices": {
                                                "?kibana-6": {
                                                                "*": ["INDICES_ALL"]
                                                },
                                                "?management-beats*": {
                                                                "*": ["INDICES_ALL"]
                                                },
                                                "?kibana_*": {
                                                                "*": ["INDICES_ALL"]
                                                },
                                                "?monitoring*": {
                                                                "*": ["INDICES_ALL"]
                                                },
                                                "?reporting*": {
                                                                "*": ["INDICES_ALL"]
                                                },
                                                "*": {
                                                                "*": ["indices:admin/aliases*"]
                                                },
                                                "?kibana": {
                                                                "*": ["INDICES_ALL"]
                                                },
                                                "?tasks": {
                                                                "*": ["INDICES_ALL"]
                                                }
                                },
                                "readonly": "true"
                }
}

 

In the end we've worked around this for our dev cluster by disabling Search Guard for one restart, letting Kibana completing its reindex before re-enabling - but this won't be appropriate for our other clusters.

Are we missing something here?

[SG]

Are your action groups up to date? CLUSTER_COMPOSITE_OPS should include "indices:data/write/reindex", see https://github.com/floragunncom/search-guard/blob/13e90b17509e95413e85ea67d2cf105935204dac/sgconfig/sg_action_groups.yml

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/680fe687-fce6-40eb-948d-fb07534d8194%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[jbec...@ft-services.com]

On Tuesday, December 11, 2018 at 5:17:06 PM UTC, Search Guard wrote:

Are your action groups up to date? CLUSTER_COMPOSITE_OPS should include "indices:data/write/reindex"

 

Looks like that was it. I've added that, and I can run a reindex from .kibana to .kibana_xxx with the `kibanaserver` user, which I couldn't before.

Two things we don't understand though:

* Why does the index-level wildcarded INDICES_ALL permission, which includes `indices:*` not give the required access

* How were reindex requests working at all, prior to this? We're performed many of these in the last year or so.

[SG]

Search Guard Community Forum - We are moving!

We are moving the Search Guard Community Forum to a new home:
https://forum.search-guard.com/

All content hosted has been migrated to the new forum.

Starting from 2019/03/30 please ask your questions on https://forum.search-guard.com/ only.

The Google Group forum will not be maintained anymore.
Thanks!

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/5194e3e8-00a6-43a3-a7e5-20b183a87008%40googlegroups.com.