no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview]

[Lance Zukel]

When asking questions, please provide the following information:

* Search Guard version: 6.1.2x

* Elasticsearch version: 6.1.2

* Installed and used enterprise modules, if any

* JVM version: java version "1.8.0_112"

                        Java(TM) SE Runtime Environment (build 1.8.0_112-b15)

                        Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

* Operating system version: Linux <hostname> 2.6.32-642.11.1.el6.x86_64 #1 SMP Wed Oct 26 10:25:23 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

1st - I am using the elasticsearch-hadoop-6.1.2 plugin in Hive to insert tables into elasticsearch index (test_index)

ADD JAR hdfs:///udf/elasticsearch-hadoop-6.1.2.jar;

ADD JAR /usr/hdp/2.5.5.0-157/hive/lib/commons-httpclient-3.0.1.jar;

CREATE EXTERNAL TABLE dev.test_table (

                

                company              STRING,

          sys                         STRING,

  date_time                   STRING

)

STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'

TBLPROPERTIES(

            'es.nodes' = "esnode",

            'es.port'='9200',

            'es.net.ssl'='true',

            'es.net.ssl.keystore.location'='file:///etc/pki/ca-trust/extracted/java/cacerts',

            'es.net.ssl.keystore.pass'='pass',

            'es.net.ssl.keystore.type'='JKS',

            'es.net.ssl.truststore.location'='file:///etc/pki/ca-trust/extracted/java/cacerts',

            'es.net.ssl.truststore.pass'='pass',

            'es.net.http.auth.user'='testuser',

            'es.net.http.auth.pass'='dummy_password',

            'es.net.ssl.cert.allow.self.signed'='true',

            'es.resource.write' = 'test_index/account',

            'es.query'='?q=*'

            );

2nd - Search-guard is enabled and configured on 51 node elasticsearch cluster

3rd - I created a user through the searchguard api to manage the index test_index

curl -XPUT 'https://esnode:9200/_searchguard/api/user/testuser' -H "Content-Type: application/json" -d '{"password":"dummy_password", "roles":["testrole"]}'

4th - I created a role with permissions that should allow access to test_index*

curl -XPUT 'https://esnode:9200/_searchguard/api/roles/testrole' -H "Content-Type: application/json" -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"test_index*":{"*":["UNLIMITED","MANAGE"]}}}'

curl -XPUT 'https://esnode:9200/_searchguard/api/rolesmapping/testrole' -H "Content-Type: application/json" -d '{"users":["testuser"]}'

5th - I can use elasticsearch.hadoop to create the index and update the index perfectly (as long as there is not a date in the index)

6th - Update the external hive table mapping as follows (date_time is actually just the date):

ADD JAR hdfs:///udf/elasticsearch-hadoop-6.1.2.jar;

ADD JAR /usr/hdp/2.5.5.0-157/hive/lib/commons-httpclient-3.0.1.jar;

CREATE EXTERNAL TABLE dev.test_table (

                

                company              STRING,

           sys                         STRING,

   date_time                   STRING

)

STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'

TBLPROPERTIES(

            'es.nodes' = "esnode",

            'es.port'='9200',

            'es.net.ssl'='true',

            'es.net.ssl.keystore.location'='file:///etc/pki/ca-trust/extracted/java/cacerts',

            'es.net.ssl.keystore.pass'='pass',

            'es.net.ssl.keystore.type'='JKS',

            'es.net.ssl.truststore.location'='file:///etc/pki/ca-trust/extracted/java/cacerts',

            'es.net.ssl.truststore.pass'='pass',

            'es.net.http.auth.user'='testuser',

            'es.net.http.auth.pass'='dummy_password',

            'es.net.ssl.cert.allow.self.signed'='true',

            'es.resource.write' = 'test_index_{date_time}/account',

            'es.query'='?q=*'

            );

ISSUE:

as soon as I add the date to the index name I get the following error: 

Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

The data is inserted into the elasticsearch test_index_2018-05-23 - 6 times

I have confirmed this is not an elasticsearch hadoop issue by using the admin user for the whole cluster (which works as expected)

[SG]

the user "elasticview" does not have the permission for "indices:admin/refresh".
Can you post the roles/roles-mapping for this user?

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/c738739c-4483-4295-93b8-d25ad860a063%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Lance Zukel]

roles: 

{"elasticview":{"cluster":["UNLIMITED","indices:*","indices:admin/exists*","indices:admin/refresh","indices:admin/refresh*","indices:data/read/scroll","indices:data/write/bulk*","cluster:monitor/nodes/info","indices:admin/create"],"indices":{"*test_index_*":{"*":["UNLIMITED","*","MANAGE","admin/create"]}}}}

rolesmapping:

{"elasticview":{"users":["evtest","elasticview"]}}

The error only happens when I use append a date to the index. And the admin user does not the same issue 

This appears to be a bug with the way wildcards are handled in the index name of roles.

[SG]

"indices:admin/refresh" is not a cluster level permissions but an indices one.
So it appears that you execute a refresh on an index not matching "*spectrum_guide_rollout_*"

Try adding a role to elasticview which permitts to execute a "indices:admin/refresh" on every index.

> Am 24.05.2018 um 21:56 schrieb Lance Zukel <lzu...@gmail.com>:
>
> roles:

> {"elasticview":{"cluster":["UNLIMITED","indices:*","indices:admin/exists*","indices:admin/refresh","indices:admin/refresh*","indices:data/read/scroll","indices:data/write/bulk*","cluster:monitor/nodes/info","indices:admin/create"],"indices":{"*spectrum_guide_rollout_*":{"*":["UNLIMITED","*","MANAGE","admin/create"]}}}}

>
> rolesmapping:
> {"elasticview":{"users":["evtest","elasticview"]}}
>
>

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/c13de74e-818a-402c-863c-1859ba7fed73%40googlegroups.com.

[Lance Zukel]

After giving the following config: 

'{"cluster":["cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_*":["UNLIMITED","*","MANAGE","refresh","admin/refresh"],"*":["indices:admin/refresh","indices:data/write/bulk","indices:data/write/bulk*"]}}'

I now get:

returned Internal Server Error(500) - Unexpected exception indices:data/write/bulk[s]; Bailing out

[SG]

indices:data/write/bulk is a cluster permission - please read the docs

https://docs.search-guard.com/latest/action-groups#multi--and-bulk-requests
https://docs.search-guard.com/latest/roles-permissions

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/167b15d7-1b9d-4dab-8ff6-f5fced27e817%40googlegroups.com.

[Lance Zukel]

- The searchguard API documentation is very vague and does not seem to be very helpful. 

- Most of the docs reference yml format, yet the api references json format

- There seems to not be any explanation for most of the values.

- I feel that clearer documentation about searchguard is needed
- Per the documentation "UNLIMITED" on an index should allow that user to perform any action on that index

- "MANAGE" on an index should allow the user with that role to perform nearly any action on that index

Let me see if I can explain this better...

---------------------------------

Sending data such as 'es.resource.write' = "spectrum_guide_rollout{date_time}/account"

curl -XPUT 'https://d03nappp0102:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_*":{"*":["UNLIMITED","MANAGE"]}}}' -> does not work and returns error --- > org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"*":{"*":["UNLIMITED","MANAGE"]}}}' -> does not work -> works but allows the user any index

----------------------------------

Sending data such as 'es.resource.write' = "spectrum_guide_rollout/account"

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_*":{"*":["UNLIMITED","MANAGE"]}}}' -> works

____________________

Sending data such as 'es.resource.write' = "spectrum_guide_rollout_2018_05_24/account"

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_2018_05_24":{"*":["UNLIMITED","MANAGE"]}}}' -> works

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_*":{"*":["UNLIMITED","MANAGE"]}}}' -> does not work and returns error --- > org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

___________________

The issue seems to be a bug with the way that wildcard chars are handled when there is a date involved in the index name

 

If there is a better way to create roles with the appropriate permissions I would appreciate an example, or a detailed explaination. I have found previous responses not very helpful, Documentation is referenced, but the documentation is vague on the points that were made. This is going on 2 days worth of work to implement what should've been a simple task. This is not a very intuitive nor user friendly product.

[Lance Zukel]

I mean even this fails:

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["CLUSTER_ALL"],"indices":{"*spectrum_guide_rollout_*":{"*":["UNLIMITED","*","MANAGE","INDICES_ALL"]},"*":{"*":["admin/refresh"]}}}'

Error:

no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

From the docs:

Cluster-level action groups

Name	Description
CLUSTER_ALL	Grants all cluster permissions. Equates to cluster:*

General

Name	Description
UNLIMITED	Grants complete access, can be used on index- and cluster-level. Equates to "*".

Index-level action groups

Name	Description
INDICES_ALL	Grants all permissions on the index. Equates to indices:*

 

[Search Guard]

Let me quickly outline the problem

- You try to insert data via hadoop into ES

- As long as you use the admin user all works as expected

- You are dealing with two users: testuser an elasticview

- With the later there seems to be a problem if a date pattern is in the indexname (in combination with wildcards)

pls see more comments inline below:

On Thursday, 24 May 2018 23:53:47 UTC+2, Lance Zukel wrote:

- The searchguard API documentation is very vague and does not seem to be very helpful. 

- Most of the docs reference yml format, yet the api references json format

- There seems to not be any explanation for most of the values.

- I feel that clearer documentation about searchguard is needed
- Per the documentation "UNLIMITED" on an index should allow that user to perform any action on that index

- "MANAGE" on an index should allow the user with that role to perform nearly any action on that index

Let me see if I can explain this better...

---------------------------------

Sending data such as 'es.resource.write' = "spectrum_guide_rollout{date_time}/account"

curl -XPUT 'https://d03nappp0102:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_*":{"*":["UNLIMITED","MANAGE"]}}}' -> does not work and returns error --- > org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

That is because "spectrum_guide_rollout{date_time}" will not match "spectrum_guide_rollout_*" but "spectrum_guide_rollout*" i guess (mind the underscore)

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"*":{"*":["UNLIMITED","MANAGE"]}}}' -> does not work -> works but allows the user any index

----------------------------------

Sending data such as 'es.resource.write' = "spectrum_guide_rollout/account"

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_*":{"*":["UNLIMITED","MANAGE"]}}}' -> works

____________________

pattern match unlike above

 

Sending data such as 'es.resource.write' = "spectrum_guide_rollout_2018_05_24/account"

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_2018_05_24":{"*":["UNLIMITED","MANAGE"]}}}' -> works

pattern match

 

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_*":{"*":["UNLIMITED","MANAGE"]}}}' -> does not work and returns error --- > org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

___________________

this one is really strange because this should also match, i need to investigate and try to reproduce

[Search Guard]

On Friday, 25 May 2018 00:17:08 UTC+2, Lance Zukel wrote:

I mean even this fails:

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["CLUSTER_ALL"],"indices":{"*spectrum_guide_rollout_*":{"*":["UNLIMITED","*","MANAGE","INDICES_ALL"]},"*":{"*":["admin/refresh"]}}}'

should be

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["CLUSTER_ALL"],"indices":{"*spectrum_guide_rollout_*":{"*":["UNLIMITED","*","MANAGE","INDICES_ALL"]},"*":{"*":["indices:admin/refresh"]}}}'

[Lance Zukel]

First, I would like to apologize for not getting back to you sooner, I was out on Friday and did not have a chance to test this until this morning.

Secondly, seems the issue is only appears when the date is dynamically generated as shown below:

Here this is sending a hard coded date ->

Sending data such as 'es.resource.write' = "spectrum_guide_rollout_2018_05_24/account"

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_2018_05_24":{"*":["UNLIMITED","MANAGE"]}}}' -> works

pattern match

This should not have a pattern match ->

Sending data such as 'es.resource.write' = "spectrum_guide_rollout/account"

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_*":{"*":["UNLIMITED","MANAGE"]}}}' -> works

I have been trying multiple permutations but still not working as expected ( if I posted all the attempts here it would probably be overwhelming )  ->

Sending data such as 'es.resource.write' = "spectrum_guide_rollout{date_time}/account"

curl -XPUT 'https://d03nappp0102:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_*":{"*":["UNLIMITED","MANAGE"]}}}' -> does not work and returns error --- > org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

That is because "spectrum_guide_rollout{date_time}" will not match "spectrum_guide_rollout_*" but "spectrum_guide_rollout*" i guess (mind the underscore)

(example I have also tried) Sending data such as 'es.resource.write' = "spectrum_guide_rollout_{date_time}/account"

curl -XPUT 'https://d03nappp0102:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_*":{"*":["UNLIMITED","MANAGE"]}}}' -> does not work and returns error --- > org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

Third:

This is exactly the point I have been trying to make ->

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["indices:admin/*","indices:data/read/scroll","cluster:monitor/nodes/info"],"indices":{"spectrum_guide_rollout_*":{"*":["UNLIMITED","MANAGE"]}}}' -> does not work and returns error --- > org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

___________________

this one is really strange because this should also match, i need to investigate and try to reproduce

Fourth:

In response to the below comment:

I have tried both ways with additional options.

I mean even this fails:

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["CLUSTER_ALL"],"indices":{"*spectrum_guide_rollout_*":{"*":["UNLIMITED","*","MANAGE","INDICES_ALL"]},"*":{"*":["admin/refresh"]}}}'

should be

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["CLUSTER_ALL"],"indices":{"*spectrum_guide_rollout_*":{"*":["UNLIMITED","*","MANAGE","INDICES_ALL"]},"*":{"*":["indices:admin/refresh"]}}}' -> I tried this as well

 

[Lance Zukel]

Role, Rolesmapping, and internal user config 

$ curl -XGET 'https:/es_node:9200/_searchguard/api/roles/elasticview'

{"elasticview":{"cluster":["cluster:monitor/nodes/info","indices:data/write/bulk","indices:admin/create"],"indices":{"spectrum_guide_*":["UNLIMITED","*","MANAGE","indices:admin/create"]}}}

    Based on the documentation - This should work... 

        - The elasticview role should be able to perform any action on any index starting with "spectrum_guide_"

        - UNLIMITED and * are both redundant and makes no difference one with or without on or the other. Both were left in for the purposes of this post

        - indices:admin/create is included as well due to error received

$ curl -XGET 'https://es_node:9200/_searchguard/api/rolesmapping/elasticview'

{"elasticview":{"users":["evtest","elasticview"]}}

    Confirmed the elasticview role is mapped to the elasticview user

$ curl -XGET 'https://es_node:9200/_searchguard/api/internalusers/elasticview'

{"elasticview":{"roles":["elasticview"],"hash":"somehashedoutvalue"}}

    Confirming the elasticview user is assigned the elasticview role as above

Hadoop table config (for reference):

ADD JAR hdfs:///udf/elasticsearch-hadoop-6.1.2.jar;

ADD JAR /usr/hdp/2.5.5.0-157/hive/lib/commons-httpclient-3.0.1.jar;

CREATE EXTERNAL TABLE dev.es_spec_guide_rollout_zuk_test (

                

                legacy_company              STRING,

  sys                                    STRING,

  date_time                          STRING

)

STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'

TBLPROPERTIES(

            'es.nodes' = "es_node",

            'es.port'='9200',

            'es.net.ssl'='true',

            'es.net.ssl.keystore.location'='file:///etc/pki/ca-trust/extracted/java/cacerts',

            'es.net.ssl.keystore.pass'='changeit',

            'es.net.ssl.keystore.type'='JKS',

            'es.net.ssl.truststore.location'='file:///etc/pki/ca-trust/extracted/java/cacerts',

            'es.net.ssl.truststore.pass'='changeit',

            'es.net.http.auth.user'='elasticview',

            'es.net.http.auth.pass'='password',

            'es.net.ssl.cert.allow.self.signed'='true',

            'es.resource.write' = "spectrum_guide_rollout_zuk_{date_time}/account",

            'es.query'='?q=*'

            );

Hive Command to insert into table:

insert into table es_spec_guide_rollout_zuk_test values ('zukel','lance','2018-05-29');

or 

insert into table es_spec_guide_rollout_zuk_test values ('zukel','lance','2018_05_29');

or

insert into table es_spec_guide_rollout_zuk_test values ('zukel','lance',"2018_05_29");

Error code returned from HIVE for above commands:

Found unrecoverable error [es_node:9202] returned Internal Server Error(500) - Unexpected exception indices:admin/create; Bailing out..

Curl Command attempting to create same index:

curl -XPUT 'https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/' -u "elasticview:password"

Error returned from curl:

{"error":{"root_cause":[{"type":"security_exception","reason":"Unexpected exception indices:admin/create"}],"type":"security_exception","reason":"Unexpected exception indices:admin/create"},"status":500}

Verify admin user working:

curl -XGET 'https://es_node:9200/_searchguard/api/rolesmapping'

  "sg_all_access" : {

    "users" : [

      "admin",

      "throttle"

    ]

  },

    Only relevant results shown above

curl -XGET 'https://es_node:9200/_searchguard/api/roles/sg_all_access'

{"sg_all_access":{"cluster":["UNLIMITED"],"tenants":{"test_tenant_ro":"RW","adm_tenant":"RW"},"indices":{"*":{"*":["UNLIMITED"]}}}}

curl -XPUT 'https://esnode:9200/spectrum_guide_rollout_zuk_2018_05_29/' -u "admin:Password"

{"acknowledged":true,"shards_acknowledged":true,"index":"spectrum_guide_rollout_zuk_2018_05_29"}

    Works as Expected

curl -XDELETE 'https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/' -u "admin:Password"

{"acknowledged":true,"shards_acknowledged":true,"index":"spectrum_guide_rollout_zuk_2018_05_29"}

    Delete the index to test from the beginning

Adjust Hadoop Table to verify Admin User working:

ADD JAR hdfs:///udf/elasticsearch-hadoop-6.1.2.jar;

ADD JAR /usr/hdp/2.5.5.0-157/hive/lib/commons-httpclient-3.0.1.jar;

CREATE EXTERNAL TABLE dev.es_spec_guide_rollout_zuk_test (

                

                legacy_company              STRING,

   sys                                    STRING,

   date_time                          STRING

)

STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'

TBLPROPERTIES(

            'es.nodes' = "es_node",

            'es.port'='9200',

            'es.net.ssl'='true',

            'es.net.ssl.keystore.location'='file:///etc/pki/ca-trust/extracted/java/cacerts',

            'es.net.ssl.keystore.pass'='changeit',

            'es.net.ssl.keystore.type'='JKS',

            'es.net.ssl.truststore.location'='file:///etc/pki/ca-trust/extracted/java/cacerts',

            'es.net.ssl.truststore.pass'='changeit',

            'es.net.http.auth.user'='admin',

            'es.net.http.auth.pass'='Password',

            'es.net.ssl.cert.allow.self.signed'='true',

            'es.resource.write' = "spectrum_guide_rollout_zuk_{date_time}/account",

            'es.query'='?q=*'

            );

Hive> insert into table es_spec_guide_rollout_zuk_test values ('zukel','lance',"2018_05_29");

Success

Rollback Hadoop table to use elasticview user:

Attempt to solve:

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -H "Content-Type: application/json" -d '{"cluster":["UNLIMITED"],"tenants":{"adm_tenant":"RW","test_tenant":"RW"},"indices":{"spectrum_guide_*":{"*":["UNLIMITED"]}}}'

    Again should above should work - This time mimicked the admin user just narrowed down the index access

curl -XDELETE 'https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/' -u "admin:Password"

{"acknowledged":true,"shards_acknowledged":true,"index":"spectrum_guide_rollout_zuk_2018_05_29"}

    Delete the index to test from the beginning

curl -XPUT 'https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/' -u "elasticview:password"

{"acknowledged":true,"shards_acknowledged":true,"index":"spectrum_guide_rollout_zuk_2018_05_29"}

    This is an improvement

Hive> insert into table es_spec_guide_rollout_zuk_test values ('zukel','lance',"2018_05_29");

no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

    Failing again - when by all logical reasoning this should be working

curl -XDELETE 'https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/' -u "admin:Password"

{"acknowledged":true,"shards_acknowledged":true,"index":"spectrum_guide_rollout_zuk_2018_05_29"}

    Delete the index again to test hive from the beginning

Hive> insert into table es_spec_guide_rollout_zuk_test values ('zukel','lance',"2018_05_29");

no permissions for [indices:admin/refresh] and User [name=elasticview, roles=[elasticview], requestedTenant=null]

    Failing again - when by all logical reasoning this should be working

curl 'https://es_node:9200/_cat/indices' --key "/etc/pki/tls/private/esadmin.key" --cert "/etc/pki/tls/private/esadmin.crt:Pl@f0rmPiNiFi" | grep spec

green open  spectrum_guide_rollout_zuk_2018_05_29

    Out of curiosity check to see if index was created

By all logical reasoning, following the documentation with the above configuration this should be working, I gave the elasticview the same permissions the admin user has, except narrowed down to only any index starting with "spectrum_guide_" The admin user works, The elasticview user does not work when the index is restricted.

[Jochen Kressin]

I know I'm a bit late to the game, but I think your last observations help to narrow down the issue. I think that this is unrelated the REST API or the permission settings in SG, but more to Hadoop:

I have created the elasticview role, the corresponding user and the roles mapping as described by you:

curl -u ...  -H "Content-Type: application/json" -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -d '{"cluster":["CLUSTER_ALL"],"indices":{"spectrum_guide_rollout_*":{"*":["UNLIMITED"]}}}'

This should give the user all access to any index spectrum_guide_rollout_* 

Then - similar to you - I tried the access permission in isolation, means without Hadoop. With the elasticview user I am able to create an index like:

curl -u elasticview:password -XPUT 'https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/' 

And I am also able to perform the indices/refresh command that Hadoop complains about:

curl -u elasticview:password -XPOST 'https://es_node:9200/spectrum_guide_rollout_zuk_2018_05_29/_refresh' 

From this observation, my conclusion is that the user and permissions are working. Let's put Hadoop into the picture: It seems to work when you use the admin role, which has access to all indices. But it fails when you use the elasticview user. I suspect that Hadoop is submitting some ES calls involving other indices than the spectrum_guide_* index, maybe uses a wildcard or something like that.

To pinpoint we need to see what query Hadoop is issuing and which indices are affected. The easiest way would be to:

1) Set the log level for Search Guard to debug

1.1) This is described here: https://docs.search-guard.com/latest/troubleshooting-tls

2) Set your Hadoop table to use the elasticview user

3) Issue the (failing): Hive> insert into table es_spec_guide_rollout_zuk_test values ('zukel','lance',"2018_05_29");

After you execute the failing call you should see an entry in the logfile stating which request has been issued, which indices were affected and what permissions were missing.

Please post this logfile so we can analyze further.

[Lance Zukel]

After thinking about what you indicated in your post, I decided to try the following:

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -H "Content-Type: application/json" -d '{"cluster":["UNLIMITED"],"tenants":{"adm_tenant":"RW","test_tenant":"RW"},"indices":{"spectrum_guide_*":{"*":["UNLIMITED"]},"*":{"*":["indices:admin/*"]}}}'              <---This works

    However, I am concerned about the permissions that indices:admin/* has

Next I tried: 

curl -XPUT 'https://es_node:9200/_searchguard/api/roles/elasticview' -H "Content-Type: application/json" -d '{"cluster":["UNLIMITED"],"tenants":{"adm_tenant":"RW","test_tenant":"RW"},"indices":{"spectrum_guide_*":{"*":["UNLIMITED"]},"*":{"*":["indices:admin/refresh"]}}}'     <----This does not work

    This is the direction I need to be going with only the required access, I also tried several other single use permissions none of which worked

    This returns: Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest:

null

In response to your post:

1) Set the log level for Search Guard to debug

1.1) This is described here: https://docs.search-guard.com/latest/troubleshooting-tls

2) Set your Hadoop table to use the elasticview user

3) Issue the (failing): Hive> insert into table es_spec_guide_rollout_zuk_test values ('zukel','lance',"2018_05_29");

1.  I will configure my test cluster with searchguard installed and enable debug - this is a 51 node production clsuter

2. I will test after the test cluster is configured I will create a new table using the elasticview user with the assumed appropriate role

Additional concerns:

1. I cannot find any documentation listing ALL Available -  cluster: indices: permissions (seems like this should be included in the searchguard docs, I have seen dozens of issues from a quick google search)

[Jochen Kressin]

So it's definitely the right direction and confirms that Hadoop is issuing additional wildcard queries, which cause the security exception. In order to see what queries are issued the ES logs are the right place to go. You should see something like (taken from a Kibana example):

2018-05-31T17:44:36,511][INFO ][c.f.s.c.PrivilegesEvaluator] No index-level perm match for User [name=hr_employee, roles=[kibanauser], requestedTenant=null] [IndexType [index=searchguard, type=*], IndexType [index=sg6-auditlog-2018.05.31, type=*]] [Action [[indices:data/read/search]]] [RolesChecked [sg_kibana_user, sg_own_index]]

[2018-05-31T17:44:36,511][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {sg_own_index=[IndexType [index=sg6-auditlog-2018.05.31, type=*], IndexType [index=searchguard, type=*]], sg_kibana_user=[IndexType [index=sg6-auditlog-2018.05.31, type=*], IndexType [index=searchguard, type=*]]}

Maybe it's sufficient to set up a simple one node cluster for testing? The basic goal is to see the exact request that Hadoop issues to debug the permission problems, running it on your 51 node prod cluster is probably not needed.

Regarding your permission settings: Can you try to use:

indices:admin/refresh*

Instead of the non-working

indices:admin/refresh

If this also does not work there must be some other indices:admin/* query that Hadoop issues.

Unfortunately, the list of available indices: and cluster: permissions changes from ES release to ES release, and there is also no official publication from Elasticsearch regarding this. That's why we recommend using the pre-defined action groups that ship with SG if possible. 

[Lance Zukel]

I was able to setup a 1 node cluster with debug enabled.

After making the request and searching through logs. HERE is what worked:

'{"cluster":["UNLIMITED"],"tenants":{"adm_tenant":"RW","test_tenant":"RW"},"indices":{"spectrum_guide_*":{"*":["UNLIMITED"]},"*":{"*":["indices:admin/refresh*","indices:admin/create*"]}}}'

Thanks for your help