Permissions issue with indices:data/write/bulk[s]

[la...@usermind.com]

* Search Guard and Elasticsearch version

{

  "name" : "vK2vBkK",

  "cluster_name" : "elasticsearch",

  "cluster_uuid" : "t_EwPLkSRDWMrUAcaae9Uw",

  "version" : {

    "number" : "5.3.0",

    "build_hash" : "3adb13b",

    "build_date" : "2017-03-23T03:31:50.652Z",

    "build_snapshot" : false,

    "lucene_version" : "6.4.1"

  },

  "tagline" : "You Know, for Search"

}

search-guard-5-5.3.0-11.jar

* JVM version and operating system version

java version "1.8.0_121"

Java(TM) SE Runtime Environment (build 1.8.0_121-b13)

Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

* Number of nodes in your cluster

3

* Description of the bug

After upgrading to 5.3.0 we started getting errors on our ES servers for permissions errors for data/write/bulk[s] when logging in from our Kibana servers:

[2017-04-06T23:57:08,636][INFO ][c.f.s.c.PrivilegesEvaluator] No perm match for User [name=user, roles=[]] [IndexType [index=.kibana-367, type=*]] [Action [indices:data/write/bulk[s]]] [RolesChecked [sg_kibana_optimizely, sg_public]]

Here is an example of the permissions set in sg_roles.yml.

 sg_kibana_user:

  cluster:

    - '*'

    - indices:data/write/bulk* <-- Added after finding an article that seemed related but no change in behavior

  indices:

    '?kibana-367':

      'index-pattern':

        - KIBANA_INDEX_PATTERNS

      '*':

        - KIBANA_INDEX_OTHER

    '367-*':

      '*':

        - SPARK_ORG_USER

        - READ

        - SEARCH

        - indices:admin/mappings/fields/get*

    'travelers-367-*':

      '*':

        - SPARK_ORG_USER

        - READ

        - SEARCH

        - indices:admin/mappings/fields/get*

Unsure if it's related but there was an ES crash before this started happening.  I tried to open an issue with ES as well but they closed it immediately citing lack of information.

[2017-04-06T20:10:15,489][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[MkZ0lPb][bulk][T#1]], exiting

java.lang.StackOverflowError: null

I don't have much else to give you.  There's nothing else in the logs that seems related or interesting.

[SG]

Can you please post your complete sg_roles.yml and sg_action_groups.yml?

Is the ES crash reproducible? (I guess thats the github issue: https://github.com/elastic/elasticsearch/issues/23955)

Elasticsearch does have a breaking change in 5.3 regarding how index is handled, see also https://github.com/elastic/elasticsearch/pull/22812
and https://groups.google.com/d/msg/search-guard/pgwf1VsUL2s/jYfL7dFnAgAJ

> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/ed478e17-65e5-4bf8-937f-f31ab2694a62%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Search Guard]

do you use regex patterns somewhere? seems the crash (caused by a stackoverflow) is related to regex pattern matching

[2017-04-06T20:10:15,489][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[MkZ0lPb][bulk][T#1]], exiting
java.lang.StackOverflowError: null
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) ~[?:1.8.0_121]
	at java.util.regex.Pattern$Loop.match(Pattern.java:4785) ~[?:1.8.0_121]
	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717) ~[?:1.8.0_121]
	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568) ~[?:1.8.0_121]
	at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777) ~[?:1.8.0_121]
	at java.util.regex.Pattern$Branch.match(Pattern.java:4604) ~[?:1.8.0_121]
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) ~[?:1.8.0_121]
	at java.util.regex.Pattern$Loop.match(Pattern.java:4785) ~[?:1.8.0_121]
	at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717) ~[?:1.8.0_121]
	at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568) ~[?:1.8.0_121]
	at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777) ~[?:1.8.0_121]
	at java.util.regex.Pattern$Branch.match(Pattern.java:4604) ~[?:1.8.0_121]
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) ~[?:1.8.0_121]

http://www.regular-expressions.info/catastrophic.html

[la...@usermind.com]

I've attached the roles and action groups files.  I'm discussing your other questions with my coworkers.  Appreciate the quick response.

[la...@usermind.com]

Appreciate you taking a look.  We have rolled back to 5.2.2 after three consecutive crashes.  I'll see if we can reproduce in a test environment.