ES authentication fails in kibana

[Peter Horvath]

Hi,

I have a kibana installed with searchguard in kubernetes, and a 3 node ES cluster on VMs

I am using the official kibana-oss docker image. There is an kubernetes nginx ingress in front of kibana interface 

- ES cluster works perfectly.

- Application with the defined usernam/password for them working nicely

- Curl with any user including kibanaserver works for the role defined things

- On kibana interface i can login with the username/password of any user defined with sgadmin on the ES cluster

My problem is that i get the following errors

Error from kibana log:

{"type":"log","@timestamp":"2018-04-18T20:11:53Z","tags":["status","plugin:elasti...@6.2.2","error"],"pid":8,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

Error from ES:

[2018-04-18T20:18:07,559][WARN ][c.f.s.a.BackendRegistry  ] Authentication finally failed for null

[2018-04-18T20:18:10,541][WARN ][c.f.s.a.BackendRegistry  ] Authentication finally failed for null

How can searchguard login work on the kibana interface if kibana itself can't connect to the ES cluster where the users are defined?

Versions:

plugin:kib...@6.2.2	Ready
plugin:elasti...@6.2.2	Authentication Exception
plugin:time...@6.2.2	Ready
plugin:searc...@6.2.2	Search Guard plugin initialised.
plugin:con...@6.2.2	Ready
plugin:met...@6.2.2	Ready

SearchGuard config

sg_config

searchguard:

  dynamic:

    authc:

      basic_internal_auth_domain:

        http_enabled: true

        transport_enabled: true

        order: 0

        http_authenticator:

          type: basic

          challenge: true

        authentication_backend:

          type: internal

      clientcert_auth_domain:

        transport_enabled: false

        order: 1

        http_authenticator:

          type: clientcert

          config:

            username_attribute: cn

          challenge: false

        authentication_backend:

          type: noop

sg_internal_user

kibanaserver:

  readonly: true

  hash: $2y$12$AoboyrVUI2A9y1yxOiyi5yx5Ni68eQLeo2BcSCNK16d6TboO

sg_roles

sg_kibana_server:

  readonly: true

  cluster:

      - CLUSTER_MONITOR

      - CLUSTER_COMPOSITE_OPS

      - cluster:admin/xpack/monitoring*

      - indices:admin/template*

  indices:

    '?kibana':

      '*':

        - INDICES_ALL

    '?reporting*':

      '*':

        - INDICES_ALL

    '?monitoring*':

      '*':

        - INDICES_ALL

sg_roles_mapping

sg_kibana_server:

  readonly: true

  users:

    - kibanaserver

Kibana config:

    server.host: "0.0.0.0"

    server.name: "gh-kibana-stage.eur.xxx.com"

    searchguard.cookie.secure: true

    searchguard.cookie.password: "xxxxxxxxxxxx"

    logging.verbose: false

    elasticsearch.ssl.verificationMode: none

    elasticsearch.url: "https://eurwebstageghes01.eurweb.xxx.com:9200"

    elasticsearch.username: "kibanaserver"

    elasticsearch.password: "xxxxxxxxxxx"

ES config:

network.bind_host: 0.0.0.0

searchguard.enterprise_modules_enabled: false

searchguard.ssl.transport.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.transport.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.transport.pemkey_password: xxxxx

searchguard.ssl.transport.pemtrustedcas_filepath: certs/ca.pem

searchguard.ssl.transport.enforce_hostname_verification: true

searchguard.ssl.transport.resolve_hostname: true

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: certs/eurwebstageghes01.pem

searchguard.ssl.http.pemkey_filepath: certs/eurwebstageghes01.key

searchguard.ssl.http.pemkey_password: xxxxx

searchguard.ssl.http.pemtrustedcas_filepath: certs/ca.pem

searchguard.nodes_dn:

- CN=eurwebstageghes01.eurweb.xxxx.com,OU=XXX,O=XXX,DC=xxx,DC=com

- CN=eurwebstageghes02.eurweb.xxxx.com,OU=XXX,O=XXX,DC=xxx,DC=com

- CN=eurwebstageghes03.eurweb.xxx.com,OU=XXX,O=XXX,DC=xxx,DC=com

searchguard.authcz.admin_dn:

- CN=see-admin,OU=XXXX,O=XXX,DC=xxxx,DC=com

searchguard.roles_mapping_resolution: MAPPING_ONLY

searchguard.restapi.roles_enabled: ["sg_all_access"]

searchguard.audit.enable_rest: true

searchguard.audit.resolve_bulk_requests: true

searchguard.audit.type: internal_elasticsearch

[SG]

plse see

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/search-guard/ZQ_-SL1tQ9k/75chW1BCAAAJ

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Peter Horvath]

Hi,

I've read that thread but it is nothing like my case.

If you check my full config you can see that searchguard login on kibana works only with the defined users not any random and i don't have certificates setup on kibana side.

My problem is that the ES cluster plugin can't connect

Peter

> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.

[SG]

Sorry for the wrong link

- remove the clientcert_auth_domain from sg_config.yml if you dont need them, reload config via sgadmin
- make sure that the password for kibanaserver is correct.
- enable debug logs like described here https://docs.search-guard.com/latest/troubleshooting-tls#setting-the-log-level-to-debug and post the logs
- please also post the complete kibana.yml as file attachment

> > To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

> > To post to this group, send email to search...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.
> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAO9xhubo7pUph74OjxrX-KW2F-gJhHEtnKjWseLvfu_9un8QZw%40mail.gmail.com.

[Peter Horvath]

Thanks! the debug mode was very useful.

My kibana config was that much i posted by the way.

I've tried it with " " encapsulation and without both yields the same

Kibana config:

    server.host: 0.0.0.0

    server.name: gh-kibana-stage.eur.xxx.com

    searchguard.cookie.secure: true

    searchguard.cookie.password: xxxxxxxxxxxx

    logging.verbose: false

    elasticsearch.ssl.verificationMode: none

    elasticsearch.url: https://eurwebstageghes01.eurweb.xxx.com:9200

    elasticsearch.username: kibanaserver

    elasticsearch.password: xxxxxxxxxxx

So if i cat the kibana config on the kibana server and i copy paste the kibana username and password and use curl i can check health of the cluster.

elasticsearch_1  | [2018-04-19T20:06:48,971][DEBUG][c.f.s.a.BackendRegistry  ] User 'User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null]' is authenticated

elasticsearch_1  | [2018-04-19T20:06:48,971][DEBUG][c.f.s.a.BackendRegistry  ] sgtenant 'null'

elasticsearch_1  | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] ### evaluate permissions for User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null] on S6VgCOD

elasticsearch_1  | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] requested cluster:monitor/health from 10.66.3.25:34102

elasticsearch_1  | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] Resolve [] from class org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest for action cluster:monitor/health

elasticsearch_1  | [2018-04-19T20:06:48,972][DEBUG][c.f.s.c.PrivilegesEvaluator]   found a match for 'sg_kibana_server' and cluster:monitor/health, skip other roles

But the kibana server still ends up triggering a password missmatch:

elasticsearch_1  | [2018-04-19T20:06:49,733][DEBUG][c.f.s.a.BackendRegistry  ] Try to extract auth creds from basic http authenticator

elasticsearch_1  | [2018-04-19T20:06:49,733][DEBUG][c.f.s.a.BackendRegistry  ] kibanaserver not cached, return from internal backend directly

elasticsearch_1  | [2018-04-19T20:06:50,207][DEBUG][c.f.s.a.BackendRegistry  ] Can not authenticate kibanaserver due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1  | com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1  |      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2218) ~[guava-23.0.jar:?]

elasticsearch_1  |      at com.google.common.cache.LocalCache.get(LocalCache.java:4147) ~[guava-23.0.jar:?]

Any idea?

> > To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

> > To post to this group, send email to search...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAO9xhubo7pUph74OjxrX-KW2F-gJhHEtnKjWseLvfu_9un8QZw%40mail.gmail.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/45CB4889-26F1-4755-BB3D-59665CCD2FC0%40search-guard.com.

[Peter Horvath]

I've also tried to upgrade the kibana stack to 6.2.3 and changed the password to short simple one in the hope that it might be too long or too complex.

None of them helped curl works kibana es plugin doesn't.

[Jochen Kressin]

This seems pretty strange.

From your first post I see that the kibanaserver user cannot connect to ES/SG:

plugin:elasti...@6.2.2

Authentication Exception

Which is also resembled in this error here:

elasticsearch_1  | [2018-04-19T20:06:50,207][DEBUG][c.f.s.a.BackendRegistry  ] Can not authenticate kibanaserver due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

elasticsearch_1  | com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

So one might think this is just a password error. If the password works with curl, the only possibility that comes to my mind would be that the pwd is somehow trashed when it is transmitted from Kibana to ES/SG. So one thing you can do is to use the developer tools and having a look at the HTTPS calls on login. You should see one POST to:

/api/v1/auth/login

Can you check that the password used here is 1:1 the password you also use in your curl call?

[Peter Horvath]

I actually did the curl from the kibana host and 1:1 copy pasted the password from the kibana.yml and curl worked. I can as well login on the kibana gui with it. Kibana elasticsearch plugin is the only one failing.

> > To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

> > To post to this group, send email to search...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAO9xhubo7pUph74OjxrX-KW2F-gJhHEtnKjWseLvfu_9un8QZw%40mail.gmail.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/45CB4889-26F1-4755-BB3D-59665CCD2FC0%40search-guard.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/de4d8cde-7acd-493d-a7a0-7d6e498fe237%40googlegroups.com.

[Peter Horvath]

Just to conclude this thread. 

The problem was a leftover ENV var which overwrote the password.

> > To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

> > To post to this group, send email to search...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAO9xhubo7pUph74OjxrX-KW2F-gJhHEtnKjWseLvfu_9un8QZw%40mail.gmail.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/45CB4889-26F1-4755-BB3D-59665CCD2FC0%40search-guard.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

[Search Guard]

so is this thread resolved?

[Peter Horvath]

yes thank you

On 23 April 2018 at 09:38, Search Guard <in...@search-guard.com> wrote:

so is this thread resolved?

--

You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/31489aef-1b84-4f8e-a12c-bc326ff73dd2%40googlegroups.com.