ES authentication fails in kibana

948 views
Skip to first unread message

Peter Horvath

unread,
Apr 18, 2018, 4:41:40 PM4/18/18
to Search Guard Community Forum
Hi,

I have a kibana installed with searchguard in kubernetes, and a 3 node ES cluster on VMs
I am using the official kibana-oss docker image. There is an kubernetes nginx ingress in front of kibana interface 

- ES cluster works perfectly.
- Application with the defined usernam/password for them working nicely
- Curl with any user including kibanaserver works for the role defined things
- On kibana interface i can login with the username/password of any user defined with sgadmin on the ES cluster

My problem is that i get the following errors
Error from kibana log:
{"type":"log","@timestamp":"2018-04-18T20:11:53Z","tags":["status","plugin:elasti...@6.2.2","error"],"pid":8,"state":"red","message":"Status changed from yellow to red - Authentication Exception","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
Error from ES:
[2018-04-18T20:18:07,559][WARN ][c.f.s.a.BackendRegistry  ] Authentication finally failed for null
[2018-04-18T20:18:10,541][WARN ][c.f.s.a.BackendRegistry  ] Authentication finally failed for null

How can searchguard login work on the kibana interface if kibana itself can't connect to the ES cluster where the users are defined?

Versions:
plugin:kib...@6.2.2 Ready
plugin:elasti...@6.2.2 Authentication Exception
plugin:time...@6.2.2 Ready
plugin:searc...@6.2.2 Search Guard plugin initialised.
plugin:con...@6.2.2 Ready
plugin:met...@6.2.2 Ready


SearchGuard config

sg_config
searchguard:
  dynamic:
    authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: internal
      clientcert_auth_domain:
        transport_enabled: false
        order: 1
        http_authenticator:
          type: clientcert
          config:
            username_attribute: cn
          challenge: false
        authentication_backend:
          type: noop

sg_internal_user
kibanaserver:
  readonly: true
  hash: $2y$12$AoboyrVUI2A9y1yxOiyi5yx5Ni68eQLeo2BcSCNK16d6TboO

sg_roles
sg_kibana_server:
  readonly: true
  cluster:
      - CLUSTER_MONITOR
      - CLUSTER_COMPOSITE_OPS
      - cluster:admin/xpack/monitoring*
      - indices:admin/template*
  indices:
    '?kibana':
      '*':
        - INDICES_ALL
    '?reporting*':
      '*':
        - INDICES_ALL
    '?monitoring*':
      '*':
        - INDICES_ALL

sg_roles_mapping
sg_kibana_server:
  readonly: true
  users:
    - kibanaserver

Kibana config:
    server.host: "0.0.0.0"
    searchguard.cookie.secure: true
    searchguard.cookie.password: "xxxxxxxxxxxx"
    logging.verbose: false
    elasticsearch.ssl.verificationMode: none
    elasticsearch.username: "kibanaserver"
    elasticsearch.password: "xxxxxxxxxxx"

ES config:
network.bind_host: 0.0.0.0
searchguard.enterprise_modules_enabled: false
searchguard.ssl.transport.pemcert_filepath: certs/eurwebstageghes01.pem
searchguard.ssl.transport.pemkey_filepath: certs/eurwebstageghes01.key
searchguard.ssl.transport.pemkey_password: xxxxx
searchguard.ssl.transport.pemtrustedcas_filepath: certs/ca.pem
searchguard.ssl.transport.enforce_hostname_verification: true
searchguard.ssl.transport.resolve_hostname: true
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certs/eurwebstageghes01.pem
searchguard.ssl.http.pemkey_filepath: certs/eurwebstageghes01.key
searchguard.ssl.http.pemkey_password: xxxxx
searchguard.ssl.http.pemtrustedcas_filepath: certs/ca.pem
searchguard.nodes_dn:
- CN=eurwebstageghes01.eurweb.xxxx.com,OU=XXX,O=XXX,DC=xxx,DC=com
- CN=eurwebstageghes02.eurweb.xxxx.com,OU=XXX,O=XXX,DC=xxx,DC=com
- CN=eurwebstageghes03.eurweb.xxx.com,OU=XXX,O=XXX,DC=xxx,DC=com
searchguard.authcz.admin_dn:
- CN=see-admin,OU=XXXX,O=XXX,DC=xxxx,DC=com
searchguard.roles_mapping_resolution: MAPPING_ONLY
searchguard.restapi.roles_enabled: ["sg_all_access"]
searchguard.audit.enable_rest: true
searchguard.audit.resolve_bulk_requests: true
searchguard.audit.type: internal_elasticsearch

SG

unread,
Apr 19, 2018, 11:53:21 AM4/19/18
to search...@googlegroups.com
plse see

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/search-guard/ZQ_-SL1tQ9k/75chW1BCAAAJ
> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Peter Horvath

unread,
Apr 19, 2018, 1:10:27 PM4/19/18
to search...@googlegroups.com
Hi,

I've read that thread but it is nothing like my case.
If you check my full config you can see that searchguard login on kibana works only with the defined users not any random and i don't have certificates setup on kibana side.
My problem is that the ES cluster plugin can't connect

Peter

> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

SG

unread,
Apr 19, 2018, 1:34:22 PM4/19/18
to search...@googlegroups.com
Sorry for the wrong link

- remove the clientcert_auth_domain from sg_config.yml if you dont need them, reload config via sgadmin
- make sure that the password for kibanaserver is correct.
- enable debug logs like described here https://docs.search-guard.com/latest/troubleshooting-tls#setting-the-log-level-to-debug and post the logs
- please also post the complete kibana.yml as file attachment
> > To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> > To post to this group, send email to search...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAO9xhubo7pUph74OjxrX-KW2F-gJhHEtnKjWseLvfu_9un8QZw%40mail.gmail.com.

Peter Horvath

unread,
Apr 19, 2018, 4:15:01 PM4/19/18
to search...@googlegroups.com
Thanks! the debug mode was very useful.
My kibana config was that much i posted by the way.
I've tried it with " " encapsulation and without both yields the same
Kibana config:
    server.host: 0.0.0.0
    searchguard.cookie.secure: true
    searchguard.cookie.password: xxxxxxxxxxxx
    logging.verbose: false
    elasticsearch.ssl.verificationMode: none
    elasticsearch.username: kibanaserver
    elasticsearch.password: xxxxxxxxxxx



So if i cat the kibana config on the kibana server and i copy paste the kibana username and password and use curl i can check health of the cluster.

elasticsearch_1  | [2018-04-19T20:06:48,971][DEBUG][c.f.s.a.BackendRegistry  ] User 'User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null]' is authenticated
elasticsearch_1  | [2018-04-19T20:06:48,971][DEBUG][c.f.s.a.BackendRegistry  ] sgtenant 'null'
elasticsearch_1  | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] ### evaluate permissions for User [name=kibanaserver, roles=[kibanaserver], requestedTenant=null] on S6VgCOD
elasticsearch_1  | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] requested cluster:monitor/health from 10.66.3.25:34102
elasticsearch_1  | [2018-04-19T20:06:48,971][DEBUG][c.f.s.c.PrivilegesEvaluator] Resolve [] from class org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest for action cluster:monitor/health
elasticsearch_1  | [2018-04-19T20:06:48,972][DEBUG][c.f.s.c.PrivilegesEvaluator]   found a match for 'sg_kibana_server' and cluster:monitor/health, skip other roles

But the kibana server still ends up triggering a password missmatch:

elasticsearch_1  | [2018-04-19T20:06:49,733][DEBUG][c.f.s.a.BackendRegistry  ] Try to extract auth creds from basic http authenticator
elasticsearch_1  | [2018-04-19T20:06:49,733][DEBUG][c.f.s.a.BackendRegistry  ] kibanaserver not cached, return from internal backend directly
elasticsearch_1  | [2018-04-19T20:06:50,207][DEBUG][c.f.s.a.BackendRegistry  ] Can not authenticate kibanaserver due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]
elasticsearch_1  | com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]
elasticsearch_1  |      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2218) ~[guava-23.0.jar:?]
elasticsearch_1  |      at com.google.common.cache.LocalCache.get(LocalCache.java:4147) ~[guava-23.0.jar:?]


Any idea?

> > To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

> > To post to this group, send email to search...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

Peter Horvath

unread,
Apr 19, 2018, 5:46:52 PM4/19/18
to Search Guard Community Forum
I've also tried to upgrade the kibana stack to 6.2.3 and changed the password to short simple one in the hope that it might be too long or too complex.
None of them helped curl works kibana es plugin doesn't.

Jochen Kressin

unread,
Apr 19, 2018, 9:43:33 PM4/19/18
to Search Guard Community Forum
This seems pretty strange.

From your first post I see that the kibanaserver user cannot connect to ES/SG:

plugin:elasti...@6.2.2 Authentication Exception

Which is also resembled in this error here:

elasticsearch_1  | [2018-04-19T20:06:50,207][DEBUG][c.f.s.a.BackendRegistry  ] Can not authenticate kibanaserver due to com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]
elasticsearch_1  | com.google.common.util.concurrent.UncheckedExecutionException: ElasticsearchSecurityException[password does not match]

So one might think this is just a password error. If the password works with curl, the only possibility that comes to my mind would be that the pwd is somehow trashed when it is transmitted from Kibana to ES/SG. So one thing you can do is to use the developer tools and having a look at the HTTPS calls on login. You should see one POST to:

/api/v1/auth/login

Can you check that the password used here is 1:1 the password you also use in your curl call?

Peter Horvath

unread,
Apr 19, 2018, 9:46:55 PM4/19/18
to search...@googlegroups.com
I actually did the curl from the kibana host and 1:1 copy pasted the password from the kibana.yml and curl worked. I can as well login on the kibana gui with it. Kibana elasticsearch plugin is the only one failing.

> > To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

> > To post to this group, send email to search...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

Peter Horvath

unread,
Apr 23, 2018, 7:17:53 AM4/23/18
to search...@googlegroups.com
Just to conclude this thread. 
The problem was a leftover ENV var which overwrote the password.

> > To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

> > To post to this group, send email to search...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/06657b9c-6388-48a3-a3db-5dcd8109c368%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4E6A4BF2-0DDC-4F8C-9334-F17A161EB4AB%40search-guard.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

Search Guard

unread,
Apr 23, 2018, 9:38:01 AM4/23/18
to Search Guard Community Forum
so is this thread resolved?

Peter Horvath

unread,
Apr 23, 2018, 9:40:18 AM4/23/18
to search...@googlegroups.com
yes thank you

On 23 April 2018 at 09:38, Search Guard <in...@search-guard.com> wrote:
so is this thread resolved?

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/5LHmSK1jjNU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages