SearchGuard 5.x with LDAP authentication issues

[Brian Sanders]

I am testing out the LDAP authentication module with Kibana multi tenancy to see if this will fit our needs, and I am having some strange login issues.  I believe I have configured it as described in the example documentation, but the FIRST time I attempt to log in I always end up at the 500 internal server error.  If I attempt to log in a few more times, eventually it works and seems to keep working for a bit.  I have also logged in and then seen this issue when switching tenants.  After logging back in a second time everything works including switching.

I don't really know how to proceed troubleshooting this, but I can't demo this to anyone and expect them to want me to take this live to production if LDAP is flaky.  

I am still using the basic_internal_auth, since I don't want to add a kibanaserver or logstash user to my LDAP directory, and would prefer those just be in the local database.  Authz is being done by mapping users to rolls, which does work once I am logged in.

Here is the relevant section from my sg_config.yml 

searchguard:

  dynamic:

    kibana:

      multitenancy_enabled: true

      server_username: "kibanaserver"

      index: '.kibana'

    http:

      anonymous_auth_enabled: false

      xff:

        enabled: false

        remoteIpHeader:  'x-forwarded-for'

        proxiesHeader:   'x-forwarded-by'

    authc:

      basic_internal_auth_domain:

        enabled: true

        order: 4

        http_authenticator:

          type: basic

          challenge: false

        authentication_backend:

          type: intern

      ldap:

        enabled: true

        order: 1

        http_authenticator:

          type: basic

          challenge: true

        authentication_backend:

          config:

            enable_ssl: false

            enable_start_tls: true

            enable_ssl_client_auth: false

            verify_hostnames: false

            hosts:

              - ipa1.example.com:389

            bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'

            password: 'FAKEPASS'

            userbase: 'cn=users,cn=accounts,dc=example,dc=com'

            usersearch: '(uid={0})'

            username_attribute: uid

            skip_users:

              - kibanaserver

              - logstash

[SG]

Which ES/SG/LDAP/MT versions are you using?

Does it work without LDAP (internal users only)?

> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9eba0f76-cbbe-4ea7-85f2-a2a90d750f26%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Brian Sanders]

Sorry I just saw this question, I never had the issue with the local only accounts.  I didn't test that extensively as I was looking to test the LDAP, but it seemed that it worked just fine.  

We are running the current versions:

ES: 5.4.2
kibana-multitenancy-5.4-3
authbackend-ldap-5.0-7

And after looking these up it would appear there is now a 5.5 version of elasticsearch.  This being in dev, I can always attempt to upgrade, but these were all the latest available versions as of sometime last week when I was starting my build.

[Brian Sanders]

Seems I didn't include the Search Guard version: search-guard-5:5.4.2-12

[Search Guard]

can you try this?

searchguard:

  dynamic:

    kibana:

      multitenancy_enabled: true

      server_username: "kibanaserver"

      index: '.kibana'

    http:

      anonymous_auth_enabled: false

      xff:

        enabled: false

        remoteIpHeader:  'x-forwarded-for'

        proxiesHeader:   'x-forwarded-by'

    authc:

      basic_internal_auth_domain:

        enabled: true

        order: 0

        http_authenticator:

          type: basic

          challenge: false

        authentication_backend:

          type: intern

      ldap:

        enabled: true

        order: 1

        http_authenticator:

          type: basic

          challenge: false

        authentication_backend:

          config:

            enable_ssl: false

            enable_start_tls: true

            enable_ssl_client_auth: false

            verify_hostnames: false

            hosts:

              - ipa1.example.com:389

            bind_dn: 'uid=admin,cn=users,cn=accounts,dc=example,dc=com'

            password: 'FAKEPASS'

            userbase: 'cn=users,cn=accounts,dc=example,dc=com'

            usersearch: '(uid={0})'

            username_attribute: uid

            

            #only valid for authz (not authc)

            #skip_users:

            #  - kibanaserver

            #  - logstash

(more complete example here https://gist.github.com/floragunncom/fdad5c8f1cab838962842d590c408538)

[Brian Sanders]

This is a configuration I have tried previously, but I tried again this morning before logging in to anything to document the exact order of tests and errors seen.

When I first log in, I do see the log message that it can not authenticate the user with ad 0 (the internal database) and it says it says it is going to the next (eg "try next")  However after an extended period the attempted login just shows

{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

After that shows up, I simply reselect my URL and press enter again, which this time loads Kibana with out any further prompting for user input.  I can perform searches on my data an things appear normal.  If I select tenants tab, and change to another tenant, I receive the same:

{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

This time pressing enter to reload the page eventually brings me back to the Search Guard login screen.  After providing my ldap credentials, I am finally authentiacted and put into Kibana as the tenant.  Now I can switch between the 2 tenants just fine... but if I select yet another tenant it takes a long time and eventually puts me at the Search Guard login website, at least it didn't 500 error this time.  

So it would seem that first login does this, as well as having issues with switching to a tenant for the first time as well.  I am using FreeIPA as an ldap backend, which is a fairly popular user management group of packages, but as far as this is concerned, it is simply authing against the LDAP portion which clearly CAN work.  It just doesn't seem to work on the first try.

[Search Guard]

I verified that the configuration i sent was is working locally for me.

I suggest to reinstall Kibana and the SG Kibana plugin https://github.com/floragunncom/search-guard-kibana-plugin/releases/download/v5.4.2-3/searchguard-kibana-5.4.2-3.zip

The relevant config in kibana.yml (maybe you miss that) looks like:

elasticsearch.requestHeadersWhitelist: ["sg_tenant", "x-authenticated-user", "authorization", "x-forwarded-for", "x-forwarded-server", "x-forwarded-by", "x-proxy-user", "x-proxy-roles"]

searchguard.multitenancy.enabled: true

searchguard.multitenancy.show_roles: true

searchguard.multitenancy.enable_filter: true

searchguard.multitenancy.tenants.enable_global: true

searchguard.multitenancy.tenants.enable_private: true

searchguard.cookie.password: "<long secure string>"

elasticsearch.username: "kibanaserver"

elasticsearch.password: "kibanaserver"

see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md#kibana-plugin-configuration

[Brian Sanders]

I'll have to try reinstalling, perhaps I'll upgrade to the most current versions of plugins as well since this isn't live and there are new versions.

From your suggested settings in kibana.yml
1) what is searchguard.cookie.password  I haven't seen mention of that before

I have the other settings, and some of those are defaults so I omitted them.  I will specificaly add them and see if this makes much difference before trying to re-install.

[SG]

Regarding "searchguard.cookie.password" pls read https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md#configuring-the-search-guard-kibana-plugin
(at least in production you should not use the default)

If its still not working pls. turn debug logging on and send the logfile from elasticsearch:

PUT /_cluster/settings
{
"transient" : {
"logger._root" : "DEBUG"

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d487734a-1beb-4a13-93d1-4686a32fc203%40googlegroups.com.

[Brian Sanders]

Thank you, I will activate this logging and see what I can get.  I tried all the settings above with no difference.  I also tried Firefox (was using chrome) with the hope that it was browser related, but it still does the same thing.  I'll see if I c

[Brian Sanders]

I have a log file, is there some way to send it other than attaching it here?  I don't know exactly what all is in it, but I generally don't like posting debug logs to the internet directly.

[SG]

Why not write a PGP encrypted mail? Here is our public key https://floragunn.com/search-guard-public-key/

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4c3d66b4-bdd4-4616-93a4-7fe9ea926f6c%40googlegroups.com.

[Brian Sanders]

Yup, that is a good suggestion.  Log file attached.

If it helps, my users that fails then succeeds is "brians", and I have the global tenant turned off but the private tenant turned on.  I have also added a few other tenants to his user to test switching.

[Brian Sanders]

Any chance someone has been able to find anything in the logs?  It very consistently fails the first login, but then after that seems to work.  I have been working on getting alerting in place along side this with elastalert, and I think between your plugin and elastalert this could work for us.  I just really have problems demoing the setup with login/tenant switching taking 30s due to the timeout.  I have avoided upgrading to 5.5, as I don't want to invalidate the logs I captured if someone finds something telling in them.

[Jochen Kressin]

Hi Brian,

we've analyzed the log files and tried to replicate the problem here, but no luck so far. The log file also looks fine, I can't find anything regarding the login or the internal server error.

The first login attempt starts here:

[2017-07-13T13:58:56,298][DEBUG][c.f.s.a.BackendRegistry  ] Try to extract auth creds from http basic 

[2017-07-13T13:58:56,298][DEBUG][c.f.s.a.BackendRegistry  ] User 'XXX' is in cache? false (cache size: 1)

... LDAP stuff ...

[2017-07-13T13:58:56,466][DEBUG][c.f.d.a.l.b.LDAPAuthenticationBackend] Authenticated username brians

[2017-07-13T13:58:56,468][DEBUG][c.f.s.a.BackendRegistry  ] User 'User [name=XXX, roles=[]]' is authenticated

which means that the authentication process takes around 150ms. Which is still a bit high, but depends on network latency and LDAP server performance. So, this seems fine. After that, the user is cached (default is 1 hour), so even the LDAP server should not have any impact anymore:

[2017-07-13T13:58:56,609][DEBUG][c.f.s.a.BackendRegistry  ] Try to extract auth creds from http basic 

[2017-07-13T13:58:56,609][DEBUG][c.f.s.a.BackendRegistry  ] User 'XXX' is in cache? true (cache size: 2)

[2017-07-13T13:58:56,610][DEBUG][c.f.s.a.BackendRegistry  ] User 'User [name=XXX, roles=[]]' is authenticated

Cache size seems ok, since both your user and the Kibana server user is cached. Also, there are no Exceptions in the log, which I would have expected since Kibana logs an 500 internal server error.

So, at the moment I can't see anything in the ES log that seems related to the issues you're facing. Also, LDAP is a major use case for Search Guard, and we did not hear about such a behaviour before. Therefore I guess the problem is not related to the LDAP module at the moment. But just to make sure:

- If you only use the internal user database and disable LDAP, do you still see these issues?

- Apart from Search Guard, do you have any other ES plugins installed

- Apart from the SG Kibana plugin, do you have any other Kibana plugins installed

- Are there any other services running against ES?

In other words - can you replicate the problem with a minimum installation of ES/SG and KI/SG Kibana plugin?

Next, let's look at the actual HTTP requests from Kibana. Can you please:

- Delete all log files

- Fire up ES and KI

- In Chrome, open the Developer tools and switch to the Network tab

- Replicate the login problem

In the network tab, can you spot which request is actually taking so long? If you find anything there, please post (or email) the request incl. headers and payload, and the reponse? And send the ES log file as well?

Sorry for the delayed response, but since we can't reproduce the problem we're having a hard time pinpointing the problem.

[Brian Sanders]

Thanks for the detailed reply!  I'll quickly answer what I can and get back to you as soon as I can get these tests run.

-I do not have any other ES plugins (not even the X-pack free version)
-Nothing else in SG plugins besides the tenants
-logstash is pushing into elasticsearch.  I have Cerebro for checking index status, but that is now a web app and no longer a plugin.


I'll try local only and also get back to you with the test results you suggested.

So is there some guaranteed way I can clear the Cache?  I am assuming that once cached I am good, so if I could do something to clear the cache and force the issue again I could test much quicker.

[Jochen Kressin]

Yes - the cache is cleared each time you either upload new configurations, or reload the existing configuration.

Reloading the existing configuration can be done with the -rl/--reload flag like:

./sgadmin.sh -ts ... -tspass ... -ks ... -kspass ... -icl -nhnv -rl 

[Brian Sanders]

I was about to reply with tests results, when I discovered that doing the reload or even editing and changing the configuration, doesn't seem to cause the issue to re-surface as I was assuming it would.  I will have to let it sit for a while and test again after lunch.

[Brian Sanders]

I was able to see the issue again this morning after clearing logs from kibana and elasticsearch.  I captured all the data from the chrome developer tools and have exported them (it seems to be a .har file json formatted).  I have attached all the logs to this with the searchguard gpgkey encryption again.

For this test I did get the error, and developer tools shows the timeout of 30s before I get the 500 internal error.  I then left the capture running, and logged in a second time.  This second attempt works.  So this log should contain both the failed 1st and then successful 2nd login attempt.  I did not then attempt switching tenants (which generally gives the same issue the first time).

I can try disabeling LDAP module all together, but if clearing the cache doesn't cause the error on demand, it will be difficult for me to say that disabeling it works, or if it just works because I have already logged in recently. 

[Jochen Kressin]

Hi Brian,

I have a couple of questions regarding the logs - since I don't want to post your hostnames and IPs and other internal content here, can we move this over to email? I'd post the end results here for other users once we figured out the problem.

[Search Guard]

This should now be fixed with version 4 of the multitenancy module

ES 5.4/5.5
http://search.maven.org/remotecontent?filepath=com/floragunn/dlic-search-guard-module-kibana-multitenancy/5.4-4/dlic-search-guard-module-kibana-multitenancy-5.4-4-jar-with-dependencies.jar

ES 5.3
http://search.maven.org/remotecontent?filepath=com/floragunn/dlic-search-guard-module-kibana-multitenancy/5.3-4/dlic-search-guard-module-kibana-multitenancy-5.3-4-jar-with-dependencies.jar

Relates https://github.com/floragunncom/search-guard-kibana-plugin/issues/34