Config hot reloading not working from Elastic index

[Philipp Meier]

Hi,

In the documented concept of SearchGuard (https://github.com/floragunncom/search-guard-docs/blob/master/concepts.md#configuration-settings-the-search-guard-index) I read about the Elastic index to store SearchGuard users. For my understanding this documentation clearly strengthen the Elastic Index as single source of truth:

Keeping the configuration settings in an Elasticsearch index enables hot config reloading. This means that you can change any of the user-, role- and permission settings at runtime , without restarting your nodes. Configuration changes will take effect nearly immediately. You can load and change the settings from any machine which has access to your Elasticsearch cluster.

So I tested this behavior with inserting an additional user to the index directly:

1. Call the internal users
1. curl -k -s -X GET --cert /opt/searchguard/config/admin-all.pem --key /opt/searchguard/config/admin-key.pem 'https://127.0.0.1:27791/searchguard/internalusers/0?pretty'
2. Update the document with an additional user called "cust2" and the same password hash as user a previous inserted user (so that I'm sure the password works for the new user)
1. curl -k -s -X POST --cert /opt/searchguard/config/admin-all.pem --key /opt/searchguard/config/admin-key.pem 'https://127.0.0.1:27791/searchguard/internalusers/0/_update?pretty' -d '{"doc": {"internalusers": "eyJhZG1pbiI6eyJoYXNoIjoiJDJhJDEyJFZjQ0RnaDJORGswN0pHTjByakdiTS5BZDQxcVZSL1lGSmNnSHAwVUduczVKRHltdi4uVE9HIn0sImN1c3QiOnsiaGFzaCI6IiQyYSQxMiQ1M0pEZXNINkZTaTlTWWxvby9zcWVPM3czV0k3bjBCb1JoV2NKZjBWOEt6b2NwaXdnNmVYQyJ9LCJjdXN0MiI6eyJoYXNoIjoiJDJhJDEyJDUzSkRlc0g2RlNpOVNZbG9vL3NxZU8zdzNXSTduMEJvUmhXY0pmMFY4S3pvY3Bpd2c2ZVhDIn19"}}'
3. Add user to rolesmapping
1. curl -k -s -X POST --cert /opt/searchguard/config/admin-all.pem --key /opt/searchguard/config/admin-key.pem 'https://127.0.0.1:27791/searchguard/rolesmapping/0/_update?pretty' -d '{"doc": {"rolesmapping": "eyJzZ19hbGxfYWNjZXNzIjp7InVzZXJzIjpbImFkbWluIl19LCJzZ19jdXN0X2FjY2VzcyI6eyJ1c2VycyI6WyJjdXN0IiwiY3VzdDIiXX19"}}'
4. Try to login
1. curl -k -s -X GET --user cust2 'https://127.0.0.1:27791/_cat?pretty'

   Enter host password for user 'cust2':

   Unauthorized
   
   

So the login failed, even after a few minutes. When I restart the whole Elastic service, the login with the new user works as expected.

What do I miss here?

Thank and best regards,

Philipp

[Urs]

Hi Phillip,

if you want to make changes to the Elastic index you need to either use sgadmin or the REST Api to do this, do not use curle to directly manipulate the index!
Search Guard uses an internal cache and that is why changes will not be visible instantly.
The default setting for caching is currently 1 hour, with the next release SG15 the timeout will be adjustable. In the meantime please use sgadmin or the REST Api to flush the cache.

So, yes, SG Index is the single source of truth and sgadmin and the REST Api are the weapons of choice to make changes ;)
Hope this answer helps,

Urs

--
Urs Armbruster
Community Manager

floragunn GmbH
Tempelhofer Ufer 16
10963 Berlin

Amtsgericht Charlottenburg HRB 147010
USt-IdNr.: DE287373363


+++ like us on facebook: https://www.facebook.com/searchguard
+++ follow us on twitter: https://twitter.com/searchguard

[Adrian K.]

Hi Urs,

Since SG15 is now released I've tried setting searchguard.cache.ttl_minutes to 1 but I get the following error:

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [searchguard.cache.ttl_minutes] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

I'm using Elasticsearch 5.4.0 and search-guard-5-5.4.0-15. Any ideas what the problem could be?

Kind regards

Adrian

[SG]

According to our EOL policy the new features are only in >= 5.4.3

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/42846950-67b7-4ea3-be16-14e368da4db0%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Philipp Meier]

Hi,

thank you for your replies! I tested it today with the new cache feature but it doesn't address my issue because all the configuration is being kept in the field typeToConfig (https://github.com/floragunncom/search-guard/blob/es-5.4.3/src/main/java/com/floragunn/searchguard/configuration/IndexBaseConfigurationRepository.java#L58). So is there away to reload this Map typeToConfig? What's the purpose of the caches in BackendRegistry?

Thank you in advance,

Philipp

[SG]

There two different things here to distinguish:

- Backend Cache (which is now configurable since SG 15 and ES >= 5.4.3)
This cache exists to avoid hitting every backend for each auth request. Thats especially helpful with remote backend like LDAP where a network connection needs to be established.
So if you modifiy or delete a user in an auth backend it can took the configured ttl (default 60 min) until your changes are reflected within SG. But you can always force a invalidation of this cache (see below) or switch it off by setting ttl = 0

- The configuration cache of SG
Search Guard stores all of its configuration in a special index. Upon node startup it loads the config from this index into memory. If you change the contents in the index the changes are not automatically reflected into memory.
You need to issue a "configuration update" request to load the current config from the index into memory (and make it effective). This "update" does also clear the "Backend cache" discussed above.

You can trigger a "configuration update" (and Backend Cache invalidation):
- By restarting a node (but thats not a good solution is its solely to update the configs/clear backend cache)
- Run sgadmin with the -rl or --reload option
- Via the /_searchguard/api/cache REST endpoint (this needs the REST mngt. module to be installed: https://github.com/floragunncom/search-guard-docs/blob/master/managementapi.md)

We recommend to use sgadmin for config updates because its the default tool/prodecure and there is always a automatic config update/backend refresh executed.
If you need to do it via curl/HTTPS then use the REST management module/api: https://github.com/floragunncom/search-guard-rest-api

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d8ca9aa4-a5a7-4b26-8150-f9ecc59ca049%40googlegroups.com.

[Terry Quigley]

Hi 

What is the full name of the ttl property ?

Is there a property to turn off this cache completely ? Is any of this documented ?

I tried a couple of setting I found in forums and change requests but they are not recognized, e.g. searchguard.authentication.authentication_backend.cache.enable: false

[Terry Quigley]

I also just tried this:

https://github.com/floragunncom/search-guard-docs/blob/master/configuration_cache.md

It states that it is available in SG15 and I use version 5.1.1-15. I get the message:

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [searchguard.cache.ttl_minutes] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.1.1.jar:5.1.1]

[SG]

only available in ES 5.4.3 and higher like described here https://github.com/floragunncom/search-guard/wiki/Changelog

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/b2700686-529d-4a78-8efa-01847873840c%40googlegroups.com.

[Terry Quigley]

Is there another setting we can use ? This is currently affecting functionality of our app. 

[SG]

how does it affect the functionality of your app?

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/582ca8b8-edc6-4145-a516-5e2946f3c1fb%40googlegroups.com.

[Terry Quigley]

We use the JWT plugin to match groups to roles. If we create a new group the user cannot see what they should for up to an hour. We just want to be able to disable caching, whether that be by using ttl or any of the other settings 
otherwise it looks like a bug

[SG]

i guess the only chance you have (if you are not a paying customer) is to upgrade to 5.4.3-15 or later
We do only backport security critical fixes to older versions, pls. refer to our EOL policy: https://github.com/floragunncom/search-guard-docs/blob/master/eol.md

> Am 25.10.2017 um 14:49 schrieb Terry Quigley <topqu...@gmail.com>:
>

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/582ca8b8-edc6-4145-a516-5e2946f3c1fb%40googlegroups.com.