SG + ES + X-Pack Monitoring + ActiveDirectory = I need help.

[mcost...@np6.com]

Hello,

I'm trying to build a local test setup with the following components:

- ElasticSearch 5.2.2

- SearchGuard 5.2.2 (With Enterprise extension for ActiveDirectory usage)
- X-Pack 5.2.2 (Monitoring only)

Without X-Pack, the environment works fine. When I install X-Pack, I get errors in stdout. With the help of an existing Github issue, I've been able to work my way past a few of the errors, but at this point, I am stuck. I'll add some details below. I'd like to mention that I intend to add Kibana to the mix, but I'd like to keep the environment minimal while sorting out these errors. 

So, at the moment this is where I am stuck:

Caused by: org.elasticsearch.ElasticsearchSecurityException: unauthenticated request indices:data/write/bulk for user User [name=_sg_internal, roles=[]]

As recommended by the above-mentioned Github issue (https://github.com/floragunncom/search-guard-ssl/issues/43), I've changed my `elasticsearch/config/elasticsearch.yml` to have the following settings:

searchguard.ssl.transport.keystore_filepath: keystore.jks

searchguard.ssl.transport.truststore_filepath: truststore.jks

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: false

searchguard.ssl.http.keystore_filepath: keystore.jks

searchguard.ssl.http.truststore_filepath: truststore.jks

searchguard.authcz.admin_dn:

  - CN=kirk,OU=client,O=client,L=test, C=de

cluster.name: docker-ezcore-cluster

network.host: 0.0.0.0

######## End Search Guard Demo Configuration ########

xpack.security.enabled: false

xpack.monitoring.enabled: true

xpack.graph.enabled: false

xpack.watcher.enabled: false

xpack.monitoring.exporters:

   my_local:

     type: local

I've tried the other changes mentioned in the issue, but they seem to have no effect for me.

Here are some things I think are remarkable:

- I've successfully hooked up SearchGuard with ActiveDirectory. Yet, I don't know what role this `admin_dn` plays, and have not yet created an account for this admin. Also, I don't feel like this is related to this problem I'm blocked on, so I've left it alone for now.
- I'm curious if Kibana is ESSENTIAL to the environment, and whether or not excluding it for now is exacerbating the problem.

- I'm constrained to working with version 5.2.2 of ES.

- I am testing with the demo versions of the keystores, although I use the `sg_admin.sh` tool to initialize SearchGuard. I provide the keystore file paths in the command arguments.

Any ideas or help is greatly appreciated.
Please and thank you,

Marco.

[Jochen Kressin]

Hi,

we're working on the monitoring integration. At the moment, you would need to use an HTTP exportert instead of local, like:

xpack.monitoring.exporters:

  id1:

    type: http

    host: ["https://127.0.0.1:9200"]

    auth.username: monitoring

    auth.password: ...

    ssl:

      truststore.path: truststore.jks

      truststore.password: changeit

And then map the monitoring user to the role with the permissions explained in the GitHub issue you referenced:

sg_monitor:

  cluster:

    - cluster:admin/ingest/pipeline/get

    - cluster:admin/xpack/monitoring/bulk

    - indices:admin/template/get

    - indices:admin/template/put

    - indices:admin/get

    - cluster:monitor/nodes/info

    - cluster:monitor/health

    - cluster:monitor/main

    - cluster:monitor/xpack/info

    - indices:data/write/bulk

  indices:

    '?monitoring*':

      '*':

        - ALL

If you also use the Kibana monitoring plugin from X-Pack, you will run into the following issue:

https://github.com/floragunncom/search-guard/issues/345

We're working on the compatibility of Search Guard and X-Pack Monitoring. It's high on our priority list, but it might take some time until the integration runs smoothly on both ES and KI side.

As to your other question, no admin_dn has nothing to do with this issue here. It simply configures which TLS certificate(s) can be used as admin certificate. An admin certificate has full access to the cluster, and you need it whenever you make changes to the SG index. If you use our demo certificates, it's the "kirk" certificate, and it's contained in the keystore you use when executing sgadmin.

[mcost...@np6.com]

Thank you for the help. No worries on it not being finalized as of yet. Sadly, I still have some problems with the environment.

I have the following normally set:

searchguard.ssl.http.enabled: false

...I get:

org.elasticsearch.client.ResponseException: GET http://127.0.0.1:9200/?filter_path=version.number: HTTP/1.1 401 Unauthorized
Authentication finally failed

I see someone also had this issue, and the role_mapping (with new role) seemed to fix it for him/her. Not for me sadly. Maybe I've done something wrong. Here are my pertinent configs:

sg_roles.yml

sg_monitor:
  cluster:
    - cluster:admin/ingest/pipeline/get
    - cluster:admin/xpack/monitoring/bulk
    - indices:admin/template/get
    - indices:admin/template/put
    - indices:admin/get
    - cluster:monitor/nodes/info
    - cluster:monitor/health
    - cluster:monitor/main
    - cluster:monitor/xpack/info
    - indices:data/write/bulk
  indices:
    '?monitoring*':
      '*':
        - ALL

sg_roles_mapping.yml

sg_monitor:

  users:

    - monitoring

elasticsearch.yml

######## Start Search Guard Demo Configuration ########

searchguard.ssl.transport.keystore_filepath: keystore.jks
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: false
searchguard.ssl.http.keystore_filepath: keystore.jks
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de


cluster.name: docker-ezcore-cluster
network.host: 0.0.0.0

######## End Search Guard Demo Configuration ########

xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.watcher.enabled: false
xpack.monitoring.exporters:

  id1:
    type: http
    host: ["http://127.0.0.1:9200"]
    auth.username: monitoring
    auth.password: pword
#    ssl:
#      truststore.path: truststore.jks
#      truststore.password: changeit

As you can see, the query is sent without basic auth credentials. I had my doubts that the mentioned solution was relevant to me as I am using ActiveDirectory.

Any thoughts?

Please and thank you,

Marco.

[Jochen Kressin]

Where do you configure/store the monitoring user? In ActiveDirectory or in the SG internal user database?

I would first check that the user "monitoring" can log in with password "pword", as configured in your elasticsearch.yml. If not already enabled, set the challenge flag to true:

For AD:

      ldap:

        enabled: true

        order: ...

        http_authenticator:

          type: basic

          challenge: true

        authentication_backend:

Or for the internal user DB:

      basic_internal_auth_domain: 

        enabled: true

        order: 1

        http_authenticator:

          type: basic

          challenge: true

        authentication_backend:

          type: intern

Note: You can only enable it for one HTTP authenticator, so choose the auth domain where you configured the monitoring user.

And then try to log in by using a browser. Best is to access the authinfo endpoint of SG:

http://localhost:9200/_searchguard/authinfo

The browser should prompt you for username/password - use the monitoring user for that. If authentication succeeded, you will see a JSON snippet containing the roles of the user. Check that the monitoring user is assigned to the sg_monitor role.

[mcost...@np6.com]

Another very helpful reply. Thank you. 

Yes, my account was not set up properly. I changed the http exporter to use an account that is known to be working. With that, I was able to get further. I ended up needing to add a few new entries under `cluster` as shown below:

sg_monitor:

  cluster:

    - cluster:admin/mappings/get               <--------- added

    - cluster:admin/ingest/pipeline/put        <--------- added        

    - cluster:admin/ingest/pipeline/get

    - cluster:admin/xpack/monitoring/bulk

    - indices:admin/template/get

    - indices:admin/template/put

    - indices:admin/get

    - cluster:monitor/nodes/info

    - cluster:monitor/health

    - cluster:monitor/main

    - cluster:monitor/xpack/info

    - indices:data/write/bulk

  indices:

    '?monitoring*':

      '*':

        - ALL

    '?marvel*':           <------ added

      '*':

        - ALL

I added these entries because I received errors concerning permissions for them. One by one my ES progressed. Finally, it complained about being able to access the `.marvel-es-1` index. That was a red flag for me. Anyway, As shown above, I added this index to the role, but this time, my environment did not progress. It still complained about permission and access to this index. The error observed is:

[2017-06-22T18:21:28,870][INFO ][c.f.s.c.PrivilegesEvaluator] No index-level perm match for User [name=CN=Kibana SearchGuard Dev,OU=Comptes de services,OU=hidden,DC=hidden,DC=hidden,DC=hidden, roles=[]] [IndexType [index=_all, type=*]] [Action [indices:admin/get]] [RolesChecked [sg_kibana_server, sg_monitor, sg_own_index, sg_public]]
[2017-06-22T18:21:28,871][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {sg_public=[IndexType [index=_all, type=*]], sg_own_index=[IndexType [index=_all, type=*]], sg_monitor=[IndexType [index=_all, type=*]], sg_kibana_server=[IndexType [index=_all, type=*]]}
[2017-06-22T18:21:28,874][ERROR][o.e.x.m.e.h.BackwardsCompatibilityAliasesResource] org.elasticsearch.xpack.monitoring.exporter.http.BackwardsCompatibilityAliasesResource$$Lambda$1681/1710762394@59f4ec16
org.elasticsearch.client.ResponseException: GET http://127.0.0.1:9200/.marvel-es-1-*?filter_path=*.aliases: HTTP/1.1 403 Forbidden
{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for indices:admin/get"}],"type":"security_exception","reason":"no permissions for indices:admin/get"},"status":403}

To repeat, this error occurs even after I added the config (above) and ran `sg_admin.sh` AND restarted my ES cluster. You can see from the logs that user is correctly mapped to the `sg_monitor` role. 

Please let me know what you think,

[mcost...@np6.com]

Respectful bump. Thank you.

[Search Guard]

sg_monitor:

  cluster:

    - cluster:admin/mappings/get

    - cluster:admin/ingest/pipeline/put       

    - cluster:admin/ingest/pipeline/get

    - cluster:admin/xpack/monitoring/bulk

    - indices:admin/template/get

    - indices:admin/template/put

    - indices:admin/get

    - cluster:monitor/nodes/info

    - cluster:monitor/health

    - cluster:monitor/main

    - cluster:monitor/xpack/info

    - indices:data/write/bulk

  indices:

    '*':

      '*':

        - indices:admin/get

    '?monitoring*':

      '*':

        - '*'

    '?marvel*':

      '*':

        - '*'

[Jochen Kressin]

We've released a preliminary documentation: https://github.com/floragunncom/search-guard-docs/blob/master/monitoring.md

This issue here requires an updated version of the Kibana Plugin, we're working on it:

https://github.com/floragunncom/search-guard/issues/345

[mcost...@np6.com]

Thank you very much for your hard work and that prelim document. On my end, I've had some success, but it's not 100% and I think you may be interested in my observations.

So, to repeat, I am using SG 5.2.2 v12 WITH the enterprise extension.

I was not able to get things going with the `sg_monitor` role described in the document. I was able to make things work with the `sg_monitor` role described earlier in this thread, but with the following message being found repeatedly in the console output of ES:

[2017-07-04T10:04:27,435][INFO ][c.f.s.c.PrivilegesEvaluator] No index-level perm match for User [name=<<<my active directory user>>>, roles=[]] [IndexType [index=_all, type=esqueue]] [Action [indices:data/read/search]] [RolesChecked [sg_kibana_server, sg_monitor, sg_own_index, sg_public]]

[2017-07-04T10:04:27,435][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {sg_public=[IndexType [index=_all, type=esqueue]], sg_own_index=[IndexType [index=_all, type=esqueue]], sg_monitor=[IndexType [index=_all, type=esqueue]], sg_kibana_server=[IndexType [index=_all, type=esqueue]]}

However, monitoring in Kibana seems to be working.

Now, if I open up the permissions for the monitoring user entirely (i.e. add him to the sg_all_access role), that message goes away. In fact, I am only left with a warning:

[2017-07-04T10:06:36,597][WARN ][c.f.s.c.PrivilegesEvaluator] _all does not exist in cluster metadata


I guess what I'm saying is, with the stack I'm using, the document may not be 100% correct. Also, I'd love your help diagnosing the permission problem.

Many thanks,

Marco.

[mcost...@np6.com]

Respectful bump. Question distilled: What should I set the permissions to for kibana user in order to avoid the above, but not open up the permissions entirely.

Thank you,

[Jochen Kressin]

Hi Marco,

yes, the docs are geared towards ES > 5.3.0 and SG 14. Plus, we assume that your monitoring user has also the Kibana User role. I guess we should make that clearer in the docs. However, we've tested the setup now with:

- ES 5.2.2

- SG 5.2.2-12

- KI 5.2.2

- X-Pack 5.2.2, both on ES and KI

Here's the role definition that works for us with the said versions:

sg_monitor:

  cluster:

    - "cluster:admin/xpack/monitoring/*"

    - "cluster:admin/ingest/pipeline/*"

    - "indices:admin/template/get"

    - "indices:admin/template/put"

    - "indices:admin/*get"

    - CLUSTER_MONITOR

    - CLUSTER_COMPOSITE_OPS

  indices:

    '?monitoring*':

      '*':

        - UNLIMITED

    '?marvel*':

      '*':

        - UNLIMITED

    '*':

      '*':

        - READ

        - indices:admin/get

    '?kibana':

      '*':

        - INDICES_ALL   

The monitoring user needs full control over the monitoring and marvel (for backwards compatibility) indices, and also access to the .kibana index.

Since monitoring affects all indices, the monitoring user must have READ and admin/get permissions on all indices, hence the wildcard.

The last message:

[2017-07-04T10:06:36,597][WARN ][c.f.s.c.PrivilegesEvaluator] _all does not exist in cluster metadata

is fixed in later versions of ES/SG, so for the moment with your versions you either need to ignore it, or just set the log level of the PrivilegesEvaluator to ERROR. In config/log4j2.properties, set:

logger.pe.name = com.floragunn.searchguard.configuration.PrivilegesEvaluator

logger.pe.level = error

Let me know if this works for you!

[Marco Costantini]

I was just able to try this out and it works great. Thank you for helping me with this. Excellent support.

Marco.