Kibana Multi Tenancy Issue "It seems that the Multitenancy module is not installed on your ... "

[.mni]

Hi I am running into the following issue with Kibana multi tenancy:

When I click on the Tenants link inside Kibana, I get:

It seems that the Multitenancy module is not installed on your Elasticsearch cluster, or it is disabled. Multitenancy will not work, please check your installation.

Here is my kibana config:

console.enabled: false
elasticsearch.requestTimeout: 600000
elasticsearch.shardTimeout:   595000
elasticsearch.ssl.verificationMode: none
elasticsearch.url: "https://localhost:9200"
elasticsearch.username: "kibanaserver"
elasticsearch.password: "kibanaserver"
logging.verbose: false
server.host: "0.0.0.0"
kibana.index: ".kibana"
searchguard.basicauth.enabled: false
searchguard.jwt.enabled: true
searchguard.multitenancy.enable_filter: true
searchguard.multitenancy.enabled: true
searchguard.multitenancy.tenants.enable_global: true
searchguard.multitenancy.tenants.enable_private: true
elasticsearch.requestHeadersWhitelist: [ "Authorization", "sgtenant", "jwt" ]

Here is my sg_config:

searchguard:
  dynamic:
      kibana:
      multitenancy_enabled: true
      server_username: 'kibanaserver'
      index: '.kibana'
      do_not_fail_on_forbidden: true
    http: ......

[Search Guard]

It would really help if you let us at least know which version of ES and SG you are using.

And pls make sure you have installed the multitenancy jar (for SG 5.x) and not disabled the enterprise features (SG 6.x).

Multitenancy is an enterprise feature and not part of the free community version. 

It also looks like the indentation in sg_config.yml is wrong, should more look like

searchguard:
  dynamic:
      kibana:
        multitenancy_enabled: true
        server_username: 'kibanaserver'
        index: '.kibana'
        do_not_fail_on_forbidden: true

[.mni]

MY ES and SearchGuard versions are 6.2.4 and I am using the JWT module so that proves I have the enterprise features enabled. I am also under the impression that I don't need to install a separate multitenancy JAR for Search-Guard 6.

Sorry, I must have messed up the indentation while trying to remove comments from config here. Here's my unmodified config and the indentation looks good. 

searchguard:
  dynamic:
    # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
    # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
    # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
    #filtered_alias_mode: warn
    kibana:
      # Kibana multitenancy - NOT FREE FOR COMMERCIAL USE
      # see https://github.com/floragunncom/search-guard-docs/blob/master/multitenancy.md
      # To make this work you need to install https://github.com/floragunncom/search-guard-module-kibana-multitenancy/wiki

      multitenancy_enabled: true
      server_username: 'kibanaserver'
      index: '.kibana'
      do_not_fail_on_forbidden: true
    http:

      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
        #internalProxies: '.*' # trust all internal proxies, regex pattern
        remoteIpHeader:  'x-forwarded-for'
        proxiesHeader:   'x-forwarded-by'
        #trustedProxies: '.*' # trust all external proxies, regex pattern
        ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
        ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
        ###### and here https://tools.ietf.org/html/rfc7239
        ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
    authc:

[Jochen Kressin]

If you are using SG 6.x then you don't need to install any additional jars, that's right. 

In order to see what modules are activated, can you please post the output of:

https://es_node:http_port/_searchguard/license

and the output of:

https://es_node:http_port/_searchguard/kibanainfo

Thanks!

[.mni]

vagrant@packer-virtualbox-iso-1524410689:~$ curl -k -u admin:admin https://localhost:9200/_searchguard/license

{
    "_nodes": {
        "total": 1,
        "successful": 1,
        "failed": 0
    },
    "cluster_name": "searchguard_demo",
    "sg_license": {
        "uid": "00000000-0000-0000-0000-000000000000",
        "type": "TRIAL",
        "issue_date": "2018-04-27",
        "expiry_date": "2018-06-27",
        "issued_to": "The world",
        "issuer": "floragunn GmbH",
        "start_date": "2018-04-27",
        "major_version": 6,
        "cluster_name": "*",
        "msgs": [],
        "expiry_in_days": 22,
        "is_expired": false,
        "is_valid": true,
        "action": "",
        "prod_usage": "Yes, one cluster with all commercial features and unlimited nodes per cluster.",
        "license_required": true,
        "allowed_node_count_per_cluster": "unlimited"
    },
    "modules": {
        "NOOP_AUTHENTICATION_BACKEND": {
            "default_implementation": "com.floragunn.searchguard.auth.internal.NoOpAuthenticationBackend",
            "gitsha1": "ea2622a7df024e3b2a20d50928a62589322d26d7",
            "buildTime": "2018-05-11T15:21:02Z",
            "is_enterprise": "false",
            "actual_implementation": "com.floragunn.searchguard.auth.internal.NoOpAuthenticationBackend",
            "description": "Noop authentication backend",
            "type": "NOOP_AUTHENTICATION_BACKEND",
            "version": "6.2.4-22.1"
        },
        "JWT_AUTHENTICATION_BACKEND": {
            "default_implementation": "com.floragunn.dlic.auth.http.jwt.HTTPJwtAuthenticator",
            "gitsha1": "ea2622a7df024e3b2a20d50928a62589322d26d7",
            "buildTime": "2018-05-11T15:21:02Z",
            "is_enterprise": "true",
            "actual_implementation": "com.floragunn.dlic.auth.http.jwt.HTTPJwtAuthenticator",
            "description": "JWT authorization backend",
            "type": "JWT_AUTHENTICATION_BACKEND",
            "version": "6.2.4-22.1"
        },
        "INTERNAL_USERS_AUTHENTICATION_BACKEND": {
            "default_implementation": "com.floragunn.searchguard.auth.internal.InternalAuthenticationBackend",
            "gitsha1": "ea2622a7df024e3b2a20d50928a62589322d26d7",
            "buildTime": "2018-05-11T15:21:02Z",
            "is_enterprise": "false",
            "actual_implementation": "com.floragunn.searchguard.auth.internal.InternalAuthenticationBackend",
            "description": "Internal users authorization backend",
            "type": "INTERNAL_USERS_AUTHENTICATION_BACKEND",
            "version": "6.2.4-22.1"
        },
        "HTTP_BASIC_AUTHENTICATOR": {
            "default_implementation": "com.floragunn.searchguard.http.HTTPBasicAuthenticator",
            "gitsha1": "ea2622a7df024e3b2a20d50928a62589322d26d7",
            "buildTime": "2018-05-11T15:21:02Z",
            "is_enterprise": "false",
            "actual_implementation": "com.floragunn.searchguard.http.HTTPBasicAuthenticator",
            "description": "HTTP Basic Authenticator",
            "type": "HTTP_BASIC_AUTHENTICATOR",
            "version": "6.2.4-22.1"
        }
    },
    "compatibility": {
        "modules_mismatch": false
    }
}

vagrant@packer-virtualbox-iso-1524410689:~$ curl -k -u admin:admin https://localhost:9200/_searchguard/kibanainfo

{
    "user_name": "admin",
    "not_fail_on_forbidden_enabled": false,
    "kibana_mt_enabled": false,
    "kibana_index": ".kibana",
    "kibana_server_user": "kibanaserver",
    "kibana_index_readonly": false
}

[Jochen Kressin]

So it seems the MT module is indeed not loaded. Can you attach the complete sg_config.yml file, because from what you posted it should in fact work.