Unable to restore an Elasticsearch snapshot using ES 5.5 & SG 5 on Kubernetes

[Luca Floris]

Hopefully someone can tell me where I'm going wrong here...

Using Elasticsearch 5.5  with Searchguard 5 deployed to Kubernetes. I'm trying to enable snapshot restores from Kibana but I keep getting the following when making API calls using the request 'POST snapshot/es_backup/logstash_02-03-2019/_restore'

{
  "error": {
    "root_cause": [
      {
        "type": "repository_missing_exception",
        "reason": "[es_backup] missing"
      }
    ],
    "type": "repository_missing_exception",
    "reason": "[es_backup] missing"
  },
  "status": 404
}

It's a simple cluster - 1 x master, 1 x data, 1 x client, 1 x logstash and filebeat(s)

I've added the following

elasticsearch.yml (on master, data and client)

searchguard:
  enable_snapshot_restore_privilege: true

role_mapping.yml (master, data and client)

sg_snapshot_restore:
- "CN=curator,OU=IBM Cloud Private"
- "CN=kibana,OU=IBM Cloud Private"

roles.yml

    sg_snapshot_restore:
      cluster:
        - cluster:admin/repository/put
        - cluster:admin/repository/get
        - cluster:admin/snapshot/status
        - cluster:admin/snapshot/get
        - cluster:admin/snapshot/create
        - cluster:admin/snapshot/restore
        - cluster:admin/snapshot/delete
      indices:
        '*':
          '*':
            - indices:data/write/index

sg_action_groups.yml

MANAGE_SNAPSHOTS:

- "cluster:admin/snapshot/*"
- "cluster:admin/repository/*"

sg_roles.yml

    sg_snapshot_restore:
      cluster:
        - cluster:admin/repository/put
        - cluster:admin/repository/get
        - cluster:admin/snapshot/status
        - cluster:admin/snapshot/get
        - cluster:admin/snapshot/create
        - cluster:admin/snapshot/restore
        - cluster:admin/snapshot/delete
      indices:
        '*':
          '*':
            - indices:data/write/index
            - indices:admin/create

Kibana server also has the MANAGE_SNAPSHOT action group in sg_roles.yml

    kibana_server:
      readonly: true
      cluster:
          - CLUSTER_MONITOR
          - CLUSTER_COMPOSITE_OPS
          - MANAGE_SNAPSHOTS
          - "cluster:admin/xpack/monitoring*"
          - "indices:admin/template*"
      indices:
        '?kibana':
          '*':
            - INDICES_ALL
        '?reporting*':
          '*':
            - INDICES_ALL
        '?monitoring*':
          '*':

sg_roles_mapping.yml

    sg_snapshot_restore:
      host:
      - "*"
      users:
      - "superuser"
      # this allows the kibana console to send snapshotst restore REST calls to elasticsearch API
      - "kibana"
      - "curator"

Can anyone shed some light on which config is wrong? I'm not trying to restore the all the indices, just one for testing, and no global state.

[SG]

Which exact ES and SG version are you using?

Please make sure the file system permissions are correct as outlined here:

https://discuss.elastic.co/t/repository-missing-exception-with-existing-repo/109369
https://github.com/elastic/elasticsearch/issues/9650

Please also check https://github.com/floragunncom/search-guard/issues/476

Workaround: Enable http/s on data or master eligible nodes and execute the restore command against these

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/ba9b464b-165c-4a6c-8d37-d3c3efe853d0%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Luca Floris]

Using Elasticsearch 5.5.1 and Searchguard 5-5.5.1-16

The filesystem permissions are correct within the  data and master containers, as I am able to snapshot indices just fine. Does this shared directory also need to be added to the client node?

I can try the workaround and update here, but it is not a long term solution.

[SG]

Only master and data nodes needs access to the shared directory.

Please report back if workaround is working (issue restore against elected master node). If so we know what the problem is and can fix it.

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/c20e041c-6c26-4fe3-a6c1-69450d5ba069%40googlegroups.com.

[Luca Floris]

Could you advise on whether the configuration in my original post is correct? Does the sg_snapshot_restore role need to be applied to the roles and role mapping for ES as well as the sg roles and role mapping? Is the action group also necessary?

I'll look at enabling the workaround this weekend