curator - no permissions for cluster:admin/snapshot/restore

[Patrick Monnoire]

Hi,

We are currently evaluating Search Guard to replace Shield.
We still have one blocking point:

We are able to generate some snapshots using curator.
We are able to restore from these snapshots using a CURL request.
But, we are not able to restore using curator. It always fail with the following error:


2017-03-01 09:53:38,206 INFO      Preparing Action ID: 1, "restore"
2017-03-01 09:53:38,295 INFO      Trying Action ID: 1, "restore": Restore all indices in the most recent snapshot-* snapshot with state SUCCESS.  Wait for the restore to complete before continuing.  Do not skip the repository filesystem access check.  Use the other options to define the index/shard settings for the restore.
2017-03-01 09:53:38,515 INFO      Restoring indices "['.kibana', 'searchguard']" from snapshot: snapshot-20170301090739
2017-03-01 09:53:38,532 ERROR     Failed to complete action: restore.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: TransportError(403, 'security_exception', 'no permissions for cluster:admin/snapshot/restore')


We are using:
- ES                      2.3.3
- search-guard-2     2.3.3.11
- search-guard-ssl  2.3.3.20

We have used example PKI scripts to generate keys & self signed certificate

Using curl to restore is working:
cat /data/elasticsearch/cert/admin.crt.pem /data/elasticsearch/cert/ca/chain-ca.pem > /tmp/ch.pem
curl -Ss --insecure -XPOST -E /tmp/ch.pem --key /data/elasticsearch/cert/admin.key.pem 'https://dev-ct-elasticsearch01:9200/_snapshot/backup_repository/snapshot-20170223104445/_restore?pretty'

But curator fails to do the same:
curator --config /tmp/curator.yml /etc/curator.restore.yml
2017-03-01 09:53:38,532 ERROR     Failed to complete action: restore.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: TransportError(403, 'security_exception', 'no permissions for cluster:admin/snapshot/restore')

How can I get ride of this error?
What did I miss in curator config?

Here are the curator yml files:

curator.yml:
client:
  hosts: [dev-ct-elasticsearch04]
  port: 9200
  url_prefix:
  use_ssl: true
  certificate:
  client_cert: /tmp/ch.pem
  client_key:
  aws_key:
  aws_secret_key:
  aws_region:
  ssl_no_validate: true
  http_auth: admin:Superuser
  timeout:
  master_only: False
logging:
  loglevel: INFO
  logfile: /var/log/curator.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

curator.restore.yml:
actions:
  1:
    action: restore
    description: >-
      Restore all indices
    options:
      repository: backup_repository
      name:
      indices:
      include_aliases: False
      ignore_unavailable: False
      include_global_state: True
      partial: False
      rename_pattern:
      rename_replacement:
      extra_settings:
      wait_for_completion: True
      skip_repo_fs_check: False
      timeout_override:
      continue_if_exception: False
      disable_action: false
    filters:
    - filtertype: pattern
      kind: prefix
      value: snapshot-
      exclude:
    - filtertype: state
      state: SUCCESS
      exclude:


Any help will be appreciated.

Thanks in advance,
Patrick

[SG]

You have two options here:

- User curator with ssl client authentication and run it with an admin certificate (like sgadmin) but we had seen problems with this approach: https://github.com/floragunncom/search-guard/issues/196
- We are working currently on this feature (pull requests 245 and 257) and here is preliminary version for ES 2.3.3. which includes PR #245 https://gist.github.com/floragunncom/fe1f5d88837b1e3d96db82a844f75bf7
This approach does not (yet) allow to restore the global state! Do you really need that?

> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9238b352-3c53-4e92-ad16-67f6f4f0f007%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Patrick Monnoire]

Hi,

I've already tried SSL authentication without any success.
But, never mind.
The fact is that your preliminary version of SG release 12 is solving the issue. I've installed it and was able to restore from a snapshot.
Thanks for having shared this preliminary version.

So, one more question: Do you have yet scheduled a delivery date for SG 2 release 12? If not, just a estimate (couple of weeks or months) will be also appreciated.


Thanks a lot,
Patrick

[SG]

2-3 weeks

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9ce0fcf8-f40d-45ea-b88b-2c8cb91b6230%40googlegroups.com.

[SG]

Release is scheduled for Tue 18th of April

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9ce0fcf8-f40d-45ea-b88b-2c8cb91b6230%40googlegroups.com.

[Patrick Monnoire]

Hi,

Even if preliminary version of SG release 12 is solving the issue, I'm still facing the same issue using the official release.

I've configured my group like this:

sg_admin:
  cluster:
    - cluster:admin/snapshot/restore
    - '*'
  indices:
    '*':
      '*':
        - '*'

Then I've added these 2 lines in /etc/elasticsearch/elasticsearch.yml

searchguard.enable_snapshot_restore_privilege: true
searchguard.check_snapshot_restore_write_privileges: false

My snapshot does not contain the searchguard index
and finaly, when calling currator to restore, I set

include_global_state: False

But I still get this error:

[2017-05-05 09:34:22,783][WARN ][com.floragunn.searchguard.configuration.PrivilegesEvaluator] cluster:admin/snapshot/restore is not allowed for a regular user

What am I doing wrong?


Regards,
Patrick

[SG]

Did you set "searchguard.enable_snapshot_restore_privilege: true" in elasticsearch on *all* nodes and are you sure you running SG 12 on all nodes too?

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/385341ea-8a60-49ed-9fd4-eb98c4c7feb4%40googlegroups.com.

[Patrick Monnoire]

GET _cat/plugins?v
name           component        version  type url
ci-ct-es02-778 search-guard-2   2.4.4.12 j       
ci-ct-es02-778 search-guard-ssl 2.4.4.21 j       
ci-ct-es03-778 search-guard-2   2.4.4.12 j       
ci-ct-es03-778 search-guard-ssl 2.4.4.21 j       
ci-ct-es04-778 search-guard-2   2.4.4.12 j       
ci-ct-es04-778 search-guard-ssl 2.4.4.21 j       
ci-ct-es01-778 search-guard-2   2.4.4.12 j       
ci-ct-es01-778 search-guard-ssl 2.4.4.21 j       


And on all 4 nodes:

searchguard.ssl.transport.keystore_filepath: dev-ct-elasticsearch03-keystore.jks
searchguard.ssl.transport.keystore_password: xxxxx
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: xxxx
searchguard.ssl.transport.enforce_hostname_verification: true
searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: dev-ct-elasticsearch03-keystore.jks
searchguard.ssl.http.keystore_password: xxxx
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.truststore_password: xxxx

searchguard.authcz.admin_dn:
  - cn=admin,ou=client,o=client,l=Test,c=DE

searchguard.enable_snapshot_restore_privilege: true
searchguard.check_snapshot_restore_write_privileges: false


Regards,
Patrick

[SG]

Ah sorry, this feature is available only for ES >= 5.0.0 (and therefore SG >= 5)

See https://github.com/floragunncom/search-guard/wiki/Changelog

Any chance to upgrade?

> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/150df126-c258-484c-ac07-b85f0951096b%40googlegroups.com.

[Patrick Monnoire]

:( :( :(

But it was working with the 2.3.3.12 pre-release you sent me for test purpose.
Any chance to have this feature for ES < 5.0?

For sure we will migrate to ES 5.x
But it is still no yet scheduled.

[Search Guard]

Hi Patrick,

we just decided that we will backport the functionality for ES 2.4.5 but not for ES 2.3.x.

Can you upgrade to ES 2.4.5? 

ES 2.3 has EOL in September 2017 (https://www.elastic.co/de/support/eol) and yesterday the first 

Alpha of Elasticsearch 6 was announced. So i think it would make really sense to upgrade at least to 2.4.5 (or even better to 5.4).

[Patrick Monnoire]

Hi,

We are currently using ES 2.4.4.
I think it will not be a big deal to migrate to 2.4.5

thanks for the backport to ES 2.4.5,
Patrick

[Search Guard]

do you need "searchguard.check_snapshot_restore_write_privileges" functionality or is "searchguard.enable_snapshot_restore_privilege" sufficient?

[Patrick Monnoire]

Hi,

searchguard.enable_snapshot_restore_privilege is enough because our backups do not contain the searchguard index.


Thanks,
Patrick

[Search Guard]

here is the release candidate: https://oss.sonatype.org/content/repositories/snapshots/com/floragunn/search-guard-2/2.4.5.13-SNAPSHOT/search-guard-2-2.4.5.13-20170518.181116-1.zip

Release is planned for Sunday.

[SG]

Officially released today https://groups.google.com/forum/#!topic/search-guard/5nK3Yhn-w3Q

> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/8ed3d00b-7be7-4952-aacc-f5dde2dd5bfc%40googlegroups.com.