Unable to restore the index from s3

[Pradeep]

Hi Team,

Getting this below error while trying to restore the index from s3 bucket.

URL: https://my-ip:9200/_snapshot/backup-1/my-backup*/_restore

Request Type: POST

Body

{

  "indices": "archives-2023.01.20"

}

Response:

{

    "error": {

        "root_cause": [

            {

                "type": "security_exception",

                "reason": "no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"

            }

        ],

        "type": "security_exception",

        "reason": "no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"

    },

    "status": 403

}

Kindly look into this.

Thanks,
Pradeep

[Pradeep]

No issue while fetching the snapshot data from s3

{

            "snapshot": "my-backup-2023.01.20-00:11:57.764",

            "uuid": "sdfwerf34tregdfvdsf34",

            "version_id": 135238227,

            "version": "1.2.4",

            "indices": [

                "archives-default-2023.01.20"

            ],

            "data_streams": [],

            "include_global_state": true,

            "metadata": {

                "snapshot_created": "Open Distro for Elasticsearch Index Management"

            },

            "state": "SUCCESS",

            "start_time": "2023-01-20T00:12:00.332Z",

            "start_time_in_millis": 1674173520332,

            "end_time": "2023-01-20T00:12:37.791Z",

            "end_time_in_millis": 1674173557791,

            "duration_in_millis": 37459,

            "failures": [],

            "shards": {

                "total": 3,

                "failed": 0,

                "successful": 3

            }

        },

[Federico Gustavo Galland]

Hi Pradeep,

As of right now, Wazuh uses OpenSearch 1.3 as a basis for its Wazuh Indexer.

The documentation for creating and restoring indices to S3 buckets can be found below:

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html#managedomains-snapshot-restore

Bear in mind that this procedure lies outside of normal Wazuh operation and as such, it is not within the scope of this community group.

With that out of the way, I can tell you from your output shared on the first e-mail, that the user you tried to restore your snapshot with doesn't have sufficient privileges on your backup indices. 

There seems to be an opensearch forum thread about this problem as well with some hints as to how to solve it:

https://forum.opensearch.org/t/security-issue-no-permissions-for-indices-admin-resolve-index/3959/11

Let us know if there is anything wazuh-related that we can help you with further.

Regards,

Federico

[unknown]

Hello Federico,

Thanks for the quick response.

I appreciate you are helping us outside of normal Wazuh operation,

As I checked, We are using default admin user for this operation which have all the required permission (which cannot be changed),
and after your suggestion we have created a user with snapshotrestore role, but getting the same error.

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/-47_XxOCgv8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/e7581624-29f3-4956-9d4c-bee2c8f1ee60n%40googlegroups.com.

[Federico Gustavo Galland]

This could very well be related to a possible bug in opensearch itself:

https://forum.opensearch.org/t/security-issue-no-permissions-for-indices-admin-resolve-index/3959/11

Check out that thread for possible solutions.

--

Federico Galland Pre-sales Engineer +1 (844) 349 2984 wazuh.com federico...@wazuh.com f-galland

[Federico Gustavo Galland]

Pradeep,

I actually managed to get ahold of an S3 bucket to test this on and was successful in saving and restoring indices with Wazuh 4.3.10.

I did run into a few road bumps which I hope I can help you avoid.

Most of the procedure was taken from here:

• https://opensearch.org/docs/1.2/opensearch/snapshot-restore/#about-snapshots
  

So first thing you should create your S3 bucket and give your user/role the proper policy over it:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Action": [
            "s3:*"
        ],
        "Effect": "Allow",
        "Resource": [
            "arn:aws:s3:::your-bucket",
            "arn:aws:s3:::your-bucket/*"
        ]
    }]
}

Now, before you proceed, you are going to need to change line 81 of the /usr/share/wazuh-indexer/bin/opensearch-env script 

from this:

if [ -z "$OPENSEARCH_PATH_CONF" ]; then OPENSEARCH_PATH_CONF="$OPENSEARCH_HOME"/config; fi

to this:

if [ -z "$OPENSEARCH_PATH_CONF" ]; then OPENSEARCH_PATH_CONF="/etc/wazuh-indexer/"; fi

With that done, you now need to install the proper opensearch plugin:

/usr/share/wazuh-indexer/bin/opensearch-plugin install repository-s3

Add the following lines to /etc/wazuh-indexer/opensearch.yml

s3.client.default.endpoint: s3.amazonaws.com
s3.client.default.max_retries: 3
s3.client.default.path_style_access: false
s3.client.default.protocol: https
s3.client.default.read_timeout: 50s
s3.client.default.use_throttle_retries: true

Now replace the access and secret keys in the following commands and run them. 

export AWS_ACCESS_KEY_ID='YOUR_ACCESS_KEY'
export AWS_SECRET_ACCESS_KEY='YOUR_SECRET_KEY'
echo $AWS_ACCESS_KEY_ID | /usr/share/opensearch/bin/opensearch-keystore add --stdin s3.client.default.access_key
echo $AWS_SECRET_ACCESS_KEY | /usr/share/wazuh-indexer/bin/opensearch-keystore add --stdin s3.client.default.secret_key
chown wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/opensearch.keystore
systemctl restart wazuh-indexer

If everything went well, you should now be able to login to the Wazuh Dashboard, go to the Stack Icon, select "Dev Tools" and run the following commands to register the repository and take snapshots:

This will reload the credentials from the keystore:

POST _nodes/reload_secure_settings

With the following we are registering the S3 bucket as a snapshot repo:

PUT _snapshot/my-s3-repository
{
  "type": "s3",
  "settings": {
    "bucket": "your_bucket_name",
    "region": "your_bucket_region"
}
}

We can now start backing up data:

PUT _snapshot/my-s3-repository/1
{
  "indices": "YOUR_INDICES_NAMES_HERE",
  "ignore_unavailable": true,
  "include_global_state": false,
  "partial": false
}

You can check your snapshot data by running something like so:

GET _snapshot/my-s3-repository/_all

So far, you would have backed up your indexed data.

You can either remove the original index to test restoring it, our you can use a renaming pattern to test restoring without modifying current indices:

POST _snapshot/my-repository/2/_restore
{
  "indices": "YOUR_INDICES_NAMES_HERE",
  "ignore_unavailable": true,
  "include_global_state": false,
  "include_aliases": false,
  "partial": false,
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1",
  "index_settings": {
    "index.blocks.read_only": false
  },
  "ignore_index_settings": [
    "index.refresh_interval"
  ]
}

This is pretty much the full circle.

Once again, this was mostly taken from here:

• https://opensearch.org/docs/1.2/opensearch/snapshot-restore/#about-snapshots

Save for the modification to the opensearch-env script.

Let me know if this worked.

Regards,

Federico

[unknown]

Hello Federico,

I was able to do the s3 setup in wazuh indexer after few tweaks in the configurations,
As I may have told you, our snapshots are successfully getting saved in the s3 bucket, but the issue was with the retrieval/restore api of open search snapshots,

After giving some time to this I managed to resolve this issue as well, actual the issue is with the wildcard in the opensearch restore api,

For eg.

GET /_snapshots/my-repo/my-snaps-*?pretty 

We get all our snapshots whose name starts with "my-snaps"

And when I do the same for restore api

POST /_snapshots/my-repo/my-snaps-*/_restore  

Body {"indices": ["archives-2023.01.24"]}

I get this response "no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"

and the same response even if the snapshot "my-snaps-*" is not present.

So, To restore the snapshots we are now putting the full name of snapshot (with date and timestamp)

For eg.

POST /_snapshots/my-repo/my-snaps-2023.01.25-06:16:47.597/_restore  

Body {"indices": ["archives-2023.01.24"]}

Response: {accepted: true}

Which is now a 2 way process for us, because if we want to restore alerts of some specific date we first need to find the full snapshot name and then using that we have to put the same for restore api.

If there is any other way around, do let us know,

Thanks,

Pradeep

[Federico Gustavo Galland]

Hi Pradeep,

Opensearch tends to show these permission errors when you use the wrong API endpoint.

Usage of wildcards is not mentioned in the official docs (it tends to be whenever it is supported):

https://opensearch.org/docs/latest/api-reference/snapshots/restore-snapshot/

So in this case, the best option is probably to write a script that retrieves all the snapshots within a repo and then issues the specific calls to retrieve each of these.

Have a good weekend,

Federico