[COMMIT scylla-cluster-tests master] improvement(k8s): run fstrim command with retry

[Commit Bot]

From: Valerii Ponomarov <valerii....@scylladb.com>
Committer: Israel Fruchter <israel....@gmail.com>
Branch: master

improvement(k8s): run fstrim command with retry

It happens that pod which is going to be used for running
fstrim command may be temporary unavailable and fail a kubectl call.
So, decrease the probability of the false failure by making it run with
the retry mechanism.

---
diff --git a/sdcm/cluster_k8s/__init__.py b/sdcm/cluster_k8s/__init__.py
--- a/sdcm/cluster_k8s/__init__.py
+++ b/sdcm/cluster_k8s/__init__.py
@@ -44,6 +44,7 @@
from kubernetes.client import exceptions as k8s_exceptions
from kubernetes.client import V1ConfigMap
from kubernetes.dynamic.resource import Resource, ResourceField, ResourceInstance, ResourceList, Subresource
+import invoke
from invoke.exceptions import CommandTimedOut

from sdcm import sct_abs_path, cluster
@@ -1967,6 +1968,9 @@ def refresh_ip_address(self):

self._init_port_mapping()

+ @retrying(n=60, sleep_time=10,
+ allowed_exceptions=(k8s_exceptions.ApiException, invoke.exceptions.UnexpectedExit),
+ message="Failed to run fstrim command...")
def fstrim_scylla_disks(self):
# NOTE: to be able to run 'fstrim' command in a pod, it must have direct device mount and
# appropriate priviledges.