Recovering from catastrophic failures

[Tin Le]

We are running 19.3.0 with awx-operator 0.14.0 on k8s.  We suffered catastrophic failures and lost the cluster.  The only thing we have is the external pair of postgresql.  The data is fine.  How do we recover from this?

Look like the issue is that we've lost the secret_key, which is preventing us from recreating the cluster with our data.

The error we are getting is:

ERROR    [-] awx.main.utils.encryption Failed to decrypt `Setting(pk=97).value`; if you've recently restored from a database backup or are running in a clustered environment, check that your `SECRET_KEY` value is correct
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/cryptography/fernet.py", line 104, in _verify_signature
    h.verify(data[-32:])
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/cryptography/hazmat/primitives/hmac.py", line 66, in verify
    ctx.verify(signature)
  File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/cryptography/hazmat/backends/openssl/hmac.py", line 74, in verify
    raise InvalidSignature("Signature did not match digest.")
cryptography.exceptions.InvalidSignature: Signature did not match digest.

[Tin Le]

I thought I post a followup to this.  I was able to recover most of the data.  It turns out most of the database content in postgresql is not encrypted.  So I was able to dump the tables and massage the data into importable json, then feed it to the new AWX.

FYI for anyone who run into the same issue.

:-)

Tin

[ponnurangam h]

Hi, we ran into similar problem, no we are at recovering the data from the DB and feed it to the AWX, can you please share the steps. thanks in advance