Getting wait-for-migrations log inside awx-task and awx-web pods

[rabin sedhain]

I have deployed awx-operator on kubernetes cluster and everything seems smooth from outside, every service and pods were running perfectly. Then when i walked through logs of the awx pods i saw that the pods were waiting for database migration . this is a fresh installation and it is looping through waiting for the migration. Any idea on what exactly is the issue.

# kubectl logs pod/awx-web-6b4767c86f-lsjnp
[wait-for-migrations] Waiting for database migrations...
[wait-for-migrations] Attempt 1 of 30
[wait-for-migrations] Waiting 0.5 seconds before next attempt
[wait-for-migrations] Attempt 2 of 30
[wait-for-migrations] Waiting 1 seconds before next attempt
[wait-for-migrations] Attempt 3 of 30
[wait-for-migrations] Waiting 2 seconds before next attempt
[wait-for-migrations] Attempt 4 of 30
[wait-for-migrations] Waiting 4 seconds before next attempt
[wait-for-migrations] Attempt 5 of 30
[wait-for-migrations] Waiting 8 seconds before next attempt
[wait-for-migrations] Attempt 6 of 30
[wait-for-migrations] Waiting 16 seconds before next attempt
[wait-for-migrations] Attempt 7 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 8 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 9 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 10 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 11 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 12 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 13 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt

[AWX Project]

Hello,

Migrations may take a bit of time to run. Did this eventually complete? If not, you can shell into your task container and run `awx-manage showmigrations` to see where it might be getting caught up. If that is the case, and migrations are not resolving, please provide us with the output of `awx-manage showmigrations`.  

- AWX Team

[rabin sedhain]

bash-5.1$ awx-manage migrate
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
    raise ex.with_traceback(None)
psycopg.OperationalError: connection is bad: Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 184, in manage
    if (connection.pg_version // 10000) < 12:
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 436, in pg_version
    with self.temporary_connection():
  File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 705, in temporary_connection
    with self.cursor() as cursor:
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 330, in cursor
    return self._cursor()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 306, in _cursor
    self.ensure_connection()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
    raise ex.with_traceback(None)
django.db.utils.OperationalError: connection is bad: Name or service not known

[AWX Project]

Hello,

If you are using an external database, could you please provide your secret Posgress configuration spec and your AWX spec? Please be sure to redact any sensitive information.  This will better enable us to assist you. 
-AWX Team 

[rabin sedhain]

I am doing basic installation as documented in awx operator repo.

https://github.com/ansible/awx-operator/blob/devel/docs/installation/basic-install.md 
I just cloned the repo and run 2 simple commands

`

$ make deploy # from root of repo

 &&

$ kubectl apply -f awx-demo ## this also from root