Refresh postgres database?

442 views
Skip to first unread message

Kevin Knox

unread,
Jan 4, 2023, 8:59:46 AM1/4/23
to AWX Project
I'm experimenting with 21.10.2 with postgres 13 on OpenShift. I hung my installation in Terminating mode last week when I was trying to delete and reinstall it, and had to forcefully delete it. Now, when I reinstall it, it seems like the postgres database has issues. When I do an awx-manage migrate, I get inconsistent results. Sometimes it runs into a dozen errors before I give up and reinstall it, sometimes it runs into no errors but the final running AWX install has no objects in it. A clean AWX install has Demo projects, inventories and the like and at the very least an admin superuser. When this install error happens, I have none of those things. And once in a while, it installs perfectly and I celebrate that it's fixed, only to find out the next reinstall fails again. 

Is there any way to do a full purge and refresh of the database?

Thank you,

Kevin

AWX Project

unread,
Jan 4, 2023, 2:38:08 PM1/4/23
to AWX Project
are you using an external db or the one that is auto deployed with AWX?

| awx-manage migrate, I get inconsistent results.

what errors are you seeing in the migration step?

AWX Team

Kevin Knox

unread,
Jan 4, 2023, 4:31:11 PM1/4/23
to AWX Project
I'm using the internal postgres automatically deployed with AWX. The install acts differently every time. So, I just did a delete and reran the full install script. It created everything perfectly, but the site would not come up. The route was not published in OpenShift. I deleted the AWX pod, and let it recreate itself and that time it worked. In fact, it worked perfectly. I have all the missing components. So, I'm not able to demonstrate this time what happens when it's broken. 

When the site was not browseable, I ran the command at the web node the first time and received: (After getting this I did the pod refresh, and everything is working like a dream now.)
h-5.1$ awx-manage migrate --noinput
Operations to perform:
  Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit
Running migrations:
  Applying main.0002_squashed_v300_release...Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL:  Key (typname, typnamespace)=(main_towersettings_id_seq, 2200) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 201, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py", line 92, in database_forwards
    schema_editor.create_model(model)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/schema.py", line 345, in create_model
    self.execute(sql, params or None)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/schema.py", line 145, in execute
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL:  Key (typname, typnamespace)=(main_towersettings_id_seq, 2200) already exists.

sh-5.1$ 

Kevin Knox

unread,
Jan 4, 2023, 4:39:04 PM1/4/23
to AWX Project
Just deleted and rebuild again. After the fresh start, I have this in the Web log:
[wait-for-migrations] Waiting for database migrations... [wait-for-migrations] Attempt 1 of 30 [wait-for-migrations] Waiting 0.5 seconds before next attempt [wait-for-migrations] Attempt 2 of 30 [wait-for-migrations] Waiting 1 seconds before next attempt [wait-for-migrations] Attempt 3 of 30 [wait-for-migrations] Waiting 2 seconds before next attempt [wait-for-migrations] Attempt 4 of 30

So, doing the migration I get:
sh-5.1$ awx-manage migrate --noinput

Operations to perform:
  Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit
Running migrations:
  Applying main.0144_event_partitions...Traceback (most recent call last):

  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
psycopg2.errors.DuplicateTable: relation "_unpartitioned_main_jobevent" already exists
...

After that, the site comes up, and it's in good order. I have all the doohickeys, including the admin user and the EEs. So, it worked twice for me, both times after attempting the migration and getting an error. 

Is there any chance if I run the migration multiple times it could hose the database? Because when I see that first error, I have retried as a rule. 

Kevin

Kevin Knox

unread,
Jan 5, 2023, 8:59:59 AM1/5/23
to AWX Project
Driving me nuts, here. It's nothing to do with leftover data. I just created a new AWX namespace, and had the same issue. Deleted it, recreated it, and had the same issue. Why does it not always work and why does it sometimes work? It has to be something I'm doing, but I have no idea what.

Kevin Knox

unread,
Jan 5, 2023, 9:23:32 AM1/5/23
to AWX Project

A list of all the errors in the database startup is:
Search "Error:" (58 hits in 1 file of 1 searched)
    Line  57: 2023-01-05 14:16:09.323 UTC [60] ERROR:  relation "conf_setting" does not exist at character 158
    Line  59: 2023-01-05 14:16:09.931 UTC [61] ERROR:  relation "conf_setting" does not exist at character 158
    Line  61: 2023-01-05 14:16:12.079 UTC [64] ERROR:  relation "conf_setting" does not exist at character 158
    Line  63: 2023-01-05 14:16:12.358 UTC [64] ERROR:  relation "django_migrations" does not exist at character 124
    Line  65: 2023-01-05 14:16:12.360 UTC [64] ERROR:  relation "django_migrations" does not exist at character 124
    Line  67: 2023-01-05 14:16:12.371 UTC [64] ERROR:  relation "django_migrations" does not exist at character 124
    Line  69: 2023-01-05 14:16:12.372 UTC [64] ERROR:  relation "django_migrations" does not exist at character 124
    Line  71: 2023-01-05 14:16:12.381 UTC [64] ERROR:  relation "django_migrations" does not exist at character 124
    Line  73: 2023-01-05 14:16:12.406 UTC [64] ERROR:  relation "django_migrations" does not exist at character 124
    Line  75: 2023-01-05 14:16:12.408 UTC [64] ERROR:  relation "django_migrations" does not exist at character 124
    Line  77: 2023-01-05 14:16:12.643 UTC [65] ERROR:  relation "conf_setting" does not exist at character 158
    Line  79: 2023-01-05 14:16:12.970 UTC [65] ERROR:  relation "django_migrations" does not exist at character 124
    Line  81: 2023-01-05 14:16:12.972 UTC [65] ERROR:  relation "django_migrations" does not exist at character 124
    Line  83: 2023-01-05 14:16:13.128 UTC [65] ERROR:  relation "django_migrations" does not exist at character 124
    Line  85: 2023-01-05 14:16:13.130 UTC [65] ERROR:  relation "django_migrations" does not exist at character 124
    Line  87: 2023-01-05 14:16:13.132 UTC [65] ERROR:  relation "django_migrations" does not exist at character 124
    Line  89: 2023-01-05 14:16:13.143 UTC [65] ERROR:  relation "django_migrations" does not exist at character 124
    Line  91: 2023-01-05 14:16:13.144 UTC [65] ERROR:  relation "django_migrations" does not exist at character 124
    Line  93: 2023-01-05 14:16:14.149 UTC [67] ERROR:  relation "conf_setting" does not exist at character 158
    Line  95: 2023-01-05 14:16:14.380 UTC [67] ERROR:  relation "django_migrations" does not exist at character 124
    Line  97: 2023-01-05 14:16:14.389 UTC [67] ERROR:  relation "django_migrations" does not exist at character 124
    Line  99: 2023-01-05 14:16:14.390 UTC [67] ERROR:  relation "django_migrations" does not exist at character 124
    Line 101: 2023-01-05 14:16:14.416 UTC [67] ERROR:  relation "django_migrations" does not exist at character 124
    Line 103: 2023-01-05 14:16:14.418 UTC [67] ERROR:  relation "django_migrations" does not exist at character 124
    Line 105: 2023-01-05 14:16:14.561 UTC [67] ERROR:  relation "django_migrations" does not exist at character 124
    Line 107: 2023-01-05 14:16:14.563 UTC [67] ERROR:  relation "django_migrations" does not exist at character 124
    Line 109: 2023-01-05 14:16:15.534 UTC [70] ERROR:  relation "conf_setting" does not exist at character 158
    Line 111: 2023-01-05 14:16:16.286 UTC [71] ERROR:  relation "conf_setting" does not exist at character 158
    Line 113: 2023-01-05 14:16:18.248 UTC [75] ERROR:  relation "conf_setting" does not exist at character 158
    Line 115: 2023-01-05 14:16:18.354 UTC [76] ERROR:  relation "django_migrations" does not exist at character 124
    Line 117: 2023-01-05 14:16:18.371 UTC [76] ERROR:  relation "django_migrations" does not exist at character 124
    Line 119: 2023-01-05 14:16:18.372 UTC [76] ERROR:  relation "django_migrations" does not exist at character 124
    Line 121: 2023-01-05 14:16:18.384 UTC [76] ERROR:  relation "django_migrations" does not exist at character 124
    Line 123: 2023-01-05 14:16:18.386 UTC [76] ERROR:  relation "django_migrations" does not exist at character 124
    Line 125: 2023-01-05 14:16:18.501 UTC [75] ERROR:  relation "django_migrations" does not exist at character 124
    Line 127: 2023-01-05 14:16:18.510 UTC [75] ERROR:  relation "django_migrations" does not exist at character 124
    Line 129: 2023-01-05 14:16:18.515 UTC [75] ERROR:  relation "django_migrations" does not exist at character 124
    Line 131: 2023-01-05 14:16:18.517 UTC [75] ERROR:  relation "django_migrations" does not exist at character 124
    Line 133: 2023-01-05 14:16:18.520 UTC [75] ERROR:  relation "django_migrations" does not exist at character 124
    Line 135: 2023-01-05 14:16:18.536 UTC [76] ERROR:  relation "django_migrations" does not exist at character 124
    Line 137: 2023-01-05 14:16:18.538 UTC [76] ERROR:  relation "django_migrations" does not exist at character 124
    Line 139: 2023-01-05 14:16:18.936 UTC [77] ERROR:  relation "conf_setting" does not exist at character 158
    Line 141: 2023-01-05 14:16:22.253 UTC [80] ERROR:  relation "conf_setting" does not exist at character 158
    Line 143: 2023-01-05 14:16:23.054 UTC [81] ERROR:  relation "conf_setting" does not exist at character 158
    Line 145: 2023-01-05 14:16:25.052 UTC [84] ERROR:  relation "conf_setting" does not exist at character 158
    Line 147: 2023-01-05 14:16:25.833 UTC [85] ERROR:  relation "conf_setting" does not exist at character 158
    Line 149: 2023-01-05 14:16:30.196 UTC [88] ERROR:  relation "conf_setting" does not exist at character 158
    Line 151: 2023-01-05 14:16:30.800 UTC [89] ERROR:  relation "conf_setting" does not exist at character 158
    Line 153: 2023-01-05 14:16:33.031 UTC [92] ERROR:  relation "conf_setting" does not exist at character 158
    Line 155: 2023-01-05 14:16:33.666 UTC [93] ERROR:  relation "conf_setting" does not exist at character 158
    Line 157: 2023-01-05 14:16:40.339 UTC [97] ERROR:  relation "conf_setting" does not exist at character 158
    Line 159: 2023-01-05 14:16:40.647 UTC [98] ERROR:  relation "conf_setting" does not exist at character 158
    Line 161: 2023-01-05 14:16:43.035 UTC [101] ERROR:  relation "conf_setting" does not exist at character 158
    Line 163: 2023-01-05 14:16:43.415 UTC [102] ERROR:  relation "conf_setting" does not exist at character 158
    Line 165: 2023-01-05 14:16:54.027 UTC [105] ERROR:  relation "conf_setting" does not exist at character 158
    Line 167: 2023-01-05 14:16:54.276 UTC [106] ERROR:  relation "conf_setting" does not exist at character 158
    Line 169: 2023-01-05 14:16:56.351 UTC [109] ERROR:  relation "conf_setting" does not exist at character 158
    Line 171: 2023-01-05 14:16:56.497 UTC [110] ERROR:  relation "conf_setting" does not exist at character 158

In the database log, I see this:
chmod: changing permissions of '/var/run/postgresql': Operation not permitted
The files belonging to this database system will be owned by user "1005300000".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok


Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data/pgdata -l logfile start

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
waiting for server to start....2023-01-05 14:15:36.006 UTC [36] LOG:  starting PostgreSQL 13.9 (Debian 13.9-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-01-05 14:15:36.008 UTC [36] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-01-05 14:15:36.018 UTC [37] LOG:  database system was shut down at 2023-01-05 14:15:35 UTC
2023-01-05 14:15:36.025 UTC [36] LOG:  database system is ready to accept connections
 done
server started
CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

2023-01-05 14:15:36.308 UTC [36] LOG:  received fast shutdown request
waiting for server to shut down....2023-01-05 14:15:36.313 UTC [36] LOG:  aborting any active transactions
2023-01-05 14:15:36.315 UTC [36] LOG:  background worker "logical replication launcher" (PID 43) exited with exit code 1
2023-01-05 14:15:36.315 UTC [38] LOG:  shutting down
2023-01-05 14:15:36.335 UTC [36] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2023-01-05 14:15:36.443 UTC [1] LOG:  starting PostgreSQL 13.9 (Debian 13.9-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-01-05 14:15:36.443 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-01-05 14:15:36.443 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2023-01-05 14:15:36.449 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-01-05 14:15:36.456 UTC [51] LOG:  database system was shut down at 2023-01-05 14:15:36 UTC
2023-01-05 14:15:36.464 UTC [1] LOG:  database system is ready to accept connections
2023-01-05 14:16:09.323 UTC [60] ERROR:  relation "conf_setting" does not exist at character 158

Kevin Knox

unread,
Jan 5, 2023, 9:54:24 AM1/5/23
to AWX Project
Could this just be a matter of impatience? I left this one sit, untouched, for an hour and when I came back it was fully built and working. 

AWX Project

unread,
Jan 6, 2023, 1:26:03 PM1/6/23
to AWX Project
When first installing, until migrations are completed, there could be errors in the services that depend on the database -- this is to be expected. After migrations complete, things should sort themselves out and run successfully. Is this what you are seeing?

You can always check the status of migrations by shelling into a running container and running `awx-manage showmigrations`.

AWX Team

Kevin Knox

unread,
Jan 6, 2023, 3:35:41 PM1/6/23
to AWX Project
That helps. Thank you! 

Kevin
Reply all
Reply to author
Forward
0 new messages