migrate exception if multiple databases migrated in a specific order

[Robert Leach]

I just added a second database to our project and was questioning my sanity when I kept getting the exception below in the remote github action tests only (not locally).  I eventually noticed that I was doing the migrations in the reverse order locally.  Once I noticed that, I was able to both reproduce the error locally and avoid the error in the github action.

There are the commands whose order matters:

Works:

python manage.py migrate
python manage.py migrate --database=validation

Results int eh exception:

python manage.py migrate --database=validation
python manage.py migrate

I do not understand why the order matters, but my guess is that there is some code that runs in the migrate command that ignores the --database setting when creating relations.  Is this true?  Or is there a reasonable reason why it MUST be done in the order `default` then `non-default`?

Here is what I have in settings.py:

```
DATABASES = {
   "default": {
       "ENGINE": "django.db.backends.postgresql",
       "NAME": env("DATABASE_NAME"),
       "USER": env("DATABASE_USER"),
       "PASSWORD": env("DATABASE_PASSWORD"),
       "HOST": env("DATABASE_HOST"),
       "PORT": env("DATABASE_PORT"),
   }
}

VALIDATION_ENABLED = False
# If the validation database is configured in the .env file...
if env("VALIDATION_DATABASE_NAME"):
   try:
       DATABASES["validation"] = {
           "ENGINE": "django.db.backends.postgresql",
           "NAME": env("VALIDATION_DATABASE_NAME"),
           "USER": env("VALIDATION_DATABASE_USER"),
           "PASSWORD": env("VALIDATION_DATABASE_PASSWORD"),
           "HOST": env("VALIDATION_DATABASE_HOST"),
           "PORT": env("VALIDATION_DATABASE_PORT"),
       }
       VALIDATION_ENABLED = True
   except Exception as e:
       print(
           f"Could not configure access to the {env('VALIDATION_DATABASE_NAME')} database: {e}"
       )
```

The exception I get is below.  So have I found a bug or is there something for me to learn here?

Thanks,
Rob


The exception:

Running migrations:
 Applying DataRepo.0001_initial... OK
 Applying DataRepo.0002_RemoveDefaultProtocolCategory... OK
 Applying DataRepo.0003_remove_animal_state... OK
 Applying DataRepo.0004_StandardizeMsrun... OK
 Applying DataRepo.0005_auto_20210721_1444... OK
 Applying DataRepo.0006_auto_20210909_1607... OK
 Applying DataRepo.0007_tissue_description... OK
 Applying DataRepo.0008_null_original_peakdata... OK
 Applying DataRepo.0009_compound_synonym... OK
Traceback (most recent call last):
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
   return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "DataRepo_compound" does not exist
LINE 1: ...nd"."formula", "DataRepo_compound"."hmdb_id" FROM "DataRepo_...
                                                            ^
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "manage.py", line 22, in <module>
   main()
 File "manage.py", line 18, in main
   execute_from_command_line(sys.argv)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
   utility.execute()
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
   self.fetch_command(subcommand).run_from_argv(self.argv)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
   self.execute(*args, **cmd_options)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
   output = self.handle(*args, **options)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped
   res = handle_func(*args, **kwargs)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 244, in handle
   post_migrate_state = executor.migrate(
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
   state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
   state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
   state = migration.apply(state, schema_editor)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/migrations/migration.py", line 126, in apply
   operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
   self.code(from_state.apps, schema_editor)
 File "/home/runner/work/tracebase/tracebase/DataRepo/migrations/0010_default_compound_synonyms.py", line 9, in create_default_synonyms
   for compound in Compound.objects.all():
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/models/query.py", line 280, in __iter__
   self._fetch_all()
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
   self._result_cache = list(self._iterable_class(self))
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/models/query.py", line 51, in __iter__
   results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
   cursor.execute(sql, params)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
   return super().execute(sql, params)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
   return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
   return executor(sql, params, many, context)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
   return self.cursor.execute(sql, params)
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
   raise dj_exc_value.with_traceback(traceback) from exc_value
 File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
   return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "DataRepo_compound" does not exist
LINE 1: ...nd"."formula", "DataRepo_compound"."hmdb_id" FROM "DataRepo_...
                                                            ^
 Applying DataRepo.0010_default_compound_synonyms...
Error: 
Process completed with exit code 1.
Make the cache table in the validation database