Migration running in shell, but no change in DB

[Martin Heitmann]

Hello everyone

I have a project with multiple apps in it. As database I use MariaDB. Have not touched it for a while, but now I had to add a field to the models of one app. makemigrations and migrate run without any indication of an error. But no change occurs in the db. Tested it with an altered models.py in another app and the result is the same. Do you have any advice how to narrow this down?

Best regards

Martin

[David Nugent]

I would check in the django_migrations table to ensure that the migration has successfully been run.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/be9ef414-4b51-4072-a1b7-b7ba036016aan%40googlegroups.com.

[Martin Heitmann]

The migrations do not show up in the django_mirations table. The problem must occur earlier.

[Jason]

part of getting effective help is sharing how you're doing things and the code.  for example, sharing the migration generated would be helpful. you do have the context of how and what is being executed, we don't, so sharing that with your original questions is a big part in getting effective help

[Muhammad Juwaini Abdul Rahman]

Try running:

```

./manage.py makemigrations <app_name>

```

If there are no changes detected by makemigrations, something wrong with your app. Check INSTALLED_APPS in settings.py for any error.

[Martin Heitmann]

Dear Jason
Thanks for your reply. I will gladly do that. This is the latest migration file:

# Generated by Django 3.2.10 on 2023-04-01 13:45

from django.db import migrations, models
import django.utils.timezone


class Migration(migrations.Migration):

    dependencies = [
        ('ObjPLW2', '0037_alter_waeschepaket_w_bezahlt'),
    ]

    operations = [
        migrations.CreateModel(
            name='Wartungsarbeit',
            fields=[
                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
                ('wa_titel', models.CharField(max_length=201)),
                ('wa_beschreibung', models.TextField()),
                ('wa_turnus', models.PositiveIntegerField(default=1)),
                ('wa_startdatum', models.DateField(default=django.utils.timezone.now)),
                ('wa_erstelldatum', models.DateTimeField(auto_now_add=True)),
                ('wa_bearbeitungsdatum', models.DateTimeField(auto_now=True)),
            ],
        ),
    ]

Does this help?

[Martin Heitmann]

Dear Muhammad

Thanks, but I did that several times. And I also check the INSTALLED_APPS. It is listed there, as it was before. :/

Best regards

Martin

[Martin Heitmann]

As far as I understand it, the migration file looks fine, but for some reason the changes are not applied to the database. The database is otherwise working, I can call the app, work with it and make entries in the old data model. But for whatever reasons the structure of the data model won't change. Therefore I need some kind of debugging to see what's happening under the hood ... or actually what's not happening.

[Martin Heitmann]

Dear Jason

I went two migration steps back, then again I did makemigrations and then I went for sqlmigrate. This was the output:

python3 manage.py sqlmigrate ObjPLW2 0037
--
-- Create model Wartungsarbeit
--
--
-- Alter field w_bezahlt on waeschepaket
--
It does not include any SQL.

Does anyone know where this is no SQL in the output?

[Martin Heitmann]

I have a little update, but not a solution. Adding the database to sqlmigrate will give me the SQL command, but nothing more:

python3 manage.py sqlmigrate ObjPLW2 0037 --database=db_obj_plw2

--
-- Create model Wartungsarbeit
--

CREATE TABLE `ObjPLW2_wartungsarbeit` (`id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY, `wa_titel` varchar(201) NOT NULL, `wa_beschreibung` longtext NOT NULL, `wa_turnus` integer UNSIGNED NOT NULL CHECK (`wa_turnus` >= 0), `wa_startdatum` date NOT NULL, `wa_erstelldatum` datetime(6) NOT NULL, `wa_bearbeitungsdatum` datetime(6) NOT NULL);

--
-- Alter field w_bezahlt on waeschepaket
--

And for whatever reason it includes only the table creation and not the change in the other field.

[Martin Heitmann]

A little PS: I logged in via phpmyadmin and saw that my user has all privileges. So this should not be the source of the problem.

[Jason]

this is all a pretty weird experience, I have to say.  One thing that is suggesting to me that your migration history is.out of sync is that migration you shared earlier is not what I would expect an addField operation, not a CreateModel. 

Also, django migrations are required when model changes, but not all model changes require emitting SQL.  What exactly is the change you're expecting to see? What field, what was it before, and what do you want it to go to? 

[Martin Heitmann]

Dear Jason

What could I do to get it back in sync? I tried several recommendations from the web, but perhaps the devil is in the details. If someone could tell me which steps I should take, I would gladly roll back the migrations and sync it again. And I would like to share the output. However, I have the existing database and may not loose the existing content.

As it seems to me I wanted to change a field more than a year ago. Perhaps that did not work. I do not remember. And now I wanted to add a new model to the app.

Best regards

Martin

[Martin Heitmann]

Would you recommend using inspectdb?

[Jason]

What specifically was the changeset occurring and what isn't?  Is the model being created in the db, or is it the field change that's not replicating?  In your comments, you showed just one migration, but seems from additional comments you made, there are two or more?  What are they, specifically.  And what is the model field change you're making?

As mentioned earlier,  django model field changes do not always equate to a migration for that model.  So django made a migration for your model, but might also be an empty/no op one for your field change

[Martin Heitmann]

Dear Jason

Thanks for your reply. I am not absolulety sure what this question means "What specifically was the changeset occurring and what isn't? ", but I try to answer it the best way I can. I worked on the app quite a while ago and back then everything seemed to work. As far as I can understand my own steps back then I tried to alter only one field.

extract from migrations file:

 migrations.AlterField(
            model_name='waeschepaket',
            name='w_bezahlt',
            field=models.BooleanField(choices=[(False, 'Nein'), (True, 'Ja')], default=False),
        ),

Here is a small screenshot from phpmyadmin

I guess that if that field alteration would have worked than the default value should be False or something else but not None.

Then I guess that I forgot about it and just worked with app as it was for more than a year and now I wanted to add a new model:

operations = [
        migrations.CreateModel(
            name='Wartungsarbeit',
            fields=[
                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
                ('wa_titel', models.CharField(max_length=201)),
                ('wa_beschreibung', models.TextField()),
                ('wa_turnus', models.PositiveIntegerField(default=1)),
                ('wa_startdatum', models.DateField(default=django.utils.timezone.now)),
                ('wa_erstelldatum', models.DateTimeField(auto_now_add=True)),
                ('wa_bearbeitungsdatum', models.DateTimeField(auto_now=True)),
            ],
        ),

But this does not show up in any way. During the last weeks I have tried a lot of things that I read on the web and I am afraid that I may have made a mess. There were many instructions about inspectdb, --fake-initial, removing the app-specific entries from the table django_migrations and so on. But nothing of that worked. Now I am at a point where I merged the last two, obviously not working migrations into one, which is currently the 37th for this app.

As I already have a lot of content in the database I don't want to loose that and instead I search for a way to start fresh and get rid of that error. And if possible I would like to know where that error originated. But currently I am at my wits end. First I thought that it must have to do with the database access, but then the app would not work and other content could not be entered and changed and so on. I checked my privileges via phpmyadmin. Everything is fine there, too. And of course the app is in INSTALLED_APPS, otherwise it would not work as well. One more thing that I am not sure of is the database routing. I have a router.py and there I have pretty much the same entries for all apps. Looks similar to this:

def allow_migrate(self, db, app_label, model_name=None, **hints):
        if app_label == 'app_name':
            return db == 'db_app_name'
        return None

I don't know if this can be relevant for this problem.

@Jason: Could I answer your questions?

Best regards

Martin

[Jason]

so, for your original boolean migration, 

https://code.djangoproject.com/ticket/470

https://stackoverflow.com/questions/32280343/default-value-of-djangos-model-doesnt-appear-in-sql

Adding a default value doesn't go to SQL, although there's work in the django ticket above.  So its inserted via app code and the original question might have been a red herring and the actual issue is your CreateModel is never applied.

ouch, you have one db per application?  that could likely be it, when was this routing put in place?  Did it exist before the last working migration on this project?  Reason I ask, 

python3 manage.py sqlmigrate ObjPLW2 0037 --database=db_obj_plw2

--
-- Create model Wartungsarbeit
--

CREATE TABLE `ObjPLW2_wartungsarbeit` (`id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY, `wa_titel` varchar(201) NOT NULL, `wa_beschreibung` longtext NOT NULL, `wa_turnus` integer UNSIGNED NOT NULL CHECK (`wa_turnus` >= 0), `wa_startdatum` date NOT NULL, `wa_erstelldatum` datetime(6) NOT NULL, `wa_bearbeitungsdatum` datetime(6) NOT NULL);

is vaid SQL and that's what django would send to your db.  So if its not being applied, the allow_migrate might be at cause here.  What happens when you put some logging there, or raise an exception on None/False?  The docs at https://docs.djangoproject.com/en/4.1/topics/db/multi-db/#allow_migrate state

makemigrations always creates migrations for model changes, but if allow_migrate() returns False, any migration operations for the model_name will be silently skipped when running migrate on the db. 

[Martin Heitmann]

Dear Jason

Thanks for all your effort.

Yes, I have one db per application. Isn't that a way to keep the databases clean and strictly focused on one purpose? Thought that that would be a good idea. Regarding the router.py: It worked for quite a while, but I am not sure if I made changes to it previous to the first occurence of the problem. But should it not work when every router has an entry like this:

def allow_migrate(self, db, app_label, model_name=None, **hints):
        if app_label == 'app_name':
            return db == 'db_app_name'
        return None

You wrote "What happens when you put some logging there, or raise an exception on None/False? ". I would love to do that but I do not know, where I must put that. I guess I must find some place in the code that calls allow_migrate and check what the return value is, right?

[Jason]

ahhh, no not at all.  One db per project, yes.  Application, no.  That's really expanding complexity. If you need that requirement, that's one thing, but its something you really should have an explicit requirement for before designing.

As far as the second question, no not at all.  Just add logging there, or raise an exception

ie, logger.info("db does not match db app name"), or raise Exception("db does not match app name")

[Martin Heitmann]

Dear Jason

db per project/app: Okay, I will keep that in mind for the future. I just thought that it would be a cleaner way when each db and app would serve a single purpose

logging: I am sorry, but I have to ask again, where do I put that what you wrote ( logger.info("db does not match db app name"), or raise Exception("db does not match app name"))? Does this belong in the router.py or migration file or another place?

Best regards

Martin

[Jason]

you said you have a router.py file in your project?  put it in there.

[Martin Heitmann]

Yes I will do that. In the meantime I tried this and that and saw that the migrations do not lead to new tables in the database, but the entries appear in the default db's django_migrations table and not in the one of this app.

[Martin Heitmann]

So,

I put this at the start of my router.py:

import logging

logger = logging.getLogger(__name__)

I have never used that, but I hope it is right.

Then below at the app's router and the allow_migrate def I did this:

 def allow_migrate(self, db, app_label, model_name=None, **hints):
        if app_label == 'app_name':

            logger.info("case A")
            return db == 'db_app_name'
        else:
            logger.info("case B")
            return None

Then I returned to the shell and I also reverted to the previous migration. Then I entered

python3 manage.py migrate app_name
Operations to perform:
  Apply all migrations: app_name
Running migrations:
  Applying app_name.0037_auto_20230406_1837... OK

Again I turned to phpymadmin and saw that neither the apps db had changed or that there was an entry in app's db table for django_migrations. On the contrary there was a new entry in the django_migrations table of the default db.

Does this help?

[Martin Heitmann]

Oh, it might be that I got it to work by adding --database=db_app_name to the migrate command. But I don't know why this helped.