Dynamically adding a field to a model - via contribute_to_class

[gerno...@gmail.com]

Hi,

I don't know if crossposting with stackoverflow is allowed here but essentially my problem is explained in https://stackoverflow.com/questions/46162104/django-dynamic-model-fields-and-migrations.

What I want to create is a custom ModelField of type DecimalField, that dynamically adds another ModelField (CharField) to the source model. As far as I can tell, this is nicely explained in https://blog.elsdoerfer.name/2008/01/08/fuzzydates-or-one-django-model-field-multiple-database-columns/.

Let's assume I start with a fresh project and app and add this code to models.py:

from django.db import models
from django.db.models import signals


_currency_field_name = lambda name: '{}_extension'.format(name)


class PriceField(models.DecimalField):

    def contribute_to_class(self, cls, name):
        # add the extra currency field (CharField) to the class
        if not cls._meta.abstract:
            currency_field = models.CharField(
                max_length=3, 
                editable=False,
                null=True, 
                blank=True
            )
            cls.add_to_class(_currency_field_name(name), currency_field)
        # add the original price field (DecimalField) to the class
        super().contribute_to_class(cls, name)

        # TODO: set the descriptor
        # setattr(cls, self.name, FooDescriptor(self))


class FooModel(models.Model):

    price = PriceField('agrhhhhh', decimal_places=3, max_digits=10, blank=True, null=True)

If I then call ./manage.py makemigrations the following migration file for the app is created:

# Generated by Django 1.11.4 on 2017-09-11 18:02
from __future__ import unicode_literals

from django.db import migrations, models
import testing.models


class Migration(migrations.Migration):

    initial = True

    dependencies = [
    ]

    operations = [
        migrations.CreateModel(
            name='FooModel',
            fields=[
                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
                ('price', testing.models.PriceField(blank=True, decimal_places=3, max_digits=10, null=True, verbose_name='agrhhhhh')),
                ('price_extension', models.CharField(blank=True, editable=False, max_length=3, null=True)),
            ],
        ),
    ]

All good, as far as I can tell. The problem comes up if I then call ./manage.py migrate which errors with the following exception:

./manage.py migrate testing
Operations to perform:
  Apply all migrations: testing
Running migrations:
  Applying testing.0001_initial...Traceback (most recent call last):
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/utils.py", line 63, in execute
    return self.cursor.execute(sql)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
sqlite3.OperationalError: duplicate column name: price_extension

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/migrations/operations/models.py", line 97, in database_forwards
    schema_editor.create_model(model)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/base/schema.py", line 303, in create_model
    self.execute(sql, params or None)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/base/schema.py", line 120, in execute
    cursor.execute(sql, params)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/utils.py", line 80, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/cachalot/monkey_patch.py", line 113, in inner
    out = original(cursor, sql, *args, **kwargs)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/utils.py", line 63, in execute
    return self.cursor.execute(sql)
  File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
django.db.utils.OperationalError: duplicate column name: price_extension

As said, I have a fresh project with no exisiting DB and tables so far. Why is it complaining that there allready exista a column named "price_extension"?. The migration file only contains one field called "price_extension"?

A sample project can be cloned from https://github.com/hetsch/django_testing. 

Thank you a lot for your help!

[Michal Petrucha]

> If I then call *./manage.py makemigrations* the following migration file

> for the app is created:
>
> # Generated by Django 1.11.4 on 2017-09-11 18:02
> from __future__ import unicode_literals
>
> from django.db import migrations, models
> import testing.models
>
>
> class Migration(migrations.Migration):
>
> initial = True
>
> dependencies = [
> ]
>
> operations = [
> migrations.CreateModel(
> name='FooModel',
> fields=[
> ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
> ('price', testing.models.PriceField(blank=True, decimal_places=3, max_digits=10, null=True, verbose_name='agrhhhhh')),
> ('price_extension', models.CharField(blank=True, editable=False, max_length=3, null=True)),
> ],
> ),
> ]
>
>

> All good, as far as I can tell. The problem comes up if I then call *./manage.py
> migrate* which errors with the following exception:

>
> ./manage.py migrate testing
> Operations to perform:
> Apply all migrations: testing
> Running migrations:
> Applying testing.0001_initial...Traceback (most recent call last):
> File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/utils.py", line 63, in execute
> return self.cursor.execute(sql)
> File "/usr/local/var/pyenv/versions/stockmanagement-3.6.2/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
> return Database.Cursor.execute(self, query)
> sqlite3.OperationalError: duplicate column name: price_extension
>

[...]

> As said, I have a fresh project with no exisiting DB and tables so far. Why
> is it complaining that there allready exista a column named
> "price_extension"?. The migration file only contains one field called
> "price_extension"?

I'm not quite certain about this, but I believe that when the
migration runner reconstructs a version of your FooModel, it iterates
over the list of fields, and adds each of them to the reconstructed
model class one by one. So what happens is that your custom PriceField
gets added, which in turn creates its price_extension field, and then
next thing, the migration runner adds the price_extension field one
more time. So you end up with two instances of the price_extension
field on the same model, and when eventually the create_model schema
operation is executed, it adds each of them as another column to the
table, which is obviously wrong.

As for what you could do to avoid this situation – I'd suggest that
you add an extra argument to PriceField which tells the field that it
shouldn't create the additional extension field. Then, you implement
the deconstruct method on the field, and include this additional flag.
That way, auto-detected migrations will include the extra argument to
the field instance, and there shouldn't be any duplication of the
extension field on the resulting models reconstructed by migrations.

Good luck,

Michal

[Melvyn Sopacua]

If I may suggest a better approach:

- Make the code as ImageField does (see it's width_field and height_field)
- Then focus on writing the migration for this field

Unfortunately, I don't see a way for custom fields to provide hints to
the migrations framework. So a field that automatically creates
another field is hard to deal with.

Another different approach is to use a custom ForeignKey [1]. Then you
create only one field, but at the cost of a join.

[1] https://github.com/furious-luke/django-address/blob/master/address/models.py#L289

> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/20170913120505.GO8762%40koniiiik.org.
> For more options, visit https://groups.google.com/d/optout.



--
Melvyn Sopacua