virtual fields and the migration framework

[Federico Capoano]

Hi everyone,

some time ago I submitted this ticket: https://code.djangoproject.com/ticket/23159 - because we had problems with the migration framework.

To sum up, the django-hstore library has a VirtualField implementation which maps hstore keys (by supplying a schema parameter) to virtual fields so that they can be used on models, the admin, and even in django-rest-framework APIs. 

After some interactions with Andrew Godwin, I implemented his suggestion, that was to let db_type() return None, which worked well, until this other issue came up: https://github.com/djangonauts/django-hstore/issues/103

In that case the migration framework would throw an error even if db_type returned None.

Luckily, Tim Graham suggested me how to reach the no-op branch of the migration framework: https://github.com/django/django/blob/190afb86187e42c449afe626cff31f65b4781aa2/django/db/backends/base/schema.py#L470-L475

I met Florian Apolloner at Pycon Italy, so I took advantage of his presence to ask some questions and hear some feedback, which I really appreciated and which really helped me.

I have been able to replicate the bug in a test case and implement the workaround, now comes the interesting part: here's the workaround code:

https://github.com/djangonauts/django-hstore/commit/47f5aa05834365e2d6c41234e03b028a827ee91c

As you can see, I had to add quite some ugly code just to enter that no-op branch.... there must be a better way! :-D

I talked with Florian about the possibility to have some way to flag virtual fields in order to enter that no-op branch more easily and he was not very sure about it but nonetheless he encouraged me to bring up this issue here. One of the reasons he was not sure is "what happens if a field has the skip flag and then it doesn't or viceversa?"

So I thought about proposing a definition of virtual field:

"A virtual field is a model field which it correlates to one or multiple concrete fields, but doesn't add or alter columns in the database."

By following this definition I suppose we can:

• allow the migration framework to enter the no-op branch if analyzing a virtual field
• if a concrete field becomes virtual (i can't immagine the use case but i'm reasoning just in case), the migration framework will drop the column, and the developer will have to adjust the migration in order to avoid losing data
• if a virtual field becomes concrete, the migration framework will add the right column and the developer will have to adjust the migration in order to correctly fill the concrete field with the data that is probably stored in another concrete field

Thank you for your attention.

Best regards

Federico Capoano

[Markus Holtermann]

Hey Federico,

I just had a brief look at the code. If I understand the HStoreVirtualMixin.contribute_to_class() correctly and its implications regarding add_field(), I'd try to use cls._meta.add_field(self, virtual=True) instead of a simple cls._meta.add_field(self). That way the virtual fields shouldn't show up in migrations at all: https://github.com/django/django/blob/master/django/db/migrations/state.py#L357 -- The way the e.g. DictionaryField is implemented, when you put the schema into the deconstruct() output, contribute_to_class() should still be called generating the additional fields on its own.

Note that this is just a shot in the dark. I didn't try it out myself.

/Markus

[Federico Capoano]

Hi Markus,

thank you very much for your suggestion. I'll try it later this afternoon and come back to report.

Federico

[Federico Capoano]

cls._meta.add_field(self, virtual=True)

gives:

  File "/var/www/django-hstore/django_hstore/virtual.py", line 29, in contribute_to_class

    cls._meta.add_field(self, virtual=True)

TypeError: Error when calling the metaclass bases

    add_field() got an unexpected keyword argument 'virtual'

I remember that I already tried flagging the field as virtual, but the result was not what I wanted, that is a field that would also show up in the admin.

There is an interesting ticket:

Provide real support for virtual fields

https://code.djangoproject.com/ticket/16508

Federico

[Federico Capoano]

I think the correct syntax is:

cls._meta.add_virtual_field(self)

The problem is that if I use this syntax it seems that the admin ignores these fields, what happens is that the changes I make to the virtual fields are ignored.

These are the test results related to the VirtualField implementation:

Creating test database for alias 'default'...

.FFF..E.......E.F.....F......

======================================================================

ERROR: test_create (django_hstore_tests.tests.SchemaTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/var/www/django-hstore/tests/django_hstore_tests/tests.py", line 1427, in test_create

    text='create3'

  File "/home/nemesis/.virtualenvs/django-hstore/local/lib/python2.7/site-packages/django/db/models/manager.py", line 92, in manager_method

    return getattr(self.get_queryset(), name)(*args, **kwargs)

  File "/home/nemesis/.virtualenvs/django-hstore/local/lib/python2.7/site-packages/django/db/models/query.py", line 370, in create

    obj = self.model(**kwargs)

  File "/home/nemesis/.virtualenvs/django-hstore/local/lib/python2.7/site-packages/django/db/models/base.py", line 452, in __init__

    raise TypeError("'%s' is an invalid keyword argument for this function" % list(kwargs)[0])

TypeError: 'boolean_true' is an invalid keyword argument for this function

======================================================================

ERROR: test_migrations (django_hstore_tests.tests.SchemaTests)

failing test for https://github.com/djangonauts/django-hstore/issues/103

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/var/www/django-hstore/tests/django_hstore_tests/tests.py", line 1565, in test_migrations

    call_command('migrate', 'django_hstore_tests')

  File "/home/nemesis/.virtualenvs/django-hstore/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 115, in call_command

    return klass.execute(*args, **defaults)

  File "/home/nemesis/.virtualenvs/django-hstore/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute

    output = self.handle(*args, **options)

  File "/home/nemesis/.virtualenvs/django-hstore/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 161, in handle

    executor.migrate(targets, plan, fake=options.get("fake", False))

  File "/home/nemesis/.virtualenvs/django-hstore/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 68, in migrate

    self.apply_migration(migration, fake=fake)

  File "/home/nemesis/.virtualenvs/django-hstore/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 102, in apply_migration

    migration.apply(project_state, schema_editor)

  File "/home/nemesis/.virtualenvs/django-hstore/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 108, in apply

    operation.database_forwards(self.app_label, schema_editor, project_state, new_state)

  File "/home/nemesis/.virtualenvs/django-hstore/local/lib/python2.7/site-packages/django/db/migrations/operations/fields.py", line 127, in database_forwards

    from_field = from_model._meta.get_field_by_name(self.name)[0]

  File "/home/nemesis/.virtualenvs/django-hstore/local/lib/python2.7/site-packages/django/db/models/options.py", line 420, in get_field_by_name

    % (self.object_name, name))

FieldDoesNotExist: SchemaDataBag has no field named u'datetime'

======================================================================

FAIL: test_admin_add (django_hstore_tests.tests.SchemaTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/var/www/django-hstore/tests/django_hstore_tests/tests.py", line 1305, in test_admin_add

    self.assertEqual(d.number, 3)

AssertionError: 0 != 3

======================================================================

FAIL: test_admin_add_utf8 (django_hstore_tests.tests.SchemaTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/var/www/django-hstore/tests/django_hstore_tests/tests.py", line 1317, in test_admin_add_utf8

    self.assertEqual(d.number, 3)

AssertionError: 0 != 3

======================================================================

FAIL: test_admin_change (django_hstore_tests.tests.SchemaTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/var/www/django-hstore/tests/django_hstore_tests/tests.py", line 1334, in test_admin_change

    self.assertEqual(d.number, 6)

AssertionError: 1 != 6

======================================================================

FAIL: test_reload_schema (django_hstore_tests.tests.SchemaTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/var/www/django-hstore/tests/django_hstore_tests/tests.py", line 1472, in test_reload_schema

    self.assertEqual(len(SchemaDataBag._meta.fields), len(original_schema) + concrete_fields_length)

AssertionError: 3 != 17

======================================================================

FAIL: test_schemadatabag_validation_error (django_hstore_tests.tests.SchemaTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/var/www/django-hstore/tests/django_hstore_tests/tests.py", line 1277, in test_schemadatabag_validation_error

    d.full_clean()

AssertionError: ValidationError not raised

----------------------------------------------------------------------

Ran 29 tests in 1.749s

FAILED (failures=5, errors=2)

Federico

[Florian Apolloner]

On Thursday, April 23, 2015 at 6:59:58 PM UTC+2, Federico Capoano wrote:

cls._meta.add_field(self, virtual=True)

gives:

  File "/var/www/django-hstore/django_hstore/virtual.py", line 29, in contribute_to_class

    cls._meta.add_field(self, virtual=True)

TypeError: Error when calling the metaclass bases

    add_field() got an unexpected keyword argument 'virtual'

This should definitely work on master/1.8: https://github.com/django/django/blob/master/django/db/models/options.py#L294 -- seems like you are on 1.7 or something.

cheers,
florian

[Federico Capoano]

Yes I am on 1.7 because 1.8 is not supported yet but I'm working on it.

I'll come back when I'll have more info.

Federico

[Federico Capoano]

Thank you very much for the suggestions.

So after using:

cls._meta.add_field(self, virtual=True)

instead of:

cls._meta.add_field(self)

cls._meta.virtual_fields.append(self)

The VirtualField implementation of django-hstore is much cleaner and does not conflict with the migration framework.

I wonder if it's possible to achieve a similar result in django 1.7?

Regarding this VirtualField implementation, in the near future I'd like to extract it in a separate python package. It doesn't have to be tied to django-hstore like it is now, it can potentially be used with any text field.

The idea is that you have something like a (hidden) JSONField or a TextField which contains the data of several other VirtualFields.

This would go in the direction of experimenting more with dynamic models in django.

I read something in the responses to the django community survey, so it seems I'm not the only one who would like to have flexible/dynamic models :-)

Federico