Custom Model field, to_python() / from_db_value() and unittests...

[Jens Diemer]

I try to create a custom model field, that should "Converting values to Python
objects" as described in the documentation here:

<https://docs.djangoproject.com/en/dev/howto/custom-model-fields/#converting-values-to-python-objects>

It doesn't work with Python 2.7 and 3.4 with Django 1.7.x and 1.8.x...


I search around and found no unittest case for this "howto/custom-model-fields"

So i created a simple test case to investigate why my code not worked.


I made a unittest against "stable/1.7.x", "stable/1.8.x" and "master" here:

https://github.com/jedie/django/branches/yours

compare links:

stable/1.7.x
<https://github.com/jedie/django/compare/stable/1.7.x...test_custom_model_fields_1.7.x>

stable/1.8.x
<https://github.com/jedie/django/compare/stable/1.8.x...test_custom_model_fields_1.8.x>

master
<https://github.com/jedie/django/compare/master...test_custom_model_fields_master>


I run these test and here the result:


*** 1.7.x with Py2:
test_custom_model_field (custom_model_fields.tests.TestModel1Tests) ... FAIL
test_values (custom_model_fields.tests.TestModel1Tests) ... FAIL
test_custom_model_field (custom_model_fields.tests.TestModel2Tests) ... ok
test_values (custom_model_fields.tests.TestModel2Tests) ... FAIL


*** 1.7.x with Py3:
test_custom_model_field (custom_model_fields.tests.TestModel1Tests) ... FAIL
test_values (custom_model_fields.tests.TestModel1Tests) ... FAIL
test_custom_model_field (custom_model_fields.tests.TestModel2Tests) ... FAIL
test_values (custom_model_fields.tests.TestModel2Tests) ... FAIL



*** 1.8.x with Py2:
test_custom_model_field (custom_model_fields.tests.TestModel1Tests) ... ok
test_values (custom_model_fields.tests.TestModel1Tests) ... ok


*** 1.8.x with Py3
test_custom_model_field (custom_model_fields.tests.TestModel1Tests) ... ok
test_values (custom_model_fields.tests.TestModel1Tests) ... ok



*** master with Py2 - doesn't run:
Traceback (most recent call last):
File "/home/jens/PyLucid_env/src/django/tests/runtests.py", line 12, in <module>
from django.apps import apps
File "/home/jens/PyLucid_env/src/django/django/apps/__init__.py", line 1, in
<module>
from .config import AppConfig # NOQA
File "/home/jens/PyLucid_env/src/django/django/apps/config.py", line 6, in
<module>
from django.utils.module_loading import module_has_submodule
File "/home/jens/PyLucid_env/src/django/django/utils/module_loading.py", line
4, in <module>
from importlib import import_module
File "/home/jens/PyLucid_env/src/django/django/utils/importlib.py", line 6,
in <module>
ImportError: cannot import name RemovedInDjango19Warning



*** master with Py3:
test_custom_model_field (custom_model_fields.tests.TestModel1Tests) ... ok
test_values (custom_model_fields.tests.TestModel1Tests) ... ok
test_custom_model_field (custom_model_fields.tests.TestModel2Tests) ... ok
test_values (custom_model_fields.tests.TestModel2Tests) ... ok




So the biggest problem is django 1.7.x...
But it should work in the same way:
<https://docs.djangoproject.com/en/1.7/howto/custom-model-fields/#converting-database-values-to-python-objects>


With 1.8.x i must remove the test with existing:
__metaclass__ = models.SubfieldBase


Otherwise the test will not run:

Testing against Django installed in '/home/jens/PyLucid_env/src/django/django'
Importing application custom_model_fields
Traceback (most recent call last):
File "/home/jens/PyLucid_env/src/django/tests/runtests.py", line 448, in <module>
options.debug_sql)
File "/home/jens/PyLucid_env/src/django/tests/runtests.py", line 235, in
django_tests
state = setup(verbosity, test_labels)
File "/home/jens/PyLucid_env/src/django/tests/runtests.py", line 214, in setup
apps.set_installed_apps(settings.INSTALLED_APPS)
File "/home/jens/PyLucid_env/src/django/django/apps/registry.py", line 324,
in set_installed_apps
self.populate(installed)
File "/home/jens/PyLucid_env/src/django/django/apps/registry.py", line 108,
in populate
app_config.import_models(all_models)
File "/home/jens/PyLucid_env/src/django/django/apps/config.py", line 198, in
import_models
self.models_module = import_module(models_module_name)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/jens/PyLucid_env/src/django/tests/custom_model_fields/models.py",
line 41, in <module>
class CommaSeparatedModelField2(CommaSeparatedModelField1):
File
"/home/jens/PyLucid_env/src/django/django/db/models/fields/subclassing.py", line
22, in __new__
RemovedInDjango20Warning)
django.utils.deprecation.RemovedInDjango20Warning: SubfieldBase has been
deprecated. Use Field.from_db_value instead.



Maybe i miss something to handle warnings with tests, but didn't find something
about warnings here:

https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/



--


Mfg.

Jens Diemer


----
http://www.jensdiemer.de

[Tim Graham]

The doc about how to ignore warnings in tests is here: https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/submitting-patches/#deprecating-a-feature

Alternatively, you can temporarily remove these lines in runtests.py:

warnings.simplefilter("error", RemovedInDjango20Warning)
warnings.simplefilter("error", RemovedInDjango21Warning)

[Jens Diemer]

Am 16.06.2015 um 18:43 schrieb Tim Graham:
> The doc about how to ignore warnings in tests is here:
> https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/submitting-patches/#deprecating-a-feature
>
> Alternatively, you can temporarily remove these lines in runtests.py:
>
> warnings.simplefilter("error", RemovedInDjango20Warning)
> warnings.simplefilter("error", RemovedInDjango21Warning)

Thanks!

But then, there can't exists tests that will raise a warning while importing ?!?




So i try to make it clear:

With v1.7 the "__metaclass__ = models.SubfieldBase" is needed. So i remove the
tests without it.

And i found a existing Bug https://code.djangoproject.com/ticket/9619 for:

to_python not called when fetching data with .values(...)


I update the tests and create a ticket and pull request here:
* https://code.djangoproject.com/ticket/24993
* https://github.com/django/django/pull/4874

I can also made pull request for v1.8.x and master...





I also found the Solution for: "to_python() didn't call with Python 3":
The "__metaclass__" syntax changed in Python 3.

The Problem: I didn't read the doc carefully here:

https://docs.djangoproject.com/en/1.7/howto/custom-model-fields/#the-subfieldbase-metaclass

There are three code examples:
* for Python 2 only
* for Python 3 only
* for Python 2+3 using six.with_metaclass()

What's about to remove the first two examples and leave only the
six.with_metaclass() example?!?

I made also a ticket/pull request for this:
* https://code.djangoproject.com/ticket/24992
* https://github.com/django/django/pull/4873

> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to django-develop...@googlegroups.com
> <mailto:django-develop...@googlegroups.com>.
> To post to this group, send email to
> django-d...@googlegroups.com
> <mailto:django-d...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/b7082532-d985-477a-947e-5afe39772e9c%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/b7082532-d985-477a-947e-5afe39772e9c%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[charettes]

Hi Jens,

I'm not sure I understand what you are trying to achieve but I assume you want to write a third-party field that supports both Django 1.7 and 1.8 without raising deprecating warnings?

I suggest you use the following pattern which also accounts for py2/3:

import django
from django.db import models
from django.utils.six import with_metaclass

MyFieldBase = type if django.VERSION >= (1, 8) else models.SubfieldBase

class MyField(with_metaclass(MyFieldBase, models.Field)):
    def from_db_value(self, value, expression, connection, context):
        pass

    def to_python(self, value):
        pass

This is stepping into the django-user@ territory so I suggest we move the discussion over there if the provided example doesn't match your needs but you are really just trying to write a portable third-party field.

Simon

> <mailto:django-developers+unsub...@googlegroups.com>.

[Jens Diemer]

Am 17.06.2015 um 16:48 schrieb charettes:
> I suggest you use the following pattern which also accounts for py2/3:
>

> ...

>
> This is stepping into the django-user@ territory so I suggest we move the
> discussion over there if the provided example doesn't match your needs but you
> are really just trying to write a portable third-party field.

Thanks for you suggestion.

I "moved" it to django-user list ;)