1.11: Value error on related user name during save of user model

[Axel Rau]

While upgrading from 1.9 to 1.11, ForwardManyToOneDescriptor.__set__()
tries to assign a string (description of the related instance) to the related user name field
of user model.

class AbstractEmailUser(AbstractBaseUser, PermissionsMixin, FieldlistForDetailTemplateMixin):
localemail = models.OneToOneField('Mailbox', verbose_name=_('Local E-mail'),
related_name='localemail', db_column='localemail',
editable=('UR', 'UE', 'UL'))

objects = UserManager()

USERNAME_FIELD = 'localemail'
REQUIRED_FIELDS = []

class Meta:
abstract = True
ordering = ['localemail']

def get_username(self):
return getattr(self, self.USERNAME_FIELD)

class Mailbox(models.Model):
id = models.AutoField(primary_key=True)
localpart = models.CharField(_('Localpart'), max_length=40)
localdomainfk = models.ForeignKey(Localdomain, verbose_name=_('Domain'), db_column='localdomainfk', editable=('AL',))
…

def __str__(self):
return self.localpart+ '@'+self.localdomainfk.name


Internal Server Error: /admin/erdb/account/19/change/
Traceback (most recent call last):
File "...python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "...python3.5/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
response = self._get_response(request)
File "...python3.5/site-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "...python3.5/site-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/contextlib.py", line 30, in inner
return func(*args, **kwds)
File "...python3.5/site-packages/django/contrib/admin/options.py", line 551, in wrapper
return self.admin_site.admin_view(view)(*args, **kwargs)
File "...python3.5/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "...python3.5/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "...python3.5/site-packages/django/contrib/admin/sites.py", line 224, in inner
return view(request, *args, **kwargs)
File "...python3.5/site-packages/django/contrib/admin/options.py", line 1511, in change_view
return self.changeform_view(request, object_id, form_url, extra_context)
File "...python3.5/site-packages/django/utils/decorators.py", line 67, in _wrapper
return bound_func(*args, **kwargs)
File "...python3.5/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "...python3.5/site-packages/django/utils/decorators.py", line 63, in bound_func
return func.__get__(self, type(self))(*args2, **kwargs2)
File "...python3.5/site-packages/django/contrib/admin/options.py", line 1408, in changeform_view
return self._changeform_view(request, object_id, form_url, extra_context)
File "...python3.5/site-packages/django/contrib/admin/options.py", line 1440, in _changeform_view
if form.is_valid():
File "...python3.5/site-packages/django/forms/forms.py", line 183, in is_valid
return self.is_bound and not self.errors
File "...python3.5/site-packages/django/forms/forms.py", line 175, in errors
self.full_clean()
File "...python3.5/site-packages/django/forms/forms.py", line 386, in full_clean
self._post_clean()
File "...python3.5/site-packages/django/forms/models.py", line 408, in _post_clean
self.instance.full_clean(exclude=exclude, validate_unique=False)
File "...python3.5/site-packages/django/db/models/base.py", line 1234, in full_clean
self.clean()
File "...python3.5/site-packages/django/contrib/auth/base_user.py", line 77, in clean
setattr(self, self.USERNAME_FIELD, self.normalize_username(self.get_username()))
File "...python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 216, in __set__
self.field.remote_field.model._meta.object_name,
ValueError: Cannot assign "'unp...@framailx.de'": "Account.localemail" must be a "Mailbox" instance.
[21/Aug/2017 16:08:37] "POST /admin/erdb/account/19/change/ HTTP/1.1" 500 166385

Any help to resolve this appreciated,
Axel
---
PGP-Key:29E99DD6 ☀ computing @ chaos claudius

[James Schneider]

On Mon, Aug 21, 2017 at 7:50 AM, Axel Rau <Axel...@chaos1.de> wrote:

While upgrading from 1.9 to 1.11, ForwardManyToOneDescriptor.__set__()
tries to assign a string (description of the related instance) to the related user name field
of user model.

class AbstractEmailUser(AbstractBaseUser, PermissionsMixin, FieldlistForDetailTemplateMixin):
    localemail = models.OneToOneField('Mailbox', verbose_name=_('Local E-mail'),
            related_name='localemail', db_column='localemail',
            editable=('UR', 'UE', 'UL'))

    objects = UserManager()

    USERNAME_FIELD = 'localemail'
    REQUIRED_FIELDS = []

    class Meta:
        abstract = True
        ordering = ['localemail']

    def get_username(self):
       return getattr(self, self.USERNAME_FIELD)

class Mailbox(models.Model):
    id = models.AutoField(primary_key=True)
    localpart = models.CharField(_('Localpart'), max_length=40)
    localdomainfk = models.ForeignKey(Localdomain,  verbose_name=_('Domain'), db_column='localdomainfk', editable=('AL',))
        …

    def __str__(self):
        return self.localpart+ '@'+self.localdomainfk.name

<snip>

 

  File "...python3.5/site-packages/django/forms/models.py", line 408, in _post_clean
    self.instance.full_clean(exclude=exclude, validate_unique=False)
  File "...python3.5/site-packages/django/db/models/base.py", line 1234, in full_clean
    self.clean()
  File "...python3.5/site-packages/django/contrib/auth/base_user.py", line 77, in clean
    setattr(self, self.USERNAME_FIELD, self.normalize_username(self.get_username()))
  File "...python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 216, in __set__
    self.field.remote_field.model._meta.object_name,
ValueError: Cannot assign "'unp...@framailx.de'": "Account.localemail" must be a "Mailbox" instance.
[21/Aug/2017 16:08:37] "POST /admin/erdb/account/19/change/ HTTP/1.1" 500 166385

Any help to resolve this appreciated,
Axel

You've set your USERNAME_FIELD to a field that is an FK to another table. I've never seen that before, but apparently the docs say it is supported. To be fair, that's the extent of the documentation on how to utilize an FK as the username field. 

https://docs.djangoproject.com/en/1.11/topics/auth/customizing/#django.contrib.auth.models.CustomUser.USERNAME_FIELD

It makes sense to me that this particular operation doesn't work. You're asking (by way of the auth model definition through a ModelForm, I'm assuming) for the string value of 'unp...@framailx.de' to be set as the value of an FK field on the Account model (which I assume is the one that inherits from AbstractEmailUser) for the localemail field, when in reality you would want to set a value that matches the PK of the related row in the Mailbox model. Django will perform this PK magic automatically if you provide a Mailbox object as the value for Account.localemail, rather than the email string itself (per the docs). It will not automatically convert an email address to a Mailbox object, or determine which row should be referenced by Account.localemail without help, as far as I know.

TL;DR; the error occurs when 'unp...@framailx.de' is set on Account.localemail, when Account.localemail is expecting either a Mailbox object (that has a PK), or an integer matching a PK in the Mailbox table. It doesn't know how to translate a raw email address string to a Mailbox object.

I suppose the 'fix' would be to retrieve a Mailbox object based on the email address provided, and use that as the value for Account.localmail. You should be able to implement that in a custom ModelForm as part of clean_localemail(). 

If this was working under 1.9 (I'd be surprised if it was), then I would file this as a backward-incompatibility bug with the bug tracker. It's possible that 1.9 did do the type/field conversion using some black magic, but with the information in your model, I don't know how it could have made that determination. 

From a design perspective, the relation you presented seems backwards to me. I would think the Account model would hold the canonical username, and Mailbox would have the FK back to the Account. That design would not require the use of a username field as an FK, simplifying everything (authentication, forms, views, etc.). Your current design would implement excessive SQL JOIN's to pull relatively simple data on every request. 

Or, I could be completely wrong. :-D

-James

[Axel Rau]

Thanks for taking the time to write such an extensive answer.

Some background info:

This is a realtime operational database, for which I try to create a GUI with Django.

The DB is fully normalized and multi-client.

To reduce the complexity, I have created a concept for shortcuts of model instances, which are used in list views or FK form fields.

Each model has a __str__() method which returns such a shortcut (like the above Mailbox) for an instance.

There are other concepts like

  Model forms just to display data, or, 

  For the multi-client part, the PK of the current user is kept in the session along with some Q-expressions to qualify account dependent queries.

My current problem with 1.11 is a model form for Account, which contains a FK field referencing Mailbox in which the above shortcut is displayed as a pop up (forms logic treats remote instance as text).

I changed something else in the form and tried to save with the Mailbox field unchanged.

It seems that forms logic does no longer manage to get the instance from its text representation.

When I access a model instance, AFAIK my context determines the data type I get (string or model instance).

Is this correct?

If this was working under 1.9 (I'd be surprised if it was), then I would file this as a backward-incompatibility bug with the bug tracker. It’s possible that 1.9 did do the type/field conversion using some black magic, but with the information in your model, I don't know how it could have made that determination. 

I have just verified: It still works with 1.9. (-;

From a design perspective, the relation you presented seems backwards to me. I would think the Account model would hold the canonical username, and Mailbox would have the FK back to the Account. That design would not require the use of a username field as an FK, simplifying everything (authentication, forms, views, etc.). Your current design would implement excessive SQL JOIN's to pull relatively simple data on every request. 

Or, I could be completely wrong. :-D

Perhaps, I should publish the project at github (it’s OSS anyway), to get more advice?

[James Schneider]

On Tue, Aug 22, 2017 at 12:31 PM, Axel Rau <Axel...@chaos1.de> wrote:

Thanks for taking the time to write such an extensive answer.

Thanks. I should really sleep more...

 

 

Some background info:

This is a realtime operational database, for which I try to create a GUI with Django.

The DB is fully normalized and multi-client.

I'll take your word that it's normalized, but I still think conceptually that the username field is in the wrong spot and IMO is the primary cause of this entire conversation. ;-)

As always, there may be a deeper context for the design choice outside the scope of this thread. It probably isn't worth the time to debate because what you are trying to do is supported AFAICT.

 

To reduce the complexity, I have created a concept for shortcuts of model instances, which are used in list views or FK form fields.

Each model has a __str__() method which returns such a shortcut (like the above Mailbox) for an instance.

There are other concepts like

  Model forms just to display data, or, 

Why use a form simply to display data? I'm assuming just for the form.as_table type shortcuts?

 

  For the multi-client part, the PK of the current user is kept in the session along with some Q-expressions to qualify account dependent queries.

Assuming you are referring to the PK of the Mailbox instance containing the username. If you make use of request.user or {{ user }} in the templates, you can access the PK of the user by using request.user.localemail_id or {{ user.localemail_id }}, rather than keeping track of stuff in the session (which the request middleware is already doing assuming you are using it). Might be easier than trying to manage the session data. It's a fun trick that Django doesn't make obvious:

https://docs.djangoproject.com/en/1.11/ref/models/querysets/#values

See the "A few subtleties that are worth mentioning:" section.

As a bonus, if you only need the PK of the related Mailbox object, using the user.localemail_id will only return the PK value, and will not perform an SQL lookup to return the entire Mailbox object just to retrieve the PK like user.localemail.pk would. Comes in handy when trying to minimize queries.

 

My current problem with 1.11 is a model form for Account, which contains a FK field referencing Mailbox in which the above shortcut is displayed as a pop up (forms logic treats remote instance as text).

I changed something else in the form and tried to save with the Mailbox field unchanged.

It seems that forms logic does no longer manage to get the instance from its text representation.

I suppose that is the more succinct way to explain what I was trying to say. As I mentioned, if the same code is working under 1.9, check the release notes for 1.10 and 1.11 to make sure that they haven't deprecated a feature you may have been taking advantage of. I "think" that jumping from 1.9 to 1.11 would skip the deprecation warnings shown in 1.10, and fully drop those features in 1.11. It's hard to tell any more with the new semantic versioning that Django is moving to. The docs have been updated to match the new intended SemVer scheme, but are clear as mud as to how older versions were treated.

 

When I access a model instance, AFAIK my context determines the data type I get (string or model instance).

Is this correct?

If this was working under 1.9 (I'd be surprised if it was), then I would file this as a backward-incompatibility bug with the bug tracker. It’s possible that 1.9 did do the type/field conversion using some black magic, but with the information in your model, I don't know how it could have made that determination. 

I have just verified: It still works with 1.9. (-;

If nothing shows in the release notes, I'd recommend a bug report to have the devs take a closer look. 

From a design perspective, the relation you presented seems backwards to me. I would think the Account model would hold the canonical username, and Mailbox would have the FK back to the Account. That design would not require the use of a username field as an FK, simplifying everything (authentication, forms, views, etc.). Your current design would implement excessive SQL JOIN's to pull relatively simple data on every request. 

Or, I could be completely wrong. :-D

Perhaps, I should publish the project at github (it’s OSS anyway), to get more advice?

It might help, but I wouldn't bother with the effort unless you were either already planning to do so, or someone asks specifically. I don't have time to read my code, let alone yours. ;-)

[Axel Rau]

Am 21.08.2017 um 23:53 schrieb James Schneider <jrschn...@gmail.com>:

If this was working under 1.9 (I'd be surprised if it was), then I would file this as a backward-incompatibility bug with the bug tracker.

https://code.djangoproject.com/ticket/28594