authentication by email

459 views
Skip to first unread message

Wim Feijen

unread,
Aug 26, 2011, 11:23:55 AM8/26/11
to Django developers
Hello,

In the past hour, I did some research on authenticating by email and I
believe Django users would benefit a lot if email authentication was
included in contrib.auth .

Many people have been working on it, and the latest code I could find
is here: https://gist.github.com/586056. I am not a very good Googler,
so there may be better patches.

Anyway, there are several problems to solve besides this:
1. the default AuthenticationForm does not accept usernames longer
than 30 characters
2. UserCreationForm and possibly the UserChangeForm need to have Email
counterparts or become more flexible
3. User emails should be unique. My first thought is to add a unique
constraint depending on an optional AUTHENTICATE_BY_EMAIL setting
which defaults to False. I find this problem the hardest to solve.

I am really open to any suggestions, so please do.

Luke Plant, Julien Phalip, I know you have looked into this before and
I am really hoping you can share your thoughts as well.

----
https://gist.github.com/586056:

from django.contrib.auth.backends import ModelBackend
from django.contrib.auth.models import User

class EmailBackend(ModelBackend):

def authenticate(self, **credentials):
if 'username' in credentials:
return super(EmailBackend,
self).authenticate(**credentials)

try:
user = User.objects.get(email=credentials.get('email'))
if user.check_password(credentials.get('password')):
return user
except User.DoesNotExist:
return None

Wim Feijen

unread,
Aug 29, 2011, 6:27:29 AM8/29/11
to Django developers
For the record and as an answer to the previous question, I'd like to
quote Russell who wrote the following in reponse to a ticket of mine:

" The core-endorsed ticket for this problem is #3011. The patch on
that
ticket isn't endorsed, but it points at the real problem - a need to
be
able to specify a custom User model. There have been several
discussions
directed at this problem, and I'm hoping to get a posse discussing
this at
the upcoming DjangoCon sprints. If you want background on some
approaches
that might work (and some that won't) search the archives for Lazy
Foreign
Key. "

Thanks Russell and I wish you good luck in the Django sprints!

- Wim

On Aug 26, 5:23 pm, Wim Feijen <wimfei...@gmail.com> wrote:
> Hello,
>
> In the past hour, I did some research on authenticating by email and I
> believe Django users would benefit a lot if email authentication was
> included in contrib.auth .
>
> Many people have been working on it, and the latest code I could find
> is here:https://gist.github.com/586056. I am not a very good Googler,
> so there may be better patches.
>
> Anyway, there are several problems to solve besides this:
> 1. the default AuthenticationForm does not accept usernames longer
> than 30 characters
> 2. UserCreationForm and possibly the UserChangeForm need to have Email
> counterparts or become more flexible
> 3. User emails should be unique. My first thought is to add a unique
> constraint depending on an optional AUTHENTICATE_BY_EMAIL setting
> which defaults to False. I find this problem the hardest to solve.
>
> I am really open to any suggestions, so please do.
>
> Luke Plant, Julien Phalip, I know you have looked into this before and
> I am really hoping you can share your thoughts as well.
>
> ----https://gist.github.com/586056:

Clay McClure

unread,
Mar 9, 2012, 12:54:16 AM3/9/12
to django-d...@googlegroups.com
"Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design"—unless you want to do something seemingly simple like using email addresses for authentication, in which case you need to monkey patch models and forms to get everything working right, which is neither rapid nor clean. What began as an innocuous feature request five years ago is now a high-level, general purpose, abstract, seemingly insurmountable design problem. The core developers are still perfectionists, but they seem to have forgotten their deadlines.

Is there not a simple, pragmatic solution (optional and for new installations—we're not talking about backwards compatibility here) that could be implemented until the panacea of pluggable User models gets figured out? Something as simple (albeit ugly) as wrapping new models and forms in:

if settings.AUTH_EMAIL_AUTHENTICATION:

Should these things really take five years? What happened to pragmatic?

Clay

Donald Stufft

unread,
Mar 9, 2012, 2:01:29 AM3/9/12
to django-d...@googlegroups.com
The major issue is that there is no way to do schema migrations in core (currently). So there's no way to handle increasing the length of the username field.

More comprehensive solutions require more thought to figure out the pluggable User models.
--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/KebjFDOOBF4J.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

Matt Pegler

unread,
Mar 9, 2012, 2:07:16 AM3/9/12
to django-d...@googlegroups.com
For the project I am working on, we solved this by making a custom
auth backend that checks the username against the email column. We've
found it to be a nice clean solution to wanting to use email addresses
instead of usernames.

Matt Pegler

unread,
Mar 9, 2012, 2:18:43 AM3/9/12
to django-d...@googlegroups.com
Sorry all, disregard my previous email, I misread one thing and
completely missed Wim's original message.

Danny Adair

unread,
Mar 9, 2012, 3:03:50 AM3/9/12
to django-d...@googlegroups.com
On Fri, Mar 9, 2012 at 20:01, Donald Stufft <donald...@gmail.com> wrote:
> The major issue is that there is no way to do schema migrations in core
> (currently). So there's no way to handle increasing the length of the
> username field.

I don't understand what the "username" field length has to do with it.

And I think that's also the problem with Clay's original post:
Do you want to authenticate against email or also change the User model?

What if I want to authenticate against email but still have a username ("nick")?
Another setting?

An auth backend that checks against the email address (which is
already in the User model just like username) _is_ simple.
Rapid and simple would be to _ship_ such a backend in contrib.auth.backends.
Then it would really just be a single setting (AUTHENTICATION_BACKENDS
= ('django.contrib.auth.backends.EmailBackend',)).
The only thing left would be the label on the login template in my
naive understanding. (another one-liner I believe)

So I agree it would be nice and useful, but where's the monkeypatching.
If you want to authenticate against email _and_ take the "username"
out of the model, it's that second part that's the problem. That's
where I believe a set of settings and conditions is a kneejerk
workaround.

Cheers,
Danny

Florian Apolloner

unread,
Mar 9, 2012, 3:13:13 AM3/9/12
to django-d...@googlegroups.com
Hi,

On Friday, March 9, 2012 6:54:16 AM UTC+1, Clay McClure wrote:
if settings.AUTH_EMAIL_AUTHENTICATION:

Hell, not another ugly setting like this.

Should these things really take five years? What happened to pragmatic?

Yes, since no one needs it. Okay no one isn't true, but no one (for true this time) who needed it stepped up and said "I'll implement it and see that it ends up in trunk"


Regards,
Florian

Danny Adair

unread,
Mar 9, 2012, 3:23:12 AM3/9/12
to django-d...@googlegroups.com
On Fri, Mar 9, 2012 at 21:13, Florian Apolloner <f.apo...@gmail.com> wrote:
>[...]

> Yes, since no one needs it. Okay no one isn't true, but no one (for true
> this time) who needed it stepped up and said "I'll implement it and see that
> it ends up in trunk"

It's the "required" of username that's the problem if you don't want a
username at all when authenticating against email.
It would have to be not required and check required fields in clean()
where the backend could be asked what's really required.
And there's Mr. Schema Migration again...

Cheers,
Danny

Tino de Bruijn

unread,
Mar 9, 2012, 7:03:29 AM3/9/12
to django-d...@googlegroups.com
My django-email-login app (https://bitbucket.org/tino/django-email-login/overview) does this by putting a hash of the email adress in the username field. It isn't as nice as it could be, but it works.

I would really like to see this solved another way, but it is a hard problem with the current restrictions.

Tino

--
You received this message because you are subscribed to the Google Groups "Django developers" group.

Clay McClure

unread,
Mar 9, 2012, 4:37:21 AM3/9/12
to django-d...@googlegroups.com
On Fri, Mar 9, 2012 at 3:23 AM, Danny Adair <danny...@unfold.co.nz> wrote:
 
It's the "required" of username that's the problem if you don't want a
username at all when authenticating against email.
It would have to be not required and check required fields in clean()
where the backend could be asked what's really required.

That's one problem. Another problem is that the default User.email field is not unique, is not required, is not indexed, and may be too short.

And there's Mr. Schema Migration again...

Who's talking about a migration? I'm asking for something that will work for *new* installations; existing installations can continue authenticating against usernames  for all I care :)

Moreover, I'm thoroughly frustrated by the fact that developers *do* bring working solutions to the table (in the form of patches in trac), but the core developers won't integrate them, either because it's not the right time (usually right before a release), because the patch isn't general enough, or because the patch doesn't meet some absurdly high bar of quality. I understand that they have a commitment to backwards compatibility and that accepting a mediocre patch could mean maintaining it for life, but I like to think that—with a framework that purports to be "pragmatic"—there are pragmatic solutions to these problems—and not, "sorry, it's not perfect, it can't go in."

The GSoC page (https://code.djangoproject.com/wiki/SummerOfCode2011#Enhancedauth.user) is a frustrating read. It goes on and on about how hard the problem is and how wrong your solution is, but doesn't provide any detail as to why it's hard or why it's wrong. Ticket #3011 was rejected without much of a reason. HM asked for an explanation both in the ticket itself and on django-dev; no explanation was ever given.

The GSoC page also mentions migrations:

"How can we roll out a new/modified User model without requiring almost every Django application on the planet to undergo a complex database modification?"

But again, why do existing databases have to change? We're talking about leaving User as-is, by default, but providing a mechanism to use a different model if the developer chooses. Clearly this is a decision the developer would not take lightly: you're not going to change from username authentication to email authentication without a bit of thought. Projects that are using username authentication will continue to use username authentication, but at least new projects that want/need email authentication will be able to do that without monkey patching models.

The GSoC page also mentions Lazy Foreign Keys, but no explanation is given there or in the linked django-dev thread as to why LFKs are required to implement a pluggable User model. LFK may be the panacea of code cleanliness and virtual awesomeness, but those of us with deadlines just need to authenticate against email addresses, like, five years ago.

Cheers,

Clay

Tom Evans

unread,
Mar 9, 2012, 9:49:42 AM3/9/12
to django-d...@googlegroups.com
On Fri, Mar 9, 2012 at 8:13 AM, Florian Apolloner <f.apo...@gmail.com> wrote:
>> Should these things really take five years? What happened to pragmatic?
>
>
> Yes, since no one needs it. Okay no one isn't true, but no one (for true
> this time) who needed it stepped up and said "I'll implement it and see that
> it ends up in trunk"
>

I'm sorry, that completely mis-characterises the situation. Lots of
people want this, but every single time this has been brought up since
I started using django (1.0), a core dev or BFDL has explicitly ruled
it out, saying "we cannot do this, requires schema migration, we'll
look at it for the next version".

At this point, the discussion is stopped. Once a proposal has the
smack down from a BFDL, what point is there trying to get it
integrated?

The next version comes, has it been addressed? No. Will it be
addressed in 1.4? No. Will it be addressed in 1.5? Possibly, but I
doubt it.

At $JOB we've given up waiting for some trivial changes, and have
simply patched django to not have these silly, arbitrary and incorrect
requirements¹ on email address and username.

We live in the real world, we have to be able to handle email
addresses that are of a valid length. We cannot tell a paying customer
"I'm sorry, your email address is too long, do you have one under 75
characters?"

It's a really invalid position to take, knowing something is broken
and incorrect, but not fixing it because 'change is hard'.

Cheers

Tom

¹ Here is my list of things that are broken with d.c.auth:

1) Length and character restrictions on username field. If I want to
use emails address as usernames, or usernames with spaces, the choices
made at lawrence.com 3+ years ago should not restrict me.
2) Email address fields must be large enough to hold valid email
addresses - that's 254 characters.

Luke Plant

unread,
Mar 9, 2012, 10:36:02 AM3/9/12
to django-d...@googlegroups.com
On 09/03/12 09:37, Clay McClure wrote:

> Who's talking about a migration? I'm asking for something that will work
> for *new* installations; existing installations can continue
> authenticating against usernames for all I care :)
>
> Moreover, I'm thoroughly frustrated by the fact that developers *do*
> bring working solutions to the table (in the form of patches in trac),
> but the core developers won't integrate them, either because it's not
> the right time (usually right before a release), because the patch isn't
> general enough, or because the patch doesn't meet some absurdly high bar
> of quality. I understand that they have a commitment to backwards
> compatibility and that accepting a mediocre patch could mean maintaining
> it for life, but I like to think that—with a framework that purports to
> be "pragmatic"—there are pragmatic solutions to these problems—and not,
> "sorry, it's not perfect, it can't go in."

What you are really saying is this: being pragmatic means that we
prioritise *your* immediate need above the need to keep the code and the
docs maintainable, and above the need to maintain compatibility with
existing installations.

There are a million-and-one hacks we would add if we took that approach,
and Django would have a million-and-one more bugs, or possibly much
worse - if you add just 10 boolean switches like the one you suggested
in an earlier email, you've got over 1000 combinations to test and
debug. The only way to avoid that in a framework like Django is well
defined APIs for components. No-one has brought such a thing to the
table and seen it through, that's why it hasn't come in.

I believe that all the problems you have can be solved already:

- you can write your own authentication backend to authenticate
by email

- you can write your own login and account creation views

- you can even override admin views by inserting items into
urls.py


Yes, these are all more-or-less hacky. And they have problems. For
instance, what if you have two accounts with the same email address? (I
have that in some of my Django projects, BTW). The current code allows
it, so we can't change anything that would make it impossible, at either
the DB level or the Python level. This is the kind of issue that means
you can't just use the existing models.

A full and correct solution that didn't introduce dozens of new bugs is
quite hard - and I'm talking about just the code required to fix this
one feature for you, without doing anything more generally useful.

That is why we're not going to put hacks like this in core - we would
have to support these hacks for at least 3 versions if we did. We are
interested in *removing* existing hacks from our code base, and this is
vital if Django is going to see increases in features and functionality.
We are not interested in adding more hacks.

We also will refuse to litter our documentation with things like: "if
you've got this setting, it does this, otherwise you get this other
behaviour, and you've got to watch out for all these bugs because this
is a gross hack".

> The GSoC page
> (https://code.djangoproject.com/wiki/SummerOfCode2011#Enhancedauth.user)
> is a frustrating read. It goes on and on about how hard the problem is
> and how wrong your solution is, but doesn't provide any detail as to why
> it's hard or why it's wrong.

It does provide detail - it gives a list of issues you have to consider,
and tells you where to search for the other issues.

Some examples from a quick Google groups search for "auth User":

http://goo.gl/swTpr

http://goo.gl/fFlKh

These provide a lot of detail.

> Ticket #3011 was rejected without much of a
> reason. HM asked for an explanation both in the ticket itself and on
> django-dev; no explanation was ever given.

I don't know what thread you are talking about. Hanne Moa brings up the
subject here and gets two replies from Russell:

http://goo.gl/7p1JN

Regards,

Luke

--
"I imagine bugs and girls have a dim suspicion that nature played a
cruel trick on them, but they lack the intelligence to really
comprehend the magnitude of it." (Calvin and Hobbes)

Luke Plant || http://lukeplant.me.uk/

Tom Evans

unread,
Mar 9, 2012, 10:52:40 AM3/9/12
to django-d...@googlegroups.com
On Fri, Mar 9, 2012 at 3:36 PM, Luke Plant <L.Pla...@cantab.net> wrote:
> What you are really saying is this: being pragmatic means that we
> prioritise *your* immediate need above the need to keep the code and the
> docs maintainable, and above the need to maintain compatibility with
> existing installations.
>
> There are a million-and-one hacks we would add if we took that approach,
> and Django would have a million-and-one more bugs, or possibly much
> worse - if you add just 10 boolean switches like the one you suggested
> in an earlier email, you've got over 1000 combinations to test and
> debug. The only way to avoid that in a framework like Django is well
> defined APIs for components. No-one has brought such a thing to the
> table and seen it through, that's why it hasn't come in.
>
> I believe that all the problems you have can be solved already:
>
>  - you can write your own authentication backend to authenticate
>   by email
>
>  - you can write your own login and account creation views
>
>  - you can even override admin views by inserting items into
>   urls.py
>

Even with these things, you are limited by the length of the username
field, the length of the email field, and the arbitrary restrictions
on what consists a username.

The length restrictions could be fixed straight away, and a note added
to the next relnotes informing users that the field has changed in
size. If they do not update their DB structure, then depending on the
database used, they will either get errors when a field value is
longer than the old limit, or silent truncation.

It's not ideal, but it doesn't cause any major problems for anyone who
reads the relnotes, and it will have to be done at some point anyway.
Even if Django did have schema migrations, the same behaviour would
occur for anyone who read the relnotes and did not apply the
migration, and many users with 'interesting' DB structures would not
be able to simply apply schema migrations either, so they would have
to do it by hand anyway.

It feels to me that only fear of change and how to manage that change
is what is stopping d.c.auth from being fixed.

>
> Yes, these are all more-or-less hacky. And they have problems. For
> instance, what if you have two accounts with the same email address? (I
> have that in some of my Django projects, BTW). The current code allows
> it, so we can't change anything that would make it impossible, at either
> the DB level or the Python level. This is the kind of issue that means
> you can't just use the existing models.
>

Good for you! I don't see that as an issue. If you are using email
address as a unique name for identifying a user, we already have a
column for that, it's called username, and it has some of the right
semantics (unique, indexed field). It is the arbitrary length and
content restrictions that stop this from happening right now.

I don't see why wanting to use email address as the unique identifier
and username for a user requires any modification to the database,
apart from (as I keep repeating) removing the arbitrary length and
content restrictions on username.

> A full and correct solution that didn't introduce dozens of new bugs is
> quite hard - and I'm talking about just the code required to fix this
> one feature for you, without doing anything more generally useful.
>
> That is why we're not going to put hacks like this in core - we would
> have to support these hacks for at least 3 versions if we did. We are
> interested in *removing* existing hacks from our code base, and this is
> vital if Django is going to see increases in features and functionality.
> We are not interested in adding more hacks.
>
> We also will refuse to litter our documentation with things like: "if
> you've got this setting, it does this, otherwise you get this other
> behaviour, and you've got to watch out for all these bugs because this
> is a gross hack".
>

I agree with this 100%. d.c.auth needs to be fixed, not have many
different settings that alter it's behaviour.

Cheers

Tom

Luke Plant

unread,
Mar 9, 2012, 10:52:47 AM3/9/12
to django-d...@googlegroups.com
On 09/03/12 14:49, Tom Evans wrote:

>> Yes, since no one needs it. Okay no one isn't true, but no one (for true
>> this time) who needed it stepped up and said "I'll implement it and see that
>> it ends up in trunk"
>>
>
> I'm sorry, that completely mis-characterises the situation. Lots of
> people want this, but every single time this has been brought up since
> I started using django (1.0), a core dev or BFDL has explicitly ruled
> it out, saying "we cannot do this, requires schema migration, we'll
> look at it for the next version".

This is not true. There have been times when we have said we cannot
consider it right now, and that a solid proposal should be brought up at
the calls for proposals for 1.1/1.2/1.3 etc., or at other times when the
core developers can look at it.

And every time, the proposal has either been missing or simply not been
adequate - for example, the GSoC 2010 and 2011 proposals regarding this.
Russell gave detailed feedback on why these solutions were not adequate
[1], and at other times has provided feedback on possible strategies
[2]. And the people who went away to work on the problem have gone
silent or admitted defeat. One presumes they fixed their immediate
problem in a way that would not be a general solution, and moved on, and
that is perfectly understandable. A adequate solution to this is very
hard, and probably requires solving several other big problems first
(like at least one of lazy foreign keys/virtual models/database migrations).

It isn't, however, impossible. Our definition of 'need' is that someone
is sufficiently motivated to overcome the obstacles, and contribute a
solution that works for other people as well as themselves.

Regards,

Luke


[1] http://goo.gl/swTpr

[2] http://goo.gl/7p1JN

--

Luke Plant || http://lukeplant.me.uk/

Tom Evans

unread,
Mar 9, 2012, 11:46:49 AM3/9/12
to django-d...@googlegroups.com

I disagree. There are small problems with the User model that have not
been fixed. Every proposal you've listed does not address these
issues, but instead looks to find ways of replacing the base user
object, which is a complex and difficult project.

There is a reason why these are the proposals. Every time someone
proposes fixing these minor bugs, the cry from on high is that
instead, d.c.auth will be overhauled to allow for more customization
by the project, immediately increasing the scope and complexity. But
this fabled d.c.auth2 never appears.

In fact, your reply alludes to this. We're talking about why a field
is too short to contain valid values, and you suggest that the
solution inevitably will involve "lazy foreign keys/virtual
models/database migrations" - it's changing one line, and adding a
comment to relnotes!

We may have to agree to disagree on how easy some of these things are to fix.

Lets look at one isolated aspect. The User email field in d.c.auth is
too short. Emails can be up to 248 characters long, and d.c.auth only
allows 75.

Clearly, this is a bug, is wrong and should be fixed. How can we fix it?

We can alter the length of the field in the model definition. What
problems does this bring?

Users who do not manually update their database will not be able to
store longer length emails, and will get 'strange effects' when they
try - either silent truncation, or database integrity errors. How can
we mitigate this for users who refuse to update?

Provide a compat shim that resets the email length to that in 1.3, and
immediately mark it PendingDeprecation. Users who read the relnotes
and don't want to change their DB can use the shim, and have a clear
expectation of when they must have updated their database.

I still think these bug fixes are paralysed by a fear of change, and a
desire to engineer a solution that is every thing to every man. Lets
just fix these glaring bugs…

Cheers

Tom

Joe & Anne Tennies

unread,
Mar 9, 2012, 12:52:57 PM3/9/12
to django-d...@googlegroups.com
While, I generally agree with the current approach, especially this close to release. I'm going to play devil's advocate for a bit.

Schema migrations have been talked about for quite a while. There are at least 3 external implementations I know of: South, nashvegas, and django-evolution. I'm unsure of the status of django-evolution, but the other two appear quite active.

A concern in my mind is that all three go off and do things totally different ways, and the migration path to a common standard is more difficult.

I love the idea of not choosing a tool to make the migrations, but instead add to the ability to execute migrations. Now comes the conversation of what that means. I would love to try to get this in to Django 1.5.

Now on to the meat of what I really want:
Is there something along the lines of the GNOME design group for Django? (example: https://live.gnome.org/Design/Apps/Web). I'd love a template for the wiki or even better an app to discuss these things. If only we had some web developers ;) So, is there a sanctioned tool already out there? Or should I start building up a page on the wiki for this? I think most things become much easier once one gets a blueprint and requirements for what the design goals are. Especially if one can get an initial "this looks acceptable" by a core committer before "wasting one's time" because the requirements weren't even correct.

Does github or bitbucket have such a feature I don't know about? Does launchpad.net blueprints meet this? Is there some other place I don't know about that someone could point me to?

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.




--
Joe & Anne Tennies
ten...@gmail.com

Łukasz Rekucki

unread,
Mar 9, 2012, 2:14:23 PM3/9/12
to django-d...@googlegroups.com
On 9 March 2012 17:46, Tom Evans <teva...@googlemail.com> wrote:
>
> Lets look at one isolated aspect. The User email field in d.c.auth is
> too short. Emails can be up to 248 characters long, and d.c.auth only
> allows 75.

The latest RFC[1] actually specifies this as 256 *octets* with max of
64 octets for the local part and 255 octets for the domain name. So
248 *characters* would actually be incorrect and all the tedious and
error prone fixing of every Django instance would just get wasted.

I don't really think this is a "fear of change" case. It's more of "we
don't want to have to fix this again" thing.

[1]: http://tools.ietf.org/html/rfc5321#section-4.5.3.1.1

--
Łukasz Rekucki

Tom Evans

unread,
Mar 9, 2012, 3:10:33 PM3/9/12
to django-d...@googlegroups.com
2012/3/9 Łukasz Rekucki <lrek...@gmail.com>:

> On 9 March 2012 17:46, Tom Evans <teva...@googlemail.com> wrote:
>>
>> Lets look at one isolated aspect. The User email field in d.c.auth is
>> too short. Emails can be up to 248 characters long, and d.c.auth only
>> allows 75.
>
> The latest RFC[1] actually specifies this as 256 *octets* with max of
> 64 octets for the local part and 255 octets for the domain name. So
> 248 *characters* would actually be incorrect and all the tedious and
> error prone fixing of every Django instance would just get wasted.
>

Sorry, 248 was a typo. If you look at my earlier reply in this thread,
I had correctly stated the maximum length of an email address as 254
*characters*.

If you check two paragraphs later in the RFC that you linked to, you
would see confirmation of this:

http://tools.ietf.org/html/rfc5321#section-4.5.3.1.3

You may also need to remind yourself what the definition of a path is.

As all email headers are either 7 or 8 bit encodings, describing 254
octets as 254 characters is perfectly valid.

None of this changes that django keeps shipping with bad defaults.
This was brought up for 1.1, 1.2, 1.3 and now 1.4.

Cheers

Tom

Łukasz Rekucki

unread,
Mar 9, 2012, 4:12:37 PM3/9/12
to django-d...@googlegroups.com
On 9 March 2012 21:10, Tom Evans <teva...@googlemail.com> wrote:
> 2012/3/9 Łukasz Rekucki <lrek...@gmail.com>:
>> On 9 March 2012 17:46, Tom Evans <teva...@googlemail.com> wrote:
>>>
>>> Lets look at one isolated aspect. The User email field in d.c.auth is
>>> too short. Emails can be up to 248 characters long, and d.c.auth only
>>> allows 75.
>>
>> The latest RFC[1] actually specifies this as 256 *octets* with max of
>> 64 octets for the local part and 255 octets for the domain name. So
>> 248 *characters* would actually be incorrect and all the tedious and
>> error prone fixing of every Django instance would just get wasted.
>>
>
> Sorry, 248 was a typo. If you look at my earlier reply in this thread,
> I had correctly stated the maximum length of an email address as 254
> *characters*.
>
> If you check two paragraphs later in the RFC that you linked to, you
> would see confirmation of this:
>
> http://tools.ietf.org/html/rfc5321#section-4.5.3.1.3
>
> You may also need to remind yourself what the definition of a path is.

And RFC 3696 originally claimed it's 320. Even after errata it still
says 256, but you are right that 254 is probably more correct.

>
> As all email headers are either 7 or 8 bit encodings, describing 254
> octets as 254 characters is perfectly valid.

UTF-8 is an 8-bit encoding that doesn't map octets to characters, so
no. But that's not really the point I wanted to make.

My point is that this kind of things change and we should have tools
to deal with that. We already have the exact same problem with IPv6
and contrib.comments, it's just people didn't noticed it yet as much.
Doing this by hand every time isn't very effective. Django needs
schema migrations.

As for the "design page" that Joe mentioned, the GSOC description is a
good start:

https://code.djangoproject.com/wiki/SummerOfCode2012#Enhancedauth.user

--
Łukasz Rekucki

Luciano Pacheco

unread,
Mar 9, 2012, 7:25:29 PM3/9/12
to django-d...@googlegroups.com

2012/3/10 Łukasz Rekucki <lrek...@gmail.com>

On 9 March 2012 21:10, Tom Evans <teva...@googlemail.com> wrote:
> 2012/3/9 Łukasz Rekucki <lrek...@gmail.com>:
>> On 9 March 2012 17:46, Tom Evans <teva...@googlemail.com> wrote:
[...] 
My point is that this kind of things change and we should have tools
to deal with that. We already have the exact same problem with IPv6
and contrib.comments, it's just people didn't noticed it yet as much.
Doing this by hand every time isn't very effective. Django needs
schema migrations.
[...]

The tool for migration can be simple as docs/release note or release warning :-) ?

We change the database schema and provide the SQL scripts to migrate the 4 supported backends (sqlite, mysql, postgres and oracle). This script are needed "only" for upgrade projects, new projects it's just a normal path.

Is that hard, inconvenient or bad?

In my humble opinion it's simple and pragmatic.

[],
--
Luciano Pacheco
blog.lucmult.com.br

Russell Keith-Magee

unread,
Mar 10, 2012, 4:16:31 AM3/10/12
to django-d...@googlegroups.com

On 10/03/2012, at 1:52 AM, Joe & Anne Tennies wrote:

> While, I generally agree with the current approach, especially this close to release. I'm going to play devil's advocate for a bit.
>
> Schema migrations have been talked about for quite a while. There are at least 3 external implementations I know of: South, nashvegas, and django-evolution. I'm unsure of the status of django-evolution, but the other two appear quite active.

Django Evolution isn't dead. It's sleeping. It's got lovely plumage, you know... :-)

Seriously -- I stopped working on Evolution several years ago. I formally handed over the project to Christian Hammond almost 2 years ago, but this has been mostly for maintenance purposes, rather than active development. For the benefit of posterity, I blogged about the reasons [1].

[1] http://cecinestpasun.com/entries/end-my-evolution/

> A concern in my mind is that all three go off and do things totally different ways, and the migration path to a common standard is more difficult.
>
> I love the idea of not choosing a tool to make the migrations, but instead add to the ability to execute migrations. Now comes the conversation of what that means. I would love to try to get this in to Django 1.5.

A lot of this conversation has already happened, and it has core team approval. Most of these discussions were elaborated in last year's aborted GSoC 2011 project; search the django-dev archives from this time last year to see the details of what was being proposed.

The short version:

* Add a db.backends module to provide an abstract interface to migration primitives (add column, add index, rename column, rename table, and so on).

* Add a contrib app that performs the high level accounting of "has migration X been applied", and management commands to "apply all outstanding migrations"

* Provide an API that allows end users to define raw-SQL migrations, or native Python migrations using the backend primitives.

* Leave the hard task of determining dependencies, introspection of database models and so on to the toolset contributed by the broader community.

Under this approach, South would still need to exist, but it would be a tool leveraging the primitives exposed by Django itself. Over time, more features from South (and any other migration support projects that emerge) could be merged into trunk as the community converged on a feature set or an implementation.

One of the big issues that needs to be addressed is testing -- we need to have the infrastructure that allows us to check that migrations have been applied. Arthur Koziel's AppRefactor (a GSoC 2010 project) has some analogous testing problems, so getting migrations into trunk may be dependent on getting the App Refactor into trunk as well. The good news is that the App Refactor also forms the likely stub for fixing the auth.User problem -- but that's a separate discussion.

> Now on to the meat of what I really want:
> Is there something along the lines of the GNOME design group for Django? (example: https://live.gnome.org/Design/Apps/Web). I'd love a template for the wiki or even better an app to discuss these things. If only we had some web developers ;) So, is there a sanctioned tool already out there? Or should I start building up a page on the wiki for this? I think most things become much easier once one gets a blueprint and requirements for what the design goals are. Especially if one can get an initial "this looks acceptable" by a core committer before "wasting one's time" because the requirements weren't even correct.

There are some historical examples where we've used the wiki to elaborate on an idea as a precursor to getting a feature into core.

https://code.djangoproject.com/wiki/SessionMessages
https://code.djangoproject.com/wiki/ClassBasedViews
https://code.djangoproject.com/wiki/LoggingProposal
https://code.djangoproject.com/wiki/ReplacingGetAbsoluteUrl

(The first two are probably the best examples to follow)

There is a wiki page which has been used in the past to discuss schema evolution (which it looks like you've found); however, it's wandered a little off the reservation. There's probably some good material in there still, but if someone is looking at this, it would benefit from some manicuring, and probably a complete reboot:

https://code.djangoproject.com/wiki/SchemaEvolution

There isn't a specific wiki template that we require -- this is mostly because every project has slightly different requirements. However, the broad format is fairly common:

* describe the problem
* describe any specific design constraints
* describe the possible approaches and their benefits/limitations
* list any prior art
* list any draft implementations

If you want to work on this, I can guarantee that you would have as much support as I and Andrew Godwin (South maintainer and Django core developer) can muster. This is one of the big ticket items missing from Django's core, and I'd *love* to see someone take a serious swing at addressing it. As with every big feature that is "missing" from Django (be it migrations, auth.User problems, or anything else), the issue is almost always finding someone who is willing to do the work -- not just for a day or two, but to actively drive a project for several months.

Yours,
Russ Magee %-)

Clay McClure

unread,
Mar 11, 2012, 4:39:55 AM3/11/12
to django-d...@googlegroups.com
On Fri, Mar 9, 2012 at 10:36 AM, Luke Plant <L.Pla...@cantab.net> wrote:

What you are really saying is this: being pragmatic means that we
prioritise *your* immediate need above the need to keep the code and the
docs maintainable, and above the need to maintain compatibility with
existing installations.

Of course not. Firstly, this is not *my* need; it's a feature that many have requested. Take a survey of the web applications that you use on a weekly basis. How many of them allow you to sign in with an email address? Compared to that, how many required you to invent a username? Of those, I would wager that a good many are Django apps.

Secondly, I'm advocating for a backwards-compatible solution that won't break existing installations. That point keeps getting missed, as you and others keep assuming that we need schema migrations for this change—we don't. People who are happy with the existing user model can keep using it, as-is. But people who aren't happy with it should be able to provide their own.
 
There are a million-and-one hacks we would add if we took that approach,
and Django would have a million-and-one more bugs, or possibly much
worse - if you add just 10 boolean switches like the one you suggested
in an earlier email, you've got over 1000 combinations to test and
debug.

I was giving an example of a pragmatic solution: sometimes an `if` statement is better than a far-reaching architectural change. I don't advocate boolean flags for solving the auth.User problem, I'm merely trying to encourage a shift in thinking from lofty aspiration to humble accomplishment.
 
I believe that all the problems you have can be solved already:

 - you can write your own authentication backend to authenticate
  by email

True.
 
 - you can write your own login and account creation views

True.
 
 - you can even override admin views by inserting items into
  urls.py

True.

But these things aren't the problem. It's the auth.User model that poses the challenge. If you haven't implemented email-based authentication in your own apps, let me suggest that you try it before you tell me how easy it is. At some point, you will run into these limitations posed by the User model:

 - The username field can't be used to store emails (it's too short and restrictive). No problem, let's use the email field;
 - But the username field is required. OK, we can just fill it with a random string;
 - The email field does not have a unique constraint. But we can do a reasonably good job applying the constraint in python;
 - The email field is not indexed. We don't want full table scans whenever someone logs in, so let's write a custom (and possibly non-portable) SQL migration to add an index.

All this, and the email field is still too short to hold all valid email addresses. On top of that, we've littered our database and admin interface with random strings for usernames, and littered our application with the code to generate them.

Yes, these are all more-or-less hacky. And they have problems. For
instance, what if you have two accounts with the same email address? (I
have that in some of my Django projects, BTW). The current code allows
it, so we can't change anything that would make it impossible, at either
the DB level or the Python level. This is the kind of issue that means
you can't just use the existing models.

Of course, which is why we need to allow developers to supply their own User model, while maintaining backwards compatibility for those who are happy with the built-in model.

A full and correct solution that didn't introduce dozens of new bugs is
quite hard - and I'm talking about just the code required to fix this
one feature for you, without doing anything more generally useful.

Again, it's not just for me. And, yes, I appreciate that it's not a trivial problem. It's hard, but all the problems worth solving are.

That is why we're not going to put hacks like this in core - we would
have to support these hacks for at least 3 versions if we did. We are
interested in *removing* existing hacks from our code base, and this is
vital if Django is going to see increases in features and functionality.
We are not interested in adding more hacks.

I can appreciate the appeal of a pristine, hack-free code base, but I've never seen one. Eventually reality always trumps grand design. If my example of the `if` statement above has led you to believe that I'm proposing a mindless hack, then I apologize. I'm proposing something between a mindless hack and grand design. Something pragmatic.

Some examples from a quick Google groups search for "auth User":

http://goo.gl/swTpr

Some hand waving ("It's deceptively difficult"), with no examples, and no mention of #3011.

http://goo.gl/fFlKh

This is someone with mediocre design skills trying to get paid to hack on Django over the summer, but failing to convince anyone that he has a clue about how to solve this problem. Save for some humorous jibes from Russell, the thread is almost devoid of good content, except for Russell's example of the COMMENTS_APP setting, which sounds a lot like "the idea of a 'pluggable' User model"a proposal that the GSoC page indicates has already been rejected, but without explanation.

> Ticket #3011 was rejected without much of a
> reason. HM asked for an explanation both in the ticket itself and on
> django-dev; no explanation was ever given.

I don't know what thread you are talking about. Hanne Moa brings up the
subject here and gets two replies from Russell:

http://goo.gl/7p1JN

I was thinking of this thread:


wherein Carl Meyer (I confused CM with HM) enquires about #3011 and receives no reply. In the thread you mention, there isn't any detail about #3011, other than that the patch it contains is "*completely* the wrong approach."

> The GSoC page
> (https://code.djangoproject.com/wiki/SummerOfCode2011#Enhancedauth.user)
> is a frustrating read. It goes on and on about how hard the problem is
> and how wrong your solution is, but doesn't provide any detail as to why
> it's hard or why it's wrong.

It does provide detail - it gives a list of issues you have to consider,
and tells you where to search for the other issues.

It poses a list of questions (how can we represent a generic user? how should we approach authentication vs. authorization? how do we handle migrations?) that all presuppose a grand redesign of auth.User instead of an incremental improvement, but it doesn't discuss at all why an incremental improvement won't be considered. And then it refers would-be implementers to read up on Lazy Foreign Keys (LFKs), which I contend are neither sufficient nor necessary for an auth.User refactor.

To illustrate this last point, consider some of the compatibility problems we're liable to face when replacing django.contrib.auth.models.User:

1. Parts of Django, numerous third-party apps, and countless bespoke software installations have models with FKs to User. LFKs could help here, in that we could delay binding to the precise model until that binding is known (through configuration, or whatever). But if, to use an LFK, any of the third-party apps or bespoke software has to be changed (say, from `ForeignKey(User)` to `LazyForeignKey('User')` or `ForeignKey(LazyModel('User'))`), then LFKs are not backwards compatible. They may be the place we want to end up, but they're not the place to start, as they require too much change in software we don't control.

2. Parts of Django, numerous third-party apps, and countless bespoke software installations have ModelForms built around the User model. The model is specified in the ModelForm's Meta class, and must be available at class definition time. LFKs don't help here. We could extend ModelForm to support delayed binding, but if that involves changing any existing software (say, `model = LazyModel('User')`), then, again, we've broken backwards compatibility.

3. Parts of Django, numerous third-party apps, and countless bespoke software installations import User from django.contrib.auth.models. But we're talking about allowing developers to specify their own User model, so code that imports d.c.a.m.User would no longer be valid. LFKs don't help here, either.

What's needed is a way to maintain the interface defined by the existing django.contrib.auth application (a User model in django.contrib.auth.models, an AuthenticationForm in django.contrib.auth.forms, a ModelBackend in django.contrib.auth.backends, etc) while allowing the developer to specify the implementation—whether the stock User app, a reusable third-party app, or their own.

I've written such a pluggable User app. By default, the User model works as before. But I've also provided a User model (and forms, admin, backends, etc) implementation that removes the limitations of the User model noted above and allows for email authentication. It's still a bit rough around the edges, but it works, insofar that `manage.py createsuperuser` doesn't ask for a username, `manage.py changepassword` takes an email address, and I can login to the admin interface with my email address.

It works for me, and that's well and good since I have a client that needs this functionality, but I don't want to maintain the fork forever. I would like to provide a patch for consideration for trunk, but before I waste my time and yours, I'd like to understand why the notion of pluggable user models was rejected (as indicated on the GSoC page), when Russell seems to be advocating for a design of that sort with his reference to the COMMENTS_APP setting.

And to be clear, I don't consider the pluggable user model approach that I'm taking and the Lazy Foreign Key approach to be mutually exclusive. Rather, I see my approach as something of a compatibility shim until sufficient time has elapsed to pull the plug on imports of django.contrib.auth.models.User. I propose that we continue to support the existing auth interface (defined more completely above, but for brevity, a User model at django.contrib.auth.models) but start raising a deprecation warning in the 1.4.1+ timeframe. Meanwhile, we introduce LFKs (which I have ideas about, but which, again, are not required for an incremental improvement to auth.User), so that maintainers of reusable apps can start to take advantage of delayed binding. At some point, say, 1.5, or 2.0, we drop support for the existing auth interface (which is to say, apps should not expect to find a User model at django.contrib.auth.models) and the LFK becomes the "one—and preferably only one—obvious way to do it."

Cheers,

Clay

Jacob Kaplan-Moss

unread,
Mar 11, 2012, 12:09:45 PM3/11/12
to django-d...@googlegroups.com
On Sunday, March 11, 2012 at 12:39 AM, Clay McClure wrote:
I've written such a pluggable User app. By default, the User model works as before. But I've also provided a User model (and forms, admin, backends, etc) implementation that removes the limitations of the User model noted above and allows for email authentication. It's still a bit rough around the edges, but it works, insofar that `manage.py createsuperuser` doesn't ask for a username, `manage.py changepassword` takes an email address, and I can login to the admin interface with my email address.
Can you please post this code somewhere? I'd love to see it.

Jacob

Joe & Anne Tennies

unread,
Mar 11, 2012, 6:58:13 PM3/11/12
to django-d...@googlegroups.com
I started a new page off the old https://code.djangoproject.com/wiki/SchemaEvolution (https://code.djangoproject.com/wiki/SchemaEvolutionDesign). It's not complete at this point, I just did a brain dump of what I remembered hearing in the past. Sounds like I'm not TOO far off with my memory.

Also, is the following the thread you are thinking of? http://groups.google.com/group/django-developers/browse_thread/thread/6239a63c9ad874b5/

Thanks in advance,
Joe Tennies

Also, Russell, I didn't know you were the one behind django-evolution. I may need to pick your brain at some point. I have a couple ideas I'd like input on, but I'm not quite ready yet. Now I wish I had tried harder to get to PyCon this year, but work was supposed to be quite hectic this week.


My goal is to

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

Joe & Anne Tennies

unread,
Mar 11, 2012, 7:08:05 PM3/11/12
to django-d...@googlegroups.com
Can I ask for one change this late in the project to the 1.4 release if it's only the documentation for this bug? I'm hoping someone at PyCon might be able to "sneak this in."

Can we add a note to the EmailField documentation that states that "The default 75 character max_length is not capable of storing all RFC3696/5321 email addresses. A max_length of 254 would be capable of storing all email addresses as defined by the RFC3696/5321 specification. This is not being changed at this time to maintain backwards compatibility." I'm thinking the equivalent of <https://docs.djangoproject.com/en/1.3/ref/models/fields/> would be a good place.

I know we aren't going to change any code for this release, as there is still discussion on this, but I thought this would be a good start.

Thanks,
Joe Tennies

Russell Keith-Magee

unread,
Mar 11, 2012, 7:28:39 PM3/11/12
to django-d...@googlegroups.com

Sure - that sounds like a reasonable addition to me. Open a documentation ticket, and we'll see if we can slip it in.

Yours,
Russ Magee %-)

Russell Keith-Magee

unread,
Mar 11, 2012, 10:32:08 PM3/11/12
to django-d...@googlegroups.com

On 12/03/2012, at 6:58 AM, Joe & Anne Tennies wrote:

> I started a new page off the old https://code.djangoproject.com/wiki/SchemaEvolution (https://code.djangoproject.com/wiki/SchemaEvolutionDesign). It's not complete at this point, I just did a brain dump of what I remembered hearing in the past. Sounds like I'm not TOO far off with my memory.
>
> Also, is the following the thread you are thinking of? http://groups.google.com/group/django-developers/browse_thread/thread/6239a63c9ad874b5/

That's the GSoC discussion I'm thinking of; it covers the db.backends part of the discussion. There is another post by Andrew from a little earlier that also includes the 'migration accounting' part of the problem:

http://groups.google.com/group/django-developers/browse_thread/thread/bac15726fa5e9662

> Also, Russell, I didn't know you were the one behind django-evolution. I may need to pick your brain at some point. I have a couple ideas I'd like input on, but I'm not quite ready yet. Now I wish I had tried harder to get to PyCon this year, but work was supposed to be quite hectic this week.

Unfortunately, I couldn't make it to PyCon this year. However, I'm more than happy to have my brain picked by someone who is volunteering to pitch in on schema migration.

Yours,
Russ Magee %-)

Clay McClure

unread,
Mar 11, 2012, 11:57:32 PM3/11/12
to django-d...@googlegroups.com
On Sun, Mar 11, 2012 at 12:09 PM, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:
I've written such a pluggable User app. By default, the User model works as before. But I've also provided a User model (and forms, admin, backends, etc) implementation that removes the limitations of the User model noted above and allows for email authentication. It's still a bit rough around the edges, but it works, insofar that `manage.py createsuperuser` doesn't ask for a username, `manage.py changepassword` takes an email address, and I can login to the admin interface with my email address.
Can you please post this code somewhere? I'd love to see it.

Sure, here's a fork of 1.4rc1 that allows for pluggable auth apps:


The patch is rather large, but represents mostly the movement of files, and some refactoring of components to allow for greater reuse. For instance, the User model (and the forms, decorators, backends, and middleware that depend on it) has moved from django.contrib.auth to a new app, django.contrib.auth.default. This new app represents the default pluggable auth app, as reflected by the default value of the AUTH_APP setting (in a manner similar to the COMMENTS_APP setting).

Backwards compatibility is maintained by continuing to honor the existing import contract, e.g., `from django.contrib.auth.models import User`. User is defined at run-time to be either the default (django.contrib.auth.default.models.User), or a user-provided model, based on the value of AUTH_APP. With the exception of having to add the new django.contrib.auth.default app to INSTALLED_APPS, existing django installations should be unaffected by this change.

New django projects can elect to use an entirely different pluggable auth app, by setting AUTH_APP and adding such app to INSTALLED_APPS. Any library or bespoke code that imports User from django.contrib.auth.models will get the new model. The burden of ensuring that third-party and bespoke code is compatible with the new User model will fall on the author of that model. In other words, if I export a User model with no username field, I shouldn't expect that model to work in tandem with a blogging app that requires a username field.

To simplify the process of writing new pluggable auth apps, I've factored out some common functionality from django.contrib.auth.default and put it in django.contrib.auth.base. These base models are intended to be an aid to reuse, but are not required, and have no special properties.

As an example of a pluggable auth app, here's one I wrote that allows for email authentication:


The pluggable-auth-apps fork is barely two days old, so it's still rough around the edges, and I've almost certainly missed some things. I haven't written any new tests for it yet, but the existing suite passes—at least the tests that aren't skipped. I've installed all the requirements listed in the testing docs—including selenium—but I get:

----------------------------------------------------------------------
Ran 4639 tests in 438.865s

OK (skipped=72, expected failures=2)

If this approach looks like something we might want to move forward with, I'll start working on some tests and docs.

Cheers,

Clay

Clay McClure

unread,
Mar 14, 2012, 6:57:46 PM3/14/12
to django-d...@googlegroups.com
On Sun, Mar 11, 2012 at 4:39 AM, Clay McClure <cl...@daemons.net> wrote:

What you are really saying is this: being pragmatic means that we
prioritise *your* immediate need above the need to keep the code and the
docs maintainable, and above the need to maintain compatibility with
existing installations.

Luke, I'm sorry for being gruff and impatient in my previous reply. I'm anxious to see this problem fixed, but I don't want to sacrifice maintainability or compatibility to do it. I hope that the community can make some progress on this issue before the next release cycle. I'm willing to devote the time to it, whether it be tweaking my patch or working on a different solution altogether.

Cheers,

Clay

Danny Adair

unread,
Mar 15, 2012, 5:53:35 AM3/15/12
to django-d...@googlegroups.com
On Mon, Mar 12, 2012 at 16:57, Clay McClure <cl...@daemons.net> wrote:
>[...]

> New django projects can elect to use an entirely different pluggable auth
>[...]

> The pluggable-auth-apps fork is barely two days old, so it's still rough
> around the edges, and I've almost certainly missed some things. I haven't
>[...]

I'm sorry I don't know if I like that. I hope I understand correctly
what you're doing, and that my criticism is seen as constructive.
This sells itself as "pluggable auth" but really only provides the
originally requested auth by email, and I don't see how it could go
from there, i.e. what else would be possible (in regards to auth).
Pulling username and email out of (Base)User and then bringing one or
both of them - and the logic of which to check - back in, is not
really a pluggable auth.

It's the mixing of profile and information relevant to authentication
(and authorization - is_staff?) that's another problem of the existing
User model, and this is where this approach allows for dangerous
developments.
You've pulled those two out and said it's up to the chosen auth to
decide a) which to even include in the User model b) which is used to
authenticate and which is just "secondary" user (profile?)
information.
What about first_name and last_name? Why can't a pluggable auth dump
those from the User model? And certainly an "auth" should get to have
a say about "password".
And then... there's not much left anymore in that (Base)User model,
right... (GSoC: "reducing the user table to little more than an
identifying primary key")

Ok email auth problem solved. But now there's a simple way of
providing any User model I like, so first thing I'm going to do is to
bring in what I think the User should look like i.e. use the "auth" to
define what really is a "profile". And so will lots of projects. "The
User model needs a username for this app to work". "The User model
needs a timezone for this app to work". etc.
I'd like to see a later, "proper" auth.user that can undo that chaos.

Cheers,
Danny

Danny Adair

unread,
Mar 15, 2012, 6:05:23 AM3/15/12
to django-d...@googlegroups.com
On Thu, Mar 15, 2012 at 22:53, Danny Adair <danny...@unfold.co.nz> wrote:
>[...]

> I'd like to see a later, "proper" auth.user that can undo that chaos.

I originally thought you intended a setting which let's the user
switch from the existing auth to an email auth with corresponding User
model.
As lame as that may be - at least a later, "proper" auth.user solution
would only have two known (shipped) auths to deal with.

Cheers,
Danny

Daniel Sokolowski

unread,
Mar 15, 2012, 12:24:01 PM3/15/12
to django-d...@googlegroups.com
Tom makes a good point, but you can already store emails in the username,
they are just limited to 30 characters or fewer. Lift this length
restriction and I will be able to do everything I need without having to
wait for contrib.auth2. In the 'I use django to make a living' world the
solution doesn't have to be perfect just good enough. The talk of lazy
foreign keys just so we can create contrib.auth2 just rubs me the wrong way,
adds additional complexity, and I see no benefit of it over a UserProfile
model or with model inheritance.

And Django developer ought to be familiar how to do a schema migration for
this simple change, and if they fail to know instructions can be provided in
the release notes; there is NO need to have progress being stalled because
there is no built automated schema migration.

Why can we not just increase the length limit on the username field?, Why
can't we just throw a validation error if data entered is to long and the
schema has not been updated? I think the answer yes we can and easily.

just fix these glaring bugs�

Cheers

Tom

Luke Sneeringer

unread,
Mar 15, 2012, 12:28:57 PM3/15/12
to django-d...@googlegroups.com

On March 15, 2012, at 11:24 , Daniel Sokolowski wrote:

> Tom makes a good point, but you can already store emails in the username, they are just limited to 30 characters or fewer. Lift this length restriction and I will be able to do everything I need without having to wait for contrib.auth2. In the 'I use django to make a living' world the solution doesn't have to be perfect just good enough. The talk of lazy foreign keys just so we can create contrib.auth2 just rubs me the wrong way, adds additional complexity, and I see no benefit of it over a UserProfile model or with model inheritance.
>
> And Django developer ought to be familiar how to do a schema migration for this simple change, and if they fail to know instructions can be provided in the release notes; there is NO need to have progress being stalled because there is no built automated schema migration.
>
> Why can we not just increase the length limit on the username field?, Why can't we just throw a validation error if data entered is to long and the schema has not been updated? I think the answer yes we can and easily.

Wouldn't increasing the length of the username field *also* incur a schema migration? After all, anyone who has spun up a Django site previous to the change would have a CHARACTER VARYING(30) (or a VARCHAR(30) or whatever the appropriate database-specific type is) in their database. So, we'd have a situation where Django would accept a 40 character email address, but then the database would truncate it or error out.

The argument that Django developers ought to know how to do the schema migrations themselves seems poor to me. I expect some don't, and it's certainly a bigger backwards incompatible change than the Django development community has traditionally been comfortable with.

Regards,
Luke

> just fix these glaring bugs…

Daniel Sokolowski

unread,
Mar 15, 2012, 12:41:37 PM3/15/12
to django-d...@googlegroups.com
Yes it clearly would, however I see two possible solutions to make it more
friendly:

1. We provide MySQL, Sqlite3, PostgreSQL instructions on how to do it - I
mean the actual commands to execute.
2. Catch the DB error and throw a form validation warning for those that
didn't.

Regards,
Luke

> just fix these glaring bugs�

Daniel Sokolowski

unread,
Mar 15, 2012, 12:44:14 PM3/15/12
to django-d...@googlegroups.com
That would be a workable compromise, yes?

Carl Meyer

unread,
Mar 15, 2012, 12:49:17 PM3/15/12
to django-d...@googlegroups.com
Hi Daniel,

On 03/15/2012 09:24 AM, Daniel Sokolowski wrote:
> Why can we not just increase the length limit on the username field?,
> Why can't we just throw a validation error if data entered is to long
> and the schema has not been updated? I think the answer yes we can and
> easily.

I don't mean to pick on you specifically, but to me this is an excellent
example of a casual assertion of something we can "easily" do that
doesn't hold up under real examination (for instance, if you tried to
write the patch to actually do it).

How do you propose to "throw a validation error" if "the schema has not
been updated"? How do we know whether it's been updated? Are you
proposing that Django should introspect the users database table every
time it starts up in order to check the length of the username field?
Where would you put this introspection check? Have you considered the
effects this would have on every user of contrib.auth (both those who do
and don't run the schema migration)? And the effect on Django
development (needing to run the tests with both an "old" and "new" table
to ensure that the backwards-compatibility handling is tested?)

Carl

signature.asc

Carl Meyer

unread,
Mar 15, 2012, 12:52:18 PM3/15/12
to django-d...@googlegroups.com
On 03/15/2012 09:41 AM, Daniel Sokolowski wrote:
> Yes it clearly would, however I see two possible solutions to make it
> more friendly:
>
> 1. We provide MySQL, Sqlite3, PostgreSQL instructions on how to do it -
> I mean the actual commands to execute.
> 2. Catch the DB error and throw a form validation warning for those that
> didn't.

Catching the database error won't work - some databases won't raise an
error, they'll just silently truncate the value, potentially leading to
all sorts of nasty data corruption problems and confusing bugs down the
road.

Carl

signature.asc

Daniel Sokolowski

unread,
Mar 15, 2012, 1:23:54 PM3/15/12
to django-d...@googlegroups.com
Carl, I sincerely appreciate your feedback, however again it seems no
answers are given except more questions and considerations to consider. Why
is it so unreasonable that we expect the end developer to be able to
manually adjust their schemas, it's part of an upgrade process and it's been
done in the past labelled backwards incompatible changes due to bugs or
security, perhaps 30 character limitation ought to be considered a design
bug - tomorrow I'll do an R&D day and see the feasibility of a solution.

Luke Sneeringer

unread,
Mar 15, 2012, 2:11:15 PM3/15/12
to django-d...@googlegroups.com
On March 15, 2012, at 12:23 , Daniel Sokolowski wrote:

> Carl, I sincerely appreciate your feedback, however again it seems no answers are given except more questions and considerations to consider. Why is it so unreasonable that we expect the end developer to be able to manually adjust their schemas, it's part of an upgrade process and it's been done in the past labelled backwards incompatible changes due to bugs or security, perhaps 30 character limitation ought to be considered a design bug - tomorrow I'll do an R&D day and see the feasibility of a solution.
>

I don't think Carl's point was to provide answers; his point was to explain why the proposal isn't a very good one. He doesn't have to propose an alternative to establish that a proposal doesn't work.

Most of Django's backwards incompatible changes are corner cases that most end developers never actually encounter. The only exception I can think of to this is the CSRF changes to AJAX requests in Django 1.3. This change, on the other hand, would affect basically every single Django installation; this is much more widespread than past Django releases.

Regards,
Luke


> -----Original Message----- From: Carl Meyer
> Sent: Thursday, March 15, 2012 12:49 PM
> To: django-d...@googlegroups.com
> Subject: Re: authentication by email
>

Daniel Sokolowski

unread,
Mar 15, 2012, 2:48:33 PM3/15/12
to django-d...@googlegroups.com
Ok to recap,

The issue here is that django auth is limited, and restrictive and needs
hacks to make it use emails as usernames, we can agree on that yes? We can
also agree that contrib.auth2 with LFK is a complex undertaking far into the
future?. Can we also agree that the 30 character limitation on the username
ought to be increased?

-----Original Message-----
From: Luke Sneeringer

Clay McClure

unread,
Mar 15, 2012, 2:50:01 PM3/15/12
to django-d...@googlegroups.com
On Thu, Mar 15, 2012 at 5:53 AM, Danny Adair <danny...@unfold.co.nz> wrote:

Hi Danny,
 
I'm sorry I don't know if I like that. I hope I understand correctly
what you're doing, and that my criticism is seen as constructive.

Thanks for reviewing the code and providing feedback.
 
This sells itself as "pluggable auth" but really only provides the
originally requested auth by email, and I don't see how it could go
from there, i.e. what else would be possible (in regards to auth).
Pulling username and email out of (Base)User and then bringing one or
both of them - and the logic of which to check - back in, is not
really a pluggable auth.

It's "pluggable" in the sense that you could write a User model with whatever fields, methods, and attributes best suits your application. You aren't required to extend BaseUser—I only added that abstract class because I figure that most deviations from the stock User won't be that drastic, and it's convenient to be able to inherit from a model that has most of what you need already.

It's the mixing of profile and information relevant to authentication
(and authorization - is_staff?) that's another problem of the existing
User model, and this is where this approach allows for dangerous
developments.

My dissatisfaction with the stock User model has less to do with the the fact that it combines authentication credentials and profile information in a single model, and more to do with the fact that it's hard to work around the stock User model if it doesn't meet your needs. The stock User model is the simplest thing that could possibly work—and indeed it does work, for most projects—but it's also one of the few components of Django that isn't configurable or replaceable. I can provide a custom ModelBackend easily enough, and a custom AuthenticationForm, but I'm always stuck with the auth.User model.

We could debate for a long time (in fact, we have—the ticket was opened five years ago) what constitutes a proper User model, but may never reach a conclusion. Some will want to separate authentication from identity from profile, others will want to keep it simple. It's not clear to me that there's a "correct" User model, just as there's not a "correct" blogging application.

So, rather than offer specific improvements to the stock User model, I aim to provide a means for developers to use their own User model, should they need to do that.
 
What about first_name and last_name? Why can't a pluggable auth dump
those from the User model? And certainly an "auth" should get to have
a say about "password".
 
Certainly, you could write a pluggable User model without first_name and last_name fields. Some have requested a User model with a simple "name" field. You could do that. It may not work in the admin, unless you define a first_name property, but your User model should work with a great many applications.

The same is true for the password field: write a User model without it, if you'd like. That should work almost everywhere, even in the admin, so long as you provide your own UserCreationForm and UserChangeForm, and your own backend that handles authentication another way.

And then... there's not much left anymore in that (Base)User model,
right... (GSoC: "reducing the user table to little more than an
identifying primary key")

Just to be clear, the BaseUser that I've proposed is there for convenience. It is not required. We could pare down BaseUser even more, but I suppose there will always be some disagreement about what should be the minimum set of fields and methods on a User model. I avoid that debate by saying that BaseUser isn't required—inherit from it, or don't, it's your choice.

Some developers may want a User object that is just a primary key. They may have factored authentication and identity out in a way that is much more flexible (and much more complex) than the stock User model. That should be possible—and with pluggable auth apps, it is.

Ok email auth problem solved. But now there's a simple way of
providing any User model I like, so first thing I'm going to do is to
bring in what I think the User should look like i.e. use the "auth" to
define what really is a "profile".  And so will lots of projects. "The
User model needs a username for this app to work". "The User model
needs a timezone for this app to work". etc.
I'd like to see a later, "proper" auth.user that can undo that chaos.

I don't know that I would call that "chaos". For many applications, it makes sense to have a simple User model with some profile data. I might not want to incur the extra JOIN or SELECT required to fetch a separate UserProfile object. Maybe it makes sense to store some profile attributes (display name, for instance) with the authentication credentials. Maybe it doesn't make sense to store email address there—what if I have several email addresses?—or timezone—what if I travel?

There isn't a one-size-fits-all solution, but I think we can at least give developers a way to use a User model that fits their needs.

Cheers,

Clay

Clay McClure

unread,
Mar 15, 2012, 3:03:48 PM3/15/12
to django-d...@googlegroups.com
On Thu, Mar 15, 2012 at 2:48 PM, Daniel Sokolowski <daniel.s...@klinsight.com> wrote:
 
The issue here is that django auth is limited, and restrictive and needs hacks to make it use emails as usernames, we can agree on that yes?

I agree with this point.
 
We can also agree that contrib.auth2 with LFK is a complex undertaking far into the future?

I agree that LFK isn't a simple undertaking, and doesn't solve all the auth.User problems, anyway.
 
Can we also agree that the 30 character limitation on the username ought to be increased?

I don't agree that changing the length of the username field is the general solution to the problem of email authentication. First, there's the issue of backwards compatibility: as Carl pointed out, you can't just change the size of the field without requiring a schema migration in every existing django installation, and that's very painful. Second, there's the issue that if you're storing emails in your username field, you've got a redundant email field floating around.

With pluggable auth apps, you could make a User model with a longer username field, if you decide that's the best solution for your app, and if you're comfortable with the schema migration. I might elect to use a different User model that foregoes the username field altogether. With pluggable auth apps, the choice is mine—and yours. :)

Cheers,

Clay

Carl Meyer

unread,
Mar 15, 2012, 7:05:30 PM3/15/12
to django-d...@googlegroups.com
Hi Daniel,

On 03/15/2012 11:48 AM, Daniel Sokolowski wrote:
> The issue here is that django auth is limited, and restrictive and needs
> hacks to make it use emails as usernames, we can agree on that yes?

Certainly.

> We
> can also agree that contrib.auth2 with LFK is a complex undertaking far
> into the future?.

It is not a simple undertaking. How far into the future it is depends
entirely on whether someone who wants to see it happen takes it upon
themselves to make it happen.

> Can we also agree that the 30 character limitation on
> the username ought to be increased?

I am not sure whether this should happen as a separate step or not. In
an ideal world, we would have a longer username field. In the real
world, we have to balance the benefit against the cost, and requiring a
schema migration from practically every Django installation on the
planet would IMO be the most significant backwards-incompatible change
Django has ever shipped, at least since Django 1.0. It is not at all
clear to me that the status quo, bad as it is, is worse than this cure.

The cost would be mitigated somewhat if we had schema migrations in
core, which would make the upgrade instructions much simpler and more
foolproof.

The cost would be reduced even more if we simply shipped this as part of
a larger customizable-auth change (which will probably require at least
a deprecation path itself).

So, in my opinion, your energies would be more productively directed
towards helping make one of those latter two things happen. But feel
free to work on a proposal to simply change the field length. If you can
provide a patch with a really compelling backwards-compatibility story,
my mind is certainly open to change.

Carl

signature.asc

Tom Evans

unread,
Mar 16, 2012, 9:08:30 AM3/16/12
to django-d...@googlegroups.com
On Thu, Mar 15, 2012 at 4:49 PM, Carl Meyer <ca...@oddbird.net> wrote:
> Hi Daniel,
>
> On 03/15/2012 09:24 AM, Daniel Sokolowski wrote:
>> Why can we not just increase the length limit on the username field?,
>> Why can't we just throw a validation error if data entered is to long
>> and the schema has not been updated? I think the answer yes we can and
>> easily.
>
> I don't mean to pick on you specifically, but to me this is an excellent
> example of a casual assertion of something we can "easily" do that
> doesn't hold up under real examination (for instance, if you tried to
> write the patch to actually do it).
>

Why would anyone waste their time working on a patch that would be
instantly rejected, as has been made patently clear throughout.

The solution to these bugs is easy, it is just being ignored because
it is 'too hard' to make these kinds of changes. Instead wishy-washy
ways of completely rewriting authentication are dreamt up, planned,
and then dropped. This is how I would fix it:

1) Fix the email length limit bug, and mark the field as a db index,
but hide this by a settings hack, called CONTRIB_AUTH_EMAIL_FIX. If
the settings hack is not set, raise a PendingDeprecationWarning

2) Add a management command 'auth_NN_inspectdb', NN being the version
number, obviously. This command introspects the auth_user table, and
determines if it needs changing, and prints out whether the table
needs updating, which could include verbatim SQL statements for their
engine. To cope with multiple databases, it takes a --database arg.

3) Add a management command 'auth_NN_updatedb'. This command
introspects the auth_user table, and if it supports the engine,
updates the DDL.To cope with mutliple databases, it takes a --database
arg.

4) Add the following release notes:
The contrib.auth.models.User model has increased the length of the
email field to 254 characters, in order to allow all legal email
addresses to be stored. In addition, the email field has been marked
as a db index, in order to more easily allow email authentication.

These changes are not made automatically by Django, sine they involve
altering the structure of your database. Instead, you must enable this
fix with by adding the settings.CONTRIB_AUTH_EMAIL_FIX. In addtion,
there are two tools included to assist you inspect your database
structure and migrate to the new structure.

<description of auth_NN_inspectdb>
<description of auth_NN_updatedb>

If you do not want to make these changes, you can instead keep using
1.4 behaviour by not adding settings.CONTRIB_AUTH_EMAIL_FIX=True. You
will receive a PendingDeprecationWarning when starting up, as this fix
will soon become default.

5) Since this scares you all so much, bump version to 2.0

6) Once released, reverse the hack in trunk so that
settings.CONTRIB_AUTH_NO_EMAIL_FIX gets you 1.4 behaviour, and warns a
DeprecationWarning.

7) Add the following relnotes to 2.1

The contrib.auth.models.User schema changes originally introduced in
2.0 are now the default. If you still do not wish to migrate your
database structure, you can keep using the 1.4 behaviour for now, by
adding settings.CONTRIB_AUTH_NO_EMAIL_FIX=True to settings.py

<info about migration commands from previous relnotes>

If you do not either set settings.CONTRIB_AUTH_NO_EMAIL_FIX or update
your database schema, then you will have problems when you encounter a
user with an email address longer than 30 characters. The exact
problem you will encounter depends upon your database:

<description of what different databases do>

I feel the amount of work involved here for your average django end
user is minimal, certainly compared to things like adding csrf tokens
throughout.

It has been made abundantly clear that this approach would not be
considered - in particular, using settings hacks to pragmatically
achieve change - so why bother pushing it any further? I love a clean
design as much as the next person, but not fixing bugs that exclude
users on grounds of 'aesthetics' is pedantic and wrong.

> How do you propose to "throw a validation error" if "the schema has not
> been updated"? How do we know whether it's been updated? Are you
> proposing that Django should introspect the users database table every
> time it starts up in order to check the length of the username field?
> Where would you put this introspection check? Have you considered the
> effects this would have on every user of contrib.auth (both those who do
> and don't run the schema migration)? And the effect on Django
> development (needing to run the tests with both an "old" and "new" table
> to ensure that the backwards-compatibility handling is tested?)

This is a bug, and fixing it will affect lots of Django users. I'm fed
up of this excuse being used to avoid fixing it.

You seem to take the view that change is too hard, and therefore we
should not do this until we have a magical solution that is all things
for all men.

Personally, I take the view that every single django site out there is
broken for 'maria.hernan...@lar.longcorpname.com', has been
for a long time, and sweet FA has been done to fix it. That view says
fix it, fix it now, because I want to be paid by the lovely Maria, not
exclude her.

I'm fed up of core devs saying that my paying users aren't important
enough to fix such a minor bug, for fear of causing issues for little
Johnny's blog.

Cheers

Tom

PS: FYI, on my django based SAML identity provider, which provides
SAML authentication for my companies network of 150+ sites has these
stats for the length of users email addresses:

mysql> select length(email) > 30 as long_email, count(*) from
auth_user group by 1;
+------------+----------+
| long_email | count(*) |
+------------+----------+
| 0 | 150493 |
| 1 | 33531 |
+------------+----------+

I'm not talking about some silly edge case that catches 0.001% of
users. 18% of my paying users have issues, if they visit a django site
made by one of my colleagues that has not been fixed to accept longer
email addresses.

Luciano Pacheco

unread,
Mar 16, 2012, 1:22:33 PM3/16/12
to django-d...@googlegroups.com

On Fri, Mar 16, 2012 at 10:05 AM, Carl Meyer <ca...@oddbird.net> wrote:
[...]

I am not sure whether this should happen as a separate step or not. In
an ideal world, we would have a longer username field. In the real
world, we have to balance the benefit against the cost, and requiring a
schema migration from practically every Django installation on the
planet would IMO be the most significant backwards-incompatible change
Django has ever shipped, at least since Django 1.0. It is not at all
clear to me that the status quo, bad as it is, is worse than this cure.

I can't understand how bad is a database schema change. Almost all web site/applications need to change they database schema. Ok, in some cases there are people that don't update their databases, but I think this cases aren't willing to update their version of software as well. 
 
If the installed a web site/app is too small to be afraid to update, the database change will be fast enough to cause a minimal downtime.

If the installed a web site/app is too big, the sysadmin/devops already know how to couple with database schema changes. And it's likely that they have a test/staging/validation environment to analyse the changes before production.

This current limitation doesn't bother me, but all this concerned about database schema change does. :-)

Regards,
--
Luciano Pacheco
blog.lucmult.com.br

Felipe Prenholato

unread,
Mar 19, 2012, 6:45:25 PM3/19/12
to django-d...@googlegroups.com
I'm not a dev that contribute too  many directly to Django Project, but aniway I use Django since 0.96, so here is my 2 cents.

While Clay's idea have many cool changes and is a really nice idea that should be evaluated I share more Tom's and Daniel's point.

Any Django Developer on World that change between Django Versions should read release notes, and documentation on new stuff.

Any Django Developer on World have knowledge enough on your applications to decide migrate or not migrate to a new Django Version (they do it all time with third part apps).

Any conscient Django Developer will test change and will assert that change works.

Change will be documented, with solutions provided, and will fix a five year old bug with a simple enough fix.

With management commands we can check every field defined by a EmailField that need to be changed and create sql for that.

So, Django Devs are not childs, they can handle that change.

 +1 on change this fields.

Cheers,



Felipe 'chronos' Prenholato.
Linux User nº 405489
Home page: http://devwithpassion.com | http://chronosbox.org/blog
Reply all
Reply to author
Forward
0 new messages