authentication by email

[Wim Feijen]

Hello,

In the past hour, I did some research on authenticating by email and I
believe Django users would benefit a lot if email authentication was
included in contrib.auth .

Many people have been working on it, and the latest code I could find
is here: https://gist.github.com/586056. I am not a very good Googler,
so there may be better patches.

Anyway, there are several problems to solve besides this:
1. the default AuthenticationForm does not accept usernames longer
than 30 characters
2. UserCreationForm and possibly the UserChangeForm need to have Email
counterparts or become more flexible
3. User emails should be unique. My first thought is to add a unique
constraint depending on an optional AUTHENTICATE_BY_EMAIL setting
which defaults to False. I find this problem the hardest to solve.

I am really open to any suggestions, so please do.

Luke Plant, Julien Phalip, I know you have looked into this before and
I am really hoping you can share your thoughts as well.

----
https://gist.github.com/586056:

from django.contrib.auth.backends import ModelBackend
from django.contrib.auth.models import User

class EmailBackend(ModelBackend):

def authenticate(self, **credentials):
if 'username' in credentials:
return super(EmailBackend,
self).authenticate(**credentials)

try:
user = User.objects.get(email=credentials.get('email'))
if user.check_password(credentials.get('password')):
return user
except User.DoesNotExist:
return None

[Wim Feijen]

For the record and as an answer to the previous question, I'd like to
quote Russell who wrote the following in reponse to a ticket of mine:

" The core-endorsed ticket for this problem is #3011. The patch on
that
ticket isn't endorsed, but it points at the real problem - a need to
be
able to specify a custom User model. There have been several
discussions
directed at this problem, and I'm hoping to get a posse discussing
this at
the upcoming DjangoCon sprints. If you want background on some
approaches
that might work (and some that won't) search the archives for Lazy
Foreign
Key. "

Thanks Russell and I wish you good luck in the Django sprints!

- Wim

On Aug 26, 5:23 pm, Wim Feijen <wimfei...@gmail.com> wrote:
> Hello,
>
> In the past hour, I did some research on authenticating by email and I
> believe Django users would benefit a lot if email authentication was
> included in contrib.auth .
>
> Many people have been working on it, and the latest code I could find
> is here:https://gist.github.com/586056. I am not a very good Googler,
> so there may be better patches.
>
> Anyway, there are several problems to solve besides this:
> 1. the default AuthenticationForm does not accept usernames longer
> than 30 characters
> 2. UserCreationForm and possibly the UserChangeForm need to have Email
> counterparts or become more flexible
> 3. User emails should be unique. My first thought is to add a unique
> constraint depending on an optional AUTHENTICATE_BY_EMAIL setting
> which defaults to False. I find this problem the hardest to solve.
>
> I am really open to any suggestions, so please do.
>
> Luke Plant, Julien Phalip, I know you have looked into this before and
> I am really hoping you can share your thoughts as well.
>

> ----https://gist.github.com/586056:

[Clay McClure]

"Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design"—unless you want to do something seemingly simple like using email addresses for authentication, in which case you need to monkey patch models and forms to get everything working right, which is neither rapid nor clean. What began as an innocuous feature request five years ago is now a high-level, general purpose, abstract, seemingly insurmountable design problem. The core developers are still perfectionists, but they seem to have forgotten their deadlines.

Is there not a simple, pragmatic solution (optional and for new installations—we're not talking about backwards compatibility here) that could be implemented until the panacea of pluggable User models gets figured out? Something as simple (albeit ugly) as wrapping new models and forms in:

if settings.AUTH_EMAIL_AUTHENTICATION:

Should these things really take five years? What happened to pragmatic?

Clay

[Donald Stufft]

The major issue is that there is no way to do schema migrations in core (currently). So there's no way to handle increasing the length of the username field.

More comprehensive solutions require more thought to figure out the pluggable User models.

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/KebjFDOOBF4J.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

[Matt Pegler]

For the project I am working on, we solved this by making a custom
auth backend that checks the username against the email column. We've
found it to be a nice clean solution to wanting to use email addresses
instead of usernames.

[Matt Pegler]

Sorry all, disregard my previous email, I misread one thing and
completely missed Wim's original message.

[Danny Adair]

On Fri, Mar 9, 2012 at 20:01, Donald Stufft <donald...@gmail.com> wrote:
> The major issue is that there is no way to do schema migrations in core
> (currently). So there's no way to handle increasing the length of the
> username field.

I don't understand what the "username" field length has to do with it.

And I think that's also the problem with Clay's original post:
Do you want to authenticate against email or also change the User model?

What if I want to authenticate against email but still have a username ("nick")?
Another setting?

An auth backend that checks against the email address (which is
already in the User model just like username) _is_ simple.
Rapid and simple would be to _ship_ such a backend in contrib.auth.backends.
Then it would really just be a single setting (AUTHENTICATION_BACKENDS
= ('django.contrib.auth.backends.EmailBackend',)).
The only thing left would be the label on the login template in my
naive understanding. (another one-liner I believe)

So I agree it would be nice and useful, but where's the monkeypatching.
If you want to authenticate against email _and_ take the "username"
out of the model, it's that second part that's the problem. That's
where I believe a set of settings and conditions is a kneejerk
workaround.

Cheers,
Danny

[Florian Apolloner]

Hi,

On Friday, March 9, 2012 6:54:16 AM UTC+1, Clay McClure wrote:

if settings.AUTH_EMAIL_AUTHENTICATION:

Hell, not another ugly setting like this.

Should these things really take five years? What happened to pragmatic?

Yes, since no one needs it. Okay no one isn't true, but no one (for true this time) who needed it stepped up and said "I'll implement it and see that it ends up in trunk"


Regards,
Florian

[Danny Adair]

On Fri, Mar 9, 2012 at 21:13, Florian Apolloner <f.apo...@gmail.com> wrote:
>[...]

> Yes, since no one needs it. Okay no one isn't true, but no one (for true
> this time) who needed it stepped up and said "I'll implement it and see that
> it ends up in trunk"

It's the "required" of username that's the problem if you don't want a
username at all when authenticating against email.
It would have to be not required and check required fields in clean()
where the backend could be asked what's really required.
And there's Mr. Schema Migration again...

Cheers,
Danny

[Tino de Bruijn]

My django-email-login app (https://bitbucket.org/tino/django-email-login/overview) does this by putting a hash of the email adress in the username field. It isn't as nice as it could be, but it works.

I would really like to see this solved another way, but it is a hard problem with the current restrictions.

Tino

--
You received this message because you are subscribed to the Google Groups "Django developers" group.

[Clay McClure]

On Fri, Mar 9, 2012 at 3:23 AM, Danny Adair <danny...@unfold.co.nz> wrote:

 

It's the "required" of username that's the problem if you don't want a
username at all when authenticating against email.
It would have to be not required and check required fields in clean()
where the backend could be asked what's really required.

That's one problem. Another problem is that the default User.email field is not unique, is not required, is not indexed, and may be too short.

And there's Mr. Schema Migration again...

Who's talking about a migration? I'm asking for something that will work for *new* installations; existing installations can continue authenticating against usernames  for all I care :)

Moreover, I'm thoroughly frustrated by the fact that developers *do* bring working solutions to the table (in the form of patches in trac), but the core developers won't integrate them, either because it's not the right time (usually right before a release), because the patch isn't general enough, or because the patch doesn't meet some absurdly high bar of quality. I understand that they have a commitment to backwards compatibility and that accepting a mediocre patch could mean maintaining it for life, but I like to think that—with a framework that purports to be "pragmatic"—there are pragmatic solutions to these problems—and not, "sorry, it's not perfect, it can't go in."

The GSoC page (https://code.djangoproject.com/wiki/SummerOfCode2011#Enhancedauth.user) is a frustrating read. It goes on and on about how hard the problem is and how wrong your solution is, but doesn't provide any detail as to why it's hard or why it's wrong. Ticket #3011 was rejected without much of a reason. HM asked for an explanation both in the ticket itself and on django-dev; no explanation was ever given.

The GSoC page also mentions migrations:

"How can we roll out a new/modified User model without requiring almost every Django application on the planet to undergo a complex database modification?"

But again, why do existing databases have to change? We're talking about leaving User as-is, by default, but providing a mechanism to use a different model if the developer chooses. Clearly this is a decision the developer would not take lightly: you're not going to change from username authentication to email authentication without a bit of thought. Projects that are using username authentication will continue to use username authentication, but at least new projects that want/need email authentication will be able to do that without monkey patching models.

The GSoC page also mentions Lazy Foreign Keys, but no explanation is given there or in the linked django-dev thread as to why LFKs are required to implement a pluggable User model. LFK may be the panacea of code cleanliness and virtual awesomeness, but those of us with deadlines just need to authenticate against email addresses, like, five years ago.

Cheers,

Clay

[Tom Evans]

On Fri, Mar 9, 2012 at 8:13 AM, Florian Apolloner <f.apo...@gmail.com> wrote:
>> Should these things really take five years? What happened to pragmatic?
>
>
> Yes, since no one needs it. Okay no one isn't true, but no one (for true
> this time) who needed it stepped up and said "I'll implement it and see that
> it ends up in trunk"
>

I'm sorry, that completely mis-characterises the situation. Lots of
people want this, but every single time this has been brought up since
I started using django (1.0), a core dev or BFDL has explicitly ruled
it out, saying "we cannot do this, requires schema migration, we'll
look at it for the next version".

At this point, the discussion is stopped. Once a proposal has the
smack down from a BFDL, what point is there trying to get it
integrated?

The next version comes, has it been addressed? No. Will it be
addressed in 1.4? No. Will it be addressed in 1.5? Possibly, but I
doubt it.

At $JOB we've given up waiting for some trivial changes, and have
simply patched django to not have these silly, arbitrary and incorrect
requirements¹ on email address and username.

We live in the real world, we have to be able to handle email
addresses that are of a valid length. We cannot tell a paying customer
"I'm sorry, your email address is too long, do you have one under 75
characters?"

It's a really invalid position to take, knowing something is broken
and incorrect, but not fixing it because 'change is hard'.

Cheers

Tom

¹ Here is my list of things that are broken with d.c.auth:

1) Length and character restrictions on username field. If I want to
use emails address as usernames, or usernames with spaces, the choices
made at lawrence.com 3+ years ago should not restrict me.
2) Email address fields must be large enough to hold valid email
addresses - that's 254 characters.

[Luke Plant]

On 09/03/12 09:37, Clay McClure wrote:

> Who's talking about a migration? I'm asking for something that will work
> for *new* installations; existing installations can continue
> authenticating against usernames for all I care :)
>
> Moreover, I'm thoroughly frustrated by the fact that developers *do*
> bring working solutions to the table (in the form of patches in trac),
> but the core developers won't integrate them, either because it's not
> the right time (usually right before a release), because the patch isn't
> general enough, or because the patch doesn't meet some absurdly high bar
> of quality. I understand that they have a commitment to backwards
> compatibility and that accepting a mediocre patch could mean maintaining
> it for life, but I like to think that—with a framework that purports to
> be "pragmatic"—there are pragmatic solutions to these problems—and not,
> "sorry, it's not perfect, it can't go in."

What you are really saying is this: being pragmatic means that we
prioritise *your* immediate need above the need to keep the code and the
docs maintainable, and above the need to maintain compatibility with
existing installations.

There are a million-and-one hacks we would add if we took that approach,
and Django would have a million-and-one more bugs, or possibly much
worse - if you add just 10 boolean switches like the one you suggested
in an earlier email, you've got over 1000 combinations to test and
debug. The only way to avoid that in a framework like Django is well
defined APIs for components. No-one has brought such a thing to the
table and seen it through, that's why it hasn't come in.

I believe that all the problems you have can be solved already:

- you can write your own authentication backend to authenticate
by email

- you can write your own login and account creation views

- you can even override admin views by inserting items into
urls.py

Yes, these are all more-or-less hacky. And they have problems. For
instance, what if you have two accounts with the same email address? (I
have that in some of my Django projects, BTW). The current code allows
it, so we can't change anything that would make it impossible, at either
the DB level or the Python level. This is the kind of issue that means
you can't just use the existing models.

A full and correct solution that didn't introduce dozens of new bugs is
quite hard - and I'm talking about just the code required to fix this
one feature for you, without doing anything more generally useful.

That is why we're not going to put hacks like this in core - we would
have to support these hacks for at least 3 versions if we did. We are
interested in *removing* existing hacks from our code base, and this is
vital if Django is going to see increases in features and functionality.
We are not interested in adding more hacks.

We also will refuse to litter our documentation with things like: "if
you've got this setting, it does this, otherwise you get this other
behaviour, and you've got to watch out for all these bugs because this
is a gross hack".

> The GSoC page
> (https://code.djangoproject.com/wiki/SummerOfCode2011#Enhancedauth.user)
> is a frustrating read. It goes on and on about how hard the problem is
> and how wrong your solution is, but doesn't provide any detail as to why
> it's hard or why it's wrong.

It does provide detail - it gives a list of issues you have to consider,
and tells you where to search for the other issues.

Some examples from a quick Google groups search for "auth User":

http://goo.gl/swTpr

http://goo.gl/fFlKh

These provide a lot of detail.

> Ticket #3011 was rejected without much of a
> reason. HM asked for an explanation both in the ticket itself and on
> django-dev; no explanation was ever given.

I don't know what thread you are talking about. Hanne Moa brings up the
subject here and gets two replies from Russell:

http://goo.gl/7p1JN

Regards,

Luke

--
"I imagine bugs and girls have a dim suspicion that nature played a
cruel trick on them, but they lack the intelligence to really
comprehend the magnitude of it." (Calvin and Hobbes)

Luke Plant || http://lukeplant.me.uk/

[Tom Evans]

On Fri, Mar 9, 2012 at 3:36 PM, Luke Plant <L.Pla...@cantab.net> wrote:
> What you are really saying is this: being pragmatic means that we
> prioritise *your* immediate need above the need to keep the code and the
> docs maintainable, and above the need to maintain compatibility with
> existing installations.
>
> There are a million-and-one hacks we would add if we took that approach,
> and Django would have a million-and-one more bugs, or possibly much
> worse - if you add just 10 boolean switches like the one you suggested
> in an earlier email, you've got over 1000 combinations to test and
> debug. The only way to avoid that in a framework like Django is well
> defined APIs for components. No-one has brought such a thing to the
> table and seen it through, that's why it hasn't come in.
>
> I believe that all the problems you have can be solved already:
>
>  - you can write your own authentication backend to authenticate
>   by email
>
>  - you can write your own login and account creation views
>
>  - you can even override admin views by inserting items into
>   urls.py
>

Even with these things, you are limited by the length of the username
field, the length of the email field, and the arbitrary restrictions
on what consists a username.

The length restrictions could be fixed straight away, and a note added
to the next relnotes informing users that the field has changed in
size. If they do not update their DB structure, then depending on the
database used, they will either get errors when a field value is
longer than the old limit, or silent truncation.

It's not ideal, but it doesn't cause any major problems for anyone who
reads the relnotes, and it will have to be done at some point anyway.
Even if Django did have schema migrations, the same behaviour would
occur for anyone who read the relnotes and did not apply the
migration, and many users with 'interesting' DB structures would not
be able to simply apply schema migrations either, so they would have
to do it by hand anyway.

It feels to me that only fear of change and how to manage that change
is what is stopping d.c.auth from being fixed.

>
> Yes, these are all more-or-less hacky. And they have problems. For
> instance, what if you have two accounts with the same email address? (I
> have that in some of my Django projects, BTW). The current code allows
> it, so we can't change anything that would make it impossible, at either
> the DB level or the Python level. This is the kind of issue that means
> you can't just use the existing models.
>

Good for you! I don't see that as an issue. If you are using email
address as a unique name for identifying a user, we already have a
column for that, it's called username, and it has some of the right
semantics (unique, indexed field). It is the arbitrary length and
content restrictions that stop this from happening right now.

I don't see why wanting to use email address as the unique identifier
and username for a user requires any modification to the database,
apart from (as I keep repeating) removing the arbitrary length and
content restrictions on username.

> A full and correct solution that didn't introduce dozens of new bugs is
> quite hard - and I'm talking about just the code required to fix this
> one feature for you, without doing anything more generally useful.
>
> That is why we're not going to put hacks like this in core - we would
> have to support these hacks for at least 3 versions if we did. We are
> interested in *removing* existing hacks from our code base, and this is
> vital if Django is going to see increases in features and functionality.
> We are not interested in adding more hacks.
>
> We also will refuse to litter our documentation with things like: "if
> you've got this setting, it does this, otherwise you get this other
> behaviour, and you've got to watch out for all these bugs because this
> is a gross hack".
>

I agree with this 100%. d.c.auth needs to be fixed, not have many
different settings that alter it's behaviour.

Cheers

Tom

[Luke Plant]

On 09/03/12 14:49, Tom Evans wrote:

>> Yes, since no one needs it. Okay no one isn't true, but no one (for true
>> this time) who needed it stepped up and said "I'll implement it and see that
>> it ends up in trunk"
>>
>
> I'm sorry, that completely mis-characterises the situation. Lots of
> people want this, but every single time this has been brought up since
> I started using django (1.0), a core dev or BFDL has explicitly ruled
> it out, saying "we cannot do this, requires schema migration, we'll
> look at it for the next version".

This is not true. There have been times when we have said we cannot
consider it right now, and that a solid proposal should be brought up at
the calls for proposals for 1.1/1.2/1.3 etc., or at other times when the
core developers can look at it.

And every time, the proposal has either been missing or simply not been
adequate - for example, the GSoC 2010 and 2011 proposals regarding this.
Russell gave detailed feedback on why these solutions were not adequate
[1], and at other times has provided feedback on possible strategies
[2]. And the people who went away to work on the problem have gone
silent or admitted defeat. One presumes they fixed their immediate
problem in a way that would not be a general solution, and moved on, and
that is perfectly understandable. A adequate solution to this is very
hard, and probably requires solving several other big problems first
(like at least one of lazy foreign keys/virtual models/database migrations).

It isn't, however, impossible. Our definition of 'need' is that someone
is sufficiently motivated to overcome the obstacles, and contribute a
solution that works for other people as well as themselves.

Regards,

Luke

[1] http://goo.gl/swTpr

[2] http://goo.gl/7p1JN

--

Luke Plant || http://lukeplant.me.uk/

[Tom Evans]

I disagree. There are small problems with the User model that have not
been fixed. Every proposal you've listed does not address these
issues, but instead looks to find ways of replacing the base user
object, which is a complex and difficult project.

There is a reason why these are the proposals. Every time someone
proposes fixing these minor bugs, the cry from on high is that
instead, d.c.auth will be overhauled to allow for more customization
by the project, immediately increasing the scope and complexity. But
this fabled d.c.auth2 never appears.

In fact, your reply alludes to this. We're talking about why a field
is too short to contain valid values, and you suggest that the
solution inevitably will involve "lazy foreign keys/virtual
models/database migrations" - it's changing one line, and adding a
comment to relnotes!

We may have to agree to disagree on how easy some of these things are to fix.

Lets look at one isolated aspect. The User email field in d.c.auth is
too short. Emails can be up to 248 characters long, and d.c.auth only
allows 75.

Clearly, this is a bug, is wrong and should be fixed. How can we fix it?

We can alter the length of the field in the model definition. What
problems does this bring?

Users who do not manually update their database will not be able to
store longer length emails, and will get 'strange effects' when they
try - either silent truncation, or database integrity errors. How can
we mitigate this for users who refuse to update?

Provide a compat shim that resets the email length to that in 1.3, and
immediately mark it PendingDeprecation. Users who read the relnotes
and don't want to change their DB can use the shim, and have a clear
expectation of when they must have updated their database.

I still think these bug fixes are paralysed by a fear of change, and a
desire to engineer a solution that is every thing to every man. Lets
just fix these glaring bugs…

Cheers

Tom

[Joe & Anne Tennies]

While, I generally agree with the current approach, especially this close to release. I'm going to play devil's advocate for a bit.

Schema migrations have been talked about for quite a while. There are at least 3 external implementations I know of: South, nashvegas, and django-evolution. I'm unsure of the status of django-evolution, but the other two appear quite active.

A concern in my mind is that all three go off and do things totally different ways, and the migration path to a common standard is more difficult.

I love the idea of not choosing a tool to make the migrations, but instead add to the ability to execute migrations. Now comes the conversation of what that means. I would love to try to get this in to Django 1.5.

Now on to the meat of what I really want:
Is there something along the lines of the GNOME design group for Django? (example: https://live.gnome.org/Design/Apps/Web). I'd love a template for the wiki or even better an app to discuss these things. If only we had some web developers ;) So, is there a sanctioned tool already out there? Or should I start building up a page on the wiki for this? I think most things become much easier once one gets a blueprint and requirements for what the design goals are. Especially if one can get an initial "this looks acceptable" by a core committer before "wasting one's time" because the requirements weren't even correct.

Does github or bitbucket have such a feature I don't know about? Does launchpad.net blueprints meet this? Is there some other place I don't know about that someone could point me to?

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

--
Joe & Anne Tennies
ten...@gmail.com

[Łukasz Rekucki]

On 9 March 2012 17:46, Tom Evans <teva...@googlemail.com> wrote:
>
> Lets look at one isolated aspect. The User email field in d.c.auth is
> too short. Emails can be up to 248 characters long, and d.c.auth only
> allows 75.

The latest RFC[1] actually specifies this as 256 *octets* with max of
64 octets for the local part and 255 octets for the domain name. So
248 *characters* would actually be incorrect and all the tedious and
error prone fixing of every Django instance would just get wasted.

I don't really think this is a "fear of change" case. It's more of "we
don't want to have to fix this again" thing.

[1]: http://tools.ietf.org/html/rfc5321#section-4.5.3.1.1

--
Łukasz Rekucki

[Tom Evans]

2012/3/9 Łukasz Rekucki <lrek...@gmail.com>:

> On 9 March 2012 17:46, Tom Evans <teva...@googlemail.com> wrote:
>>
>> Lets look at one isolated aspect. The User email field in d.c.auth is
>> too short. Emails can be up to 248 characters long, and d.c.auth only
>> allows 75.
>
> The latest RFC[1] actually specifies this as 256 *octets* with max of
> 64 octets for the local part and 255 octets for the domain name. So
> 248 *characters* would actually be incorrect and all the tedious and
> error prone fixing of every Django instance would just get wasted.
>

Sorry, 248 was a typo. If you look at my earlier reply in this thread,
I had correctly stated the maximum length of an email address as 254
*characters*.

If you check two paragraphs later in the RFC that you linked to, you
would see confirmation of this:

http://tools.ietf.org/html/rfc5321#section-4.5.3.1.3

You may also need to remind yourself what the definition of a path is.

As all email headers are either 7 or 8 bit encodings, describing 254
octets as 254 characters is perfectly valid.

None of this changes that django keeps shipping with bad defaults.
This was brought up for 1.1, 1.2, 1.3 and now 1.4.

Cheers

Tom

[Łukasz Rekucki]

On 9 March 2012 21:10, Tom Evans <teva...@googlemail.com> wrote:
> 2012/3/9 Łukasz Rekucki <lrek...@gmail.com>:
>> On 9 March 2012 17:46, Tom Evans <teva...@googlemail.com> wrote:
>>>
>>> Lets look at one isolated aspect. The User email field in d.c.auth is
>>> too short. Emails can be up to 248 characters long, and d.c.auth only
>>> allows 75.
>>
>> The latest RFC[1] actually specifies this as 256 *octets* with max of
>> 64 octets for the local part and 255 octets for the domain name. So
>> 248 *characters* would actually be incorrect and all the tedious and
>> error prone fixing of every Django instance would just get wasted.
>>
>
> Sorry, 248 was a typo. If you look at my earlier reply in this thread,
> I had correctly stated the maximum length of an email address as 254
> *characters*.
>
> If you check two paragraphs later in the RFC that you linked to, you
> would see confirmation of this:
>
> http://tools.ietf.org/html/rfc5321#section-4.5.3.1.3
>
> You may also need to remind yourself what the definition of a path is.

And RFC 3696 originally claimed it's 320. Even after errata it still
says 256, but you are right that 254 is probably more correct.

>
> As all email headers are either 7 or 8 bit encodings, describing 254
> octets as 254 characters is perfectly valid.

UTF-8 is an 8-bit encoding that doesn't map octets to characters, so
no. But that's not really the point I wanted to make.

My point is that this kind of things change and we should have tools
to deal with that. We already have the exact same problem with IPv6
and contrib.comments, it's just people didn't noticed it yet as much.
Doing this by hand every time isn't very effective. Django needs
schema migrations.

As for the "design page" that Joe mentioned, the GSOC description is a
good start:

https://code.djangoproject.com/wiki/SummerOfCode2012#Enhancedauth.user

--
Łukasz Rekucki

[Luciano Pacheco]

2012/3/10 Łukasz Rekucki <lrek...@gmail.com>

On 9 March 2012 21:10, Tom Evans <teva...@googlemail.com> wrote:
> 2012/3/9 Łukasz Rekucki <lrek...@gmail.com>:
>> On 9 March 2012 17:46, Tom Evans <teva...@googlemail.com> wrote:

[...] 

My point is that this kind of things change and we should have tools
to deal with that. We already have the exact same problem with IPv6
and contrib.comments, it's just people didn't noticed it yet as much.
Doing this by hand every time isn't very effective. Django needs
schema migrations.

[...]

The tool for migration can be simple as docs/release note or release warning :-) ?

We change the database schema and provide the SQL scripts to migrate the 4 supported backends (sqlite, mysql, postgres and oracle). This script are needed "only" for upgrade projects, new projects it's just a normal path.

Is that hard, inconvenient or bad?

In my humble opinion it's simple and pragmatic.

[],

--
Luciano Pacheco
blog.lucmult.com.br

[Russell Keith-Magee]

On 10/03/2012, at 1:52 AM, Joe & Anne Tennies wrote:

> While, I generally agree with the current approach, especially this close to release. I'm going to play devil's advocate for a bit.
>
> Schema migrations have been talked about for quite a while. There are at least 3 external implementations I know of: South, nashvegas, and django-evolution. I'm unsure of the status of django-evolution, but the other two appear quite active.

Django Evolution isn't dead. It's sleeping. It's got lovely plumage, you know... :-)

Seriously -- I stopped working on Evolution several years ago. I formally handed over the project to Christian Hammond almost 2 years ago, but this has been mostly for maintenance purposes, rather than active development. For the benefit of posterity, I blogged about the reasons [1].

[1] http://cecinestpasun.com/entries/end-my-evolution/

> A concern in my mind is that all three go off and do things totally different ways, and the migration path to a common standard is more difficult.
>
> I love the idea of not choosing a tool to make the migrations, but instead add to the ability to execute migrations. Now comes the conversation of what that means. I would love to try to get this in to Django 1.5.

A lot of this conversation has already happened, and it has core team approval. Most of these discussions were elaborated in last year's aborted GSoC 2011 project; search the django-dev archives from this time last year to see the details of what was being proposed.

The short version:

* Add a db.backends module to provide an abstract interface to migration primitives (add column, add index, rename column, rename table, and so on).

* Add a contrib app that performs the high level accounting of "has migration X been applied", and management commands to "apply all outstanding migrations"

* Provide an API that allows end users to define raw-SQL migrations, or native Python migrations using the backend primitives.

* Leave the hard task of determining dependencies, introspection of database models and so on to the toolset contributed by the broader community.

Under this approach, South would still need to exist, but it would be a tool leveraging the primitives exposed by Django itself. Over time, more features from South (and any other migration support projects that emerge) could be merged into trunk as the community converged on a feature set or an implementation.

One of the big issues that needs to be addressed is testing -- we need to have the infrastructure that allows us to check that migrations have been applied. Arthur Koziel's AppRefactor (a GSoC 2010 project) has some analogous testing problems, so getting migrations into trunk may be dependent on getting the App Refactor into trunk as well. The good news is that the App Refactor also forms the likely stub for fixing the auth.User problem -- but that's a separate discussion.

> Now on to the meat of what I really want:
> Is there something along the lines of the GNOME design group for Django? (example: https://live.gnome.org/Design/Apps/Web). I'd love a template for the wiki or even better an app to discuss these things. If only we had some web developers ;) So, is there a sanctioned tool already out there? Or should I start building up a page on the wiki for this? I think most things become much easier once one gets a blueprint and requirements for what the design goals are. Especially if one can get an initial "this looks acceptable" by a core committer before "wasting one's time" because the requirements weren't even correct.

There are some historical examples where we've used the wiki to elaborate on an idea as a precursor to getting a feature into core.

https://code.djangoproject.com/wiki/SessionMessages
https://code.djangoproject.com/wiki/ClassBasedViews
https://code.djangoproject.com/wiki/LoggingProposal
https://code.djangoproject.com/wiki/ReplacingGetAbsoluteUrl

(The first two are probably the best examples to follow)

There is a wiki page which has been used in the past to discuss schema evolution (which it looks like you've found); however, it's wandered a little off the reservation. There's probably some good material in there still, but if someone is looking at this, it would benefit from some manicuring, and probably a complete reboot:

https://code.djangoproject.com/wiki/SchemaEvolution

There isn't a specific wiki template that we require -- this is mostly because every project has slightly different requirements. However, the broad format is fairly common:

* describe the problem
* describe any specific design constraints
* describe the possible approaches and their benefits/limitations
* list any prior art
* list any draft implementations

If you want to work on this, I can guarantee that you would have as much support as I and Andrew Godwin (South maintainer and Django core developer) can muster. This is one of the big ticket items missing from Django's core, and I'd *love* to see someone take a serious swing at addressing it. As with every big feature that is "missing" from Django (be it migrations, auth.User problems, or anything else), the issue is almost always finding someone who is willing to do the work -- not just for a day or two, but to actively drive a project for several months.

Yours,
Russ Magee %-)

[Clay McClure]

On Fri, Mar 9, 2012 at 10:36 AM, Luke Plant <L.Pla...@cantab.net> wrote:

What you are really saying is this: being pragmatic means that we

prioritise *your* immediate need above the need to keep the code and the
docs maintainable, and above the need to maintain compatibility with
existing installations.

Of course not. Firstly, this is not *my* need; it's a feature that many have requested. Take a survey of the web applications that you use on a weekly basis. How many of them allow you to sign in with an email address? Compared to that, how many required you to invent a username? Of those, I would wager that a good many are Django apps.

Secondly, I'm advocating for a backwards-compatible solution that won't break existing installations. That point keeps getting missed, as you and others keep assuming that we need schema migrations for this change—we don't. People who are happy with the existing user model can keep using it, as-is. But people who aren't happy with it should be able to provide their own.

 

There are a million-and-one hacks we would add if we took that approach,
and Django would have a million-and-one more bugs, or possibly much
worse - if you add just 10 boolean switches like the one you suggested
in an earlier email, you've got over 1000 combinations to test and
debug.

I was giving an example of a pragmatic solution: sometimes an `if` statement is better than a far-reaching architectural change. I don't advocate boolean flags for solving the auth.User problem, I'm merely trying to encourage a shift in thinking from lofty aspiration to humble accomplishment.

 

I believe that all the problems you have can be solved already:

 - you can write your own authentication backend to authenticate
  by email

True.

 

 - you can write your own login and account creation views

True.

 

 - you can even override admin views by inserting items into
  urls.py

True.

But these things aren't the problem. It's the auth.User model that poses the challenge. If you haven't implemented email-based authentication in your own apps, let me suggest that you try it before you tell me how easy it is. At some point, you will run into these limitations posed by the User model:

 - The username field can't be used to store emails (it's too short and restrictive). No problem, let's use the email field;

 - But the username field is required. OK, we can just fill it with a random string;

 - The email field does not have a unique constraint. But we can do a reasonably good job applying the constraint in python;

 - The email field is not indexed. We don't want full table scans whenever someone logs in, so let's write a custom (and possibly non-portable) SQL migration to add an index.

All this, and the email field is still too short to hold all valid email addresses. On top of that, we've littered our database and admin interface with random strings for usernames, and littered our application with the code to generate them.

Yes, these are all more-or-less hacky. And they have problems. For
instance, what if you have two accounts with the same email address? (I
have that in some of my Django projects, BTW). The current code allows
it, so we can't change anything that would make it impossible, at either
the DB level or the Python level. This is the kind of issue that means
you can't just use the existing models.

Of course, which is why we need to allow developers to supply their own User model, while maintaining backwards compatibility for those who are happy with the built-in model.

A full and correct solution that didn't introduce dozens of new bugs is
quite hard - and I'm talking about just the code required to fix this
one feature for you, without doing anything more generally useful.

Again, it's not just for me. And, yes, I appreciate that it's not a trivial problem. It's hard, but all the problems worth solving are.

That is why we're not going to put hacks like this in core - we would
have to support these hacks for at least 3 versions if we did. We are
interested in *removing* existing hacks from our code base, and this is
vital if Django is going to see increases in features and functionality.
We are not interested in adding more hacks.

I can appreciate the appeal of a pristine, hack-free code base, but I've never seen one. Eventually reality always trumps grand design. If my example of the `if` statement above has led you to believe that I'm proposing a mindless hack, then I apologize. I'm proposing something between a mindless hack and grand design. Something pragmatic.

Some examples from a quick Google groups search for "auth User":

http://goo.gl/swTpr

Some hand waving ("It's deceptively difficult"), with no examples, and no mention of #3011.

http://goo.gl/fFlKh

This is someone with mediocre design skills trying to get paid to hack on Django over the summer, but failing to convince anyone that he has a clue about how to solve this problem. Save for some humorous jibes from Russell, the thread is almost devoid of good content, except for Russell's example of the COMMENTS_APP setting, which sounds a lot like "the idea of a 'pluggable' User model"—a proposal that the GSoC page indicates has already been rejected, but without explanation.

> Ticket #3011 was rejected without much of a
> reason. HM asked for an explanation both in the ticket itself and on
> django-dev; no explanation was ever given.

I don't know what thread you are talking about. Hanne Moa brings up the
subject here and gets two replies from Russell:

http://goo.gl/7p1JN

I was thinking of this thread:

http://goo.gl/nehrO

wherein Carl Meyer (I confused CM with HM) enquires about #3011 and receives no reply. In the thread you mention, there isn't any detail about #3011, other than that the patch it contains is "*completely* the wrong approach."

> The GSoC page
> (https://code.djangoproject.com/wiki/SummerOfCode2011#Enhancedauth.user)
> is a frustrating read. It goes on and on about how hard the problem is
> and how wrong your solution is, but doesn't provide any detail as to why
> it's hard or why it's wrong.

It does provide detail - it gives a list of issues you have to consider,
and tells you where to search for the other issues.

It poses a list of questions (how can we represent a generic user? how should we approach authentication vs. authorization? how do we handle migrations?) that all presuppose a grand redesign of auth.User instead of an incremental improvement, but it doesn't discuss at all why an incremental improvement won't be considered. And then it refers would-be implementers to read up on Lazy Foreign Keys (LFKs), which I contend are neither sufficient nor necessary for an auth.User refactor.

To illustrate this last point, consider some of the compatibility problems we're liable to face when replacing django.contrib.auth.models.User:

1. Parts of Django, numerous third-party apps, and countless bespoke software installations have models with FKs to User. LFKs could help here, in that we could delay binding to the precise model until that binding is known (through configuration, or whatever). But if, to use an LFK, any of the third-party apps or bespoke software has to be changed (say, from `ForeignKey(User)` to `LazyForeignKey('User')` or `ForeignKey(LazyModel('User'))`), then LFKs are not backwards compatible. They may be the place we want to end up, but they're not the place to start, as they require too much change in software we don't control.

2. Parts of Django, numerous third-party apps, and countless bespoke software installations have ModelForms built around the User model. The model is specified in the ModelForm's Meta class, and must be available at class definition time. LFKs don't help here. We could extend ModelForm to support delayed binding, but if that involves changing any existing software (say, `model = LazyModel('User')`), then, again, we've broken backwards compatibility.

3. Parts of Django, numerous third-party apps, and countless bespoke software installations import User from django.contrib.auth.models. But we're talking about allowing developers to specify their own User model, so code that imports d.c.a.m.User would no longer be valid. LFKs don't help here, either.

What's needed is a way to maintain the interface defined by the existing django.contrib.auth application (a User model in django.contrib.auth.models, an AuthenticationForm in django.contrib.auth.forms, a ModelBackend in django.contrib.auth.backends, etc) while allowing the developer to specify the implementation—whether the stock User app, a reusable third-party app, or their own.

I've written such a pluggable User app. By default, the User model works as before. But I've also provided a User model (and forms, admin, backends, etc) implementation that removes the limitations of the User model noted above and allows for email authentication. It's still a bit rough around the edges, but it works, insofar that `manage.py createsuperuser` doesn't ask for a username, `manage.py changepassword` takes an email address, and I can login to the admin interface with my email address.

It works for me, and that's well and good since I have a client that needs this functionality, but I don't want to maintain the fork forever. I would like to provide a patch for consideration for trunk, but before I waste my time and yours, I'd like to understand why the notion of pluggable user models was rejected (as indicated on the GSoC page), when Russell seems to be advocating for a design of that sort with his reference to the COMMENTS_APP setting.

And to be clear, I don't consider the pluggable user model approach that I'm taking and the Lazy Foreign Key approach to be mutually exclusive. Rather, I see my approach as something of a compatibility shim until sufficient time has elapsed to pull the plug on imports of django.contrib.auth.models.User. I propose that we continue to support the existing auth interface (defined more completely above, but for brevity, a User model at django.contrib.auth.models) but start raising a deprecation warning in the 1.4.1+ timeframe. Meanwhile, we introduce LFKs (which I have ideas about, but which, again, are not required for an incremental improvement to auth.User), so that maintainers of reusable apps can start to take advantage of delayed binding. At some point, say, 1.5, or 2.0, we drop support for the existing auth interface (which is to say, apps should not expect to find a User model at django.contrib.auth.models) and the LFK becomes the "one—and preferably only one—obvious way to do it."

Cheers,

Clay

[Jacob Kaplan-Moss]

On Sunday, March 11, 2012 at 12:39 AM, Clay McClure wrote:

I've written such a pluggable User app. By default, the User model works as before. But I've also provided a User model (and forms, admin, backends, etc) implementation that removes the limitations of the User model noted above and allows for email authentication. It's still a bit rough around the edges, but it works, insofar that `manage.py createsuperuser` doesn't ask for a username, `manage.py changepassword` takes an email address, and I can login to the admin interface with my email address.

Can you please post this code somewhere? I'd love to see it.

Jacob

[Joe & Anne Tennies]

I started a new page off the old https://code.djangoproject.com/wiki/SchemaEvolution (https://code.djangoproject.com/wiki/SchemaEvolutionDesign). It's not complete at this point, I just did a brain dump of what I remembered hearing in the past. Sounds like I'm not TOO far off with my memory.

Also, is the following the thread you are thinking of? http://groups.google.com/group/django-developers/browse_thread/thread/6239a63c9ad874b5/

Thanks in advance,
Joe Tennies

Also, Russell, I didn't know you were the one behind django-evolution. I may need to pick your brain at some point. I have a couple ideas I'd like input on, but I'm not quite ready yet. Now I wish I had tried harder to get to PyCon this year, but work was supposed to be quite hectic this week.


My goal is to

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

[Joe & Anne Tennies]

Can I ask for one change this late in the project to the 1.4 release if it's only the documentation for this bug? I'm hoping someone at PyCon might be able to "sneak this in."

Can we add a note to the EmailField documentation that states that "The default 75 character max_length is not capable of storing all RFC3696/5321 email addresses. A max_length of 254 would be capable of storing all email addresses as defined by the RFC3696/5321 specification. This is not being changed at this time to maintain backwards compatibility." I'm thinking the equivalent of <https://docs.djangoproject.com/en/1.3/ref/models/fields/> would be a good place.

I know we aren't going to change any code for this release, as there is still discussion on this, but I thought this would be a good start.

Thanks,
Joe Tennies

[Russell Keith-Magee]

Sure - that sounds like a reasonable addition to me. Open a documentation ticket, and we'll see if we can slip it in.

Yours,
Russ Magee %-)

[Russell Keith-Magee]

On 12/03/2012, at 6:58 AM, Joe & Anne Tennies wrote:

> I started a new page off the old https://code.djangoproject.com/wiki/SchemaEvolution (https://code.djangoproject.com/wiki/SchemaEvolutionDesign). It's not complete at this point, I just did a brain dump of what I remembered hearing in the past. Sounds like I'm not TOO far off with my memory.
>
> Also, is the following the thread you are thinking of? http://groups.google.com/group/django-developers/browse_thread/thread/6239a63c9ad874b5/

That's the GSoC discussion I'm thinking of; it covers the db.backends part of the discussion. There is another post by Andrew from a little earlier that also includes the 'migration accounting' part of the problem:

http://groups.google.com/group/django-developers/browse_thread/thread/bac15726fa5e9662

> Also, Russell, I didn't know you were the one behind django-evolution. I may need to pick your brain at some point. I have a couple ideas I'd like input on, but I'm not quite ready yet. Now I wish I had tried harder to get to PyCon this year, but work was supposed to be quite hectic this week.

Unfortunately, I couldn't make it to PyCon this year. However, I'm more than happy to have my brain picked by someone who is volunteering to pitch in on schema migration.

Yours,
Russ Magee %-)

[Clay McClure]

On Sun, Mar 11, 2012 at 12:09 PM, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:

I've written such a pluggable User app. By default, the User model works as before. But I've also provided a User model (and forms, admin, backends, etc) implementation that removes the limitations of the User model noted above and allows for email authentication. It's still a bit rough around the edges, but it works, insofar that `manage.py createsuperuser` doesn't ask for a username, `manage.py changepassword` takes an email address, and I can login to the admin interface with my email address.

Can you please post this code somewhere? I'd love to see it.

Sure, here's a fork of 1.4rc1 that allows for pluggable auth apps:

    https://github.com/claymation/django/tree/pluggable-auth-apps

The patch is rather large, but represents mostly the movement of files, and some refactoring of components to allow for greater reuse. For instance, the User model (and the forms, decorators, backends, and middleware that depend on it) has moved from django.contrib.auth to a new app, django.contrib.auth.default. This new app represents the default pluggable auth app, as reflected by the default value of the AUTH_APP setting (in a manner similar to the COMMENTS_APP setting).

Backwards compatibility is maintained by continuing to honor the existing import contract, e.g., `from django.contrib.auth.models import User`. User is defined at run-time to be either the default (django.contrib.auth.default.models.User), or a user-provided model, based on the value of AUTH_APP. With the exception of having to add the new django.contrib.auth.default app to INSTALLED_APPS, existing django installations should be unaffected by this change.

New django projects can elect to use an entirely different pluggable auth app, by setting AUTH_APP and adding such app to INSTALLED_APPS. Any library or bespoke code that imports User from django.contrib.auth.models will get the new model. The burden of ensuring that third-party and bespoke code is compatible with the new User model will fall on the author of that model. In other words, if I export a User model with no username field, I shouldn't expect that model to work in tandem with a blogging app that requires a username field.

To simplify the process of writing new pluggable auth apps, I've factored out some common functionality from django.contrib.auth.default and put it in django.contrib.auth.base. These base models are intended to be an aid to reuse, but are not required, and have no special properties.

As an example of a pluggable auth app, here's one I wrote that allows for email authentication:

    https://github.com/claymation/django_email_auth

The pluggable-auth-apps fork is barely two days old, so it's still rough around the edges, and I've almost certainly missed some things. I haven't written any new tests for it yet, but the existing suite passes—at least the tests that aren't skipped. I've installed all the requirements listed in the testing docs—including selenium—but I get:

----------------------------------------------------------------------

Ran 4639 tests in 438.865s

OK (skipped=72, expected failures=2)

If this approach looks like something we might want to move forward with, I'll start working on some tests and docs.

Cheers,

Clay

[Clay McClure]

On Sun, Mar 11, 2012 at 4:39 AM, Clay McClure <cl...@daemons.net> wrote:

What you are really saying is this: being pragmatic means that we

prioritise *your* immediate need above the need to keep the code and the
docs maintainable, and above the need to maintain compatibility with
existing installations.

Luke, I'm sorry for being gruff and impatient in my previous reply. I'm anxious to see this problem fixed, but I don't want to sacrifice maintainability or compatibility to do it. I hope that the community can make some progress on this issue before the next release cycle. I'm willing to devote the time to it, whether it be tweaking my patch or working on a different solution altogether.

Cheers,

Clay

[Danny Adair]

On Mon, Mar 12, 2012 at 16:57, Clay McClure <cl...@daemons.net> wrote:
>[...]

> New django projects can elect to use an entirely different pluggable auth

>[...]

> The pluggable-auth-apps fork is barely two days old, so it's still rough
> around the edges, and I've almost certainly missed some things. I haven't

>[...]

I'm sorry I don't know if I like that. I hope I understand correctly
what you're doing, and that my criticism is seen as constructive.
This sells itself as "pluggable auth" but really only provides the
originally requested auth by email, and I don't see how it could go
from there, i.e. what else would be possible (in regards to auth).
Pulling username and email out of (Base)User and then bringing one or
both of them - and the logic of which to check - back in, is not
really a pluggable auth.

It's the mixing of profile and information relevant to authentication
(and authorization - is_staff?) that's another problem of the existing
User model, and this is where this approach allows for dangerous
developments.
You've pulled those two out and said it's up to the chosen auth to
decide a) which to even include in the User model b) which is used to
authenticate and which is just "secondary" user (profile?)
information.
What about first_name and last_name? Why can't a pluggable auth dump
those from the User model? And certainly an "auth" should get to have
a say about "password".
And then... there's not much left anymore in that (Base)User model,
right... (GSoC: "reducing the user table to little more than an
identifying primary key")

Ok email auth problem solved. But now there's a simple way of
providing any User model I like, so first thing I'm going to do is to
bring in what I think the User should look like i.e. use the "auth" to
define what really is a "profile". And so will lots of projects. "The
User model needs a username for this app to work". "The User model
needs a timezone for this app to work". etc.
I'd like to see a later, "proper" auth.user that can undo that chaos.

Cheers,
Danny

[Danny Adair]

On Thu, Mar 15, 2012 at 22:53, Danny Adair <danny...@unfold.co.nz> wrote:
>[...]

> I'd like to see a later, "proper" auth.user that can undo that chaos.

I originally thought you intended a setting which let's the user
switch from the existing auth to an email auth with corresponding User
model.
As lame as that may be - at least a later, "proper" auth.user solution
would only have two known (shipped) auths to deal with.

Cheers,
Danny

[Daniel Sokolowski]

Tom makes a good point, but you can already store emails in the username,
they are just limited to 30 characters or fewer. Lift this length
restriction and I will be able to do everything I need without having to
wait for contrib.auth2. In the 'I use django to make a living' world the
solution doesn't have to be perfect just good enough. The talk of lazy
foreign keys just so we can create contrib.auth2 just rubs me the wrong way,
adds additional complexity, and I see no benefit of it over a UserProfile
model or with model inheritance.

And Django developer ought to be familiar how to do a schema migration for
this simple change, and if they fail to know instructions can be provided in
the release notes; there is NO need to have progress being stalled because
there is no built automated schema migration.

Why can we not just increase the length limit on the username field?, Why
can't we just throw a validation error if data entered is to long and the
schema has not been updated? I think the answer yes we can and easily.