Status of 3.1 release blockers.

591 views
Skip to first unread message

Mariusz Felisiak

unread,
May 6, 2020, 3:58:11 AM5/6/20
to Django developers (Contributions to Django itself)
Hi y'all,

Time to begin release process for the next major release, Django 3.1!

The 3.1 feature freeze is scheduled (according to https://code.djangoproject.com/wiki/Version3.1Roadmap) for May 11. We'll probably release the alpha a few days later.

We have a few larger patches we want to finish reviewing:

https://github.com/django/django/pull/12392 - Fixed #12990, Refs #27694 -- Added JSONField model field.
https://github.com/django/django/pull/12851 - Fixed #25236 -- Deprecated {% ifequal %} and {% ifnotequal %} template tags.
https://github.com/django/django/pull/12159 - Fixed #31034 -- Added a navigation sidebar to the admin.

Best,
Mariusz

Claude Paroz

unread,
May 8, 2020, 5:47:32 AM5/8/20
to Django developers (Contributions to Django itself)
Hi Mariusz,

I think we should also address:


as release blocker, because more and more installations will have GDAL 3 by default and the backwards compatibility issues are serious. I'll try to prepare a patch ASAP. As this is only affecting GeoDjango projects, we might make an exception to the rule and fix it between alpha and beta, depending on the time it takes to solve it.

Claude

Mariusz Felisiak

unread,
May 8, 2020, 10:40:55 AM5/8/20
to Django developers (Contributions to Django itself)
I think we should also address:


as release blocker, because more and more installations will have GDAL 3 by default and the backwards compatibility issues are serious. I'll try to prepare a patch ASAP. As this is only affecting GeoDjango projects, we might make an exception to the rule and fix it between alpha and beta, depending on the time it takes to solve it.

Agreed, I will check it on Monday.

Best,
Mariusz

Mariusz Felisiak

unread,
May 12, 2020, 5:53:05 AM5/12/20
to Django developers (Contributions to Django itself)
Hi y'all,

The feature freeze was yesterday, so we've branched stable/3.1.x today and Claude has updated the translations catalogs. We're planning to do the alpha release in this week, after fixing two confirmed regressions:


Best,
Mariusz

James Bennett

unread,
May 12, 2020, 8:33:56 PM5/12/20
to django-d...@googlegroups.com
I've been working on documentation updates to get the DEP 10
governance listed in the docs, but it's unlikely I'll be able to PR it
until this weekend. How do people feel about that also being included
in 3.1? It's not exactly a feature change, and it arguably corrects a
bug in that the docs as they exist now don't correctly describe the
way Django is governed...

Mariusz Felisiak

unread,
May 13, 2020, 12:15:00 AM5/13/20
to Django developers (Contributions to Django itself)
Hi James,

  To the first release candidate (July 20) we will backport doc fixes and even cleanups, so you have time to finish this change. Documentation fixes generally are generally more freely backported even to the last release branch.

Best,
Mariusz

Mariusz Felisiak

unread,
Jul 31, 2020, 5:28:15 AM7/31/20
to Django developers (Contributions to Django itself)
Markus reported a release blocker #31842 related with running an app on multiple servers with different versions of Django (3.0.x or 3.1). Signatures created on servers with Django 3.1 are not valid on Django 3.0, it's not only about signing.loads()/dumps but also about sessions etc. (see #27468). We have several possible approaches:
  • revert commits related with #27468,
  • change the default hashing algorithm to SHA-1,
  • add the new setting for the default hashing algorithm,
  • add the "algorithm" parameter to signing.dumps()/loads() (PR13260) and ignore the rest of the problem (it's probably not an option),
  • add the new setting "MIN_DJANGO_VERSION" and change the default hashing algorithm if it's 3.0 or less.
  • ... (ideas are welcome)
Best,
Mariusz

Markus Holtermann

unread,
Jul 31, 2020, 5:41:35 AM7/31/20
to Django developers
Thank you for summarizing our IRC discussion, Mariusz. To be clear, the problem occurs during the upgrade process where more than 1 server is involved. That might be the case in small deployments with just 2 servers, where the time of two Django versions running simultaneously is likely small, or on huge deployments of the course of days or weeks, when a staged rollout occurs.

Cheers, Markus

On Fri, Jul 31, 2020, at 11:28 AM, Mariusz Felisiak wrote:
> Markus reported a release blocker #31842
> <https://code.djangoproject.com/ticket/31842> related with running an
> app on multiple servers with different versions of Django (3.0.x or
> 3.1). Signatures created on servers with Django 3.1 are not valid on
> Django 3.0, it's not only about signing.loads()/dumps but also about
> sessions etc. (see #27468
> <https://code.djangoproject.com/ticket/27468>). We have several
> possible approaches:
> * revert commits related with #27468,
> * change the default hashing algorithm to SHA-1,
> * add the new setting for the default hashing algorithm,
> * add the "algorithm" parameter to signing.dumps()/loads() (PR13260
> <https://github.com/django/django/pull/13260>) and ignore the rest of
> the problem (it's probably not an option),
> * add the new setting "MIN_DJANGO_VERSION" and change the default
> hashing algorithm if it's 3.0 or less.
> * ... (ideas are welcome)
> Best,
> Mariusz
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/2a3d706d-6f78-406e-b7a9-3bba3ea9b7e6n%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/2a3d706d-6f78-406e-b7a9-3bba3ea9b7e6n%40googlegroups.com?utm_medium=email&utm_source=footer>.

Carlton Gibson

unread,
Jul 31, 2020, 5:47:25 AM7/31/20
to django-d...@googlegroups.com
It looks like we need to add the algorithm to the function signatures, as per the PR, but also add a (immediately deprecated) migration setting, so folks can opt-in to the new algorithm when they’re updated.

Add the new setting default to sha1.
Raise a DeprecationWarning unless it’s set to sha256 (so folks can opt-in to the future.)
Remove setting, and change default to sha256, when 4.0?

Does that sound right? (Grrr.)

Raffaele Salmaso

unread,
Jul 31, 2020, 6:09:24 AM7/31/20
to django-d...@googlegroups.com
On Fri, Jul 31, 2020 at 11:47 AM Carlton Gibson <carlton...@gmail.com> wrote:
Add the new setting default to sha1.
Raise a DeprecationWarning unless it’s set to sha256 (so folks can opt-in to the future.)
Remove setting, and change default to sha256, when 4.0?

Does that sound right? (Grrr.)
I think this just move the migration problem from 3.2 to 4.0.
What about the other way: add a migration setting set to new algorithm, so who really need sha1 can opt-in to old algorithm?

--

Carlton Gibson

unread,
Jul 31, 2020, 6:12:24 AM7/31/20
to django-d...@googlegroups.com
> I think this just move the migration problem from 3.2 to 4.0. 

My thought was the instant warning helped with that... BUT...

What about the other way: add a migration setting set to new algorithm, so who really need sha1 can opt-in to old algorithm?

Yes, perhaps: default to the future sounds better. 

אורי

unread,
Jul 31, 2020, 6:15:30 AM7/31/20
to Django developers (Contributions to Django itself)
On Fri, Jul 31, 2020 at 12:28 PM Mariusz Felisiak <felisiak...@gmail.com> wrote:
Markus reported a release blocker #31842 related with running an app on multiple servers with different versions of Django (3.0.x or 3.1).

I think it might be related to an issue I reported - #31592. Django 3.0 can't handle sessions created by Django 3.1.

Raffaele Salmaso

unread,
Jul 31, 2020, 6:24:24 AM7/31/20
to django-d...@googlegroups.com
On Fri, Jul 31, 2020 at 12:12 PM Carlton Gibson <carlton...@gmail.com> wrote:
Yes, perhaps: default to the future sounds better. 
What about:

django 3.2
* add a global settings set to sha1
* configure settings template to use sha256 so a new project will start with new algorithm
* add a warning to sha1 usage and instruction how to upgrade
django 4.0
* remove setting

Markus Holtermann

unread,
Jul 31, 2020, 6:35:04 AM7/31/20
to Django developers
No, it won't move the problem to 3.2. The problem is that 3.0 only knows about sha1. 3.1 and later know about sha1 and sha256. Meaning, any >=3.1,<4.0 version can decode and verify signed data from 4.0 and before.

Cheers

Markus
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CABgH4Jt0xXzXrFO8vAFFbF_zabF6xhFGnnTS1rJi_iWAT9fJRg%40mail.gmail.com <https://groups.google.com/d/msgid/django-developers/CABgH4Jt0xXzXrFO8vAFFbF_zabF6xhFGnnTS1rJi_iWAT9fJRg%40mail.gmail.com?utm_medium=email&utm_source=footer>.

Raffaele Salmaso

unread,
Jul 31, 2020, 6:42:41 AM7/31/20
to django-d...@googlegroups.com
On Fri, Jul 31, 2020 at 12:34 PM Markus Holtermann <in...@markusholtermann.eu> wrote:
No, it won't move the problem to 3.2. The problem is that 3.0 only knows about sha1. 3.1 and later know about sha1 and sha256. Meaning, any >=3.1,<4.0 version can decode and verify signed data from 4.0 and before.
Sorry, s/3.2/3.1/ in all emails 😰

Raffaele Salmaso

unread,
Jul 31, 2020, 6:44:40 AM7/31/20
to django-d...@googlegroups.com
On Fri, Jul 31, 2020 at 12:23 PM Raffaele Salmaso <raff...@salmaso.org> wrote:
On Fri, Jul 31, 2020 at 12:12 PM Carlton Gibson <carlton...@gmail.com> wrote:
What about:
retry 😅

django 3.1
* add a global settings set to sha1
* configure settings template to use sha256 so a new project will start with new algorithm
* add a warning to sha1 usage and instruction how to upgrade
django 3.2
* set global setting set to sha256
django 4.0
* remove setting

Carlton Gibson

unread,
Jul 31, 2020, 7:13:24 AM7/31/20
to Django developers (Contributions to Django itself)
Hi Uri. 

On 31 Jul 2020, at 12:11, ⁨אורי⁩ <⁨u...@speedy.net⁩> wrote:

On Fri, Jul 31, 2020 at 12:28 PM Mariusz Felisiak <felisiak...@gmail.com> wrote:
Markus reported a release blocker #31842related with running an app on multiple servers with different versions of Django (3.0.x or 3.1). 

I think it might be related to an issue I reported - #31592. Django 3.0 can't handle sessions created by Django 3.1.

Yes, it’s related. Your issue was downgrading IIRC. This is “can’t upgrade piecemeal” — but a solution may allow your use-case too. 

C.

אורי

unread,
Jul 31, 2020, 11:31:16 AM7/31/20
to Django developers (Contributions to Django itself)
Hi Carlton,

I think a possible solution can be if Django 3.0 will be patched to handle sessions created by 3.1. This will allow both downgrading Django and running Django on several servers with 3.0 and 3.1 in parallel. If this is too late now to do it before releasing 3.1, maybe you can postpone this change (of hashing algorithm, if I understand correctly) to Django 3.2. And then of course, patch 3.0 and 3.1 to handle sessions created by 3.2.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

Carlton Gibson

unread,
Jul 31, 2020, 11:37:41 AM7/31/20
to django-d...@googlegroups.com
Hey Uri. 

We’re not going to patch 3.0 now. (Risk of regression is too high — it’s why we have the backporting policy.) 

It’s 3.1’s job to be “forward compatible” - folks should be able to update. This is a difficult case in that it requires allowing running the old version at the same time as the new one, which isn’t something I’d like to encourage, but I agree with Simon on the issue, we need to provide a path.

Kind  regards,
Carlton

Mariusz Felisiak

unread,
Jul 31, 2020, 4:46:36 PM7/31/20
to Django developers (Contributions to Django itself)
I've created a draft PR13262  with the new setting for the default hashing algorithm.

charettes

unread,
Jul 31, 2020, 6:29:45 PM7/31/20
to Django developers (Contributions to Django itself)
I like the state of your proposed PR Mariusz. It does add one more setting but it feels like having this configurable project wide will be a plus as it will allow the ecosystem to rely on it and hopefully make audit of large Django application a bit easier when having specific hashing requirements.

Simon

אורי

unread,
Aug 1, 2020, 1:16:57 AM8/1/20
to Django developers (Contributions to Django itself)
Hi Mariusz,

How do you propose running several versions of Django 3.0 and 3.1 with this PR and setting? Does it have to be defined, or is it enough using the default setting? And when upgrading Django to 3.1, what is the default setting if this setting is not explicitly defined? And how will it affect sessions in case of downgrading back to 3.0? Will it be possible to downgrade also with the default setting?

Also, maybe it's better to document the answers to these questions in the release notes and in the definition of the new setting. And also, how to define this setting to its default value in Django versions <= 3.0. What do you think?

On Fri, Jul 31, 2020 at 11:46 PM Mariusz Felisiak <felisiak...@gmail.com> wrote:
I've created a draft PR13262  with the new setting for the default hashing algorithm.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

אורי

unread,
Aug 1, 2020, 1:21:24 AM8/1/20
to Django developers (Contributions to Django itself)
I also think, when the old hashing algorithm is deprecated, it's better to convert sessions to the new hashing algorithm without logging out users and without raising errors or exceptions. Is it possible? And notice that the Django user may change this setting only in the future, such as in Django 4.0 and even later. What will happen then?


On Fri, Jul 31, 2020 at 11:46 PM Mariusz Felisiak <felisiak...@gmail.com> wrote:
I've created a draft PR13262  with the new setting for the default hashing algorithm.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

Florian Apolloner

unread,
Aug 1, 2020, 6:53:09 AM8/1/20
to Django developers (Contributions to Django itself)
On Saturday, August 1, 2020 at 7:16:57 AM UTC+2 Uri wrote:
How do you propose running several versions of Django 3.0 and 3.1 with this PR and setting? Does it have to be defined, or is it enough using the default setting?

You have to set it to `sha1` and then when all your instances upgraded to 3.1 you will can remove it at which point all new signatures will use the new algorithm.

And when upgrading Django to 3.1, what is the default setting if this setting is not explicitly defined? And how will it affect sessions in case of downgrading back to 3.0? Will it be possible to downgrade also with the default setting?

Downgrading will __never__ be supported. This is only for the transitional period where you run 3.0 and 3.1 in tandem. Once you switched to 3.1 and changed fully to `sha256` there will be no way (never ever!) to downgrade that.

And also, how to define this setting to its default value in Django versions <= 3.0.

You don't, this is not supported.

Florian Apolloner

unread,
Aug 1, 2020, 6:58:07 AM8/1/20
to Django developers (Contributions to Django itself)
On Saturday, August 1, 2020 at 12:53:09 PM UTC+2 Florian Apolloner wrote:
And also, how to define this setting to its default value in Django versions <= 3.0.

You don't, this is not supported.

Sorry I probably didn't fully understand the question. Either way; this setting is not used (and will not be) in Django <= 3.0. You can only set it on 3.1 to `sha1` to get the pre 3.1 behavior in 3.1.

Mariusz Felisiak

unread,
Aug 3, 2020, 2:48:31 AM8/3/20
to Django developers (Contributions to Django itself)
It seems that adding DEFAULT_HASHING_ALGORITHM setting is an acceptable solution, we have one remaining question: Do we keep the setting around?
  1. we can treat DEFAULT_HASHING_ALGORITHM as a transitional setting, limit its values to `sha1` and `sha256`, and deprecate it immediately. Florian said: "From a security perspective I wonder if we should limit this that the existing sha1 and the new sha256 algos. We really don't ever want anyone to put md5 there and it is really just a transitional setting.", "...setting this to sha256, running with it and then changing it to sha512 will be a hard cut. There will be no migration path like there would be for sha1. Imo that limits the usefulness of such a setting for long term usage and makes it questionable if we want to support such usage."
  2. we can treat DEFAULT_HASHING_ALGORITHM as a new feature and allows any secure hash algorithm supported by hashlib. Simon said: "It does add one more setting but it feels like having this configurable project wide will be a plus as it will allow the ecosystem to rely on it and hopefully make audit of large Django application a bit easier when having specific hashing requirements.".
Personally I think we should choose the first option because it's safer.

We're delaying the 3.1 release until tomorrow.

Best,
Mariusz

Markus Holtermann

unread,
Aug 3, 2020, 3:48:39 AM8/3/20
to Django developers
Can we come up with a way that we have the settings variable around for now, only to allow transitioning, and then add it as a proper feature in 3.2.

As for the feature, I think we could choose a path like passlib does: A list of 2 (n>=1) algorithms. The first one will be used for signing, and all of them for verification. We could then limit the set of options to sensible algorithms (or put some algorithms on a blocklist, such as md5). But I think that's nothing we should add the day before the final release.

Cheers,

Markus

On Mon, Aug 3, 2020, at 8:48 AM, Mariusz Felisiak wrote:
> It seems that adding DEFAULT_HASHING_ALGORITHM setting is an acceptable
> solution, we have one remaining question: Do we keep the setting around?
> 1. we can treat DEFAULT_HASHING_ALGORITHM as a transitional setting,
> limit its values to `sha1` and `sha256`, and deprecate it immediately.
> Florian said: *"From a security perspective I wonder if we should limit
> this that the existing sha1 and the new sha256 algos. We really don't
> ever want anyone to put md5 there and it is really just a transitional
> setting.", "...setting this to sha256, running with it and then
> changing it to sha512 will be a hard cut. There will be no migration
> path like there would be for sha1. Imo that limits the usefulness of
> such a setting for long term usage and makes it questionable if we want
> to support such usage."*
> 2. we can treat DEFAULT_HASHING_ALGORITHM as a new feature and allows
> any secure hash algorithm supported by hashlib. Simon said: *"**It does
> add one more setting but it feels like having this configurable project
> wide will be a plus as it will allow the ecosystem to rely on it and
> hopefully make audit of large Django application a bit easier when
> having specific hashing requirements.".*
> Personally I think we should choose the first option because it's
> safer.
>
> *We're delaying the 3.1 release until tomorrow.*
>
> Best,
> Mariusz
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/023a0518-a58c-48c6-8b36-9d5fb7a7c108n%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/023a0518-a58c-48c6-8b36-9d5fb7a7c108n%40googlegroups.com?utm_medium=email&utm_source=footer>.

Mariusz Felisiak

unread,
Aug 3, 2020, 4:25:32 AM8/3/20
to Django developers (Contributions to Django itself)
Hi Markus,

    I don't see a clear path to do this, we cannot add a setting that accepts a string and immediately (in the next release) change it to a list, this will require a deprecation path etc. and will be really confusing to users. IMO we can reconsider this new feature in the next release with a different setting.

Best,
Mariusz

אורי

unread,
Aug 5, 2020, 3:53:56 AM8/5/20
to Django developers (Contributions to Django itself)
Hi,

Have you tested multiple servers with Django versions 3.0 and 3.1 with `DEFAULT_HASHING_ALGORITHM = 'sha1'`? I created a branch with this setting, with Django 3.1, and then I used it on our staging server. After I signed up and logged in, I reverted to Django 3.0, and then one user was logged out (who should be logged in) and another user got the exception `Incorrect padding` (from `base64.b64decode(session_data.encode('ascii'))`). It looks like the setting I created  `DEFAULT_HASHING_ALGORITHM = 'sha1'` is ignored with Django 3.1. I know you don't support downgrading Django, but this might also affect users who run several servers in parallel, with Django 3.0 and 3.1.

When I downgraded I rolled back 2 migrations - one I created myself (match_accounts.0008_auto_20200804_1453) and (auth.0012_alter_user_first_name_max_length). But rolling back these migrations looks like it's not related to this issue.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

Mariusz Felisiak

unread,
Aug 7, 2020, 1:24:29 AM8/7/20
to Django developers (Contributions to Django itself)
> After I signed up and logged in, I reverted to Django 3.0, and then one user was logged out (who should be logged in) and another user got the exception `Incorrect padding` (from `base64.b64decode(session_data.encode('ascii'))`).

This is a separate issue, see #31864.

אורי

unread,
Aug 7, 2020, 2:32:02 AM8/7/20
to Django developers (Contributions to Django itself)
Thank you.

On Fri, Aug 7, 2020 at 8:24 AM Mariusz Felisiak <felisiak...@gmail.com> wrote:
> After I signed up and logged in, I reverted to Django 3.0, and then one user was logged out (who should be logged in) and another user got the exception `Incorrect padding` (from `base64.b64decode(session_data.encode('ascii'))`).

This is a separate issue, see #31864.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages