Status of 3.1 release blockers.

[Mariusz Felisiak]

Hi y'all,

Time to begin release process for the next major release, Django 3.1!

The 3.1 feature freeze is scheduled (according to https://code.djangoproject.com/wiki/Version3.1Roadmap) for May 11. We'll probably release the alpha a few days later.

We have a few larger patches we want to finish reviewing:

https://github.com/django/django/pull/12392 - Fixed #12990, Refs #27694 -- Added JSONField model field.

https://github.com/django/django/pull/12851 - Fixed #25236 -- Deprecated {% ifequal %} and {% ifnotequal %} template tags.

https://github.com/django/django/pull/12159 - Fixed #31034 -- Added a navigation sidebar to the admin.

Best,
Mariusz

[Claude Paroz]

Hi Mariusz,

I think we should also address:

https://code.djangoproject.com/ticket/30678 - GDAL 3 support

as release blocker, because more and more installations will have GDAL 3 by default and the backwards compatibility issues are serious. I'll try to prepare a patch ASAP. As this is only affecting GeoDjango projects, we might make an exception to the rule and fix it between alpha and beta, depending on the time it takes to solve it.

Claude

[Mariusz Felisiak]

I think we should also address:

https://code.djangoproject.com/ticket/30678 - GDAL 3 support

as release blocker, because more and more installations will have GDAL 3 by default and the backwards compatibility issues are serious. I'll try to prepare a patch ASAP. As this is only affecting GeoDjango projects, we might make an exception to the rule and fix it between alpha and beta, depending on the time it takes to solve it.

Agreed, I will check it on Monday.

Best,

Mariusz

[Mariusz Felisiak]

Hi y'all,

The feature freeze was yesterday, so we've branched stable/3.1.x today and Claude has updated the translations catalogs. We're planning to do the alpha release in this week, after fixing two confirmed regressions:

https://code.djangoproject.com/ticket/31566

https://code.djangoproject.com/ticket/31568

Best,

Mariusz

[James Bennett]

I've been working on documentation updates to get the DEP 10
governance listed in the docs, but it's unlikely I'll be able to PR it
until this weekend. How do people feel about that also being included
in 3.1? It's not exactly a feature change, and it arguably corrects a
bug in that the docs as they exist now don't correctly describe the
way Django is governed...

[Mariusz Felisiak]

Hi James,

  To the first release candidate (July 20) we will backport doc fixes and even cleanups, so you have time to finish this change. Documentation fixes generally are generally more freely backported even to the last release branch.

Best,
Mariusz

[Mariusz Felisiak]

Markus reported a release blocker #31842 related with running an app on multiple servers with different versions of Django (3.0.x or 3.1). Signatures created on servers with Django 3.1 are not valid on Django 3.0, it's not only about signing.loads()/dumps but also about sessions etc. (see #27468). We have several possible approaches:

• revert commits related with #27468,
• change the default hashing algorithm to SHA-1,
• add the new setting for the default hashing algorithm,
• add the "algorithm" parameter to signing.dumps()/loads() (PR13260) and ignore the rest of the problem (it's probably not an option),
• add the new setting "MIN_DJANGO_VERSION" and change the default hashing algorithm if it's 3.0 or less.
• ... (ideas are welcome)
  

Best,
Mariusz

[Markus Holtermann]

Thank you for summarizing our IRC discussion, Mariusz. To be clear, the problem occurs during the upgrade process where more than 1 server is involved. That might be the case in small deployments with just 2 servers, where the time of two Django versions running simultaneously is likely small, or on huge deployments of the course of days or weeks, when a staged rollout occurs.

Cheers, Markus

On Fri, Jul 31, 2020, at 11:28 AM, Mariusz Felisiak wrote:
> Markus reported a release blocker #31842

> <https://code.djangoproject.com/ticket/31842> related with running an

> app on multiple servers with different versions of Django (3.0.x or
> 3.1). Signatures created on servers with Django 3.1 are not valid on
> Django 3.0, it's not only about signing.loads()/dumps but also about
> sessions etc. (see #27468

> <https://code.djangoproject.com/ticket/27468>). We have several
> possible approaches:
> * revert commits related with #27468,
> * change the default hashing algorithm to SHA-1,
> * add the new setting for the default hashing algorithm,
> * add the "algorithm" parameter to signing.dumps()/loads() (PR13260
> <https://github.com/django/django/pull/13260>) and ignore the rest of

> the problem (it's probably not an option),

> * add the new setting "MIN_DJANGO_VERSION" and change the default

> hashing algorithm if it's 3.0 or less.

> * ... (ideas are welcome)
> Best,
> Mariusz
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/2a3d706d-6f78-406e-b7a9-3bba3ea9b7e6n%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/2a3d706d-6f78-406e-b7a9-3bba3ea9b7e6n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Carlton Gibson]

It looks like we need to add the algorithm to the function signatures, as per the PR, but also add a (immediately deprecated) migration setting, so folks can opt-in to the new algorithm when they’re updated.

Add the new setting default to sha1.

Raise a DeprecationWarning unless it’s set to sha256 (so folks can opt-in to the future.)

Remove setting, and change default to sha256, when 4.0?

Does that sound right? (Grrr.)

[Raffaele Salmaso]

On Fri, Jul 31, 2020 at 11:47 AM Carlton Gibson <carlton...@gmail.com> wrote:

Add the new setting default to sha1.

Raise a DeprecationWarning unless it’s set to sha256 (so folks can opt-in to the future.)

Remove setting, and change default to sha256, when 4.0?

Does that sound right? (Grrr.)

I think this just move the migration problem from 3.2 to 4.0.

What about the other way: add a migration setting set to new algorithm, so who really need sha1 can opt-in to old algorithm?

--

| Raffaele Salmaso
| https://salmaso.org
| https://bitbucket.org/rsalmaso
| https://github.com/rsalmaso

[Carlton Gibson]

> I think this just move the migration problem from 3.2 to 4.0. 

My thought was the instant warning helped with that... BUT...

What about the other way: add a migration setting set to new algorithm, so who really need sha1 can opt-in to old algorithm?

Yes, perhaps: default to the future sounds better. 

[אורי‎]

On Fri, Jul 31, 2020 at 12:28 PM Mariusz Felisiak <felisiak...@gmail.com> wrote:

Markus reported a release blocker #31842 related with running an app on multiple servers with different versions of Django (3.0.x or 3.1).

I think it might be related to an issue I reported - #31592. Django 3.0 can't handle sessions created by Django 3.1.

אורי

u...@speedy.net

[Raffaele Salmaso]

On Fri, Jul 31, 2020 at 12:12 PM Carlton Gibson <carlton...@gmail.com> wrote:

Yes, perhaps: default to the future sounds better. 

What about:

django 3.2

* add a global settings set to sha1

* configure settings template to use sha256 so a new project will start with new algorithm

* add a warning to sha1 usage and instruction how to upgrade

django 4.0

* remove setting

[Markus Holtermann]

No, it won't move the problem to 3.2. The problem is that 3.0 only knows about sha1. 3.1 and later know about sha1 and sha256. Meaning, any >=3.1,<4.0 version can decode and verify signed data from 4.0 and before.

Cheers

Markus

> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-developers/CABgH4Jt0xXzXrFO8vAFFbF_zabF6xhFGnnTS1rJi_iWAT9fJRg%40mail.gmail.com <https://groups.google.com/d/msgid/django-developers/CABgH4Jt0xXzXrFO8vAFFbF_zabF6xhFGnnTS1rJi_iWAT9fJRg%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Raffaele Salmaso]

On Fri, Jul 31, 2020 at 12:34 PM Markus Holtermann <in...@markusholtermann.eu> wrote:

No, it won't move the problem to 3.2. The problem is that 3.0 only knows about sha1. 3.1 and later know about sha1 and sha256. Meaning, any >=3.1,<4.0 version can decode and verify signed data from 4.0 and before.

Sorry, s/3.2/3.1/ in all emails 😰

[Raffaele Salmaso]

On Fri, Jul 31, 2020 at 12:23 PM Raffaele Salmaso <raff...@salmaso.org> wrote:

On Fri, Jul 31, 2020 at 12:12 PM Carlton Gibson <carlton...@gmail.com> wrote:

What about:

retry 😅

django 3.1

* add a global settings set to sha1

* configure settings template to use sha256 so a new project will start with new algorithm

* add a warning to sha1 usage and instruction how to upgrade

django 3.2

* set global setting set to sha256

django 4.0

* remove setting

[Carlton Gibson]

Hi Uri. 

On 31 Jul 2020, at 12:11, ⁨אורי⁩ <⁨u...@speedy.net⁩> wrote:

On Fri, Jul 31, 2020 at 12:28 PM Mariusz Felisiak <felisiak...@gmail.com> wrote:

Markus reported a release blocker #31842related with running an app on multiple servers with different versions of Django (3.0.x or 3.1). 

I think it might be related to an issue I reported - #31592. Django 3.0 can't handle sessions created by Django 3.1.

Yes, it’s related. Your issue was downgrading IIRC. This is “can’t upgrade piecemeal” — but a solution may allow your use-case too. 

C.

[אורי‎]

Hi Carlton,

I think a possible solution can be if Django 3.0 will be patched to handle sessions created by 3.1. This will allow both downgrading Django and running Django on several servers with 3.0 and 3.1 in parallel. If this is too late now to do it before releasing 3.1, maybe you can postpone this change (of hashing algorithm, if I understand correctly) to Django 3.2. And then of course, patch 3.0 and 3.1 to handle sessions created by 3.2.

אורי

u...@speedy.net

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/AEC8D4F2-E042-4A6B-9E9D-7D0584BB42B4%40gmail.com.

[Carlton Gibson]

Hey Uri. 

We’re not going to patch 3.0 now. (Risk of regression is too high — it’s why we have the backporting policy.) 

It’s 3.1’s job to be “forward compatible” - folks should be able to update. This is a difficult case in that it requires allowing running the old version at the same time as the new one, which isn’t something I’d like to encourage, but I agree with Simon on the issue, we need to provide a path.

Kind  regards,

Carlton

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CABD5YeEVhj2T0Yq0q%3DESvKjEHfXddiTNtgo_M%2B%2BxDvSKs5xRYw%40mail.gmail.com.

[Mariusz Felisiak]

I've created a draft PR13262  with the new setting for the default hashing algorithm.

[charettes]

I like the state of your proposed PR Mariusz. It does add one more setting but it feels like having this configurable project wide will be a plus as it will allow the ecosystem to rely on it and hopefully make audit of large Django application a bit easier when having specific hashing requirements.

Simon

[אורי‎]

Hi Mariusz,

How do you propose running several versions of Django 3.0 and 3.1 with this PR and setting? Does it have to be defined, or is it enough using the default setting? And when upgrading Django to 3.1, what is the default setting if this setting is not explicitly defined? And how will it affect sessions in case of downgrading back to 3.0? Will it be possible to downgrade also with the default setting?

Also, maybe it's better to document the answers to these questions in the release notes and in the definition of the new setting. And also, how to define this setting to its default value in Django versions <= 3.0. What do you think?

אורי

u...@speedy.net

On Fri, Jul 31, 2020 at 11:46 PM Mariusz Felisiak <felisiak...@gmail.com> wrote:

I've created a draft PR13262  with the new setting for the default hashing algorithm.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/c3b3025f-3830-464a-8c8c-883ec606720an%40googlegroups.com.

[אורי‎]

I also think, when the old hashing algorithm is deprecated, it's better to convert sessions to the new hashing algorithm without logging out users and without raising errors or exceptions. Is it possible? And notice that the Django user may change this setting only in the future, such as in Django 4.0 and even later. What will happen then?

אורי

u...@speedy.net

On Fri, Jul 31, 2020 at 11:46 PM Mariusz Felisiak <felisiak...@gmail.com> wrote:

I've created a draft PR13262  with the new setting for the default hashing algorithm.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/c3b3025f-3830-464a-8c8c-883ec606720an%40googlegroups.com.

[Florian Apolloner]

On Saturday, August 1, 2020 at 7:16:57 AM UTC+2 Uri wrote:

How do you propose running several versions of Django 3.0 and 3.1 with this PR and setting? Does it have to be defined, or is it enough using the default setting?

You have to set it to `sha1` and then when all your instances upgraded to 3.1 you will can remove it at which point all new signatures will use the new algorithm.

And when upgrading Django to 3.1, what is the default setting if this setting is not explicitly defined? And how will it affect sessions in case of downgrading back to 3.0? Will it be possible to downgrade also with the default setting?

Downgrading will __never__ be supported. This is only for the transitional period where you run 3.0 and 3.1 in tandem. Once you switched to 3.1 and changed fully to `sha256` there will be no way (never ever!) to downgrade that.

And also, how to define this setting to its default value in Django versions <= 3.0.

You don't, this is not supported.

[Florian Apolloner]

On Saturday, August 1, 2020 at 12:53:09 PM UTC+2 Florian Apolloner wrote:

And also, how to define this setting to its default value in Django versions <= 3.0.

You don't, this is not supported.

Sorry I probably didn't fully understand the question. Either way; this setting is not used (and will not be) in Django <= 3.0. You can only set it on 3.1 to `sha1` to get the pre 3.1 behavior in 3.1.

[Mariusz Felisiak]

It seems that adding DEFAULT_HASHING_ALGORITHM setting is an acceptable solution, we have one remaining question: Do we keep the setting around?

1. we can treat DEFAULT_HASHING_ALGORITHM as a transitional setting, limit its values to `sha1` and `sha256`, and deprecate it immediately. Florian said: "From a security perspective I wonder if we should limit this that the existing sha1 and the new sha256 algos. We really don't ever want anyone to put md5 there and it is really just a transitional setting.", "...setting this to sha256, running with it and then changing it to sha512 will be a hard cut. There will be no migration path like there would be for sha1. Imo that limits the usefulness of such a setting for long term usage and makes it questionable if we want to support such usage."
2. we can treat DEFAULT_HASHING_ALGORITHM as a new feature and allows any secure hash algorithm supported by hashlib. Simon said: "It does add one more setting but it feels like having this configurable project wide will be a plus as it will allow the ecosystem to rely on it and hopefully make audit of large Django application a bit easier when having specific hashing requirements.".

Personally I think we should choose the first option because it's safer.

We're delaying the 3.1 release until tomorrow.

Best,
Mariusz

[Markus Holtermann]

Can we come up with a way that we have the settings variable around for now, only to allow transitioning, and then add it as a proper feature in 3.2.

As for the feature, I think we could choose a path like passlib does: A list of 2 (n>=1) algorithms. The first one will be used for signing, and all of them for verification. We could then limit the set of options to sensible algorithms (or put some algorithms on a blocklist, such as md5). But I think that's nothing we should add the day before the final release.

Cheers,

Markus

On Mon, Aug 3, 2020, at 8:48 AM, Mariusz Felisiak wrote:
> It seems that adding DEFAULT_HASHING_ALGORITHM setting is an acceptable
> solution, we have one remaining question: Do we keep the setting around?

> 1. we can treat DEFAULT_HASHING_ALGORITHM as a transitional setting,

> limit its values to `sha1` and `sha256`, and deprecate it immediately.

> Florian said: *"From a security perspective I wonder if we should limit

> this that the existing sha1 and the new sha256 algos. We really don't
> ever want anyone to put md5 there and it is really just a transitional
> setting.", "...setting this to sha256, running with it and then
> changing it to sha512 will be a hard cut. There will be no migration
> path like there would be for sha1. Imo that limits the usefulness of
> such a setting for long term usage and makes it questionable if we want

> to support such usage."*
> 2. we can treat DEFAULT_HASHING_ALGORITHM as a new feature and allows
> any secure hash algorithm supported by hashlib. Simon said: *"**It does

> add one more setting but it feels like having this configurable project
> wide will be a plus as it will allow the ecosystem to rely on it and
> hopefully make audit of large Django application a bit easier when

> having specific hashing requirements.".*

> Personally I think we should choose the first option because it's
> safer.
>

> *We're delaying the 3.1 release until tomorrow.*
>
> Best,
> Mariusz

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-developers/023a0518-a58c-48c6-8b36-9d5fb7a7c108n%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/023a0518-a58c-48c6-8b36-9d5fb7a7c108n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Mariusz Felisiak]

Hi Markus,

    I don't see a clear path to do this, we cannot add a setting that accepts a string and immediately (in the next release) change it to a list, this will require a deprecation path etc. and will be really confusing to users. IMO we can reconsider this new feature in the next release with a different setting.

Best,

Mariusz

[אורי‎]

Hi,

Have you tested multiple servers with Django versions 3.0 and 3.1 with `DEFAULT_HASHING_ALGORITHM = 'sha1'`? I created a branch with this setting, with Django 3.1, and then I used it on our staging server. After I signed up and logged in, I reverted to Django 3.0, and then one user was logged out (who should be logged in) and another user got the exception `Incorrect padding` (from `base64.b64decode(session_data.encode('ascii'))`). It looks like the setting I created  `DEFAULT_HASHING_ALGORITHM = 'sha1'` is ignored with Django 3.1. I know you don't support downgrading Django, but this might also affect users who run several servers in parallel, with Django 3.0 and 3.1.

When I downgraded I rolled back 2 migrations - one I created myself (match_accounts.0008_auto_20200804_1453) and (auth.0012_alter_user_first_name_max_length). But rolling back these migrations looks like it's not related to this issue.

אורי

u...@speedy.net

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/77dfa750-21c2-42ef-a69c-077217746098n%40googlegroups.com.

[Mariusz Felisiak]

> After I signed up and logged in, I reverted to Django 3.0, and then one user was logged out (who should be logged in) and another user got the exception `Incorrect padding` (from `base64.b64decode(session_data.encode('ascii'))`).

This is a separate issue, see #31864.

[אורי‎]

Thank you.

אורי

u...@speedy.net

On Fri, Aug 7, 2020 at 8:24 AM Mariusz Felisiak <felisiak...@gmail.com> wrote:

> After I signed up and logged in, I reverted to Django 3.0, and then one user was logged out (who should be logged in) and another user got the exception `Incorrect padding` (from `base64.b64decode(session_data.encode('ascii'))`).

This is a separate issue, see #31864.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d220613e-e717-4518-b204-14d5c2b2ac47n%40googlegroups.com.