[Django] #36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate submissions

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake | Owner: Sarah Boyce
Howard |
Type: | Status: assigned
Cleanup/optimization |
Component: Forms | Version: 6.0
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
When a ChoiceField / MultipleChoiceField has 5 possible choices, but the
form submits 25 values, the `choices` values are compared once per
submitted value. If the submitted values are duplicates, the validation
doesn't terminate early, but can still spend a lot of time unnecessarily
validating values. This can be very slow when large (~30k) numbers of
values are submitted.

A suggested fix is to only validate the unique submitted values (for
example `for val in set(value)`).

This issue was reported to the Security Team, but deemed not a security
issue due to the minimal impact when given reasonable input (in the bounds
of the security policy).
--
Ticket URL: <https://code.djangoproject.com/ticket/36913>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------

Reporter: Jake Howard | Owner: Sarah
Type: | Boyce
Cleanup/optimization | Status: assigned
Component: Forms | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jake Howard):

Note that validation that submissions (and choices) are unique is being
handled in a separate feature request: https://github.com/django/new-
features/issues/121
--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:1>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake Howard | Owner: Sarah
Type: | Boyce
Cleanup/optimization | Status: assigned
Component: Forms | Version: 6.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* stage: Unreviewed => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:2>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake Howard | Owner: Sarah
Type: | Boyce
Cleanup/optimization | Status: assigned
Component: Forms | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Abhimanyu Singh Negi):

I’d like to work on this issue.

From what I understand the slowdown happens because duplicate submitted
values are validated again and again. I’m thinking of validating only the
unique values internally while keeping the original list unchanged so
behaviour stays the same. I’ll add some tests too to make sure duplicates
and invalid values are still handled correctly.

If this sounds reasonable, i would like to assign the issue to myself.
--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:3>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake Howard | Owner: Sarah
Type: | Boyce
Cleanup/optimization | Status: assigned
Component: Forms | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Abhimanyu Singh Negi):

I’d like to work on this issue.

From what I understand the slowdown happens because duplicate submitted
values are validated again and again. I’m thinking of validating only the
unique values internally while keeping the original list unchanged so
behaviour stays the same. I’ll add some tests too to make sure duplicates
and invalid values are still handled correctly.

If this sounds reasonable and no one else is working on it, i would like

to assign the issue to myself.
--

Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:4>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake Howard | Owner: Sarah
Type: | Boyce
Cleanup/optimization | Status: assigned
Component: Forms | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Natalia Bidart):

Replying to [comment:3 Abhimanyu Singh Negi]:

> I’d like to work on this issue.
>
> From what I understand the slowdown happens because duplicate submitted
values are validated again and again. I’m thinking of validating only the
unique values internally while keeping the original list unchanged so
behaviour stays the same. I’ll add some tests too to make sure duplicates
and invalid values are still handled correctly.
>

> If this sounds reasonable and no one is working on this, i would like to

assign the issue to myself.

This ticket has been assigned already and has been reserved to evaluate
under the Djangonaut Space umbrella. Sorry!
--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:5>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake Howard | Owner: Sarah
Type: | Boyce
Cleanup/optimization | Status: assigned
Component: Forms | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by anjaniacatus):

Hello, I am Aina (Team Saturn) from Djangonaut Space. I will be looking at
this ticket
--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:6>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------

Reporter: Jake Howard | Owner: Afenomamy
Type: | Status: assigned
Cleanup/optimization |

Component: Forms | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Afenomamy):

* owner: Sarah Boyce => Afenomamy

--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:7>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake Howard | Owner: Afenomamy
Type: | Status: assigned
Cleanup/optimization |
Component: Forms | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Afenomamy):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:8>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake Howard | Owner: Afenomamy
Type: | Status: assigned
Cleanup/optimization |
Component: Forms | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Description changed by Afenomamy:

Old description:

> When a ChoiceField / MultipleChoiceField has 5 possible choices, but the
> form submits 25 values, the `choices` values are compared once per
> submitted value. If the submitted values are duplicates, the validation
> doesn't terminate early, but can still spend a lot of time unnecessarily
> validating values. This can be very slow when large (~30k) numbers of
> values are submitted.
>
> A suggested fix is to only validate the unique submitted values (for
> example `for val in set(value)`).
>
> This issue was reported to the Security Team, but deemed not a security
> issue due to the minimal impact when given reasonable input (in the
> bounds of the security policy).

New description:

When a ChoiceField / MultipleChoiceField has 5 possible choices, but the
form submits 25 values, the `choices` values are compared once per
submitted value. If the submitted values are duplicates, the validation
doesn't terminate early, but can still spend a lot of time unnecessarily
validating values. This can be very slow when large (~30k) numbers of
values are submitted.

A suggested fix is to only validate the unique submitted values (for
example `for val in set(value)`).

This issue was reported to the Security Team, but deemed not a security
issue due to the minimal impact when given reasonable input (in the bounds
of the security policy).

sugessted PR : https://github.com/django/django/pull/20960

--
--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:9>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake Howard | Owner: Afenomamy
Type: | Status: assigned
Cleanup/optimization |
Component: Forms | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Old description:

> When a ChoiceField / MultipleChoiceField has 5 possible choices, but the
> form submits 25 values, the `choices` values are compared once per
> submitted value. If the submitted values are duplicates, the validation
> doesn't terminate early, but can still spend a lot of time unnecessarily
> validating values. This can be very slow when large (~30k) numbers of
> values are submitted.
>
> A suggested fix is to only validate the unique submitted values (for
> example `for val in set(value)`).
>
> This issue was reported to the Security Team, but deemed not a security
> issue due to the minimal impact when given reasonable input (in the
> bounds of the security policy).
>

> sugessted PR : https://github.com/django/django/pull/20960

New description:

When a ChoiceField / MultipleChoiceField has 5 possible choices, but the
form submits 25 values, the `choices` values are compared once per
submitted value. If the submitted values are duplicates, the validation
doesn't terminate early, but can still spend a lot of time unnecessarily
validating values. This can be very slow when large (~30k) numbers of
values are submitted.

A suggested fix is to only validate the unique submitted values (for
example `for val in set(value)`).

This issue was reported to the Security Team, but deemed not a security
issue due to the minimal impact when given reasonable input (in the bounds
of the security policy).

--
Comment (by Jake Howard):

[https://github.com/django/django/pull/20960 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:10>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake Howard | Owner: Afenomamy
Type: | Status: assigned
Cleanup/optimization |
Component: Forms | Version: 6.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jake Howard):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:11>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake Howard | Owner: Afenomamy

Type: | Status: closed

Cleanup/optimization |
Component: Forms | Version: 6.0

Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls <jacobtylerwalls@…>):

* resolution: => fixed
* status: assigned => closed

Comment:

In [changeset:"26f8929f16c514bd9967cd78d8dcd7760b82cc92" 26f8929]:
{{{#!CommitTicketReference repository=""
revision="26f8929f16c514bd9967cd78d8dcd7760b82cc92"
Fixed #36913 -- Optimized MultipleChoiceField.validate().
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:12>