[Django] #36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate submissions

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake | Owner: Sarah Boyce
Howard |
Type: | Status: assigned
Cleanup/optimization |
Component: Forms | Version: 6.0
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
When a ChoiceField / MultipleChoiceField has 5 possible choices, but the
form submits 25 values, the `choices` values are compared once per
submitted value. If the submitted values are duplicates, the validation
doesn't terminate early, but can still spend a lot of time unnecessarily
validating values. This can be very slow when large (~30k) numbers of
values are submitted.

A suggested fix is to only validate the unique submitted values (for
example `for val in set(value)`).

This issue was reported to the Security Team, but deemed not a security
issue due to the minimal impact when given reasonable input (in the bounds
of the security policy).
--
Ticket URL: <https://code.djangoproject.com/ticket/36913>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------

Reporter: Jake Howard | Owner: Sarah
Type: | Boyce
Cleanup/optimization | Status: assigned
Component: Forms | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jake Howard):

Note that validation that submissions (and choices) are unique is being
handled in a separate feature request: https://github.com/django/new-
features/issues/121
--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:1>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake Howard | Owner: Sarah
Type: | Boyce
Cleanup/optimization | Status: assigned
Component: Forms | Version: 6.0
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* stage: Unreviewed => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:2>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake Howard | Owner: Sarah
Type: | Boyce
Cleanup/optimization | Status: assigned
Component: Forms | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Abhimanyu Singh Negi):

I’d like to work on this issue.

From what I understand the slowdown happens because duplicate submitted
values are validated again and again. I’m thinking of validating only the
unique values internally while keeping the original list unchanged so
behaviour stays the same. I’ll add some tests too to make sure duplicates
and invalid values are still handled correctly.

If this sounds reasonable, i would like to assign the issue to myself.
--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:3>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake Howard | Owner: Sarah
Type: | Boyce
Cleanup/optimization | Status: assigned
Component: Forms | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Abhimanyu Singh Negi):

I’d like to work on this issue.

From what I understand the slowdown happens because duplicate submitted
values are validated again and again. I’m thinking of validating only the
unique values internally while keeping the original list unchanged so
behaviour stays the same. I’ll add some tests too to make sure duplicates
and invalid values are still handled correctly.

If this sounds reasonable and no one else is working on it, i would like

to assign the issue to myself.
--

Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:4>

[Django]

#36913: Optimise ChoiceField / MultipleChoiceField handling of duplicate
submissions
-------------------------------------+-------------------------------------
Reporter: Jake Howard | Owner: Sarah
Type: | Boyce
Cleanup/optimization | Status: assigned
Component: Forms | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Natalia Bidart):

Replying to [comment:3 Abhimanyu Singh Negi]:

> I’d like to work on this issue.
>
> From what I understand the slowdown happens because duplicate submitted
values are validated again and again. I’m thinking of validating only the
unique values internally while keeping the original list unchanged so
behaviour stays the same. I’ll add some tests too to make sure duplicates
and invalid values are still handled correctly.
>

> If this sounds reasonable and no one is working on this, i would like to

assign the issue to myself.

This ticket has been assigned already and has been reserved to evaluate
under the Djangonaut Space umbrella. Sorry!
--
Ticket URL: <https://code.djangoproject.com/ticket/36913#comment:5>