[Django] #37126: Make Task and TaskResult comparable

[Django]

#37126: Make Task and TaskResult comparable
--------------------------------+---------------------------------------
Reporter: Johannes Maron | Type: New feature
Status: new | Component: Tasks
Version: dev | Severity: Normal
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
--------------------------------+---------------------------------------
[https://forum.djangoproject.com/t/tasks-framework-versatility-
performance/45035/4 Jake and I have been discussing changes to the Task
framework for 6.2 with a focus on performance and versatility.]

Python's [https://docs.python.org/3/library/queue.html#queue.PriorityQueue
queue.PriorityQueue] implementation requires objects to be comparable.

Since the base implementation of a `Task` implements a priority, it only
makes sense to provide basic comparability based on the priority and date.

Dataclasses have a neat `order=True` attribute to make this stupidly
simple.
--
Ticket URL: <https://code.djangoproject.com/ticket/37126>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#37126: Make Task and TaskResult comparable
--------------------------------+--------------------------------------

Reporter: Johannes Maron | Owner: (none)

Type: New feature | Status: new
Component: Tasks | Version: dev

Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
--------------------------------+--------------------------------------

Comment (by Johannes Maron):

I would also suggest adding a custom __eq__ method. We only need to
include `id` and a simple string comparison is much quicker. Benchmarks N
= 100_000_000:


{{{
Original == (equal) 18.614s (186.1 ns/call)
Original == (not equal) 7.452s (74.5 ns/call)
Id-only == (equal) 4.434s (44.3 ns/call)
Id-only == (not equal) 4.423s (44.2 ns/call)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/37126#comment:1>

[Django]

#37126: Make Task and TaskResult comparable
--------------------------------+--------------------------------------
Reporter: Johannes Maron | Owner: (none)
Type: New feature | Status: new
Component: Tasks | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
--------------------------------+--------------------------------------
Comment (by Johannes Maron):

Missing benchmark sources for my previous comment:

{{{
import timeit
from dataclasses import dataclass, field
from datetime import datetime
from typing import Any

from django.utils.json import normalize_json
from django.tasks.base import Task, TaskResult, TaskResultStatus

# --- Real Task instance ---

def my_func():
pass

real_task = Task.__new__(Task)
object.__setattr__(real_task, "func", my_func)
object.__setattr__(real_task, "priority", 0)
object.__setattr__(real_task, "backend", "default")
object.__setattr__(real_task, "queue_name", "default")
object.__setattr__(real_task, "run_after", None)
object.__setattr__(real_task, "takes_context", False)

# --- Shared kwargs ---

now = datetime.now()

common_kwargs = dict(
task=real_task,
id="abc123",
status=TaskResultStatus.SUCCESSFUL,
enqueued_at=now,
started_at=now,
finished_at=now,
last_attempted_at=now,
args=[],
kwargs={},
backend="default",
errors=[],
worker_ids=["worker-1"],
)

# --- id-only equality variant ---

@dataclass(frozen=True, slots=True, kw_only=True)
class TaskResultIdEq:
task: Any = field(compare=False)
id: str
status: Any = field(compare=False)
enqueued_at: datetime | None = field(compare=False)
started_at: datetime | None = field(compare=False)
finished_at: datetime | None = field(compare=False)
last_attempted_at: datetime | None = field(compare=False)
args: list[Any] = field(compare=False)
kwargs: dict[str, Any] = field(compare=False)
backend: str = field(compare=False)
errors: list = field(compare=False)
worker_ids: list[str] = field(compare=False)
_return_value: Any | None = field(init=False, default=None,
compare=False)

def __post_init__(self):
object.__setattr__(self, "args", normalize_json(self.args))
object.__setattr__(self, "kwargs", normalize_json(self.kwargs))

# --- Instances to compare ---

a_full = TaskResult(**common_kwargs)
b_full = TaskResult(**common_kwargs)
# equal
c_full = TaskResult(**{**common_kwargs, "id": "other"}) #
not equal

a_id = TaskResultIdEq(**common_kwargs)
b_id = TaskResultIdEq(**common_kwargs)
# equal
c_id = TaskResultIdEq(**{**common_kwargs, "id": "other"}) #
not equal

N = 100_000_000

results = {
"Original == (equal)": timeit.timeit(lambda: a_full == b_full,
number=N),
"Original == (not equal)": timeit.timeit(lambda: a_full == c_full,
number=N),
"Id-only == (equal)": timeit.timeit(lambda: a_id == b_id,
number=N),
"Id-only == (not equal)": timeit.timeit(lambda: a_id == c_id,
number=N),
}

for label, t in results.items():
print(f"{label:<30} {t:.3f}s ({t/N*1e9:.1f} ns/call)")
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/37126#comment:2>

[Django]

#37126: Make Task and TaskResult comparable
--------------------------------+--------------------------------------
Reporter: Johannes Maron | Owner: (none)
Type: New feature | Status: new
Component: Tasks | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
--------------------------------+--------------------------------------

Comment (by zky):

Hi Johannes, thanks for providing such clear benchmarks! The performance
gains for the task queue are very significant. I'm quite interested in
this and would love to help take the implementation work off your hands.
--
Ticket URL: <https://code.djangoproject.com/ticket/37126#comment:3>

[Django]

#37126: Make Task and TaskResult comparable
--------------------------------+------------------------------------

Reporter: Johannes Maron | Owner: zky
Type: New feature | Status: assigned

Component: Tasks | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
--------------------------------+------------------------------------

Changes (by zky):

* owner: (none) => zky
* stage: Unreviewed => Accepted
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/37126#comment:4>