[Django] #37220: A Count-less Paginator for high-performance navigation

[Django]

#37220: A Count-less Paginator for high-performance navigation
------------------------------------------+--------------------------------
Reporter: Harvey Bellini | Owner: Harvey Bellini
Type: Uncategorized | Status: assigned
Component: Uncategorized | Version: 6.0
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------+--------------------------------
The new paginator provides a way to navigate results without performing a
COUNT(*) query, instead determining the existence of a "next page" by
fetching `N + 1` items.

**Problem**

On large PostgreSQL or MySQL tables, `SELECT COUNT(*)` requires a full or
partial index scan that scales linearly with data size. Even if the actual
data fetch is fast (via indexed slicing), the Paginator is forced to wait
for the count to:

- Validate the page number
- Determine if a "next" page exists
- Calculate the total number of pages for the UI

In many modern UIs (like "Infinite Scroll" or "Next/Prev" only
navigation), the total page count is unnecessary and actively hurts
performance.

https://github.com/django/new-features/issues/136
--
Ticket URL: <https://code.djangoproject.com/ticket/37220>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#37220: A Count-less Paginator for high-performance navigation

--------------------------------+------------------------------------------

Reporter: Harvey Bellini | Owner: Harvey Bellini
Type: Uncategorized | Status: assigned
Component: Uncategorized | Version: 6.0

Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

--------------------------------+------------------------------------------
Comment (by Simon Charette):

Shouldn't we favour adopting cursor based pagination in core instead?

The latter doesn't require `COUNT` and avoid using `OFFSET` entirely which
can be even of a performance problem for large offsets.

To me `CursorPaginator` has the same limitation as this suggested N+1 item
approach (no listing of pages, ''has next page'' can also be achieved by
fetching an extra item) but with less drawback (no `OFFSET`) so I see
limited benefits in maintaining a solution that only address performance
concerns partially.
--
Ticket URL: <https://code.djangoproject.com/ticket/37220#comment:1>

[Django]

#37220: A Count-less Paginator for high-performance navigation
--------------------------------+------------------------------------------
Reporter: Harvey Bellini | Owner: Harvey Bellini
Type: Uncategorized | Status: assigned
Component: Uncategorized | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------+------------------------------------------

Comment (by Harvey Bellini):

I agree cursor based pagination is the ideal technical solution but that
doesn't mean there isn't value in the `CountlessPaginator` approach.

Does it solve the `COUNT(*)` penalty, yes. Does it solve the `OFFSET`
penalty, no. But it would provide a positive impact for all cases where
users rarely dig past the first few pages. Personally I can't remember the
last time I went past say page 5 on a paginated display.

I don't think these features are mutually exclusive.

Admittedly this is search engine related, and written by a company trying
to sell you their SEO software, but I think the psychology still applies:
https://kafkai.ai/articles/ai-seo/why-users-never-scroll-past-page-one/
--
Ticket URL: <https://code.djangoproject.com/ticket/37220#comment:2>

[Django]

#37220: A Count-less Paginator for high-performance navigation

-------------------------------------+-------------------------------------
Reporter: Harvey Bellini | Owner: Abhay
| Singh Jadaun

Type: Uncategorized | Status: assigned
Component: Uncategorized | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Abhay Singh Jadaun):

* owner: Harvey Bellini => Abhay Singh Jadaun

--
Ticket URL: <https://code.djangoproject.com/ticket/37220#comment:3>

[Django]

#37220: A Count-less Paginator for high-performance navigation

--------------------------------+------------------------------------------
Reporter: Harvey Bellini | Owner: Harvey Bellini

Type: Uncategorized | Status: assigned
Component: Uncategorized | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

--------------------------------+------------------------------------------
Changes (by Harvey Bellini):

* owner: Abhay Singh Jadaun => Harvey Bellini

Comment:

?
--
Ticket URL: <https://code.djangoproject.com/ticket/37220#comment:4>