[Django] #24736: Replace magic 50000 Sitemap item limit with customizable value

7 views
Skip to first unread message

Django

unread,
May 1, 2015, 8:49:36 PM5/1/15
to django-...@googlegroups.com
#24736: Replace magic 50000 Sitemap item limit with customizable value
----------------------------------+---------------------
Reporter: Miserlou | Owner: nobody
Type: New feature | Status: new
Component: contrib.sitemaps | Version: 1.8
Severity: Normal | Keywords: sitemap
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 1 | UI/UX: 0
----------------------------------+---------------------
Hey all!

I recently ran into an interesting and reasonably soluble problem. I've
switched from using a static sitemap to Django's Sitemap framework. Now,
my database has a ton of items to map (>200,000), so generating the
sitemap actually takes quite a long time. Unfortunately, it takes such a
long time that my web server actually times out waiting for the request,
which is obviously a big problem.

Rather than deal with adding exceptions in my web server or manually
creating static sitemaps, I'd greatly prefer to lower the number of items
on each page and have a larger number of entries on the index page.

Currently, the pagination uses a magic number of 50000, which is the
maximum number of items recommended by Google. That line is here:
https://github.com/django/django/blob/c2d5f2903cfaef4f8b17d86f4db706b61073a471/django/contrib/sitemaps/__init__.py#L50

I'd greatly prefer it if this was available as a configurable setting.

This would be a very simple, 2-line fix:

One to add a

SITEMAP_MAX_ITEMS = 50000

to Django's default settings,and another change the line linked about to

limit = settings.SITEMAP_MAX_ITEMS

It's a small change, but this would be greatly useful to me and other
Django sites with lots of items that use the Sitemap framework. In the
meantime, I'm just going try to override default Sitemap object.

What do you guys think? Does this make sense to you?

--
Ticket URL: <https://code.djangoproject.com/ticket/24736>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,
May 1, 2015, 9:10:47 PM5/1/15
to django-...@googlegroups.com
#24736: Replace magic 50000 Sitemap item limit with customizable value
----------------------------------+--------------------------------------

Reporter: Miserlou | Owner: nobody
Type: New feature | Status: new
Component: contrib.sitemaps | Version: 1.8
Severity: Normal | Resolution:

Keywords: sitemap | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
----------------------------------+--------------------------------------
Changes (by Miserlou):

* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0


Comment:

Pull request submitted for your gracious consideration.

--
Ticket URL: <https://code.djangoproject.com/ticket/24736#comment:1>

Django

unread,
May 1, 2015, 9:42:46 PM5/1/15
to django-...@googlegroups.com
#24736: Replace magic 50000 Sitemap item limit with customizable value
----------------------------------+--------------------------------------

Reporter: Miserlou | Owner: nobody
Type: New feature | Status: new
Component: contrib.sitemaps | Version: 1.8
Severity: Normal | Resolution:

Keywords: sitemap | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
----------------------------------+--------------------------------------

Comment (by MarkusH):

Thanks for submitting this issue and patch. However, I'm -1 on adding a
global settings variable if you can just override `limit` on your sitemap
class much like `protocol`:
https://docs.djangoproject.com/en/1.8/ref/contrib/sitemaps/#django.contrib.sitemaps.Sitemap.protocol.
Or am I missing something?

If we're pursuing this issue I'd prefer to document `limit` as public API
and mention that its value shouldn't exceed 50000.

--
Ticket URL: <https://code.djangoproject.com/ticket/24736#comment:2>

Django

unread,
May 1, 2015, 9:43:49 PM5/1/15
to django-...@googlegroups.com
#24736: Replace magic 50000 Sitemap item limit with customizable value
----------------------------------+--------------------------------------

Reporter: Miserlou | Owner: nobody
Type: New feature | Status: new
Component: contrib.sitemaps | Version: master
Severity: Normal | Resolution:

Keywords: sitemap | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
----------------------------------+--------------------------------------
Changes (by MarkusH):

* version: 1.8 => master


--
Ticket URL: <https://code.djangoproject.com/ticket/24736#comment:3>

Django

unread,
May 1, 2015, 11:15:24 PM5/1/15
to django-...@googlegroups.com
#24736: Replace magic 50000 Sitemap item limit with customizable value
----------------------------------+--------------------------------------

Reporter: Miserlou | Owner: nobody
Type: New feature | Status: new
Component: contrib.sitemaps | Version: master
Severity: Normal | Resolution:

Keywords: sitemap | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
----------------------------------+--------------------------------------

Comment (by Miserlou):

Fine by me.. that's how I actually ended up solving the problem in my
application. I think that makes more sense anyway.

--
Ticket URL: <https://code.djangoproject.com/ticket/24736#comment:4>

Django

unread,
May 2, 2015, 3:14:37 AM5/2/15
to django-...@googlegroups.com
#24736: Document how to customize the 50000 item limit in sitemaps
-------------------------------+------------------------------------

Reporter: Miserlou | Owner: nobody
Type: New feature | Status: new
Component: Documentation | Version: master
Severity: Normal | Resolution:
Keywords: sitemap | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
-------------------------------+------------------------------------
Changes (by aaugustin):

* stage: Unreviewed => Accepted
* component: contrib.sitemaps => Documentation


Comment:

I agree as well. Requalifying as a documentation ticket and accepting.

--
Ticket URL: <https://code.djangoproject.com/ticket/24736#comment:5>

Django

unread,
May 4, 2015, 1:40:43 PM5/4/15
to django-...@googlegroups.com
#24736: Document how to customize the 50000 item limit in sitemaps
-------------------------------+------------------------------------
Reporter: Miserlou | Owner: abhaga
Type: New feature | Status: assigned

Component: Documentation | Version: master
Severity: Normal | Resolution:
Keywords: sitemap | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
-------------------------------+------------------------------------
Changes (by abhaga):

* owner: nobody => abhaga
* status: new => assigned


--
Ticket URL: <https://code.djangoproject.com/ticket/24736#comment:6>

Django

unread,
May 4, 2015, 1:53:30 PM5/4/15
to django-...@googlegroups.com
#24736: Document how to customize the 50000 item limit in sitemaps
--------------------------------------+------------------------------------
Reporter: Miserlou | Owner: abhaga
Type: Cleanup/optimization | Status: assigned

Component: Documentation | Version: master
Severity: Normal | Resolution:
Keywords: sitemap | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
--------------------------------------+------------------------------------
Changes (by claudep):

* has_patch: 0 => 1
* type: New feature => Cleanup/optimization


--
Ticket URL: <https://code.djangoproject.com/ticket/24736#comment:7>

Django

unread,
May 5, 2015, 9:18:34 AM5/5/15
to django-...@googlegroups.com
#24736: Document how to customize the 50000 item limit in sitemaps
--------------------------------------+------------------------------------
Reporter: Miserlou | Owner: abhaga
Type: Cleanup/optimization | Status: closed
Component: Documentation | Version: master
Severity: Normal | Resolution: fixed

Keywords: sitemap | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
--------------------------------------+------------------------------------
Changes (by Tim Graham <timograham@…>):

* status: assigned => closed
* resolution: => fixed


Comment:

In [changeset:"9096e2b5f75abf8e8882937bd3c3d47ccdc24e25" 9096e2b]:
{{{
#!CommitTicketReference repository=""
revision="9096e2b5f75abf8e8882937bd3c3d47ccdc24e25"
Fixed #24736 -- Documented the Sitemap.limit attribute
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/24736#comment:8>

Django

unread,
May 5, 2015, 9:23:14 AM5/5/15
to django-...@googlegroups.com
#24736: Document how to customize the 50000 item limit in sitemaps
--------------------------------------+------------------------------------
Reporter: Miserlou | Owner: abhaga
Type: Cleanup/optimization | Status: closed
Component: Documentation | Version: master
Severity: Normal | Resolution: fixed
Keywords: sitemap | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
--------------------------------------+------------------------------------

Comment (by Tim Graham <timograham@…>):

In [changeset:"eb00b427fc9d12c8ba53ce3875d7e481cd825e38" eb00b427]:
{{{
#!CommitTicketReference repository=""
revision="eb00b427fc9d12c8ba53ce3875d7e481cd825e38"
[1.8.x] Fixed #24736 -- Documented the Sitemap.limit attribute

Backport of 9096e2b5f75abf8e8882937bd3c3d47ccdc24e25 from master
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/24736#comment:9>

Reply all
Reply to author
Forward
0 new messages