[Django] #29994: Document performance issues in FileBasedCache
Django
-----------------------------------------+------------------------
Reporter: matkoniecz | Owner: nobody
Type: Uncategorized | Status: new
Component: Documentation | Version: 2.1
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
https://docs.djangoproject.com/en/2.1/topics/cache/#filesystem-caching is
not mentioning any problems with FileBasedCache
According to https://github.com/grantjenks/python-diskcache#diskcache-
disk-backed-cache "Unfortunately the file-based cache in Django is
essentially broken. The culling method is random and large caches
repeatedly scan a cache directory which slows linearly with growth. Can
you really allow it to take sixty milliseconds to store a key in a cache
with a thousand items?"
From my checking it seems to not be reported so far.
According to https://github.com/grantjenks/python-
diskcache/issues/93#issuecomment-442580191 "The deficiencies of
FileBasedCache are too well known to require enumeration" but it would be
nice to warn also developers unaware of this problems.
--
Ticket URL: <https://code.djangoproject.com/ticket/29994>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: matkoniecz | Owner: nobody
Type: Uncategorized | Status: new
Component: Documentation | Version: 2.1
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 1
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by matkoniecz):
* needs_docs: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:1>
Django
Reporter: matkoniecz | Owner: nobody
Type: Uncategorized | Status: new
Component: Documentation | Version: 2.1
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 1
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Grant Jenks):
Related, and much older issue: https://code.djangoproject.com/ticket/11260
I agree that the file-based caching docs could be improved to described
the limitations. I would also welcome a link to the Disk Cache project
(disclaimer: I'm the author.)
For the Django developers, please don't think Disk Cache is a criticism of
Django's file-based cache. I love Django! The Disk Cache project is
thousands of lines of code with many non-Django users. I don't think the
code nor design is appropriate for inclusion in Django.
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:2>
Django
Reporter: matkoniecz | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* needs_docs: 1 => 0
* type: Uncategorized => Cleanup/optimization
* stage: Unreviewed => Accepted
Comment:
The Django documentation avoids endorsing third-party packages because of
the burden of curation this would present. I think we would need a
consensus on the DevelopersMailingList to do that.
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:3>
Django
Reporter: matkoniecz | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by matkoniecz):
From #11260 - ""I'm going to wontfix, on the grounds that the filesystem
cache is intended as an easy way to test caching, not as a serious caching
strategy. The default cache size and the cull strategy implemented by the
file cache should make that obvious.""
Would it be OK to submit patch that for start will add something like
"Note that this caching strategy is useful only for development and should
not be used in production" ?
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:4>
Django
Reporter: matkoniecz | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
I'm not sure. I think ticket:11260#comment:9 is interesting: "I have
Django sites of tens of thousands of pages running for over 2 years using
the above patches, so your statements about filesystem caching not a
serious strategy are irrelevant." The patch for #11260 is very simple and
seems worth considering if it makes file-based caching usable in
production. Do you have an opinion on that, Grant?
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:5>
Django
Reporter: Mateusz Konieczny | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Grant Jenks):
I looked at the patch in #11260 but as far as I can tell it only adds a
scenario where when `_max_enries` is set to 0 or None then no culling ever
takes place. I'm not sure how `anteater_sa` handles cache size in that
case. I suppose if you can guarantee that the filesystem cache will never
grow beyond a certain size then that's a reasonable strategy. It's more of
a persistent dictionary though.
I certainly think filesystem caching has serious use cases. I myself have
used diskcache for an ecommerce site for several years with tens of
thousands of pages. I started with the built-in file-based cached but
later chose sqlite3 (used by diskcache) for better performance.
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:6>
Django
Reporter: Mateusz Konieczny | Owner: Abhishek
Type: | Bera
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Abhishek Bera):
* owner: nobody => Abhishek Bera
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:7>
Django
Reporter: Mateusz Konieczny | Owner: (none)
Type: Cleanup/optimization | Status: new
Component: Documentation | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Abhishek Bera):
* owner: Abhishek Bera => (none)
* status: assigned => new
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:8>
Django
Reporter: Mateusz Konieczny | Owner: sheenarbw
Type: | Status: assigned
Cleanup/optimization |
Component: Documentation | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by sheenarbw):
* owner: nobody => sheenarbw
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:7>
Django
Reporter: Mateusz Konieczny | Owner: sheenarbw
Type: | Status: assigned
Cleanup/optimization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by sheenarbw):
The _cull function in Filesystem caching does still slow down with a large
number of files. I added a warning to the docs.
https://github.com/django/django/pull/16218
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:8>
Django
Reporter: Mateusz Konieczny | Owner: sheenarbw
Type: | Status: assigned
Cleanup/optimization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* needs_better_patch: 0 => 1
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:9>
Django
Reporter: Mateusz Konieczny | Owner: sheenarbw
Type: | Status: assigned
Cleanup/optimization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:10>
Django
Reporter: Mateusz Konieczny | Owner: sheenarbw
Type: | Status: assigned
Cleanup/optimization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* stage: Ready for checkin => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:11>
Django
Reporter: Mateusz Konieczny | Owner: sheenarbw
Type: | Status: assigned
Cleanup/optimization |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:12>
Django
Reporter: Mateusz Konieczny | Owner: (none)
Component: Documentation | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* cc: Sarah Boyce (added)
* owner: sheenarbw => (none)
* status: assigned => new
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:13>
Django
Reporter: Mateusz Konieczny | Owner: Sarah
Type: | Boyce
Cleanup/optimization | Status: assigned
Component: Documentation | Version: 2.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Sarah Boyce):
* owner: (none) => Sarah Boyce
* needs_better_patch: 1 => 0
* status: new => assigned
Comment:
https://github.com/django/django/pull/16551
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:14>
Django
Reporter: Mateusz Konieczny | Owner: Sarah
Type: | Boyce
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:15>
Django
Reporter: Mateusz Konieczny | Owner: Sarah
Type: | Boyce
Component: Documentation | Version: 2.1
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by GitHub <noreply@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"1eb94bc8dab46dfa117d21ef4f3b52aebb593615" 1eb94bc]:
{{{
#!CommitTicketReference repository=""
revision="1eb94bc8dab46dfa117d21ef4f3b52aebb593615"
Fixed #29994 -- Added warning about performance of FileBasedCache with a
large number of files.
Co-authored-by: sheenarbw <699166+s...@users.noreply.github.com>
Co-authored-by: Mariusz Felisiak <felisiak...@gmail.com>
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:16>
Django
Reporter: Mateusz Konieczny | Owner: Sarah
Type: | Boyce
Cleanup/optimization | Status: closed
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"de42d51361e316a4525440ad953e3a42ea6432b7" de42d513]:
{{{
#!CommitTicketReference repository=""
revision="de42d51361e316a4525440ad953e3a42ea6432b7"
[4.2.x] Fixed #29994 -- Added warning about performance of FileBasedCache
with a large number of files.
Co-authored-by: sheenarbw <699166+s...@users.noreply.github.com>
Co-authored-by: Mariusz Felisiak <felisiak...@gmail.com>
Backport of 1eb94bc8dab46dfa117d21ef4f3b52aebb593615 from main
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29994#comment:17>