[Django] #31599: Improve the performance of the autoreloader by ignoring third party packages
Django
packages
------------------------------------------------+------------------------
Reporter: Tom Forbes | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Utilities | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------------+------------------------
In https://github.com/django/django/pull/12750 and
https://code.djangoproject.com/ticket/30372 we stopped watching the
translation files that Django ships, to improve the performance of the
autoreloader.
I believe we can go a step further and stop monitoring all third-party
packages for changes by using the sysconfig module:
https://docs.python.org/3/library/sysconfig.html. With this we can get a
list of different Python system directories, for example:
{{{
data = "/Users/tom/PycharmProjects/github/orf/django/venv"
include =
"/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/include/python3.7m"
platinclude =
"/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/include/python3.7m"
platlib =
"/Users/tom/PycharmProjects/github/orf/django/venv/lib/python3.7/site-
packages"
platstdlib =
"/Users/tom/PycharmProjects/github/orf/django/venv/lib/python3.7"
purelib =
"/Users/tom/PycharmProjects/github/orf/django/venv/lib/python3.7/site-
packages"
scripts = "/Users/tom/PycharmProjects/github/orf/django/venv/bin"
stdlib =
"/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7"
}}}
We could simply not watch any path under the `purelib` directory to start.
This would exclude a huge number of files from being watched - Django
alone has 824 individual files that would be excluded, and it's safe to
assume that a large project with many third party dependencies would have
several thousand.
This would be a breaking change - personally I sometimes find myself
navigating into third party code and making small changes while debugging,
and I'm sure others do this from time to time. However I don't think it's
a common occurance for most developers and I'm not sure everyone paying a
cost is worth the benefit to a small number of people.
--
Ticket URL: <https://code.djangoproject.com/ticket/31599>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: Tom Forbes | Owner: nobody
Cleanup/optimization |
Component: Utilities | Version: master
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by felixxm):
* status: new => closed
* resolution: => wontfix
Comment:
IMO we should still trace changes in 3rd-party packages. It's not only
about making local changes for debugging (I do the same) but also about
changing versions of 3rd-party packages. In such cases, forcing users to
manually restart the server can be annoying.
This kind of changes should be preceded by discussion and a strong
consensus on DevelopersMailingList. Thanks.
--
Ticket URL: <https://code.djangoproject.com/ticket/31599#comment:1>
Django
-------------------------------------+-------------------------------------
Reporter: Tom Forbes | Owner: nobody
Cleanup/optimization |
Component: Utilities | Version: master
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Adam (Chainz) Johnson):
Instead of not watching third party files at all, we could check them much
less frequently. For example `StatReloader` tries to stat every file every
second - it could instead use a lower frequency for third party files,
maybe adapting to high frequency temporarily if it spots any changes
indicating debugging (and maybe even move to a higher frequency for
project files). I'm not sure how it could be extended to
`WatchmanReloader` - perhaps only files that are expected to be changed
are checked through watchman, the rest on a low-frequency state.
--
Ticket URL: <https://code.djangoproject.com/ticket/31599#comment:2>
Django
-------------------------------------+-------------------------------------
Reporter: Tom Forbes | Owner: nobody
Cleanup/optimization |
Component: Utilities | Version: master
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Adam (Chainz) Johnson):
Anyway, yes a mailing list discussion would be nice, could draw out some
alternative approaches.
--
Ticket URL: <https://code.djangoproject.com/ticket/31599#comment:3>