Future of the development server's auto-reloading
1,105 views
Skip to first unread message
Aymeric Augustin
Aug 8, 2015, 5:53:32 PM8/8/15
to django-d...@googlegroups.com
Hello,
While writing some horrific code [1] related to the development server’s auto-reloading, I realized that its design is fundamentally flawed and I think that we should reconsider it.
The current implementation walks through the list of all imported Python modules, attempts to figure out which file they come from and builds a list of these files. Then it watches all these files for changes by checking if their mtime changed every second or, since 1.7, with inotify on Linux. I tried to do it with kqueue on BSD (read: OS X) but I failed. This stuff is hard.
In addition to the source of imported Python modules, the autoreloader watches .mo files in order to catch translation changes and source files for modules that failed to import — it extracts them from exception tracebacks. This shows the limits of basing the list on imported Python modules. Even with such hacks, the auto-reloader will never handle all cases. Two examples:
- It doesn’t survive a syntax error in the settings module. I have reasons to believe that this would be extremely messy to fix.
- If a module reads a configuration file on disk at startup and caches it, the auto-reloader doesn’t detect changes to that file.
So we’re stuck with cases where the development server doesn’t reload. I can’t say they’re edge cases; who never made a typo in a settings file? People who run tutorials report that it’s a very common cause of frustration for beginners.
It's also wasteful. There’s little need to watch every file of every dependency in the virtualenv for changes.
I think it would be much better to remove the auto-reloading logic from Django and have an external process watch the current working directory for changes. (When virtualenv is considered appropriate for people with exactly 0 days of Django experience under their belt, I think we can require a dependency for the development server’s auto-reloading.)
The behavior would be much easier to describe: if you change something in your source tree, it reloads automatically; if you change something somewhere else, you must reload manually. It would also be quite easy to customize with a standard include / exclude mechanism to select which files should be watched.
We aren’t in the business of writing production web servers, neither are we in the business of writing a filesystem watcher :-) Watchman [2] appears to be a robust solution. However I would prefer a pure-Python implementation that could easily be installed in a virtualenv. Does someone have experience in this area?
I’m not making a concrete proposal as I haven’t studied the details. It this point I’d just like to hear what you think of the general concept.
Thanks!
Tino de Bruijn
Aug 9, 2015, 5:01:27 AM8/9/15
to django-d...@googlegroups.com
Hi Aymeric,
While I have wondered from time to time why runserver could not just continue after certain syntax errors, and wished it could do so, I think what you are proposing makes a lot of sense. I think having to manually reload after installing a new package is really acceptable (and more an edge-case than typing a typo).
There is also watchdog [1], which also seems to work on Windows. It does contain some C for OS X FSEvents bindings.
Tino
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8E08AB59-2450-40D5-A52B-C8803DA39D5F%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Florian Apolloner
Aug 9, 2015, 6:37:41 AM8/9/15
to Django developers (Contributions to Django itself)
On Saturday, August 8, 2015 at 11:53:32 PM UTC+2, Aymeric Augustin wrote:
- It doesn’t survive a syntax error in the settings module. I have reasons to believe that this would be extremely messy to fix.- If a module reads a configuration file on disk at startup and caches it, the auto-reloader doesn’t detect changes to that file.
It also misses the addition of a new admin.py file etc
The behavior would be much easier to describe: if you change something in your source tree, it reloads automatically; if you change something somewhere else, you must reload manually. It would also be quite easy to customize with a standard include / exclude mechanism to select which files should be watched.
Seems good to me and would indeed cover all my current pain points
Collin Anderson
Aug 9, 2015, 3:55:54 PM8/9/15
to Django developers (Contributions to Django itself)
This all makes sense to me, but it would be nice if it worked out of the box, especially for the purposes of the tutorial. :)
Daniel Moisset
Aug 9, 2015, 4:23:03 PM8/9/15
to django-d...@googlegroups.com
The idea sounds awesome. A couple of corner cases that should be taken in consideration:
- I think it is common for small simple projects (and people doing the tutorial) to have the sqlite db file in the same project directory. If we're watching for writes in every file in the project directory, every save() could trigger a reload
- It is also usual to have the media/ directory based on filesystem storage inside the project dir. So any project with file uploads could trigger a reload on each file upload.
Possibly some exclusion rules with sane defaults could help here
Regards,
D.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8E08AB59-2450-40D5-A52B-C8803DA39D5F%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Sam Cooke
Aug 10, 2015, 5:35:05 AM8/10/15
to django-d...@googlegroups.com
Hi all,
I've had experience playing around with watchdog[1] before - I've only ever used it on OS X but it should work cross platform. I've thrown together a rough proof of concept[2] for runserver that uses watchdog to watch for changes/additions/deletions to python files recursively in the same directory as manage.py and restarts the server accordingly. It could easily be extended to handle translations.
To avoid changing a lot of code I've just set it up to use watchdog if available and fall back to the current method silently if not.
I'd be happy to turn this into a more complete solution if people think it's the right route to go. We can choose to:
- make watchdog required - it's a simple pip install
- make watchdog required if you want runserver to autoreload
- fall back to the current implementation if watchdog is not on the python path (and potentially tell the user about watchdog)
Do you think it's worth carrying on down this path?
Sam
[2] https://github.com/django/django/compare/master...sdcooke:watchdog (ignore the code quality - it was thrown together quickly!)
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CALuYSZX1X8D3BNO9Mt_DuHHA34FTO2PM%3DUMYw87ueeGLnbnk_A%40mail.gmail.com.
Aymeric Augustin
Aug 10, 2015, 8:06:46 AM8/10/15
to django-d...@googlegroups.com
Hello Sam,
On 10 août 2015, at 11:34, Sam Cooke <sdc...@gmail.com> wrote:
> I've thrown together a rough proof of concept[2] for runserver that uses watchdog to watch for changes/additions/deletions to python files recursively in the same directory as manage.py and restarts the server accordingly. It could easily be extended to handle translations.
Interesting proof of concept!
I had in mind a more invasive approach. To remove almost all supporting code from Django, I’d like the implementation to look like:
(somewhere in the runserver command, with an hypothetical watchrun utility)
if autoreload:
cmd = sys.argv + ["--noreload"]
os.execlp("watchrun", "--include=*.py,*.mo", "--", *cmd)
Of course the actual implementation would have to be a bit smarter with the include / exclude patterns in order to allow customization.
This would make the reloading tool required if you want runserver to autoreload.
--
Aymeric.
On 10 août 2015, at 11:34, Sam Cooke <sdc...@gmail.com> wrote:
> I've thrown together a rough proof of concept[2] for runserver that uses watchdog to watch for changes/additions/deletions to python files recursively in the same directory as manage.py and restarts the server accordingly. It could easily be extended to handle translations.
I had in mind a more invasive approach. To remove almost all supporting code from Django, I’d like the implementation to look like:
(somewhere in the runserver command, with an hypothetical watchrun utility)
if autoreload:
cmd = sys.argv + ["--noreload"]
os.execlp("watchrun", "--include=*.py,*.mo", "--", *cmd)
Of course the actual implementation would have to be a bit smarter with the include / exclude patterns in order to allow customization.
This would make the reloading tool required if you want runserver to autoreload.
--
Aymeric.
Carl Meyer
Aug 10, 2015, 1:15:46 PM8/10/15
to django-d...@googlegroups.com
Hi Aymeric,
I actually spent the PyCon 2015 sprints working on a generic
watcher/reloader for Python servers, built on top of watchdog:
https://github.com/carljm/wsgiwatcher
It should still be considered alpha/prototype code, but I think the
approach is promising. It uses a two-process model, where the master
process just monitors for file changes, and spawns a subprocess with the
actual server. The API is just a `run()` function that takes a dotted
import path to a serve-forever callable.
This model (as opposed to the single-process threads-only model used by
runserver) fixes the "killed by settings.py SyntaxError" problem,
because a failure in the server-running subprocess never kills the
master monitor process; the monitor just waits for the next file change
and tries again.
The question of which paths to monitor is orthogonal. Wsgiwatcher
currently follows the same "monitor everything in sys.modules" pattern
as runserver, but it could easily be modified to take a fixed directory
to watch instead. In fact that would simplify it quite a bit - currently
it has to use a multiprocessing Queue and a separate thread in the child
process to poll sys.modules and communicate the file list back to the
monitor process, and that could be removed.
The main potential issue with Wsgiwatcher's multi-process model (which I
haven't explored much yet) is how it interacts with multi-process
servers like gunicorn. This wouldn't be an issue for using it (or
something like it) with runserver.
I don't know that I'll have time to get back to Wsgiwatcher soon, but if
you (or Sam) are interested in working on alternative reloaders for
runserver, you're more than welcome to use it, improve it, and/or rip
off some code from it.
Carl
I actually spent the PyCon 2015 sprints working on a generic
watcher/reloader for Python servers, built on top of watchdog:
https://github.com/carljm/wsgiwatcher
It should still be considered alpha/prototype code, but I think the
approach is promising. It uses a two-process model, where the master
process just monitors for file changes, and spawns a subprocess with the
actual server. The API is just a `run()` function that takes a dotted
import path to a serve-forever callable.
This model (as opposed to the single-process threads-only model used by
runserver) fixes the "killed by settings.py SyntaxError" problem,
because a failure in the server-running subprocess never kills the
master monitor process; the monitor just waits for the next file change
and tries again.
The question of which paths to monitor is orthogonal. Wsgiwatcher
currently follows the same "monitor everything in sys.modules" pattern
as runserver, but it could easily be modified to take a fixed directory
to watch instead. In fact that would simplify it quite a bit - currently
it has to use a multiprocessing Queue and a separate thread in the child
process to poll sys.modules and communicate the file list back to the
monitor process, and that could be removed.
The main potential issue with Wsgiwatcher's multi-process model (which I
haven't explored much yet) is how it interacts with multi-process
servers like gunicorn. This wouldn't be an issue for using it (or
something like it) with runserver.
I don't know that I'll have time to get back to Wsgiwatcher soon, but if
you (or Sam) are interested in working on alternative reloaders for
runserver, you're more than welcome to use it, improve it, and/or rip
off some code from it.
Carl
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com
> <mailto:django-develop...@googlegroups.com>.
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com
> To post to this group, send email to django-d...@googlegroups.com
> <mailto:django-d...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/8E08AB59-2450-40D5-A52B-C8803DA39D5F%40polytechnique.org
> <https://groups.google.com/d/msgid/django-developers/8E08AB59-2450-40D5-A52B-C8803DA39D5F%40polytechnique.org?utm_medium=email&utm_source=footer>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/8E08AB59-2450-40D5-A52B-C8803DA39D5F%40polytechnique.org
George Kappel
Aug 10, 2015, 2:09:04 PM8/10/15
to Django developers (Contributions to Django itself)
In case it is useful as a reference, runserver_plus from django-extensions uses Werkzeug/Watchdog for reloading
> <mailto:django-developers+unsub...@googlegroups.com>.
Burhan
Aug 14, 2015, 6:17:19 PM8/14/15
to Django developers (Contributions to Django itself)
LiveReload[1] is one more to consider; what's more it already has examples for Django and other Python frameworks. It relies on Tornado (which may cause some dependency issues? would require testing) but it works.
The only tweak that would be required would be to ensure that static files/media files continue to be served correctly.
Regards,
--
Burhan Khalid
[1] https://github.com/lepture/python-livereload
The only tweak that would be required would be to ensure that static files/media files continue to be served correctly.
Regards,
--
Burhan Khalid
[1] https://github.com/lepture/python-livereload
Elf Sternberg
Jun 4, 2016, 8:37:25 PM6/4/16
to Django developers (Contributions to Django itself)
Tim Graham
Jun 27, 2016, 8:23:05 PM6/27/16
to Django developers (Contributions to Django itself)
A pull request is proposed to add a new setting to allow specifying a custom reloader:
https://github.com/django/django/pull/6719
Is this something anyone else would find useful and does it seems like we could continue support that option even if autoreloading is refactored based on the ideas here?
https://github.com/django/django/pull/6719
Is this something anyone else would find useful and does it seems like we could continue support that option even if autoreloading is refactored based on the ideas here?
Tim Graham
Jan 4, 2017, 6:33:31 PM1/4/17
to Django developers (Contributions to Django itself)
So this idea doesn't get lost, I created a ticket for "Allow autoreloader to use watchman"
https://code.djangoproject.com/ticket/27685#ticket
https://code.djangoproject.com/ticket/27685#ticket
Tom Forbes
Jul 24, 2017, 9:58:15 AM7/24/17
to Django developers (Contributions to Django itself)
I was looking into refactoring the auto-reloading code and I wanted to gather some feedback. There is some potential to squash several bugs in one go[1][2][3].
To begin with we can get rid of the Jython specific code, Jython is 2.7 only at the moment and I don't think that will change soon.
We currently have an implicit dependency on `pyinotify`, if it's installed a more efficient inotify-based watcher is used. I've never seen this advertised anywhere, and pyinotify has not received any updates in over 3 years. There is also this[4] blog post which says perhaps pyinotify isn't the best. I think the best way forward is to use the 'watchdog' package if it's installed, as it contains platform-specific implementations for observing file modifications[6] and even their fallback polling observer[7] is better than any we could include in Django. If we want to add the ability to use the watchman service with Django (something I fully agree with) perhaps adding support for it directly to that library would be more preferable than bundling it in Django itself.
The current autoreloading code contains some translations specific code that would be good to split out. It would be nice to have the autoreload module be as focused as possible on just detecting and handling file changes in a pretty neutral way, other parts of Django could register subscriptions to be notified when files change and can handle it themselves (perhaps via signals?). For example the cached template loader could register a callback to fire whenever one if it's source files changes and delete that entry from the cache, the translations framework could handle .mo files changing itself or third party modules can use these hooks to do whatever crazy things they want.
I've got a first-draft branch with a cleanup of the autoreload code here[8], any feedback would be welcome. It currently doesn't handle syntax error reloading or the i18n resetting. Could the syntax error reloading be improved with a clean re-implementation of the autoreloader? I can't think of any nice ways to handle it.
1. https://code.djangoproject.com/ticket/27685#ticket
2. https://code.djangoproject.com/ticket/25624
3. https://code.djangoproject.com/ticket/25791
4. http://www.serpentine.com/blog/2008/01/04/why-you-should-not-use-pyinotify/
5. https://pythonhosted.org/watchdog/_modules/watchdog/observers/polling.html
6. https://pythonhosted.org/watchdog/api.html#module-watchdog.observers
7. https://pythonhosted.org/watchdog/_modules/watchdog/observers/polling.html
8. https://github.com/orf/django/blob/27685-autoreloader-refactor/django/utils/autoreload.py
To begin with we can get rid of the Jython specific code, Jython is 2.7 only at the moment and I don't think that will change soon.
We currently have an implicit dependency on `pyinotify`, if it's installed a more efficient inotify-based watcher is used. I've never seen this advertised anywhere, and pyinotify has not received any updates in over 3 years. There is also this[4] blog post which says perhaps pyinotify isn't the best. I think the best way forward is to use the 'watchdog' package if it's installed, as it contains platform-specific implementations for observing file modifications[6] and even their fallback polling observer[7] is better than any we could include in Django. If we want to add the ability to use the watchman service with Django (something I fully agree with) perhaps adding support for it directly to that library would be more preferable than bundling it in Django itself.
The current autoreloading code contains some translations specific code that would be good to split out. It would be nice to have the autoreload module be as focused as possible on just detecting and handling file changes in a pretty neutral way, other parts of Django could register subscriptions to be notified when files change and can handle it themselves (perhaps via signals?). For example the cached template loader could register a callback to fire whenever one if it's source files changes and delete that entry from the cache, the translations framework could handle .mo files changing itself or third party modules can use these hooks to do whatever crazy things they want.
I've got a first-draft branch with a cleanup of the autoreload code here[8], any feedback would be welcome. It currently doesn't handle syntax error reloading or the i18n resetting. Could the syntax error reloading be improved with a clean re-implementation of the autoreloader? I can't think of any nice ways to handle it.
1. https://code.djangoproject.com/ticket/27685#ticket
2. https://code.djangoproject.com/ticket/25624
3. https://code.djangoproject.com/ticket/25791
4. http://www.serpentine.com/blog/2008/01/04/why-you-should-not-use-pyinotify/
5. https://pythonhosted.org/watchdog/_modules/watchdog/observers/polling.html
6. https://pythonhosted.org/watchdog/api.html#module-watchdog.observers
7. https://pythonhosted.org/watchdog/_modules/watchdog/observers/polling.html
8. https://github.com/orf/django/blob/27685-autoreloader-refactor/django/utils/autoreload.py
Aymeric Augustin
Jul 29, 2017, 6:58:59 AM7/29/17
to django-d...@googlegroups.com
Hello Tom,
Your approach seems comprehensive. I'm commenting on your PR on Trac and GitHub. Let's do this!
Regarding watchdog / watchman, I think Django could select automatically one or the other depending on which on is installed. If none is installed, Django would fall back to the current approach and perhaps raise a warning to suggest installing something better.
Regarding the syntax error reloading, I believe the solution is the two-process approach described by Carl earlier in this thread. Check it out.
Thanks!
--
Aymeric.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b33d73e3-6ff5-431d-b9ff-7db1219d11cd%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages