A different approach for the auto-reloader

[Ramiro Morales]

Hi all,

I had a stab at a somewhat simpler development server automatic reloading strategy https://github.com/django/django/compare/master...ramiro:synch-reloader

Intention is to test how an implementation of a design by Gary Bernhardt would look. The best written description I could find is this:

https://github.com/devlocker/tychus/issues/3

Gary also had posted some tweets (this is how I got interested in the topic) which seems to have been deleted since then.

Main idea is: Actual checking of changes on the filesystem for modules under monitoring isn't performed in a loop or by depending on a OS kernel feature but per-HTTP request by a front-end proxy process which is in charge of restarting the 'upstream' web server process (in our case a dumbed-down runserver dev server) only when it detects there have been changes.

Been meaning to try this for some time. It would have been much harder before Tom Forbes' work on refactoring and cleaning up the reloading code for Django 2.2. IMHO Tom's code is so very well thought that for example I just had to lightly subclass StatReload to implement this totally different strategy.

Current form of the code is a new experimental 'serverrun' (for lack of a better name) added to the Django code base whose command line UI mimics 100% the runserver one.

It copies code from a few places of our code base: The runserver command, the WSGI app hosting code, etc.

I decided to implement this as a new built-in command for now a) to ease experimentation and b) because it needs some minor changes to the 'runserver' command to handle cosmetic details (logging). If the idea is accepted (read further below for reasons in favor of this) then maybe we can switch runserver to this code. Or if the idea isn't deemed appropate for Django core them I might implement it as an standalone django app/project.

If the idea of a smarter stat()-based FS status monitor like this gets actually tested and validated in the field (i.e. by users with big source code trees) it could allow us to possibly stop needing to depend on all of:

* watchman
* pyinotify
* watchdog
(and removing our support code for them from the Django code base).

Also, this would mean:

* Setup simplification for final users (no third party Python libraries or system daemon to install)
* Better cross-platform portability for Django (we go back to piggy-backing stat() from the stdlib as our only way yo trigger code reloading).

Additionally, as the reloading is performed fully (by restarting the whole HTTP server) and is triggered from another process (the transparent http proxy one) we can drop some contortions we currently need to make:

- Having to wait for the app registry stabilization
- Avoiding race conditions with the url resolver

I suspect there could be power efficiency advantages too as:

* The scanning for changes is triggered by HTTP requests which should be less frequent than periodically every N seconds.
* If the developer modifies more than one file before switching to the browser there is need of only one FS scan to cater for all these changes, which is performed just in time for the first HTTP request so the code executed to render/serve it is 100% accurate in regard to actually reflecting the state of the code on disk.

Similar projects include:
- serveit: https://github.com/garybernhardt/serveit
- tychus: https://github.com/devlocker/tychus
- wsgiwatch: https://github.com/dpk/wsgiwatch

Feedback is welcome!

Regards,

--

Ramiro Morales
@ramiromorales

[Carlton Gibson]

Hi Ramiro. 

I had a quick look at this — it looks great. (And various folks are having fun with Watchman, so if the promise delivers it'll be welcome.) 

Do you want to make an actual PR (perhaps with a ticket) so we can get a proper review going? 

Thanks for the input, as ever! 🙂

Kind Regards,

Carlton

[Andrew Godwin]

From my read this also looks like it would make the auto-reloader able to work a lot better with an async-capable server, so I would be in favour given that is likely in the future as well.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAO7PdF99hUobeXs8JQyb%3DywDJ6bkkKWyhUYC%3DEa9JzwQM%2BH_5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Forbes]

Hey Ramiro,

This sounds like a pretty awesome idea in general. It would also solve one long standing issue where the http socket is closed and re-opened during reloading, leading to pageviews after code changes failing.

I’m trying to think of some edge cases where we need to be careful, and there are three I can think of:

1. It’s really hard to know where a request is going at this level, so we could end up with a page view triggering 20 static files requests very quickly, which would lead to stating all files 20x redundantly.

2. How would we handle (potentially ridiculous) things like code imported from a thread during app ready? In general wouldn’t we need some kind of thread to poll for new imports to watch after Django has been started?

3. Are there any workflows that would be disrupted by this? I can’t think of any that I use, but it’s possible that people have come to rely on the existing “does this throw an exception” in some cases, and triggering a request only to get this could be annoying?

Overall though this feels like a good idea. Stat based reloaders are a lot simpler to deal with than platform specific fs monitoring. Ask thanks for the kind words about my refactor, I’m glad the abstractions work on something like this.

Tom

--

[Adam Johnson]

1. It’s really hard to know where a request is going at this level, so we could end up with a page view triggering 20 static files requests very quickly, which would lead to stating all files 20x redundantly.

Indeed, maybe this could be solved with a re-stat cooldown?

 

2. How would we handle (potentially ridiculous) things like code imported from a thread during app ready? In general wouldn’t we need some kind of thread to poll for new imports to watch after Django has been started?

This is less ridiculous than it sounds, I'm working on an APM package that does something like this.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9EAABCD5-1B76-4A0D-AF73-A5401AF7F427%40tomforb.es.

For more options, visit https://groups.google.com/d/optout.

--

Adam

[natali...@gmail.com]

Hello everyone!

I'm conducting a PR cleanse crusade for the project, where I'll try move forward those PRs that still make sense or close them when appropriate. Were there other developments in this topic that would be relevant to make a decision about the PR[0]?

Thank you!

Natalia.

[0] https://github.com/django/django/pull/13154

[David Arredondo]

Hola 

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/13b7e075-c3bd-4dd2-9952-fb3fc8daecden%40googlegroups.com.

[Salomon Roberto Herrera Navarro]

Buenas tardes David, cuentame

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CABe-O5ncf6pnDsKSEOqQQ9Rwt%3DoZkU03FfZN_x7ASDT_E5Qxpg%40mail.gmail.com.