A proposal for a new auto-reloader in Django

[qingn...@gmail.com]

Hi, I'm David Ma, a first-year Science student from the University of British Columbia. I'm a enthusiastic Django developer and have four years Python programming experience. I've read through the posts about replacing the current autoreloader and would like to work on this task during GSoC. I paraphrase the discussions as follows:

Currently, the default autoreloader contains a lot of problems, most of which are caused by the inefficient methods to look for file changes (#27685). Notably, it fails to work with many python libraries including Jinja (#25624). In addition, if a syntax error is in the settings file, the autoreloader would stop working. We want to get rid of this by tracking the status of an entire folder and using subprocesses to reload the development server.

On going discussions have deemed that the best way to solve this is to replace the current autoreloader with a new one implemented with Watchman (https://groups.google.com/forum/#!topic/django-developers/voXNIDdDcpU/discussion). Since inotify is a low-level library and cannot be applied to platforms including Windows and Macintosh OS, the best library to replace Watchdog is Watchman, a library that supports both of these three platforms (for Windows it's still in the alpha stage.) It runs as a standalone process and has a python wrapper library that is incompatible with Python 3. Therefore, one of the goals is to figure out how to interact with this library. There are also complaints on Django not tracking non-python files or tracking files that it is not supposed to watch. For the new version, we'd expect it to work with cached templates (#25791) and be customizable.

So my goals would be:

1. Replace the old auto-reloader with the Watchman version

2. Make sure that the new auto-reloader can work with Python3 and on other platforms

3. Make it compatible with other Python libraries

4. Allow Django users to choose the files that should be watched

5. Integrate the Watchman package and its Python wrapper into DjangoI

For Phase 1, I will write a prototype that can replace the current autoreloader. In the second phase, I'll come up with another one capable of handling cached files and excluding certain types of files based on the configuration, and the final version will be a cross-platform autoreloader that works with most of the Python libraries. Here's my development schedule:

• May 30 – June 8: Surveying the functions implemented in autoreloader.py, researching the dependency of the functions included, and constructing the development environment. 
• June 8 – June 17: Play with the Watchman library; design a way to interact with Watchman (probably as a standalone process); write wrappers for Watchman. Make a list of files that is not monitored by default (e.g. sqlite files or files under the media directory) and make sure it works with settings.py. 
• June 17 – June 25: Implement the auto-reloader from scratch; make sure the new reloader can work with Django properly (particularly can survive a syntax error and would not be affected by cached files.)
• July 26 – June 30: Phase 1 evaluations
  
• July 1 – July 8: Add appropriate benchmarks and check whether if the new implementation meets the expectation. Test whether if the new implementation can detect the addition of a file in the trivial cases and work with applications like git. 
  
• July 8 – July 15: Figure out how to monitor cached files in Django; add an option that enables Watchman to ignore all cached files. Check whether if this solution works with cached templates. 
  
• July 15 – July 24: Designate options to control the files monitored by Watchman with a standard include / exclude mechanism; write a function to read the relevant portion in the configuration file.  
  
• July 24 – July 28: Phase 2 evaluations
  
• August 2  – August 15: Test compatibility with other libraries including Jinja2 and Jython; revise the code if necessary. Test the autoreloader on Windows and Macintosh OS.
  
• August 15  – August 21: Integrate the Watchman library and its wrappers into Django (might need to submit pull requests to the pywatchman project.)
  
• August 21 – August 29: Final evaluations

Thanks for reading my proposal. If you have any questions or concerns, please do not hesitate to comment. Any advice and suggestions would be greatly appreciated!

[Brice PARENT]

Hi David,
I think it's a good idea to improve what we have now, mostly for the
problems you described.
Don't you think it would be a better idea to make a reloader which
doesn't depend on Django at all? You could make a library working with a
settings file, and create a base settings file for Django projects.
You could define whatever should be called to start and restart the
server, and be able to define file types and folders that should be seen
as changes that require a restart or not. It could then be used easily
with any kind of Python project, and even allow specific behaviours,
like launching a server reload when a new plugin is uploaded to a
specific folder, or when one of them is updated.
Every Python framework could then embed their own base settings file,
which would allow this project to touch a wider audience.

- Brice

[Aymeric Augustin]

Hello,

> On 29 Mar 2017, at 01:05, qingn...@gmail.com wrote:
>
> the best library to replace Watchdog is Watchman, a library that supports both of these three platforms (for Windows it's still in the alpha stage.)

Django currently doesn't use Watchdog. Watchdog is a cross-platform Python library that can be used for implementing an autoreloader. It's a plausible alternative to Watchman, likely easier to integrate, but Watchman has some fancy features that I'd love to have available -- like waiting for git to finish its operations before notifying a change.

> It runs as a standalone process and has a python wrapper library that is incompatible with Python 3. Therefore, one of the goals is to figure out how to interact with this library.

In order to be selected for this project, you have to convince us that you can solve this problem. At a minimum you must prove that you understand where the difficulties lie and suggest some possible approaches to tackle them.

> So my goals would be:
>
> • Replace the old auto-reloader with the Watchman version

Since installing watchman isn't trivial, you need to keep a pure-Python alternative so that beginners get an acceptable experience out of the box. This alternative may be the current "poll every second approach" or could be backed by watchdog, assuming it's pip-installable without complicated dependencies (e.g. a C compiler and development headers for a bunch of libraries) on all operating systems.

> • Make sure that the new auto-reloader can work with Python3 and on other platforms

Did you mean "work with Python 2" ? In any case, this isn't necessary, as you'd be working off the master branch which no longer supports Python 2.

> • Make it compatible with other Python libraries

> • Allow Django users to choose the files that should be watched
> • Integrate the Watchman package and its Python wrapper into Django

Watchman should be installed separately by developers who wish to use it. Django should use it when it's available (and there should be a flag to disable that if for any reason someone has watchman installed systemwide but doesn't want to use it).

I hope this helps!

--
Aymeric.

[Carl Meyer]

Anyone working on this project should at least be aware of
https://github.com/Pylons/hupper (based on work David Glick and I
originally did in https://github.com/carljm/wsgiwatcher), which aims to
be a framework-agnostic solution to this problem for any Python web
project. Docs at http://docs.pylonsproject.org/projects/hupper/en/latest/

Carl

[qingn...@gmail.com]

Hi Carl,

  Thanks for mentioning this awesome project! I saw it in one of the discussions but did not take a close look. I'll definitely check this out and try to integrate wsgiwatcher/watcher.py into Django.

[qingn...@gmail.com]

Hi Brice,

  Thanks for the suggestion! I'm not sure whether if I can successfully implement a library like you've mentioned, so for now I might stick with the Watch* libraries available. I really love the goals you've listed and would add these to my proposal.

David Ma

[Melvyn Sopacua]

On Thursday 30 March 2017 22:01:00 qingn...@gmail.com wrote:

> Hi Carl,

> Thanks for mentioning this awesome project! I saw it in one of the

> discussions but did not take a close look. I'll definitely check this

> out and try to integrate wsgiwatcher/watcher.py into Django.

>

> On Thursday, March 30, 2017 at 7:47:13 AM UTC-7, Carl Meyer wrote:

> > Anyone working on this project should at least be aware of

> > https://github.com/Pylons/hupper

> > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FPylons%2Fhu

> > pper&sa=D&sntz=1&usg=AFQjCNGVwtqvdo53UFfK80kaQ1qxL7ST8Q> (based on

> > work David Glick and I

> > originally did in https://github.com/carljm/wsgiwatcher

> > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fcarljm%2Fws

> > giwatcher&sa=D&sntz=1&usg=AFQjCNGqLfsmxsp37Lng7_d_DaVzja9c6Q>),

> > which aims to

> > be a framework-agnostic solution to this problem for any Python web

> > project. Docs at

> > http://docs.pylonsproject.org/projects/hupper/en/latest/

> >

> > Carl

I was going to suggest the same, because you already get watchdog support for free and can probably build on that integration to pick up watchman. Also, it's used in devpi, so it gets some scruteny from there (and devpi is awesome to distribute internal or augmented Django apps amongst projects).

--

Melvyn Sopacua

[qingn...@gmail.com]

Hi Aymeric,

  Thanks for this detailed and informative response! I will try to figure out the difficulties of integrating Watchman during the weekend. For the pure-Python solution, I might implement a standalone autoreloader based on Carl's work and replace the current one. Does this look good to you?

David Ma

[Aymeric Augustin]

Hello,

On 1 Apr 2017, at 05:52, qingn...@gmail.com wrote:

For the pure-Python solution, I might implement a standalone autoreloader based on Carl's work and replace the current one. Does this look good to you?

Perhaps.

It depends on the details (features, installation requirements, cross platform support, etc.).

It's up to you to argument in your proposal why this would be a good solution.

Best regards,

-- 

Aymeric.

[qingn...@gmail.com]

Hi Carl,

  I don't quite understand why get_module_paths() in your wsgiwatcher project is returning a list of python module paths. I thought it would return the directory that needs to be monitored. Could you please tell me how this part works? Thanks.

David Ma

On Thursday, March 30, 2017 at 7:47:13 AM UTC-7, Carl Meyer wrote:

[qingn...@gmail.com]

Hi Aymeric,

  I feel like using Watchman might be the easiest solution. It keeps triggering the command once an event occurs, so autoreload.py does not need to restart the server in case of a syntax error. Pywatchman does not work on Python 3 and has some dependency problems, so I'm planning to use the command line interface of the Watchman library. Watchdog only triggers the given command once, so it cannot survive a syntax error and requires the auto-reloader to deal with multi-threading, the most challenging part in this project (likely to enter the same problem described in #2330.) Therefore, I'd start with Watchman and see how to implement this with Watchdog later. How do you feel about this plan? 

Cheers,

David Ma

[Carl Meyer]

Hi David,

I wouldn't bother with the wsgiwatcher repo; it's proof-of-concept code
from one sprint's worth of hacking, not used by anyone. Look at hupper
instead, it is based on wsgiwatcher, but much changed, and actually used
in production.

Carl

On 04/02/2017 04:27 PM, qingn...@gmail.com wrote:
> Hi Carl,
> I don't quite understand why get_module_paths() in your wsgiwatcher
> project is returning a list of python module paths. I thought it would
> return the directory that needs to be monitored. Could you please tell
> me how this part works? Thanks.
>
> David Ma
>
> On Thursday, March 30, 2017 at 7:47:13 AM UTC-7, Carl Meyer wrote:
>
> Anyone working on this project should at least be aware of

> https://github.com/Pylons/hupper <https://github.com/Pylons/hupper>

> (based on work David Glick and I
> originally did in https://github.com/carljm/wsgiwatcher

> <https://github.com/carljm/wsgiwatcher>), which aims to

> be a framework-agnostic solution to this problem for any Python web
> project. Docs at
> http://docs.pylonsproject.org/projects/hupper/en/latest/

> <http://www.google.com/url?q=http%3A%2F%2Fdocs.pylonsproject.org%2Fprojects%2Fhupper%2Fen%2Flatest%2F&sa=D&sntz=1&usg=AFQjCNHhMlzZ6K6HLWsxUxOPanWzxLeMlg>
>
>
> Carl
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com
> <mailto:django-develop...@googlegroups.com>.
> To post to this group, send email to django-d...@googlegroups.com
> <mailto:django-d...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/4fac9a8a-231d-41c5-a5fb-6eaeb2bd148f%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/4fac9a8a-231d-41c5-a5fb-6eaeb2bd148f%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Aymeric Augustin]

Hello David,

This reasoning makes sense.

You say that watchman "keeps triggering the command once an event occurs". This is correct; as a consequence "the command" must stop the currently running instance of the development server and start a new one. This requires overhauling significantly the current implementation of the development server.

You need to understand the two-process structure of the current autoreloader and the workarounds for allowing runserver to start with invalid settings to get a good grasp on the requirements for a good developer experience. I hope we can do away with these workarounds with a better architecture.

Best regards,

-- 

Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3c284301-15e9-44d9-9b7c-2d866b7f7d13%40googlegroups.com.