Add custom autoreload file tracking options setting

[Bobby Mozumder]

Hi,

Right now, Django only tracks Python module files for autoreload during development. As a project starts to include more custom include files, such as Javascript, SQL, Makefiles, etc.., the autoreload function doesn't apply to these.

For my use case, I have custom view functions that call in separate SQL & Javascript files. (I don’t use the template system.)

If I edit these Javascript & SQL files, the Django server doesn’t autoreload.

So, I made a pull-request where we can add a manual list of files to track for autoreload: https://github.com/django/django/pull/7791

In your project's settings.py file, assign a variable TRACK_FILES containing a list of full file paths to track. This will track files to autoreload the development run server as these files are updated.

Is this OK? This pull request is a basic option and I’m sure it can get more complicated than that (directory tracking, Makefiles, Javscript builds, etc..)

-bobby

[Tim Graham]

Could you give us a code snippet (sample view, perhaps) demonstrating how this caching happens?

[Bobby Mozumder]

OK here is some example code snippet where I load prepared SQL statements:

class FastDetailView(DetailView,FastView):

    c = connection.cursor()

    SQL_VIEW_DIRS = {

        'fashion': (

            'include/sql/materializedviews/headlines',

            'include/sql/materializedviews/latestCollections',

            'include/sql/materializedviews/allSeasons',

            'include/sql/materializedviews/fullSeason',

            'include/sql/materializedviews/gallery',

            'include/sql/materializedviews/indexView',

            'include/sql/materializedviews/cover',

            'include/sql/materializedviews/latestSeasonView',

            'include/sql/materializedviews/seasonView',

            'include/sql/materializedviews/collectionView',

            'include/sql/materializedviews/latestCollectionsJSON',

            'include/sql/materializedviews/collectionCardJSON',

            'include/sql/materializedviews/indexJSON',

            'include/sql/materializedviews/categoryJSON',

            'include/sql/materializedviews/articleJSON',

            'include/sql/triggers/globals',

            'include/sql/triggers/brand',

            'include/sql/triggers/collection',

            'include/sql/triggers/collectionlookassignment',

            'include/sql/triggers/cover',

            'include/sql/triggers/look',

            'include/sql/triggers/photo',

            'include/sql/triggers/season',

            'include/sql/triggers/fashion_headlinesviewmat',

            'include/sql/triggers/fashion_latestcollectionsviewmat',

            'include/sql/triggers/fashion_allseasonsviewmat',

            'include/sql/triggers/fashion_fullseasonviewmat',

            'include/sql/triggers/fashion_galleryviewmat',

            'include/sql/triggers/fashion_coverviewmat',

            'include/sql/triggers/fashion_indexviewmat',

            'include/sql/triggers/fashion_latestseasonviewmat',

            'include/sql/triggers/fashion_seasonviewmat',

            'include/sql/triggers/fashion_collectionviewmat',

            'include/sql/triggers/fashion_collectioncardjsonviewmat',

        ),

        'analytics': (

            'include/sql/analytics',

        ),

    }

    MATERIALIZED_VIEWS = True

    @classmethod

    def prepare_db_queries(self):

        logger.info('Reading fashion prepared SQL statements')

        cursor = connection.cursor()

        for sql_view_dir in SQL_VIEW_DIRS['fashion']:

            file_name = sql_view_dir + '/prepare.sql'

            try:

                with open(file_name, 'r') as file:

                    sql_prepare=file.read().strip()

                    if sql_prepare:

                        cursor.execute(sql_prepare)

            except (OSError, IOError) as e:

                pass

            except e:

                logger.info('Error reading SQL file: %s' % file_name)

                raise e

            if MATERIALIZED_VIEWS:

                file_name = sql_view_dir + '/prepare_materialized.sql'

                try:

                    with open(file_name, 'r') as file:

                        sql_prepare=file.read().strip()

                        if sql_prepare:

                            cursor.execute(sql_prepare)

                except (OSError, IOError) as e:

                    pass

                except e:

                    logger.info('Error reading SQL file: %s' % file_name)

                    raise e

It’s a custom view class that basically reads SQL from a separate list of files on initialization, and executes those SQL files.  

If I edit these SQL files, it won't restart the development server.

-bobby

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/c857c334-6388-4e10-8367-ffbee08acc10%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Graham]

When is prepare_db_queries() called? During a request/response cycle? I doesn't look like any caching is happening so I still doesn't see why the server needs to restart to pickup changes to the SQL files.

[Adam Johnson]

For that use case I'd suggest just re-executing prepare_db_queries on every page view when DEBUG=True. This is similar to how Django's template loaders work without the cached loader wrapping them.

To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/c857c334-6388-4e10-8367-ffbee08acc10%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/605A1262-888E-4039-8071-F0F19640F1C0%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Adam

[Bobby Mozumder]

It’s actually called once on app startup during DB connection via a Signal.

Here is my app.py:

from django.apps import AppConfig

from .signals import *

from django.utils import autoreload

class FashionAppConfig(AppConfig):

    name = 'fashion'

    verbose_name = "Fashion"

And here is my signals.py:

from django.dispatch import receiver

from django.db.backends.signals import connection_created

from .page import factory as pageFactory

from .api import APIFactory

@receiver(connection_created, dispatch_uid="dbConnectionInitiated")

def prepareSQL(sender, **kwargs):

    pageFactory.prepare_db_queries()

    APIFactory.prepare_db_queries()

    pass

My FastDetailView class is a subclass of factory , and this is where prepare_db_queries happens.

I also have separate code that loads in Javascript into this FastDetailView class.

Basically I don’t read html from the template at all during the Request/Response cycle.  I instead pre-load all my DB Queries and Javascript.

(This whole process helps speed up the view response generation times.  I can generate a full page in about 4ms, and that includes GZip.)

-bobby

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ceaf7dcc-7cf1-42e1-be76-a528d394da63%40googlegroups.com.

[Bobby Mozumder]

That’ll make development server access times really slow.  There’s a pretty long Makefile also that builds Javascript.  We really shouldn’t have to rebuild Javascript on every page view, especially since I call interactive API requests with these views.

-bobby

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM3NKV%2BpCHFX0LRR-H%2B4RxU-6y%3DCj%3DjHwmvVk5D_rsCa7Q%40mail.gmail.com.

[Aymeric Augustin]

Hello Bobby,

> On 4 Jan 2017, at 22:25, Bobby Mozumder <bmoz...@gmail.com> wrote:
>
> It’s actually called once on app startup during DB connection via a Signal.

Unless I missed something, since the development server creates a new connection to the database for each request — Python’s threaded socket server creates a new thread for each connection and database connections are thread local in Django — prepared statements will be refreshed for each request. So I don’t think you need autoreloading here.

> (This whole process helps speed up the view response generation times. I can generate a full page in about 4ms, and that includes GZip.)

Off topic, but I’m jealous…

<rant>

I still believe we should stop maintaining an autoreloader as soon as possible. Django’s autoreloader is annoyingly slow, highly inefficient, moderately well designed, and a gigantic pain to maintain. I’m more scared of django.utils.autoreload than of django.db.models.related before it was cleaned up.

I wish one day someone will take the time to write a good autoreloading dev server based on watchman. This would solve the problem discussed here because watchman watches all files in the current directory. The correct way to do this is to throw away the current design and start from scratch.

Watchman is smart enough to wait until you’ve finished a git operation to trigger a reload. Once such polished tech has become available, trying to compete with a thread that checks the mtime of all known files every second isn’t funny anymore. In fact it’s just sad.

</rant>

While I don’t care much about django.utils.autoreloader because I want to kill it, I’m reluctant to add public APIs such as the proposed setting, which may not make sense with an hypothetical better autoreloader.

I’m -0 on the change. I could move to +0 if I understood why the use case described here requires watching additional files.

Best regards,

--
Aymeric.

[Bobby Mozumder]

> On Jan 4, 2017, at 4:47 PM, Aymeric Augustin <aymeric....@polytechnique.org> wrote:
>
> Hello Bobby,
>
>> On 4 Jan 2017, at 22:25, Bobby Mozumder <bmoz...@gmail.com> wrote:
>>
>> It’s actually called once on app startup during DB connection via a Signal.
>
> Unless I missed something, since the development server creates a new connection to the database for each request — Python’s threaded socket server creates a new thread for each connection and database connections are thread local in Django — prepared statements will be refreshed for each request. So I don’t think you need autoreloading here.

Hmm I thought the test server used permanent connections if CONN_MAX_AGE was set to None? Could have sworn it was that way for a while?

In any case, I just tried the Javascript & CSS Makefile with a simple one line change to one file, and it took 4 seconds. That’s going to be way too long to do on each request.

How do people serve development Javascript & CSS files? These days Javascript & CSS involves a large build process. Are we forced to manually restart the development server every time Javascript changes?

I think it should be just like changing Python files and it autoreloads Javascript & CSS as it changes.

Also, I haven’t used the Django Template system or Jinja in over a year. Does the Django development server restart if you edit Templates? Because it should also do that.

This is what I’m getting at with this autoreload.. it should include the full system, not just the Python files.

>
>> (This whole process helps speed up the view response generation times. I can generate a full page in about 4ms, and that includes GZip.)
>
> Off topic, but I’m jealous…
>

Also, my average page generates in 1-2ms, and fully cached html serves in 200 microseconds, and that include Gzip, since I cache Gzip (which makes my cache 10x bigger). There’s lots that I can contribute here, including non-blocking analytics stored in teh database. If I can make it “generic” enough I’ll try to publish it, but the code is really manual. Maybe instead of a formal toolset, I could post a methodology manual? It’s really tedious though, involving materialized database views & all sorts of other techniques.

BTW My site is at http://www.futureclaw.com

> <rant>
>
> I still believe we should stop maintaining an autoreloader as soon as possible. Django’s autoreloader is annoyingly slow, highly inefficient, moderately well designed, and a gigantic pain to maintain. I’m more scared of django.utils.autoreload than of django.db.models.related before it was cleaned up.
>
> I wish one day someone will take the time to write a good autoreloading dev server based on watchman. This would solve the problem discussed here because watchman watches all files in the current directory. The correct way to do this is to throw away the current design and start from scratch.
>
> Watchman is smart enough to wait until you’ve finished a git operation to trigger a reload. Once such polished tech has become available, trying to compete with a thread that checks the mtime of all known files every second isn’t funny anymore. In fact it’s just sad.
>
> </rant>
>
> While I don’t care much about django.utils.autoreloader because I want to kill it, I’m reluctant to add public APIs such as the proposed setting, which may not make sense with an hypothetical better autoreloader.
>
> I’m -0 on the change. I could move to +0 if I understood why the use case described here requires watching additional files.
>

I guess I could just use Watchman to restart the Django development server as needed?

I’m OK with that.. my only goal is to make sure I don’t have to restart Django manually every time I edit CSS/Javascript/SQL/HTML.

Is that the recommended workflow for this? I never looked into outside tools to manage this, but it seems it would be a lot easier to add a list of files or directories to our settings and have Django manage autoreload, than to bring in an outside tool. (I can add directories to watch to the pull request if needed)

> Best regards,
>
> --
> Aymeric.

>
> --
> You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.

> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/BAAB82B3-974C-4240-B9C6-E81F04D9604F%40polytechnique.org.

[Adam Johnson]

How do people serve development Javascript & CSS files?  These days Javascript & CSS involves a large build process.  Are we forced to manually restart the development server every time Javascript changes?

Django just serves them from the filesystem, and doesn't cache them itself. You reload the page in your browser, and Django serves the files from disk

Does the Django development server restart if you edit Templates?  Because it should also do that.

Similarly, by default templates aren't cached, so on each view the dev server loads the templates from disk, parses them, and renders them.

 

On 4 January 2017 at 22:31, Bobby Mozumder <bmoz...@gmail.com> wrote:

> To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.

> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/BAAB82B3-974C-4240-B9C6-E81F04D9604F%40polytechnique.org.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8ABA81DF-807E-4B22-ACCF-C4D207A3E5E4%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Adam

[Bobby Mozumder]

On Jan 4, 2017, at 5:40 PM, Adam Johnson <m...@adamj.eu> wrote:

How do people serve development Javascript & CSS files?  These days Javascript & CSS involves a large build process.  Are we forced to manually restart the development server every time Javascript changes?

Django just serves them from the filesystem, and doesn't cache them itself. You reload the page in your browser, and Django serves the files from disk

Does the Django development server restart if you edit Templates?  Because it should also do that.

Similarly, by default templates aren't cached, so on each view the dev server loads the templates from disk, parses them, and renders them.

 

Does it do the same under production environments - read and interpret each template file on every request?  

Is there interest in making a much faster version of this flow?

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM2E6JQtE9Jc_Ff-LsyDH2tj6v_YGjfmVdbe8p8%2BM4dvHg%40mail.gmail.com.

[Tim Graham]

No, there's a cached template loader: https://docs.djangoproject.com/en/dev/ref/templates/api/#django.template.loaders.cached.Loader

> To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.

> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/BAAB82B3-974C-4240-B9C6-E81F04D9604F%40polytechnique.org.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8ABA81DF-807E-4B22-ACCF-C4D207A3E5E4%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Adam

[Aymeric Augustin]

On 4 Jan 2017, at 23:31, Bobby Mozumder <bmoz...@gmail.com> wrote:

> I guess I could just use Watchman to restart the Django development server as needed?

If you find a way to tell watchman to run `django-admin runserver --noreload` and restart it whenever a file in the current directory changes, that should do the job.

Unfortunately the APIs exposed by watchman don’t make this trivial. They’re intended to run a build process that will terminate, while the development server will keep running.

--
Aymeric.

[Sam Willis]

Could one options be to replace the current devserver with the one from Werkzeug? It already uses watchdog (similar to watchman) for monitoring file system events and is well maintained. With Django now allowing dependancies, this seems like something that doesn't necessarily need to be developed internally.

The Werkzeug devserver also has some niceties like an interactive debugger and ssl hosting with an automatically issued self signed certificate. It could be implemented behind the current management command relatively easily.

There is already an implementation as part of django-extentions that I believe is well used and battle tested - http://django-extensions.readthedocs.io/en/latest/runserver_plus.html 

[Florian Apolloner]

On Thursday, January 5, 2017 at 1:38:44 PM UTC+1, Sam Willis wrote:

Could one options be to replace the current devserver with the one from Werkzeug? It already uses watchdog (similar to watchman) for monitoring file system events and is well maintained. With Django now allowing dependancies, this seems like something that doesn't necessarily need to be developed internally.

Watchdog seems to require a .c extension file on MacOSX at least, so as long as there are no wheels, that does not seem to be an option.

The Werkzeug devserver also has some niceties like an interactive debugger and ssl hosting with an automatically issued self signed certificate. It could be implemented behind the current management command

Self-Signed SSL sounds somewhat nice, I am -0 to -1 for the debugger -- I've seen to many sites out there running with DEBUG=True, enabling RCE ootb seems to be pretty horrible.

Cheers,
Florian

[Michael Manfre]

I think anyone running devserver in prod deserves a breakpoint induced outage.

Regards,
Michael Manfre

[Stratos Moros]

> I’m -0 on the change. I could move to +0 if I understood why the use
> case described here requires watching additional files.

A different use case we've run into is non-python configuration files.
Our settings.py reads a few variables off a toml file and it would be
nice if we could configure runserver to reload automatically when that
toml file changes.

That said, I agree that auto reloading should probably live outside
Django.

[Josh Smeaton]

> I am -0 to -1 for the debugger -- I've seen to many sites out there running with DEBUG=True, enabling RCE ootb seems to be pretty horrible.

But it's so incredibly useful. And we already show the django debug page for errors with DEBUG=True that exposes enough secrets to allow a sufficient attacker to gain access. If we could, by default, block the debugger in a similar way that django debug toolbar does, would that be appropriate? That is, checks for DEBUG and HOST etc?

[Florian Apolloner]

On Thursday, January 5, 2017 at 11:14:08 PM UTC+1, Josh Smeaton wrote:

> I am -0 to -1 for the debugger -- I've seen to many sites out there running with DEBUG=True, enabling RCE ootb seems to be pretty horrible.

But it's so incredibly useful. And we already show the django debug page for errors with DEBUG=True that exposes enough secrets to allow a sufficient attacker to gain access.

What exactly? Last time I checked SECRET_KEY and other dangerous stuff was blanked out as good as possible. Do not get me wrong, it is certainly not "safe" to show the debug page, but leaking information versus RCE is a different story… Even if the debug page leaks enough information to login as admin, you do not neccessarily compromise the OS, whereas the werkzeug debugger gives you at least user access to the OS.

And truth to be told, I can count the instance where the werkzeug debugger would have been useful on one hand -- the traces are usually more than enough.
 

If we could, by default, block the debugger in a similar way that django debug toolbar does, would that be appropriate? That is, checks for DEBUG and HOST etc?

I am not deploying debug toolbar anywhere, so I cannot tell. That said, people have DEBUG=True etc in prod, if we open those installations for RCE, that would still suck (defense in depth and everything). Also https://labs.detectify.com/2015/10/02/how-patreon-got-hacked-publicly-exposed-werkzeug-debugger/ -- even though it is old, it is a nice sign that it happens…

Cheers,
Florian