apps.populate deadlock

[John Bazik]

I'm porting a site from django 1.6 -> 1.8, and getting deadlock (no errors) at load time.  The problem is that apps.populate is getting called twice and hanging on the lock.  This is from executing "python manage.py help".  There are lots of moving parts, but a trace shows something like this:

django.setup -> apps.populate -> import  cmsplugin_zinnia -> django-cms discover_plugins -> load templates -> django-multisite template loader -> get site-id from locmem cache -> django.db.models.base.model_unpickle -> apps.populate

Since apps.populate isn't reentrant, and hangs when called recursively, it might be nice to have it throw an exception instead.

Can anyone suggest which component is most at fault above?  Any insights appreciated.

John

[John Bazik]

Here's the actual traceback.

John

[John Bazik]

Breaking it down:

• Django.setup wants to load all apps, and calls apps.populate
  
• Django-cms plugins register themselves at load time
• Django-cms plugin registration involves loading templates
• Django-multisite provides a template loader (for per-site templates)
• Django-multisite also monkey-patches Sites' cache (normally a dict) with a wrapper around a django cache
• The django-multisite template loader fetches the current site from cache when loading templates
• Django's LocMemCache backend fetches a pickled model from cache
• It is implicitly unpickled by django.db.models.base.Model which calls model_unpickle
• model_unpickle tests if apps.ready is true and if not, calls apps.populate

I could break the cycle any number of places, but it seems to me that all the apps are doing reasonable things (monkey patching aside), and django shouldn't be calling apps.populate to unpickle a model.  Perhaps that is unavoidable?

John

[James Schneider]

Have you tried the Django dev mailing list? You're dealing with the deep internals of Django, the black magic of which us lowly end users take for granted.

You may have better luck there, since they have a much better understanding of the gears and gizmos behind the scenes.

-James

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/f5271566-56ce-4171-a21e-fa24bbd73451%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shai]

Hi,

On Wednesday, January 13, 2016 at 12:54:59 AM UTC+2, John Bazik wrote:

Breaking it down:

• Django.setup wants to load all apps, and calls apps.populate
  
• Django-cms plugins register themselves at load time
• Django-cms plugin registration involves loading templates

I have never used Django-CMS, but according to your description, it (or its plugins) is doing what
applications should avoid doing since Django 1.7 and the app-loading refactor. Registrations which
involve interactions with applications -- and loading templates falls under this definition -- should
happen only in an app's AppConfig.ready(), after all the models are already loaded.
 

• Django-multisite provides a template loader (for per-site templates)

• Django-multisite also monkey-patches Sites' cache (normally a dict) with a wrapper around a django cache

• The django-multisite template loader fetches the current site from cache when loading template

Loading templates involves a trivial interaction with apps in the sense that apps' template directories
are used as template sources; but it can be deeper, because, as in your example, 3rd-party template
loaders may use models for loading them. It can be the site object as here, it can even be the
templates themselves stored in the database.
 

• Django's LocMemCache backend fetches a pickled model from cache
• It is implicitly unpickled by django.db.models.base.Model which calls model_unpickle
• model_unpickle tests if apps.ready is true and if not, calls apps.populate

I could break the cycle any number of places, but it seems to me that all the apps are doing reasonable things (monkey patching aside), and django shouldn't be calling apps.populate to unpickle a model.  Perhaps that is unavoidable?

The question of whether Django should be calling apps.populate() to unpickle a model (instance) will
be better discussed on django-developers, but I think that with well-behaved apps it is harmless.

HTH,
Shai.

[John Bazik]

Thanks Shai.  I'm waiting for a comment from the django-cms people.

John

[John Bazik]

Following up, the discussion continued on other lists:

https://groups.google.com/forum/#!topic/django-cms-developers/CNWHNviqS6Q
https://groups.google.com/forum/#!topic/django-developers/YvEzdQU4qUU

Problem ultimately resolved by a pending PR:

https://github.com/django/django/pull/6044

John