Dynamic app loading

691 views
Skip to first unread message

Christian González

unread,
Jul 29, 2018, 7:44:31 AM7/29/18
to django-d...@googlegroups.com
Hello DjangoDevs,

I'm new here to this group, and to be honest, just a "fake" developer.
Doing that is not my main job. So please be patient with my maybe
BadIdea(tm).

Another warning: much text. Hint: there is a TL;DR at the end.

I stumbled upon a "missing feature" in Django, which I call "dynamic"
app loading, a while ago, since I try to create a Django based
application which can dynamically add plugins to itself.

I first tried to google the internet, and found many Stackexchange Q&A
where this topic is handled, but either in an insufficient way, not
applicable to Django 2.0, or else.

Best ones:
https://stackoverflow.com/questions/24027901/dynamically-loading-django-apps-at-runtime
https://stackoverflow.com/questions/7933596/django-dynamic-model-fields

And my own question with no answer so far:
https://stackoverflow.com/questions/51234829/dynamic-django-apps

So I began implementing my own way of handling this.

Let me first tell a "user story", so you can imagine what I mean.

My application should more or less be a framework that provides a
loosely-coupled bunch of modules working together, with a dependency
tree and versioning. There is a "core module", and others that depend on
it (e.g. "notifications" etc.). Third party apps should be possible, and
something like an "app store" should be created to dynamically download
apps from within the program, and add that functionality to the main
application.

So, my first approach was creating zip files with a predefined structure
(models.py, schema.py, views.py, client stuff etc.), and tried to load
this code during runtime. I soon realized that I had to re-implement
most of the stuff Django does anyway, and doing migrations isn't an easy
task when done barefoot.

I then changed my mind, and found the best way of having "dynamic
plugins" is using "Django apps" as plugins.
But: Django apps are not pluggable. They have to be inserted hardcodedly
into INSTALLED_APPS to have a predictable order of loading. Yes, I've read
https://groups.google.com/forum/#!searchin/django-developers/app-loading%7Csort:date/django-developers/_iggZzrYtJQ/FWFPgCflSnkJ
- and I "kind of" understand the Django setup() process (see later).

I started to fiddle with INSTALLED_APPS, as recommended in Stackexchange
etc., and dynamically searched a "plugins" directory to add some plugins
into the list of other apps, just by extending INSTALLED_APPS. Django
sees no difference, has no cache problems and happily loads all my plugins.

BUT: this is no way dynamic. First thing I recognized is: You can't
simply call DB requests anywhere near the settings.py loading time.
Because there is no DB at that moment, let alone models. I then stuffed
the code into AppConfig.ready() of the core app, and was a step further,
even if it's not recommended to call models there:
I need to use models there: I want to check if a plugin app on disk is
"deactivated", and NOT load it in that case. Aaaargh. Back to the start.

* In settings.py, you can tell Django to dynamically load plugins, using
disk IO code there. BadIdea(tm).
* in AppConfig.ready(), you can use models, even if discouraged, but
it's too late to find "plugin apps" now and add it to INSTALLED_APPS.
* in a middleware, you can use Models (somehow), but same problem. It's
too late to add models.

I at least managed to add this plugins' URLs to my main urls by
providing a plugin hook in the main urls.py which is called in all the
plugins:

main urls.py:

    PluginManager.load_plugin_submodule('urls')

    for patternplugin in IURLPattern:
        urlpatterns += patternplugin.get_patterns()

Where IURLPattern is a "Interface" class that can be used in plugins:

    @implements(IURLpattern)
    class FooPluginURLs:
        def get_patterns(self():
            return [path('foo/', FooAPIView.as_view(), name='test')]

So the main URLs add all dynamically added urls. But, like I said: no
way dynamic, as it's fully deterministic at Django start.

What I wanted is: Danymically download such an app, stuff it into the
Django system and - bling - it works, with models, URLs, and everything,
after a "django_reload" magic.

Ok, next: I pimped my PluginManager, created a middleware that starts at
Django start and loads all plugins. So, no INSTALLED_APPS tweaks, done
in middleware, after all models are available.

The plugin manager now reimplements the django.apps.populate() method
and does the same things again, bypassing checks of already loaded apps.
This works somehow(tm). But there are many problems remaining, and I
think it is worth rethinking the whole Django app loading process to
make it more dynamic:


TL;DR:
https://code.djangoproject.com/ticket/29554
Could Django be changed to not load apps hardcodedly at startup, but
load them "dynamically", e.g. reusing the existing "apps.populate()" method?

I think one of the best approaches would be: let the user call
apps.populate() too, like setup() does it with INSTALLED_APPS. But let
the user dynamically add lines to that variable. or even better: Let the
user load one app at a time, like by extending the "Apps" class:

django.apps.load_app("my.own.app.apps.MyAppConfig")

Like Aymeric states in the issue: there has to be discussed what Django
needs to do about dependencies. ATM app loading is fully deterministic
(order defined by INSTALLED_APPS).
But I think this is not necessary for dynamically added apps. Django
CANNOT decide that, as it would imply a sophisticated dependency and
versioning schema of apps. Which IMHO Django should NOT provide, to be
as flexible as possible. But it SHOULD provide the ability to create
such a framework, which it does not at the moment.
The framework I try to implement should be responsible for loading the
apps in the right order (because of here: dependency tree, versions,
etc. - but this could be done completely different with another Django
application!)

So, somtehing like an "apps.load_app(dotted_appname, reload:bool =
False)" method should only be responsible for:

[X] loading AppConfigs (check if already loaded, reload?)
    -> code can be reused, as populate() ATM just creates a possibly
missing AppConfig and creates an internal list of apps
[X] loading Models (check if already loaded, reload?)
    -> code can be reused as well here.
[ ] invalidate cache (models, AppConfigs etc)

and consequently, there must be a apps.unload_app(dotted_appname)
method, to remove that again.

Aymeric writes in the issue:
> I'd like to see a thorough discussion of the pros and cons before
making this decision as well as an analysis of which caches need to be
invalidated and how this could happen.

I'm the wrong one to give input here. No knowledge of Django internals.

Please tell me what you think of that feature.

Raphael Michel

unread,
Jul 30, 2018, 2:53:29 AM7/30/18
to Christian González, django-d...@googlegroups.com
Hi Christian,

we are doing such a thing for quite a while in our open source project
pretix[1]. I'm not sure if its something that Django needs to do
better, since requirements for this tend to deviate a lot and there's
already a solid basis in the Python packaging toolchain to start from:
To discover apps/plugins, we rely on setuptools' entry point
feature[2], which allows us to easily load all compatible apps
installed in the local Python requirement.

We then disable/enable plugins on a per-client level by using a custom
subclass from Signal[4] and automatically wrapping all views with a
decorator that makes them conditional[6] through some URLConf inclusion
tricks[5].

From a plugin author perspective, this makes for a pretty clean view[7]
and we are pretty satisfied with the approach. You can also watch me
explaining it here: https://www.youtube.com/watch?v=5NxRdzLTFik

Cheers
Raphael

[1] https://github.com/pretix/pretix
[2] https://packaging.python.org/specifications/entry-points/
[3]
https://github.com/pretix/pretix/blob/master/src/pretix/settings.py#L264
[4]
https://github.com/pretix/pretix/blob/master/src/pretix/base/signals.py#L21
[5]
https://github.com/pretix/pretix/blob/master/src/pretix/multidomain/maindomain_urlconf.py#L23
[6]
https://github.com/pretix/pretix/blob/master/src/pretix/multidomain/plugin_handler.py
[7] https://docs.pretix.eu/en/latest/development/api/index.html

Am Sun, 29 Jul 2018 13:26:04 +0200
schrieb Christian González <christian...@nerdocs.at>:

Christian González

unread,
Jul 31, 2018, 5:18:51 PM7/31/18
to django-d...@googlegroups.com
Hi Raphael,
> we are doing such a thing for quite a while in our open source project
> pretix[1].
Whow, I'm quite impressed. Never stumbled upon that. I'll recommend that
for some collegues (to use it...)
But about the internals: Nice, this was the first way I implemented that
as well, and maybe I will come back to that again. You have to restart
the server as well if you want to enable a plugin, right?

> [...]
> To discover apps/plugins, we rely on setuptools' entry point
> feature[2], which allows us to easily load all compatible apps
> installed in the local Python requirement.

From a security POV, I thought: who hinders anyone to write malicious
code and provide an entry point for your system? maybe in a library you
install... (but this is merely a philosophic question... don't bother.)

* You (deprecatedly) searched for plugins in all installed apps, check
which one has a "PretixPluginMeta" attr. This is quite nice. I did just
by appname.startswith('foo'). Less pythonic... ;-)
But it's now replaced by entry points discovery. I had this too, (and
will maybe come back to it). Setuptools' entry points discovery is a bit
slow, but for the server start that doesn't matter, it's just once.
(except development, you'll have to restart your server often)
Your entrypoint is e.g. "pretix_pages=pretix_pages:PretixPluginMeta" - I
don't understand that - this is an inner class of PluginApp. How can
you  access it directly via the entry point? But generally I understand
the workflow.

* How do you use signals? e.g.

    @receiver(footer_link, dispatch_uid="pages_footer_links")
What does this mean? How do you pass this to your frontend? Could you
describe this in just a few words?

Really, I had about 70% of your system already (more or less) working)
in mine. And as I will proceed, I think I will come back to the same
system as well.
I'd like to add channels to the mix as well to enable a Vue frontend.
Instead of copying your code and forking the parts that I need, It would
be better OpenSource practice to maybe build a "plugin system" that is a
library and can be used of both - this is not very difficult IMHO,
b'cause most of it is already there.

Just tell me what you think about it - I can live with both - I just
think that stability is better if more projects rely on one library and
find security flaws and bugs alternatively ;-)

I have written a IMHO better plugin handler:
https://gitlab.com/nerdocs/medux/MedUX/blob/develop/medux/extensionsystem/__init__.py


It defines defined before to have custom attrs., like:

class URLPattern(Interface):
    def get_patterns(self):
        pass

and in the plugin, this way you can use any class and decorate it with

    @implements(URLPattern)
    class FooURLPattern:
        def get_patterns(self):
            return [path(...)]

And in Main urls.py:

for pattern in URLPattern:
    prl_patterns += pattern.get_patterns

This way all plugins can just "implement" predefined Interfaces, supereasy.


Greetings from Salzburg,
Christian
signature.asc

Raphael Michel

unread,
Aug 1, 2018, 3:32:11 AM8/1/18
to Christian González, django-d...@googlegroups.com
Hi,

Am Tue, 31 Jul 2018 23:18:32 +0200
schrieb Christian González <christian...@nerdocs.at>:
> Whow, I'm quite impressed. Never stumbled upon that. I'll recommend
> that for some collegues (to use it...)

Thanks =)

> But about the internals: Nice, this was the first way I implemented
> that as well, and maybe I will come back to that again. You have to
> restart the server as well if you want to enable a plugin, right?

If by enable you mean installing (in our terminology), yes. Normally,
you need to do something like

pip install pretix-myfancyplugin
python -m pretix migrate
python -m pretix rebuild
systemctl restart pretix

rebuild is a custom management command that compiles translation files,
SASS files and compressed JS files.

> From a security POV, I thought: who hinders anyone to write malicious
> code and provide an entry point for your system? maybe in a library
> you install... (but this is merely a philosophic question... don't
> bother.)

Nobody. But if you pip-install a library with malicious intents, you
have lost anyways, since its setup.py will likely be executed during
install (or at least some of its code will be executed once you import
it) and it can do whatever it wants anyway.

> But it's now replaced by entry points discovery. I had this too, (and
> will maybe come back to it). Setuptools' entry points discovery is a
> bit slow, but for the server start that doesn't matter, it's just
> once. (except development, you'll have to restart your server often)
> Your entrypoint is e.g. "pretix_pages=pretix_pages:PretixPluginMeta"
> - I don't understand that - this is an inner class of PluginApp. How
> can you  access it directly via the entry point? But generally I
> understand the workflow.

The part after the : in the entrypoint is probably not used at all at
the moment. With the entry point, I get the Django app. Then, with the
Django app, I get the inner class from the app registry:
https://github.com/pretix/pretix/blob/0b167aaa2cf607adddf4fe96c3da55b160b95602/src/pretix/base/plugins.py#L24

> * How do you use signals? e.g.
>
>     @receiver(footer_link, dispatch_uid="pages_footer_links")
> What does this mean? How do you pass this to your frontend? Could you
> describe this in just a few words?

These are basically standard Django signals, with only slight
modifications to be able to turn them off per-client:
https://docs.djangoproject.com/en/2.0/topics/signals/

I then use a few custom helpers like a custom template tag that allows
to call a signal directly from a template:

{% eventsignal event "path.to.signal" argument="value" ... %}

This is just to avoid repetitive code, however.

> I'd like to add channels to the mix as well to enable a Vue frontend.
> Instead of copying your code and forking the parts that I need, It
> would be better OpenSource practice to maybe build a "plugin system"
> that is a library and can be used of both - this is not very
> difficult IMHO, b'cause most of it is already there.
>
> Just tell me what you think about it - I can live with both - I just
> think that stability is better if more projects rely on one library
> and find security flaws and bugs alternatively ;-)

It would be great to have a library based on the ideas in pretix that
others can depend on. I'm not sure if pretix itself could use it, since
some of the things we do in the plugin API are very domain-specific,
like allowing plugin URLs to specify if they should be available only
when a shop is enabled or always.

A generic library would need to behave a bit different in design, and
changing that now will be hard for pretix, since there are already >50
plugins out there.

That said, I'd love to see it and maybe even collaborate on it!
> …
> This way all plugins can just "implement" predefined Interfaces,
> supereasy.

I like it! It is more framework-agnostic than our approach and
therefore probably more flexible. Our approach is a bit more tied to
Django's internals and therefore makes some things easier, but some less
clean.

Best
Raphael

Christian González

unread,
Aug 1, 2018, 7:14:53 PM8/1/18
to django-d...@googlegroups.com
Hello Raphael,

Thanks for your very, very helpful explanations.

Just another 2 questions: I saw that you did two things:

1.: you created a "class ReportsApp(AppConfig):" in
/plugins/reports/__init__.py - according to the Django docs[1] this
should go into /plugins/reports/apps.py
Why in the root plugin module?


and 2.: Why did you use "PretixPluginMeta"?
Why didn't you just use the AppConfig namespace?
Like:

class ReportsApp(AppConfig):
    name = 'pretix.plugins.reports'
    verbose_name = _("Report exporter")

    name = _("Report exporter") # name clash
    author = _("the pretix team")
    version = XYZ
    description = _("This plugin allows you to generate printable
reports about your sales.")

Except for the name clash, this would suffice?
Have you had a good reason for that? (I suppose...)

Thank you very much.

> [...]
>> I have written a IMHO better plugin handler:[...]
> I like it! It is more framework-agnostic than our approach and
> therefore probably more flexible. Our approach is a bit more tied to
> Django's internals and therefore makes some things easier, but some less
> clean.

Ok, thanks. I'm already working on extracting my plugin system and
making it more generic ;-)
I'll post a link when I can release a bit of code.

Christian

[1]
https://docs.djangoproject.com/en/2.1/ref/applications/#for-application-authors


signature.asc
Reply all
Reply to author
Forward
0 new messages