dynamically added INSTALLED_APPS

[Christian González]

Hi devs,

I am still evolving my GDAPS plugin system for Django (which works well
here), and try to overcome some shortcomings of Django that is not
really easy so solve, like dynamically added INSTALLED_APPS.

Given the need that a plugin that is installed/found via
setuptools/entry point (like GDAPS, or e.g. Pretix does), it can
sometimes be necessary that a plugin itself needs another app (like e.g.
django-money etc.) as "dependency".

Install-wise this is no problem, as it can be added as "dependency" in
setup.py, setup.cfg or, as in my case, pyproject.toml. But to make the
main app use that dependency, it must be added to the INSTALLEd_APPS
list during the setup process.

I solved that problem by adding a helper function that must be called
after INSTALLED_APPS in settings.py, which dynamically imports the
plugin class (no models, no views, just the plugin main class itself,
e.g. "foo_main_app.plugins.dynamically_added_plugin", and searches for
an INSTALLED_APPS attr in that namespace. If it exists, it is basically
appended to the main INSTALLED_APPS.

To solve ordering problems, it is a bit more complicated, and the plugin
gets the chance to reorder the INSTALLED_APPS and insert its own
dependencies (and itself) to a special index.

But basically the main module of the django (plugin) app is imported,
and read. This works, but I'd like to ask you if this violates any of
Django's principles, or if there would be a better approach in the long
term to create pluggable apps, that just can be installed by pip.

The only reason I post it here (in dev) is that this may be of interest
too for Django itself at some point in time too. But here I need your
feedback.

I first wanted to put that information into the AppConfig class - but
can't be loaded in settings.py - apps aren't loaded.
So, for my taste, plugin information is a bit scattered around
(AppConfig in apps.py, INSTALLED_APPS and more in __init__.py). It could
be added declaratively in some yaml file too, but this would feel even
creepier to me.

Eager to hear your opinion here.

--
Dr. Christian González
https://nerdocs.at

[Jacob Rief]

Isn't it a bit dangerous to auto-add a package from PyPI to a running Django installation? That module then gains full database access and could do all kind of nasty stuff.

Maybe I am a bit naive here, but 3rd party packages sometimes get installed incautiously.

– Jacob

[Christian González]

Am 20.02.23 um 14:23 schrieb Jacob Rief:

Hi Jacob,

no, I don't think so. It is generally "dangerous" to run code you don't
know what it does ;-)
In my case it is even more dangerous to run code I wrote myself, hehe.

But really, if you install ANY package via pip, you have to trust that
package. So it doesn't matter if you install a Django GDAPS auto-plugin
package or django-money. you would have to add it manually to your
settings.py/ INSTALLED_APPS anyway to use it.
GDAPS is intended to enable plugins for a main application - e.g. there
is medux, and medux.plugins.laboratory - both from the same vendor.
There is no trust problem when installing your own packages.

Christian

[Jason Johns]

I'd be a -1 on dynamic modification with installed apps.  that's the developer's responsibility to add in, and should be implicit by design.  The list of apps needs to come somewhere via configuration, and django defines that configuration to be done explicitly.

I be;ieve some of the major security holes via Wordpress come with being able to add in plugins and projects dynamically.   You might be discounting this, but it does open up a fairly large attack vector

[Christian González]

Am 21.02.23 um 14:57 schrieb Jason Johns:

I'd be a -1 on dynamic modification with installed apps.  that's the developer's responsibility to add in, and should be implicit by design.

No, that's not IMHO. In my use case (and e.g. Pretix' too, which is a great Django use case IMHO) I need a base django application and dynamically installed plugins, which may change over time. With GDAPS this is possible just by installing the plugins using pip, and running migrate/collectstatic and a custom mgmt command.
This can be done by an admin, and not necessarily by a developer.

The list of apps needs to come somewhere via configuration, and django defines that configuration to be done explicitly.

Yes, and that's what I want to extend. It's possible with some work, and I'd like to collect how it's done *best*.

I be;ieve some of the major security holes via Wordpress come with being able to add in plugins and projects dynamically.   You might be discounting this, but it does open up a fairly large attack vector

Yes, but my question is another one. GDAPS isn't about adding apps dynamically to Django. it's about zero-configuration-adding apps to a Django application. In Wordpress speech: Not like Wordpress plugins, but plugins for a specialized app created USING Wordpress.

I'd not like to exactly discuss this here on django-developers, as it's more a usage thingy, and I don't like to spam this list with django usage questions...

My original question was HOW would I implement this in a  "good-practice" way, by not violating Django's conventions, and showing a way it could be done clean.

You always can use a tool in bad ways. There are even discussions about why Django adds a SECRET_KEY in code by default, which should not be done.

But I think that it is really valuable to expand an application with plugins that can be installed via pip. It does not need to mean users can freely install plugins like in wordpress. It means that devs/vendors can plug together an appliance of GDAPS (=Django app) plugins for *one* application by just installing plugins via pip. They all can be coded by the same author.

This is not possible ATM in Django, as it would need to touch settings.py and add the app in INSTALLED_APPS manually, add some plugin specific settings etc. Which gets bulky when using 20+ plugins, and maintaining 300 different appliances (for 300 customers) with different (and dynamically changing) plugins.

So again, does anything stand against loading plugins' INSTALLED_APPS from a plugin's root module? Is there a better approach?

Thanks, Christian

On Tuesday, February 21, 2023 at 1:54:01 AM UTC-5 Christian González wrote:

Am 20.02.23 um 14:23 schrieb Jacob Rief:
> Isn't it a bit dangerous to auto-add a package from PyPI to a running
> Django installation? That module then gains full database access and
> could do all kind of nasty stuff.
> Maybe I am a bit naive here, but 3rd party packages sometimes get
> installed incautiously.

Hi Jacob,

no, I don't think so. It is generally "dangerous" to run code you don't
know what it does ;-)
In my case it is even more dangerous to run code I wrote myself, hehe.

But really, if you install ANY package via pip, you have to trust that
package. So it doesn't matter if you install a Django GDAPS auto-plugin
package or django-money. you would have to add it manually to your
settings.py/ INSTALLED_APPS anyway to use it.
GDAPS is intended to enable plugins for a main application - e.g. there
is medux, and medux.plugins.laboratory - both from the same vendor.
There is no trust problem when installing your own packages.

Christian

--
Dr. Christian González
https://nerdocs.at

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6c3ea57a-c66d-40cf-809b-899d3b0d2b98n%40googlegroups.com.

[Jörg Breitbart]

Am 21.02.23 um 22:05 schrieb Christian González:

> My original question was HOW would I implement this in a "good-practice"
> way, by not violating Django's conventions, and showing a way it could
> be done clean.

I think thats not really feasible without big restrictions on the apps
and what they might bring into a running django instance. Main reason
being the multistage bootstrapping with collecting phases, esp. on ORM
side. While it is possible to create models on the fly, they will not
fully integrate properly. You'd have shim in all the bookkeeping, cache
abstractions and static globals (django still has a few of those) and
alter their values in way, that would resemble a clean startup. And with
multithreading things will get really funny here.

If you introduce on-the-fly app loading, demand for on-the-fly app
unloading or app updating is around the corner. Thats like opening
pandora's box (from dangling refs to not gc'ing/freeing at all due to
some deeply held ref).

Overall I think any attempt into these on-the-fly directions will lead
to serious stability issues.

Cheers,
Jörg

[Christian González]

Am 22.02.23 um 22:37 schrieb Jörg Breitbart:

Am 21.02.23 um 22:05 schrieb Christian González:

My original question was HOW would I implement this in a "good-practice" way, by not violating Django's conventions, and showing a way it could be done clean.

I think thats not really feasible without big restrictions on the apps and what they might bring into a running django instance. Main reason being the multistage bootstrapping with collecting phases, esp. on ORM side. While it is possible to create models on the fly, they will not fully integrate properly. You'd have shim in all the bookkeeping, cache abstractions and static globals (django still has a few of those) and alter their values in way, that would resemble a clean startup. And with multithreading things will get really funny here.

Hm - maybe I don't understand this (& Django bootstrap process) in depth, or I didn't make myself clear enough...

I don't want to interfer with Djangos setup process in various ways. What you say would be true if GDAPS would try to hook into many processes Django does at the start.

I just add some strings to INSTALLED_APPS at the time of parsing settings.py. In fact, there is:

INSTALLED_APPS = [
    "django.contrib..."
    "..."
    "myapp.core"
]

PluginManager.find_apps(INSTALLED_APPS, "myapp.plugins")

Here, PluginManager has a method that receives the INSTALLED_APPS, loads all plugins' main modules, extracts their INSTALLED_APPS list, and merges them into the main.
So after this line, INSTALLED_APPS contains additionally e.g. "myapp.plugins.foo_plugin" and "djmoney".

If you introduce on-the-fly app loading, demand for on-the-fly app unloading or app updating is around the corner. Thats like opening pandora's box (from dangling refs to not gc'ing/freeing at all due to some deeply held ref).

Overall I think any attempt into these on-the-fly directions will lead to serious stability issues.

I don't want to introduce on-the-fly-anything... This would, as you say correctly, interfere with Django's internals and eat your dog, or worse. Never wanted to do that. After that line (code above), everything goes the normal Django way of loading.

Of course, to add or remove a plugin, you have to restart the server, and run migrate etc mgmt commands, as I said earlier.

And it **works** perfectly this way, I use it (even unofficially in production) on a daily base. I just want to know if there is a better approach than placing an INSTALLED_APPS list (or a method) into the **main module** of a plugin...

Cheers,

Christian

Cheers,
Jörg

[Jörg Breitbart]

Ah sorry for the fuzz, I read too much into your idea and jumped to the
conclusion, that you want to extend INSTALLED_APPS at runtime.

What you describe here

> INSTALLED_APPS = [
> "django.contrib..."
> "..."
> "myapp.core"
> ]
>
> PluginManager.find_apps(INSTALLED_APPS, "myapp.plugins")

makes perfect sense, as it only decorates INSTALLED_APPS very early
during settings.py loading, so the boostrapping should be fine. I dont
see any issues with that - beside hiding the explicit loading nature, as
Jacob pointed out, but thats indeed already an issue baked into python
with setup.py deps or carelessly pulling untrusted stuff via pip. So
nope, the explicit nature of INSTALLED_APPS is a "false friend" in terms
of security concerns from 3rd party modules.

As you already pointed out, the proper app ordering might be a bigger
issue for an automated app discovery. Not sure how you solved that, this
might need some tree balance logic, but the question remains, where to
get the info "xy must be loaded before/after z" from in the first place.
Ideally apps should be loading position independent, but thats not
always the case, and you might even run into edge cases, where a proper
loading strategy cannot be found due to incompatible before/after
requirements.

Cheers,
Jörg

[Christian González]

Am 23.02.23 um 17:09 schrieb Jörg Breitbart:

Ah sorry for the fuzz, I read too much into your idea and jumped to the conclusion, that you want to extend INSTALLED_APPS at runtime.

no prob ;-)

As you already pointed out, the proper app ordering might be a bigger issue for an automated app discovery. Not sure how you solved that, this might need some tree balance logic, but the question remains, where to get the info "xy must be loaded before/after z" from in the first place. Ideally apps should be loading position independent, but thats not always the case, and you might even run into edge cases, where a proper loading strategy cannot be found due to incompatible before/after requirements.

I am exploring 2 ways of adding app strings to INSTALLED_APPS. Code says more than words:

class PluginManager:
    ...
    @classmethod
    def alter_installed_apps(cls, installed_apps: list[str], group: str) -> list[str]:
        """Lets plugins update INSTALLED_APPS and add their own apps to it, in arbitrary order.
    
        Call this method directly after your settings.INSTALLED_APPS declaration.
        """

I'm not sure if altering the passed list is better, or returning a new one. would be more explicit to see what it does.

The method

1. searches for an alter_installed_apps method in the main plugin module, which takes the INSTALLED_APPS as first argument, and calls it. So I give the plugin itself the possibility to place itself (and evtl. needed other modules) at indexes of the INSTALLED_APPS list they need and find appropriate. This should work well for many apps that have special requirements like "should be placed after django.contrib.staticfiles", but does not cover real dependency checking, especially does not cover inter-dependencies of 2 plugins - the first loaded plugin does not know about the second, as at time of calling the first one, INSTALLED_APPS does not contain the second one.

2. if this does not exist, it searches for an INSTALLED_APPS list in the main module as stated earlier, for dumb modules that just need to be loaded, no matter which order. This list is plainly appended to the main INSTALLED_APPS.

At time of calling plugin implementations via their hooks, they provide "weight" attrs and get ordered by this attr when iterating over them. So each interface hook can have it's own ordering, which works perfectly.

But that doesn't solve the dependency resolution problem of INSTALLED_APPS. One of my solutions is: Use Python's way: I declare dependencies in pyproject.toml, so dependency resolution is done by pip. And in Django, I can already be sure everything is installed as intended, and just order INSTALLED_APPS like I want to.

But it's not perfect, I know.

So thanks Jörg for your helpful comment. I wasn't quite sure about if I did something stupid.

And thanks for your all engaging in Django, it's really a great piece of software.

Yours,

Christian

[Silent Coder]

Hi Christian and Group,

I'm new to Django and still learning the ropes, but I'm curious to know what the use cases for something like this might be.

Best,

Silent

[Christian González]

Am 25.02.23 um 17:58 schrieb Silent Coder:

Hi Christian and Group,

I'm new to Django and still learning the ropes, but I'm curious to know what the use cases for something like this might be.

I don't want to use django-developers for this topic more than intended (django development); I'll answer this, but if you have further questions, just send me a PM or use django-users - ok?

The main reason is Django does not allow apps to be "plugged" into a main application. If you want to extend an application woth other plugins, you are alone...

GDAPS tries to fill this gap.

I started this because I want to create a medical application that has a "core" app, and can be extended  using plugins like "laboratory", "demographics/statistics", "ledger" etc.. Those all are apps that fit into the main system, with own models and views, but use the core's templates, models, etc.

If you want to create applicances that can be easily extended by plugins, you can do this with GDAPS by just installing the app with pip and restart the server. With Django alone, you would have to add all the apps manually into settings.py, and step through dependency installation, etc.

GDAPS tries to create a framework for that, at least for the backend (+  Django templates). No frontend supported any more (Vue.js was in exploration).

Hope you can see what I want to achieve.

Christian

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/e0a655b6-dc48-4a32-9851-f4dc573dbe93n%40googlegroups.com.