Discussing improvements of django.setup()

[Pkl]

Hello everyone,

I'm bringing here, for broader review and discussion, the subject of making django setup more "tweakable":

https://code.djangoproject.com/ticket/30536

https://github.com/django/django/pull/11435

There is indeed a need for doing pre/post operations at that crucial moment of the framework lifetime, especially to apply compatibility shims (greening-related, API stability-related...) and setup test scaffolding, stubs etc.

A separate PR has been created to handle threadsafety/idempotence (https://github.com/django/django/pull/11440) when multiple actors try to setup django concurrently, but a discussion remains on how to potentially allow users to specify an alternative setup

I originally did a POC with a new django setting, but if we want to have full control, even settings might not be normally loaded at that stage. So maybe, as apollo13 suggested, an environment variable, like DJANGO_SETUP_CALLABLE, complementing the DJANGO_SETTINGS_MODULE one?

What are your thoughts about this ?

thanks,

regards,

Pakal

[Aymeric Augustin]

Hello Pakal,

Making django.setup() idempotent

In https://code.djangoproject.com/ticket/28752, you say:

> I've been bitten numerous times by the impredictable behaviour of django when django.setup() was called numerous times.
> In the old days I had exceptions, now it's mainly subtle breakages of logging configuration.

This is fuzzy for a bug report. Ideally, you should say:

- What you're doing (minimal reproduction instructions, within supported use of Django)

- What results you're expecting

- What results you're getting

Anyway it suggests that the problem is about logging. In this comment I said that the only possible bug is that `configure_logging()` isn't idempotent. Your answer confirms that the issue is really about logging. Let's fix configure_logging(), then!

Barring new arguments in favor of the proposed approach, which involves nested locks, I'm -1. Nested locks create a possibility of deadlocks. See my comment on the PR for details.

Making django.setup() tweakable

Again, the rationale is fuzzy. I'm seeing two use cases:

- pytest-django

- django-compat-patchers

One more or one less monkey patch isn't going to make a difference to these projects ;-)

I'm ready to accept that there's a need for projects that aren't big hacks but I need to see it explained clearly and specifically. I'd really like to see a feature request in the form of:

- I want to do X, which is a reasonable thing to do and stays within supported use of Django, except I need to override django.setup()

- Here's what the code looks like with a workaround (monkey-patching, probably)

- Here's what the code would look like with the improvement I'm asking for (hopefully much better)

- Here are the alternatives I considered and why I rejected them

- Other use cases also include Y and Z

I'm asking this because concrete use cases will put us in a better position to find the best solution.

Best regards,

-- 

Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/0206c98c-199f-49f9-9ff7-cba736e71224%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pkl]

Hello Aymeric,

regarding django.setup() idempotence I continued discussion in the PR.

Regarding django.setup() tweakability, the generic need is simply to have a hook, somewhere for users to put early init code, regardless of the way django is launched. Code that needs to execute this early are mostly (if not all) monkey-patching the environnement, a little like a LD_PRELOAD logic python-style.

This can be:

- for (of course) the init of django-compat-patcher

- greening of the environment (e.g. as https://github.com/erickponce/django-db-geventpool recommends, for Gevent or Eventlet)

- early tweaking of some process aspects likes socket timeouts (they broke for me when used along with mutiprocessing, but several years ago)

- possibly setup of specific test scaffolding, for which late mock()-style things aren't enough (I have no precise scenario in mind for this one, even though I know we did suffer with a Jpype bridge to a java LDAP client, which required tons of care to not break, including being initialized rather early along with process signal handlers)

Without a hook built into django, users that need such early setup have to duplicate their efforts for each way Django can be launched, and they must call their patching code in manage.py, in the wsgi file, in their custom scripts, and in their specific runners likes pytest-django (this last case is quite uneasy to handle at the moment, due to its very early setup of Django).

I'm afraid I can't provide much code to help the pondering; simply, without builtin hook, the user has to battle to ensure the kind of lines like the following are always executed before django.setup(). If Django provides a hook, it's so much less to worry about.

from psycogreen.gevent import patch_psycopg
patch_psycopg()

There are surely lots of ways to provide such a hook, via new settings, or new environment variables, or modules following precise naming conventions that django would systematically try to import... I have no idea how exactly my approach is rated amongst them.

As for alternatives to these early setup hooks, I'm alas clueless...

regards,

Pakal

-- 

Aymeric.

To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

[Christian González]

Am 01.07.19 um 23:24 schrieb Pkl:

>
> Regarding django.setup() tweakability, the generic need is simply to
> have a hook, somewhere for users to put early init code, regardless of
> the way django is launched. Code that needs to execute this early are
> mostly (if not all) monkey-patching the environnement, a little like a
> LD_PRELOAD logic python-style.

> [snip]

> There are surely lots of ways to provide such a hook, via new
> settings, or new environment variables, or modules following precise
> naming conventions that django would systematically try to import... I
> have no idea how exactly my approach is rated amongst them.
>
> As for alternatives to these early setup hooks, I'm alas clueless...

Why not a setup() method in the Apps' AppConfig, like ready()? This way
the AppConfig would be used more. Problem here is: when setup runs, it
is loading the settings. And THERE INSTALLED_APPS first time tell Django
which apps are installed. So before that point, it's not possible to 
"import modules from apps following a precise naming convention" etc. Or
did you mean "modules" outside of apps like e.g. a
"django_hook_setup_foo" python module? (Grrrrrr.)

Maybe here could again help to extract the apps list out of settings,
and use a descriptive list that is parsed VERY early in the setup
process. Then the hook could be placed (with imports from all apps).

This would help improving my GDAPS plugins system as well, instead of
hacking myself into INSTALLED_APPS within settings.py.

Just my 2 cents.

Christian

[Pkl]

hello,

AppConfig could provide hooks, but it'd indeed be already quite late in the setup process : settings are already loaded, logging already configured, script prefix already set, and just accessing AppConfig could already trigger the import of advanced submodules (models etc.), which could break the early setup.

I guess an environment variable, or a naming convention, could allow early hooks similar to those of Gunicorn for example, and unrelated to installed apps.

regards,

Pascal Chambon

[Pkl]

Hello,

any news about this issue?

Concerning the thread-safety of setup(), the PR is here - https://github.com/django/django/pull/11440

Concerning the tweakability of setup(), I don't get your argument Aymeric: "One more or one less monkey patch isn't going to make a difference to these projects".

That's precisely why we need a hook for custom setup operations. To monkey-patch test environments, apply DCP fixers, setup gevent-like runtimes... for now, there is no single point where one can inject early customization. One needs to change uwsgi.py, maybe manage.py, test launchers, potential scripts...

AppConfig.ready() was introduced so that applications have a single point to initialize signals and the likes, this proposal only extends this to support early-stage setup customization, which is currently missing.

Wouldn't an environment variable, pointing to a custom setup function, do the job?

thanks,

regards,

Pascal

[Christian González]

Hi,

i just stumbled upon logging in management commands I saw that there is
an own system of logging.

When creating custom mgmt cmds, you have to write logging output to

    if options["verbosity"] >= 2:
        sys.stdout.write(message)

this is a little inconvenient, and much less DRY as e.g.

    logger.info(message)

But logger does not really work as intended in commands.

Was this a good choice? Yes, log levels can be changed using settings.py.

But: wouldn't it be convenient to use the system logger for that too in
Django?


My background is: I have made a plugin system, and I would have to add
the "verbosity" parameter to every plugin hook that is ought to print
out some logging strings. It would be a lot easier to use logger.warning
e.g. for that.

Could the 4 verbosity levels just be tied to debug,info, warning, error
of python's logging module as default?

Thanks,
Christian

[charettes]

Hello Christian,

I don't have an exact answer for you but there's open tickets about making management

commands use logging instead of custom output wrappers that might interest you.

Simon

[0] https://code.djangoproject.com/ticket/21429

[1] https://code.djangoproject.com/ticket/27877