Resource loading (Django without a filesystem)

[Peter Baumgartner]

I'm interested in using PyOxidizer [1] to create single-file executable Django projects. This currently isn't possible because of all the places Django assumes it is operating on a proper filesystem. I haven't done a full audit, but templates, migrations, and static files come to mind. Python has full support this scenario (aka resource loading), but it would require some changes to Django internals to use it. One of the biggest hurdles I see on the surface is that you can only load a resource from a Python module (not from a sub-directory). That being said, I have a feeling resource loading could be added such that users could opt-in or backwards compatibility is maintained with the current implementation.

Some additional reading/watching on the subject:

* Topic overview: https://pyoxidizer.readthedocs.io/en/latest/packaging_pitfalls.html#reliance-on-file

* Barry Warsaw talk on resource loading: https://www.youtube.com/watch?v=ZsGFU2qh73E

I'm posting here to see if there is general support for this and to discuss what sort of changes would be required. Thanks!

[1] https://pyoxidizer.readthedocs.io/

[Markus Holtermann]

Hi Peter,

PyOxidizer looks indeed super interesting. Talking about templates and specifically Jinja2 templates, they are internally converted to the Python AST if I'm not mistaking. Turning them into Python modules that a new Jinja2ModuleTemplateLoader could load doesn't seem like that far fetched.

Similarly for Django's migration files. Swapping out the MigrationLoader would already be sufficient.

I'd definitely be interested to see what's needed to change in core to have a 3rd party package provide the necessary support.

Cheers,

Markus

> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com.
> To post to this group, send email to
> django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/b6229b20-0647-4e0a-b9e6-31136a774e13%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/b6229b20-0647-4e0a-b9e6-31136a774e13%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Andrew Godwin]

My impression reading over the problem a little yesterday was that we could work to provide a common "get me a resource" abstraction in Django that papers over the couple of different ways it has to work, though I haven't looked super far into things that require directory listing (e.g. migrations) rather than direct file access (like templates).

It would be nice to investigate this a bit more, though. If we can get Django compatible, or work with PyOxidiser if we find a reasonable workaround they could implement, it would be great.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/2faec64a-da9f-4d55-9378-a3cb6a308d53%40www.fastmail.com.

[Peter Baumgartner]

The big issue I see is that a resource must reside directly in a
Python module. You can not load a resource from a child directory,
e.g. I can load "index.html" from the Python module
"myproject.templates", but I can't load "app1/index.html" from the
same module.

This would require developers to scatter __init__.py files in every
template directory. I think migrations would be easy (you can list
Python files in a module without the filesystem). I need to look at
how staticfiles works a little closer. I think we could still use
__file__ for collectstatic (you would run that before converting your
project), but I'm not sure what Django needs access to at runtime
there.

Pytz is also an issue. It looks like an easier fix there because the
project controls the "resource" directories and could sprinkle in the
necessary __init__.py files. I filed an issue to start the discussion
there as well, https://bugs.launchpad.net/pytz/+bug/1834363

> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1ur36dYm7xohN0PQRkY3Y8H5m8-Ws307Wo-u%2B3xMtjLBKg%40mail.gmail.com.

[Andrew Godwin]

Well, remember that in Python 3 you don't need an __init__.py file to have something be a module (because of  https://www.python.org/dev/peps/pep-0420/), but I wonder if there still needs to be a proper discovery mechanism that flags that they should be considered as implicit packages/modules.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6K9zn5jMw2QQxtAHMrmZMopxqPMLd7jrzi5-S4ryPMa7ZM-Q%40mail.gmail.com.

[Peter Baumgartner]

Unfortunately you can't load resources from implicit namespace
packages. It raises `FileNotFoundError: Package has no location
<module 'myapp' (namespace)>`
The __init__.py appears to be required.

> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1uofCa292Jv7A9Xj_WqyYC0sJJJe%2BJ%2BmhGzKc6cUkknbMg%40mail.gmail.com.