Static media handling - ticket 12323

15 views
Skip to first unread message

Luke Plant

unread,
May 27, 2010, 11:23:25 AM5/27/10
to django-d...@googlegroups.com
A Django 1.3 proposal. It is a pretty small feature/change in some
ways, but needs some discussion.

= Motivation =

This is intended to make it easier to cope with the distinction
between user provided media and developer provided media. This is a
significant pain point when it comes to source
control/deployment/backup.

For example, it would be much better when uploading my source if I
could do 'rsync --delete', so that the files on the server are exactly
what I expect - extra files can cause all kinds of bugs. But I can't
do that, as it would delete all the user uploaded media (or at least
the symlinks to it). Nor can I simply upload to a fresh directory - I
would still need to mess around with symlinks or moving files to get
user uploaded media where it is expected to be.

Also, for security it would be good to have the whole source code
directory (including templates, javascript etc) to be write protected
from the point of the webserver. But user uploaded files mess that
up.

= Options =

We could:
1) add 'STATIC_URL' and use it for widgets and all other media.
2) add 'USER_MEDIA_URL' and 'USER_MEDIA_ROOT' and use them for file
storage.

(using backwards compatible defaults either way).

If 1), then, additionally, do we need 'STATIC_ROOT' as well? What
for? It wouldn't be used anywhere in Django.

I was going to go for 1), like the ticket suggests, but I now think
that 2) is a much better idea.

I strongly suspect that static media are far more common than user
uploaded files, so doing 2) will require far fewer changes to existing
apps. Also, I suspect that almost all direct use of MEDIA_URL in apps
will be for static media: file fields provide a URL attribute which
automatically includes MEDIA_URL, but it will be common to use
MEDIA_URL for calculating URLs (e.g. in templates).

Also, use of 'media' for static media is consistent with
ADMIN_MEDIA_PREFIX, which is lost with option 1)

If we go with 2), there is a good case for not adding USER_MEDIA_URL
to the media context processor - I can't imagine know when it would be
needed in the normal case. If you want to generate a URL to a file
stored in a model, the only sensible way to do it is
instance.somefile.url, which handles it for you. If we go for 1) we
will need both STATIC_URL and MEDIA_URL in the media context processor
for backwards compatibility.

Are there any common uses cases I have not accounted for?

Finally, once these things are sorted out, is this a small enough
change that I should go ahead and commit it, or should I wait for
voting on Django 1.3 features?

Thanks,

Luke

--
"Oh, look. I appear to be lying at the bottom of a very deep, dark
hole. That seems a familiar concept. What does it remind me of? Ah,
I remember. Life." (Marvin the paranoid android)

Luke Plant || http://lukeplant.me.uk/

Brian Rosner

unread,
May 27, 2010, 12:09:58 PM5/27/10
to django-d...@googlegroups.com

On May 27, 2010, at 9:23 AM, Luke Plant wrote:

> A Django 1.3 proposal. It is a pretty small feature/change in some
> ways, but needs some discussion.
>
> = Motivation =
>

[...]

I am on board with the motivation here. The reasons given are exactly why
we made a similar change to Pinax.

> = Options =
>
> We could:
> 1) add 'STATIC_URL' and use it for widgets and all other media.
> 2) add 'USER_MEDIA_URL' and 'USER_MEDIA_ROOT' and use them for file
> storage.

I am torn between these two. I am in favor of both to some extent. I like the
direction of being more clear about which is used by FileFields. On the other
hand calling the static resources for a site STATIC_* seems the most correct
to me.

Of course Pinax had to take some direction and since we couldn't go changing
all the FileFields of the world we had to introduce new settings. My stance
would definitely favor option 2 for the added clarity it provides. Pinax could
end up with best of both worlds ;-)

> (using backwards compatible defaults either way).
>
> If 1), then, additionally, do we need 'STATIC_ROOT' as well? What
> for? It wouldn't be used anywhere in Django.

For Pinax we added STATIC_ROOT because we support a command called
build_static which collects static files from apps and writes them to
STATIC_ROOT. It has worked *extremely* well for us, but I don't think you
are looking at going this far yet. If so, under option 2 essentially
MEDIA_ROOT could serve that purpose. However, going with option 2 would make
MEDIA_ROOT obsolete until something like this is added? Might make that a bit
tricky.

> I strongly suspect that static media are far more common than user
> uploaded files, so doing 2) will require far fewer changes to existing
> apps. Also, I suspect that almost all direct use of MEDIA_URL in apps
> will be for static media: file fields provide a URL attribute which
> automatically includes MEDIA_URL, but it will be common to use
> MEDIA_URL for calculating URLs (e.g. in templates).

This is a decent assumption as when I went through my code to move to
STATIC_URL it was largely changing all MEDIA_URL references.

> Also, use of 'media' for static media is consistent with
> ADMIN_MEDIA_PREFIX, which is lost with option 1)

Good point.

> Finally, once these things are sorted out, is this a small enough
> change that I should go ahead and commit it, or should I wait for
> voting on Django 1.3 features?

I suppose this is dependent on whether we want to introduce app media handling
or if we want this to separate from that. In my opinion I can see them as two
different bits that be done independently.

Brian Rosner
http://oebfare.com
http://twitter.com/brosner

Luke Plant

unread,
May 27, 2010, 12:38:16 PM5/27/10
to django-d...@googlegroups.com
On Thursday 27 May 2010 17:09:58 Brian Rosner wrote:

> > = Options =
> >
> > We could:
> > 1) add 'STATIC_URL' and use it for widgets and all other media.
> > 2) add 'USER_MEDIA_URL' and 'USER_MEDIA_ROOT' and use them for
> > file storage.
>
> I am torn between these two. I am in favor of both to some extent.
> I like the direction of being more clear about which is used by
> FileFields. On the other hand calling the static resources for a
> site STATIC_* seems the most correct to me.
>
> Of course Pinax had to take some direction and since we couldn't go
> changing all the FileFields of the world we had to introduce new
> settings. My stance would definitely favor option 2 for the added
> clarity it provides. Pinax could end up with best of both worlds
> ;-)

Thanks for your feedback, it's good to have some input from Pinax
developers on this.

I knew you had adopted method 1), but I don't think Django doing 2)
causes you much pain - you just need to set STATIC_ROOT = MEDIA_ROOT,
STATIC_URL = MEDIA_URL in your settings, and add USER_MEDIA_* and
nothing else would need to change.

> > Finally, once these things are sorted out, is this a small enough
> > change that I should go ahead and commit it, or should I wait for
> > voting on Django 1.3 features?
>
> I suppose this is dependent on whether we want to introduce app
> media handling or if we want this to separate from that. In my
> opinion I can see them as two different bits that be done
> independently.

Agreed. It could be done separately, but that might some confusion.
I'll probably make a branch in my bitbucket repository, and wait and
see if anyone proposes app media handling for 1.3.

Cheers,

Jannis Leidel

unread,
May 27, 2010, 4:22:22 PM5/27/10
to django-d...@googlegroups.com
Am 27.05.2010 um 17:23 schrieb Luke Plant:

> A Django 1.3 proposal. It is a pretty small feature/change in some
> ways, but needs some discussion.
>
> = Motivation =
>
> This is intended to make it easier to cope with the distinction
> between user provided media and developer provided media. This is a
> significant pain point when it comes to source
> control/deployment/backup.
>
> For example, it would be much better when uploading my source if I
> could do 'rsync --delete', so that the files on the server are exactly
> what I expect - extra files can cause all kinds of bugs. But I can't
> do that, as it would delete all the user uploaded media (or at least
> the symlinks to it). Nor can I simply upload to a fresh directory - I
> would still need to mess around with symlinks or moving files to get
> user uploaded media where it is expected to be.
> Also, for security it would be good to have the whole source code
> directory (including templates, javascript etc) to be write protected
> from the point of the webserver. But user uploaded files mess that
> up.

Couldn't agree more.

> = Options =
>
> We could:
> 1) add 'STATIC_URL' and use it for widgets and all other media.
> 2) add 'USER_MEDIA_URL' and 'USER_MEDIA_ROOT' and use them for file
> storage.
>
> (using backwards compatible defaults either way).
>
> If 1), then, additionally, do we need 'STATIC_ROOT' as well? What
> for? It wouldn't be used anywhere in Django.
> I was going to go for 1), like the ticket suggests, but I now think
> that 2) is a much better idea.

I also prefer 2), the ambiguity of MEDIA_* strikes me as unfortunate and the Django core seems like the right place to fix this. For django-staticfiles I used STATIC_* for the media files included in apps (et al) which is of course only slightly better.

> I strongly suspect that static media are far more common than user
> uploaded files, so doing 2) will require far fewer changes to existing
> apps. Also, I suspect that almost all direct use of MEDIA_URL in apps
> will be for static media: file fields provide a URL attribute which
> automatically includes MEDIA_URL, but it will be common to use
> MEDIA_URL for calculating URLs (e.g. in templates).
>
> Also, use of 'media' for static media is consistent with
> ADMIN_MEDIA_PREFIX, which is lost with option 1)

Yeah, good point.

> If we go with 2), there is a good case for not adding USER_MEDIA_URL
> to the media context processor - I can't imagine know when it would be
> needed in the normal case. If you want to generate a URL to a file
> stored in a model, the only sensible way to do it is
> instance.somefile.url, which handles it for you. If we go for 1) we
> will need both STATIC_URL and MEDIA_URL in the media context processor
> for backwards compatibility.
>
> Are there any common uses cases I have not accounted for?
>
> Finally, once these things are sorted out, is this a small enough
> change that I should go ahead and commit it, or should I wait for
> voting on Django 1.3 features?

I'd like to extend this proposal after having talked to a few people at the sprints to include the following features of django-staticfiles [1]:

- a media files path resolver -- following a similar directory structure as the app templates loader (<app>/media/<app> vs. <app>/templates/<app>)
- build_static -- a mangement command that'll collect media files from apps from various locations using the media files path resolver and uses a file storage backend to copy (or link to) the found files (file storage backend by default), used during deployment
- an extended media file serving view that uses the media files path resolver (in debug mode), backwards compatible replacement for django.core.servers.basehttp.AdminMediaHandler

I'd be happy to commit myself to that feature which proved itself invaluable in real projects.

Jannis


1: http://pypi.python.org/pypi/django-staticfiles


bur...@gmail.com

unread,
May 27, 2010, 7:27:08 PM5/27/10
to django-d...@googlegroups.com
Hi all,

> - a media files path resolver -- following a similar directory structure as the app templates loader (<app>/media/<app> vs. <app>/templates/<app>)
> - build_static -- a mangement command that'll collect media files from apps from various locations using the media files path resolver and uses a file storage backend to copy (or link to) the found files (file storage backend by default), used during deployment
> - an extended media file serving view that uses the media files path resolver (in debug mode), backwards compatible replacement for django.core.servers.basehttp.AdminMediaHandler

Couldn't agree more. But three more points:
1) Media files are linked from applications forms, so what will happen
if user rename an app?
Do we need some config piece for this kind of reverse?
Something like app_media(app_name=None) function and APP_MEDIA?

2) More generally, we need per-application settings to handle this
sort of configuration problems.
One that will be autodiscovered like templates and admin. And it can
contain additive variables for MEDIA, like suggested in my
configuration proposal:
http://groups.google.com/group/django-developers/browse_thread/thread/b7339d8577567d95

3) I've seen a solution recently that generates apache and lighttpd
install scripts, found it much better than symlinks for media in
production.
But this one definitely needs additive variables and per-application
configuration!

--
Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
MSN: bu...@live.com

Brian Rosner

unread,
May 27, 2010, 7:59:13 PM5/27/10
to django-d...@googlegroups.com

On May 27, 2010, at 5:27 PM, bur...@gmail.com wrote:

> Hi all,
>
>> - a media files path resolver -- following a similar directory structure as the app templates loader (<app>/media/<app> vs. <app>/templates/<app>)
>> - build_static -- a mangement command that'll collect media files from apps from various locations using the media files path resolver and uses a file storage backend to copy (or link to) the found files (file storage backend by default), used during deployment
>> - an extended media file serving view that uses the media files path resolver (in debug mode), backwards compatible replacement for django.core.servers.basehttp.AdminMediaHandler
>
> Couldn't agree more. But three more points:
> 1) Media files are linked from applications forms, so what will happen
> if user rename an app?

If someone changes the name of the app there is no way they will get by
without modifying source code in the app. Why should this be any different?
I wouldn't really buy the argument that it reduces the amount of change they'd
have to do. It applies to the convention people use when storing/referencing
templates too.

BrettH

unread,
May 29, 2010, 11:54:28 PM5/29/10
to Django developers
+1 on Option 2.

I have written a basic deployment app (not quite ready for release
yet) that deploys a virtualenv for each version of your project,
roughly equivalent to Google App Engine. I specifically need to split
out the USER_MEDIA so that it isn't versioned. Internally if the
developer does not use the setting I will default to the MEDIA_ROOT
anyway.

I like the limited focus of this proposal. USER_MEDIA nicely describes
something that is not specifically meant or available to be deployed
by default as part of the app, and is a reasonable convention to
encourage developers to split their directories from the start.

A more complicated discussion is that around assumed app media
directories. I'm inclined to think that admin gives the precedent to
declare a default media directory per app, a convention that can be
backed up by management command to link/consolidate by app name for
production which other apps already do, but this in turn opens the can
of worms that is settings when overriding defaults. I think ponies
will cry if this proposal gets extended to try and include too much.

Jannis Leidel

unread,
Jun 3, 2010, 8:16:23 AM6/3/10
to django-d...@googlegroups.com

Am 28.05.2010 um 01:27 schrieb bur...@gmail.com:

> Hi all,
>
>> - a media files path resolver -- following a similar directory structure as the app templates loader (<app>/media/<app> vs. <app>/templates/<app>)
>> - build_static -- a mangement command that'll collect media files from apps from various locations using the media files path resolver and uses a file storage backend to copy (or link to) the found files (file storage backend by default), used during deployment
>> - an extended media file serving view that uses the media files path resolver (in debug mode), backwards compatible replacement for django.core.servers.basehttp.AdminMediaHandler
>
> Couldn't agree more. But three more points:
> 1) Media files are linked from applications forms, so what will happen
> if user rename an app?
> Do we need some config piece for this kind of reverse?
> Something like app_media(app_name=None) function and APP_MEDIA?
>
> 2) More generally, we need per-application settings to handle this
> sort of configuration problems.
> One that will be autodiscovered like templates and admin. And it can
> contain additive variables for MEDIA, like suggested in my
> configuration proposal:
> http://groups.google.com/group/django-developers/browse_thread/thread/b7339d8577567d95

That seems overly complicated for the use case of app media. For the app level configuration though, I'd say we should wait a little for the summer of code to progress.

> 3) I've seen a solution recently that generates apache and lighttpd
> install scripts, found it much better than symlinks for media in
> production.
> But this one definitely needs additive variables and per-application
> configuration!

In case you mean the django-staticmedia [1] app -- I think it's doing fine outside of Django -- given the fact that generating deployment config files isn't a core goal of Django.

Jannis


1: http://pypi.python.org/pypi/django-staticmedia

Jannis Leidel

unread,
Jun 3, 2010, 8:18:30 AM6/3/10
to django-d...@googlegroups.com
Am 30.05.2010 um 05:54 schrieb BrettH:

> +1 on Option 2.
>
> I have written a basic deployment app (not quite ready for release
> yet) that deploys a virtualenv for each version of your project,
> roughly equivalent to Google App Engine. I specifically need to split
> out the USER_MEDIA so that it isn't versioned. Internally if the
> developer does not use the setting I will default to the MEDIA_ROOT
> anyway.

Note, my propsal isn't about creating a deployment tool but to make sure Django is prepared for the increasing number of apps that ship with media files. django-staticfiles is in use today due to Pinax and doesn't make assumptions about your custom deployments. (e.g. override the storage backend to build the media on S3)

> I like the limited focus of this proposal. USER_MEDIA nicely describes
> something that is not specifically meant or available to be deployed
> by default as part of the app, and is a reasonable convention to
> encourage developers to split their directories from the start.

Agreed, the ambiguity of the MEDIA_ROOT setting is unfortunate but not that easy to fix in a backwards incompatible way if we also want to solve the app media problem.

> A more complicated discussion is that around assumed app media
> directories. I'm inclined to think that admin gives the precedent to
> declare a default media directory per app, a convention that can be
> backed up by management command to link/consolidate by app name for
> production which other apps already do, but this in turn opens the can
> of worms that is settings when overriding defaults. I think ponies
> will cry if this proposal gets extended to try and include too much.

Honestly, I strongly believe this is a topic in which Django needs to provide optional helpers to make it easier to ship non-Python files. Since a vast amount of apps out there already ship with media files, nicely wrapped in Python packages, ready to be installed with pip, Django only needs to be made aware of those conventions.

Speaking of which, the admin is actually a bad example since it uses an own legacy WSGI handler ("AdminMediaHandler") from long ago to serve its media files. staticfiles abstracts that feature, is backwards compatible and follows the simple idea of namespaces -- similar to how app template directories are prefixed with the app name, <app_label>/media/<app_label>/*. So say you want to ship media files with your app 'my_blog', it'd look like this:

my_blog
├── __init__.py
├── admin.py
├── media
│ └── my_blog
│ ├── css
│ │ ├── base.css
│ │ └── forms.css
│ ├── img
│ │ └── logo.png
│ └── js
│ └── ads.js
├── models.py
└── views.py


And you'd be able to use <static URL>/<app_label>/img/logo.png in your templates to point to the right location of your app media files. Either use staticfiles' file serving view (for debugging), which knows how to resolve the relative path to the file included in the app, or use the build_static management command, that can be called from a fabric script during deployment.

FWIW, I've started with an experimental branch at http://github.com/jezdez/django/tree/staticfiles if you want to follow along.

Jannis

> --
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>

BrettH

unread,
Jun 3, 2010, 10:21:35 AM6/3/10
to Django developers
Ok yes seems like a slam dunk.. hell how about an --install-media for
python setup.py install ;)
> FWIW, I've started with an experimental branch athttp://github.com/jezdez/django/tree/staticfilesif you want to follow along.
Reply all
Reply to author
Forward
0 new messages