Overwrite mode in collectstatic

[Prathik Rajendran M]

Hi all,

This is to initiate a discussion on whether we should add a new overwrite method to collectstatic.

Although I agree that we should careful in adding new features, I think this one is pretty useful.

Here is why:

• We want the files in STATIC_ROOT to be replaced even if the one there is newer. Currently I don't see a way to do this apart from using --clear which deletes the entire directory, where other apps could be writing too. We could implement overwrite where it does a filecmp and then overwrites.
• The clear method deletes the files irrespective of where they are, an accident could mean that data could be lost for ever. In one instance our production box had STATIC_ROOT point to a folder in the app accidentally, then collectstatic was not fetching a file from a sub app which was linked to the main project, we ran collectstatic --clear and it deleted the static folder of the main project which had some base css/js files. This could have brought down production for a while if not for a caching mechanism.
• Django should be able to run in a safe mode where no unintended side-effects should happen.

What are your thoughts on this?

[Tim Graham]

For other readers, my analysis of the issue and why I don't think it's appropriate can be found on the ticket where the idea was first raised: https://code.djangoproject.com/ticket/23724

[Marc Tamlyn]

I'd be inclined to agree with Tim, this seems to simply be a less useful version of --clear, for which the primary motivation was "we messed up".

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/e16d9821-95ca-43c8-930b-3e05f3a447d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Prathik Rajendran M]

I'm happy if it is a fix for --clear either. The motivation is not "we messed up" but if in case someone else accidentally messed up nothing would happen unless they are aware of the what is going to happen, we make it clear that it will delete the existing files and get a prompt from them. My fix or patch doesn't have to be the ultimate solution but this mess did happen and in my opinion mess happens, the best thing we can do is prevent any damage happening because of it. I'm happy if any alternative solution comes out of it either. And who cares about what the motivation is, shouldn't we discuss the solution instead?

[Prathik Rajendran M]

And would love to hear on why you think of this as a less useful version. Would you suggest any changes to make it more useful? The end goal being that in scenarios where modified time of files in STATIC_ROOT is greater than that of the files in the APP, the files of the APP should replace the files in STATIC_ROOT without disturbing other contents in STATIC_ROOT.

Unless the design decision of Django is that files directly written to STATIC_ROOT shouldn't be preserved, I think overwrite is pretty useful.

An alternate solution in cases where the files in STATIC_ROOT have a timestamp greater that of the APP and there are files there which are used by other non-django APPs or there are files present there like csv generated by django, we go to STATIC_ROOT, manually delete the files that need to be replaced and run collectstatic.

On Wednesday, October 29, 2014 9:58:51 PM UTC+5:30, Marc Tamlyn wrote:

[Tim Graham]

"we make it clear that it will delete the existing files and get a prompt from them"

Doesn't this already happen per the commits I linked in the ticket?

"Unless the design decision of Django is that files directly written to STATIC_ROOT shouldn't be preserved,"

Yes, as the documentation for STATIC_ROOT says, "This should be an (initially empty) destination directory for collecting your static files from their permanent locations into one directory for ease of deployment; it is not a place to store your static files permanently."

[Prathik Rajendran M]

Hi Tim,

Doesn't this already happen per the commits I linked in the ticket?

As far as I can see it tells that the existing files will be deleted, which is true, but it doesn't tell that the files of the STATIC_ROOT are same as the APP (in the scenario I mentioned) and that it can't be restored. It should tell this in order to mitigate unknown damage. This can be inferred from the location of the STATIC_ROOT if it is printed while clear is being done but it is possible that the person who is running the clear command missed this.

Yes, as the documentation for STATIC_ROOT says, "This should be an (initially empty) destination directory for collecting your static files from their permanent locations into one directory for ease of deployment; it is not a place to store your static files permanently."

This makes it necessary that a STATIC_ROOT location is fixed for Django itself but it then adds a friction for other apps running with it. We should minimize the requirement that people learn certain laws before using Django because it is possible that they don't and ideally we should have mechanisms to figure out that they are not and then alert them, I am not sure if these alerts are there.

Thanks,

Prathik

[Tim Graham]

I just don't see the value here. A problem happens if a user:

1) doesn't read the docs for STATIC_ROOT and sets it incorrectly AND
2) doesn't understand the following prompt:

You have requested to collect static files at the destination
location as specified in your settings:

    /home/tim/code/mysite/static

This will DELETE ALL FILES in this location!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel:

Maybe we can tweak the message if it's not clear, but adding more code logic to try to make the error message "smarter" seems overkill to me.

[Shai Berger]

I agree with Tim completely.

I do see one thing that could help here: Add a check that STATIC_ROOT is not a
subfolder of any app. This check cannot be really reliable (that is, a
particularly careless and unlucky user could set things up so that their
STATIC_ROOT is in an app but the check will not notice it -- a couple of soft-
links will do the trick), but a blunder as described here (STATIC_ROOT is
simply the same as the static folder of an app) can be identified and warned
against.

I don't think such a check belongs in Django, given its inherent unreliability
and the low probabiity of such errors (this is the first time I've heard of
one), but with 1.7, you could just add it yourself; you can publish it and it
can live as a 3rd-party library until it proves me wrong by gaining wide-
spread use.

Shai.

> >> deployment; it is *not* a place to store your static files

> >> permanently."
> >
> > This makes it necessary that a STATIC_ROOT location is fixed for Django
> > itself but it then adds a friction for other apps running with it. We
> > should minimize the requirement that people learn certain laws before
> > using Django because it is possible that they don't and ideally we
> > should have mechanisms to figure out that they are not and then alert
> > them, I am not sure if these alerts are there.
> >
> > Thanks,
> > Prathik
> >
> > On Wednesday, October 29, 2014 10:51:13 PM UTC+5:30, Tim Graham wrote:
> >> "we make it clear that it will delete the existing files and get a
> >> prompt from them"
> >>
> >> Doesn't this already happen per the commits I linked in the ticket?
> >>
> >> "Unless the design decision of Django is that files directly written to
> >> STATIC_ROOT shouldn't be preserved,"
> >>
> >> Yes, as the documentation for STATIC_ROOT says, "This should be an
> >> (initially empty) destination directory for collecting your static files
> >> from their permanent locations into one directory for ease of

> >> deployment; it is *not* a place to store your static files

> >>>>>> - We want the files in STATIC_ROOT to be replaced even if the one

> >>>>>> there is newer. Currently I don't see a way to do this apart from
> >>>>>> using --clear which deletes the entire directory, where other
> >>>>>> apps could be writing too. We could implement overwrite where it
> >>>>>> does a filecmp and then overwrites.

> >>>>>> - The clear method deletes the files irrespective of where they

> >>>>>> are, an accident could mean that data could be lost for ever. In
> >>>>>> one instance our production box had STATIC_ROOT point to a
> >>>>>> folder in the app accidentally, then collectstatic was not
> >>>>>> fetching a file from a sub app which was linked to the main
> >>>>>> project, we ran collectstatic --clear and it deleted the static
> >>>>>> folder of the main project which had some base css/js files.
> >>>>>> This could have brought down production for a while if not for a
> >>>>>> caching mechanism.

> >>>>>> - Django should be able to run in a safe mode where no unintended

> >>>>>> side-effects should happen.
> >>>>>>
> >>>>>> What are your thoughts on this?
> >>>>>

> >>>>> You received this message because you are subscribed to the Google
> >>>>> Groups "Django developers (Contributions to Django itself)" group.
> >>>>> To unsubscribe from this group and stop receiving emails from it,
> >>>>> send an email to django-develop...@googlegroups.com.
> >>>>> To post to this group, send email to django-d...@googlegroups.com.
> >>>>> Visit this group at http://groups.google.com/group/django-developers.
> >>>>> To view this discussion on the web visit
> >>>>> https://groups.google.com/d/msgid/django-developers/e16d9821-95ca-43c
> >>>>> 8-930b-3e05f3a447d0%40googlegroups.com

> >>>>> <https://groups.google.com/d/msgid/django-developers/e16d9821-95ca-4
> >>>>> 3c8-930b-3e05f3a447d0%40googlegroups.com?utm_medium=email&utm_source=
> >>>>> footer> .

[Prathik Rajendran M]

Let's say you run --clear often and in most cases it won't cause a problem and the same message is shown, let's say someone else in your team set the STATIC_FOLDER to a file in the app, in this case wouldn't you want an explicit message that unlike in the previous scenarios where STATIC_FOLDER is separate, in this scenario the files deleted cannot be brought back? I think this would make django safer.

Shai, overwrite is just a proposal, I basically wanted a way in which even if the files in STATIC_ROOT were newer they'd get replaced without having to delete all the files.