Feature request: collectstatic shouldn't recopy files that already exist in destination - Django developers

The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Message from discussion Feature request: collectstatic shouldn't recopy files that already exist in destination
Received-SPF: pass (google.com: domain of dloewenh...@gmail.com designates 74.125.82.42 as permitted sender) client-ip=74.125.82.42;
MIME-Version: 1.0
Sender: dloewenh...@gmail.com
In-Reply-To: <BFC83DCA-157E-4577-A32D-C52302300...@gmail.com>
References: <CAOKWLrHqJW+Db18n2Qbs8z4n4qVcfEJ5aZRPGuT6HkN47G5...@mail.gmail.com>
 <7516c233-e5f2-4109-8ecf-6f4a0054b798@googlegroups.com> <CAOKWLrHydT4M8LX8j9n5SrJRghZ1gWGM4RFbPGbO_zS_AzE...@mail.gmail.com>
 <BFC83DCA-157E-4577-A32D-C52302300...@gmail.com>
From: Dan Loewenherz <d...@dlo.me>
Date: Sat, 6 Oct 2012 10:48:56 -0700
Message-ID: <CAOKWLrGhKnhR7tiBhCO1UPp2-v-Ri5J1bTPyn-m2oPJtU1K...@mail.gmail.com>
Subject: Re: Feature request: collectstatic shouldn't recopy files that
 already exist in destination
To: django-developers@googlegroups.com
Content-Type: multipart/alternative; boundary=f46d0438eba99d51fb04cb679c13

--f46d0438eba99d51fb04cb679c13
Content-Type: text/plain; charset=UTF-8

Hey Jannis,

On Mon, Oct 1, 2012 at 12:47 AM, Jannis Leidel <lei...@gmail.com> wrote:

>
> On 30.09.2012, at 23:41, Dan Loewenherz <d...@dlo.me> wrote:
>
> > Many backends don't support last modified times, and even if they all
> did, it's incorrect to assume that last modified time is an accurate
> heuristic for whether a file has already been uploaded or not.
>
> Well but it's an accurate way to decide whether a file has been changed on
> the filesystem, and that's what collectstatic cares about. The storage
> backend *is* the API to extend that when needed, so feel free to use it.
>

It's accurate *only* in certain situations. And on a distributed
development team, I've run into a lot of issues with developers re-upload
files that have already been uploaded because they just recently updated
their repo.

A checksum is the only true accurate method to determine if a file has
changed.

Additionally, you didn't address my point that I quoted from. Storage
backends don't just reflect filesystems--they could reflect files stored in
a database, S3, etc. And some of these filesystems don't support last
modified times.

> It might be a better idea to let the backends decide when a file has been
> changed (instead of just calling the backend's last modified method).
>
> I don't understand, you can easily implement exactly that in the
> last_modified method if you'd like.
>

This is a bit confusing...why call it last_modified when that's doesn't
necessarily reflect what it's doing? It would be more flexible to create
two methods:

def modification_identifier(self):

def has_changed(self):

Then, any backend could implement these however they might like, and
collectstatic would have no excuse in uploading the same file more than
once. Overloading last_modified to also do things like calculate md5's
seems a bit hacky to me, and confusing for any developer maintaining a
custom storage backend that doesn't support last modified.

Dan

--f46d0438eba99d51fb04cb679c13
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hey Jannis,<div><br><div class=3D"gmail_quote">On Mon, Oct 1, 2012 at 12:47=
 AM, Jannis Leidel <span dir=3D"ltr">&lt;<a href=3D"mailto:lei...@gmail.com=
" target=3D"_blank">lei...@gmail.com</a>&gt;</span> wrote:<br><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;=
padding-left:1ex">

<div class=3D"im"><br>
On 30.09.2012, at 23:41, Dan Loewenherz &lt;<a href=3D"mailto:d...@dlo.me">d=
a...@dlo.me</a>&gt; wrote:<br>
<br>
&gt; Many backends don&#39;t support last modified times, and even if they =
all did, it&#39;s incorrect to assume that last modified time is an accurat=
e heuristic for whether a file has already been uploaded or not.<br>
<br>
</div>Well but it&#39;s an accurate way to decide whether a file has been c=
hanged on the filesystem, and that&#39;s what collectstatic cares about. Th=
e storage backend *is* the API to extend that when needed, so feel free to =
use it.<br>

</blockquote><div><br></div><div>It&#39;s accurate *only* in certain situat=
ions. And on a distributed development team, I&#39;ve run into a lot of iss=
ues with developers re-upload files that have already been uploaded because=
 they just recently updated their repo.</div>

<div><br></div><div>A checksum is the only true accurate method to determin=
e if a file has changed.</div><div><br></div><div>Additionally, you didn&#3=
9;t address my point that I quoted from. Storage backends don&#39;t just re=
flect filesystems--they could reflect files stored in a database, S3, etc. =
And some of these filesystems don&#39;t support last modified times.</div>

<div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex=
;border-left:1px #ccc solid;padding-left:1ex"><div class=3D"im">
&gt; It might be a better idea to let the backends decide when a file has b=
een changed (instead of just calling the backend&#39;s last modified method=
).<br>
<br>
</div>I don&#39;t understand, you can easily implement exactly that in the =
last_modified method if you&#39;d like.<br></blockquote><div><br></div><div=
>This is a bit confusing...why call it last_modified when that&#39;s doesn&=
#39;t necessarily reflect what it&#39;s doing? It would be more flexible to=
 create two methods:</div>

<div><br></div><div>def modification_identifier(self):</div><div><br></div>=
<div>def has_changed(self):</div><div><br></div><div>Then, any backend coul=
d implement these however they might like, and collectstatic would have no =
excuse in uploading the same file more than once. Overloading last_modified=
 to also do things like calculate md5&#39;s seems a bit hacky to me, and co=
nfusing for any developer maintaining a custom storage backend that doesn&#=
39;t support last modified.</div>

<div><br></div><div>Dan</div></div></div>

--f46d0438eba99d51fb04cb679c13--
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
Account Options
