Install over old level produces mutant

[Karen Tracey]

We've had at least three people run into a problem (#9090) after installing 1.0 over an old level.  Now, we do document that before installing you should remove any old level.  But I find the results when you don't rather odd. It's not just old files renamed/deleted in the new level that wind up being 'leftover' in the site-packages directory. Rather, some files that exist in both levels, and changed between the two, are not updated by install.  So you wind up with a mixture of 0.96.x and 1.0 level files, which is pretty confusing and subject to odd failures.

Does anyone know if there is something we could do to prevent this situation?  I'm not very familiar with Python installation procedures.  It seems like it would be better, though, if our installer flat-out refused to install over a pre-existing installation or offered to delete the pre-existing install before continuing with the new install.  What's the norm for Python packages?

Karen

[alex....@gmail.com]

We could definately have it do the check pretty easily just by shoe
horning some stuff at the top of setup.py, actually offering to
uninstall might be more complicated, although it could probably be
done in the same way.

[Steve Holden]

I suspect what's happening is that the install keeps the distribution
dates on the .py files, but that the people who see the problems have
installed (or at least recompiled) the older versions more recently, so
the interpreter doesn't require an overwrite of the 0.96 files.

I've just done an install of 1.0 on my Python 2.6 alpha2 distribution
and I see, for example, that .../django/views/static.py is dated
12/17/2007. In this case there wasn't a previous installation, so I
can't verify that the compilation won't take place if there's an
existing .pyc file. However I *did* verify that a clean re-install
(after deletion of the appropriate build subdirectories) did *not*
change the creation time of the .py file.

So the answer is to put another step in the build process that touches
all files before bundling them up for distribution. Obviously this is
best done outside the version control system.

regards
Steve

[Steve Holden]

Steve Holden wrote:
[...]

> I've just done an install of 1.0 on my Python 2.6 alpha2 distribution
> and I see, for example, that .../django/views/static.py is dated
> 12/17/2007. In this case there wasn't a previous installation, so I
> can't verify that the compilation won't take place if there's an
> existing .pyc file. However I *did* verify that a clean re-install
> (after deletion of the appropriate build subdirectories) did *not*
> change the creation time of the .py file.

Sorry, that should have been ".pyc file".
>
regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/

[Fredrik Lundh]

Karen Tracey wrote:

> We've had at least three people run into a problem (#9090) after
> installing 1.0 over an old level. Now, we do document that before
> installing you should remove any old level. But I find the results when
> you don't rather odd. It's not just old files renamed/deleted in the new
> level that wind up being 'leftover' in the site-packages directory.
> Rather, some files that exist in both levels, and changed between the
> two, are not updated by install. So you wind up with a mixture of
> 0.96.x and 1.0 level files, which is pretty confusing and subject to odd
> failures.

does that apply to PY files, or is it just that you may have PYC files
from 0.96.x but PY files from 1.0? (Python only looks at the timestamp
when deciding whether to recompile a PY file, so if your PYC files for
0.96.x were created later than the corresponding 1.0 file, Python will
pick up the wrong file).

is the installer you're referring to here "setup.py", or something else?

</F>

[Steve Holden]

See ticket #9112 for a purported explanation.

regards
Steve

[Karen Tracey]

I do not believe the description/explanation in that ticket is correct.  It says the new .py files will be copied but .pyc files will not be created, which is not what I have observed.  What I have seen is that new .py files are not copied at all.  The tree is left with a mixture of old and new .py files.  (And to answer Fredrik's question yes, I'm talking about 'setup.py install'.)

It does seem to have been caused by inconsistent file dating during the release process over time.  One problematic file is django/utils/dates.py.  This file was last modified in SVN on 7/4/2007 at 8:11AM, and that is the date/time on the file when extracted from the 1.0 release tar file.

However, going back to the 0.96.2 tarball, the date on its django/utils/dates.py file is 10/29/2007.  I'm not sure where that date came from.  0.96.2 was announced on django-announce on 5/14/2008, so it's not when 0.96.2 came out.  The actual revision level for the file is [2912], which was checked in on 5/16/06.  And there's the problem -- it's an earlier level of the file than what's in 1.0, but it appears based on the date to be more recent.

Going back to the 0.96.1 tarball (yes I'm a packrat), the date on its django/utils/dates.py is 10/26/2007, which is the same date as 0.96.1 was announced. 

So the problem seems to be that some of the older tar files for releases used tar file creation date (or ? since I'm not sure about the 0.96.2 one) for files contained in the release, while 1.0 somehow put last-modified in SVN dates on the files (I didn't know SVN would even do that, actually).  Seems it would be safer to use tarfile creation date always.

That still leaves potential problems for people doing install-overs.  Old files that have been renamed/deleted won't get deleted, and that can cause problems.  Even though we say you should first delete old levels, and even if the release process is adjusted to fix up the dates, I think it would be a good idea to refuse to install if an old level is detected. 

Karen