Disable use of pyc file with no matching py file

[Roy Smith]

Every so often (typically when refactoring), I'll remove a .py file and forget to remove the corresponding .pyc file. If I then import the module, python finds the orphaned .pyc and happily imports it. Usually leading to confusing and hard to debug failures.

Is there some way to globally tell python, "Never import a .pyc unless the corresponding .py file exits"? Perhaps some environment variable I could set in my .login file?

[Miki Tebeka]

Not that I'm aware of.

I have a script that run the test suite, one of the first commands (before calling nosetests) is:
find . -name '*.py[co]' -exec rm {} \;

This makes sure the tests run in a "clean" environment.

[Terry Reedy]

On 1/30/2012 4:30 PM, Roy Smith wrote:
> Every so often (typically when refactoring), I'll remove a .py file
> and forget to remove the corresponding .pyc file. If I then import
> the module, python finds the orphaned .pyc and happily imports it.
> Usually leading to confusing and hard to debug failures.
>
> Is there some way to globally tell python, "Never import a .pyc
> unless the corresponding .py file exits"?

Upgrade to 3.2.

--
Terry Jan Reedy

[Jean-Michel Pichavant]

Terry Reedy wrote:
> On 1/30/2012 4:30 PM, Roy Smith wrote:

>> Every so often (typically when refactoring), I'll remove a .py file
>> and forget to remove the corresponding .pyc file. If I then import
>> the module, python finds the orphaned .pyc and happily imports it.
>> Usually leading to confusing and hard to debug failures.
>>
>> Is there some way to globally tell python, "Never import a .pyc
>> unless the corresponding .py file exits"?
>

> Upgrade to 3.2.
>
No.

JM

[Andrea Crotti]

On 01/30/2012 09:30 PM, Roy Smith wrote:
> Every so often (typically when refactoring), I'll remove a .py file and forget to remove the corresponding .pyc file. If I then import the module, python finds the orphaned .pyc and happily imports it. Usually leading to confusing and hard to debug failures.
>
> Is there some way to globally tell python, "Never import a .pyc unless the corresponding .py file exits"? Perhaps some environment variable I could set in my .login file?

If you want to stay python only I have this function:

def clean_orphaned_pycs(directory, simulate=False):
"""Remove all .pyc files without a correspondent module
and return the list of the removed files.
If simulate=True don't remove anything but still return
the list of pycs
"""

exists, join = path.exists, path.join
rm = (lambda _: None) if simulate else remove
removed = []

for root, dirs, files in walk(directory):
for f in files:
if f.endswith(".pyc"):
pyc = join(root, f)
if not exists(pyc[:-1]):
logger.debug("DELETING orphaned file %s" % pyc)
removed.append(pyc)
rm(pyc)

# ignore certain sub-dirs
for i in reversed(range(len(dirs))):
d = dirs[i]
if d == ".svn" or d.endswith(".egg-info"):
dirs.pop(i) # dirs.remove(d)

return removed

[Ben Finney]

Terry's response was an answer to “Is there some way to [do what Roy
Smith asked how to do]?”. Are you saying that upgrading to Python 3.2
doesn't work for that?

--
\ “Faith is belief without reason. I think as soon as you turn |
`\ your back on reason, you turn your back on what makes us |
_o__) human.” —Iain M. Banks, 2010-10-07 |
Ben Finney

[Steven D'Aprano]

Is that intended as "No, I won't upgrade" or "No, Python 3.2 doesn't do
the job"?

If the first one, then that's your decision, of course, but there really
is no other straight-forward way to have Python ignore left-over .pyc
files. Like it or not, there is no way to disable the use of .pyc files
in older versions of Python.

(I suppose you could write a patch for Python, and use your custom
version, but that's it.)

The ability to use .pyc files without providing source code is considered
by some people a feature, not a bug, and in fact even in Python 3.2 the
same applies. The difference in 3.2 is that you can't *accidentally* use
old left-over .pyc files, you have to move and rename them before they
can be used as sourceless modules.



--
Steven

[Jean-Michel Pichavant]

Steven D'Aprano wrote:
> On Tue, 31 Jan 2012 11:26:10 +0100, Jean-Michel Pichavant wrote:
>
>
>> Terry Reedy wrote:
>>
>>> On 1/30/2012 4:30 PM, Roy Smith wrote:
>>>
>>>> Every so often (typically when refactoring), I'll remove a .py file
>>>> and forget to remove the corresponding .pyc file. If I then import
>>>> the module, python finds the orphaned .pyc and happily imports it.
>>>> Usually leading to confusing and hard to debug failures.
>>>>
>>>> Is there some way to globally tell python, "Never import a .pyc unless
>>>> the corresponding .py file exits"?
>>>>
>>> Upgrade to 3.2.
>>>
>>>
>> No.
>>
>
>
> Is that intended as "No, I won't upgrade" or "No, Python 3.2 doesn't do
> the job"?
>

To answer Ben's mail as well, the "No" would be more of the "don't do
it". My answer was as argued as Terry's one anyway (it was quite intended).
Steven, you often use analogies/similarities, here's one:

A: "My wheel is flat"
B: "Buy a new car"

Buying a new car would solve A's problem : yes
Should A buy a new car : probably no

Python 3.2 is fine, but someone could run into several issues while
migrating. This is quite a pretty huge decision to make (I dedicate this
sentence to Rick, I hope he's trolling fine).

JM

[John Roth]

Terry,

I've noticed that the tutorial (section 6.1.3) hasn't been updated for
PEP 3147; there's no way of telling that this is the behavior from
reading the tutorial. The development doc for 3.3 hasn't been updated
either.

[Terry Reedy]

On 1/31/2012 3:20 PM, John Roth wrote:
> On Jan 30, 3:43 pm, Terry Reedy<tjre...@udel.edu> wrote:
>> On 1/30/2012 4:30 PM, Roy Smith wrote:
>>
>>> Every so often (typically when refactoring), I'll remove a .py file
>>> and forget to remove the corresponding .pyc file. If I then import
>>> the module, python finds the orphaned .pyc and happily imports it.
>>> Usually leading to confusing and hard to debug failures.
>>
>>> Is there some way to globally tell python, "Never import a .pyc
>>> unless the corresponding .py file exits"?
>>
>> Upgrade to 3.2.

I tested before writing this. The caveat is that x.pyc in the same
directly as x.py will not be ignored (for back compatibility). However,
this only happens intentionally as .pyc files are put in __pycache__/
with name x.<version>.pyc, where <version> is 'cpython-32' or something
similar for another version or implementation.

> I've noticed that the tutorial (section 6.1.3) hasn't been updated for
> PEP 3147; there's no way of telling that this is the behavior from
> reading the tutorial. The development doc for 3.3 hasn't been updated
> either.

You are right. An oversight. Thanks for noticing.
http://bugs.python.org/issue13915
Suggested rewrites are welcome.

--
Terry Jan Reedy

[Terry Reedy]

On 1/31/2012 9:19 AM, Jean-Michel Pichavant wrote:

> A: "My wheel is flat"
> B: "Buy a new car"

A better analogy would be

Q. "How do I make my old model car do something (it cannot do)?"
A. "Get the free new model that has that feature added."

Of course, there is a cost to giving up the old and familiar and
learning and adjusting to the new, even when it is available gratis. A
husband wearing an old sweater after his wife gives him a new one, and
even retrieving it from the trash when she tosses it out, is a classic
(and true) cartoon joke.

But I am sure that 95% of readers here will be using 3.x withing 10
years. The only question for them is "When?". This not-well-known new
feature is one straw that some will put on the 'sooner' side of the balance.

--
Terry Jan Reedy

[Roy Smith]

In article <mailman.5264.1328054...@python.org>,

Terry Reedy <tjr...@udel.edu> wrote:

> But I am sure that 95% of readers here will be using 3.x withing 10
> years. The only question for them is "When?". This not-well-known new
> feature is one straw that some will put on the 'sooner' side of the balance.

We would love to move to 3.x, for the better unicode support, if nothing
else. What's keeping us from doing so is the host of third-party
modules and tools we depend on that don't yet support 3.x. Off the top
of my head (it's possible some of these already have support):

django (I understand it's been done, if not yet officially supported)
pymongo
mongoengine
tornado
pybeanstalk
dateutils
requests
lxml
blinker
gunicorn
I'm sure I've missed a few

It's getting to the point where any open-source python project needs to
either jump on the 3.x bandwagon or get consigned to obscurity. I'm
guessing that's going to start happening in a big way in 2012.

[Jean-Michel Pichavant]

Terry Reedy wrote:
> On 1/31/2012 9:19 AM, Jean-Michel Pichavant wrote:
>
>> A: "My wheel is flat"
>> B: "Buy a new car"
>
> A better analogy would be
>
> Q. "How do I make my old model car do something (it cannot do)?"
> A. "Get the free new model that has that feature added."
>
> Of course, there is a cost to giving up the old and familiar and
> learning and adjusting to the new, even when it is available gratis. A
> husband wearing an old sweater after his wife gives him a new one, and
> even retrieving it from the trash when she tosses it out, is a classic
> (and true) cartoon joke.
>

> But I am sure that 95% of readers here will be using 3.x withing 10
> years. The only question for them is "When?". This not-well-known new
> feature is one straw that some will put on the 'sooner' side of the
> balance.
>

A simple solution to that problem has been provided by Miki. Someone
should read at least http://wiki.python.org/moin/Python2orPython3 before
migrating to python 3.

Speaking for myself, I'm stuck with python 2.5 :-/

JM

[Terry Reedy]

On 1/31/2012 11:14 PM, Roy Smith wrote:

> We would love to move to 3.x, for the better unicode support, if nothing
> else. What's keeping us from doing so is the host of third-party
> modules and tools we depend on that don't yet support 3.x.

Tell that to the authors of packages you use so they no longer say that
they have not converted for lack of demand ;-)

--
Terry Jan Reedy

[Devin Jeanpierre]

On Tue, Jan 31, 2012 at 6:55 PM, Terry Reedy <tjr...@udel.edu> wrote:
> Q. "How do I make my old model car do something (it cannot do)?"
> A. "Get the free new model that has that feature added."
>
> Of course, there is a cost to giving up the old and familiar and learning
> and adjusting to the new, even when it is available gratis. A husband
> wearing an old sweater after his wife gives him a new one, and even
> retrieving it from the trash when she tosses it out, is a classic (and true)
> cartoon joke.

It really bothers me that you imagine that there are no other problems
than the newness. It's disheartening, because the problems are not
that trivial and the world would be better if people were less callous
about it, and realized that they exist. Python 3 is not very different
from Python 2, as far as humans are concerned
semantically/syntactically -- but, hell, just pick any project that
uses PyPy, or Jython, or IronPython, or Twisted, or Zope, etc. -- it
can be a lot of effort (sometimes infeasibly much) to port something
dependent on these things, and it's taken years to get the (smallish)
set of dependencies ported that we have now [and we literally paid
people to do it, too!], and still many large projects haven't made the
transition, and many small projects never will.

Anyone that relies on those projects is stuck, and your "free car"
metaphor completely ignores the true cost of wasting that much time
porting everything for a tiny little feature. Evaluating only the
monetary amounts can be misleading as to what the rational decision is
(in particular when there are no monetary amounts). The only true
notion of cost is the alternatives you sacrifice in making a decision:
opportunity cost. The car is not free.

-- Devin

[John Roth]

> You are right. An oversight. Thanks for noticing.http://bugs.python.org/issue13915

> Suggested rewrites are welcome.
>
> --
> Terry Jan Reedy

I'll see if I can put a suggestion in the bug report.

One other point: I'm unclear if a compiled module in the source
directory would be named spam.pyc or spam.cpython-32.pyc. I'd think
the latter to allow two versions of a compiled-only distribution.

John Roth

[Terry Reedy]

On 2/1/2012 6:14 AM, Devin Jeanpierre wrote:

> It really bothers me that you imagine that there are no other problems
> than the newness.

And it bothers me that you imput such ignorance to me. You made what I
think was a bad analogy and I made a better one of the same type, though
still imperfect. I acknowledged that the transition will take years.

--
Terry Jan Reedy

[Terry Reedy]

On 2/1/2012 8:11 AM, John Roth wrote:

> One other point: I'm unclear if a compiled module in the source
> directory would be named spam.pyc or spam.cpython-32.pyc. I'd think
> the latter to allow two versions of a compiled-only distribution.

By test, it has to be spam.pyc, as before.

--
Terry Jan Reedy

[Devin Jeanpierre]

On Wed, Feb 1, 2012 at 2:53 PM, Terry Reedy <tjr...@udel.edu> wrote:
> And it bothers me that you imput such ignorance to me. You made what I think
> was a bad analogy and I made a better one of the same type, though still
> imperfect. I acknowledged that the transition will take years.

Ah. It is a common attitude among those that make these sorts of
comments about Python 3, and I hadn't read anything in what you said
that made me think that you were considering more than the superficial
costs of moving. I am sorry that I did not give you the benefit of the
doubt.

-- Devin

[Terry Reedy]

On 2/2/2012 1:42 AM, Devin Jeanpierre wrote:
> On Wed, Feb 1, 2012 at 2:53 PM, Terry Reedy<tjr...@udel.edu> wrote:
>> And it bothers me that you imput such ignorance to me. You made what I think
>> was a bad analogy and I made a better one of the same type, though still
>> imperfect. I acknowledged that the transition will take years.
>
> Ah. It is a common attitude among those that make these sorts of
> comments about Python 3, and I hadn't read anything in what you said
> that made me think that you were considering more than the superficial
> costs of moving.

I thought '95% in 10 years' would be a hint that I know upgrading is not
trivial for every one ;-).

> I am sorry that I did not give you the benefit of the doubt.

Apology accepted.

--
Terry Jan Reedy

[Brian]

This is OT, but relevant to the side discussion. Also note that
the community respects and appreciates the T.J. Reedy
contributions and hard work.

Reality. It was a monumental task to convert ATE driver dev and
related factory automation stuff from C to Python starting in
2000. Most was done guerrilla-style. My employer now appreciates
the resultant infrastructure that even the Mexico factory
engineering team can use, and is good for their pride because
they are no longer totally dependent on gringo engineers.

Am currently being disruptive to their little world with my
recent (1Q 2011) switch to 3.x. The corporate natives are
restless and there is talk of an armed insurrection. Humans, at
the tribal level, do not adapt to change. Expect a long series
of battles that are ruthless, bloody, and have a high body
count. Vive la Revolution.