Python 3.3 and .pyo files

[Marco]

I was trying to import a pyo module in Python 3.3, but Python does not
find it:

$ echo "print(__file__)" > foo.py
$ python3.3 -O -m foo
/home/marco/temp/foo.py
$ ls
foo.py __pycache__
$ rm foo.py
$ mv __pycache__/foo.cpython-33.pyo foo.pyo
$ rm __pycache__ -r
$ ls
foo.pyo
# The following works in Python3.2, but not in 3.3
$ python3.3 -O -m foo
/usr/local/bin/python3.3: No module named foo

How come? Thanks in advance, Marco

--
Marco

[Steven D'Aprano]

On Fri, 21 Sep 2012 11:10:07 +0200, Marco wrote:

> I was trying to import a pyo module in Python 3.3, but Python does not
> find it:
>
> $ echo "print(__file__)" > foo.py
> $ python3.3 -O -m foo
> /home/marco/temp/foo.py
> $ ls
> foo.py __pycache__
> $ rm foo.py
> $ mv __pycache__/foo.cpython-33.pyo foo.pyo

I cannot duplicate the creation of the foo.cpython-33.pyo file using just
the -m option. I believe that you created the foo*.pyo file some other
way. Nevertheless, moving along:

> $ rm __pycache__ -r
> $ ls
> foo.pyo
> # The following works in Python3.2, but not in 3.3
> $ python3.3 -O -m foo
> /usr/local/bin/python3.3: No module named foo

I can confirm that (1) it works using Python 3.2; (2) it doesn't work
using Python 3.3; and (3) it does work in Python 3.3 if you don't use the
-O option.

I believe that is a bug.

(Tested using Python 3.2.2 and Python 3.3.0a1)


--
Steven

[Marco]

On 09/21/2012 02:55 PM, Steven D'Aprano wrote:

>> $ ls
>> >foo.pyo
>> ># The following works in Python3.2, but not in 3.3
>> >$ python3.3 -O -m foo
>> >/usr/local/bin/python3.3: No module named foo
>
> I can confirm that (1) it works using Python 3.2; (2) it doesn't work
> using Python 3.3; and (3) it does work in Python 3.3 if you don't use the
> -O option.

It doesn't work with Python 3.3.0rc2 too.

--
Marco

[Terry Reedy]

On 9/21/2012 5:10 AM, Marco wrote:
> I was trying to import a pyo module in Python 3.3, but Python does not
> find it:

You appear to be trying to *run*, not *import* a .pyo module.

> $ echo "print(__file__)" > foo.py
> $ python3.3 -O -m foo

Since foo.py is in the current directory, I am not sure why you use '-m
foo' instead of 'foo.py'. -m is for running a module somewhere on sys.path.

> /home/marco/temp/foo.py
> $ ls
> foo.py __pycache__
> $ rm foo.py
> $ mv __pycache__/foo.cpython-33.pyo foo.pyo
> $ rm __pycache__ -r
> $ ls
> foo.pyo
> # The following works in Python3.2, but not in 3.3
> $ python3.3 -O -m foo

I would try just 'foo.pyo' in case the -m part is the problem.
Also, the -O is sort of redundant, or perhaps interfering, since its
usual effect to to say 'get and put, from and to the cache, .pyo instead
of .pyc'.

> /usr/local/bin/python3.3: No module named foo
>
> How come? Thanks in advance, Marco

You might read some of http://bugs.python.org/issue12982

in particular, from http://bugs.python.org/issue12982#msg162814

"Indeed, since I posted last night, the pydev discussion has moved to
the question of whether -O, __debug__, and .pyo as now defined are worth
the nuisance they cause or whether some or all should be deprecated.
(Docstring stripping for saving space could then be a separate tool.)
---
Python interpreters exist to run Python code. The existence,
persistence, and other details of compilation caches are
version-dependent implementation details. Being able to execute from
such caches without source present is also an implementation detail, and
for CPython, it gets secondary support at best. (This is a compromise
between full support and no support.)"

--
Terry Jan Reedy

[Steven D'Aprano]

On Fri, 21 Sep 2012 22:46:08 -0400, Terry Reedy wrote:

> On 9/21/2012 5:10 AM, Marco wrote:
>> I was trying to import a pyo module in Python 3.3, but Python does not
>> find it:
>
> You appear to be trying to *run*, not *import* a .pyo module.

Marco is using the standard mechanism for finding, importing, and running
a module. I don't believe his use of -m should be a problem. It works in
3.2, and it works with .pyc files in 3.3, I see nothing to suggest it
shouldn't work with .pyo files in 3.3.

>> $ echo "print(__file__)" > foo.py
>> $ python3.3 -O -m foo
>
> Since foo.py is in the current directory, I am not sure why you use '-m
> foo' instead of 'foo.py'. -m is for running a module somewhere on
> sys.path.

Yes, and the current directory is on sys.path.

I would be astonished if python -m could not find a module that happened
to be in the current directory.


[...]

> Also, the
> -O is sort of redundant, or perhaps interfering, since its usual effect
> to to say 'get and put, from and to the cache, .pyo instead of .pyc'.

No it is not redundant. You link specifically to an bug tracker issue
below where is is clearly decided that if you want to run a .pyo file you
*must* use the -O switch. (I approve of this decision.)

>> /usr/local/bin/python3.3: No module named foo
>>
>> How come? Thanks in advance, Marco
>
> You might read some of http://bugs.python.org/issue12982
>
> in particular, from http://bugs.python.org/issue12982#msg162814

Whose words are these following?

> Python interpreters exist to run Python code. The existence,
> persistence, and other details of compilation caches are
> version-dependent implementation details. Being able to execute from
> such caches without source present is also an implementation detail, and
> for CPython, it gets secondary support at best. (This is a compromise
> between full support and no support.)"

I'm not sure if these are your words, or if you are quoting some random
commenter on the pydev list, or one of the lead developers who might
actually know what he is talking about.

As I recall, some time in the recent past Guido came down *hard* against
the suggestion that support for running sourceless files (.pyc and .pyo)
should be dropped. Even if I'm misremembering, it is the case that the
Python 3.3 will find and run a .pyc file, but not a .pyo file. There's a
new candidate release of 3.3 due out over the next couple of days. If it
shows the same behaviour, it should be reported as a bug.



--
Steven

[Ramchandra Apte]

It is Terry J. Reedy's words - see end of http://bugs.python.org/issue12982#msg162814

[Terry Reedy]

My words summarizing the discussion on pydev which included at least a
few lead developers. My initial post was probably 6/12/2012


--
Terry Jan Reedy

[Steven D'Aprano]

On Fri, 21 Sep 2012 11:10:07 +0200, Marco wrote:

> I was trying to import a pyo module in Python 3.3, but Python does not
> find it:

[...]

Marco, this bug is apparently now fixed:

http://bugs.python.org/issue16046

Please download the latest version of Python 3.3 and try again.

(Folks, this is why testing and reporting bugs to beta and rc versions is
so useful. You can often get them fixed quite quickly.)



--
Steven