Relative import from script with same name as package
OKB (not okblacke)
thetest/
__init__.py
thetest.py
theother.py
__init__.py is an empty file. theother.py contains a function foo().
The package is accessible from sys.path, so that if I open the
interpreter and do "import thetest" or "from thetest import thetest" or
"import thetest.thetest", it works fine.
Inside thetest.py I have code like this:
###
from __future__ import absolute_import
if __name__ == "__main__" and __package__ is None:
import thetest
__package__ = "thetest"
from .theother import foo
###
Note that I need the "import thetest" line to avoid a "parent
module not loaded" error, as described here:
http://stackoverflow.com/questions/2943847/nightmare-with-relative-
imports-how-does-pep-366-work
If I run foo.py directly, I receive a traceback like this:
Traceback (most recent call last):
File "C:\...\thetest\thetest.py", line 4, in <module>
import thetest
File "C:\...\thetest\thetest.py", line 11, in <module>
from .theother import foo
ValueError: Attempted relative import in non-package
It appears that Python is reading "import thetest" as importing
thetest.py (the same file that is currently being run). When it tries
to run that file a second time, the relative import fails.
But why? That __future__ import is supposed to make absolute
imports the default, so why is "import thetest" importing thetest.py
instead of the package called thetest? The absolute import should make
it look in sys.path first and not try to import from the script
directory, right?
If I change the outer directory name and change the code in
thetest.py to match, it works fine. But I shouldn't have to do this.
How can I get relative imports to work correctly when running a script
whose filename is the same as that of the directory (and thus the
package) in which it resides?
--
--OKB (not okblacke)
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is
no path, and leave a trail."
--author unknown
OKB (not okblacke)
> But why? That __future__ import is supposed to make
> absolute
> imports the default, so why is "import thetest" importing
> thetest.py instead of the package called thetest? The absolute
> import should make it look in sys.path first and not try to import
> from the script directory, right?
>
> If I change the outer directory name and change the code
> in
> thetest.py to match, it works fine. But I shouldn't have to do
> this. How can I get relative imports to work correctly when
> running a script whose filename is the same as that of the
> directory (and thus the package) in which it resides?
After a bit more googling I discovered the answer here:
http://stackoverflow.com/questions/1959188/absolute-import-failing-in-
subpackage-that-shadows-a-stdlib-package-name
The deal is that sys.path by default has the empty string as the
first element, which tells Python to look first in the directory of the
script being executed. This is unfortunate, but can worked around this
way:
import sys
sys.path = sys.path[1:] + ['']
(That is, move the current directory to the end of the search path
instead of the beginning.)
Chris Angelico
<brenNOS...@nobrenspambarn.net> wrote:
> sys.path = sys.path[1:] + ['']
>
> (That is, move the current directory to the end of the search path
> instead of the beginning.)
>
Or, equivalently:
sys.path.append(sys.path.pop(0))
ChrisA
Oktay Şafak
> I'm using Python 2.6.5. I have a directory structure like this:
>
> thetest/
> __init__.py
> thetest.py
> theother.py
>
> __init__.py is an empty file. theother.py contains a function foo().
> The package is accessible from sys.path, so that if I open the
> interpreter and do "import thetest" or "from thetest import thetest" or
> "import thetest.thetest", it works fine.
It should.
>
> Inside thetest.py I have code like this:
>
> ###
> from __future__ import absolute_import
>
> if __name__ == "__main__" and __package__ is None:
> import thetest
> __package__ = "thetest"
>
> from .theother import foo
> ###
>
...
>
> If I run foo.py directly, I receive a traceback like this:
>
Wait! What's foo.py? I guess you mean thetest.py as you say "(the same
file that is currently being run)" below.
> Traceback (most recent call last):
> File "C:\...\thetest\thetest.py", line 4, in<module>
> import thetest
> File "C:\...\thetest\thetest.py", line 11, in<module>
> from .theother import foo
> ValueError: Attempted relative import in non-package
>
> It appears that Python is reading "import thetest" as importing
> thetest.py (the same file that is currently being run). When it tries
> to run that file a second time, the relative import fails.
>
No, there is no such thing happening. Read the error message more
carefully: the error happens when your code reaches the line "from
.theother import foo", and it fails because you are trying to execute an
"explicit" relative import statement (with leading dot notation) as
introduced by PEP 328. What you see is perfectly expected behaviour as
explained in detail in the PEP because the python interpreter can only
make sense of that statement if that code is *imported* for use by code
that resides *outside* the package. That error message is what you see
when you try to *run* a package member module which uses explicit
relative imports. Let me try to explain a bit further:
You say that your package is accessible from sys.path. So let's say you
fire up your interpreter and type
import thetest.thetest
Here the interpreter first imports the *package* thetest and then the
module *thetest.py*, therefore it knows that the module is a member of
the package. And when executing the code in the module thetest.py, it
can make sense of the leading dot in "import .theother", it can see that
the leading dot means the packege thetest and find the right module
inside that package.
But when you run thetest.py directly, the interpreter has no idea what
the leading dot means because it has not imported any package
beforehand. So it complains about that.
> But why? That __future__ import is supposed to make absolute
> imports the default, so why is "import thetest" importing thetest.py
> instead of the package called thetest?
This is not what is happening.
> The absolute import should make
> it look in sys.path first and not try to import from the script
> directory, right?
As I explained above, that's not the error but I think this sentence
shows your misunderstanding of this relative imports issue. Never mind,
I found the documentation very confusing at first and took me a great
deal of effort to finally "get it". Let me share my "distilled wisdom"
about this:
- First of all, never perform your experiments about relative/absolute
imports using IDLE! I guess IDLE has its own path manipulation that
doesn't do *the right thing*.
- "Absolute import" means importing from sys.path only.
- "Relative import" means a "package member" module importing another
"package member", using a not fully qualified name (i.e X.Y.a importing
X.Y.b as import b, instead of using import X.Y.b). When Python saw
(before PEP 328) "import b" or "from b import ..." *in a package
member*, it first imported the local b if available. (This business of a
package member importing another package member is called relative
import, only) Since this had the same notation (syntax) with other (non
package member) imports, intra-package imports were considered
"ambiguous" when the module to be imported had the same name with a
module reachable from sys.path: it was not immediately clear which one
was meant for untrained eyes. Also, it had the potential of shadowing a
stdlib module: in that case no clean method of getting at the stdlib
module was available.
- PEP 328 is about giving "relative imports" a new syntax so that this
ambiguity no longer exists and the possibility of shadowing a stdlib
module is removed altogether. The new syntax is called "*explicit*
relative imports". The former method can thus be called "implicit
relative imports" or "ambiguous relative imports"
- BIG SOURCE OF CONFUSION: Relative imports (both implicit and explicit)
is all about imports *inside* packages. When you have 2 scripts app.py
and string.py sitting in a *non-package* folder (they are called
top-level modules), that folder is always the first entry in sys.path
when you run either of those modules directly, so "import string" in
app.py will always import the string.py in that folder, not the stdlib
one. (But not when you try with IDLE!) In this case, if you really want
the standard module, you must manipulate sys.path before your import
statement (or change the name of your local string.py). THE PEP (328)
HAS NO EFFECT IN THAT CASE!
- So, "relative imports" is not a concept introduced by PEP 328, it
means "importing a package member from another member of that package"
and it existed well before PEP 328. But its syntax was ambiguous so it
was clarified by PEP 328 by giving it a special syntax (the leading dot
notation). But that change introduced many small gotchas, including the
one you are seeing: you can't directly run a package member module if
you have used this new syntax (leading dot notation) in it. It works
fine only when you import the package members for use by a module
outside of that package. This of course brings difficulties with testing
but it is another story.
I think you will now find it much easier to get what PEP 328 is all about.
Oktay
OKB (not okblacke)
>> It appears that Python is reading "import thetest" as
>> importing
>> thetest.py (the same file that is currently being run). When it
>> tries to run that file a second time, the relative import fails.
>>
>
> No, there is no such thing happening. Read the error message more
> carefully: the error happens when your code reaches the line "from
> .theother import foo", and it fails because you are trying to
> execute an "explicit" relative import statement (with leading dot
> notation) as introduced by PEP 328. What you see is perfectly
> expected behaviour as explained in detail in the PEP because the
> python interpreter can only make sense of that statement if that
> code is *imported* for use by code that resides *outside* the
> package. That error message is what you see when you try to *run* a
> package member module which uses explicit relative imports. Let me
> try to explain a bit further:
Yes, such a thing was happening. (I described the fix in an
answer to my own post.) You should read PEP 366 to understand what I
was talking about, and what the __package__ variable does.
Oktay Şafak
> =?UTF-8?B?T2t0YXkgxZ5hZmFr?= wrote:
>
>>> It appears that Python is reading "import thetest" as
>>> importing
>>> thetest.py (the same file that is currently being run). When it
>>> tries to run that file a second time, the relative import fails.
>>>
>>
>> No, there is no such thing happening. Read the error message more
>> carefully: the error happens when your code reaches the line "from
>> .theother import foo", and it fails because you are trying to
>> execute an "explicit" relative import statement (with leading dot
>> notation) as introduced by PEP 328. What you see is perfectly
>> expected behaviour as explained in detail in the PEP because the
>> python interpreter can only make sense of that statement if that
>> code is *imported* for use by code that resides *outside* the
>> package. That error message is what you see when you try to *run* a
>> package member module which uses explicit relative imports. Let me
>> try to explain a bit further:
>
> Yes, such a thing was happening. (I described the fix in an
> answer to my own post.) You should read PEP 366 to understand what I
> was talking about, and what the __package__ variable does.
>
Hmmm, I have overlooked the middle part of your code. Now I get what you
mean. Yes that is happening, you are right and I have given the source
of that behaviour in my previous post:
> - BIG SOURCE OF CONFUSION: Relative imports (both implicit and explicit) is all about imports *inside* packages. When you have 2 scripts app.py and string.py sitting in a *non-package* folder (they are called top-level modules), that folder is always the first entry in sys.path when you run either of those modules directly, so "import string" in app.py will always import the string.py in that folder, not the stdlib one. (But not when you try with IDLE!) In this case, if you really want the standard module, you must manipulate sys.path before your import statement (or change the name of your local string.py). THE PEP (328) HAS NO EFFECT IN THAT CASE!
In your case, you are not using the package by importing its members
from code outside the package but you are running them directly. So,
your files thetest.py and theother.py are now *top-level* modules for
the interpreter, not package members. So as per PEP 366, you need to add
that boilerplate to set the __package__ attribute.
But the "from __future__ import absolute_import" statement has no effect
on the "import thetest" statement when you run the file directly: then
thetest.py is a top level module, and when the interpreter encounters
the statement "import thetest" *it is already absolute import* (whether
you issed the __future__ statement or not). It still is in Python 2.7.
PEP 328 has no effect in this case. Remember that absolute import means
importing from sys.path only. In this case, thetest.py's folder is
always the first entry in sys.path so it tries to import itself first.
You would also see the same behaviour if your "import thetest" statement
was in theother.py, it would import thetest.py rather than the package.
So as I have said before and quoted above, in this case you have to
manipulate sys.path, which you have found out yourself.
So what is this "absolute import" thing that is introduced by PEP 328
then? It is all about import statements in package members when they are
imported from code outside the package. Before PEP 328, if theother.py
contained an "import thetest" statement and if you fired up your
interpreter and issued
import thetest.theother
that "import thetest" in theother.py would find thetest.py. It was
considered to be a(n implicit) package-relative import. After PEP 328,
if you add to the top of theother.py or issue at the interpreter
from __future__ import absolute_import
and then issue
import thetest.theother
the "import thetest" statement in theother.py will find the *package*
thetest because sys.path is given priority over the package dir by the
__future__ statement. So the PEP has effectively overthrown the
convention of treating such imports as special. They are not treated as
special anymore and receive the same treatment as other imports. This is
one aspect of the PEP.
The PEP also defines a new notation (leading dot) for those who want the
sibling thetest.py, which is another aspect.
But when you run the theother.py directly, the PEP and the __future__
statement has NO EFFECT. The "import thetest" statement in theother.py
is not seen as intra package import by the interpreter so it is already
absolute import (uses sys.path) So it finds thetest.py because it is in
the same folder as theother.py, which is the first folder in sys.path.
Many people are confused and think that this is relative import. It's
not. It is already absolute import (from sys.path) and the PEP has
nothing to do with it. Now we can answer your critical question:
> But why? That __future__ import is supposed to make absolute
> imports the default, so why is "import thetest" importing thetest.py
> instead of the package called thetest?
Bacause you are running it directly and the PEP and the __future__
statement does not apply here.
I hope it is clear now.
--
Oktay Şafak