Relative Imports, why the hell is it so hard?
CinnamonDonkey
I'm fairly new to Python so I still have a lot to learn. But I'd like
to know how to correectly use relative imports.
Please, please... please! don't go off on rants about why you think
relative imports should not be used. I've got 15+ years in C++ and
relative inclusion of other sections of code has never been a problem.
As far as I am concerned what I am trying to do is perfectly
reasonable and valid.
Thank you in advance to everyone who helps solve this, because I just
don't get it.
Example:
\ App
| main.py
+--\subpack1
| | __init__.py
| | module1.py
|
+--\subpack2
| | __init__.py
| | module2.py
Module1 needs to access functionality in Module2.
#module1.py
from ..subpack2 import module2
Seems reasonable to me... but it just does not work and I was so
liking Python. :(
Maxim Khitrov
Relative imports are perfectly fine, in my opinion. Do you have "from
__future__ import absolute_import" at the top of module1.py? Should
work fine once you add that line.
- Max
sk...@pobox.com
>> Please, please... please! don't go off on rants about why you think
>> relative imports should not be used. I've got 15+ years in C++ and
>> relative inclusion of other sections of code has never been a
>> problem. As far as I am concerned what I am trying to do is
>> perfectly reasonable and valid.
However, C++ != Python. Regardless whether or not you can "make it work",
translating idioms from one language to another is often suboptimal. That
may not be the case here, but it bears keeping in mind.
>> Example:
>> \ App
>> | main.py
>> +--\subpack1
>> | | __init__.py
>> | | module1.py
>> |
>> +--\subpack2
>> | | __init__.py
>> | | module2.py
>> Module1 needs to access functionality in Module2.
>> #module1.py
>> from ..subpack2 import module2
>> Seems reasonable to me... but it just does not work and I was so
>> liking Python. :(
You might try removing a dot from your import statement. Python is also not
Unix. Popping up one level in the package hierarchy is done with ".", not
"..".
--
Skip Montanaro - sk...@pobox.com - http://www.smontanaro.net/
CinnamonDonkey
Thanx for the quick responses, it is very much appreciated!
Skip, that's a good point about "C++ != Python" and I assure you I am
very much aware of that ;-).
Looking at http://www.python.org/dev/peps/pep-0328/#guido-s-decision
would suggest, unless I am completely miss-understanding the example,
that '.' refers to the current level and '..' pops up a level. First
three uses:
# Access moduleY in same level as ModuleX
from .moduleY import spam
from .moduleY import spam as ham
from . import moduleY
Just to be sure though I tried both ;):
from ..subpack2 import module1 #ValueError: Attempted relative import
beyond toplevel package
from .subpack2 import module1 #ImportError: No module named subpack2
from . import subpack2 #ImportError: cannot import name subpack2
from .. import subpack2 #ValueError: Attempted relative import beyond
toplevel package
Max, thank you for the response... I tried adding "from __future__
import absolute_import" which made no difference. I still get exactly
the same error messages. Perhaps I should have mentioned that I am
using Python 2.5, which I understand alread supports relative imports
out of the box. I'll keep this line in for now anyway though :-)
Cheers!
#subpack1.module1
from __future__ import absolute_import
from .. import subpack2
def subpack1_module1_foo():
print "subpack1_module1_foo()"
call_subpack2_module1()
def call_subpack2_module1():
subpack2.module2.subpack2_module2_foo()
#subpack2.module2
def subpack2_module2_foo():
print "subpack2_module2_foo()"
#main.py
import subpack1.module1
if __name__ == "__main__":
subpack1.module1.subpack1_module1_foo()
I am starting the sandbox app with the command line: python main.py
with the current working directory in \app where main.py is located.
Shaun >8)
> Skip Montanaro - s...@pobox.com -http://www.smontanaro.net/
Maxim Khitrov
<Cinnamo...@googlemail.com> wrote:
> Looking at http://www.python.org/dev/peps/pep-0328/#guido-s-decision
> would suggest, unless I am completely miss-understanding the example,
> that '.' refers to the current level and '..' pops up a level.
That is correct, but you cannot jump beyond the parent package, which
is why your code isn't working.
> Max, thank you for the response... I tried adding "from __future__
> import absolute_import" which made no difference. I still get exactly
> the same error messages. Perhaps I should have mentioned that I am
> using Python 2.5, which I understand alread supports relative imports
> out of the box. I'll keep this line in for now anyway though :-)
> Cheers!
Sorry, I use that line to avoid conflicts with standard modules, and
forgot that relative imports are already enabled.
Basically, the reason your code doesn't work is because you're trying
to use relative imports between two separate packages. As far as I
know, this isn't possible. What I did to get your code working was
move main.py one directory up and create an empty __init__.py under
\App. The following code should then work:
# main.py
import App.subpack1.module1
if __name__ == "__main__":
App.subpack1.module1.subpack1_module1_foo()
# App.subpack1.module1
from ..subpack2 import module2
def subpack1_module1_foo():
print "subpack1_module1_foo()"
call_subpack2_module1()
def call_subpack2_module1():
module2.subpack2_module2_foo()
# App.subpack2.module2
def subpack2_module2_foo():
print "subpack2_module2_foo()"
- Max
Gabriel Genellina
<Cinnamo...@googlemail.com> escribió:
>> >> \ App
>> >> | main.py
>> >> +--\subpack1
>> >> | | __init__.py
>> >> | | module1.py
>> >> |
>> >> +--\subpack2
>> >> | | __init__.py
>> >> | | module2.py
>>
>> >> Module1 needs to access functionality in Module2.
>>
>> >> #module1.py
>> >> from ..subpack2 import module2
>>
>> >> Seems reasonable to me... but it just does not work and I was so
>> >> liking Python. :(
Another name for relative imports is "intra-package imports". They work
*inside* a package, and you cannot go out of the package.
If App is not a package, then subpack1 and subpack2 are separate packages
and you cannot use relative imports between them. So module1 must refer to
module2 absolutely:
from subpack2 import module2
> from ..subpack2 import module1 #ValueError: Attempted relative import
> beyond toplevel package
See the exception message.
> Max, thank you for the response... I tried adding "from __future__
> import absolute_import" which made no difference. I still get exactly
> the same error messages. Perhaps I should have mentioned that I am
> using Python 2.5, which I understand alread supports relative imports
> out of the box. I'll keep this line in for now anyway though :-)
That __future__ line is not to enable relative imports (since they have
incompatible syntax, don't require anything special) but to ensure Python
interprets "normal" imports (that is, without leading dots) always as
absolute. The default behavior in 2.5 is to try *both* ways before failing.
--
Gabriel Genellina
CinnamonDonkey
package? and does that mean that what I am trying to do is not
possible ?
:(
On 23 Mar, 15:53, "Gabriel Genellina" <gagsl-...@yahoo.com.ar> wrote:
> En Mon, 23 Mar 2009 12:22:21 -0300, CinnamonDonkey
> <CinnamonDon...@googlemail.com> escribió:
Maxim Khitrov
<Cinnamo...@googlemail.com> wrote:
> My applogies if this is a silly question... but what makes something a
> package? and does that mean that what I am trying to do is not
> possible ?
A package is a directory that has an __init__.py file. That file can
be empty, or contain some initialization code. In your original
example, subpack1 and subpack2 are packages. By adding an empty
__init__.py file under \App, I made App into a package, which allowed
me to execute "import App.subpack1.module1" in main.py.
See the following url for additional info:
http://docs.python.org/tutorial/modules.html
- Max
Gabriel Genellina
<Cinnamo...@googlemail.com> escribió:
> My applogies if this is a silly question... but what makes something a
> package?
A package is a directory with an __init__.py file [that Python is aware
of].
> and does that mean that what I am trying to do is not
> possible ?
You can do an "absolute" import of subpack1 and subpack2. But you cannot
import them "relatively" - not using your current configuration, but see
Maxim Khitrov post for an alternate disposition that works.
--
Gabriel Genellina
rocky
wrote:
> Hi Guys,
>
> Thanx for the quick responses, it is very much appreciated!
>
> Skip, that's a good point about "C++ != Python" and I assure you I am
> very much aware of that ;-).
>
> would suggest, unless I am completely miss-understanding the example,
> that '.' refers to the current level and '..' pops up a level. First
> three uses:
>
> # Access moduleY in same level as ModuleX
> from .moduleY import spam
> from .moduleY import spam as ham
> from . import moduleY
>
> Just to be sure though I tried both ;):
>
> from ..subpack2 import module1 #ValueError: Attempted relative import
> beyond toplevel package
> from .subpack2 import module1 #ImportError: No module named subpack2
> from . import subpack2 #ImportError: cannot import name subpack2
> from .. import subpack2 #ValueError: Attempted relative import beyond
> toplevel package
>
> Max, thank you for the response... I tried adding "from __future__
> import absolute_import" which made no difference. I still get exactly
> the same error messages. Perhaps I should have mentioned that I am
> using Python 2.5, which I understand alread supports relative imports
> out of the box. I'll keep this line in for now anyway though :-)
> Cheers!
I wrote http://code.google.com/p/pyimport-relative so I could have a
simple and more reliable way to specify a *file* relative import in
Python 2.5 (and it works with 2.6 as well).
I'm not sure it's for everyone, or that it is flawless. Actually, it's
not. But I use it for in pydbgr (http://code.google.com/p/pydbgr) so
see that for examples of how to use. It allows me to run the
development code out of the source tree while having possibly a
different version installed as an egg.
bearoph...@lycos.com
>what makes something a package?
If you don't know what a package is, then maybe you don't need
packages.
In your project is it possible to avoid using packages and just use
modules in the same directory?
Bye,
bearophile
CinnamonDonkey
everyone else also!
From the various posts, Python considers any directory containing the
__init__.py file to be a package. The top level package is the highest
directory (closest to root) with a __init__.py file.
Inter-package communication is not allowed unless the packages
themselves are contained by a parent package.
How does this relate to the site-packages folder? Is it a top level
package for all installed packages?
Let's say I have installed the "Trac" system which uses "Genshi", they
are both packages. They are also both installed at the same level and
I know "Trac" uses "Genshi" to work. \Python25\Lib\site-packages does
not contain a __init__.py file so it is not a package (i.e. not a
parent package to "Trac" and "Genshi") :0.
-=- bearophile -=-
Hi Bearophile,
Thanx for taking the time to post a response but I am afraid I feel
the need to point out that it is exactly this kind of response that I
find un-helpful. It is neither constructive nor educational.
It's a bit like saying "If you don't know what a function is, then
maybe you don't need it. ... have you tried having a single block of
code?"
The point of people coming to these forums is to LEARN and share
knowledge. Perhaps it's not the best solution for me right now but
without trying it I won't know when or how to apply it as a solution.
By the way, my project has about 50 files (modules) in it with a lot
of shared code that could be used across other projects... seems as
good a reason as any to try packages out ;-)
Thanx anyway :)
Maxim Khitrov
<Cinnamo...@googlemail.com> wrote:
> Thanx Max - your explanation sorted it :-), and a big thank you to
> everyone else also!
>
> >From the various posts, Python considers any directory containing the
> __init__.py file to be a package. The top level package is the highest
> directory (closest to root) with a __init__.py file.
>
> Inter-package communication is not allowed unless the packages
> themselves are contained by a parent package.
>
> How does this relate to the site-packages folder? Is it a top level
> package for all installed packages?
>
> Let's say I have installed the "Trac" system which uses "Genshi", they
> are both packages. They are also both installed at the same level and
> I know "Trac" uses "Genshi" to work. \Python25\Lib\site-packages does
> not contain a __init__.py file so it is not a package (i.e. not a
> parent package to "Trac" and "Genshi") :0.
Trac does not use relative imports to access genshi. When relative
imports are not used, python goes through sys.path list to find
modules (with a small exception made when absolute_imports are not
enabled, but that should be default in 2.7). The site-packages
directory is added to sys.path, so when trac executes something like
"from genshi import some_module", python will look in site-packages,
among other directories, for a directory called "genshi" that contains
an __init__.py file.
When you execute a script, the directory of that script is
automatically added to sys.path, so with your example you could have
used absolute imports between subpack1 and subpack2, with the \App
directory performing the same function as site-packages (Gabriel's
suggestion). This is for your original version of the code when
main.py was under App.
Once you moved main.py outside of \App, running "import sybpack2"
would no longer work. You can, however, append directories to
sys.path, so by doing the following in main.py you could again allow
non-relative imports between subpack1 and subpack2:
import os
import sys
sys.path.append(os.path.realpath('App'))
- Max
R. David Murray
CinnamonDonkey <Cinnamo...@googlemail.com> wrote:
> On 23 Mar, 18:57, bearophileH...@lycos.com wrote:
> > CinnamonDonkey:
> >
> > >what makes something a package?
> >
> > If you don't know what a package is, then maybe you don't need
> > packages.
> >
> > In your project is it possible to avoid using packages and just use
> > modules in the same directory?
> >
> > Bye,
> > bearophile
>
> Hi Bearophile,
>
> Thanx for taking the time to post a response but I am afraid I feel
> the need to point out that it is exactly this kind of response that I
> find un-helpful. It is neither constructive nor educational.
>
> It's a bit like saying "If you don't know what a function is, then
> maybe you don't need it. ... have you tried having a single block of
> code?"
>
> The point of people coming to these forums is to LEARN and share
> knowledge. Perhaps it's not the best solution for me right now but
> without trying it I won't know when or how to apply it as a solution.
>
> By the way, my project has about 50 files (modules) in it with a lot
> of shared code that could be used across other projects... seems as
> good a reason as any to try packages out ;-)
>
> Thanx anyway :)
I think bearophile could have left out the first sentence, but otherwise
his question is perfectly sensible. If you have a bunch of independent
modules, then you don't need to put them in packages. Your example
only showed one module file in each package...I understand now that was
just for simplicity of the example, but we had no way of knowing that.
We've had newbies come in and think they _need_ to put a module file into
a subpackage even when they'd only have one module file per subdirectory
and they don't really know what a package is...thus bearophile's (perhaps
poorly phrased) question.
Now that you know what packages are and what the restrictions on relative
imports are, and you've told us that you have '50 modules' with a lot
of shared code that 'could be used across other projects', perhaps you
see why relative imports are generally discouraged. If you use only
relative imports, the code _can't_ be shared across multiple projects,
because all that project code would have to be in one monster package,
and not be separate projects at all.
So now you'll know better where it makes Pythonic (as opposed to C++)
sense to use it and where not.
--
R. David Murray http://www.bitdance.com
bearoph...@lycos.com
> It is neither constructive nor educational.
>
> It's a bit like saying "If you don't know what a function is, then
> maybe you don't need it. ... have you tried having a single block of
> code?"
>
> The point of people coming to these forums is to LEARN and share
> knowledge. Perhaps it's not the best solution for me right now but
> without trying it I won't know when or how to apply it as a solution.
>
> By the way, my project has about 50 files (modules) in it with a lot
> of shared code that could be used across other projects... seems as
> good a reason as any to try packages out ;-)
I don't agree. My answer can be wrong for your situation, but I can't
know much about you from the start, and for most people my answer was
the right one.
When a student asks me how to improve his/her usage of metaclasses I
usually answer that metaclasses aren't required to solve that small
problem.
Generally in engineering the simplest solution is the one you have to
try first (and often second and third), and only later, if practical
experience shows the simple solution doesn't work, you try a more
complex solution.
So I have suggested you a simple solution by "default". If later you
see that you have many modules and you really need packages, then it's
easy to ignore my first answer.
Bye,
bearophile
CinnamonDonkey
Bearophile,
My applogies if I have offended you, but:
1. "I can't know much about you from the start" - Is kind of my point.
Perhaps it would be better to avoid jumping to conclusions and pre-
judging someones abilities simply because they are lacking knowledge
in a particular area.
Would it have been better if I had opened my thread with a copy of my
CV? I've got a Degree in Digital Systems Engineering (yes I am also an
Engineer)... I went on to do a Phd in AI and Robotics where I also
developed embedded systems. I bummed out on that after 3 years and
went on to work for a games company where I worked on 6 game titles
across 5 different platforms. I was a Lead Software Engineer for the
last 5 years. Before now moving on again.
2. I am also very much a follower of the K.I.S.S. approach, 9 times
out of 10 the simplest solution is often the best.
As an Engineer though I would also suggest that the best way to learn
is to try solving a problem being sure to constantly assess your
approach and then your final solution. By dismissing a possible avenue
you are dismissing a whole new path of knowledge. Is it not better to
try and fail than never try at all? Surely this is how we gain the
valuable experience we need.
By simply suggesting a "simple default" solution, I may never have
consider the alternatives nor understand why they are or are not
suitable.
3. I get your point about Students, sometimes there is such a thing as
too much knowledge or information overload. Whilst doing a PhD I had
to take labs and teach students... The approach I tried to take was
one off, "You may not need packages, why do you think you need
packages?" or "This is how packages would be used and why they would
be used... do you still think you need packages" or better still, for
a capable student, "This is how packages work, try solving your
problem and then tell me if you think it was a good solution."
Going with R. David Murray, perhaps I also jumped too my own
conclusion too quickly and for that I appologise.
Cheers,
Shaun
Istvan Albert
wrote:
> I'm fairly new to Python so I still have a lot to learn. But I'd like
> to know how to correectly use relative imports.
Relative imports are *fundamentally* broken in python. You will soon
see that code using relative import will break if you attempt to run
the module on its own. Yes, it is mindboggling!
Why is it so you ask? It is one of those issue that would be trivial
to implement correctly (it is relative to the current file location,
duh!!!), would be tremendously useful yet for some reason it is
controversial with those who would need to implement it.
It looks like they think that the expected mode of operation would
greatly confuse the 'rest' of us. So that is the reason you end up
with a broken implementation that is worse than not having it at all.
All I can do is recommend that you avoid relative imports.
The easiest way around the issue is to create a module named
pathfix.py like the one below and import it in all of your modules.
This is the only way to fix this issue in a way that does not come
back and bite you, it is ugly, you are repeating yourself in multiple
places, but it is still better than the alternative.
-------------------
import os, sys
def path_join(*args):
return os.path.abspath(os.path.join(*args))
# adds base directory to import path to allow relative imports
__currdir = os.path.dirname( __file__ )
__basedir = path_join(__currdir, '..' )
if __basedir not in sys.path:
sys.path.append( __basedir )
Gabriel Genellina
<istvan...@gmail.com> escribió:
> On Mar 23, 10:16 am, CinnamonDonkey <CinnamonDon...@googlemail.com>
> wrote:
>
>> I'm fairly new to Python so I still have a lot to learn. But I'd like
>> to know how to correectly use relative imports.
>
> Relative imports are *fundamentally* broken in python. You will soon
> see that code using relative import will break if you attempt to run
> the module on its own. Yes, it is mindboggling!
How import works in general is hard to grasp -- the semantics are rather
complicated, partly because the import system has grown patch over patch
over the years, and because it's mostly underdocumented.
But I would not say that relative imports are "fundamentally broken" --
they're one of the best improvements to the import system!
> Why is it so you ask? It is one of those issue that would be trivial
> to implement correctly (it is relative to the current file location,
> duh!!!), would be tremendously useful yet for some reason it is
> controversial with those who would need to implement it.
A module doesn't *have* to reside in the filesystem - so in the general
case, there is no such thing as "current file location". Packages provide
such hierarchical disposition - and relative imports work with packages
only.
> It looks like they think that the expected mode of operation would
> greatly confuse the 'rest' of us. So that is the reason you end up
> with a broken implementation that is worse than not having it at all.
> All I can do is recommend that you avoid relative imports.
I'd recommend the oposite - use relative (intra-package) imports when
possible. Explicit is better than implicit - and starting with 2.7 -when
"absolute" import semantics will be enabled by default- you'll *have* to
use relative imports inside a package, or fail.
> The easiest way around the issue is to create a module named
> pathfix.py like the one below and import it in all of your modules.
> This is the only way to fix this issue in a way that does not come
> back and bite you, it is ugly, you are repeating yourself in multiple
> places, but it is still better than the alternative.
"import it in all of your modules"?
Did you know, once a module is imported by the first time, subsequent
imports simply return the same module instance? The initialization code is
not executed again.
> import os, sys
>
> def path_join(*args):
> return os.path.abspath(os.path.join(*args))
>
> # adds base directory to import path to allow relative imports
> __currdir = os.path.dirname( __file__ )
> __basedir = path_join(__currdir, '..' )
> if __basedir not in sys.path:
> sys.path.append( __basedir )
I don't understand, how is this supposed to help relative imports?
Bindly inserting directories into sys.path can easily confuse the import
system (and you!)
--
Gabriel Genellina
Gabriel Genellina
<rdmu...@bitdance.com> escribió:
> CinnamonDonkey <Cinnamo...@googlemail.com> wrote:
>> On 23 Mar, 18:57, bearophileH...@lycos.com wrote:
>> > CinnamonDonkey:
>> >
>> > >what makes something a package?
>> >
>> > If you don't know what a package is, then maybe you don't need
>> > packages.
>>
>> the need to point out that it is exactly this kind of response that I
>> find un-helpful. It is neither constructive nor educational.
>
> his question is perfectly sensible. If you have a bunch of independent
> modules, then you don't need to put them in packages. Your example
> only showed one module file in each package...I understand now that was
> just for simplicity of the example, but we had no way of knowing that.
Especially, since people coming from Java:
- think that they *have* to use a package
- think that they *have* to put a single class per file
Neither is true in Python. And from the example alone, one could infer the
OP was making such mistakes. Glad to see it was not the case.
--
Gabriel Genellina
Istvan Albert
wrote:
> Did you know, once a module is imported by the first time
yeah yeah, could we not get sidetracked with details that are not
relevant? what it obviously means is to import it in all of your
modules that need to access to relative paths
> I don't understand, how is this supposed to help relative imports?
That's only because you have not had to deal with the problem that it
solves.
If you need to have a module that can do both:
1. access relative paths (even other packages)
2. be executable on its own (for example a modules may execute its own
doctests when running them directly)
this is the only way to achieve it.
> I'd recommend the oposite - use relative (intra-package) imports when possible.
Does it not bother you that a module that uses relative imports cannot
be run on its own anymore? To me that is irritating because it forces
me to change a habit (running the doctests when the module is
executed) that I consider a very good practice. It is extremely handy
to be writing a module, press a key and the module is executed and I
can see the tests results. The relative import takes away this from
me. Like I said, it is irritating.
> Bindly inserting directories into sys.path can easily confuse the import systemn
confuse the import system? what the heck does that mean? You either
have a path in the sys.path or not. FWIW it is far cleaner than doing
a relative import that does not work correctly.
Istvan
Maxim Khitrov
> Does it not bother you that a module that uses relative imports cannot
> be run on its own anymore?
$ python --help
<snip>
-m mod : run library module as a script (terminates option list)
<snip>
$ python -m some.module.name
Works perfectly fine with relative imports.
- Max
Steve Holden
> Top responses guys! This has all helped increadibly.
>
> Bearophile,
>
> My applogies if I have offended you, but:
>
> 1. "I can't know much about you from the start" - Is kind of my point.
> Perhaps it would be better to avoid jumping to conclusions and pre-
> judging someones abilities simply because they are lacking knowledge
> in a particular area.
>
proved that it's important to establish the correct level of discourse
before proceeding to discuss solutions.
> Would it have been better if I had opened my thread with a copy of my
> CV? I've got a Degree in Digital Systems Engineering (yes I am also an
> Engineer)... I went on to do a Phd in AI and Robotics where I also
> developed embedded systems. I bummed out on that after 3 years and
> went on to work for a games company where I worked on 6 game titles
> across 5 different platforms. I was a Lead Software Engineer for the
> last 5 years. Before now moving on again.
>
No, none of that would have helped ;-)
> 2. I am also very much a follower of the K.I.S.S. approach, 9 times
> out of 10 the simplest solution is often the best.
>
> As an Engineer though I would also suggest that the best way to learn
> is to try solving a problem being sure to constantly assess your
> approach and then your final solution. By dismissing a possible avenue
> you are dismissing a whole new path of knowledge. Is it not better to
> try and fail than never try at all? Surely this is how we gain the
> valuable experience we need.
>
> By simply suggesting a "simple default" solution, I may never have
> consider the alternatives nor understand why they are or are not
> suitable.
>
That's true, but we seem to be converging to rational discussion.
> 3. I get your point about Students, sometimes there is such a thing as
> too much knowledge or information overload. Whilst doing a PhD I had
> to take labs and teach students... The approach I tried to take was
> one off, "You may not need packages, why do you think you need
> packages?" or "This is how packages would be used and why they would
> be used... do you still think you need packages" or better still, for
> a capable student, "This is how packages work, try solving your
> problem and then tell me if you think it was a good solution."
>
>
> Going with R. David Murray, perhaps I also jumped too my own
> conclusion too quickly and for that I appologise.
>
People round here generally have broad shoulders and are slow to take
offense. No need to worry - and welcome to c.l.py!
regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/
Want to know? Come to PyCon - soon! http://us.pycon.org/
Istvan Albert
> Works perfectly fine with relative imports.
This only demonstrates that you are not aware of what the problem
actually is.
Try using relative imports so that it works when you import the module
itself. Now run the module as a program. The same module that worked
fine when you imported it will raise the exception:
ValueError: Attempted relative import in non-package
when running it on its own.
Istvan
Carl Banks
> On Mar 24, 9:35 pm, Maxim Khitrov <mkhit...@gmail.com> wrote:
>
> > Works perfectly fine with relative imports.
>
> This only demonstrates that you are not aware of what the problem
> actually is.
>
> Try using relative imports so that it works when you import the module
> itself. Now run the module as a program. The same module that worked
> fine when you imported it will raise the exception:
PEP 366 addresses this issue.
Not the best solution, one that still involves boilerplate, but it is
much less of a hack than your version, and at least it has the
blessing of the language designers so it won't unceremoniously break
at some point.
Carl Banks
Kay Schluehr
A workaround that is hardly acceptable when we are working with /
debugging 3rd party packages. Python was simpler without relative
imports and occasional ambiguity resolutions by means of absolute
imports. Unfortunately Brett Cannons reconstruction of import
semantics comes a little late for Python 3 and I suppose we have to
live with the current mess.
Gabriel Genellina
<istvan...@gmail.com> escribió:
> On Mar 24, 3:16 pm, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
> wrote:
>
>> Did you know, once a module is imported by the first time
>
> yeah yeah, could we not get sidetracked with details that are not
> relevant? what it obviously means is to import it in all of your
> modules that need to access to relative paths
Uh? You posted a module containing a function and some code that modifies
sys.path. I imagine you are interested in those side effects in sys.path
-- which will only happen the first time it is imported. Isn't it relevant?
>> I don't understand, how is this supposed to help relative imports?
>
> That's only because you have not had to deal with the problem that it
> solves.
But I had to deal with the problem that it *creates*, even before relative
imports existed, and it's a lot worse.
>> Bindly inserting directories into sys.path can easily confuse the
>> import systemn
>
> confuse the import system? what the heck does that mean? You either
> have a path in the sys.path or not. FWIW it is far cleaner than doing
> a relative import that does not work correctly.
Others have posted the way to execute modules inside a package. For the
problem of adding random directories to sys.path, see:
http://groups.google.com/group/comp.lang.python/browse_thread/thread/ed47d8e31ca3d411/e22039a2cf166f42?#e22039a2cf166f42
--
Gabriel Genellina
Carl Banks
> On 25 Mrz., 05:56, Carl Banks <pavlovevide...@gmail.com> wrote:
>
>
>
>
>
> > On Mar 24, 8:32 pm, Istvan Albert <istvan.alb...@gmail.com> wrote:
>
> > > On Mar 24, 9:35 pm, Maxim Khitrov <mkhit...@gmail.com> wrote:
>
> > > > Works perfectly fine with relative imports.
>
> > > This only demonstrates that you are not aware of what the problem
> > > actually is.
>
> > > Try using relative imports so that it works when you import the module
> > > itself. Now run the module as a program. The same module that worked
> > > fine when you imported it will raise the exception:
>
> > PEP 366 addresses this issue.
>
> > Not the best solution, one that still involves boilerplate, but it is
> > much less of a hack than your version, and at least it has the
> > blessing of the language designers so it won't unceremoniously break
> > at some point.
>
> debugging 3rd party packages. Python was simpler without relative
> imports and occasional ambiguity resolutions by means of absolute
> imports. Unfortunately Brett Cannons reconstruction of import
> semantics comes a little late for Python 3 and I suppose we have to
> live with the current mess
Out of curiosity, is there anything--aside from explicit relative
imports--worked before but doesn't now?
The issue Istvan is talking about exists even with the implicit
imports.
Carl Banks
Aahz
Gabriel Genellina <gags...@yahoo.com.ar> wrote:
>
>I'd recommend the oposite - use relative (intra-package) imports when
>possible. Explicit is better than implicit - and starting with 2.7 -when
>"absolute" import semantics will be enabled by default- you'll *have* to
>use relative imports inside a package, or fail.
Really? I thought you would still be able to use absolute imports; you
just won't be able to use implied relative imports instead of explicit
relative imports.
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it." --Brian W. Kernighan
Gabriel Genellina
> In article <mailman.2591.1237922...@python.org>,
> Gabriel Genellina <gags...@yahoo.com.ar> wrote:
>>
>> I'd recommend the oposite - use relative (intra-package) imports when
>> possible. Explicit is better than implicit - and starting with 2.7 -when
>> "absolute" import semantics will be enabled by default- you'll *have* to
>> use relative imports inside a package, or fail.
>
> Really? I thought you would still be able to use absolute imports; you
> just won't be able to use implied relative imports instead of explicit
> relative imports.
You're right, I put it wrongly. To make things clear, inside a package
"foo" accessible thru sys.path, containing a.py and b.py:
site-packages/
foo/
a.py
b.py
__init__.py
Currently, the "a" module can import "b" this way:
from foo import b
import foo.b
from . import b
import b
When implicit relative imports are disabled ("from __future__ import
absolute_import", or after 2.7 supposedly) the last one won't find b.py
anymore.
(I hope I put it right this time).
--
Gabriel Genellina
Kay Schluehr
> En Mon, 30 Mar 2009 21:15:59 -0300, Aahz <a...@pythoncraft.com> escribió:
>
> > In article <mailman.2591.1237922208.11746.python-l...@python.org>,
So it even breaks more code which is great ;)
Do you know of any near or far past attempts to re-design the import
system from the ground up? I do not mean a rather faithful and
accessible reconstruction such as Brett Cannons work but a radical re-
design which starts with a domain model and does not end with Loaders,
Importers and Finders which are actually services that pretend to be
objects.
Kay
s4g
I was looking for a nice idiom for interpackage imports as I found
this thread.
Here come a couple of solutions I came up with. Any discussion is
welcome.
I assume the same file structure
\ App
| main.py
+--\subpack1
| | __init__.py
| | module1.py
|
+--\subpack2
| | __init__.py
| | module2.py
When you run main.py all imports relative to \App work fine, so the
only problem is running a module from within a subpackage as a script.
I therefore assume we want to run module1.py as a script, which wants
to import module2.
I hope the following solutions are self-evident
================= solution 1
--> in module1.py
import sys, os
if __name__ == '__main__':
sys.path.append(os.path.normpath(__file__+'/../..'))
import subpack2.module2
================= solution 2
--> in subpack1/__init__.py
import sys, os
_top_package_level = 1 # with current package being level 0
_top_package = os.path.normpath(__file__ + '/..'*(_top_package_level
+1))
if _top_package not in sys.path:
sys.path.append(_top_package)
--> in module1 or any module in the package, which requires import
relative to the package top
import __init__
import subpack2.module2
================= solution 3
--> in package_import.py, somewhere on the PYTHONPATH ( perhaps in
standard lib ;)
def set_top_package(module, level):
_top_package = os.path.normpath(module + '/..'*(level+1))
if _top_package not in sys.path:
sys.path.append(_top_package)
class absolute_import(object):
def __init__(self, module, level):
self.level = level
self.module = module
def __enter__(self):
sys.path.insert( 0,
os.path.normpath(self.module + '/..'*(self.level+1))
)
def __exit__(self, exc_type, exc_val, exc_tb):
del sys.path[0]
--> in module1
import package_import
package_import.set_top_package(__file__, 1)
import subpack2.module2
--> or in module1
import package_import
with package_import.absolute_import(__file__, 1):
import subpack2.module2
...
Kay Schluehr
This and similar solutions ( see Istvan Alberts ) point me to a
fundamental problem of the current import architecture. Suppose you
really want to run a module as a script without a prior import from a
module path:
...A\B\C> python my_module.py
then the current working directory C is added to sys.path which means
that the module finder searches in C but C isn't a known package.
There is no C package in sys.modules even if the C directory is
"declared" as a package by placing an __init__.py file in it. Same
goes of course with B and A. Although the ceremony has been performed
basically correct the interpreter god is not pacified and doesn't
respond. But why not? Because it looks up for *living* imported
packages in the module cache ( in sys.modules ).
I don't think there is any particular design idea behind it. The
module cache is just a simple flat dictionary; a no-brainer to
implement and efficient for look ups. But it counteracts a domain
model. All you are left with is those Finders, Loaders and Importers
in Brett Cannons importlib. Everything remains deeply mysterious and I
don't wonder that it took long for him to work this out.
Terry Reedy
> On 31 Mrz., 18:48, s4g <rafals...@gmail.com> wrote:
> This and similar solutions ( see Istvan Alberts ) point me to a
> fundamental problem of the current import architecture. Suppose you
> really want to run a module as a script without a prior import from a
> module path:
>
> ...A\B\C> python my_module.py
>
> then the current working directory C is added to sys.path which means
> that the module finder searches in C but C isn't a known package.
> There is no C package in sys.modules even if the C directory is
> "declared" as a package by placing an __init__.py file in it. Same
> goes of course with B and A.
Nothing is added to sys.modules, except the __main__ module, unless
imported (which so are on startup).
> Although the ceremony has been performed
> basically correct the interpreter god is not pacified and doesn't
> respond.
But the import 'ceremony' has not been performed.
> But why not? Because it looks up for *living* imported
> packages in the module cache ( in sys.modules ).
>
> I don't think there is any particular design idea behind it. The
> module cache is just a simple flat dictionary; a no-brainer to
> implement and efficient for look ups.
This all dates to the time before packages and imports from zip files
and such.
> But it counteracts a domain model.
What is that?
> All you are left with is those Finders, Loaders and Importers
> in Brett Cannons importlib. Everything remains deeply mysterious and I
> don't wonder that it took long for him to work this out.
And your proposal is?
tjr
Kay Schluehr
> Nothing is added to sys.modules, except the __main__ module, unless
> imported (which so are on startup).
Yes. The startup process is opaque but at least user defined modules
are not accidentally imported.
>
> > Although the ceremony has been performed
> > basically correct the interpreter god is not pacified and doesn't
> > respond.
>
> But the import 'ceremony' has not been performed.
There is no import ceremony. Imports are just stated in the source.
There is a package ceremony for whatever reasons.
> > But why not? Because it looks up for *living* imported
> > packages in the module cache ( in sys.modules ).
>
> > I don't think there is any particular design idea behind it. The
> > module cache is just a simple flat dictionary; a no-brainer to
> > implement and efficient for look ups.
>
> This all dates to the time before packages and imports from zip files
> and such.
>
> > But it counteracts a domain model.
>
> What is that?
Object oriented programming.
>
> > All you are left with is those Finders, Loaders and Importers
> > in Brett Cannons importlib. Everything remains deeply mysterious and I
> > don't wonder that it took long for him to work this out.
>
> And your proposal is?
I have still more questions than answers.
Carl Banks
> > And your proposal is?
>
> I have still more questions than answers.
That's obvious.
Perhaps you should also refrain from making sweeping negative
judgments about a system you have more questions than answers about.
(Feel free to make sweeping negative judgments once you have the
answers, though.)
Carl Banks
Terry Reedy
> On 31 Mrz., 20:50, Terry Reedy <tjre...@udel.edu> wrote:
>>> Although the ceremony has been performed
>>> basically correct the interpreter god is not pacified and doesn't
>>> respond.
>> But the import 'ceremony' has not been performed.
>
> There is no import ceremony. Imports are just stated in the source.
> There is a package ceremony for whatever reasons.
Sorry, I have no idea what 'package ceremony' means.
>
>>> But why not? Because it looks up for *living* imported
>>> packages in the module cache ( in sys.modules ).
>>> I don't think there is any particular design idea behind it. The
>>> module cache is just a simple flat dictionary; a no-brainer to
>>> implement and efficient for look ups.
>> This all dates to the time before packages and imports from zip files
>> and such.
>>
>> > But it counteracts a domain model.
>>
>> What is that?
>
> Object oriented programming.
Domain model == oop? New one for me.
>>> All you are left with is those Finders, Loaders and Importers
>>> in Brett Cannons importlib. Everything remains deeply mysterious and I
>>> don't wonder that it took long for him to work this out.
>> And your proposal is?
>
> I have still more questions than answers.
OK. I will just note that import statements are syntactic sugar for
__import__ calls and name bindings. One could try out other import
schemes, just without the sugar.
tjr
Kay Schluehr
Diagnosing a problem means having a detailed cure? Wow, that's
critical thinking I guess.
O.K. I have got some ideas for a new import system and I'm going to
blog about them within the next days. If I have some results I'll
leave a notice in this thread.
Carl Banks
> On 1 Apr., 00:38, Carl Banks <pavlovevide...@gmail.com> wrote:
>
> > On Mar 31, 12:08 pm, Kay Schluehr <kay.schlu...@gmx.net> wrote:
>
> > > > And your proposal is?
>
> > > I have still more questions than answers.
>
> > That's obvious.
>
> > Perhaps you should also refrain from making sweeping negative
> > judgments about a system you have more questions than answers about.
>
> > (Feel free to make sweeping negative judgments once you have the
> > answers, though.)
>
> > Carl Banks
>
> Diagnosing a problem means having a detailed cure? Wow, that's
> critical thinking I guess.
A diagnosis commesurate with the understanding you've displayed in
this thread so far would be something like "Python's import system is
too complicated", not "Python's import system is fundamentally
broken".
The latter is not something someone who has more questions than
answers has any grounds to claim.
But, as I said, once you have learned the answers to your questions
and you claim that, it's all good.
Carl Banks