changing sys.path

[Andrea Crotti]

So suppose I want to modify the sys.path on the fly before running some code
which imports from one of the modules added.

at run time I do
sys.path.extend(paths_to_add)

but it still doesn't work and I get an import error.

If I take these paths and add them to site-packages/my_paths.pth
everything works, but at run-time the paths which I actually see before
importing are exactly the same.

So there is something I guess that depends on the order, but what can I
reset/reload to make these paths available (I thought I didn't need
anything in theory)?

[jmfauth]

>>> import mod
Traceback (most recent call last):
File "<eta last command>", line 1, in <module>
ImportError: No module named mod
>>> sys.path.append(r'd:\\jm\\junk')
>>> import mod
>>> mod
<module 'mod' from 'd:\\jm\\junk\mod.py'>
>>> mod.hello()
fct hello in mod.py


sys.path? Probably, the most genious Python idea.

jmf

[Andrea Crotti]

On 02/01/2012 05:13 PM, Eric Snow wrote:

> On Wed, Feb 1, 2012 at 9:15 AM, Andrea Crotti<andrea....@gmail.com> wrote:
>> So suppose I want to modify the sys.path on the fly before running some code
>> which imports from one of the modules added.
>>
>> at run time I do
>> sys.path.extend(paths_to_add)
>>
>> but it still doesn't work and I get an import error.

> Make sure you are adding to sys.path the directories that your
> packages/modules are in, and not the actual package directories.
> During import Python looks for modules/packages _in_ each of the
> directories on sys.path, but not _at_ those directories.

Yes sure I do this..

>
>> If I take these paths and add them to site-packages/my_paths.pth
>> everything works, but at run-time the paths which I actually see before
>> importing are exactly the same.

> You mean sys.path looks exactly the same in the two cases?
>
> -eric

Yes they are exactly the same, because in that file I just write exactly
the same list,
but when modifying it at run-time it doesn't work, while if at the
application start
there is this file everything works correctly...

That's what really puzzles me.. What could that be then?

[Rick Johnson]

On Feb 1, 10:15 am, Andrea Crotti <andrea.crott...@gmail.com> wrote:
> So suppose I want to modify the sys.path on the fly before running some code
> which imports from one of the modules added.
>
> at run time I do
> sys.path.extend(paths_to_add)
>
> but it still doesn't work and I get an import error.
>
> If I take these paths and add them to site-packages/my_paths.pth
> everything works, but at run-time the paths which I actually see before
> importing are exactly the same.

1. Is paths_to_add a nested list?
2. Have you tried inspecting the contents of sys.path AFTER calling
extend method?

Consider:
py> sys.path.__len__()
14
py> sys.path.extend([[1,2,3]])
py> sys.path.__len__()
15

[Steven D'Aprano]

On Wed, 01 Feb 2012 17:47:22 +0000, Andrea Crotti wrote:

> Yes they are exactly the same, because in that file I just write exactly
> the same list,
> but when modifying it at run-time it doesn't work, while if at the
> application start
> there is this file everything works correctly...
>
> That's what really puzzles me.. What could that be then?

Are you using IDLE or WingIDE or some other IDE which may not be
honouring sys.path? If so, that's a BAD bug in the IDE.

Are you changing the working directory manually, by calling os.chdir? If
so, that could be interfering with the import somehow. It shouldn't, but
you never know...

Are you adding absolute paths or relative paths?

You say that you get an ImportError, but that covers a lot of things
going wrong. Here's a story. Could it be correct? I can't tell because
you haven't posted the traceback.

When you set site-packages/my_paths.pth you get a sys path that looks
like ['a', 'b', 'fe', 'fi', 'fo', 'fum']. You then call "import spam"
which locates b/spam.py and everything works.

But when you call sys.path.extend(['a', 'b']) you get a path that looks
like ['fe', 'fi', 'fo', 'fum', 'a', 'b']. Calling "import spam" locates
some left over junk file, fi/spam.py or fi/spam.pyc, which doesn't
import, and you get an ImportError.


--
Steven

[Andrea Crotti]

On 02/02/2012 12:51 AM, Steven D'Aprano wrote:
> On Wed, 01 Feb 2012 17:47:22 +0000, Andrea Crotti wrote:
>
>> Yes they are exactly the same, because in that file I just write exactly
>> the same list,
>> but when modifying it at run-time it doesn't work, while if at the
>> application start
>> there is this file everything works correctly...
>>
>> That's what really puzzles me.. What could that be then?
>
> Are you using IDLE or WingIDE or some other IDE which may not be
> honouring sys.path? If so, that's a BAD bug in the IDE.
> Are you changing the working directory manually, by calling os.chdir? If
> so, that could be interfering with the import somehow. It shouldn't, but
> you never know...
>
> Are you adding absolute paths or relative paths?

No, no and absolute paths..

>
> You say that you get an ImportError, but that covers a lot of things
> going wrong. Here's a story. Could it be correct? I can't tell because
> you haven't posted the traceback.
>
> When you set site-packages/my_paths.pth you get a sys path that looks
> like ['a', 'b', 'fe', 'fi', 'fo', 'fum']. You then call "import spam"
> which locates b/spam.py and everything works.
>
> But when you call sys.path.extend(['a', 'b']) you get a path that looks
> like ['fe', 'fi', 'fo', 'fum', 'a', 'b']. Calling "import spam" locates
> some left over junk file, fi/spam.py or fi/spam.pyc, which doesn't
> import, and you get an ImportError.
>
>

And no the problem is not that I already checked inspecting at run-time..
This is the traceback and it might be related to the fact that it runs
from the
.exe wrapper generated by setuptools:

Traceback (most recent call last):

File "c:\python25\scripts\dev_main-script.py", line 8, in <module>
load_entry_point('psi.devsonly==0.1', 'console_scripts', 'dev_main')()
File "h:\git_projs\psi\psi.devsonly\psi\devsonly\bin\dev_main.py",
line 152, in main
Develer(ns).full_run()
File "h:\git_projs\psi\psi.devsonly\psi\devsonly\bin\dev_main.py",
line 86, in full_run
run(project_name, test_only=self.ns.test_only)
File "h:\git_projs\psi\psi.devsonly\psi\devsonly\environment.py",
line 277, in run
from psi.devsonly.run import Runner
File "h:\git_projs\psi\psi.devsonly\psi\devsonly\run.py", line 7, in
<module>
from psi.workbench.api import Workbench, set_new_dev_main
ImportError: No module named workbench.api


Another thing which might matter is that I'm launching Envisage
applications, which
heavily rely on the use of entry points, so I guess that if something is
not in the path
the entry point is not loaded automatically (but it can be forced I
guess somehow).

I solved in another way now, since I also need to keep a dev_main.pth in
site-packages
to make Eclipse happy, just respawning the same process on ImportError works
already perfectly..

[jmfauth]

There is something strange here. I can not figure
out how correct code will fail with the sys.path.
It seems to me, the lib you are using is somehow not
able to recognize its own structure ("his own sys.path").

Idea. Are you sure you are modifying the sys.path at
the right place, understand at the right time
when Python processes?

I'm using this sys.path tweaking at run time very often;
eg to test or to run different versions of the same lib
residing in different dirs, and this, in *any* dir and
independently of *any* .pth file.

jmf

[Tim Roberts]

Andrea Crotti <andrea....@gmail.com> wrote:
>
>So suppose I want to modify the sys.path on the fly before running some code
>which imports from one of the modules added.
>
>at run time I do
>sys.path.extend(paths_to_add)
>
>but it still doesn't work and I get an import error.

Are you actually adding multiple paths? One possible cause for error would
be this:
sys.path.extend( '/usr/local/lib' )

That succeeds, but it doesn't do what you meant. It adds "/" as a path,
then "u", then "s", then "r", and so on.
--
Tim Roberts, ti...@probo.com
Providenza & Boekelheide, Inc.

[John Nagle]

On 2/1/2012 8:15 AM, Andrea Crotti wrote:
> So suppose I want to modify the sys.path on the fly before running some
> code
> which imports from one of the modules added.
>
> at run time I do
> sys.path.extend(paths_to_add)
>
> but it still doesn't work and I get an import error.

Do

import sys

first.

John Nagle

[Andrea Crotti]

I think I finally located the issue with the sys.path extension.

The problem is that I have many namespace directories, for example

lib:
- sub1
- sub2

lib:
- sub3
- sub4

But to have everything working I had lib.sub3 in easy-install.pth.
Now if I try to add something else to the path it doesn't take care of
the namespace
declaration
(every __init__.py in the packages contains:
__import__('pkg_resources').declare_namespace(__name__))
and just doesn't find the other submodules..

If I try to add manually lib.sub1, lib.sub2 changing the sys.path the
imports will only work for the first one.

Strangely if I just create a dev_main.pth in site-packages containing
the same paths, everything works perfectly.

Any suggestions now that the problem is more clear?
Thanks,
Andrea

[Dave Angel]

The only code I saw in this thread was:

sys.path.extend(paths_to_add)

Can you add a print of paths_to_add, and of sys.path after you execute it?

If there's only one path, are you putting it in a list anyway? If not
then it won't do what you expect.



--

DaveA

[Andrea Crotti]

Ok now it's getting really confusing, I tried a small example to see
what is the real behaviour,
so I created some package namespaces (where the __init__.py declare the
namespace package).

/home/andrea/test_ns:
total used in directory 12 available 5655372
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:46 a.b
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:46 a.c
-rw-r--r-- 1 andrea andrea 125 Feb 10 14:46 test.py

/home/andrea/test_ns/a.b:
total 8
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:47 a
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.b/a:
total 8
drwxr-xr-x 2 andrea andrea 4096 Feb 10 14:47 b
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.b/a/b:
total 12
-rw-r--r-- 1 andrea andrea 25 Feb 10 14:36 api.py
-rw-r--r-- 1 andrea andrea 153 Feb 10 14:37 api.pyc
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.c:
total 8
drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:47 a
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.c/a:
total 8
drwxr-xr-x 2 andrea andrea 4096 Feb 10 14:47 c
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py

/home/andrea/test_ns/a.c/a/c:
total 12
-rw-r--r-- 1 andrea andrea 20 Feb 10 14:36 api.py
-rw-r--r-- 1 andrea andrea 148 Feb 10 14:38 api.pyc
-rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py


So this test.py works perfectly:
import sys
sys.path.insert(0, 'a.c')
sys.path.insert(0, 'a.b')

from a.b import api as api_ab

from a.c import api as api_ac

While just mixing the order:
import sys
sys.path.insert(0, 'a.b')

from a.b import api as api_ab

sys.path.insert(0, 'a.c')
from a.c import api as api_ac

Doesn't work anymore

[andrea@precision test_ns]$ python2 test.py

Traceback (most recent call last):

File "test.py", line 7, in <module>
from a.c import api as api_ac
ImportError: No module named c



Am I missing something/doing something stupid?

[Dave Angel]

Yes, you've got periods in your directory names. A period means
something special within python, and specifically within the import.

When you say from a.c import api

You're telling it: from package a get module c, and from there
impoort the symbol api

But package a has no module c, so it complains.


In an earlier message you asserted you were using all absolute paths in
your additions to sys.path. Here you're inserting relative ones. How come?

--

DaveA

[Andrea Crotti]

On 02/10/2012 03:06 PM, Dave Angel wrote:
>
> Yes, you've got periods in your directory names. A period means
> something special within python, and specifically within the import.
>
> When you say from a.c import api
>
> You're telling it: from package a get module c, and from there
> impoort the symbol api
>
> But package a has no module c, so it complains.
>
>
> In an earlier message you asserted you were using all absolute paths
> in your additions to sys.path. Here you're inserting relative ones.
> How come?
>

Well yes I have periods, but that's also the real-world situation.
We have many directories that are contributing to the same namespace in
the same superdirectory
which should not interfere, so it was decided to give this naming.
It would be quite hard to change I guess and I have to prove that this
is problem.

I renamed everything and this
import sys
from os import path
sys.path.insert(0, path.abspath('ab'))

from a.b import api as api_ab

sys.path.insert(0, path.abspath('ac'))

from a.c import api as api_ac

still fails, so the period in the name is not the problem.

Also absolute or relative paths in this small example doesn't make any
difference.

Adding all the paths in one go works perfectly fine anyway, so I
probably have to make sure
I add them *all* before anything is imported.

If there are better solutions I would like to hear them :)

[Peter Otten]

> from a.b import api as api_ab
>

> from a.c import api as api_ac
>

> While just mixing the order:
> import sys
> sys.path.insert(0, 'a.b')
>

> from a.b import api as api_ab
>

> sys.path.insert(0, 'a.c')

> from a.c import api as api_ac
>

> Doesn't work anymore
>
> [andrea@precision test_ns]$ python2 test.py
> Traceback (most recent call last):
> File "test.py", line 7, in <module>

> from a.c import api as api_ac

> ImportError: No module named c
>
>
>
> Am I missing something/doing something stupid?

The package a will be either a.c/a/ or a.b/a/ depending on whether a.c/ or
a.b/ appears first in sys.path.

If it's a.c/a, that does not contain a c submodule or subpackage.

[Andrea Crotti]

On 02/10/2012 03:27 PM, Peter Otten wrote:
> The package a will be either a.c/a/ or a.b/a/ depending on whether
> a.c/ or a.b/ appears first in sys.path. If it's a.c/a, that does not
> contain a c submodule or subpackage.

I would agree if I didn't have this declaration
__import__('pkg_resources').declare_namespace(__name__)
in each subdirectory.

And how do you explain the fact that changing the order everything works?
Namespace packages are supposed to work exactly like this, if it doesn't
resolve the
"c" instead of raising an Exception it goes forward in the sys.path and
try again, which
is what actually happens when I do this

sys.path.append(path.abspath('ab'))
sys.path.append(path.abspath('ac'))

from a.b import api as api_ab
from a.c import api as api_ac

Maybe this:
Definition: pkgutil.extend_path(path, name)
Docstring:
Extend a package's path.

Intended use is to place the following code in a package's __init__.py:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)


might come handy, from what I'm gathering is the only way to have a more
dynamic path
manipulation with namespace packages..

[Peter Otten]

Peter Otten wrote:

> If it's a.c/a, that does not contain a c submodule or subpackage.

Sorry, I meant a.b/a

[Peter Otten]

Andrea Crotti wrote:

> On 02/10/2012 03:27 PM, Peter Otten wrote:
>> The package a will be either a.c/a/ or a.b/a/ depending on whether

>> a.c/ or a.b/ appears first in sys.path. If it's a.c/a, that does not

>> contain a c submodule or subpackage.
>
>

> I would agree if I didn't have this declaration
> __import__('pkg_resources').declare_namespace(__name__)
> in each subdirectory.

Sorry, you didn't mention that in the post I responded to and I didn't
follow the thread closely.

I found a description for declare_namespace() at
http://peak.telecommunity.com/DevCenter/PkgResources

but the text explaining the function is completely unintelligible to me, so
I cannot contribute anything helpful here :(

[Andrea Crotti]

On 02/10/2012 04:00 PM, Peter Otten wrote:
> Sorry, you didn't mention that in the post I responded to and I didn't
> follow the thread closely.
>
> I found a description for declare_namespace() at
> http://peak.telecommunity.com/DevCenter/PkgResources
>
> but the text explaining the function is completely unintelligible to me, so
> I cannot contribute anything helpful here :(
>

Well in the end I submitted a bug report
http://bugs.python.org/issue13991
I'm not sure it's really a bug and maybe I'm just doing something wrong,
but to me the behavior is at least unexpected..