Generated code that is exec-ed (to simulate import) cannot import os.path??

[Irmen de Jong]

Hello,
I don't understand why the following doesn't work.
What I want to do is dynamically import some generated
Python code and I'm doing this using compile and exec'ing
it in the dict of a new empty module object.
That works okay, but as soon as the generated code
tries do perform certain imports, it fails!
Certain other imports succeed. Consider this example code:

-----SNIP----
import imp

source="""
print 'importing random'
import random
print 'importing os'
import os
print 'bye!'

def getName():
return 'Hello there'
"""

newmod = imp.new_module('generated.testmodule')
code=compile(source,'<generated code>','exec')
print 'got code...'
exec code in newmod.__dict__
print 'done, newmod.getname(): ',newmod.getName()
-----/SNIP----

When run, it produces the following output:

[E:\test]python dynimport.py
got code...
importing random
importing os
Traceback (most recent call last):
File "dynimport.py", line 17, in ?
exec code in newmod.__dict__
File "<generated code>", line 5, in ?
File "C:\Python23\lib\os.py", line 131, in ?
from os.path import curdir, pardir, sep, pathsep, defpath, extsep, altsep
ImportError: No module named path

What's going on? Why can't it find os.path?

(Python 2.3.2, tested on linux and windows)

Confused,

Irmen.

[Michael Hudson]

Irmen de Jong <irmen@-NOSPAM-REMOVETHIS-xs4all.nl> writes:

> Hello,
> I don't understand why the following doesn't work.
> What I want to do is dynamically import some generated
> Python code and I'm doing this using compile and exec'ing
> it in the dict of a new empty module object.
> That works okay, but as soon as the generated code
> tries do perform certain imports, it fails!
> Certain other imports succeed. Consider this example code:

[snip]

> What's going on? Why can't it find os.path?

Dunno. Two things to try:

1) using new.module instead of the imp function
2) run python -v

Cheers,
mwh

--
CLiki pages can be edited by anybody at any time. Imagine the most
fearsomely comprehensive legal disclaimer you have ever seen, and
double it -- http://ww.telent.net/cliki/index

[Peter Otten]

Irmen de Jong wrote:

> Hello,
> I don't understand why the following doesn't work.
> What I want to do is dynamically import some generated
> Python code and I'm doing this using compile and exec'ing
> it in the dict of a new empty module object.
> That works okay, but as soon as the generated code
> tries do perform certain imports, it fails!
> Certain other imports succeed. Consider this example code:

[...]

> What's going on? Why can't it find os.path?

My trial and error findings (Python 2.3.2 on Linux):

Python gets confused by the module name "generated.testmodule", when
a "generated" package does not exist; it seems to look for
"generated.posixpath" when it should for "os.posixpath" during the import
of the os module.

Two fixes are possible:

(1) Change

newmod = imp.new_module('generated.testmodule')

to, e. g.

newmod = imp.new_module('generated_testmodule')

(2) Create a dummy "generated" module and insert it into sys.modules:

import imp, sys

source="""
print 'importing random'
import random
print 'importing os'
import os
print 'bye!'

def getName():
return 'Hello there'
"""

sys.modules["generated"] = imp.new_module("generated")

newmod = imp.new_module('generated.testmodule')
code=compile(source,'<generated code>','exec')
print 'got code...'
exec code in newmod.__dict__
print 'done, newmod.getname(): ',newmod.getName()

While this works, a helpful error message would be nice when an intermediate
package is not found. Unfortunately I was not able to track down the actual
point of failure yet.

Peter

[Irmen de Jong]

Michael Hudson wrote:
>>What's going on? Why can't it find os.path?
>
>
> Dunno. Two things to try:
>
> 1) using new.module instead of the imp function

Tried it, no difference. I think they are equivalent.

> 2) run python -v

A relevant piece of the trace is found below, using
the same code that I posted before. I've done this already
but couldn't find anything that helps me understand why
os.path cannot be imported.

--Irmen

[.... partial 'python -v dynimport.py' output....]
got code...
importing random
# /usr/local/lib/python2.3/random.pyc matches
/usr/local/lib/python2.3/random.py
import generated.random # precompiled from
/usr/local/lib/python2.3/random.pyc
dlopen("/usr/local/lib/python2.3/lib-dynload/math.so", 2);
import generated.math # dynamically loaded from
/usr/local/lib/python2.3/lib-dynload/math.so
dlopen("/usr/local/lib/python2.3/lib-dynload/_random.so", 2);
import generated._random # dynamically loaded from
/usr/local/lib/python2.3/lib-dynload/_random.so
dlopen("/usr/local/lib/python2.3/lib-dynload/time.so", 2);
import generated.time # dynamically loaded from
/usr/local/lib/python2.3/lib-dynload/time.so
importing os
# /usr/local/lib/python2.3/os.pyc matches /usr/local/lib/python2.3/os.py
import generated.os # precompiled from /usr/local/lib/python2.3/os.pyc
import sys # previously loaded (sys)
import posix # previously loaded (posix)
import posix # previously loaded (posix)
# /usr/local/lib/python2.3/posixpath.pyc matches
/usr/local/lib/python2.3/posixpath.py
import generated.posixpath # precompiled from
/usr/local/lib/python2.3/posixpath.pyc
import sys # previously loaded (sys)
# /usr/local/lib/python2.3/stat.pyc matches /usr/local/lib/python2.3/stat.py
import generated.stat # precompiled from /usr/local/lib/python2.3/stat.pyc
import posix # previously loaded (posix)

Traceback (most recent call last):

File "dynimport.py", line 18, in ?

exec code in newmod.__dict__
File "<generated code>", line 5, in ?

File "/usr/local/lib/python2.3/os.py", line 131, in ?

from os.path import curdir, pardir, sep, pathsep, defpath, extsep,
altsep
ImportError: No module named path

# clear __builtin__._
# clear sys.path
# clear sys.argv

[.... end snipped...]

[Irmen de Jong]

Peter Otten wrote:

> Python gets confused by the module name "generated.testmodule", when
> a "generated" package does not exist; it seems to look for
> "generated.posixpath" when it should for "os.posixpath" during the import
> of the os module.

Terrific, that was the problem! Thanks a lot Peter!
After changing the module name to 'generated_testmodule' as you
suggested, it works :-)
Perhaps a more structural improvement is possible but I'm happy
with just changing my generated module name to a name that is
not in a (non-existing) package.

> While this works, a helpful error message would be nice when an intermediate
> package is not found.

Indeed...

--Irmen de Jong.