Strange import problem: is sys.path not used in web2py?

[Daniel Gonzalez]

Hi,

I have a problem with an import in my model:

Traceback (most recent call last):
  File "/installdir/web2py/gluon/restricted.py", line 212, in restricted
    exec ccode in environment
  File "/installdir/web2py/applications/wavilon_portal/models/db.py", line 10, in <module>
    import nltk.data
  File "/installdir/web2py/gluon/custom_import.py", line 77, in custom_importer
    raise ImportError, 'Cannot import module %s' % str(e)
ImportError: Cannot import module 'nltk'

I thought this would be a problem with sys.path, so I am displaying it in the top of the controller (<sys_path_in_model>)

Then I am doing, manually in the python interpreter:

import sys
sys.path=<sys_path_in_model>
import nltk.data

But this is giving no error!

What is happening here? Is the import mechanism in web2py using something different than sys,path?

Thanks,

Daniel

[Alan Etkin]

I have a problem with an import in my model:

What version are you running?, in what OS?. As a temporary workaround and If it's a path error you can try fixing sys.path so nltk is found.

[Daniel Gonzalez]

Version 2.3.2 (2012-12-17 08:59:58) stable, commit 9557c46, in Ubuntu 12.04.1 LTS. I am running from a virtualenv with python 2.7.2.

I have no idea how to force nltk to be found: the sys.path is correct, as I have verified by setting manually in the interpreter and running the import statement. That is why I mean this is a "strange" error: everything seems to be set up correctly, but running from web2py I get an import error, and doing the same thing directly in the interpreter does not.

I am taking a look at the custom_importer: I do no know why this import is falling to line 77 (if I understand correctly, that part is doing the custom web2py modules/models/controllers importing). It should go to line 96:

    return NATIVE_IMPORTER(name, globals, locals, fromlist, level)

since nltk.data is a system import. It has nothing to do with web2py.

[Daniel Gonzalez]

The problem is this code in custom_import.py

    # if not relative and not from applications:
    if hasattr(current, 'request') \
            and level <= 0 \
            and not name.split('.')[0] in INVALID_MODULES \
            and isinstance(globals, dict):

"import nltk.data" goes through, but it shouldn't. Why could that be? (I do not understand the code very well)

[Alan Etkin]

The problem is this code in custom_import.py

With a very similar configuration but the trunk version I could not reproduce the problem (altough the nltk library has taken about 20 seconds to import in the model)

My system config:
Python 2.7.3 Ubuntu 12.04 Rocket server

This has something to do with virtualenv surely. BTW, there's a recent issue posted about module headers that might be related.

[Daniel Gonzalez]

I have logged the import activity, modifying custom_import like this:

    FORMAT="%s -> %-40s %-10s %-10s %-10s %-10s"

    # if not relative and not from applications:
    if hasattr(current, 'request') \
            and level <= 0 \
            and not name.split('.')[0] in INVALID_MODULES \
            and isinstance(globals, dict):

        print FORMAT % ("custom", name, hasattr(current, 'request'), level, name.split('.')[0] in INVALID_MODULES, isinstance(globals, dict))
    ...
        finally:
            if import_tb:
                import_tb = None
    print FORMAT % ("native", name, hasattr(current, 'request'), level, name.split('.')[0] in INVALID_MODULES, isinstance(globals, dict))

    return NATIVE_IMPORTER(name, globals, locals, fromlist, level)

Here you see some of the data:

native -> random                                   True       -1         True       True      
custom -> nltk.data                                True       -1         False      True      
native -> __future__                               True       -1         True       True      
native -> os                                       True       -1         True       True      
custom -> internals                                True       -1         False      True      
native -> subprocess                               True       -1         True       True      
native -> os                                       True       -1         True       True      
native -> os.path                                  True       -1         True       True      
native -> re                                       True       -1         True       True      
native -> warnings                                 True       -1         True       True      
native -> textwrap                                 True       -1         True       True      
native -> types                                    True       -1         True       True      
native -> sys                                      True       -1         True       True      
native -> stat                                     True       -1         True       True  

For nltk the test name.split('.')[0] in INVALID_MODULES is False, but should be True. Why could that be?

- I had to type this 7 times! the google groups message composer is not very stable :(

[Daniel Gonzalez]

I have found a workaround (hack) to my problem. I had to modify:

INVALID_MODULES = set(('', 'gluon', 'applications', 'custom_import'))

to

INVALID_MODULES = set(('', 'gluon', 'applications', 'custom_import', 'nltk', 'collocations', 'numpy', 'testing'))

(there were more modules affected by the same problem). This is ugly, so I hope we can find a real fix for this.

It seems the custom_import_install:

def custom_import_install():
    if __builtin__.__import__ == NATIVE_IMPORTER:
        INVALID_MODULES.update(sys.modules.keys())
        __builtin__.__import__ = custom_importer

Is not working as expected?

[Massimo Di Pierro]

I do not know why that is.

For now you can also do:

from gluon.custom_import import INVALID_MODULES

for key in ('nltk', 'collocations', 'numpy', 'testing'):

   INVALID_MODULES.add(key)