Why am I getting "unable to import requests" in web2py even though requests is accessible directly through python

[vald...@gmail.com]

Asked this one on SO here as I was just reminded of how much nicer this community tends to be to newb questions.

I'm attempting to integrate MSAL which requires the requests module.

I'm running python 3.7 on Linux and using pipenv to manage the environment. I'm also using web2py 2.24.1 from source (as in I download the web2py framework via the source button on the web2py website).

When I am in pipenv shell and go into the python shell, I can access the requests, however when I try to access it from web2py (running in same shell), I get the "module not found" error.

When I check the site-packages folder the requests packages is present. I have checked the pythonpath and seen that the path from the virtual environment is present.

When attempting to load the web2py python shell, it gives the same error.

I'm probably missing something, but it sometimes appears as if web2py does some code compilation and then uses the compiled stuff and ignores code changes after a certain point. Asking as I have commented out all the code involving the requests module in an effort to get the web2py shell working, but still get the error and now the entire app is not working even on the master branch which was previously working.

Not sure what to try next. Any ideas are appreciated.

[Tom Clerckx]

Good point, I'm curious as well.

I did a few experiments and I noticed that:

Running a virtualenv built on python 2.7.18 and using web2py (python2 version) "Version 2.24.1-stable+timestamp.2023.03.23.05.07.17", the import goes OK within a web2py shell.

Running a virtualenv built on python 3.10.10 and using web2py (python3 version) "Version 2.24.1-stable+timestamp.2023.03.23.05.07.17", the import FAILS within a web2py shell.

In both cases the import of the requests library works fine when running just the python interpreter.

[vald...@gmail.com]

What also gets me is that when I switched back to my original branch that doesn't have the msal and requests stuff - I'm now still getting the same error in the master branch..

[Massimiliano]

Maybe one thing you can do is to launch web2py shell and check paths.

./web2py.py -S <your-app-name> 

import sys

print(sys.path)

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/83b8a4ab-120a-4106-83f2-d37652c23093n%40googlegroups.com.

--

Massimiliano

[Tom Clerckx]

See below. In both tests the requests library is in the last element of the path.

TEST1, plain python shell - import OK

(test) tclerckx@stark:~/temp/test$ python
Python 3.10.6 (main, Jan 14 2023, 23:48:13) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__file__
'/home/tclerckx/.pyenv/versions/test/lib/python3.10/site-packages/requests/__init__.py'
>>> import sys
>>> sys.path
['', '/home/tclerckx/.pyenv/versions/3.10.6/lib/python310.zip', '/home/tclerckx/.pyenv/versions/3.10.6/lib/python3.10', '/home/tclerckx/.pyenv/versions/3.10.6/lib/python3.10/lib-dynload', '/home/tclerckx/.pyenv/versions/test/lib/python3.10/site-packages']
>>>

TEST2, python web2py shell - import FAILS

(test) tclerckx@stark:~/temp/test/web2py$ python web2py.py -S welcome -M
web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2023
Version 2.24.1-stable+timestamp.2023.03.23.05.07.17
Database drivers available: sqlite3, imaplib, pymysql
WARNING:web2py:import IPython error; use default python shell
Python 3.10.6 (main, Jan 14 2023, 23:48:13) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import requests
Traceback (most recent call last):
  File "/home/tclerckx/temp/test/web2py/gluon/custom_import.py", line 78, in custom_importer
    result = sys.modules[modules_prefix]
KeyError: 'applications.welcome.modules.requests'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/tclerckx/temp/test/web2py/gluon/custom_import.py", line 80, in custom_importer
    raise ImportError("No module named %s" % modules_prefix)
ImportError: No module named applications.welcome.modules.requests
>>> import sys
>>> sys.path
['', '/home/tclerckx/temp/test/web2py/site-packages', '/home/tclerckx/temp/test/web2py', '/home/tclerckx/temp/test/web2py/gluon/packages/yatl', '/home/tclerckx/temp/test/web2py/gluon/packages/dal', '/home/tclerckx/.pyenv/versions/3.10.6/lib/python310.zip', '/home/tclerckx/.pyenv/versions/3.10.6/lib/python3.10', '/home/tclerckx/.pyenv/versions/3.10.6/lib/python3.10/lib-dynload', '/home/tclerckx/.pyenv/versions/test/lib/python3.10/site-packages']
>>>

[Massimiliano]

I'm also using pyenv and having the same paths order, ma importing requests is working with no problem, on macos and on linux.

Have you already tried to get rid of all pyc files and or to uninstall and reinstall requests?

To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/9e3df18e-403a-4bd2-b42b-8c9028da601fn%40googlegroups.com.

--

Massimiliano

[Tom Clerckx]

With pyenv itself it goes OK, it's only when using a pyenv-virtualenv that things fail.

I tried on two different machines.

Newly created pyenv (3.10.6)

Newly  created virtualenv

Newly unzipped web2py directory (2.24.1)

Same result.

When launching the web2py-shell in the virtualenv, the import fails.

Have you also tried with a virtualenv?

Regards,

Tom.

[Massimiliano]

Yes it's a virtualenv also mine. 

How do you start web2py and how do you set your virtual env for web2py? 

pyenv local yourenv 

or what?

Another thing. web2py.py has #!/usr/bin/env python shebang so it knows what python to use.

Have you tried to launch 

./web2py.py -S 

instead of 

python web2py.py

as last test you can try to call web2py with the full path of your virtualenv:

<your pyenv path>/.pyenv/versions/<your venv>/bin/python web2py.py -S <yourapp>

To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/7e1b981a-d2fe-4af2-8e42-57acc4ca1e7cn%40googlegroups.com.

--

Massimiliano

[vald...@gmail.com]

I'm using pipenv to run the virtual environment.

So I install pyenv first and then python 3.7 with it and set it to the default.

Then I install pipenv so that it uses the python 3.7 as its default.

Then I activate the virtual environment  with pipenv shell and then run python web2py/web2py.py -a pwd -p 7000.

To get the web2py shell, I run python web2py/web2py.py -S init

I pretty much got the same results -- see below

I actually have. I'm running a virtual environment with Pipenv.

Noting essentially the same results.

(t2d) popos@popos2004:~/code/t2d

$ python web2py/web2py.py -S init

web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2023

Version 2.22.5-stable+timestamp.2022.06.04.18.13.51

Database drivers available: sqlite3, imaplib, pymysql
WARNING:web2py:import IPython error; use default python shell

Python 3.7.12 (default, Feb 11 2023, 17:19:18)
[GCC 9.3.0] on linux

Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)

>>> import sys
>>> print(sys.path)
['', '/home/popos/code/t2d/web2py/site-packages', '/home/popos/code/t2d/web2py', '/home/popos/code/t2d/web2py/gluon/packages/yatl', '/home/popos/code/t2d/web2py/gluon/packages/dal', '/home/popos/.pyenv/versions/3.7.12/lib/python37.zip', '/home/popos/.pyenv/versions/3.7.12/lib/python3.7', '/home/popos/.pyenv/versions/3.7.12/lib/python3.7/lib-dynload', '/home/popos/.local/share/virtualenvs/t2d-ueLDNtkj/lib/python3.7/site-packages']
>>>
now exiting InteractiveConsole...

The site-packages folder has the requests module files in it.

Able to see it in plain python, but not in web2py shell or the web2py app.

[Massimiliano]

What I can say i that with pyenv and virtualenv created in pyenv it works.

Just tried on a new vm with debian 11

Steps: 

apt update

apt install curl git

install pyenv:

curl https://pyenv.run | bash

setup the build environment to build python version, from pyenv docs:

apt install build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev curl \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

Installing python 3.7.16:

pyenv install 3.7.16

wait for the build process... 

adding this in .profile 

export PYENV_ROOT="$HOME/.pyenv"

command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"

eval "$(pyenv init -)"

eval "$(pyenv virtualenv-init -)"

restart the shell or logout... 

create an virtual env based on 3.7.16:

pyenv virtualenv 3.7.16 testvenv

getting web2py:

git clone https://github.com/web2py/web2py.git

cd web2py/

git submodule update --init --recursive

Activate the venv:

pyenv local testvenv
pip install requests

(testvenv) max@d11:~/web2py$ ./web2py.py -S welcome

web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2023

Version 2.24.1-stable+timestamp.2023.03.22.21.39.14

Database drivers available: sqlite3, imaplib, pymysql
WARNING:web2py:import IPython error; use default python shell

Python 3.7.16 (default, May 28 2023, 09:25:55)
[GCC 10.2.1 20210110] on linux

To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/da72c846-1b5a-448f-a51c-32be529a1935n%40googlegroups.com.

--

Massimiliano

[Massimiliano]

Sorry it's not clear that only the apt stuffs are done as root.

Installing pyenv

curl https://pyenv.run | bash 

and from:

pyenv install 3.7.16

is all done as normal user.

--

Massimiliano

[vald...@gmail.com]

Interesting

I had actually put together a little script to setup my environment. The difference is that I download the source from the website instead of cloning from Github. Wondering if that could be the cause. Here's my entire script:

#!/usr/bin/env bash

FILE=web2py/web2py.py

echo

echo "Checking for $FILE..."

if [ -f "$FILE" ];

then

echo "..."

else

echo "File does not exist."

wget https://mdipierro.pythonanywhere.com/examples/static/web2py_src.zip

unzip web2py_src.zip

rm web2py_src.zip 2>/dev/null \

&& echo "Deleted web2py_src.zip" \

|| echo "Unable to delete as web2py_src.zip does not exist"

fi

CONFIGFILE=web2py/applications/init/private/appconfig.ini

echo

echo "Checking for $CONFIGFILE..."

if [ -f "$CONFIGFILE" ];

then

echo "..."

else

echo "File does not exist. Copying from welcome app..."

mkdir web2py/applications/init/private

cp web2py/applications/welcome/private/appconfig.ini web2py/applications/init/private

sleep 2

fi

if [ -f "$CONFIGFILE" ];

then

echo "Starting server..."

pipenv run python web2py/web2py.py -a pwd -p 7000

else

echo "$CONFIGFILE file does not exist."

[Massimiliano]

Tried right now.

 extracting: web2py/applications/__init__.py
 extracting: web2py/site-packages/__init__.py
max@d11:~/aaa$ cd web2py/
max@d11:~/aaa/web2py$ pyenv local testvenv
(testvenv) max@d11:~/aaa/web2py$ ./web2py.py -S welcome

web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2023

Version 2.24.1-stable+timestamp.2023.03.23.05.07.17
Database drivers available: sqlite3, imaplib, pymysql
WARNING:web2py:import IPython error; use default python shell

To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/3c344817-a69f-427b-ac52-4e1b3918cd85n%40googlegroups.com.

--

Massimiliano

[Massimiliano]

Just for the records:

clean python 3.7.16 w/o requests:

pyenv local 3.7.16

pip install pipenv

pipenv install requests

max@d11:~/aaa/web2py$ pipenv shell

Launching subshell in virtual environment...
max@d11:~/aaa/web2py$  . /home/max/.local/share/virtualenvs/web2py-ifCUjADX/bin/activate

(web2py) max@d11:~/aaa/web2py$ python

Python 3.7.16 (default, May 28 2023, 09:25:55)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import requests
>>>

(web2py) max@d11:~/aaa/web2py$ ./web2py.py -S welcome

web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2023
Version 2.24.1-stable+timestamp.2023.03.23.05.07.17
Database drivers available: sqlite3, imaplib, pymysql
WARNING:web2py:import IPython error; use default python shell
Python 3.7.16 (default, May 28 2023, 09:25:55)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import requests

Traceback (most recent call last):

  File "/home/max/aaa/web2py/gluon/custom_import.py", line 78, in custom_importer

    result = sys.modules[modules_prefix]
KeyError: 'applications.welcome.modules.requests'

It seems that pipenv introduces some oddity that broke the custom_import in web2py.

--

Massimiliano

[Saifuddin Rangwala]

I had the same issue recently, but on windows. I did a workaround by importing requests before importing gluon in web2py.py. Looks like something is broken, after gluon is imported.

To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/CANPTPxK2oeg_5KxvEjzVoYU_xo0LLJKZWZGvP5Mr%2BoGVcSHrYQ%40mail.gmail.com.

--

Saifuddin,
---------------------------------------------
Reach me @ +91 9850815274
---------------------------------------------

"The secret to find something is knowing where it is"

[Leonel Câmara]

I had this same problem recently, in my case I thought the issue was because I had some python modules installed using pip and other modules installed using apt.

I started by removing all the modules installed by apt and installing them using pip. The issue persisted, but I solved it by pip installing brotli, urllib3 and urllib3_secure_extra. Apparently I was missing some requirements the "requests" module has. That finally solved the problem.

To check for missing dependencies. You can change line 58-61 in gluon/custom_import.py from:

try:
     return NATIVE_IMPORTER(name, globals, locals, fromlist, level)
except (ImportError, KeyError):

     pass

To

try:
     return NATIVE_IMPORTER(name, globals, locals, fromlist, level)
except (ImportError, KeyError) as e:

     print(e)

Then open a web2py shell in your application and do a "import requests", this should show you what's going wrong importing the requests module.

[Massimiliano]

Just tried and it works flawless! :-) 

To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/97b021b0-ae9d-4395-8692-9669993e7c0bn%40googlegroups.com.

--

Massimiliano

[Tom Clerckx]

I also confirm I get the web2py shell working after installing some extra pip package...

Actually, I only had to add urllib3_secure_extra to get requests working in the web2py shell.

Checking the python site-packages, I see the following in the urllib3 package __init__.py:

# === NOTE TO REPACKAGERS AND VENDORS ===
# Please delete this block, this logic is only
# for urllib3 being distributed via PyPI.
# See: https://github.com/urllib3/urllib3/issues/2680
try:
    import urllib3_secure_extra  # type: ignore # noqa: F401
except ImportError:
    pass
else:
    warnings.warn(
        "'urllib3[secure]' extra is deprecated and will be removed "
        "in a future release of urllib3 2.x. Read more in this issue: "
        "https://github.com/urllib3/urllib3/issues/2680",
        category=DeprecationWarning,
        stacklevel=2,
    )

So there is no real need for urllib3_secure_extra (as is shown in the plain python shelll), but somehow in the web2py shell, this dependency comes back.

Tom.

[vald...@gmail.com]

Hey guys,

Apparently this is due to a "a buggy interaction between web2py's custom importer and the urllib3 module (which is imported by requests)"

Ended up placing the code below in the __init__.py for the app which worked:

import builtins

_real_import = builtins.__import__

def _import_with_modulenotfounderror(*args, **kwargs):

try:

return _real_import(*args, **kwargs)

except ImportError as e:

if e.__class__ is ImportError and str(e).startswith("No module named "):

raise ModuleNotFoundError(str(e)) from e

else:

raise

builtins.__import__ = _import_with_modulenotfounderror

See solution in context of question here on SO.

[Krzysztof Socha]

So... I ran into a similar problem as well... In my case - after debugging with the approach proposed by Leonel above - I found out that I had to additionally _manually_ import `importlib_resources` (https://pypi.org/project/importlib-resources/). No need to do so if using just the Python console, but somehow (I do not have time to debug) the `custom_import.py` is unable to import some packages (in my case `pyvat`) unless this additional package is manually installed.

I would also be useful to have a bit more elaborate error message from web2py rather than ignore the errors and _assume_ that orignal package being imported is not installed. 

Good luck everyone trying to use web2py with the new Python versions in general... 

Cheers,

Krzysztof. 

[Massimo Di Pierro]

Thank you for bringing this problem up.

Once again I encourage people to move to py4web which is a solid and better supported replacement for web2py.