Why am I getting "unable to import requests" in web2py even though requests is accessible directly through python

431 views
Skip to first unread message

vald...@gmail.com

unread,
May 26, 2023, 10:02:36 AM5/26/23
to web2py-users

Asked this one on SO here as I was just reminded of how much nicer this community tends to be to newb questions.

I'm attempting to integrate MSAL which requires the requests module.

I'm running python 3.7 on Linux and using pipenv to manage the environment. I'm also using web2py 2.24.1 from source (as in I download the web2py framework via the source button on the web2py website).

When I am in pipenv shell and go into the python shell, I can access the requests, however when I try to access it from web2py (running in same shell), I get the "module not found" error.

When I check the site-packages folder the requests packages is present. I have checked the pythonpath and seen that the path from the virtual environment is present.

When attempting to load the web2py python shell, it gives the same error.

I'm probably missing something, but it sometimes appears as if web2py does some code compilation and then uses the compiled stuff and ignores code changes after a certain point. Asking as I have commented out all the code involving the requests module in an effort to get the web2py shell working, but still get the error and now the entire app is not working even on the master branch which was previously working.

Not sure what to try next. Any ideas are appreciated.

Tom Clerckx

unread,
May 26, 2023, 7:06:56 PM5/26/23
to web2py-users
Good point, I'm curious as well.

I did a few experiments and I noticed that:

Running a virtualenv built on python 2.7.18 and using web2py (python2 version) "Version 2.24.1-stable+timestamp.2023.03.23.05.07.17", the import goes OK within a web2py shell.
Running a virtualenv built on python 3.10.10 and using web2py (python3 version) "Version 2.24.1-stable+timestamp.2023.03.23.05.07.17", the import FAILS within a web2py shell.

In both cases the import of the requests library works fine when running just the python interpreter.

vald...@gmail.com

unread,
May 26, 2023, 8:05:08 PM5/26/23
to web2py-users
What also gets me is that when I switched back to my original branch that doesn't have the msal and requests stuff - I'm now still getting the same error in the master branch..

Massimiliano

unread,
May 27, 2023, 6:11:19 AM5/27/23
to web...@googlegroups.com
Maybe one thing you can do is to launch web2py shell and check paths.

./web2py.py -S <your-app-name> 

import sys
print(sys.path)




--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/83b8a4ab-120a-4106-83f2-d37652c23093n%40googlegroups.com.


--
Massimiliano

Tom Clerckx

unread,
May 27, 2023, 8:19:57 AM5/27/23
to web2py-users
See below. In both tests the requests library is in the last element of the path.

TEST1, plain python shell - import OK

(test) tclerckx@stark:~/temp/test$ python
Python 3.10.6 (main, Jan 14 2023, 23:48:13) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__file__
'/home/tclerckx/.pyenv/versions/test/lib/python3.10/site-packages/requests/__init__.py'
>>> import sys
>>> sys.path
['', '/home/tclerckx/.pyenv/versions/3.10.6/lib/python310.zip', '/home/tclerckx/.pyenv/versions/3.10.6/lib/python3.10', '/home/tclerckx/.pyenv/versions/3.10.6/lib/python3.10/lib-dynload', '/home/tclerckx/.pyenv/versions/test/lib/python3.10/site-packages']
>>>

TEST2, python web2py shell - import FAILS

(test) tclerckx@stark:~/temp/test/web2py$ python web2py.py -S welcome -M
web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2023
Version 2.24.1-stable+timestamp.2023.03.23.05.07.17
Database drivers available: sqlite3, imaplib, pymysql
WARNING:web2py:import IPython error; use default python shell
Python 3.10.6 (main, Jan 14 2023, 23:48:13) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import requests
Traceback (most recent call last):
  File "/home/tclerckx/temp/test/web2py/gluon/custom_import.py", line 78, in custom_importer
    result = sys.modules[modules_prefix]
KeyError: 'applications.welcome.modules.requests'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/tclerckx/temp/test/web2py/gluon/custom_import.py", line 80, in custom_importer
    raise ImportError("No module named %s" % modules_prefix)
ImportError: No module named applications.welcome.modules.requests
>>> import sys
>>> sys.path
['', '/home/tclerckx/temp/test/web2py/site-packages', '/home/tclerckx/temp/test/web2py', '/home/tclerckx/temp/test/web2py/gluon/packages/yatl', '/home/tclerckx/temp/test/web2py/gluon/packages/dal', '/home/tclerckx/.pyenv/versions/3.10.6/lib/python310.zip', '/home/tclerckx/.pyenv/versions/3.10.6/lib/python3.10', '/home/tclerckx/.pyenv/versions/3.10.6/lib/python3.10/lib-dynload', '/home/tclerckx/.pyenv/versions/test/lib/python3.10/site-packages']
>>>

Massimiliano

unread,
May 27, 2023, 8:38:28 AM5/27/23
to web...@googlegroups.com
I'm also using pyenv and having the same paths order, ma importing requests is working with no problem, on macos and on linux.

Have you already tried to get rid of all pyc files and or to uninstall and reinstall requests?



--
Massimiliano

Tom Clerckx

unread,
May 27, 2023, 12:12:52 PM5/27/23
to web2py-users
With pyenv itself it goes OK, it's only when using a pyenv-virtualenv that things fail.
I tried on two different machines.

Newly created pyenv (3.10.6)
Newly  created virtualenv
Newly unzipped web2py directory (2.24.1)

Same result.
When launching the web2py-shell in the virtualenv, the import fails.

Have you also tried with a virtualenv?

Regards,
Tom.

Massimiliano

unread,
May 27, 2023, 12:59:18 PM5/27/23
to web...@googlegroups.com
Yes it's a virtualenv also mine. 
How do you start web2py and how do you set your virtual env for web2py? 

pyenv local yourenv 
or what?

Another thing. web2py.py has #!/usr/bin/env python shebang so it knows what python to use.

Have you tried to launch 
./web2py.py -S 
instead of 
python web2py.py

as last test you can try to call web2py with the full path of your virtualenv:

<your pyenv path>/.pyenv/versions/<your venv>/bin/python web2py.py -S <yourapp>





--
Massimiliano

vald...@gmail.com

unread,
May 27, 2023, 7:25:36 PM5/27/23
to web2py-users
I'm using pipenv to run the virtual environment.

So I install pyenv first and then python 3.7 with it and set it to the default.

Then I install pipenv so that it uses the python 3.7 as its default.

Then I activate the virtual environment  with pipenv shell and then run python web2py/web2py.py -a pwd -p 7000.

To get the web2py shell, I run python web2py/web2py.py -S init

I pretty much got the same results -- see below

I actually have. I'm running a virtual environment with Pipenv.

Noting essentially the same results.

(t2d) popos@popos2004:~/code/t2d
$ python web2py/web2py.py -S init

web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2023
Version 2.22.5-stable+timestamp.2022.06.04.18.13.51

Database drivers available: sqlite3, imaplib, pymysql
WARNING:web2py:import IPython error; use default python shell
Python 3.7.12 (default, Feb 11 2023, 17:19:18)
[GCC 9.3.0] on linux

Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import sys
>>> print(sys.path)
['', '/home/popos/code/t2d/web2py/site-packages', '/home/popos/code/t2d/web2py', '/home/popos/code/t2d/web2py/gluon/packages/yatl', '/home/popos/code/t2d/web2py/gluon/packages/dal', '/home/popos/.pyenv/versions/3.7.12/lib/python37.zip', '/home/popos/.pyenv/versions/3.7.12/lib/python3.7', '/home/popos/.pyenv/versions/3.7.12/lib/python3.7/lib-dynload', '/home/popos/.local/share/virtualenvs/t2d-ueLDNtkj/lib/python3.7/site-packages']
>>>
now exiting InteractiveConsole...

The site-packages folder has the requests module files in it.

Able to see it in plain python, but not in web2py shell or the web2py app.

Massimiliano

unread,
May 28, 2023, 3:56:01 AM5/28/23
to web...@googlegroups.com
What I can say i that with pyenv and virtualenv created in pyenv it works.

Just tried on a new vm with debian 11

Steps: 

apt update
apt install curl git

install pyenv:

curl https://pyenv.run | bash

setup the build environment to build python version, from pyenv docs:

apt install build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev curl \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

Installing python 3.7.16:

pyenv install 3.7.16

wait for the build process... 

adding this in .profile 

export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"


restart the shell or logout... 

create an virtual env based on 3.7.16:

pyenv virtualenv 3.7.16 testvenv

getting web2py:

cd web2py/
git submodule update --init --recursive

Activate the venv:

pyenv local testvenv
pip install requests

(testvenv) max@d11:~/web2py$ ./web2py.py -S welcome

web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2023
Version 2.24.1-stable+timestamp.2023.03.22.21.39.14

Database drivers available: sqlite3, imaplib, pymysql
WARNING:web2py:import IPython error; use default python shell
Python 3.7.16 (default, May 28 2023, 09:25:55)
[GCC 10.2.1 20210110] on linux


--
Massimiliano

Massimiliano

unread,
May 28, 2023, 4:04:33 AM5/28/23
to web...@googlegroups.com
Sorry it's not clear that only the apt stuffs are done as root.

Installing pyenv
curl https://pyenv.run | bash 

and from:

pyenv install 3.7.16

is all done as normal user.

--
Massimiliano

vald...@gmail.com

unread,
May 29, 2023, 9:21:23 AM5/29/23
to web2py-users
Interesting
I had actually put together a little script to setup my environment. The difference is that I download the source from the website instead of cloning from Github. Wondering if that could be the cause. Here's my entire script:
#!/usr/bin/env bash

FILE=web2py/web2py.py
echo
echo "Checking for $FILE..."
if [ -f "$FILE" ];
then
echo "..."
else
echo "File does not exist."
unzip web2py_src.zip
rm web2py_src.zip 2>/dev/null \
&& echo "Deleted web2py_src.zip" \
|| echo "Unable to delete as web2py_src.zip does not exist"
fi
CONFIGFILE=web2py/applications/init/private/appconfig.ini
echo
echo "Checking for $CONFIGFILE..."
if [ -f "$CONFIGFILE" ];
then
echo "..."
else
echo "File does not exist. Copying from welcome app..."
mkdir web2py/applications/init/private
cp web2py/applications/welcome/private/appconfig.ini web2py/applications/init/private
sleep 2
fi
if [ -f "$CONFIGFILE" ];
then
echo "Starting server..."
pipenv run python web2py/web2py.py -a pwd -p 7000
else
echo "$CONFIGFILE file does not exist."

Massimiliano

unread,
May 29, 2023, 9:29:51 AM5/29/23
to web...@googlegroups.com
Tried right now.

 extracting: web2py/applications/__init__.py
 extracting: web2py/site-packages/__init__.py
max@d11:~/aaa$ cd web2py/
max@d11:~/aaa/web2py$ pyenv local testvenv
(testvenv) max@d11:~/aaa/web2py$ ./web2py.py -S welcome

web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2023
Version 2.24.1-stable+timestamp.2023.03.23.05.07.17
Database drivers available: sqlite3, imaplib, pymysql
WARNING:web2py:import IPython error; use default python shell

Massimiliano

unread,
May 29, 2023, 9:43:11 AM5/29/23
to web...@googlegroups.com
Just for the records:

clean python 3.7.16 w/o requests:
pyenv local 3.7.16

pip install pipenv
pipenv install requests
max@d11:~/aaa/web2py$ pipenv shell
Launching subshell in virtual environment...
max@d11:~/aaa/web2py$  . /home/max/.local/share/virtualenvs/web2py-ifCUjADX/bin/activate
(web2py) max@d11:~/aaa/web2py$ python

Python 3.7.16 (default, May 28 2023, 09:25:55)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>>

(web2py) max@d11:~/aaa/web2py$ ./web2py.py -S welcome

web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2023
Version 2.24.1-stable+timestamp.2023.03.23.05.07.17
Database drivers available: sqlite3, imaplib, pymysql
WARNING:web2py:import IPython error; use default python shell
Python 3.7.16 (default, May 28 2023, 09:25:55)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import requests
Traceback (most recent call last):
  File "/home/max/aaa/web2py/gluon/custom_import.py", line 78, in custom_importer

    result = sys.modules[modules_prefix]
KeyError: 'applications.welcome.modules.requests'

It seems that pipenv introduces some oddity that broke the custom_import in web2py.


--
Massimiliano

Saifuddin Rangwala

unread,
May 29, 2023, 11:57:46 AM5/29/23
to web...@googlegroups.com
I had the same issue recently, but on windows. I did a workaround by importing requests before importing gluon in web2py.py. Looks like something is broken, after gluon is imported.



--
Saifuddin,
---------------------------------------------
Reach me @ +91 9850815274
---------------------------------------------

"The secret to find something is knowing where it is"

Leonel Câmara

unread,
May 30, 2023, 9:44:28 AM5/30/23
to web2py-users
I had this same problem recently, in my case I thought the issue was because I had some python modules installed using pip and other modules installed using apt.

I started by removing all the modules installed by apt and installing them using pip. The issue persisted, but I solved it by pip installing brotli, urllib3 and urllib3_secure_extra. Apparently I was missing some requirements the "requests" module has. That finally solved the problem.

To check for missing dependencies. You can change line 58-61 in gluon/custom_import.py from:

try:

     return NATIVE_IMPORTER(name, globals, locals, fromlist, level)

except (ImportError, KeyError):

     pass

To

try:
     return NATIVE_IMPORTER(name, globals, locals, fromlist, level)

except (ImportError, KeyError) as e:

     print(e)

Then open a web2py shell in your application and do a "import requests", this should show you what's going wrong importing the requests module.

Massimiliano

unread,
May 30, 2023, 11:55:26 AM5/30/23
to web...@googlegroups.com
Just tried and it works flawless! :-) 



--
Massimiliano

Tom Clerckx

unread,
May 30, 2023, 5:19:14 PM5/30/23
to web2py-users
I also confirm I get the web2py shell working after installing some extra pip package...

Actually, I only had to add urllib3_secure_extra to get requests working in the web2py shell.

Checking the python site-packages, I see the following in the urllib3 package __init__.py:

# === NOTE TO REPACKAGERS AND VENDORS ===
# Please delete this block, this logic is only
# for urllib3 being distributed via PyPI.
# See: https://github.com/urllib3/urllib3/issues/2680
try:
    import urllib3_secure_extra  # type: ignore # noqa: F401
except ImportError:
    pass
else:
    warnings.warn(
        "'urllib3[secure]' extra is deprecated and will be removed "
        "in a future release of urllib3 2.x. Read more in this issue: "
        "https://github.com/urllib3/urllib3/issues/2680",
        category=DeprecationWarning,
        stacklevel=2,
    )


So there is no real need for urllib3_secure_extra (as is shown in the plain python shelll), but somehow in the web2py shell, this dependency comes back.

Tom.

vald...@gmail.com

unread,
Jun 1, 2023, 9:51:24 AM6/1/23
to web2py-users
Hey guys,

Apparently this is due to a "a buggy interaction between web2py's custom importer and the urllib3 module (which is imported by requests)"

Ended up placing the code below in the __init__.py for the app which worked:

import builtins
_real_import = builtins.__import__
def _import_with_modulenotfounderror(*args, **kwargs):
try:
return _real_import(*args, **kwargs)
except ImportError as e:
if e.__class__ is ImportError and str(e).startswith("No module named "):
raise ModuleNotFoundError(str(e)) from e
else:
raise
builtins.__import__ = _import_with_modulenotfounderror

See solution in context of question here on SO.

Krzysztof Socha

unread,
Dec 13, 2023, 5:40:51 AM12/13/23
to web2py-users
So... I ran into a similar problem as well... In my case - after debugging with the approach proposed by Leonel above - I found out that I had to additionally _manually_ import `importlib_resources` (https://pypi.org/project/importlib-resources/). No need to do so if using just the Python console, but somehow (I do not have time to debug) the `custom_import.py` is unable to import some packages (in my case `pyvat`) unless this additional package is manually installed.

I would also be useful to have a bit more elaborate error message from web2py rather than ignore the errors and _assume_ that orignal package being imported is not installed. 

Good luck everyone trying to use web2py with the new Python versions in general... 

Cheers,
Krzysztof. 

Massimo Di Pierro

unread,
Dec 28, 2023, 3:26:22 AM12/28/23
to web2py-users
Thank you for bringing this problem up.
Once again I encourage people to move to py4web which is a solid and better supported replacement for web2py.
Reply all
Reply to author
Forward
0 new messages