from ... import ...

[Lucas]

hello one and all,

back in the old days of web2py, we can develop modules and place them under the modules subdirectory and use them simply by "import chmpy" or "from chmpy import Gtest, grubbs" in any controller or even html view.  the really amazing thing was I could develop code, store it in a text field in db, and compile it on-the-fly when told to do so.  I could access these modules just like the latter quoted imports.

I'm trying to do the same thing in py4web and say my controller file is under the controllers subdirectory while the module is under the modules subdirectory.  in order to access the module I have to write "from ..modules.chmpy import Gtest, grubbs" into the controller file.  however, I can not figure a way to import that chmpy file from the db-stored compiled on-the-fly code.

am I missing something with the __init__.py file, which I've tried many options but I'm just guessing at this point?  how to I create an app environment that the modules are accessible to all controllers/*.py, templates/*.html, and on-the-fly code?

thank you in advance, Lucas

[Massimo DiPierro]

web2py was criticized a lot because of the custom importer. It was one of the limiting factors to its adoption because it "is not pythonic". So we moved away from it intentionally.

Yet I understand the problem and there is a solution.

1) create a folder in your app called "modules" and, like in web2py put modules in it, for example:

mkdir -p modules

mkdir -p modules/mymodule

echo "x=7" > modules/mymodule/__init__,py

2) in you app common.py add

def custom_import(name):

import importlib.util

spec = importlib.util.spec_from_file_location("add", settings.APP_FOLDER)

spec = importlib.util.spec_from_file_location("add", os.path.join(settings.APP_FOLDER, "modules", name, '__init__.py'))

module = importlib.util.module_from_spec(spec)

spec.loader.exec_module(module)

return module

3) wherever you need to import the module do something like this

from .common import custom_import

mymodule = custom_import("mymodule")

@action("index")

@action.uses("generic.html", auth, T)

def index():

x = mymodule.x

return locals()

This allows you to distribute the source of custom modules along with your projects.

I am not sure if this is something I want to encourage but it is possible.

Perhaps we should put that function in a helper or perhaps we should add a mechanism to add a list of requirements.txt to each app and tools to create a venv common to the installed apps by merging the requirements.txt.

All solutions are not very pythonic so I am going to think some more about this but open to ideas.

[Lucas]

hey Massimo,

I have about 8 very intense apps with web2py.  3 of those allow code to be written and stored in a db and run dynamically using exec and render.  like for my teaching website, my question pool and test banks are written with views for the question itself, a controller to setup the parameters for the view/question, and an answer controller.  it has worked great for 13 years.  I've done similar for dynamically run code for python programming and chemical laboratory computations and data analysis.  so it is critical that I get these systems back up and running under py4web.  

your proposal is in way of web2py's dynamic module loader and very important especially when the developer is working on the module that he's developing.  those modules in my old systems are extremely stable and I don't need dynamic loading and so when I make changes I restart py4web.  and to that end, I brute forced my modules into an "iquanta" subdirectory under my python's virtual environment "site-packages", creating an ad hoc package in the common space of python.  put, for instance chmpy, in there, made a few import changes necessary for py4web, and then on the app and dynamically-run db-stored code simply put "import iquanta.chmpy as chmpy" or "from iquanta.chmpy import Gtest, grubbs"

that is not to say that I won't need a module that is very app focused and still want a common modules subdirectory like before, but I think it should be used with the traditional from...import structure as it was with web2py.  I always thought that was very pythonic.

simply my opinion and I hope it helps,  lucas

[laundmo]

heya all,

i'm not sure i understand either your problem, Lucas, or your solution, Massimo.

I've been working with modules just fine by using relative imports instead. This doesn't require any custom loader or other changes to pythons import system.

I have the following folder structure:
apps/<myapp>/
├─ src/
│  ├─ controllers/
│  ├─ early_modules/
│  ├─ late_modules/
│  ├─ __init__.py
│  ├─ common.py
├─ __init__.py
├─ settings.py

The __init__.py next to settings just contains "from . import src"

The __init__.py inside src/ contains

# isort: off
# the order of these is important: early_modules can't import things set up by common.py
from . import early_modules

from .common import db, T, auth, cache, flash, session, # ...
from . import late_modules
from . import controllers

This allows me to write modules which don't need db, auth, cache, flash etc. inside early_modules, these will be useable by common.py. Here i've done things like define my own Auth subclass, my own Cache, etc.
Inside late_modules, all the imports from common.py are available simply by doing "from .. import db, auth", and this is where i've done things like define custom fixtures etc.

Inside the controllers folder, i have multiple python files with controllers. These can use the following style of imports: "from .. import db, T, auth", "from ..early_modules import helper_func", "from ..late_modules import custom_db_expression"

This gives you all the flexibility of modules using pythons import system in the way it was intended.

I hope that helps some

[Christian Varas]

Hello all 👋 

This is an interesting suggestion, but I have some concerns. While it might seem convenient to import code from the database, this approach can introduce significant risks and maintenance problems.

For this reason, I strongly advise against adding this functionality to the py4web core. While a workaround might be okay for specific use cases, it isn't a common practice. 

I agree with @laundmo, storing modules as Python files is a much cleaner and more standard approach.

​More importantly, introducing this feature would create a significant security risk. If a malicious user were able to modify these functions in the database, whether through a SQL injection or even just a regular web form, they could execute arbitrary code. This would completely compromise your application. 

Py4web is built to be clean, minimal and fast, and this functionality would go against that core principle. Let's maintain the current structure, which keeps the framework secure and easy to manage.

​Greetings,

--
You received this message because you are subscribed to the Google Groups "py4web" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py4web+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/py4web/b420540a-5666-4d79-a10a-b1c95855d157n%40googlegroups.com.

[Massimo DiPierro]

If this is not per app you can do

sys.path.append(...path to iquanta)

Will that work for you?

--

You received this message because you are subscribed to the Google Groups "py4web" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py4web+un...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/py4web/80e2d455-18d3-4c9b-a075-5954d392966dn%40googlegroups.com.

[laundmo]

Hi Massimo,

Instead of modifying sys.path, I would highly recommend using normal python packages.
If you have code which should be shared across multiple apps, I recommend making it a python package by including one of the packaging definitions (setup.py, pyproject.toml).
You can then list those packages in requirements.txt, even as relative paths (note: relative paths in requirements.txt are relative to the current working directory, not the requirements.txt file)
or just pip install from the path, from a git repo, a private package registry (built in to some git forges) or if its generally useful even publish on pypi and install from there.

- laund