Using python as an user rules engine in a web2py application

[Carlos Kitu]

Hellow everyone,

I'm developing an application for a consulting company. To make a long story short, there is some data in the database that is user administered, and some formulas to execute with that data. Those formulas should be administered by the consultant.

The database data can be easily manageable by the end user in web2py. 

With regard to the formulas, I considered to use some business rules engine. Then I realised that python is such a nice programming language that who needs a business rules engine? I gave a try to python as a user formulas specification.

I tried with a user module file with a function. That function has the following parameters:

• parameteres: where selected data from the database is injected and available to be used in formulas
• results: another dictionary where the user can store the results back to the calling function. 

This works quite fine. 

Just to give an idea with a simplified sample code:

controllers/controller.py:

from gluon.storage import Storage

def calling_function():

    parameters, results = Storage(), Storage()

    parameters=get_data_from_database() 

    # Now we have defined things like:

    # parameters.var1, parameters.var2, ...

    

    from user_module import user_function

      user_function(parameters, results)

    # Now we have in results the key value1 with the resulting value

modules/user_module.py

def user_function(p, r):

    # For instance if we want to add two values:

    r.value1=p.var1 + p.var2

    return None

My question is:

I need different versions of the user function and allow the end user to edit those functions. This is needed because different organizational units(OU) would have different formula definition. 

Those formulas would be edited by a trusted and trained user. Obviously there are some risks in allowing the end user to edit  python code. To minimize the risk, it is isolated in one module and the call to that user function would be controlled by a try: clause.

I considered several alternatives:

1. Using just one user module and different functions, like user_function1, user_function2, and calling usermodule.user_function1(), user_module.user_function2() from the calling_function depending on the OU related.
   
2. Using different user modules with the same user_function. That means calling user_module1.userfunction(), user_module2.userfunction()

Obviously there is a need of creating a kind of scaffolding function user_module.user_functionX() to be edited by the user, or an scaffolding module user_moduleX with a user_function() to be edited by the user. The scaffolding function or module would be created by a controller.

Can you recommend any of those approaches, or a better one?

Thanks in advance.

Best regards

[Alfonso Serra]

Allowing users to execute random code within the webapp environment is a really bad idea.
A simple db.auth_user.truncate() within any of those user functions would tear your whole app apart.

Some suggestion is the creation of any kind of engine or environment in which to execute code safely.
In order to do that you can use the app to inspect the code for forbidden commands or check to whether the function returns a result or not.

In python you can create context managers that could take care of that.

You can also try to launch user defined code as independent threads. See threading module.

In a sense, those user functions are dynamic, so should be text that you can store in the database or text files in a private folder. The application could pull that data and use exec(random_code) within a safe environment. There are some examples in david beazly's video about metaprogramming.

[Carlos Kitu]

Thanks a lot Alfonso for your quick and useful answer.

I agree with your concerns about security. It's a real issue.

On the other side, I'm taking into account some issues that help to mitigate the risks:

• It will be a trusted and trained user of my company(non IT person), not the customer, with an specific role limiting the access to that funcionallity.
• Inside of a module there is a lower risk if somebody is messing around because all the critical objects like db, request, ... are not available as happens in the controllers, unless explicitly passed as arguments, and that is not gonna happen. I'm just using dictionaries to send variables, receive results.

Until now, my greatest concern is about unintended user errors in the formulas, causing an exception. I thought about creating a file with an scaffolding module, letting the user download it, edit it and upload again. Then the uploaded code could be checked as you suggest, compiled to detect errors, etc. 

I'm considering that it will be safer/easier to have different user modules(actually files), for every different set of business rules, than one module with different functions.

I find the options you mentioned, like context managers and threads really interesting. Need to consider it carefully because I haven't deal with them until now. And thanks a lot for the video reference. Most interesting.

Any other hints will be welcome.

Thank you and best regards.

[Leonel Câmara]

Have you considered creating a DSL? Creating a DSL in python is quite easy. Python might be easy but it is still an imperative language, non-programmers don't think in either imperative or functional terms, they think declaratively, they want to declare what they want to the computer and let it figure it out how to do it.

[Carlos Kitu]

Thank you very much Leonel. This is also a very useful hint. As with the previous recommendation I need a deeper knowledge about the subject to make a decission, and this is a good point to start with.

Thank you and best regards.

[Richard Vézina]

Why not just found a way to attach Ipython notebook (which seem to had been renamed Jupyter) to your app?? 

https://jupyter.org/

By providing dataset over which users can interact with... I am not sure exactly how to do that, but one idea could be to db select data that user can play with inside a module that can be import in an notebook...

It would be very powerful and you don't have to care about security as it assured by ipython notebook...

Richard

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave S]

On Thursday, April 28, 2016 at 12:01:50 PM UTC-7, Richard wrote:

Why not just found a way to attach Ipython notebook (which seem to had been renamed Jupyter) to your app?? 

https://jupyter.org/

By providing dataset over which users can interact with... I am not sure exactly how to do that, but one idea could be to db select data that user can play with inside a module that can be import in an notebook...

It would be very powerful and you don't have to care about security as it assured by ipython notebook...

Richard

That would be appropriate if the user just needs to play around with the data because they're looking for something interesting, or need a one-time calculation.

If the user ("semi-administrator") needs to change how the website does a calculation that other users will see, a DSL would seem to be more appropriate.  Or maybe one of those visual programming things where you use "virtual Lego(tm) pieces" which are compute blocks that you connect up (I don't know of any of those that would fit in as an IDE the way the web2py IDE fits the admin app).  Yet Another Option would be a server-side sandbox that could flag unsafe actions, but I have no information on such a beast.

/dps

[Richard Vézina]

Good points... I forgot about the aspect of changing site way to make calculation... At the same time, I understand the need as more specific for some user the rest of the crowd stock with the basic calculation if they don't know how to hack their own calculation function... In this context Jupyter seems a good choice... It may seems odd to add such a dependancy for the need at first but considering that your application can evolve and you may be victim of the success you may need to extend the user base of the feature and at some point you will not be able to "trust" your user(s) (if you ever be able too)...

[Carlos Kitu]

Thanks again Richard, Dave for your useful hints.

Just to clarify, the need of trusting a user with formula editing is for allowing a non-IT trusted user to edit the business logic that has to be processed in an application. That means getting data from the database, processing it, storing the results in the database. Dictionaries proved to be a safe way to deliver and receive data to/from the business rules.

Traditionally, this issue has been solved before with a business rules engine. Drools (http://www.drools.org/) is a good example I got to know, and I was looking for something similar for python when I realized that it is not easy to find those tools for python, probably because this language is easy enough.

With regard to Jupyter, it has been for a long time in my radar, but I had no time to get to it. As for what I have read, Jupyter would be safe place to allow people to operate with the data but maybe there is no easy way to allow the user to give back the processed data, as Richard mentiond, but maybe I'm wrong.

I don't foresee a lot of people editing formulas. This would be a critical task assigned to a well known and trusted user.

Until now, as for the suggestiongs received I'm more inclined to explore the DSL way or creating a safe sandbox in python.

Thank you guys for your useful help and interest.

Best regards.