How to call PRC session (or a custom python function) in PEPS with delay

[mustafa dikmen]

Hello, I would like to invoke a custom function which has some parameters and also uses the PRC session. This invoke will be done by the delay mechanism inside a PEP. But I am not able to succeed this although I tried various things, getting errors .

 
My findings::
- My function can be accessed and called via the delay syntax only if it is defined with serialized irods parameters (rule_args, callback, rei).
- But these irods parameters dont follow/keep the embedded arrays of the PEP that it is invoked by.
- I cannot access the function if it is defined inside the PEP - I mean define a custom function inside a PEP and call it via the delay msi indside the same pep.
- I guess there seems no way to call a PRC session object with the delay unless using a wrapper function? Something like:
callback.delayExec("directives", "session.acls.set(new_permission)", "")
- I am partially able to achive what I want to do only if I define my functon with the serialized irods parameters outside the pep and if I use PRC session inside this function. But I cannot pass some information I capture from the PEP to the PRC session object somehow. 

More concrete example that I wan to achieve:

def pep_api_mod_avu_metadata_post(rule_args, callback, rei):
    user_name = str(session_vars.get_map(rei)["client_user"]["user_name"])
    items = dict(rule_args[2].items())
    obj_path = str(items["arg2"])
    obj_type = items["arg1"]
    metadata_operation = items["arg0"]
    metadata_attribute = items["arg3"]
    metadata_value = str(items["arg4"])
    metadata_unit = str(items.get("arg5"))
    if metadata_operation in ("rm", "rmw") and metadata_attribute.startswith("rules::") and user_name not in ("rods", "operator"):
        error_code_with_message(irods_errors.USER_NOT_ALLOWED_TO_EXEC_CMD,
                                        "ADMIN METADATA cannot be removed or altered!", callback)
    if obj_type == "-d" and metadata_attribute == "rules::soft_delete":
        #def change_permission(callback):
        #    new_permission = iRODSAccess('own', obj_path, user_name)
        #    with SWITCH() as session:
        #        session.acls.set(new_permission)
        callback.delayExec(f"<INST_NAME>irods_rule_engine_plugin-python-instance</INST_NAME><PLUSET>{metadata_value}{metadata_unit}</PLUSET>",
        "callback.change_permission_as_admin()", "")
        #"callback.change_permission()", "")
        #"callback.change_permission_as_admin('{obj_path}', '{user_name}')", "")

# def change_permission_as_admin(callback, obj_path, user_name):
def change_permission_as_admin(rule_args, callback, rei):
    #new_permission = iRODSAccess('own', obj_path, user_name)
    new_permission = iRODSAccess('own', '/u0137480/home/SWITCH/hello1.txt', 'mustafa')
    #new_permission = iRODSAccess('own', str(items["arg2"]), str(session_vars.get_map(rei)["client_user"]["user_name"]))
    with SWITCH() as session:
            session.acls.set(new_permission)

Commented lines represent options I tried and explained above. Could you show how I can call a PRC method in PEPs by the delay msi?

  

Thanks you.

[dmoore.renci]

hi Mustafa,

  I can answer this more to your point in a bit, but I was wondering what motivation there was for using PRC?  

 Is it because the delay rule running on the server has access to an .irods environment file that can be used to jump into an admin status?.

 If so,  maybe we could do this via a microservice -  like msiSetACL, but allowing the required privilege escalation in a controlled way?

 

-Daniel Moore / Applications Engineer / iRODS Consortium

[mustafa dikmen]

Hi Daniel,

Indeed that was the main reason why I need to use the PRC. I did already try the msi you pointed (msiSetACL) and I think it can not be used in a PEP because I am getting permission errors and someone who triggers PEPs should be an admin user according to its documentation. What do you mean by "but allowing the required privilege escalation in a controlled way"? As you might suggest I tried `msiSudoObjAclSet` developed by UU (btw I use it already for other projects/rules) and I am getting errors in this use case. You can pls see https://github.com/UtrechtUniversity/irods-sudo-microservices/issues/6

We have already been using the PRC in the server side for some of our projects owing to some other motivations. These are:
- Since some MSIs like msiDataObjPut() require client interaction, they cannot be executed by the delayExec.
- Act as a proxy user - such as creating a collection on behalf of someone.

Thank you.

Regards,

Mustafa

[dmoore.renci]

hello Mustafa,

   Sorry for the late answer.  You are correct in that the PRC session object must be instantiated in the delay rule in which it is used, and cannot be carried over from the PEP  call instance from which the delay rule was scheduled. 

   Things aren't so strict for arrays and other data structures involving Python primitive types, eg strings, integers, etc.  These could be serialized using Python's pickle module, or similar. I  could work up an example if you're interested.

- Daniel Moore - Applications Engineer / iRODS Consortium.

[dmoore.renci]

Mustafa ,

  Just curious, but is the system Python (ie the one being run by your python rule engine) at least Python 3.7 ?  This would make it easier to convey the value of the pickle-serialized values into the quoted rule code for the delay task because you'd be able to use format (f"..."-style) strings.

    - Daniel

[mustafa dikmen]

Hello Dainel,

Yes, we are interested in an example that will show how to pass information captured from the said PEP to the PRC function that will be executed by the delayExec in the same PEP.  Thanks.

We use the default python version of the operating system where our iRods server run. So it is Python 3.6.8 in AlmaLinux8.

But if I am not mistaken, I guess "f-strings" was introduced with the Python 3.6 which is what we have already. If you are implying something else with Python 3.7, I can say that for now we may probably not install a higher python version.

Have a nice weekend!
Mustafa

[dmoore.renci]

Mustafa,

  you are correct, i often forget that 3.6 has format strings and is a reasonable if not happy medium between Python's comfortable present and glorious future! : )

  I was able to achieve, I think, what you were after. Here is my /etc/irods/core.py :

import session_vars

def pep_api_mod_avu_metadata_post(rule_args, callback, rei):
    user_name = str(session_vars.get_map(rei)["client_user"]["user_name"])
    items = dict(rule_args[2].items())
    obj_path = str(items["arg2"])
    obj_type = items["arg1"]
    metadata_operation = items["arg0"]
    metadata_attribute = items["arg3"]
    metadata_value = str(items["arg4"])
    metadata_unit = str(items.get("arg5"))

    if metadata_attribute.startswith("rules::"):
        if metadata_operation in ("rm", "rmw") and user_name not in ("rods", "operator"):
            # original code:
            #       error_code_with_message(irods_errors.USER_NOT_ALLOWED_TO_EXEC_CMD,
            #                               "ADMIN METADATA cannot be removed or altered!", callback)
            callback.writeLine("serverLog","DEBUG - preventing rm,rmw by other than rodsadmin.")
            return;
        else:

            if obj_type == "-d" and metadata_attribute == "rules::soft_delete":

                callback.delayExec(f"<INST_NAME>irods_rule_engine_plugin-python-instance</INST_NAME><PLUSET>{metadata_value}{metadata_unit}</PLUSET>",

                                   f"callback.change_permission_as_admin({obj_path!r},{user_name!r})",
                                   "")

import irods.test.helpers
from irods.access import iRODSAccess

def SWITCH():
    return irods.test.helpers.make_session()

def change_permission_as_admin(rule_args, callback, rei):
    (obj_path, user_name) = rule_args

    new_permission = iRODSAccess('own', obj_path, user_name)

    with SWITCH() as session:
            session.acls.set(new_permission, admin = True)

If I then  iadmin set -M -d /tempZone/home/alice/adat rules::soft_delete 10 s

then I will see in iqstat:

id     name
10030 callback.change_permission_as_admin('/tempZone/home/alice/adat','rods')

[dmoore.renci]

hi again, Mustafa. 

Sorry, hit the 'post' button too soon there, but I think my example above is essentially correct and, in some variation, will work for you.

The format strings contain !r which force a Python representation to be printed instead of the raw __str__ value, and this is what's needed to create the Python code string for the delay rule. 

You don't need to use the pickle module as I'd suggested before , unless of course  you wanted to serialize more complicated data structures than strings.

If you did want to convey arrays, dicts, etc. of primitive Python types into the delayed code, they could be  injected into the f""-string Python text  in delayExec as: 

  pickle.loads( {pickled_var!r} )

. 

The variable pickled_var must then exist within the  Python PEP code that launches the delayed task, and must contain the result of pickling the data structure using the  pickle.dumps function.

Hope all of this helps,

   - Daniel

[dmoore.renci]

Of course, I could have done something this sort of thing in the Python code fed to delayExec:

  """ #...

      x = {some_data_structure!r}

      func ( {some_other_data_structure!r} )

  """

and that would've worked without the pickle stage, but then we aren't guaranteed that we wouldn't pick up a stray boost-python type (frequently passed into various PEPs via the rule framework) within such a data structure, and then we'd have to register that class to the pickle system. 

[mustafa dikmen]

Hi Daniel,

I really appreciate your help provided even on saturday! I now tested only your first solution - python representational string !r (repr) - since I don't need to use more complicated data structures than strings and it meets my use case/need. However, I will still test your other hints (pickle and copyreg) that are needed to be used for the complex data structures in order for me to use in case we need.

Best wishes,

Mustafa

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/0ef3b790-06d3-448d-a5f2-66a39a9039c1n%40googlegroups.com.

[dmoore.renci]

Mustafa,

  No problem, glad to follow up.  Will be interesting to hear what uses you find for this approach and for copyreg/pickle (a combination I've only played with so far.)

-Daniel