How to find out which dynamic peps are triggered by which action?

[joris luijsterburg]

Hi all,

I am relatively new to iRODS, and I was looking at the dynamic pep's as described here: https://docs.irods.org/4.3.0/plugins/dynamic_policy_enforcement_points/

It gives me a big list of possible PEP's I could use. I was playing around with it, but find it hard to determine exactly when a specific PEP is triggered. For example, I tried the pep_api_data_obj_put_pre, put an msiWriteRodsLog in it, and then it wrote a line in the log when I did an iput. Success!

After that I tried to execute an  msiWriteRodsLog on pep_resource_getfs_freespace_pre, pep_resource_read_pre and pep_api_obj_stat_pre, but I was unable to generate logging there. So either I misunderstood how the signature of these PEP's works(very well possible), or I just executed the wrong icommands and didn't trigger them.

Although the naming for pep_api_data_obj_put_pre does suggest that it might get triggered by an iput, I can't find in the documentation that it does. This makes it harder for me to debug, and to see if I have to change something in my rule definition, or if I just triggered the wrong PEP. Is this documentation available somewhere?

Regards,

Joris

[Kory Draughn]

Hi Joris,

There is no documentation that defines mappings between APIs and PEPs. Hence this new issue we've created:

• https://github.com/irods/irods/issues/6679
  

We'll work on providing documentation for this, but until then, the best source for seeing which PEP an API will trigger is:

• https://github.com/irods/irods/blob/6e0cc93ba0b87763755c2a0a51f56085b4d85514/lib/api/include/irods/apiTable.hpp#L303

That file defines API definitions for all APIs except API plugins.

Below is the API definition for the PUT API. I'll highlight the important information regarding your question.

{
    DATA_OBJ_PUT_AN, RODS_API_VERSION, REMOTE_USER_AUTH, REMOTE_USER_AUTH,
    "DataObjInp_PI", 1,  "PortalOprOut_PI", 0,
    boost::any(std::function<int(rsComm_t*,dataObjInp_t*,bytesBuf_t*,portalOprOut_t**)>(RS_DATA_OBJ_PUT)),
    "api_data_obj_put", clearDataObjInp,
    (funcPtr)CALL_DATAOBJINP_BYTESBUFINP_PORTALOPROUT
},

Each entry will contain a string starting with api_ (See the 5th line in the example above)

• This string is part of the PEP signature that will be invoked as a result of calling the function, RS_DATA_OBJ_PUT (explained later)
  
• Upon invoking this API, the server will generate the full PEP name using this string (e.g. <prefix> + "api_data_obj_put" + <class>)
  
• <prefix> will be "pep_" 
  
• <class> will be expanded to pre, post, except, or finally
  

Now, RS_DATA_OBJ_PUT is a macro that expands to a function or nothing. If you search the file for that macro, you'll come across the following line:

• #define RS_DATA_OBJ_PUT NULLPTR_FOR_CLIENT_TABLE(rsDataObjPut)
  

In parentheses, notice the rsDataObjPut symbol. That is the server-side API function that will be run upon invoking rcDataObjPut.

iput calls rcDataObjPut which will cause the server to lookup the API definition for the PUT API. With the API definition in hand, the server knows which server-side function and PEP to invoke.

Hope that helps,

Kory Draughn

Chief Technologist

iRODS Consortium

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/d0686b18-6cc0-438e-bd1c-31f62430132fn%40googlegroups.com.

[joris luijsterburg]

Kory,

Thanks for the quick answer! It is not completely clear yet though. How do I get from iput to rcDataObjPut? In here I can see a reference to pututil, but there I can't find rcDataObjPut either. 

In the meantime, what I was actually looking for was a way to keep my resource usage up to date. There are examples that use acPostProcForParallelTransferReceived, acPostProcForDataCopyReceived, acPostProcForPut and acPostProcForDelete, but I think you want to get rid of these in the future, hence I was lookin for alternatives.

Or, a PEP that is triggered whenever I try to get resourceinformation, or scheduling a rule every X minutes could also do the trick (hence I was guessing to use pep_resource_getfs_freespace_pre, pep_resource_read_pre and pep_api_obj_stat_pre)

Best regards,

Joris

[Kory Draughn]

Joris,

Many of the icommands invoke functions such as putUtil() (or lsUtil() for ils). The logic for these util functions live in the irods/irods repository. In these functions you'll find calls to APIs such as rcDataObjPut().

As for the resource usage question ... Can you elaborate more on what you want to happen?

I want to make sure I understand your use-case before providing an answer.

And yes, you are correct in avoiding the static PEPs (i.e. ac* rules).

Kory Draughn

Chief Technologist

iRODS Consortium

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/a6c7b305-e830-460f-a34e-ebfe5cb0c5b6n%40googlegroups.com.

[joris luijsterburg]

Kory, 

Thanks again, this does help me! 

I see I was looking at the wrong pututilfile( .h instead of .cpp, not a C++ programmer here). In there I indeed find the link!  This does give some generic guideline on how to find which PEP's are triggered. 

E.g. if I execute an ilsresc, I can see it will use rcGenQuery, which corresponds to  RS_GEN_QUERY and  pep_api_gen_query_(pre,post,except,finally). (Naming suggests that a lot more functions apart from ilsresc use this)

Creating a writerodslog in pep_api_gen_query_pre should then result in a logline when calling ilsresc. 

It does(success!), but apparently there is also some other process calling rsgenquery very often, so best to proceed cautiously here;).

The usecase is monitoring the storage limits on your resource, and potentially warn if it exceeds x%, or even block files from being created. For this a requirement is probably that the free space entry is accurately filled in the resource, which you can probably do in various ways, but since I am still learning, I also just wanted to test out a couple of PEP's to see how it ties together and how we can service the kind of requests that we will be getting from our research teams, since this will not be the only thing we want to automate.

Best regards,

Joris

[Kory Draughn]

Yes, GenQuery is used frequently so it's likely not what you want.

Regarding your resource question, the following documentation contains information about tracking resource usage:

• https://docs.irods.org/4.3.0/plugins/composable_resources/#unixfilesystem
  

You may also be interested in the Logical Quotas Rule Engine Plugin. It tracks the number of data objects in a collection tree and their sizes.

• https://github.com/irods/irods_rule_engine_plugin_logical_quotas/tree/4.3.0.0
  

Finally, there's the built in resource quotas. The following should be helpful:

• https://github.com/irods/irods_docs/blob/main/docs/administrators/policy_cookbook.md#simulating-user-quotas (not yet published)
  
• https://docs.irods.org/4.3.0/icommands/administrator/#sgq
  
• https://docs.irods.org/4.3.0/icommands/administrator/#lq
• https://docs.irods.org/4.3.0/icommands/administrator/#cu

Kory Draughn

Chief Technologist

iRODS Consortium

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/17582060-c0e2-4be6-90e7-e2973fb73d35n%40googlegroups.com.

[joris luijsterburg]

As a second answer to my own question: 

I tried the pepview-tool of cyverse, and that also helped me find the pep's I needed to build logic in. It is explained here as a step in creating an automatic trash management policy:

https://www.youtube.com/watch?v=hAaJWdXg69M

https://irods.org/uploads/2023/Gola-CyVerse-Arizona-Using_iRODS_Rules_to_Automate_Trash_Management_Policy-slides.pdf