acPostProcForPut and replication

112 views
Skip to first unread message

tedgin...@gmail.com

unread,
Apr 6, 2021, 8:23:04 PM4/6/21
to iRODS-Chat

Hi guys.

I'm preparing to upgrade a iRODS 4.1.10 grid to 4.2.8. While testing our current set of rules, I noticed that replication triggers the acPostProcForPut PEP in 4.2.8 where it doesn't in 4.1.10. I need to handle replication differently than uploads. From inside a rule attached to this PEP, is there a good way to determine if it was triggered by replication?

Cheers,
Tony

tedgin...@gmail.com

unread,
Apr 6, 2021, 9:07:37 PM4/6/21
to iRODS-Chat
I see that this change is an open bug. See https://github.com/irods/irods/issues/4614.  Does anyone have a good workaround, or is there a dynamic PEP that can easily replace acPostProcForPut?

Alan King

unread,
Apr 7, 2021, 12:57:22 PM4/7/21
to irod...@googlegroups.com
Hi Tony,

I've written up an explanation as to when and how this behavior broke here: https://github.com/irods/irods/issues/4614#issuecomment-815068203

In it, I make a recommendation for using the dynamic policy enforcement points introduced in 4.2.0. The equivalent dynamic PEP for the static PEP acPostProcForPut is pep_api_data_obj_put_post

Hope that helps!

Alan

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/1388021b-3171-458e-a767-706921f52c88n%40googlegroups.com.


--
Alan King
Software Developer | iRODS Consortium

Tony Edgin

unread,
Apr 7, 2021, 5:04:11 PM4/7/21
to irod...@googlegroups.com
Hi Alan.

Very helpful! Thanks.

In your write-up, I see the expression L1desc[l1descInx].openType == CREATE_TYPE.  I notice that the dataObjInp_t.openType field is available to rules attached to the PEP pep_api_data_obj_put_post. Is the constant CREATE_TYPE or something like it available as well? I could use that to distinguish data object creates from data object overwrites.  If not, do you know if it will always be the value 1 (I learned that from experimentation.)

Cheers,
Tony

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CADnp3x7JyzjwpiSWqp4uxGQVg-nGYjn%2BOYSXFMjKxQiR0xAy5g%40mail.gmail.com.

Alan King

unread,
Apr 8, 2021, 1:48:30 PM4/8/21
to irod...@googlegroups.com
For your overwrite detection use case, you could stat the object in pep_api_data_obj_put_pre and capture whether or not it exists already before the put begins; then pep_data_obj_put_post can check if the object existed before the put was issued.

As for the openType suggestion, the DataObjInp struct has no member named openType: https://github.com/irods/irods/blob/9eb6c23df45cdedff8ee9c3af71a65d304635037/lib/api/include/dataObjInpOut.h#L65-L75

openType is part of the L1 descriptor which only exists as long as the data object is open. By the time the pep_data_obj_put_post is invoked, the L1 descriptor will have been freed. Was there another reason you might need the openType in the post-PEP implementation? Otherwise, I think the stat suggested above should fill your needs.

Additional info: APIs such as rcDataObjPut use other, server-side APIs in the server such as rsDataObjOpen and rsDataObjClose to accomplish their tasks. However, these "rs" calls do not trigger policy. We in the Consortium have discussed implementing rsDataObjPut such that the rsDataObjOpen and rsDataObjClose calls will trigger policy using irods::server_api_call. This would allow for another solution where you could implement pep_api_data_obj_open_post or pep_api_data_obj_close_pre to capture the openType from the L1 descriptor, for example.

Alan

Terrell Russell

unread,
Apr 8, 2021, 1:52:32 PM4/8/21
to irod...@googlegroups.com

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CADnp3x6_S-4eE6O7BrzCrQw-n_%3DmBGXs5w8U0PWi-xVdQrE22g%40mail.gmail.com.

Ilari Korhonen

unread,
Apr 8, 2021, 3:31:22 PM4/8/21
to iRODS-Chat
Hi,

I struggled with the very same thing. It seems that using acPostProcForPut is no longer viable option in 4.2.x.

The problem is that pep_api_data_obj_put_post doesn't get invoked by all the clients, like Jargon and Davrods,
so one needs to implement further PEPs to catch all put-like streaming operations into objects.


You need to load an API plugin, which puts in the required serializers for data types not being serialized in 4.2.8,
those are now merged upstream, so will become available in 4.2.9 out-of-the-box.


Then you need microservices to map the object path to its open L1 descriptor index.


That one also is merged into upstream and should be released with 4.2.9 (different name though).

Let me know if you need to know more. You know how to reach me :)

- ilari

Ilari Korhonen

unread,
Apr 8, 2021, 3:33:51 PM4/8/21
to iRODS-Chat
Also,

Reading in case one is interested:

Tony Edgin

unread,
Apr 8, 2021, 5:09:33 PM4/8/21
to irod...@googlegroups.com
Hi Alan and Ilari.

Ilari, thanks for the additional information.  I'll take a look.

Alan, when I list the key-value pairs of the serialized dataObjInp_t structure from within a rule attached to pep_api_data_obj_put_post, openType is present.

pep_api_data_obj_put_post(*INSTANCE, *COMM, *DATA_OBJ_INP, *_, *_) {
  writeLine('serverLog', 'pep_api_data_obj_put_post called');
  foreach(*kv in *DATA_OBJ_INP) {
    writeLine('serverLog', '\*DATA_OBJ_INP.*kv = ' ++ *DATA_OBJ_INP.'*kv');
  }
}

Here's the logged output.

Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = pep_api_data_obj_put_post called
Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = *DATA_OBJ_INP.create_mode = 0
Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = *DATA_OBJ_INP.dataIncluded =
Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = *DATA_OBJ_INP.dataType = generic
Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = *DATA_OBJ_INP.data_size = 6536
Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = *DATA_OBJ_INP.noOpenFlag =
Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = *DATA_OBJ_INP.num_threads = 0
Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = *DATA_OBJ_INP.obj_path = /cyverse/home/tedgin/count-sessions
Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = *DATA_OBJ_INP.offset = 0
Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = *DATA_OBJ_INP.openType = 1
Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = *DATA_OBJ_INP.open_flags = 2
Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = *DATA_OBJ_INP.opr_type = 1
Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = *DATA_OBJ_INP.resc_hier = mainIngestRes;mainIngest
Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = *DATA_OBJ_INP.selObjType = dataObj
Apr  8 13:50:18 pid:50304 NOTICE: writeLine: inString = *DATA_OBJ_INP.translatedPath =

I guess I can't rely on openType always being there. I'll use the suggestion Terrell and you made.

Cheers,
Tony

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to a topic in the Google Groups "iRODS-Chat" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/irod-chat/iRUXFpj5fnI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/0d494579-74a7-4d4b-b72b-4c8edc652cfen%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages