Silly question about acPostProcForPut()

[Samuel Viscapi]

Dear all,

For quite a long time now we've used acPostProcForPut() in our local rules file to trigger workflows at CINES.

So far, $userNameClient was used to discriminate between the possible actions:

acPostProcForPut{

ON($userNameClient != "rods")
{

        transferFinished( $objPath )    ;
}

}

acPostProcForPut{}

I wanted to experiment something new for a change, and toy with the --metadata option of the iput command:

On the client: iput --metadata "ARCHIVING;True;Bool" myfile

And on the server:

acPostProcForPut{

ON( *AName == "ARCHIVING" )
{

        transferFinished( $objPath )    ;
}

}

acPostProcForPut{}

But for some reason it doesn't work...

I can see the metadata is there with imeta ls -d myfile but it doesn't trigger anything...

What am I doing wrong ?

Thanks for your help :)

Best regards (and take care),

Samuel, from CINES (Montpellier, France)

https://www.cines.fr/en/

[Terrell Russell]

Hi Samuel,

We're looking to write a set of 'best practices' around this transition from the legacy static policy enforcement points (PEPs) to the dynamic PEPs... but for now, I'll try to answer here...

The best candidate for replacing acPostProcForPut() is with pep_api_data_obj_put_post()

Its signature is pep_api_data_obj_put_post(*INSTANCE_NAME, *COMM, *DATAOBJINP, *BUFFER, *PORTAL_OPR_OUT).

Please note that the pep_api_data_obj_put_post() will not fire for clients that are using open/write/close semantics, rather than put semantics.  This currently includes some of Jargon, and the Python iRODS Client.  We are working on new event handlers for the iRODS server that will make it easier to understand this difference and work with all clients equally.

As a separate note, you should consider using the if() notation, rather than the ON() notation for your logic at the beginning of the rule.

  See https://github.com/irods/irods/issues/3897 and https://github.com/irods/irods/issues/4614

Terrell

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/718ad587-4026-4078-b951-ea92e596bdac%40googlegroups.com.

[Samuel Viscapi]

Hi Terrell,

Thanks for the quick reply, that's really appreciated.

Looks like I have a bit of reading to do now ;)

https://docs.irods.org/4.2.7/plugins/dynamic_policy_enforcement_points/

But that's a good thing though, an update to our local rules file is long overdue.

Cheers,

Samuel

To unsubscribe from this group and stop receiving emails from it, send an email to irod...@googlegroups.com.

[Samuel Viscapi]

Hi Terrell,

I'm sorry to bother you again with this :(

Based on a recent forum post and the documentation about dynamic PEPs, I decided to give pep_api_data_obj_put_post a try:

https://groups.google.com/forum/#!msg/irod-chat/bcQZS2PFeps/X2vsveybBAAJ

https://docs.irods.org/4.2.7/plugins/dynamic_policy_enforcement_points/

pep_api_data_obj_put_post(*INSTANCE_NAME, *COMM, *DATAOBJINP, *BUFFER, *PORTAL_OPR_OUT) {

on(*COMM.user_user_name != "rods")

{

        transferFinished( *DATAOBJINP.obj_path )    ;
}

}

pep_api_data_obj_put_post{}

It seems to be working almost correctly :) The workflow crashes further down but I think that's unrelated to dynamic PEPs.

Now, where could I find all the possible values of a variable, say *COMM ? And what if I want to use metadata to discriminate between actions ?

In that case, I suppose *DATAOBJINP comes into play, but again, which values should I test ?

if(*DATAOBJINP.metadata == "ARCHIVING") ?

I'm really sorry if my question sounds utterly stupid :)

Cheers, Samuel

[Samuel Viscapi]

Hi Terrell,

I think I'm starting to understand how it works :)

pep_api_data_obj_put_post(*INSTANCE_NAME, *COMM, *DATAOBJINP, *BUFFER, *PORTAL_OPR_OUT) {

on(*COMM.user_user_name != "rods")
{

        writeLine("serverLog" , "COMM is: *COMM");
        *usr = *COMM.user_user_name ;
        writeLine("serverLog" , "User is: *usr");

}

}

pep_api_data_obj_put_post{}

And then in the logs :

Mar 26 12:13:14 pid:14102 NOTICE: writeLine: inString = User is: viscapi

Cheers, Samuel

[Samuel Viscapi]

Hi again :)

There's still a thing I don't quite understand...

How do one call a rule from a dynamic PEP ?

For example:

pep_api_data_obj_put_post(*INSTANCE_NAME, *COMM, *DATAOBJINP, *BUFFER, *PORTAL_OPR_OUT) {

on(*COMM.user_user_name != "rods")
{

        *obj_path = *DATAOBJINP.obj_path ;
        writeLine("serverLog" , "Path is: *obj_path");

        *usr = *COMM.user_user_name ;
        writeLine("serverLog" , "User is: *usr");

        transferFinished( *obj_path );
}

}

transferFinished( *obj_path )

{
        msiModAVUMetadata( "-d" , *obj_path , "add" , "OWNER" , *usr , "string" ) ;

}

But it fails with the following error message:

remote addresses: 10.6.10.18 ERROR: putUtil: put error for /CINESDEV/home/viscapi/test31, status = -130000 status = -130000 SYS_INVALID_INPUT_PARAM

Mar 26 14:19:30 pid:15576 DEBUG: execMicroService3: error when executing microservice
line 19, col 1, rule base test
        msiModAVUMetadata( "-d" , *obj_path , "add" , "OWNER" , *usr , "string" ) ;

I guess my logic is wrong somewhere, but I honestly can't see where...

Cheers, Samuel

[Coposky, Jason Matthew]

Samuel,

The variable *usr does not appear to be in scope with your transferFinished() function?  Perhaps the error is just misleading.

Thanks,

------
Jason Coposky
Executive Director, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
w: (919)445-9675
m: (919)522-0517
jas...@renci.org

---

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Samuel Viscapi <svis...@gmx.com>
Sent: Thursday, March 26, 2020 9:28 AM
To: iRODS-Chat <irod...@googlegroups.com>
Subject: Re: [iROD-Chat:18805] Silly question about acPostProcForPut()

 

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/e3e116ac-2688-4362-8cbf-15898a7301ab%40googlegroups.com.

[Samuel Viscapi]

Hi Jason,

You are right, the *usr variable doesn't seem to be declared within transferFinished()

I'll have a look at my rules again :)

Thanks,

Samuel

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/e3e116ac-2688-4362-8cbf-15898a7301ab%40googlegroups.com.

[Samuel Viscapi]

Hi Jason,

My updated rules file for dynamic PEPs is below.

What am I trying to achieve with it: as part of an ongoing POC, one of our users (not rods) submits a new file to iRODS. That file is to be archived in our new long-term preservation platform, Vitam (http://www.programmevitam.fr/pages/english/pres_english/). That file has to be converted to SEDA 2.1 archiving standard prior to being allowed in the archiving platform. For that purpose we're using "Resip", wich is part of the "sedatools" suite (https://github.com/ProgrammeVitam/sedatools) through msiExecCmd (see archive.sh, also provided below). First we retrieve (iget) the file from the user as rods, then we convert the file to a SEDA compliant archive, and finally we submit (iput) as rods user that archive back to iRODS. When the original submission from the user has been successfully converted, it can safely be removed from iRODS. When everything is OK, we try to POST the SEDA 2.1 archive to Vitam API using msiCurlPost (work in progress, see my related post post here: https://groups.google.com/forum/#!topic/irod-chat/AXJOFwWoAqc). Vitam successfully processes that archive and replies with an HTTP code 200. The SEDA 2.1 archive can safely be removed from iRODS.

What do you think of it ?

Opinions, pieces of advice and constructive criticism welcome :)

N.B.: a large French natural history museum could very well send us 700TB of data in the near future using this proposed workflow, so I'm really looking forward to reading your comments.

Cheers, Samuel

pep_api_data_obj_put_post(*INSTANCE_NAME, *COMM, *DATAOBJINP, *BUFFER, *PORTAL_OPR_OUT) {

if(*COMM.user_user_name != "rods")
{
        *obj_path = *DATAOBJINP.obj_path ;
        *user = *COMM.user_user_name ;
        writeLine("serverLog" , "*user stored object *obj_path");
                *cmd = "archive.sh" ;
        *par = *obj_path ;
        msiSetACL( "default" , "read" , "rods" , *obj_path );
        writeLine("serverLog" , "Sending *obj_path to SEDA 2.1 generator");
        msiExecCmd( *cmd , *par , "null" , "null" , "null" , *Result );
        writeLine("serverLog" , "SEDA 2.1 generation is OK");
        msiDataObjUnlink( "objPath=*obj_path++++forceFlag=" , *Status );
        writeLine("serverLog" , "Removed *obj_path from the collection");

}

if(*COMM.user_user_name == "rods")
{
        *obj_path = *DATAOBJINP.obj_path ;
        *user = *COMM.user_user_name ;
        writeLine("serverLog" , "*user stored object *obj_path");
        msiModAVUMetadata( "-d" , *obj_path , "add" , "ARCHIVED" , "False" , "Bool" );
        writeLine("serverLog" , "Setting ARCHIVED metadata to False on *obj_path");
        writeLine("serverLog" , "Posting *obj_path to Vitam");
        *url="https://some.ip:8443/ingest-external/v1/ingests"
        *data="*obj_path"
        *postFields."data" = *data
        msiCurlPost(*url , *postFields , *response);
        writeLine("serverLog" , "server response: "++*response);

}

}

And for the sake of completeness, the contents of archive.sh:

#!/bin/bash

# cd to workdir

if [  ! -d /tmp/herbadrop ]; then
mkdir /tmp/herbadrop
fi

cd /tmp

# iget the file

/bin/iget $1 /tmp/herbadrop/$i
my_file=`echo $1 | cut -d "/" -f 5`
my_archive="$my_file.zip"

# SEDA 2.1 conversion

java -jar /opt/test-sedatools/sedatools/resip/target/resip-2.3.0-SNAPSHOT-shaded.jar -c /var/lib/irods/msiExecCmd_bin/ExportContext.config -d /tmp/herbadrop -g $my_archive -i -w /tmp/ -x

# Submission of archive with write rights to API user

/bin/iput --acl 'write vitamapi;' -R access $my_archive

[Terrell Russell]

As a follow up - this successful POC was presented by Viscapi at UGM 2020.

Lightning Talk - Using iRODS as an entry point to VITAM for long-term
data preservation
Samuel Viscapi - CINES

Slides:
https://irods.org/uploads/2020/Viscapi-CINES-iRODS_as_Entry_Point_to_VITAM_for_Long_Term_Data_Preservation-slides.pdf

Video:
https://www.youtube.com/watch?v=jQVdfY0WLNI&list=PL29FhEN41mZPxeQLdPBYTq8Ze9yEDK0Tz

Terrell

> To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/946f23c9-5564-4afc-9062-323fa8785285%40googlegroups.com.