Enforce file inmutable read-only

[Andreas Kuntzagk]

Hi,

 

for an archive like iRODS setup we want to disable all file changes (including deleting, renaming, metadata change) except by special admin account.

Even the user who uploaded the file should not be able to change it afterwards.

Adding more files to Collections should still be possible.

My tests seem to indicate that this is not possible with ACLs only and we probably to write a rule for data-changing PEPs?

How do I find out which PEPs this are?

Looking at the list https://docs.irods.org/4.2.4/plugins/dynamic_policy_enforcement_points/ what is the difference between pep_api_file_* and pep_api_data_* ?

 

Regards, Andreas

[dmoore.renci]

Andreas,

   If you wish to limit the scope of data writes to creating new data objects in a collection and/or on a given resource, but stop updates to existing data objects, you may want to

use :

pep_resource_resolve_hierarchy_pre

in a manner similar to what is displayed in this storage balancing example (modulo your choice of rule language) :

  https://slides.com/irods/ugm2018-rule-engine-plugins#/15

The code you end up with might even be a tad simpler, as you'll have no use for the "percent full" calculations. Basically you'll need to set the "write=" value in the result string to 0.0 in the _pre PEP when the operation string is not  "CREATE" (ie, it's a write to an object).

Then restore the previous value in the _post PEP.  The arguments passed into the PEP should allow you to filter on collection name or resource.

-Dan Moore / Applications Engineer ( iRODS Consortium)

[Smeele, A.P.M. (Ton)]

Hi Andreas,

 

Indeed ACL's are not enough to accomplish this.  The creator of the data object can reestablish acces to the object even if you set "null" access.  Currently there is no way to change creatorship of an object.  

As a practice we have the rodsadmin user account make a copy of the relevant data object(s) so that the rodsadmin user is the creator. The resulting object is ACL protected and hence inmutable to regular users.

 

Kind regards,

 

Ton

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andreas Kuntzagk]

Hi Ton,

 

interesting approach. You do that by a rule? Could you maybe share that?

 

Regards, Andreas

[Andreas Kuntzagk]

Hi Dan,

 

thanks for the hint. Can you please explain why this is the PEP to use. I would have never guessed that “Resolve Resource Hierarchy” is a good point to go for.

 

Regards, Andreas

--

[Andreas Kuntzagk]

Hi,

 

one more question: does that also fire on “imeta” changes?

 

Regards, Andreas

[Daniel Moore]

Hi Andreas: are you receptive to Ton's idea of simply making an admin-owned immutable copy? Depending on your application it could be more suitable and easier to maintain.

Otherwise yes I could possibly get something worked out today if you're still interested in the resource oriented approach. Which language do you favor (python, or the native irods rule language) for writing the rules?

Dan

[Smeele, A.P.M. (Ton)]

Hi Andreas,

 

In our case it is part of a workflow, see pointer here: https://github.com/UtrechtUniversity/irods-ruleset-research/blob/release-1.3/iiVault.r  procedure name is iiCopyFolderToVault.   Note that this executes as an irods admin job separate from the regular user context.

 

Cheers,

 

Ton

[Andreas Kuntzagk]

Hi Ton,

 

I’m not sure I understand that rule. This is called manually for moving data into iRODS? Or is that triggered by a PEP?

What part is the copy-by-rodsadmin you mentioned?

[Andreas Kuntzagk]

Hi Dan,

 

it’s not decided yet since I did not really understand Ton’s approach. In the meantime I experiment (for the first time) with python rules.

(I would much prefer python over the irods rule language)

Currently I have problems to setup the python rules at all.

 

In my server_config_json I have:

 

        "rule_engines": [

            {

                "instance_name" : "irods_rule_engine_plugin-python-instance",

                 "plugin_name" : "irods_rule_engine_plugin-python",

                 "plugin_specific_configuration" : {

"re_rulebase_set": ["core", "avoid_file_changes"] }

            },

 

In /etc/irods/core.py I have

 

def pep_resource_resolve_hierarchy_pre(rule_args, callback, rei):

    return avoid_file_changes.pep_resource_resolve_hierarchy_pre(rule_args, callback, rei)

 

In /etc/irods/avoid_file_changes.py

 

def pep_resource_resolve_hierarchy_pre(rule_args, callback, rei):

    callback.writeLine('serverLog', 'pep_resource_resolve_hierarchy_pre' + rule_args)

    if rule_args[3] == 'CREATE':

        rule_args[2] = 'read=1.0;write=0.0'

 

An iput commands triggers this in thr rodsLog:

 

Nov 21 17:04:51 pid:7184 remote addresses: 10.205.112.133, 127.0.0.1 ERROR: caught python exception: Traceback (most recent call last):

  File "/etc/irods/core.py", line 2, in pep_resource_resolve_hierarchy_pre

    return avoid_file_changes.pep_resource_resolve_hierarchy_pre(rule_args, callback, rei)

NameError: global name 'avoid_file_changes' is not defined

 

How do I correctly setup Python rules?

 

Regards, Andreas

[Terrell Russell]

You do not need 

"re_rulebase_set": ["core", "avoid_file_changes"]

The python rule engine plugin only loads core.py.    From core.py, you can import whatever you want.

Terrell

[Ton Smeele]

Hi Andreas,

In your situation you could do:

1) user annotates the object/collection with an arbitrary AVU to signal that this object needs to become read only.

2) on a regular basis the rodsadmin user runs a rule using irule to scan the zone for objects with the AVU flag. When found, just use a microservice e.g. msiDataObjCopy to copy object "X" to "Y". Y is now owned by the irods admin and hence cannot be edited/updated by the regular user.

Cheers,

Ton

-----Original Message-----

Date: Wed, 21 Nov 2018 15:49:40 +0000

Subject: [iROD-Chat:17782] AW: Enforce file inmutable read-only

To: irod...@googlegroups.com <irod...@googlegroups.com>

Reply-to: irod...@googlegroups.com

From: Andreas Kuntzagk <andreas.ku...@bayer.com>

--

[dmoore.renci]

hello Andreas -

 

To answer your previous question of

why the pep_resource_resolve_hierarchy_XXX calls are the appropriate choice, I believe it's because controlling object writes it is often a

choice made on  a per-resource basis, although the  information passed into this pep is comprehensive enough to be able to find out the

relevant user, collection, and name of the data object involved, even when the destination resource is not the main concern.

  I'm working on an example for the approach I suggested,  but am still in beginning stages. I should have more information (possibly

code as well) in a while.  If you use Ton's approach, I'll be just as glad because I'll have gleaned an education on two different fronts!

  As for the mechanics of importing a rule defined in a different python module, I have done it on more than one occasion using this approach:

__core.py __

from My_other_module import *

# ... pep_XXX_yyy gets put into the core.py's namespace and is callable as a rule function

__ My_other_module.py __

def pep_XXX_yyy( rule_args,  , rei):

   #...define body of PEP here.

Which will work as long as My_other_module.py is in /etc/irods or reachable from the Python interpreter that runs core.py.  If necessary, you can even

do the following in core.py before the import occurs:

   sys.path.insert(0, "/my/other/python/path")

Be in contact soon. In the meantime,  happy rule-base hacking!

Best regards,

  - Dan

[Jean-Yves Nief]

hello Andreas,

            have you considered using msiDeleteDisallowed micro-service
in acDataDeletePolicy ?
cheers,
JY

> --
> --
> "iRODS: the Integrated Rule-Oriented Data-management System; A
> community driven, open source, data grid software solution"
> https://www.irods.org
>
> iROD-Chat: http://groups.google.com/group/iROD-Chat
>
> ---
> You received this message because you are subscribed to the Google
> Groups "iRODS-Chat" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to irod-chat+...@googlegroups.com

> <mailto:irod-chat+...@googlegroups.com>.

[Andreas Kuntzagk]

Hi Jean,

no, I did not look into this. But this would only prevent delete operations not overwrite or metadata change, right?

Regards, Andreas

-----Ursprüngliche Nachricht-----
Von: irod...@googlegroups.com [mailto:irod...@googlegroups.com] Im Auftrag von Jean-Yves Nief
Gesendet: Donnerstag, 22. November 2018 12:02
An: irod...@googlegroups.com; Andreas Kuntzagk
Betreff: Re: [iROD-Chat:17787] Enforce file inmutable read-only

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.

[Andreas Kuntzagk]

Hi Ton,

 

would it be possible to “catch” the put operation and replace it with a put by the rodsadmin user?

Like a rule for acPreProcForPut calling msiDataObjCreate.

 

Regards, Andreas

 

Von: irod...@googlegroups.com [mailto:irod...@googlegroups.com] Im Auftrag von Ton Smeele

Gesendet: Mittwoch, 21. November 2018 19:55
An: irod...@googlegroups.com

[Andreas Kuntzagk]

Oops, looks like there is no acPreProcForPut , only acPostProcForPut

 

Regards, Andreas

 

Von: irod...@googlegroups.com [mailto:irod...@googlegroups.com] Im Auftrag von Andreas Kuntzagk

Gesendet: Donnerstag, 22. November 2018 15:30
An: irod...@googlegroups.com

[Jason Coposky]

We provide dynamic policy enforcement points to address this limitation:

https://docs.irods.org/4.2.4/plugins/dynamic_policy_enforcement_points/

------

Jason Coposky
Executive Director, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
w: (919)445-9675

m: (919)522-0517
jas...@renci.org
linkedin

twitter

irods.org

[Andreas Kuntzagk]

Hi,

 

so this would probably be pep_api_data_obj_put_pre.

But I still don’t see if and how I can change the user for which this put is done.

 

Regards, Andreas

[Andreas Kuntzagk]

Hi,

 

Today I investigated copying the file in an acPostProcForPut rule. Seems that this rule is executed with the user doing the “iput” and so the copied file is also owned by the user and not the rodsadmin.

 

Btw. on the net I found different examples for msiDataObjCopy with 2 parameters. When I use this I get an ACTION_ARG_COUNT_MISMATCH, seems that 4 parameters are needed despite the documentation saying that 2 are optional.

[dmoore.renci]

hello Andreas, (and Ton?),

I wonder if we could use following iRODS sudo-like capability for the problem at hand:

  https://github.com/UtrechtUniversity/irods-sudo-microservices

I'd like to look into how applicable it is to implementing your immutable data objects,  Andreas. My feeling is, it should be....

Ton and/or Chris Smeele would  know more about the microservice's appropriate uses than I would, of course.... But

presumably, it would work this way:

The post-processing PEP for a"put" or "create" operation on a qualifying data object would

  (1) downgrade ACL's for the user  creating the object from "own" to "read"; and,

  (2) add an ownership ACL for  the admin.

  

-Dan

On Friday, November 23, 2018 at 7:29:37 AM UTC-5, Andreas Kuntzagk wrote:

Hi,

 

so this would probably be pep_api_data_obj_put_pre.

But I still don’t see if and how I can change the user for which this put is done.

 

Regards, Andreas

 

Von: irod...@googlegroups.com [mailto:irod...@googlegroups.com] Im Auftrag von Jason Coposky
Gesendet: Donnerstag, 22. November 2018 17:21
An: irod...@googlegroups.com
Betreff: Re: AW: [iROD-Chat:17791] AW: Enforce file inmutable read-only

 

We provide dynamic policy enforcement points to address this limitation:

 

https://docs.irods.org/4.2.4/plugins/dynamic_policy_enforcement_points/

------

Jason Coposky
Executive Director, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
w: (919)445-9675

m: (919)522-0517
jas...@renci.org
linkedin

twitter

irods.org

On Nov 22, 2018, at 10:32 AM:

 

;;; deleted ;;;

[Smeele, A.P.M. (Ton)]

Hi Dan,

 

Our Sudo service microservices can establish own *rights* on an object yet we have not (yet) created a microservice that transfers *ownership* to e.g. the rodsadmin. 

 

Kind regards,

 

Ton

--

[Andreas Kuntzagk]

Hi Dan,

 

the topic is not to set the “own” ACL but the “ownership” (as shown by “irods –l” ) because this still allows to delete files without “own” ACL.

 

Regards, Andreas

--

[Andreas Kuntzagk]

Hi,

 

just for the record: I did not find a good solution using dynamic enforcement points.

So I went for a rule which will be run by cronjob of the rodsadmin-user that copies the objects.

Here is my rule that finally looks quite simple (thanks to todays discussion about recursive changes to collections)

 

$ cat make_immutable2.r

make_immutable{

   foreach (*f in select COLL_NAME,DATA_NAME where COLL_NAME = '*RootColl'|| like '*RootColl/%' and DATA_OWNER_NAME != '*RodsUser') {

         *logicalDataPath = *f.COLL_NAME++'/'++*f.DATA_NAME;

         *logicalDataPathCopy = *logicalDataPath++'.temp_copy';

         writeLine("stdout","reowning *logicalDataPath");

         msiDataObjCopy(*logicalDataPath,*logicalDataPathCopy,"forceFlag=",*Status);

         msiDataObjUnlink("objPath=*logicalDataPath++++forceFlag=",*Status);

         msiDataObjRename(*logicalDataPathCopy,*logicalDataPath,"0",*Status);

    }

}

 

INPUT *RootColl="/mytestfolder", *RodsUser="rods"

OUTPUT ruleExecOut

 

Any comments?

 

Regards, Andreas

--

[Terrell Russell]

Looks pretty good to me.

Another wrinkle to consider is that with the msiDataObjUnlink, *all* the replicas of that data object will be removed from the catalog and vault(s).

If you're working with multiple replicas, be careful about replicating the new copy into the desired multiple locations...

Terrell

[Andreas Kuntzagk]

Hi,

 

Until now there is no replication but in the future there will. I will keep that in mind.

I guess the msiDataObjCopy copies to the default resource if not told otherwise?

 

One more thing to consider: I also have to manually copy the metadata. What would be the cleanest way to do that?

 

Regards, Andreas

 

Von: irod-...@googlegroups.com [mailto:irod...@googlegroups.com] Im Auftrag von Terrell Russell

Gesendet: Mittwoch, 28. November 2018 14:57
An: irod...@googlegroups.com

[Terrell Russell]

I believe it copies to the same as from where it reads (avoiding any network traffic)... as it is a ‘logical’ operation (at the data object level rather than the replica level).

Regarding the metadata, yes you’d have to copy it... probably most cleanly within a GenQuery result loop, writing each result to the new object (before removing the original object).

Terrell

[Adil Hasan]

Hello folks,
regarding copying the data as the irods admin or manager account: you
probably want to keep the information on who uploaded the data and from
where as additional metadata. I guess you could collect the info from
the irods log file.
hth
adil

[Andreas Kuntzagk]

In the hope that it will be helpful for others: this is the rule now to also copy the metadata (without units since we don’t use them, see my other email)

 

reown{

   foreach (*f in select COLL_NAME,DATA_NAME where COLL_NAME = '*RootColl'|| like '*RootColl/%' and DATA_OWNER_NAME != '*RodsUser') {

         *Coll = *f.COLL_NAME;

         *Data = *f.DATA_NAME;

         *Copy = *Data++'.temp_copy';

         *logicalDataPath = *Coll++'/'++*Data;

         *logicalDataPathCopy = *logicalDataPath++'.temp_copy';

 

         writeLine("stdout","reowning *logicalDataPath");

         msiDataObjCopy(*logicalDataPath,*logicalDataPathCopy,"forceFlag=",*Status);

         if (*Status != 0) then {

             failmsg(-1, "msiDataObjCopy did not return 0 in reown rule");

         }

         writeLine("stdout","msiDataObjCopy status *Status");

         copyMetadata(*Data,*Copy, *Coll);

         msiDataObjUnlink("objPath=*logicalDataPath++++forceFlag=",*Status);

         msiDataObjRename(*logicalDataPathCopy,*logicalDataPath,"0",*Status);

    }

}

 

copyMetadata(*Data,*Copy,*Coll) {

   # copy all metadata (attribute-value pairs, units not copied since not supported by microservice and not used in BMDA

   foreach ( *f in select META_DATA_ATTR_NAME,META_DATA_ATTR_VALUE where COLL_NAME = '*Coll' and DATA_NAME = '*Data') {

       *att_name=*f.META_DATA_ATTR_NAME;

       *att_value=*f.META_DATA_ATTR_VALUE;

       msiString2KeyValPair("*att_name=*att_value",*att)

       msiSetKeyValuePairsToObj(*att,*Coll++'/'++*Copy, "-d");

   }

}

 

INPUT *RootColl="/rootCollection, *RodsUser="rods"

OUTPUT ruleExecOut

 

Regards, Andreas

 

Von: irod...@googlegroups.com [mailto:irod...@googlegroups.com] Im Auftrag von Andreas Kuntzagk

Gesendet: Donnerstag, 29. November 2018 16:35
An: irod...@googlegroups.com