How to disable or stop iput usage when the storage resource is going to be full

[Mauro Tridici]

Dear users, 

Is there a simple way to temporary disable or stop iput and irsync usage when the storage resource manager by irods is almost full? 

I would like to stop users that are trying to puttana new files when the Unix file system usage  is at 80%.

Many thanks in advance, 

Mauro

[Terrell Russell]

Hi Mauro,

Please look at "minimum_free_space_for_create_in_bytes":

   https://docs.irods.org/4.2.11/plugins/composable_resources/#unixfilesystem

Terrell

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CA%2B_W2a04t2o7aS7xTo54%3DgG7x7Rif%3DxfSh8GyKUozo8XXGTEmw%40mail.gmail.com.

[Mauro Tridici]

Thank you, Terrell.

Great! It is exactly what I need.

So, in order to keep 5TB free, I should execute the following command on the iCAT server:

irods@hostname:~/ $ iadmin modresc unixResc context 'minimum_free_space_for_create_in_bytes=5497558138880'

And, if I’m not wrong, I can update the “free_space” value automatically or manually.

Since I need to check the available free space on a regular basis, I’m thinking if I should do it “manually” every 15 minutes or when a put action is fired.

For this reason, I would like to ask you the following questions:

Regarding manual free_space update procedure:

- is there a way to update the free_space value using a unix crontab task + iadmin command?

Regarding automatic free_space update procedure:

- when a file can be considered “large” ? ( we implemented a file size check to be sure that 1GB <= size <= 30GB );

- in order to config the automatic update should I simply add these lines in /etc/core.re file in the iRODS resource server?

acPostProcForParallelTransferReceived(*leaf_resource) { msi_update_unixfilesystem_resource_free_space(*leaf_resource); } acPostProcForDataCopyReceived(*leaf_resource) { msi_update_unixfilesystem_resource_free_space(*leaf_resource); }

Last question: 

Since the unix file system is an HSM managed file system, I would like, may be in a second step, to stop also every get users activities ( because a “read” action will result in a file recall from tape  ).

Is there a way to temporarily put offline the entire iRODS resource?

Sorry for the large number of questions...

Many thanks in advance,

Mauro

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CAFaqteZ%3D0p1iMTp9tbW1hYsqNruv2SP_fLyZVtcUbY2P48ERrQ%40mail.gmail.com.

[mauro....@cmcc.it]

Hi all,

did anyone have the same questions or needs?

Thank you in advance,

Mauro

[John Constable]

We do exactly this, with some slight exceptions;

1. we add the acPostProcForParallelTransferReceived two rules to a custom rulebase; https://docs.irods.org/4.2.7/system_overview/best_practices/#adding-custom-rulebases-rather-than-editing-corere

2. We have a cron job that runs every 10 mins on each Consumer that calculates and updates the free space for each resource hosted on that server. This helps avoid long-running uploads running a resource out of space mid-transfer, as well as the resource voting it has enough space for other files as the large file is ingested. It's a bit race-condition-y but better than not doing it at all.

Regarding making resources unavailable there are two approaches;

1. marking a resource down `iadmin modresc <resc> status down`

2. using passthru weighting ( https://docs.irods.org/4.2.7/plugins/composable_resources/#passthru ) to set the resource to vote zero to read activities.

We've been using (1) happily for years, but recently hit a wrinkle when it's combined with passthru voting resources - https://github.com/irods/irods/issues/6260.

However, given all the edge cases in https://github.com/irods/irods/issues/4508, if starting now, I would recommend setting up passthru weighted resources.

We plan to script the equivalent of `iadmin modresc status` for our DC team folks and have a separate branch for each server. However, this requires a fair amount of 'tree surgery', not to mention some coding, so we've not done it yet, but we do need to...

hope that helps!

John

[Mauro Tridici]

Hello John,

many thanks for sharing your experience with us.

I really appreciated your explanation.

I will try to replicate your setup (custom rule + crontab job on consumer) in our environment.

At this moment, we don’t use passthru resource, but a unix storage resource: it’s a GPFS remote file system (managed by HSM software) mounted on the iRODS Resource server via NFS.

In the past, we tried to use passthru and replica in order to make easy a future storage decomissioning, but it seems that it is not a great idea when the file system is managed by HSM.

In case of “rebalance”, a lot of petabytes of migrated data will be recalled from tapes to the old resource and then from the old resource to the new one.

During the next months, a storage extension will be done using GPFS data migration features: the storage running in the backend will be changed and extended, but the GPFS file system, impemented on top of the storage, will be preserved…also the NFS server export will be unchanged. In your opinion, do you think that, from the iRODS point of view, it will be necessary to reconfigure the resource and "manually" ingest the data or everything will continue to work as before?

Please, correct me if I’m wrong.

Sorry, if my last question is out of topic.

Regards,

Mauro

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/22bdc793-ed6b-48f1-88a0-6e6b37727e84n%40googlegroups.com.

[jc...@sanger.ac.uk]

>  During the next months, a storage extension will be done using GPFS data migration features: the storage running in the backend will be changed and extended, but the GPFS file system, impemented on top of the storage, will be preserved…also the NFS server export will be unchanged. In your opinion, do you think that, from the iRODS point of view, it will be necessary to reconfigure the resource and "manually" ingest the data or everything will continue to work as before?

If the data is still going to reside the in the file system in the same place iRODS has it registered before the upgrade, I would imagine that it will continue working as before?

If you are changing the NFS path or sub directories though you should look into `iadmin modrescdatapaths` to update that.

so, if data was registered before the upgrade to /nfs/gpfs/irods, and it remains that path after, you shouldn't need to do anything as long as the Consumer that has the resource on it can still see that path.

If on the other hand that path becomes /nfs/gpfs_new/irods then you will need to update the catalog with modrescdatapaths (but this is a catalog operation so wont touch the data).

Hope that helps!

John

[jc...@sanger.ac.uk]

Oh, more more thing I noticed;

>  In the past, we tried to use passthru and replica in order to make easy a future storage decomissioning, but it seems that it is not a great idea when the file system is managed by HSM.

You don't need to use a replica resource to make use of the composite trees, so you could have;

root:passthru

--> GPFS:passthru

----> GPFS Resource: filesystem

On Tuesday, April 5, 2022 at 8:58:38 AM UTC+1 mauro....@cmcc.it wrote:

[Mauro Tridici]

Thank you very much for your help, John.

What a pity! It is too late to come back e revert to passthru, right?

Have a great day,

Mauro

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/720c6cf5-1f8d-4e41-b84b-841db7d89cb0n%40googlegroups.com.

[Terrell Russell]

You can definitely change the resource hierarchy now, your data will not be affected.

John's proposed hierarchy seems very nice - gives you a stable root resource (for your users), gives you the passthru for manipulating read/write weights (for when you want to prevent activity), and your GPFS NFS mount point (which should not need to be touched/manipulated, you should have stable paths according to your explanation).

Remember, you can test this in a separate resource hierarchy first to make sure you like how it works.

maurotestroot (passthru)

 --> maurotestpassthru (passthru)

    ----> maurotestgpfs (unixfilesystem)

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/81CBD1E0-6878-4322-A6BE-C227A02B2086%40cmcc.it.

[Mauro Tridici]

Thank you, Terrell.

I will create a "devel" instance of iRODS and I will try to understand the steps (the commands list) I need to execute to do this work safely.

Many thanks again,

Mauro

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CAFaqteaLBT9SAAaB4vh3LxJQ%2BJ0i-OzLxBuZARNEB6%2BMqfEd6Q%40mail.gmail.com.

-------------------------
Mauro Tridici

Fondazione CMCC
CMCC Supercomputing Center
presso Complesso Ecotekne - Università del Salento -
Strada Prov.le Lecce - Monteroni sn
73100 Lecce  IT
http://www.cmcc.it

mobile: (+39) 327 5630841
email: mauro....@cmcc.it
https://it.linkedin.com/in/mauro-tridici-5977238b

-------------------------

Le informazioni contenute in questo messaggio di posta elettronica e negli allegati se presenti sono riservate e confidenziali: ne è vietata la diffusione in qualsiasi modo o forma (GDPR 2016/679).

Qualora lei non fosse il destinatario del messaggio, la invito a non diffonderlo e ad eliminarlo dandone gentilmente comunicazione al mittente.

 

The information included in this e-mail and any attachments are confidential and may also be privileged (GDPR 2016/679).

If you are not the correct recipient, you are kindly requested to notify the sender immediately, to cancel it and not disclose the contents to any other person.

[mauro....@cmcc.it]

Hi Terrell, Hi John,

the following command set minimum free space value to 5TB.

irods@hostname:~/ $ iadmin modresc unixResc context 'minimum_free_space_for_create_in_bytes=5497558138880'

How can I know the currrent (default) value in order to set it to the original value in case of  problem?

Thanks again,

Mauro

[Terrell Russell]

> How can I know the currrent (default) value in order to set it to the original value in case of  problem?

I'm not quite sure what you're asking.   If you want to change the value, just change it again.  If you set it to 1, then this setting will never block replica creation.

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/3c75cd8b-80c7-4505-ab4e-c59cf9bd97c9n%40googlegroups.com.

[mauro....@cmcc.it]

Sorry Terrell, you are right. My english is not so good.

I will try to explain my needs.

Recently you provided this link https://docs.irods.org/4.2.11/plugins/composable_resources/#unixfilesystem.

I read that:

"A free_space check capability has been added to the unixfilesystem resource in 4.1.10. The free_space check can be configured with the context string using the following syntax: irods@hostname:~/ $ iadmin modresc unixResc context 'minimum_free_space_for_create_in_bytes=21474836480'

The example requires this unixfilesystem plugin instance (unixResc) to keep 20GiB free when considering whether to accept a create operation. If a create operation would result in the bytes free on disk being smaller than the set value, then the resource will return USER_FILE_TOO_LARGE and the create operation will not occur"

So, if I want to enable this kind of check I only need to set minimum_free_space_for_create_in_bytes to the desired value.

What should I do in order to disable this check and return to the default/original situation?
I think that I should set minimum_free_space_for_create_in_bytes to the original/default value but I don't know this value.

irods@hostname:~/ $ iadmin modresc unixResc context 'minimum_free_space_for_create_in_bytes=???'

Replica is not implemented.

Thank you,

Mauro

[Terrell Russell]

Same answer - just set it to 1, or remove the entry from the context string entirely.

Then...

"If a create operation would result in the bytes free on disk being smaller than the set value, then the resource will return USER_FILE_TOO_LARGE and the create operation will not occur"

Still true :)

The default/original setting was that it *was not set*, so there was no protection / safety check.  The disk could fill up.

I hope that answers your question.

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/0271b7fb-bb05-4416-95c6-27c44c2256f2n%40googlegroups.com.

[Mauro Tridici]

Yes, you helped me again :)

Thank you very much, Terrell.

Mauro

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CAFaqteZi3JmiaFJ54%2BwW69nYbdz9y_tB432GnYnVtMaMF%2Bcx%3DA%40mail.gmail.com.

[Mauro Tridici]

Houston, we have a problem :)

Let’s see what I have done:

1) on the unique iRODS resource server that is mounting the NFS remote unix resource, I created freespace.re rule in /etc/irods/ with the following content:

acPostProcForParallelTransferReceived(*leaf_resource) { 

msi_update_unixfilesystem_resource_free_space(*leaf_resource); 

}

acPostProcForDataCopyReceived(*leaf_resource) { 

msi_update_unixfilesystem_resource_free_space(*leaf_resource);

}

2) I added the freespace.re rule to server_config.json file and I restarted the iRODS server 

[…]

        "rule_engines": [

            {

                "instance_name": "irods_rule_engine_plugin-irods_rule_language-instance",

                "plugin_name": "irods_rule_engine_plugin-irods_rule_language",

                "plugin_specific_configuration": {

                    "re_data_variable_mapping_set": [

                        "core"

                    ],

                    "re_function_name_mapping_set": [

                        "core"

                    ],

                    "re_rulebase_set": [

"checkmig",

"fileType",

"freespace",

                        "core"

                    ],

[…]

3) I exectued this command to set to 5TB the amount of free space to be preserved: iadmin modresc LtfsCache01 context 'minimum_free_space_for_create_in_bytes=5497558138880’ 

PROBLEM:

When I try to put a 4GB file I received this error:

[user@test][/tmp]> du -hs ISO.iso 

4.3G ISO.iso

remote addresses: 10.0.1.77 ERROR: putUtil: put error for /idas/home/sysm02/ISO.iso, status = -311000 status = -311000 USER_FILE_TOO_LARGE

The resource free space is equal to 51TB.

In order to restore the previous situation I executed: iadmin modresc LtfsCache01 context 'minimum_free_space_for_create_in_bytes=1’

Could you please help me to understand what I’m doing wrong?

Thank you in advance,

Mauro

[Mauro Tridici]

It seems I solved running again the following command:

iadmin modresc LtfsCache01 context 'minimum_free_space_for_create_in_bytes=5497558138880’ 

Now, I’m able to put new files on iRODS resource.

The tests I have done seem to be ok.

Now, I would like to complete this task "marking a resource down using `iadmin modresc <resc> status down`” when USER_FILE_TOO_LARGE error is detected by the rule.

Do you have some suggestions to implement (and log) it?

Thank you in advance,

Mauro

[Terrell Russell]

Glad you got the freespace check working.

You want to mark the resource down if someone tries to upload a file past the threshold and gets denied?

You could add some code to pep_api_data_obj_except() to check the freespace against the minimum_free_space_for_create_in_bytes, and then run something from msiExecCmd_bin to mark a resource as down - but I'm not sure you'd get much value from that... the resource is already 'protected' from disk full events by using the minimum_free_space_for_create_in_bytes mechanism in the first place.

Terrell

 

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/D6921809-7F06-4D74-8D20-E151A5845D74%40cmcc.it.