Can an irdos user check the storage resource used/available space?

[mauro....@cmcc.it]

Hi All,

some user ask me to provide a way to check the used/available space of the storage resource managed by irods.

In other words, at this moment, each user can check the current "used and available user quota" using iRODS icommands. But, due to some technical reasons, users also needs to obtain something similar to the linux "df -h" command output.

Let's say, something similar to the following kind of output.

/ltfscache file system is mounted and used as storage resource by the iRODS resource server.

[admin@nfs-server ~]# df -h
Filesystem              Size  Used Avail Use% Mounted on
ltfscache                660T   410T   260T  62% /ltfscache

Can an irods user obtain it using iRODS?

Thank you in advance,

Mauro

[Jean-Yves Nief]

hello Mauro,

            you could achieve this using msiExecCmd msi which could
execute a shell script installed in the msiExecCmd_bin of your
installation. The rule could be defined on the server side and the
client could trigger this rule using irule or any kind of wrapper script
that would call "irule": that way you will be sure that end clients
won't be aware of the gory details of the implementation (hence have no
clue of what is being ran on the server side: better for security reasons).
cheers,
JY

> --
> --
> The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
>
> iROD-Chat: http://groups.google.com/group/iROD-Chat
> ---
> You received this message because you are subscribed to the Google
> Groups "iRODS-Chat" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to irod-chat+...@googlegroups.com
> <mailto:irod-chat+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/irod-chat/003574d9-bbc5-46cf-8492-f66f29db253dn%40googlegroups.com
> <https://groups.google.com/d/msgid/irod-chat/003574d9-bbc5-46cf-8492-f66f29db253dn%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Terrell Russell]

Hi Mauro,

Another method...

If you are keeping the free space entries in the catalog updated... that information would be directly available via iquest.

See:

   https://docs.irods.org/4.2.11/plugins/composable_resources/#unixfilesystem

To update the 'free_space' value after every large file put and replication (automatically), the following rules can be used:

acPostProcForParallelTransferReceived(*leaf_resource) {
    msi_update_unixfilesystem_resource_free_space(*leaf_resource);
}
acPostProcForDataCopyReceived(*leaf_resource) {
    msi_update_unixfilesystem_resource_free_space(*leaf_resource);
}

And then any iRODS user could see the free space on demoResc with...

$ iquest "SELECT RESC_FREE_SPACE where RESC_NAME = 'demoResc'"

Terrell

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/91c5fc5d-e065-3f66-7df5-e10556429c7e%40cc.in2p3.fr.

[mauro....@cmcc.it]

Hi Terrell, Hi Jean-Yves,

many thanks for your help and suggestions.

I will try to implement one of the solutions you provided.

Thanks again,

Mauro

[mauro....@cmcc.it]

Hello Terrell,

I was thinking about your solution.

I have a question: since sometime we don't use parallel upload and since we are not replicating any file between resources, I would like to ask you if the solution you provided here https://groups.google.com/g/Irod-Chat/c/oViKLJg0LaI/m/jvHjp7leAgAJ could be used in my case.

If yes, I can add the follwoing code in a dedicated rule

acPostProcForPut {

  msi_update_unixfilesystem_resource_free_space('yourRescName')

}

and I can create a wrapper containing ilsresc command.

Does this solution take into account also the data that have been already stored in the storage resource?

[Terrell Russell]

I think that would probably work okay.

You will end up running the free space calculation *very* often (every put), which could slow down your system.

Here is the code if you'd like to see what it's doing.

   https://github.com/irods/irods/blob/4-2-stable/plugins/microservices/administration/msi_update_unixfilesystem_resource_free_space/libmsi_update_unixfilesystem_resource_free_space.cpp#L17

Alternatively, you can write a rule that you place on the delay queue and which fires periodically, at whatever rate makes sense for your use cases.

Or combine these methods - an automatic rule for large file transfers and replications, and a time-based rule to keep the stats always relatively up-to-date.

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/c664b010-e685-4ac9-939c-74d364296d95n%40googlegroups.com.

[John Constable]

This is what we do - we have a script that runs on every Consumer which iterates through all the resources on the server and updates its Free Space.

We originally needed this because we also have Minimum Free Space in Bytes set to ensure we don't run a disk out of space.

In addition, we have msi_update_unixfilesystem_resource_free_space on acPostProcForPut, but also, if memory serves, for similar upload PEP's.

Be aware also that how your users upload data might affect this; for instance, using the Python API doesn't count as a put (or it didn't)?, and some uses of the C API can avoid triggering acPostProcForPut as well.

This is why we have a simple cron based rule as well. A delay rule would also work but would be quite complex if you have lots of Consumers, whereas the cron job for us scales with the number of servers.

John

[Mauro Tridici]

Hello John, Hello Terrell and Jean-Yves,

I really appreciated your support.

Thanks to your support, I have been able to find the right solution for my needs.

Have a great day!

Mauro

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/96465db0-f3b8-48ed-b5c9-6fb0de97c155n%40googlegroups.com.

 be privileged (GDPR 2016/679).

If you are not the correct recipient, you are kindly requested to notify the sender immediately, to cancel it and not disclose the contents to any other person.

[Mauro Tridici]

Good morning guys,

this is just to say that I satisfied my needs following your suggestions.

Many thanks to all of you :)

Have a great day,

Mauro

On 31 Jan 2022, at 14:41, John Constable <johnsco...@gmail.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/96465db0-f3b8-48ed-b5c9-6fb0de97c155n%40googlegroups.com.

[Terrell Russell]

Excellent.

Can you share the approach you chose and why?

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/6A4D21D1-B0B5-478B-B9E8-C3F3254E43AB%40cmcc.it.

[Mauro Tridici]

Hello Terrell, 

Sure! I can share it, it is a very simple approach, but it works :)

I created a rule (in the iRODS client side) containing msiExecCmd msi which execute a shell script installed in the msiExecCmd_bin directory in the iRODS server side.

After that, I created a wrapper (even in the client side) which trigger this rule using irule command.

usedbuffer.r rule:

usedbuffer {

   msiExecCmd(*Cmd,*Arg,"irs01","null","null",*Result);

   msiGetStdoutInExecCmdOut(*Result,*Out);

   writeLine("stdout","iDAS buffer filesystem used space");

   writeLine("stdout","*Out");

}

INPUT *Cmd="usedbuffer"

OUTPUT ruleExecOut

usedbuffer.sh script:

[irods@irs01 msiExecCmd_bin]# cat usedbuffer 

#!/bin/sh

/usr/bin/df -h|grep -E "ltfs|Filesystem"|awk '{print $2"\t"$3"\t"$4"\t"$5}'

Please note that “irs01” is the iRODS resource server that mounts, via NFS, the storage resource.

"cbuffer" is the wrapper that uses “irule”:

[test@login7][/home/test]> cbuffer

iDAS buffer filesystem used space

Size Used Avail Use%

660T 180T 480T 28%

I chose this kind of solution because, in this way, nothing will be executed periodically in the backend and there is no constraints related to the size of the files that the users will store.

I know that this solution will never stop a “create” operation if the free space on disk will be not enough to accept new data, but, at this moment, users data are jailed in the irods user quota (the sum of users quota should not overcome the storage resource total capacity).

Please, let me know if you have some additional ideas that can improve this solution.

Thank you,

Mauro

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CAFaqteYp-x5hUtajTdV64XyuvbAserbE6npMm8Raq6AAVvkKrA%40mail.gmail.com.

[Terrell Russell]

Seems to work great.

Thanks for sharing!

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/319D5D13-DCAB-4D78-821D-B10871A63A99%40cmcc.it.

[mustafa dikmen]

Hello Jean-Yves, Terrell,

I am a bit confused with what you offered here regarding with the rule part. Could you verify what I wrote below is correct?

- Say I have "du" script in msiExecCmd_bin

#!/usr/bin/sh

echo "The total size in $1"
iquest "select sum(DATA_SIZE) where COLL_NAME like '$2%'"

- In order to call du script I need to use irule. That is;

the test.r file a client will call is:

test_rule{

   msiExecCmd(*Cmd,"*Arg","null","null","null",*Result);
   msiGetStdoutInExecCmdOut(*Result,*Out);

   writeLine("stdout","*Out");

}

INPUT *Cmd="du", *Arg="/myZone/home/myUser"
OUTPUT ruleExecOut

the command that iCommands' user will execute:

irule -r irods_rule_engine_plugin-irods_rule_language-instance -F test.r

But how can someone use "irule" to call "du" as a command instead of calling it in a rule file? (to call the "du" script in msiExecCmd_bin and see the result in a user's stdout)

Would there be an approach like here?  irule -r irods_rule_engine_plugin-irods_rule_language-instance "msiExecCmd(...)"; "msiExecCmd(...)"; writeLine("stdout","*Out") null null

If what you meant by "The rule could be defined on the server side" is that we can put "msiExecCmd" in lets say in core.re, then how can a user see the output by using irule invoking that rule?

Thanks,

Mustafa