Irods setup for HPC / Servers

[Jay Linux]

Dear Team,

I would like to setup Irods v4.3.0 for managing the data residing across our infrastructure like HPC, Virtual machines, Servers, Storage box etc..

Let me explain my requirements below, please let me know if this can be doable using Irods.

We dealt with massive data let's say  4 Petabytes of data which sits across in HPC ( scratch and archive ) , Physical servers, Netapp Storage drive etc only in our Onpremise. We don't have data over the cloud.

Here I need a centralized setup where I can take the control of all the data. This means I will have a dedicated host where the iRods server running. From here i can communicate to other hosts datasets.

Requirements

===========

a) Data Connection:- Able to connect to HPC/servers to get the data. Here how can I establish the communication between the iRods server to HPC/Servers?

b) Data Movement:-  I have the ability from iRods to modify or update the data. 

c) Data Tagging: I can tag the data as per the host to identify from where the data comes in.

d) Data Classification:  I can seggregate the data as per the data types. Let's say for example based on file extension like bams, vcf , tgz etc..

As of now, I started setup with iRoDS 4.3.0 server running on Ubuntu 20.04 and tested with local data and integrated with metalnx as the Web UI.

Setup questions:-

=============

a) Do you have the Docker setup for irods-server for v4.3.0? I can see the latest docker version for iRods as v4.0.3. In future, i would like to move to kubernetes setup. Please advise the pathway/pointer how to proceed.

b) Once the irods-server is up and running. How can i import the storage from other nodes. Do i need to install any agent on the remote server? or i can pull the information using SSH/NFS/SMB protocol ? Please advise.

c) For the UI part, I'm using the metalnx docker to visualize the data. Do you support any other software for UI? How can I customize the UI?

d) Do you provide API? My plan is to communicate to iRODS via API to fetch the data information from a custom developed inhouse software.

Kindly advise for my above questions and concerns with irods.

Thanks

Jay

[Kory Draughn]

Hi Jay,

a) Do you have the Docker setup for irods-server for v4.3.0? I can see the latest docker version for iRods as v4.0.3. In future, i would like to move to kubernetes setup. Please advise the pathway/pointer how to proceed.

We do not host or provide pre-built docker images. We offer a few repositories that allow you to test and try various things in iRODS.

You can find the repositories at the following:

• https://github.com/irods/irods_development_environment
  
• This repo makes it easy to compile iRODS source code and generate installable packages for various OS
• https://github.com/irods/irods_testing_environment
  
• This repo allows you to run tests against iRODS
• The packages produced by the the development environment are normally used with this project
• This repository also makes it easy to stand up an environment and try various things, so it's very good for learning
• https://github.com/irods/irods_demo
  
• This repo is for demonstration purposes
• It stands up an iRODS server along with several other clients so users can try them out

If you want to know more about the first two repos, watch the following UGM video:

• https://youtu.be/tm6DY83KWW0
  

The development and testing environment repos contain several Dockerfiles which capture how to put iRODS in a container.

I can't offer any advice on deploying in Kubernetes since I've never used it. Perhaps someone within the community can assist with that.

b) Once the irods-server is up and running. How can i import the storage from other nodes. Do i need to install any agent on the remote server? or i can pull the information using SSH/NFS/SMB protocol ? Please advise.

To make iRODS aware of storage, you have to tell iRODS about it.

You do not have to install any agents on the storage devices. You tell iRODS where the storage is (i.e. hostname) and use the appropriate resource plugin so iRODS can access it.

The following should be helpful in learning how to do this.

• Read the beginner training to learn the concepts and basics
• https://github.com/irods/irods_training/blob/main/beginner/irods_beginner_training_2023.pdf
  
• Use this slide deck to learn about resource hierarchies
• https://slides.com/irods/ugm2023-resource-hierarchies-and-composition

Feel free to browse other slide decks to learn about other tools and concepts.

• https://slides.com/irods
  

c) For the UI part, I'm using the metalnx docker to visualize the data. Do you support any other software for UI? How can I customize the UI?

By customize, I'm assuming you mean adding logos and changing colors, etc. If so, then yes, you can customize the Metalnx UI. I've never tried it, but I've seen others do it. I'll see if I can locate documentation for that.

As for other clients, have a look at this page:

• https://irods.org/clients/

d) Do you provide API? My plan is to communicate to iRODS via API to fetch the data information from a custom developed inhouse software.

Yes. There are several client-side libraries for interacting with iRODS.

The libraries are listed below:

• C/C++ - https://packages.irods.org/
• irods-dev (DEB package) or irods-devel (RPM package)
• Python - https://github.com/irods/python-irodsclient
  
• Java - https://github.com/DICE-UNC/jargon
  
• Go - https://github.com/cyverse/go-irodsclient

We also have a REST API which makes interacting with iRODS simpler. You can find that at the following:

• https://github.com/irods/irods_client_rest_cpp.

We're currently redesigning the REST API based on feedback from the community. This was presented at our annual UGM meeting. The presentation will be available soon.

If you have more questions, please reach out.

Good luck!

Kory Draughn

Chief Technologist

iRODS Consortium

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/4c616895-64d6-43d4-85f4-9abf64d7d81an%40googlegroups.com.