Adding remote host issues with Unixfilesystem

[Jay Linux]

Hi Team,

I'm a newbie to irods software.

Currently I managed to setup Irods server v4.3.0 on a ubuntu system. Now I need to connect to a remote linux server and get the files on a remote server mount point path /data.

Here is the remote connection information.

cat .irods/irods_environment.json

{
    "irods_host": "irods.example.com",
    "irods_port": 1247,
    "irods_user_name": "rods",
    "irods_zone_name": "tempZone",
    "irods_remote_resources": {
        "guruResc": {
            "plugin_name": "unixfilesystem",
            "plugin_specific_configuration": {
                "resource_location": "/data",
                "remote_scheme": "ssh",
                "remote_host": "data.example.com",
                "remote_zone": "tempZone",
                "ssh_user": "testuser"
            }
        }
    }
}

I excecuted below command to initiate the  connection to the remote linux server.

$ iadmin mkresc dataResc unixfilesystem  data.example.com:/var/lib/irods/new_vault1
Creating resource:
Name: "dataResc"
Type: "unixfilesystem"
Host: "data.example.com"
Path: "/var/lib/irods/new_vault1"
Context: ""

 $ilsresc -l dataResc
resource name: dataResc
id: 10053
zone: tempZone
type: unixfilesystem
location: data.example.com
vault: /var/lib/irods/new_vault1
free space:
free space time: : Never
status:
info:
comment:
create time: 01691063836: 2023-08-03.15:57:16
modify time: 01691063836: 2023-08-03.15:57:16
context:
parent:
parent context:

The problem is that there is no connection attempt establishing to the remote server. I'm not sure why this above command, not looking the remote server credentails from this file .irods/irods_environment.json. I already enabled the SSH key based authentication to remote server.

Note:- There is no any firewall between Irods server and the remote server (/data/ ). If I'm doing the wrong procedure, please pin point the correct method.

My goal is to get the files/directories on the remote server (path - /data ) should be reflected in the irods connection, and then i need to manage and access the data from metalnx.

Please advise.

Thanks

Jay

[Kory Draughn]

Hi Jay,

A few things ...

cat .irods/irods_environment.json
{
    "irods_host": "irods.example.com",
    "irods_port": 1247,
    "irods_user_name": "rods",
    "irods_zone_name": "tempZone",
    "irods_remote_resources": {
        "guruResc": {
            "plugin_name": "unixfilesystem",
            "plugin_specific_configuration": {
                "resource_location": "/data",
                "remote_scheme": "ssh",
                "remote_host": "data.example.com",
                "remote_zone": "tempZone",
                "ssh_user": "testuser"
            }
        }
    }
}

 

irods_remote_resources is not a valid property.

See https://docs.irods.org/4.3.0/system_overview/configuration/ to learn what properties are supported by various files.

Consider reading the beginner training PDF to learn basic concepts and commands for iRODS. It is highly recommended to help jump start things. You can find the PDF at the following link:

• https://github.com/irods/irods_training/blob/main/beginner/irods_beginner_training_2023.pdf
  

I excecuted below command to initiate the  connection to the remote linux server.

$ iadmin mkresc dataResc unixfilesystem  data.example.com:/var/lib/irods/new_vault1
Creating resource:
Name: "dataResc"
Type: "unixfilesystem"
Host: "data.example.com"
Path: "/var/lib/irods/new_vault1"
Context: ""

That command does not initiate a connection to the remote machine. It updates the catalog with information which helps iRODS know where to find data. It also assumes there is an iRODS server running on the machine identified as data.example.com.

Now that that's out of the way. I have a question.

Q. Can the computer running iRODS see /data?

Q. Am I correct in that you only want to present the files under /data through iRODS? That is, nothing is moved/copied from /data.

Thanks,

Kory Draughn

Chief Technologist

iRODS Consortium

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/c0954a8a-1f75-4d85-b6fa-08d14f1bb1f7n%40googlegroups.com.

[Jay Linux]

Dear Kory,

Thanks for your update.

Answering to your query

Q. Can the computer running iRODS see /data?

The remote machine doesn't install any irods package. A system user named "testuser" have full privilege to access /data path.

Q. Am I correct in that you only want to present the files under /data through iRODS? That is, nothing is moved/copied from /data.

Yes correct, i will not modify the data.

I have some queries for you.

1) Inorder to add a remote linux server, do i need to install the irods package first and then only i can add as a unixfilesystem resource ?

2) I read about the network share protocols like NFS/SMB, but in my case, the data is not distributed and it is available local to each server. In our case, i have 10-20 physical servers and if i need to manage each server /data mountpoint to a centralized irods server, do i need to install irods software on each node?

"This type of storage resource communicates with a storage device through the standard POSIX interface -> Unix filesystem " -> this means, this resource is only work whether the data mountpoint will be available local to the irods server.

Below is my actual requirement:-

----

As per my expectation, 

Server A connects to Server B via SSH with key based authentication as a "testuser" and it will fetch the /data mountpoint from ServerB.

Server A connects to Server C via SSH with key based authentication as a "testuser" and it will fetch the /scratch/share mountpoint from ServerB.

Server A -> Running irods 4.3.0  -> This acts as a centralized data warehouse system

Server B -> Normal linux server, without irods packages installed.

Server C -> HPC system, without irods packages installed.

---

Please let me know.

Thanks

Jay

[Kory Draughn]

Jay,

1) Inorder to add a remote linux server, do i need to install the irods package first and then only i can add as a unixfilesystem resource ?

It depends.

If the data is accessible via the mount command, then there's no need to deploy iRODS on the remote server.

If the data is not accessible via the mount command, then you need to deploy iRODS on the remote server.

For example, let's assume the following ...

• Server A is running iRODS 4.3.0
• Server B is running an NFS server that exports /data
• Server A has mounted /data

With this setup, there's no need to deploy iRODS on Server B because Server A can access /data directly. All that needs to be done is the following:

• Create a unixfilesystem resource on Server A that points to the location where /data is mounted
• Walk the filesystem at /data and register all files into the iRODS catalog
• For this, consider using something like the iRODS automated ingest tool (see https://github.com/irods/irods_capability_automated_ingest)
• Keep in mind that registration of a file only informs iRODS of its existence

Now, let's look at a different example.

Given the following:

• Server A is running iRODS 4.3.0
• Server B has data (i.e /data) which you want to access
• Server B does NOT export /data in any way (i.e. Server A cannot see /data at all)

In this case, you need to deploy iRODS 4.3.0 on Server B. The reason being the data on Server B is not reachable by Server A.

Deploying iRODS on Server B gives the iRODS application running on Server A a way to access the data on Server B. This is achieved by creating a unixfilesystem resource that references Server B and the directory of interest.

With that in place, when a client connects to Server A and requests data that is only available on Server B, Server A will detect this and forward the request to Server B.

Again, I highly recommend reading the beginner training PDF. I've also provided a link to the slide decks which cover many aspects of iRODS. I've also included a link to the website.

• https://github.com/irods/irods_training/blob/main/beginner/irods_beginner_training_2023.pdf
  
• https://slides.com/irods?search=advanced
• https://docs.irods.org

Hope that helps!

Kory Draughn

Chief Technologist

iRODS Consortium

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/1132c159-163e-4974-af76-d56961a1fa5dn%40googlegroups.com.

[Jay Linux]

Hi Kory,

Thanks, this makes more clear now.

Then I will proceed with mapping the remote mount point using SSHFS, by this way i can see the remote mountpoint available local to the iRODS server. Because installing irods software on remote hosts would not be a good option in our environment. Hence i will opt with fuse mount.

I still have some issues with accessing the resource collection. I'm good, once the below issue will be sorted.

===

/share is a local mount point on the iRODS server

$ls -ld /share

drwxrwxrwx 2 root root 4096 Aug  9 00:54 /share

$ls -l /share/
-rw-r--r-- 1 irods irods 1048576000 Aug  8 23:29 test.bg

I created the resource as below

$iadmin mkresc dataResc unixfilesystem  `hostname`:/share

Currently I'm "rods" user

$ils
/tempZone/home/rods:

Once I create the resource, I'm unable to switch to /share

$icd /share
No such directory (collection): /share

$icd /tempZone/home/rods/share
No such directory (collection): /tempZone/home/rods/share

===

Kindly advise what is blocking me to unable to view the file "/share/test.bg" using icommands?

Thanks

Jay

[Kory Draughn]

Jay,

You've made the storage device available to iRODS, but you haven't told iRODS what files are there.

To do that, you need to walk the /share directory and register all files in the iRODS catalog.

Your iRODS server as of now thinks /share is an empty resource.

Building a strong mental model will make using iRODS much easier. The material I shared will help with that.

Hope that helps :-)

Kory Draughn

Chief Technologist

iRODS Consortium

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/0deafb1f-213a-41f3-b0d7-4f0ce776fcf5n%40googlegroups.com.

[Kory Draughn]

Also, you may find this command useful.

• https://docs.irods.org/4.3.0/icommands/user/#ireg

I recommend learning iRODS using a test environment. That will give you a place to experiment and learn the basics.

Kory Draughn

Chief Technologist

iRODS Consortium

[Jay Linux]

Hi Kory,

Awesome, thanks for the pointer.

After registering the files, I'm able to view perfectly. Now i need to learn in depth how to enable tagging, automate ingest of data etc..

Many thanks for your help and advise.

Thanks

Jay