usng ireg fails

[chris vandert]

Some help required for a newbi.

ireg -C /home/chrisvdt/images /FYD/home/chrisvdt/nw/images

results in this error message:

remote addresses: 127.0.0.1 ERROR: regUtil: reg error for /FYD/home/chrisvdt/nw/images, status = -511013

status = -511013 UNIX_FILE_CREATE_ERR, Permission denied
 ERROR: main: regUtil error.  status = -511013 UNIX_FILE_CREATE_ERR, Permission denied

Why do I get this error. the irods service account(rods)  has access to these files.

I need to register a huge amount of files on the iRods server, but I cannont simply iput them there.

regards

[Tony Edgin]

Hi Chris.

I had this same problem on Saturday. Sometimes iRODS needs a hint about which storage resource to use.  The -R option tells ireg which storage resource to use. 

Cheers,

Tony

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/a625ea74-ac7e-4702-96f4-1101cd921cafn%40googlegroups.com.

[Terrell Russell]

Oh, interesting.   We could probably give a better error message there....

What's the simplest way to reproduce this?

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CACkmaQVg8u_Wg%2BH7qV-hj%2Bs5aGYrVUJcy7gkQ6ojo28YJv0_gA%40mail.gmail.com.

[Tony Edgin]

I have no idea.  Running `ireg -C -f /vault/path//to/folder /zone/path/to/collection` on a resource server will sometimes register the folder's contents and sometimes it won't. If it fails to register, it will always fail. We only host one resource per resource server. Maybe it's something to do with our resource hierarchy.  Here's a reduced version of its structure.

CyVerseRes:random

├── ds01Broker:passthru
│   └── ds01:unixfilesystem
└── ds02Broker:passthru
    └── ds02:unixfilesystem

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CAFaqteZK_DfkGkV6hoT6vOxoveV9TvBAt_6nfqvu3_RUGX9O%3Dw%40mail.gmail.com.

[Terrell Russell]

It would be helpful to see a server-side log (rodsLog - perhaps both provider and consumer) of this event if you can catch one.

Thanks,

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CACkmaQUYwEdbDwwab01YqqYAKyksKWWVFJ6piWpB--XaQgEHsA%40mail.gmail.com.

[Coposky, Jason Matthew]

Tony,

This will only work if the directory to be registered is on a network shared folder across both servers hosting those storage resources, otherwise the vote may go to a leaf resource which cannot reach out and stat() those files. 

Thanks,

------
Jason Coposky
Executive Director, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
w: (919)445-9675
m: (919)522-0517
jas...@renci.org

---

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Tony Edgin <tedgin...@gmail.com>
Sent: Monday, November 9, 2020 2:42 PM
To: irod...@googlegroups.com <irod...@googlegroups.com>
Subject: Re: [iROD-Chat:19345] usng ireg fails

 

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CACkmaQUYwEdbDwwab01YqqYAKyksKWWVFJ6piWpB--XaQgEHsA%40mail.gmail.com.

[Tony Edgin]

Hi Jason.

I may be misunderstanding what you are saying, so I'll add some details to what I was saying.

All files being registered are in local storage on a single resource server, e.g., ds01.cyverse.org.  I run ireg on that server. ds01.cyverse.org hosts a single unixfilesystem storage resource, ds01.  `ireg -C -f ...` may or may not work, but `ireg -C -f -R ds01 ...` works. ireg can sometimes figure out the correct storage server, I guess.

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/BN7PR03MB36507D28DA96FD3629462378C7EA0%40BN7PR03MB3650.namprd03.prod.outlook.com.

[Coposky, Jason Matthew]

Tony,

Your root resource in that hierarchy is a random node with two leaves which I presume are on separate servers.  If you leave out the -R <target resource> the voting mechanism kicks in and you may or may not get a root resource which is on the same server as your directory of data (unless that directory is shared across both servers at the exact same path).

Hope that makes sense.

Thanks,

------
Jason Coposky
Executive Director, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
w: (919)445-9675
m: (919)522-0517
jas...@renci.org

---

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Tony Edgin <tedgin...@gmail.com>

Sent: Monday, November 9, 2020 3:40 PM
To: irod...@googlegroups.com <irod...@googlegroups.com>
Subject: Re: [iROD-Chat:19348] usng ireg fails

 

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CACkmaQWXRpEVTUsmKHsp1cC25w4tthAcf_DLa0gj6ONaGuRmXA%40mail.gmail.com.

[Tony Edgin]

That makes sense. Thanks for the clarification.

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/BN7PR03MB3650258B11448370E516CEA6C7EA0%40BN7PR03MB3650.namprd03.prod.outlook.com.

[chris vandert]

I don't know whats wrong in my case but it never seems to work;

ireg  -C -f -R bundleResc --repl /home/chrisvdt/images  /FYD/home/chrisvdt/images

remote addresses: 127.0.0.1 ERROR: regUtil: reg error for /FYD/home/chrisvdt/images, status = -511013 status = -511013 UNIX_FILE_CREATE_ERR, Permission denied

 ERROR: main: regUtil error.  status = -511013 UNIX_FILE_CREATE_ERR, Permission denied

rodsLog:

Nov 11 16:16:49 pid:2068 remote addresses: 127.0.0.1 ERROR: dirPathReg: rsFileOpendir for /home/chrisvdt/images error, status = -511013

Alternatively:

 ireg  -R demoResc --repl /home/chrisvdt/images/football.jpg /FYD/home/chrisvdt/images/football.jpg
 ERROR: main: regUtil error.  status = -814000 CAT_UNKNOWN_COLLECTION

rodsLog

Nov 11 16:05:19 pid:1816 remote addresses: 127.0.0.1 ERROR: filePathRegRepl: getDataObjInfo for /FYD/home/chrisvdt/images/football.jpg

Op maandag 9 november 2020 om 18:35:53 UTC+1 schreef tedgin...@gmail.com:

[Terrell Russell]

Not clear what's going on here - but you should definitely not be using bundleResc directly - that's only for internal iRODS things (aka mounted files, tar, zip, etc.).

If the resource you're registering into is connected/hosted on a server that has access to that physical path, then that iRODS server should be happy to stat() the things and register them into the catalog.

Your UNIX_FILE_CREATE_ERR error there suggests permissions on the physical disk.

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/e671a0b5-5e2a-4253-9ded-2aaf8c4daeb2n%40googlegroups.com.

[chris vandert]

if I use -R demoResc ==>      ERROR: main: regUtil error.  status = -516013 UNIX_FILE_STAT_ERR, Permission denied

or -R demoResc --repl ==>  ERROR: main: regUtil error.  status = -814000 CAT_UNKNOWN_COLLECTION

i've tried all possible combinations without result.

Op woensdag 11 november 2020 om 16:29:02 UTC+1 schreef Terrell Russell:

[Terrell Russell]

Can you share some relevant logs from /var/lib/irods/log/rodsLog* ?

Shouldn't be any secrets in there...

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/05e7e1cb-191c-4ba1-a6e1-cd592f53a901n%40googlegroups.com.

[chris vandert]

these are the last errors corresponding with my last ireg commands;

Nov 11 16:39:17 pid:2591 remote addresses: 127.0.0.1 ERROR: filePathReg: getFileMetadataFromVault for /FYD/home/chrisvdt/images/football.jpg failed, status = -516013
Nov 11 16:39:55 pid:2611 remote addresses: 127.0.0.1 ERROR: filePathRegRepl: getDataObjInfo for /FYD/home/chrisvdt/images/football.jpg

there are no secrets because this is a demo instance. :-))

the messages don't seem to be very helpfull

Op woensdag 11 november 2020 om 16:56:46 UTC+1 schreef Terrell Russell:

[Terrell Russell]

It seems the file to be registered cannot be seen by the iRODS server process (aka service account).

$ ierror 516013
irods error: -516013 UNIX_FILE_STAT_ERR Permission denied

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/e28da737-57c9-44ff-88d8-4579a329fac9n%40googlegroups.com.

[chris vandert]

Here are some details about my system and environment;

A vm with Red Hat Enterprise Linux Server release 7.6

ICAT db  is on MySQL (MariaDB)

my irods service account name is  ;    irods

I changed the ownership on the files to :  chown irods:irods

The files were chmod to 666  rw rw rw

the user (chrisvdt) performing the ireg is a rodsadmin

I don't see how I can make my files more permissive for irods.

Op woensdag 11 november 2020 om 19:04:04 UTC+1 schreef Terrell Russell:

[chris vandert]

so looking at the source code i see where the error is occuring;

if ( dataObjInfo.dataSize <= 0 &&

  ( dataObjInfo.dataSize = getFileMetadataFromVault( rsComm, &dataObjInfo ) ) < 0 &&

  dataObjInfo.dataSize != UNKNOWN_FILE_SZ ) {

  status = ( int ) dataObjInfo.dataSize;

  rodsLog( LOG_ERROR,

  "filePathReg: getFileMetadataFromVault for %s failed, status = %d",

However, what I don't understand is why is ireg trying to obtain metadata from a file in the vault, since the file is not copyied there in the first place!!!

ireg should only be registering a file in the ICAT database and metadata should be extracted from the file in its original/ physical location.

Op donderdag 12 november 2020 om 10:29:13 UTC+1 schreef chris vandert:

[Terrell Russell]

Does this work for you?

$ cd /tmp
$ mkdir christest
$ echo "hello" > christest/newfile
$ ireg -C $(pwd)/christest $(ipwd)/ichristest
$ ils -L ichristest
/tempZone/home/rods/ichristest:
  rods              0 demoResc            6 2020-11-12.08:14 & newfile
        generic    /tmp/christest/newfile

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/8bb18404-ab98-4c58-95df-fd6d22e3be1fn%40googlegroups.com.

[chris vandert]

Yes, to my astonishment it works fine.

Op donderdag 12 november 2020 om 14:16:41 UTC+1 schreef Terrell Russell:

[chris vandert]

Why does htis make a difference!!!????

Op donderdag 12 november 2020 om 15:18:57 UTC+1 schreef chris vandert:

[Terrell Russell]

The stat/permissions error you were seeing suggests the service account couldn't see all the way to the file it was asked to register.

$ sudo su - irods -c 'stat /tmp/christest/newfile'
  File: '/tmp/christest/newfile'
  Size: 6               Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d      Inode: 131608      Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/     terrell)   Gid: ( 1000/     terrell)
Access: 2020-11-12 08:14:16.922461073 -0500
Modify: 2020-11-12 08:14:16.922461073 -0500
Change: 2020-11-12 08:14:16.922461073 -0500
 Birth: -

If the permissions of /tmp/christest are reduced for the irods service account...

$ chmod 700 /tmp/christest/
$ sudo su - irods -c 'stat /tmp/christest/newfile'
stat: cannot stat '/tmp/christest/newfile': Permission denied

Does yours succeed?

$ sudo su - irods -c 'stat /home/chrisvdt/images/football.jpg'

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/325b9153-03cc-4561-87ea-7ba46edf31d0n%40googlegroups.com.

[chris vandert]

Okay Terell,

thank you very much, this was helpfull.

Actually, this did not succeed in my case.

$ sudo su - irods -c 'stat /home/chrisvdt/images/football.jpg'

But why doesn't the service account have admin rights?

Op donderdag 12 november 2020 om 15:39:05 UTC+1 schreef Terrell Russell:

[Terrell Russell]

The iRODS service account is just another unix account - we definitely do not want it to have admin/sudo rights on the host computer.

If it does not have permission to see the files its registering/managing, it cannot do its job.

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/fb54ac91-6e36-4358-998a-cb19c6d00162n%40googlegroups.com.