usng ireg fails

148 views
Skip to first unread message

chris vandert

unread,
Nov 8, 2020, 7:51:10 AM11/8/20
to iRODS-Chat

Some help required for a newbi.

ireg -C /home/chrisvdt/images /FYD/home/chrisvdt/nw/images

results in this error message:
remote addresses: 127.0.0.1 ERROR: regUtil: reg error for /FYD/home/chrisvdt/nw/images, status = -511013
status = -511013 UNIX_FILE_CREATE_ERR, Permission denied
 ERROR: main: regUtil error.  status = -511013 UNIX_FILE_CREATE_ERR, Permission denied

Why do I get this error. the irods service account(rods)  has access to these files.

I need to register a huge amount of files on the iRods server, but I cannont simply iput them there.

regards

Tony Edgin

unread,
Nov 9, 2020, 12:35:53 PM11/9/20
to irod...@googlegroups.com
Hi Chris.

I had this same problem on Saturday. Sometimes iRODS needs a hint about which storage resource to use.  The -R option tells ireg which storage resource to use. 

Cheers,
Tony

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/a625ea74-ac7e-4702-96f4-1101cd921cafn%40googlegroups.com.

Terrell Russell

unread,
Nov 9, 2020, 1:08:51 PM11/9/20
to irod...@googlegroups.com
Oh, interesting.   We could probably give a better error message there....

What's the simplest way to reproduce this?



Tony Edgin

unread,
Nov 9, 2020, 2:43:01 PM11/9/20
to irod...@googlegroups.com
I have no idea.  Running `ireg -C -f /vault/path//to/folder /zone/path/to/collection` on a resource server will sometimes register the folder's contents and sometimes it won't. If it fails to register, it will always fail. We only host one resource per resource server. Maybe it's something to do with our resource hierarchy.  Here's a reduced version of its structure.

CyVerseRes:random
├── ds01Broker:passthru
│   └── ds01:unixfilesystem
└── ds02Broker:passthru
    └── ds02:unixfilesystem


Terrell Russell

unread,
Nov 9, 2020, 3:08:39 PM11/9/20
to irod...@googlegroups.com
It would be helpful to see a server-side log (rodsLog - perhaps both provider and consumer) of this event if you can catch one.

Thanks,

Terrell




Coposky, Jason Matthew

unread,
Nov 9, 2020, 3:10:20 PM11/9/20
to irod...@googlegroups.com
Tony,

This will only work if the directory to be registered is on a network shared folder across both servers hosting those storage resources, otherwise the vote may go to a leaf resource which cannot reach out and stat() those files. 

Thanks,

------
Jason Coposky
Executive Director, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
w: (919)445-9675
m: (919)522-0517
jas...@renci.org



From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Tony Edgin <tedgin...@gmail.com>
Sent: Monday, November 9, 2020 2:42 PM
To: irod...@googlegroups.com <irod...@googlegroups.com>
Subject: Re: [iROD-Chat:19345] usng ireg fails
 

Tony Edgin

unread,
Nov 9, 2020, 3:40:49 PM11/9/20
to irod...@googlegroups.com
Hi Jason.

I may be misunderstanding what you are saying, so I'll add some details to what I was saying.

All files being registered are in local storage on a single resource server, e.g., ds01.cyverse.org.  I run ireg on that server. ds01.cyverse.org hosts a single unixfilesystem storage resource, ds01.  `ireg -C -f ...` may or may not work, but `ireg -C -f -R ds01 ...` works. ireg can sometimes figure out the correct storage server, I guess.

Coposky, Jason Matthew

unread,
Nov 9, 2020, 3:51:56 PM11/9/20
to irod...@googlegroups.com
Tony,

Your root resource in that hierarchy is a random node with two leaves which I presume are on separate servers.  If you leave out the -R <target resource> the voting mechanism kicks in and you may or may not get a root resource which is on the same server as your directory of data (unless that directory is shared across both servers at the exact same path).

Hope that makes sense.

Thanks,

------
Jason Coposky
Executive Director, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
w: (919)445-9675
m: (919)522-0517
jas...@renci.org



Sent: Monday, November 9, 2020 3:40 PM
To: irod...@googlegroups.com <irod...@googlegroups.com>
Subject: Re: [iROD-Chat:19348] usng ireg fails
 

Tony Edgin

unread,
Nov 9, 2020, 4:10:16 PM11/9/20
to irod...@googlegroups.com
That makes sense. Thanks for the clarification.

chris vandert

unread,
Nov 11, 2020, 10:24:34 AM11/11/20
to iRODS-Chat
I don't know whats wrong in my case but it never seems to work;
ireg  -C -f -R bundleResc --repl /home/chrisvdt/images  /FYD/home/chrisvdt/images

remote addresses: 127.0.0.1 ERROR: regUtil: reg error for /FYD/home/chrisvdt/images, status = -511013 status = -511013 UNIX_FILE_CREATE_ERR, Permission denied

 ERROR: main: regUtil error.  status = -511013 UNIX_FILE_CREATE_ERR, Permission denied

rodsLog:
Nov 11 16:16:49 pid:2068 remote addresses: 127.0.0.1 ERROR: dirPathReg: rsFileOpendir for /home/chrisvdt/images error, status = -511013


Alternatively:
 ireg  -R demoResc --repl /home/chrisvdt/images/football.jpg /FYD/home/chrisvdt/images/football.jpg
 ERROR: main: regUtil error.  status = -814000 CAT_UNKNOWN_COLLECTION

rodsLog
Nov 11 16:05:19 pid:1816 remote addresses: 127.0.0.1 ERROR: filePathRegRepl: getDataObjInfo for /FYD/home/chrisvdt/images/football.jpg
Op maandag 9 november 2020 om 18:35:53 UTC+1 schreef tedgin...@gmail.com:

Terrell Russell

unread,
Nov 11, 2020, 10:29:02 AM11/11/20
to irod...@googlegroups.com
Not clear what's going on here - but you should definitely not be using bundleResc directly - that's only for internal iRODS things (aka mounted files, tar, zip, etc.).

If the resource you're registering into is connected/hosted on a server that has access to that physical path, then that iRODS server should be happy to stat() the things and register them into the catalog.

Your UNIX_FILE_CREATE_ERR error there suggests permissions on the physical disk.

Terrell





chris vandert

unread,
Nov 11, 2020, 10:45:00 AM11/11/20
to iRODS-Chat
if I use -R demoResc ==>      ERROR: main: regUtil error.  status = -516013 UNIX_FILE_STAT_ERR, Permission denied

or -R demoResc --repl ==>  ERROR: main: regUtil error.  status = -814000 CAT_UNKNOWN_COLLECTION

i've tried all possible combinations without result.

Op woensdag 11 november 2020 om 16:29:02 UTC+1 schreef Terrell Russell:

Terrell Russell

unread,
Nov 11, 2020, 10:56:46 AM11/11/20
to irod...@googlegroups.com
Can you share some relevant logs from /var/lib/irods/log/rodsLog* ?

Shouldn't be any secrets in there...

Terrell



chris vandert

unread,
Nov 11, 2020, 11:20:35 AM11/11/20
to iRODS-Chat
these are the last errors corresponding with my last ireg commands;

Nov 11 16:39:17 pid:2591 remote addresses: 127.0.0.1 ERROR: filePathReg: getFileMetadataFromVault for /FYD/home/chrisvdt/images/football.jpg failed, status = -516013
Nov 11 16:39:55 pid:2611 remote addresses: 127.0.0.1 ERROR: filePathRegRepl: getDataObjInfo for /FYD/home/chrisvdt/images/football.jpg

there are no secrets because this is a demo instance. :-))
the messages don't seem to be very helpfull
Op woensdag 11 november 2020 om 16:56:46 UTC+1 schreef Terrell Russell:

Terrell Russell

unread,
Nov 11, 2020, 1:04:04 PM11/11/20
to irod...@googlegroups.com
It seems the file to be registered cannot be seen by the iRODS server process (aka service account).

$ ierror 516013
irods error: -516013 UNIX_FILE_STAT_ERR Permission denied

Terrell




chris vandert

unread,
Nov 12, 2020, 4:29:13 AM11/12/20
to iRODS-Chat
Here are some details about my system and environment;

A vm with Red Hat Enterprise Linux Server release 7.6
ICAT db  is on MySQL (MariaDB)
my irods service account name is  ;    irods
I changed the ownership on the files to :  chown irods:irods
The files were chmod to 666  rw rw rw

the user (chrisvdt) performing the ireg is a rodsadmin

I don't see how I can make my files more permissive for irods.

Op woensdag 11 november 2020 om 19:04:04 UTC+1 schreef Terrell Russell:

chris vandert

unread,
Nov 12, 2020, 8:00:47 AM11/12/20
to iRODS-Chat
so looking at the source code i see where the error is occuring;

if ( dataObjInfo.dataSize <= 0 &&
  ( dataObjInfo.dataSize = getFileMetadataFromVault( rsComm, &dataObjInfo ) ) < 0 &&
  dataObjInfo.dataSize != UNKNOWN_FILE_SZ ) {
  status = ( int ) dataObjInfo.dataSize;
  rodsLog( LOG_ERROR,
  "filePathReg: getFileMetadataFromVault for %s failed, status = %d",

However, what I don't understand is why is ireg trying to obtain metadata from a file in the vault, since the file is not copyied there in the first place!!!
ireg should only be registering a file in the ICAT database and metadata should be extracted from the file in its original/ physical location.

Op donderdag 12 november 2020 om 10:29:13 UTC+1 schreef chris vandert:

Terrell Russell

unread,
Nov 12, 2020, 8:16:41 AM11/12/20
to irod...@googlegroups.com
Does this work for you?

$ cd /tmp
$ mkdir christest
$ echo "hello" > christest/newfile
$ ireg -C $(pwd)/christest $(ipwd)/ichristest
$ ils -L ichristest
/tempZone/home/rods/ichristest:
  rods              0 demoResc            6 2020-11-12.08:14 & newfile
        generic    /tmp/christest/newfile

Terrell



chris vandert

unread,
Nov 12, 2020, 9:18:57 AM11/12/20
to iRODS-Chat
Yes, to my astonishment it works fine.

Op donderdag 12 november 2020 om 14:16:41 UTC+1 schreef Terrell Russell:

chris vandert

unread,
Nov 12, 2020, 9:20:48 AM11/12/20
to iRODS-Chat
Why does htis make a difference!!!????


Op donderdag 12 november 2020 om 15:18:57 UTC+1 schreef chris vandert:

Terrell Russell

unread,
Nov 12, 2020, 9:39:05 AM11/12/20
to irod...@googlegroups.com
The stat/permissions error you were seeing suggests the service account couldn't see all the way to the file it was asked to register.

$ sudo su - irods -c 'stat /tmp/christest/newfile'
  File: '/tmp/christest/newfile'
  Size: 6               Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d      Inode: 131608      Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/     terrell)   Gid: ( 1000/     terrell)
Access: 2020-11-12 08:14:16.922461073 -0500
Modify: 2020-11-12 08:14:16.922461073 -0500
Change: 2020-11-12 08:14:16.922461073 -0500
 Birth: -

If the permissions of /tmp/christest are reduced for the irods service account...

$ chmod 700 /tmp/christest/
$ sudo su - irods -c 'stat /tmp/christest/newfile'
stat: cannot stat '/tmp/christest/newfile': Permission denied


Does yours succeed?

$ sudo su - irods -c 'stat /home/chrisvdt/images/football.jpg'


Terrell





chris vandert

unread,
Nov 12, 2020, 9:58:57 AM11/12/20
to iRODS-Chat
Okay Terell,

thank you very much, this was helpfull.

Actually, this did not succeed in my case.
$ sudo su - irods -c 'stat /home/chrisvdt/images/football.jpg'

But why doesn't the service account have admin rights?


Op donderdag 12 november 2020 om 15:39:05 UTC+1 schreef Terrell Russell:

Terrell Russell

unread,
Nov 12, 2020, 10:11:07 AM11/12/20
to irod...@googlegroups.com
The iRODS service account is just another unix account - we definitely do not want it to have admin/sudo rights on the host computer.

If it does not have permission to see the files its registering/managing, it cannot do its job.

Terrell




Reply all
Reply to author
Forward
0 new messages