Unable to register files from unixfilesystem

[Marcin Stolarek]

I'm new to irods, so maybe it's something obvious. I want to register files i have on the filesystem to add some metadata and play with irods a little. However when I try to register files I see:

[root@plkra-com05 log]# ireg -C /mnt/plkra/users/qjy4vw/testIrods/ /Test

remote addresses: 10.234.56.28 ERROR: regUtil: reg error for /Test, status = -78000 status = -78000 SYS_RESC_DOES_NOT_EXIST

however the path exists: 

[root@plkra-com05 log]# ls -l /mnt/plkra/users/qjy4vw/testIrods/

total 0

-rw-r--r-- 1 root root 0 Aug 31 13:34 asd

and the resource is defined:

resource name: MarcinProject

id: 10028

zone: XXX

type: unixfilesystem

class: cache

location: plkra-com05

vault: /mnt/plkra/users/qjy4vw/testIrods/

free space:

free space time: : Never

status:

info:

comment:

create time: 01504699688: 2017-09-06.14:08:08

modify time: 01504699688: 2017-09-06.14:08:08

context:

parent:

parent context:

thanks in advance, 
Marcin

[Terrell Russell]

Hi Marcin,

You're missing the -R option to tell the register operation to use the MarcinProject resource.  

$ ireg -h

...

 -R  resource - specifies the target storage resource. This can also be specified

     in your environment or via a rule set up by the administrator.

...

This also suggests your server is still configured to use a different resource by default, possibly one that you've removed (demoResc?).

You need to update /etc/irods/core.re to update the default from demoResc:

acSetRescSchemeForCreate {msiSetDefaultResc("demoResc","null"); }

acSetRescSchemeForRepl {msiSetDefaultResc("demoResc","null"); }

Terrell

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marcin Stolarek]

Terrell, 

Thanks for the hint, I'm one step further, but... what I see know is:

[root@plkra-com05 ~]# ireg -R MarcinProject -C /mnt/plkra/users/qjy4vw/testIrods/ /MarcinProject

remote addresses: 10.234.56.28 ERROR: regUtil: reg error for /MarcinProject, status = -827000 status = -827000 CAT_INVALID_USER

I did iexit/iinit, but it didn't fix nor give any hint..

cheers,

Marcin

[Terrell Russell]

CAT_INVALID_USER suggests that the user you are trying to use to authenticate does not exist in the catalog.  If you're able to do things like 'ils', then this error means you're trying to connect to a different zone (and not finding your username in that other namespace).

/MarcinProject is the logical path you're trying to write to?

Probably you want a full path that looks more like /[zonename]/home/[username]/[collection]

Try

$ ipwd

And then use that fullpath as the prefix for where you would like to register your new collection.

Terrell

--

[Marcin Stolarek]

still no success, what I see looks like:

[root@plkra-com05 ~]# ipwd

/XHPC

[root@plkra-com05 ~]# ils

/XHPC:

  C- /XHPC/home

  C- /XHPC/MarcinTest

  C- /XHPC/trash

[root@plkra-com05 ~]# ireg -R MarcinInternal -C /mnt/plkra/users/qjy4vw/testIrods/ /XHPC/MarcinInternal

remote addresses: 10.234.56.28 ERROR: regUtil: reg error for /XHPC/MarcinInternal, status = -78000 status = -78000 SYS_RESC_DOES_NOT_EXIST

[root@plkra-com05 ~]# ireg -R MarcinInternal -C /mnt/plkra/users/qjy4vw/testIrods/ /XHPC/home/MarcinProject

remote addresses: 10.234.56.28 ERROR: regUtil: reg error for /XHPC/home/MarcinProject, status = -78000 status = -78000 SYS_RESC_DOES_NOT_EXIST

[root@plkra-com05 ~]# ienv

irods_version - 4.2.1

irods_cwd - /XHPC

irods_session_environment_file - /root/.irods/irods_environment.json.9381

irods_environment_file - /root/.irods/irods_environment.json

irods_default_resource - marcinTest

irods_user_name - rods

irods_zone_name - XHPC

irods_port - 1247

irods_host - plkra-com05.europe.delphiauto.net

cheers,

Marcin

You received this message because you are subscribed to a topic in the Google Groups "iRODS-Chat" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/irod-chat/QeOQRaWSITw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to irod-chat+unsubscribe@googlegroups.com.

[Adil Hasan]

Hello Marcin,
I think your resource is named 'MarcinTest' and not 'MarcinInternal'.
hth
adil

> >> email to irod-chat+...@googlegroups.com.

> >> For more options, visit https://groups.google.com/d/optout.
> >>
> >
> > --
> > --
> > "iRODS: the Integrated Rule-Oriented Data-management System; A community
> > driven, open source, data grid software solution" https://www.irods.org
> >
> > iROD-Chat: http://groups.google.com/group/iROD-Chat
> >
> > ---
> > You received this message because you are subscribed to a topic in the
> > Google Groups "iRODS-Chat" group.
> > To unsubscribe from this topic, visit https://groups.google.com/d/
> > topic/irod-chat/QeOQRaWSITw/unsubscribe.
> > To unsubscribe from this group and all its topics, send an email to

> > irod-chat+...@googlegroups.com.

> > For more options, visit https://groups.google.com/d/optout.
> >
>
> --
> --
> "iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
>
> iROD-Chat: http://groups.google.com/group/iROD-Chat
>
> ---

> You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.

[Marcin Stolarek]

yes.. there was an error in the listing I've pasted. Correct me if I'm wrong. My resource name is MarcinProject? 

[root@plkra-com05 tomcat]# ilsresc -l

resource name: MarcinProject

id: 10028

zone: XHPC

type: unixfilesystem

class: cache

location: plkra-com05

vault: /mnt/plkra/users/qjy4vw/testIrods/

free space:

free space time: : Never

status:

info:

comment:

create time: 01504699688: 2017-09-06.14:08:08

modify time: 01504699688: 2017-09-06.14:08:08

context:

parent:

parent context:

[root@plkra-com05 tomcat]# icd /XHPC

[root@plkra-com05 tomcat]# ils

/XHPC:

  C- /XHPC/home

  C- /XHPC/MarcinTest

  C- /XHPC/trash

[root@plkra-com05 tomcat]# ireg -R MarcinProject -C /mnt/plkra/users/qjy4vw/testIrods/ /XHPC/MarcinProject

remote addresses: 10.234.56.28 ERROR: regUtil: reg error for /XHPC/MarcinProject, status = -827000 status = -827000 CAT_INVALID_USER

> >> email to irod-chat+unsubscribe@googlegroups.com.

> >> For more options, visit https://groups.google.com/d/optout.
> >>
> >
> > --
> > --
> > "iRODS: the Integrated Rule-Oriented Data-management System; A community
> > driven, open source, data grid software solution" https://www.irods.org
> >
> > iROD-Chat: http://groups.google.com/group/iROD-Chat
> >
> > ---
> > You received this message because you are subscribed to a topic in the
> > Google Groups "iRODS-Chat" group.
> > To unsubscribe from this topic, visit https://groups.google.com/d/
> > topic/irod-chat/QeOQRaWSITw/unsubscribe.
> > To unsubscribe from this group and all its topics, send an email to

> > irod-chat+unsubscribe@googlegroups.com.

> > For more options, visit https://groups.google.com/d/optout.
> >
>
> --
> --
> "iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution"   https://www.irods.org
>
>  iROD-Chat:  http://groups.google.com/group/iROD-Chat
>
> ---
> You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution"   https://www.irods.org

 iROD-Chat:  http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to a topic in the Google Groups "iRODS-Chat" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/irod-chat/QeOQRaWSITw/unsubscribe.

To unsubscribe from this group and all its topics, send an email to irod-chat+unsubscribe@googlegroups.com.

[Adil Hasan]

Hello Marcin,
sorry you are right indeed. Your resource is named MarcinProject.
hth
adil

> > > >> email to irod-chat+...@googlegroups.com.

> > > >> For more options, visit https://groups.google.com/d/optout.
> > > >>
> > > >
> > > > --
> > > > --
> > > > "iRODS: the Integrated Rule-Oriented Data-management System; A
> > community
> > > > driven, open source, data grid software solution"
> > https://www.irods.org
> > > >
> > > > iROD-Chat: http://groups.google.com/group/iROD-Chat
> > > >
> > > > ---
> > > > You received this message because you are subscribed to a topic in the
> > > > Google Groups "iRODS-Chat" group.
> > > > To unsubscribe from this topic, visit https://groups.google.com/d/
> > > > topic/irod-chat/QeOQRaWSITw/unsubscribe.
> > > > To unsubscribe from this group and all its topics, send an email to

> > > > irod-chat+...@googlegroups.com.

> > > > For more options, visit https://groups.google.com/d/optout.
> > > >
> > >
> > > --
> > > --
> > > "iRODS: the Integrated Rule-Oriented Data-management System; A community
> > driven, open source, data grid software solution" https://www.irods.org
> > >
> > > iROD-Chat: http://groups.google.com/group/iROD-Chat
> > >
> > > ---
> > > You received this message because you are subscribed to the Google
> > Groups "iRODS-Chat" group.
> > > To unsubscribe from this group and stop receiving emails from it, send

> > an email to irod-chat+...@googlegroups.com.

> > > For more options, visit https://groups.google.com/d/optout.
> >
> > --
> > --
> > "iRODS: the Integrated Rule-Oriented Data-management System; A community
> > driven, open source, data grid software solution" https://www.irods.org
> >
> > iROD-Chat: http://groups.google.com/group/iROD-Chat
> >
> > ---
> > You received this message because you are subscribed to a topic in the
> > Google Groups "iRODS-Chat" group.
> > To unsubscribe from this topic, visit https://groups.google.com/d/
> > topic/irod-chat/QeOQRaWSITw/unsubscribe.
> > To unsubscribe from this group and all its topics, send an email to

> > irod-chat+...@googlegroups.com.

> > For more options, visit https://groups.google.com/d/optout.
> >
>
> --
> --
> "iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
>
> iROD-Chat: http://groups.google.com/group/iROD-Chat
>
> ---
> You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.

[Terrell Russell]

What happens when you just `iput` a file rather than `ireg`?  Do you get any resource or user/zone-related errors?

Is there anything helpful in the /var/lib/irods/log/rodsLog* to explain what is happening on the server side?

Are you trying to test federation?  Did you remove demoResc?   Are there any other users on your system or in the other Zone?

Your first error should look something like:

$ ireg -R doesNotExist -C /tmp/irods_sockets_Agib2s/ /NOPEZone/regtest1

rodsLog:

"status [SYS_RESC_DOES_NOT_EXIST]  errno [] -- message [no resource found for name [doesNotExist]]"

In testing this myself, I also saw this in the log:

$ ireg -R newResc -C /tmp/irods_sockets_Agib2s/ /NOPEZone/regtest2

rodsLog:

"status [SYS_INVALID_INPUT_PARAM]  errno [] -- message [a valid zone name does not appear at the root of the object path [/NOPEZone/regtest2]]"

I was able to generate an CAT_INVALID_USER error, but only when I hadn't yet completed all the steps of changing my Zone name from tempZone:

https://docs.irods.org/4.2.1/getting_started/installation/#changing-the-zone-name

rodsLog:

"status [CAT_INVALID_USER]  errno [] -- message [rcAuthCheck failed.]"

It's a configuration mismatch somewhere - I think you're close,

Terrell

> > > >> email to irod-chat+unsubscribe@googlegroups.com.

> > > >> For more options, visit https://groups.google.com/d/optout.
> > > >>
> > > >
> > > > --
> > > > --
> > > > "iRODS: the Integrated Rule-Oriented Data-management System; A
> > community
> > > > driven, open source, data grid software solution"
> > https://www.irods.org
> > > >
> > > > iROD-Chat: http://groups.google.com/group/iROD-Chat
> > > >
> > > > ---
> > > > You received this message because you are subscribed to a topic in the
> > > > Google Groups "iRODS-Chat" group.
> > > > To unsubscribe from this topic, visit https://groups.google.com/d/
> > > > topic/irod-chat/QeOQRaWSITw/unsubscribe.
> > > > To unsubscribe from this group and all its topics, send an email to

> > > > irod-chat+unsubscribe@googlegroups.com.

> > > > For more options, visit https://groups.google.com/d/optout.
> > > >
> > >
> > > --
> > > --
> > > "iRODS: the Integrated Rule-Oriented Data-management System; A community
> > driven, open source, data grid software solution"   https://www.irods.org
> > >
> > >  iROD-Chat:  http://groups.google.com/group/iROD-Chat
> > >
> > > ---
> > > You received this message because you are subscribed to the Google
> > Groups "iRODS-Chat" group.
> > > To unsubscribe from this group and stop receiving emails from it, send

> > an email to irod-chat+unsubscribe@googlegroups.com.

> > > For more options, visit https://groups.google.com/d/optout.
> >
> > --
> > --
> > "iRODS: the Integrated Rule-Oriented Data-management System; A community
> > driven, open source, data grid software solution"   https://www.irods.org
> >
> >  iROD-Chat:  http://groups.google.com/group/iROD-Chat
> >
> > ---
> > You received this message because you are subscribed to a topic in the
> > Google Groups "iRODS-Chat" group.
> > To unsubscribe from this topic, visit https://groups.google.com/d/
> > topic/irod-chat/QeOQRaWSITw/unsubscribe.
> > To unsubscribe from this group and all its topics, send an email to

> > irod-chat+unsubscribe@googlegroups.com.

> > For more options, visit https://groups.google.com/d/optout.
> >
>
> --
> --
> "iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution"   https://www.irods.org
>
>  iROD-Chat:  http://groups.google.com/group/iROD-Chat
>
> ---
> You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution"   https://www.irods.org

 iROD-Chat:  http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+unsubscribe@googlegroups.com.

[Marcin Stolarek]

Terrell,

In fact yes - I've removed demoResc and  I wanted to add a new "iRODS path" bound to my unixfilesystem. When I try to iput in any location it fails (probably becaus there is no resource where data can be stored)...

[root@plkra-com05 qjy4vw]# iput asd .

remote addresses: 10.234.56.28 ERROR: putUtil: put error for /XHPC/home/public/asd, status = -130000 status = -130000 SYS_INVALID_INPUT_PARAM

Level 0: DEBUG:

[root@plkra-com05 qjy4vw]# ls asd

asd

[root@plkra-com05 qjy4vw]# ipwd

/XHPC/home/public

[root@plkra-com05 qjy4vw]# ils -l /XHPC/home/public

/XHPC/home/public:

I didn't attempt to configure the federation, I'm stucked in some basics :) This is just test environment the only user is root.

When I'm creating the resoruce with iadmin mkresc I don't have to specify the "irods path" (not sure if this is correct terminology) that will be handled by this resource, how can I do this? 

cheers,

Marcin

[Terrell Russell]

The `iadmin mkresc` command requires that you define the name of your resource, the type, and then the host and vaultpath.

  Something has gone sideways in that definition, I think.  The `irods` unix service account needs write permission to that vaultpath on the physical disk.

Have you gone through the beginner training?

   https://github.com/irods/irods_training/blob/master/beginner/irods_beginner_training_2017.pdf

We've also got slides on composable resources:

  http://slides.com/irods/ugm2017-composable-resources#/

Again, check your rodsLog and you should see more helpful information than is being sent back to the iCommand that is failing.

Terrell

[Marcin Stolarek]

OK.. I just dropped the mysql database and reconfigured everything from scratch. This time using idamin command with the knowleadge I've gained during last few days. 

Finally I'm backwith ireg issue :)

First ireg run was fine, after some modifications I've checked iscan:

[root@plkra-com05 qjy4vw]# iscan -r  /mnt/plkra/users/qjy4vw/FAS/ | wc -l

29316

but unfortunately ireg (even with -f ends up very quickly without registering any files, the result doesn't differ whether I specify ):

[root@plkra-com05 qjy4vw]# time ireg -f  -R fas   -C /mnt/plkra/users/qjy4vw/FAS/  /HPC/fas

real    0m0.393s

user    0m0.019s

sys     0m0.002s

[root@plkra-com05 qjy4vw]# iscan -r  /mnt/plkra/users/qjy4vw/FAS/ | wc -l

29316

I really appreciate your help.

cheers,
Marcin

[Marcin Stolarek]

one thing I've noticed is that after I issue ireg -f .... command I see two lines in rodsLog:

Sep  7 18:37:44 pid:1075 remote addresses: 10.234.56.28 ERROR: prepareNonServerSharedMemory: failed to get shared memory object [irods_re_cache_shared_memory_irods_rule_language_rule_engine_pid20024_3a83b68025f610d0b4284ab064565630c9467b2f3bbac1610b68fa75736d601ab615a6c1940d5104]. Exception caught [No such file or directory]

Sep  7 18:37:50 pid:1082 remote addresses: 10.234.56.28 ERROR: prepareNonServerSharedMemory: failed to get shared memory object [irods_re_cache_shared_memory_irods_rule_language_rule_engine_pid20024_3a83b68025f610d0b4284ab064565630c9467b2f3bbac1610b68fa75736d601ab615a6c1940d5104]. Exception caught [No such file or directory]

[Marcin Stolarek]

OK... maybe this will help someone in the future, I did first ireg without -R option, so it was registered in demoResc (my default resource) not in the resource which path was correct for that. 
If I'll have some time I'll try to check sources I think it should not be difficult to implement a check in ireg to prevent such an issue... Am I correct that this will be never desired behavior ?

cheers,

Marcin 

[Terrell Russell]

Hi Marcin,

The default resource is demoResc, until you change/override it in the server's policy (/etc/irods/core.re).

It is definitely best practice to do so.

  https://docs.irods.org/4.2.1/system_overview/best_practices/#do-not-use-demoresc-in-production

If things are registered correctly/cleanly, iscan should return no items.

Is there still something that's not behaving as you would like or expect?

Terrell

[Marcin Stolarek]

Unfortunately yes... iscan still shows some missing files and number of those files doesn't change after ireg...

[root@plkra-com05 labelDir]# iscan  -r /mnt/plkra/tmp/irodsTest/  | wc -l

21259

[root@plkra-com05 labelDir]# ireg -f  -R zfas -C /mnt/plkra/tmp/irodsTest  /HPC/zfas                                                                               [root@plkra-com05 labelDir]# iscan  -r /mnt/plkra/tmp/irodsTest/  | wc -l

21259

however when I check arbitrary files from the list returned by iscan, they are actually registered

[root@plkra-com05 labelDir]# iscan  -r /mnt/plkra/tmp/irodsTest/  | head -1

/mnt/plkra/tmp/irodsTest/17CW20_EP/MSR/IN-MC_7353_20160607_232722_split_033.dat is not registered in iRODS

[root@plkra-com05 labelDir]# ils -l /HPC/zfas/17CW20_EP/MSR/IN-MC_7353_20160607_232722_split_033.dat

  rods              0 zfas            0 2017-09-08.13:45 & IN-MC_7353_20160607_232722_split_033.dat

There is one error repeatedly logged into rodsLog:

Sep  9 07:51:12 pid:25286 remote addresses: 10.234.56.28 ERROR: prepareNonServerSharedMemory: failed to get shared memory object [irods_re_cache_shared_memory_irods_rule_language_rule_engine_pid32735_1639d6622e4a0fff086326e3706aec2e3cf281d713c1ed430ed7a1b602033300dcfddb06d1e19545]. Exception caught [No such file or directory]

Do you have any idea what may be the issue?

cheers,

Marcin

[Terrell Russell]

Please share the output of:

$ ilsresc -l zfas

Terrell

[Marcin Stolarek]

[root@plkra-com05 labelDir]# ilsresc -l zfas

resource name: zfas

id: 10016

zone: HPC

type: unixfilesystem

class: cache

location: plkra-com05.europe.delphiauto.net

vault: /mnt/plkra/tmp/irodsTest/

free space:

free space time: : Never

status:

info:

comment:

create time: 01504870196: 2017-09-08.13:29:56

modify time: 01504870902: 2017-09-08.13:41:42

context:

parent:

parent context:

[Terrell Russell]

I think I've figured out what's going on.

On my local machine, I've reproduced the `iscan -r ... | wc -l` confusion.

I think it has to do with trailing slashes...

$ ils -L /tempZone/home/public/destcollection/LICENSE

  rods              0 newResc         1693 2017-09-12.09:25 & LICENSE

        generic    /tmp/thesourceondisk//LICENSE

$ iscan /tmp/thesourceondisk/LICENSE

/tmp/thesourceondisk/LICENSE is not registered in iRODS

ERROR(192) $ iscan /tmp/thesourceondisk//LICENSE

$

Note the "//" in the physical path in the catalog.

When I ask (via iscan) whether the single slash version is in the catalog, it reports no, not registered.

When I ask if the double slash version is in the catalog, it is quiet, meaning, yes, it's registered.

I had registered this content into the system with:

$ time ireg -R newResc -C /tmp/thesourceondisk/ /tempZone/home/public/destcollection

Note the trailing slash on "/tmp/thesourceondisk/".

You can avoid this issue by registering without the trailing slash.  iscan will be happy and report as expected.

I have created a new issue:

  https://github.com/irods/irods/issues/3658

Thanks,

Terrell

[Marcin Stolarek]

I can confirm that checking with iscan and dobule slash it looks like file is registered. Doing iscan with -r still shows those files regrdless of / or // at the end. 

thanks for your help!

cheers,

Marcin