storage tiering not happening

kenneth....@ugent.be

unread,

Mar 29, 2019, 11:50:14 AM3/29/19

to iRODS-Chat

Hi all,

I tried to get the storage tiering working following the https://slides.com/irods/sc18-storage-tiering#/ slides.

I'm running irods 4.2.5, installed the pkg irods-rule-engine-plugin-storage-tiering, added this to server_config

        {
           "instance_name": "irods_rule_engine_plugin-storage_tiering-instance",
           "plugin_name": "irods_rule_engine_plugin-storage_tiering",
           "plugin_specific_configuration": {
                }
        },

added the needed metadata;

[irods@rodsqa02 ~]$ imeta ls -R irodsResc
AVUs defined for resource irodsResc:
attribute: irods::storage_tiering::group
value: tier_group_A
units: 0
----
attribute: irods::storage_tiering::time
value: 5
units:
[irods@rodsqa02 ~]$ imeta ls -R irodsRescArch
AVUs defined for resource irodsRescArch:
attribute: irods::storage_tiering::group
value: tier_group_A
units: 1

and ran the rule

[irods@rodsqa02 ~]$ cat tier_rule.r

{
   "rule-engine-instance-name": "irods_rule_engine_plugin-storage_tiering-instance",
   "rule-engine-operation": "apply_storage_tiering_policy",
   "delay-parameters": "<PLUSET>1s</PLUSET><EF>REPEAT FOR EVER</EF>",
   "storage-tier-groups": [
       "tier_group_A"
   ]
}

[irods@rodsqa02 ~]$ irule -r irods_rule_engine_plugin-storage_tiering-instance -F tier_rule.r
[irods@rodsqa02 ~]$ iput tier_rule.r
[irods@rodsqa02 ~]$ ils -L
/UGentqa02/home/rods:
rods              0 irodsResc          304 2019-03-29.16:30 & tier_rule.r
        generic    /var/lib/irods/Resc01/home/rods/tier_rule.r
[irods@rodsqa02 ~]$ iqstat
id     name
10088 {"rule-engine-operation":"apply_storage_tiering_policy","storage-tier-groups":["tier_group_A"]}

So the rule is in the delay queue, but the file is never moving to the next tier

No clear error in the log file or on command line running the rule, but something clearly is wrong:

Mar 29 16:30:02 pid:23120 remote addresses: 10.141.10.152 ERROR: parse error - unexpected 'p'
Mar 29 16:30:02 pid:23120 remote addresses: 10.141.10.152 ERROR: rsExecMyRule : -66000, [-] /tmp/tmpDKiaty/libirods_rule_engine_plugin-storage_tiering.cpp:185:irods::error exec_rule_text(irods::default_re_ct
x &, const std::string &, msParamArray_t *, const std::string &, irods::callback) : status [SYS_NOT_SUPPORTED] errno [] -- message [parse error - unexpected 'p']

Mar 29 16:30:02 pid:23127 remote addresses: 10.141.10.152 ERROR: parse error - unexpected 'r'
Mar 29 16:30:02 pid:23127 remote addresses: 10.141.10.152 ERROR: rsExecMyRule : -66000, [-] /tmp/tmpDKiaty/libirods_rule_engine_plugin-storage_tiering.cpp:185:irods::error exec_rule_text(irods::default_re_ct
x &, const std::string &, msParamArray_t *, const std::string &, irods::callback) : status [SYS_NOT_SUPPORTED] errno [] -- message [parse error - unexpected 'r']

Someone a clue what I'm missing ?

Thanks!!

Kenneth

Coposky, Jason Matthew

unread,

Mar 29, 2019, 12:38:34 PM3/29/19

to irod...@googlegroups.com

Kenneth,

That error is coming out of the json parser, which implies the rule on the queue is not being parsed properly. We will try to reproduce this issue, is this iRODS zone used for any other purpose or a dev zone? Can you share your /etc/irods/server_config.json offline?

Thanks,

------
Jason Coposky
Executive Director, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
w: (919)445-9675
m: (919)522-0517
jas...@renci.org

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of kenneth....@ugent.be <kenneth....@ugent.be>
Sent: Friday, March 29, 2019 11:50 AM
To: iRODS-Chat
Subject: [iROD-Chat:18106] storage tiering not happening

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org

iROD-Chat: http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Terrell Russell

unread,

Mar 29, 2019, 12:51:19 PM3/29/19

to irod...@googlegroups.com

The unexpected letters coming from the parse error... do they spell anything?

I see a 'p' and an 'r' in the two errors you pasted. Are there more?

Terrell

Kenneth Waegeman

unread,

Apr 1, 2019, 4:09:04 AM4/1/19

to irod...@googlegroups.com, Terrell Russell

Hi Terrell, Jason,

Thanks for looking into this!

I'll send you a copy of my server_config right away. This Zone is a dev setup, just a all-in-one machine for functional tests.

In one of the first attempt, there were other characters in the log:

Mar 29 15:40:01 pid:16628 remote addresses: 10.141.10.152 ERROR: parse error - unexpected 'p'
Mar 29 15:50:02 pid:17597 remote addresses: 10.141.10.152 ERROR: parse error - unexpected 'p'
Mar 29 15:50:02 pid:17604 remote addresses: 10.141.10.152 ERROR: parse error - unexpected 'r'
Mar 29 15:58:29 pid:18572 remote addresses: 10.141.10.152 ERROR: parse error - unexpected ']'
Mar 29 16:00:01 pid:18850 remote addresses: 10.141.10.152 ERROR: parse error - unexpected 'p'
Mar 29 16:00:01 pid:18849 remote addresses: 10.141.10.152 ERROR: parse error - unexpected 'r'
Mar 29 16:02:15 pid:19280 remote addresses: 10.141.10.152 ERROR: parse error - unexpected 'u'
Mar 29 16:02:15 pid:19280 remote addresses: 10.141.10.152 ERROR: rsExecRuleExpression : -66000, [-]    /tmp/tmpDKiaty/libirods_rule_engine_plugin-storage_tiering.cpp:264:irods::error exec_rule_expression(irods::default_re_ctx &, const std::string &, msParamArray_t *, irods::callback) : status [SYS_NOT_SUPPORTED] errno [] -- message [parse error - unexpected 'u']
Mar 29 16:03:16 pid:19514 remote addresses: 10.141.10.152 ERROR: parse error - unexpected 'u'
Mar 29 16:03:16 pid:19514 remote addresses: 10.141.10.152 ERROR: rsExecRuleExpression : -66000, [-]    /tmp/tmpDKiaty/libirods_rule_engine_plugin-storage_tiering.cpp:264:irods::error exec_rule_expression(irods::default_re_ctx &, const std::string &, msParamArray_t *, irods::callback) : status [SYS_NOT_SUPPORTED] errno [] -- message [parse error - unexpected 'u']
Mar 29 16:05:19 pid:19848 remote addresses: 10.141.10.152 ERROR: parse error - unexpected 'u'
Mar 29 16:05:19 pid:19848 remote addresses: 10.141.10.152 ERROR: rsExecRuleExpression : -66000, [-]    /tmp/tmpDKiaty/libirods_rule_engine_plugin-storage_tiering.cpp:264:irods::error exec_rule_expression(irods::default_re_ctx &, const std::string &, msParamArray_t *, irods::callback) : status [SYS_NOT_SUPPORTED] errno [] -- message [parse error - unexpected 'u']
Mar 29 16:09:24 pid:20419 remote addresses: 10.141.10.152 ERROR: parse error - unexpected 'u'
Mar 29 16:09:24 pid:20419 remote addresses: 10.141.10.152 ERROR: rsExecRuleExpression : -66000, [-]    /tmp/tmpDKiaty/libirods_rule_engine_plugin-storage_tiering.cpp:264:irods::error exec_rule_expression(irods::default_re_ctx &, const std::string &, msParamArray_t *, irods::callback) : status [SYS_NOT_SUPPORTED] errno [] -- message [parse error - unexpected 'u']
Mar 29 16:10:02 pid:20535 remote addresses: 10.141.10.152 ERROR: parse error - unexpected 'r'

But since then until now only 'p' and 'r' . It is probable I made a change to the rulefile between those attempts (but I *think* it was just newlines)

Cheers,

Kenneth

You received this message because you are subscribed to a topic in the Google Groups "iRODS-Chat" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/irod-chat/RG1jfTrnT98/unsubscribe.
To unsubscribe from this group and all its topics, send an email to irod-chat+...@googlegroups.com.

kenneth....@ugent.be

unread,

Apr 4, 2019, 5:09:03 AM4/4/19

to iRODS-Chat

Hi,

I also see this on our other environments, where I only installed the tiering rpm and the plugin in server_config.

I didn't set any tiering metadata or delay rule

Only this in server_config:

"rule_engines" : [

         {
            "instance_name" : "irods_rule_engine_plugin-storage_tiering-instance",
            "plugin_name" : "irods_rule_engine_plugin-storage_tiering",
            "plugin_specific_configuration" : {}
         },

{
"instance_name" : "irods_rule_engine_plugin-irods_rule_language-instance",

.....

I also got an additional character there :)

Apr 4 10:52:26 pid:1940283 remote addresses: 10.141.10.151, 157.193.252.76 ERROR: parse error - unexpected 'm'
Apr 4 10:52:26 pid:1940283 remote addresses: 10.141.10.151, 157.193.252.76 ERROR: rsExecMyRule : -66000, [-] /tmp/tmpDKiaty/libirods_rule_engine_plugin-storage_tiering.cpp:185:irods::error exec_rule_text(irods::default_re_ctx &, const std::string &, msParamArray_t *, const std::string &, irods::callback) : status [SYS_NOT_SUPPORTED] errno [] -- message [parse error - unexpected 'm']

Thanks again!

Kenneth

Alan King

unread,

Apr 4, 2019, 11:57:48 AM4/4/19

to irod...@googlegroups.com

Hi Kenneth,

The problem here is a bug in genQuery which is being tripped by storage tiering. Your resource "irodsResc" is a substring of "irodsRescArch". The implementation of the "IN" construct in genQuery, which is in use in the storage tiering plugin, has a bug with respect to handling substrings in this manner. If you rename your resources such that they are not substrings of one another, storage tiering will work as expected.

Thanks!

Alan

--

Kenneth Waegeman

unread,

Apr 9, 2019, 5:22:14 AM4/9/19

to irod...@googlegroups.com, Alan King

Hi Alan,

That did the trick, tiering works now!

Thanks!

Kenneth

You received this message because you are subscribed to a topic in the Google Groups "iRODS-Chat" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/irod-chat/RG1jfTrnT98/unsubscribe.
To unsubscribe from this group and all its topics, send an email to irod-chat+...@googlegroups.com.

Kenneth Waegeman

unread,

Apr 9, 2019, 8:48:30 AM4/9/19

to irod...@googlegroups.com, Alan King

Hi all,

Well, it started working, but very soon I saw issues again. Tiering to the archive seems to work ok, but tiering back does not always work

I first saw an error I even couldn't access the file:

[kwaegema@rodsqa02 ~]$ ils -L /UGentqa02/home/kwaegema: kwaegema 1 irodsArch 76 2019-04-09.12:50 & V1.0.1__BasicAdvance_version.sql generic /var/lib/irods/Resc03/home/kwaegema/V1.0.1__BasicAdvance_version.sql [kwaegema@rodsqa02 ~]$ iget V1.0.1__BasicAdvance_version.sql - ALTER TABLE users ADD COLUMN advanced_view BOOLEAN NOT NULL DEFAULT FALSE;[kwaegema@ls -L /UGentqa02/home/kwaegema: kwaegema 2 irodsResc 76 2019-04-09.12:54 & V1.0.1__BasicAdvance_version.sql generic /var/lib/irods/Resc01/home/kwaegema/V1.0.1__BasicAdvance_version.sql kwaegema 3 irodsResc 76 2019-04-09.12:54 & V1.0.1__BasicAdvance_version.sql generic /var/lib/irods/Resc01/home/kwaegema/V1.0.1__BasicAdvance_version.sql.0 /UGentqa02/home/kwaegema: kwaegema 2 irodsResc 76 2019-04-09.12:54 & V1.0.1__BasicAdvance_version.sql generic /var/lib/irods/Resc01/home/kwaegema/V1.0.1__BasicAdvance_version.sql kwaegema 3 irodsResc 76 2019-04-09.12:54 & V1.0.1__BasicAdvance_version.sql generic /var/lib/irods/Resc01/home/kwaegema/V1.0.1__BasicAdvance_version.sql.0 [kwaegema@rodsqa02 ~]$ iget V1.0.1__BasicAdvance_version.sql - remote addresses: 10.141.10.152 ERROR: getUtil: get error for - status = -510002 UNIX_FILE_OPEN_ERR, No such file or directory [kwaegema@rodsqa02 ~]$ ils -L /UGentqa02/home/kwaegema: kwaegema 2 irodsResc 76 2019-04-09.12:54 & V1.0.1__BasicAdvance_version.sql generic /var/lib/irods/Resc01/home/kwaegema/V1.0.1__BasicAdvance_version.sql kwaegema 3 irodsResc 76 2019-04-09.12:54 & V1.0.1__BasicAdvance_version.sql generic /var/lib/irods/Resc01/home/kwaegema/V1.0.1__BasicAdvance_version.sql.0

irods@rodsqa02 ~]$ tail log/rodsLog.2019.04.06 -n 100 [-] /tmp/tmppTB_kL/plugins/resources/unixfilesystem/libunixfilesystem.cpp:724:irods::error unix_file_open(irods::plugin_context &) : status [UNIX_FILE_OPEN_ERR] errno [No such file or directory] -- message [Open error for "/var/lib/irods/Resc01/home/kwaegema/V1.0.1__BasicAdvance_version.sql", errno = "No such file or directory", status = "-510002", flags = "0".] Apr 9 12:55:26 pid:14586 NOTICE: _rsFileGet: fileGet for /var/lib/irods/Resc01/home/kwaegema/V1.0.1__BasicAdvance_version.sql, status = -510002 Apr 9 12:55:30 pid:14595 NOTICE: writeLine: inString = Agent process started for puser=kwaegema and cuser=kwaegema from 10.141.10.152 Apr 9 12:55:32 pid:14606 NOTICE: writeLine: inString = Agent process started for puser=kwaegema and cuser=kwaegema from 10.141.10.152 Apr 9 12:55:35 pid:14615 NOTICE: writeLine: inString = Agent process started for puser=kwaegema and cuser=kwaegema from 10.141.10.152 Apr 9 12:55:49 pid:14630 NOTICE: writeLine: inString = Agent process started for puser=rods and cuser=rods from 10.141.10.152 Apr 9 12:55:49 pid:14636 NOTICE: writeLine: inString = Agent process started for puser=rods and cuser=rods from 10.141.10.152 Apr 9 12:55:49 pid:14642 NOTICE: writeLine: inString = Agent process started for puser=rods and cuser=rods from 10.141.10.152 Apr 9 12:55:50 pid:14704 NOTICE: writeLine: inString = Agent process started for puser=rods and cuser=rods from 10.141.10.152 Apr 9 12:55:50 pid:14710 NOTICE: writeLine: inString = Agent process started for puser=rods and cuser=rods from 10.141.10.152 Apr 9 12:55:50 pid:14636 NOTICE: _rsGenQuery: genQuery status = -818000 Apr 9 12:55:50 pid:14636 NOTICE: rsGenQuery: _rsGenQuery failed, status = -818000 Apr 9 12:55:50 pid:14636 NOTICE: getDataObjInfo: rsGenQuery error, status = -818000 Apr 9 12:55:50 pid:14636 remote addresses: 10.141.10.152 ERROR: [-] /tmp/tmppTB_kL/server/api/src/rsFileOpen.cpp:164:int _rsFileOpen(rsComm_t *, fileOpenInp_t *) : status [UNIX_FILE_OPEN_ERR] errno [No such file or directory] -- message [Open error for "/var/lib/irods/Resc01/home/kwaegema/V1.0.1__BasicAdvance_version.sql", errno = "No such file or directory", status = "-510002", flags = "0".] [-] /tmp/tmppTB_kL/server/drivers/src/fileDriver.cpp:69:irods::error fileOpen(rsComm_t *, irods::first_class_object_ptr) : status [UNIX_FILE_OPEN_ERR] errno [No such file or directory] -- message [Open error for "/var/lib/irods/Resc01/home/kwaegema/V1.0.1__BasicAdvance_version.sql", errno = "No such file or directory", status = "-510002", flags = "0".] [-] /tmp/tmppTB_kL/plugins/resources/unixfilesystem/libunixfilesystem.cpp:724:irods::error unix_file_open(irods::plugin_context &) : status [UNIX_FILE_OPEN_ERR] errno [No such file or directory] -- message [Open error for "/var/lib/irods/Resc01/home/kwaegema/V1.0.1__BasicAdvance_version.sql", errno = "No such file or directory", status = "-510002", flags = "0".] Apr 9 12:55:50 pid:14636 NOTICE: _rsFileGet: fileGet for /var/lib/irods/Resc01/home/kwaegema/V1.0.1__BasicAdvance_version.sql, status = -510002 Apr 9 12:55:50 pid:14642 NOTICE: _rsDataObjRepl - Failed to resolve a single replication copy. Apr 9 12:55:50 pid:14642 NOTICE: rsDataObjRepl - Failed to replicate data object. Apr 9 12:55:50 pid:14642 remote addresses: 10.141.10.152 ERROR: rsExecRuleExpression : -808000, [-] /tmp/tmpDKiaty/libirods_rule_engine_plugin-storage_tiering.cpp:250:irods::error exec_rule_expression(irods::default_re_ctx &, const std::string &, msParamArray_t *, irods::callback) : status [CAT_NO_ROWS_FOUND] errno [] -- message [iRODS Exception: file: /tmp/tmpDKiaty/irods_object_migrator.cpp function: void irods::object_migrator::replicate_object_to_resource(const std::string &, const std::string &, const std::string &, const std::string &) line: 185 code: -808000 (CAT_NO_ROWS_FOUND) message: failed to migrate [/UGentqa02/home/kwaegema/V1.0.1__BasicAdvance_version.sql] to [irodsResc] stack trace: -------------- Dumping stack trace <0> Offset: 0x65 Address: 0x7f3c1887ad95 irods::exception::exception(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <1> Offset: 0x43 Address: 0x7f3c1887af13 irods::exception::exception(long, boost::basic_format<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <2> Offset: 0x554 Address: 0x7f3c15e64614 irods::object_migrator::replicate_object_to_resource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <3> Offset: 0x10b Address: 0x7f3c15e6315b irods::object_migrator::operator()() <4> Offset: 0xc3 Address: 0x7f3c15e75c33 irods::storage_tiering::move_data_object(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <5> Offset: 0x3f2 Address: 0x7f3c15e4d052 exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback) <6> Offset: 0x2f Address: 0x7f3c15e62d4f std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*&&, irods::callback&&) <7> Offset: 0xf0 Address: 0x7f3c1b904a90 irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback) <8> Offset: 0x354 Address: 0x7f3c1b904924 irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule_expression(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*) <9> Offset: 0x25e Address: 0x7f3c1b9042fe rsExecRuleExpression(rsComm_t*, exec_rule_expression_t*) <10> Offset: 0x44 Address: 0x7f3c1bb5da24 irods::api_call_adaptor<exec_rule_expression_t*>::operator()(irods::plugin_context&, rsComm_t*, exec_rule_expression_t*) <11> Offset: 0x13 Address: 0x7f3c1bb5d9a3 std::__1::__function::__func<irods::api_call_adaptor<exec_rule_expression_t*>, std::__1::allocator<irods::api_call_adaptor<exec_rule_expression_t*> >, irods::error (irods::plugin_context&, rsComm_t*, exec_rule_expression_t*)>::operator()(irods::plugin_context&, rsComm_t*&&, exec_rule_expression_t*&&) <12> Offset: 0x93b Address: 0x7f3c1ba512cb int irods::api_entry::call_handler<exec_rule_expression_t*>(rsComm_t*, exec_rule_expression_t*) <13> Offset: 0x75b Address: 0x7f3c1bbdfdcb rsApiHandler(rsComm_t*, int, BytesBuf*, BytesBuf*) <14> Offset: 0x8f2 Address: 0x7f3c1bbe1b42 readAndProcClientMsg(rsComm_t*, int) <15> Offset: 0x9c7 Address: 0x7f3c1bbd55e7 agentMain(rsComm_t*) <16> Offset: 0x1cc3 Address: 0x7f3c1bbd3f63 runIrodsAgentFactory(sockaddr_un) <17> Offset: Address: 0x4215ad /usr/sbin/irodsServer(main+0x4dd) [0x4215ad] <18> Offset: Address: 0x7f3c17f893d5 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f3c17f893d5] <19> Offset: Address: 0x42100d /usr/sbin/irodsServer() [0x42100d] ]

I am not able to reproduce this every time, eg next test files I didn't get a problem accessing the file, but the errors were in the logs, and still showing after irm:

Apr 9 14:34:03 pid:26607 NOTICE: rsApiHandler: User has no permission for apiNumber 708 Apr 9 14:34:03 pid:26616 NOTICE: _rsDataObjRepl - Failed to resolve a single replication copy. Apr 9 14:34:03 pid:26616 NOTICE: rsDataObjRepl - Failed to replicate data object. Apr 9 14:34:03 pid:26595 NOTICE: _rsDataObjRepl - Failed to resolve a single replication copy. Apr 9 14:34:03 pid:26595 NOTICE: rsDataObjRepl - Failed to replicate data object. Apr 9 14:34:03 pid:26601 NOTICE: _rsDataObjRepl - Failed to resolve a single replication copy. Apr 9 14:34:03 pid:26601 NOTICE: rsDataObjRepl - Failed to replicate data object. Apr 9 14:34:03 pid:26616 remote addresses: 10.141.10.152 ERROR: rsExecRuleExpression : -808000, [-] /tmp/tmpDKiaty/libirods_rule_engine_plugin-storage_tiering.cpp:250:irods::error exec_rule_expression(irods::default_re_ctx &, const std::string &, msParamArray_t *, irods::callback) : status [CAT_NO_ROWS_FOUND] errno [] -- message [iRODS Exception: file: /tmp/tmpDKiaty/irods_object_migrator.cpp function: void irods::object_migrator::replicate_object_to_resource(const std::string &, const std::string &, const std::string &, const std::string &) line: 185 code: -808000 (CAT_NO_ROWS_FOUND) message: failed to migrate [/UGentqa02/home/kwaegema/ctest] to [irodsResc] stack trace: -------------- Dumping stack trace <0> Offset: 0x65 Address: 0x7f3c1887ad95 irods::exception::exception(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <1> Offset: 0x43 Address: 0x7f3c1887af13 irods::exception::exception(long, boost::basic_format<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <2> Offset: 0x554 Address: 0x7f3c15e64614 irods::object_migrator::replicate_object_to_resource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <3> Offset: 0x10b Address: 0x7f3c15e6315b irods::object_migrator::operator()() <4> Offset: 0xc3 Address: 0x7f3c15e75c33 irods::storage_tiering::move_data_object(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <5> Offset: 0x3f2 Address: 0x7f3c15e4d052 exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback) <6> Offset: 0x2f Address: 0x7f3c15e62d4f std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*&&, irods::callback&&) <7> Offset: 0xf0 Address: 0x7f3c1b904a90 irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback) <8> Offset: 0x354 Address: 0x7f3c1b904924 irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule_expression(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*) <9> Offset: 0x25e Address: 0x7f3c1b9042fe rsExecRuleExpression(rsComm_t*, exec_rule_expression_t*) <10> Offset: 0x44 Address: 0x7f3c1bb5da24 irods::api_call_adaptor<exec_rule_expression_t*>::operator()(irods::plugin_context&, rsComm_t*, exec_rule_expression_t*) <11> Offset: 0x13 Address: 0x7f3c1bb5d9a3 std::__1::__function::__func<irods::api_call_adaptor<exec_rule_expression_t*>, std::__1::allocator<irods::api_call_adaptor<exec_rule_expression_t*> >, irods::error (irods::plugin_context&, rsComm_t*, exec_rule_expression_t*)>::operator()(irods::plugin_context&, rsComm_t*&&, exec_rule_expression_t*&&) <12> Offset: 0x93b Address: 0x7f3c1ba512cb int irods::api_entry::call_handler<exec_rule_expression_t*>(rsComm_t*, exec_rule_expression_t*) <13> Offset: 0x75b Address: 0x7f3c1bbdfdcb rsApiHandler(rsComm_t*, int, BytesBuf*, BytesBuf*) <14> Offset: 0x8f2 Address: 0x7f3c1bbe1b42 readAndProcClientMsg(rsComm_t*, int) <15> Offset: 0x9c7 Address: 0x7f3c1bbd55e7 agentMain(rsComm_t*) <16> Offset: 0x1cc3 Address: 0x7f3c1bbd3f63 runIrodsAgentFactory(sockaddr_un) <17> Offset: Address: 0x4215ad /usr/sbin/irodsServer(main+0x4dd) [0x4215ad] <18> Offset: Address: 0x7f3c17f893d5 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f3c17f893d5] <19> Offset: Address: 0x42100d /usr/sbin/irodsServer() [0x42100d] ]

Apr 9 14:34:03 pid:26595 NOTICE: rsApiHandler: User has no permission for apiNumber 708Apr 9 14:34:03 pid:26607 remote addresses: 10.141.10.152 ERROR: [-] /tmp/tmppTB_kL/server/api/src/rsDataObjRepl.cpp:153:int rsDataObjRepl(rsComm_t *, dataObjInp_t *, transferStat_t **) : status [CAT_NO_ROWS_FOUND] errno [] -- message [failed in call to getDataObjInfoIncSpecColl for [/UGentqa02/home/kwaegema/btest] CAT_NO_ROWS_FOUND ] [-] /tmp/tmppTB_kL/server/core/src/irods_resource_redirect.cpp:610:irods::error irods::resolve_resource_hierarchy(const std::string &, rsComm_t *, dataObjInp_t *, std::string &, dataObjInfo_t **) : status [CAT_NO_ROWS_FOUND] errno [] -- message [failed in call to getDataObjInfoIncSpecColl for [/UGentqa02/home/kwaegema/btest] CAT_NO_ROWS_FOUND ] [-] /tmp/tmppTB_kL/server/core/src/irods_file_object.cpp:334:irods::error irods::file_object_factory(rsComm_t *, dataObjInp_t *, file_object_ptr, dataObjInfo_t **) : status [CAT_NO_ROWS_FOUND] errno [] -- message [failed in call to getDataObjInfoIncSpecColl for [/UGentqa02/home/kwaegema/btest] CAT_NO_ROWS_FOUND ]Apr 9 14:34:03 pid:26607 remote addresses: 10.141.10.152 ERROR: rsExecRuleExpression : -808000, [-] /tmp/tmpDKiaty/libirods_rule_engine_plugin-storage_tiering.cpp:250:irods::error exec_rule_expression(irods::default_re_ctx &, const std::string &, msParamArray_t *, irods::callback) : status [CAT_NO_ROWS_FOUND] errno [] -- message [iRODS Exception: file: /tmp/tmpDKiaty/irods_object_migrator.cpp function: void irods::object_migrator::replicate_object_to_resource(const std::string &, const std::string &, const std::string &, const std::string &) line: 185 code: -808000 (CAT_NO_ROWS_FOUND) message: failed to migrate [/UGentqa02/home/kwaegema/btest] to [irodsResc]stack trace:--------------Dumping stack trace<0> Offset: 0x65 Address: 0x7f3c1887ad95 irods::exception::exception(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)<1> Offset: 0x43 Address: 0x7f3c1887af13 irods::exception::exception(long, boost::basic_format<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)<2> Offset: 0x554 Address: 0x7f3c15e64614 irods::object_migrator::replicate_object_to_resource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)<3> Offset: 0x10b Address: 0x7f3c15e6315b irods::object_migrator::operator()()<4> Offset: 0xc3 Address: 0x7f3c15e75c33 irods::storage_tiering::move_data_object(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)<5> Offset: 0x3f2 Address: 0x7f3c15e4d052 exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)<6> Offset: 0x2f Address: 0x7f3c15e62d4f std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*&&, irods::callback&&)<7> Offset: 0xf0 Address: 0x7f3c1b904a90 irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)<8> Offset: 0x354 Address: 0x7f3c1b904924 irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule_expression(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*)<9> Offset: 0x25e Address: 0x7f3c1b9042fe rsExecRuleExpression(rsComm_t*, exec_rule_expression_t*)<10> Offset: 0x44 Address: 0x7f3c1bb5da24 irods::api_call_adaptor<exec_rule_expression_t*>::operator()(irods::plugin_context&, rsComm_t*, exec_rule_expression_t*)<11> Offset: 0x13 Address: 0x7f3c1bb5d9a3 std::__1::__function::__func<irods::api_call_adaptor<exec_rule_expression_t*>, std::__1::allocator<irods::api_call_adaptor<exec_rule_expression_t*> >, irods::error (irods::plugin_context&, rsComm_t*, exec_rule_expression_t*)>::operator()(irods::plugin_context&, rsComm_t*&&, exec_rule_expression_t*&&)<12> Offset: 0x93b Address: 0x7f3c1ba512cb int irods::api_entry::call_handler<exec_rule_expression_t*>(rsComm_t*, exec_rule_expression_t*)<13> Offset: 0x75b Address: 0x7f3c1bbdfdcb rsApiHandler(rsComm_t*, int, BytesBuf*, BytesBuf*)<14> Offset: 0x8f2 Address: 0x7f3c1bbe1b42 readAndProcClientMsg(rsComm_t*, int)<15> Offset: 0x9c7 Address: 0x7f3c1bbd55e7 agentMain(rsComm_t*)<16> Offset: 0x1cc3 Address: 0x7f3c1bbd3f63 runIrodsAgentFactory(sockaddr_un)<17> Offset: Address: 0x4215ad /usr/sbin/irodsServer(main+0x4dd) [0x4215ad]<18> Offset: Address: 0x7f3c17f893d5 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f3c17f893d5]<19> Offset: Address: 0x42100d /usr/sbin/irodsServer() [0x42100d]]

I saw also things like this, not sure if this is how tiering is supposed to work:

kwaegema@rodsqa02 ~]$ ils -L
/UGentqa02/home/kwaegema:
kwaegema 2 irodsResc 10 2019-04-09.14:19 & atestfile
generic /var/lib/irods/Resc01/replica/home/kwaegema/atestfile.2582636266

My resources were now renamed:

[kwaegema@rodsqa02 ~]$ ilsresc
demoResc:unixfilesystem
irodsArch:unixfilesystem
irodsRepl:unixfilesystem
irodsResc:unixfilesystem

Someone a clue what is happening?

Thanks a lot !!

Kenneth

Alan King

unread,

Apr 12, 2019, 3:35:48 PM4/12/19

to irod...@googlegroups.com

Hi Kenneth,

Thanks for the logs and info. Could you please recount the order of events from the client and the server side as part of your story? The usual order of events for this kind of scenario when things are working as expected might be, e.g.:

1. File put to tier 0 resource

2. File tiered out to tier 1 resource

3. Get file from tier 1 resource

4. File restaged to tier 0 resource

5. File tiered out to tier 1 resource

It looks like something goes wrong between step 3 and step 4 from your story so far. The following pieces of information can help us figure out what might be going on, if you could please provide them:

- Another printout of the metadata query results from your unixfilesystem resources (irodsArch, irodsRepl, and irodsResc) for knowing how the tier group(s) is configured.

- Metadata query results for the problematic data object (before/after tiering out, before/after restage) (Is the access time being updated properly?)

- Is the file on the disk in the correct location with the correct contents (UNIX_FILE_OPEN_ERR says it's not there, but the catalog says it is)?

- What appears in the log after the first attempt to get the file? This will reveal how the restage fails.

In the meantime, have you tried removing/re-adding the resources from/to the tier group(s)? I wonder if renaming the resource trips some bug in iRODS or the plugin somehow...

Thanks!

Alan

Kenneth Waegeman

unread,

May 7, 2019, 12:45:58 PM5/7/19

to irod...@googlegroups.com, Alan King

Hi Alan,

It took some time, but I finally got back to the issue: I removed and readded the metadata to the tier groups, but that didnt solve it

I've got the requested data here,

so the tier groups

[irods@rodsqa02 ~]$ imeta ls -R irodsResc
AVUs defined for resource irodsResc:
attribute: irods::storage_tiering::group
value: tier_group_A
units: 0
----
attribute: irods::storage_tiering::time

value: 60
units:
[irods@rodsqa02 ~]$ imeta ls -R irodsArch
AVUs defined for resource irodsArch:

attribute: irods::storage_tiering::group
value: tier_group_A
units: 1

So this happens when I try to get file that is in archive resource Resc03

/UGentqa02/home/kwaegema:
  kwaegema          0 irodsResc           10 2019-05-07.18:29 & copy1
        generic    /var/lib/irods/Resc01/home/kwaegema/copy1
  kwaegema          1 irodsArch           10 2019-05-07.18:31 & copy2
        generic    /var/lib/irods/Resc03/home/kwaegema/copy2
  kwaegema          2 irodsResc           10 2019-05-07.18:28 & newtesta
        generic    /var/lib/irods/Resc01/home/kwaegema/newtesta
[kwaegema@rodsqa02 ~]$ ils -L
/UGentqa02/home/kwaegema:
  kwaegema          0 irodsResc           10 2019-05-07.18:29 & copy1
        generic    /var/lib/irods/Resc01/home/kwaegema/copy1
  kwaegema          1 irodsArch           10 2019-05-07.18:31 & copy2
        generic    /var/lib/irods/Resc03/home/kwaegema/copy2
  kwaegema          2 irodsResc           10 2019-05-07.18:28 & newtesta
        generic    /var/lib/irods/Resc01/home/kwaegema/newtesta
[kwaegema@rodsqa02 ~]$ iget copy2 -
justatest
[kwaegema@rodsqa02 ~]$ ils -L
/UGentqa02/home/kwaegema:
  kwaegema          1 irodsArch           10 2019-05-07.18:31 & copy1
        generic    /var/lib/irods/Resc03/home/kwaegema/copy1
  kwaegema          2 irodsResc           10 2019-05-07.18:31 & copy2
        generic    /var/lib/irods/Resc01/replica/home/kwaegema/copy2.3856308997
  kwaegema          3 irodsResc           10 2019-05-07.18:31 & copy2
        generic    /var/lib/irods/Resc01/home/kwaegema/copy2
  kwaegema          3 irodsArch           10 2019-05-07.18:31 & newtesta
        generic    /var/lib/irods/Resc03/home/kwaegema/newtesta
[kwaegema@rodsqa02 ~]$ iget copy2 -
justatest
[kwaegema@rodsqa02 ~]$ ils -L
/UGentqa02/home/kwaegema:
  kwaegema          1 irodsArch           10 2019-05-07.18:31 & copy1
        generic    /var/lib/irods/Resc03/home/kwaegema/copy1
  kwaegema          2 irodsResc           10 2019-05-07.18:31 & copy2
        generic    /var/lib/irods/Resc01/replica/home/kwaegema/copy2.3856308997
  kwaegema          3 irodsResc           10 2019-05-07.18:31 & copy2
        generic    /var/lib/irods/Resc01/home/kwaegema/copy2
  kwaegema          3 irodsArch           10 2019-05-07.18:31 & newtesta
        generic    /var/lib/irods/Resc03/home/kwaegema/newtesta
[kwaegema@rodsqa02 ~]$ iget copy2 -
justatest
[kwaegema@rodsqa02 ~]$ ils -L
/UGentqa02/home/kwaegema:
  kwaegema          1 irodsArch           10 2019-05-07.18:31 & copy1
        generic    /var/lib/irods/Resc03/home/kwaegema/copy1
  kwaegema          2 irodsResc           10 2019-05-07.18:31 & copy2
        generic    /var/lib/irods/Resc01/replica/home/kwaegema/copy2.3856308997
  kwaegema          3 irodsArch           10 2019-05-07.18:31 & newtesta
        generic    /var/lib/irods/Resc03/home/kwaegema/newtesta

The first error I saw:

May 7 18:31:43 pid:3594 remote addresses: 10.141.10.152 ERROR: [-]   /tmp/tmppTB_kL/server/api/src/rsFileCreate.cpp:196:int _rsFileCreate(rsComm_t *, fileCreateInp_t *, rodsServerHost_t *, fileCreateOut_t **) : status [UNIX_FILE_RMDIR_ERR] errno [Not a directory] -- message [Rmdir error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "Not a directory", status = -521020.]
   [-]   /tmp/tmppTB_kL/server/drivers/src/fileDriver.cpp:352:irods::error fileRmdir(rsComm_t *, irods::first_class_object_ptr) : status [UNIX_FILE_RMDIR_ERR] errno [Not a directory] -- message [Rmdir error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "Not a directory", status = -521020.]
       [-]   /tmp/tmppTB_kL/plugins/resources/unixfilesystem/libunixfilesystem.cpp:1020:irods::error unix_file_rmdir(irods::plugin_context &) : status [UNIX_FILE_RMDIR_ERR] errno [Not a directory] -- message [Rmdir error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "Not a directory", status = -521020.]

May 7 18:31:43 pid:3594 remote addresses: 10.141.10.152 ERROR: [-]   /tmp/tmppTB_kL/server/api/src/rsFileCreate.cpp:210:int _rsFileCreate(rsComm_t *, fileCreateInp_t *, rodsServerHost_t *, fileCreateOut_t **) : status [UNIX_FILE_CREATE_ERR] errno [File exists] -- message [create error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "File exists".]
   [-]   /tmp/tmppTB_kL/server/drivers/src/fileDriver.cpp:38:irods::error fileCreate(rsComm_t *, irods::first_class_object_ptr) : status [UNIX_FILE_CREATE_ERR] errno [File exists] -- message [create error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "File exists".]
       [-]   /tmp/tmppTB_kL/plugins/resources/unixfilesystem/libunixfilesystem.cpp:625:irods::error unix_file_create(irods::plugin_context &) : status [UNIX_FILE_CREATE_ERR] errno [File exists] -- message [create error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "File exists".]

May 7 18:31:43 pid:3594 NOTICE: bindVar[1]=16204
May 7 18:31:43 pid:3594 NOTICE: bindVar[2]=16101
May 7 18:31:43 pid:3594 NOTICE: bindVar[3]=copy2
May 7 18:31:43 pid:3594 NOTICE: bindVar[4]=2
May 7 18:31:43 pid:3594 NOTICE: bindVar[5]=
May 7 18:31:43 pid:3594 NOTICE: bindVar[6]=generic
May 7 18:31:43 pid:3594 NOTICE: bindVar[7]=10
May 7 18:31:43 pid:3594 NOTICE: bindVar[8]=EMPTY_RESC_GROUP_NAME
May 7 18:31:43 pid:3594 NOTICE: bindVar[9]=EMPTY_RESC_NAME
May 7 18:31:43 pid:3594 NOTICE: bindVar[10]=EMPTY_RESC_HIER
May 7 18:31:43 pid:3594 NOTICE: bindVar[11]=10015
May 7 18:31:43 pid:3594 NOTICE: bindVar[12]=/var/lib/irods/Resc01/home/kwaegema/copy2
May 7 18:31:43 pid:3594 NOTICE: bindVar[13]=kwaegema
May 7 18:31:43 pid:3594 NOTICE: bindVar[14]=UGentqa02
May 7 18:31:43 pid:3594 NOTICE: bindVar[15]=1
May 7 18:31:43 pid:3594 NOTICE: bindVar[16]=
May 7 18:31:43 pid:3594 NOTICE: bindVar[17]=
May 7 18:31:43 pid:3594 NOTICE: bindVar[18]=00000000000
May 7 18:31:43 pid:3594 NOTICE: bindVar[19]=0
May 7 18:31:43 pid:3594 NOTICE: bindVar[20]=0
May 7 18:31:43 pid:3594 NOTICE: bindVar[21]=
May 7 18:31:43 pid:3594 NOTICE: bindVar[22]=01557246703
May 7 18:31:43 pid:3594 NOTICE: bindVar[23]=01557246703
May 7 18:31:43 pid:3594 NOTICE: bindVar[24]=10015
May 7 18:31:43 pid:3594 NOTICE: bindVar[25]=/var/lib/irods/Resc01/home/kwaegema/copy2
May 7 18:31:43 pid:3594 NOTICE: bindVar[26]=16204
May 7 18:31:43 pid:3594 NOTICE: _cllExecSqlNoResult: SQLExecDirect error: -1 sql:insert into R_DATA_MAIN ( data_id,                        coll_id,                         data_name,                        data_repl_num,                        data_version,                        data_type_name,                        data_size,                        resc_group_name,                        resc_name,                        resc_hier,                        resc_id,                        data_path,                        data_owner_name,                        data_owner_zone,                        data_is_dirty,                        data_status,                        data_checksum,                        data_expiry_ts,                        data_map_id,                        data_mode,                        r_comment,                        create_ts,                        modify_ts ) select ?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,? where not exists (select data_id from R_DATA_MAIN where resc_id=? and data_path=? and data_id=?)
May 7 18:31:43 pid:3594 NOTICE: SQLSTATE: S1010
May 7 18:31:43 pid:3594 NOTICE: SQLCODE: 0
May 7 18:31:43 pid:3594 NOTICE: SQL Error message: [unixODBC][Driver Manager]Function sequence error
May 7 18:31:43 pid:3594 NOTICE: SQLSTATE: 23505
May 7 18:31:43 pid:3594 NOTICE: SQLCODE: 7
May 7 18:31:43 pid:3594 NOTICE: SQL Error message: ERROR: duplicate key value violates unique constraint "idx_data_main2"
Key (coll_id, data_name, data_repl_num, data_version)=(16101, copy2, 2, ) already exists.;
Error while executing the query
May 7 18:31:43 pid:3594 NOTICE: chlRegReplica cmlExecuteNoAnswerSql(insert) failure -809000
May 7 18:31:43 pid:3594 NOTICE: chlRegReplica cmlExecuteNoAnswerSql(rollback) succeeded
May 7 18:31:43 pid:3612 remote addresses: 10.141.10.152 ERROR: [-]   /tmp/tmppTB_kL/server/api/src/rsFileCreate.cpp:196:int _rsFileCreate(rsComm_t *, fileCreateInp_t *, rodsServerHost_t *, fileCreateOut_t **) : status [UNIX_FILE_RMDIR_ERR] errno [Not a directory] -- message [Rmdir error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "Not a directory", status = -521020.]
   [-]   /tmp/tmppTB_kL/server/drivers/src/fileDriver.cpp:352:irods::error fileRmdir(rsComm_t *, irods::first_class_object_ptr) : status [UNIX_FILE_RMDIR_ERR] errno [Not a directory] -- message [Rmdir error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "Not a directory", status = -521020.]
       [-]   /tmp/tmppTB_kL/plugins/resources/unixfilesystem/libunixfilesystem.cpp:1020:irods::error unix_file_rmdir(irods::plugin_context &) : status [UNIX_FILE_RMDIR_ERR] errno [Not a directory] -- message [Rmdir error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "Not a directory", status = -521020.]

May 7 18:31:43 pid:3612 remote addresses: 10.141.10.152 ERROR: [-]   /tmp/tmppTB_kL/server/api/src/rsFileCreate.cpp:210:int _rsFileCreate(rsComm_t *, fileCreateInp_t *, rodsServerHost_t *, fileCreateOut_t **) : status [UNIX_FILE_CREATE_ERR] errno [File exists] -- message [create error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "File exists".]
   [-]   /tmp/tmppTB_kL/server/drivers/src/fileDriver.cpp:38:irods::error fileCreate(rsComm_t *, irods::first_class_object_ptr) : status [UNIX_FILE_CREATE_ERR] errno [File exists] -- message [create error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "File exists".]
       [-]   /tmp/tmppTB_kL/plugins/resources/unixfilesystem/libunixfilesystem.cpp:625:irods::error unix_file_create(irods::plugin_context &) : status [UNIX_FILE_CREATE_ERR] errno [File exists] -- message [create error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "File exists".]

May 7 18:31:43 pid:3606 NOTICE: chlRegReplica cmlExecuteNoAnswerSql(insert) failure -819000
May 7 18:31:43 pid:3606 NOTICE: chlRegReplica cmlExecuteNoAnswerSql(rollback) succeeded
May 7 18:31:43 pid:3612 remote addresses: 10.141.10.152 ERROR: [-]   /tmp/tmppTB_kL/server/api/src/rsFileCreate.cpp:196:int _rsFileCreate(rsComm_t *, fileCreateInp_t *, rodsServerHost_t *, fileCreateOut_t **) : status [UNIX_FILE_RMDIR_ERR] errno [Not a directory] -- message [Rmdir error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "Not a directory", status = -521020.]
   [-]   /tmp/tmppTB_kL/server/drivers/src/fileDriver.cpp:352:irods::error fileRmdir(rsComm_t *, irods::first_class_object_ptr) : status [UNIX_FILE_RMDIR_ERR] errno [Not a directory] -- message [Rmdir error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "Not a directory", status = -521020.]
       [-]   /tmp/tmppTB_kL/plugins/resources/unixfilesystem/libunixfilesystem.cpp:1020:irods::error unix_file_rmdir(irods::plugin_context &) : status [UNIX_FILE_RMDIR_ERR] errno [Not a directory] -- message [Rmdir error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "Not a directory", status = -521020.]

May 7 18:31:43 pid:3612 remote addresses: 10.141.10.152 ERROR: [-]   /tmp/tmppTB_kL/server/api/src/rsFileCreate.cpp:210:int _rsFileCreate(rsComm_t *, fileCreateInp_t *, rodsServerHost_t *, fileCreateOut_t **) : status [UNIX_FILE_CREATE_ERR] errno [File exists] -- message [create error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "File exists".]
   [-]   /tmp/tmppTB_kL/server/drivers/src/fileDriver.cpp:38:irods::error fileCreate(rsComm_t *, irods::first_class_object_ptr) : status [UNIX_FILE_CREATE_ERR] errno [File exists] -- message [create error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "File exists".]
       [-]   /tmp/tmppTB_kL/plugins/resources/unixfilesystem/libunixfilesystem.cpp:625:irods::error unix_file_create(irods::plugin_context &) : status [UNIX_FILE_CREATE_ERR] errno [File exists] -- message [create error for "/var/lib/irods/Resc01/home/kwaegema/copy2", errno = "File exists".]

May 7 18:31:43 pid:3594 remote addresses: 10.141.10.152 ERROR: [-]   /tmp/tmppTB_kL/server/api/src/rsUnregDataObj.cpp:85:int _rsUnregDataObj(rsComm_t *, unregDataObj_t *) : status [CAT_UNKNOWN_FILE] errno [] -- message [_rsUnregDataObj - Failed to unregister the data object "/UGentqa02/home/kwaegema/copy2" - CAT_UNKNOWN_FILE ]

May 7 18:31:43 pid:3594 NOTICE: dataObjUnlinkS: rsUnregDataObj error for /UGentqa02/home/kwaegema/copy2. status = -817000
May 7 18:31:43 pid:3594 remote addresses: 10.141.10.152 ERROR: rsExecRuleExpression : -817000, [-]   /tmp/tmpDKiaty/libirods_rule_engine_plugin-storage_tiering.cpp:250:irods::error exec_rule_expression(irods::default_re_ctx &, const std::string &, msParamArray_t *, irods::callback) : status [CAT_UNKNOWN_FILE] errno [] -- message [iRODS Exception:
    file: /tmp/tmpDKiaty/irods_object_migrator.cpp
    function: void irods::object_migrator::trim_replica_on_resource(const std::string &, const std::string &)
    line: 356
    code: -817000 (CAT_UNKNOWN_FILE)
    message:
        failed to trim obj path [/UGentqa02/home/kwaegema/copy2] src resc [irodsArch]

stack trace:
--------------

Dumping stack trace

<0>   Offset: 0x65     Address: 0x7ff916268d95   irods::exception::exception(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<1>   Offset: 0x43     Address: 0x7ff916268f13   irods::exception::exception(long, boost::basic_format<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<2>   Offset: 0x467    Address: 0x7ff9138539f7   irods::object_migrator::trim_replica_on_resource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<3>   Offset: 0x1cb    Address: 0x7ff91385121b   irods::object_migrator::operator()()
<4>   Offset: 0xc3     Address: 0x7ff913863c33   irods::storage_tiering::move_data_object(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<5>   Offset: 0x3f2    Address: 0x7ff91383b052   exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)
<6>   Offset: 0x2f     Address: 0x7ff913850d4f   std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*&&, irods::callback&&)
<7>   Offset: 0xf0     Address: 0x7ff9192f3a90   irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)
<8>   Offset: 0x354    Address: 0x7ff9192f3924   irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule_expression(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*)
<9>   Offset: 0x25e    Address: 0x7ff9192f32fe   rsExecRuleExpression(rsComm_t*, exec_rule_expression_t*)
<10>   Offset: 0x44     Address: 0x7ff91954ca24   irods::api_call_adaptor<exec_rule_expression_t*>::operator()(irods::plugin_context&, rsComm_t*, exec_rule_expression_t*)
<11>   Offset: 0x13     Address: 0x7ff91954c9a3   std::__1::__function::__func<irods::api_call_adaptor<exec_rule_expression_t*>, std::__1::allocator<irods::api_call_adaptor<exec_rule_expression_t*> >, irods::error (irods::plugin_context&, rsComm_t*, exec_rule_expression_t*)>::operator()(irods::plugin_context&, rsComm_t*&&, exec_rule_expression_t*&&)
<12>   Offset: 0x93b    Address: 0x7ff9194402cb   int irods::api_entry::call_handler<exec_rule_expression_t*>(rsComm_t*, exec_rule_expression_t*)
<13>   Offset: 0x75b    Address: 0x7ff9195cedcb   rsApiHandler(rsComm_t*, int, BytesBuf*, BytesBuf*)
<14>   Offset: 0x8f2    Address: 0x7ff9195d0b42   readAndProcClientMsg(rsComm_t*, int)
<15>   Offset: 0x9c7    Address: 0x7ff9195c45e7   agentMain(rsComm_t*)
<16>   Offset: 0x1cc3   Address: 0x7ff9195c2f63   runIrodsAgentFactory(sockaddr_un)

<17> Offset: Address: 0x4215ad /usr/sbin/irodsServer(main+0x4dd) [0x4215ad]

<18> Offset: Address: 0x7ff9159773d5 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7ff9159773d5]

<19> Offset: Address: 0x42100d /usr/sbin/irodsServer() [0x42100d]

]

May 7 18:31:43 pid:3594 NOTICE: rsApiHandler: User has no permission for apiNumber 708
May 7 18:31:43 pid:3600 NOTICE: _rsDataObjRepl - Failed to resolve a single replication copy.
May 7 18:31:43 pid:3600 NOTICE: rsDataObjRepl - Failed to replicate data object.
May 7 18:31:43 pid:3600 remote addresses: 10.141.10.152 ERROR: rsExecRuleExpression : -808000, [-] /tmp/tmpDKiaty/libirods_rule_engine_plugin-storage_tiering.cpp:250:irods::error exec_rule_expression(irods::default_re_ctx &, const std::string &, msParamArray_t *, irods::callback) : status [CAT_NO_ROWS_FOUND] errno [] -- message [iRODS Exception:

    file: /tmp/tmpDKiaty/irods_object_migrator.cpp
    function: void irods::object_migrator::replicate_object_to_resource(const std::string &, const std::string &, const std::string &, const std::string &)
    line: 185
    code: -808000 (CAT_NO_ROWS_FOUND)
    message:

failed to migrate [/UGentqa02/home/kwaegema/copy2] to [irodsResc]

stack trace:
--------------

Dumping stack trace

<0>   Offset: 0x65     Address: 0x7ff916268d95   irods::exception::exception(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<1>   Offset: 0x43     Address: 0x7ff916268f13   irods::exception::exception(long, boost::basic_format<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<2>   Offset: 0x554    Address: 0x7ff913852614   irods::object_migrator::replicate_object_to_resource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<3>   Offset: 0x10b    Address: 0x7ff91385115b   irods::object_migrator::operator()()
<4>   Offset: 0xc3     Address: 0x7ff913863c33   irods::storage_tiering::move_data_object(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<5>   Offset: 0x3f2    Address: 0x7ff91383b052   exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)
<6>   Offset: 0x2f     Address: 0x7ff913850d4f   std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*&&, irods::callback&&)
<7>   Offset: 0xf0     Address: 0x7ff9192f3a90   irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)
<8>   Offset: 0x354    Address: 0x7ff9192f3924   irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule_expression(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*)
<9>   Offset: 0x25e    Address: 0x7ff9192f32fe   rsExecRuleExpression(rsComm_t*, exec_rule_expression_t*)
<10>   Offset: 0x44     Address: 0x7ff91954ca24   irods::api_call_adaptor<exec_rule_expression_t*>::operator()(irods::plugin_context&, rsComm_t*, exec_rule_expression_t*)
<11>   Offset: 0x13     Address: 0x7ff91954c9a3   std::__1::__function::__func<irods::api_call_adaptor<exec_rule_expression_t*>, std::__1::allocator<irods::api_call_adaptor<exec_rule_expression_t*> >, irods::error (irods::plugin_context&, rsComm_t*, exec_rule_expression_t*)>::operator()(irods::plugin_context&, rsComm_t*&&, exec_rule_expression_t*&&)
<12>   Offset: 0x93b    Address: 0x7ff9194402cb   int irods::api_entry::call_handler<exec_rule_expression_t*>(rsComm_t*, exec_rule_expression_t*)
<13>   Offset: 0x75b    Address: 0x7ff9195cedcb   rsApiHandler(rsComm_t*, int, BytesBuf*, BytesBuf*)
<14>   Offset: 0x8f2    Address: 0x7ff9195d0b42   readAndProcClientMsg(rsComm_t*, int)
<15>   Offset: 0x9c7    Address: 0x7ff9195c45e7   agentMain(rsComm_t*)
<16>   Offset: 0x1cc3   Address: 0x7ff9195c2f63   runIrodsAgentFactory(sockaddr_un)

<17> Offset: Address: 0x4215ad /usr/sbin/irodsServer(main+0x4dd) [0x4215ad]

<18> Offset: Address: 0x7ff9159773d5 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7ff9159773d5]

<19> Offset: Address: 0x42100d /usr/sbin/irodsServer() [0x42100d]

The file copy2 isn't anywhere I would expect, but in an orphan directory:

[root@rodsqa02 ~]# ls /var/lib/irods/Resc03/home/kwaegema/
copy1 newtesta
[root@rodsqa02 ~]# ls /var/lib/irods/Resc01/home/kwaegema/
[root@rodsqa02 ~]# ls /var/lib/irods/Resc01/orphan/home/kwaegema/
copy2.1290957432 copy2.1670744615 copy2.331412819 copy2.3771183873 copy2.3939224961 copy2.896905754

Thanks again!!

Kenneth

Kenneth Waegeman

unread,

May 22, 2019, 3:38:20 AM5/22/19

to irod...@googlegroups.com, Alan King

Hi All,

does someone has a clue what could be going on here?

Thanks!

Kenneth

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/cbfc31e9-dc89-79b5-89be-3846dabfc973%40ugent.be.

Terrell Russell

unread,

May 23, 2019, 8:51:04 AM5/23/19

to irod...@googlegroups.com

We're going to take this offline - will report back when we know what's going on.

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/e081002d-f204-8259-c6cf-1721ecf2e23b%40ugent.be.

Alan King

unread,

Jun 17, 2019, 2:33:14 PM6/17/19

to irod...@googlegroups.com

Hi,

After an offline conversation, we have determined the cause and captured the issue here:

https://github.com/irods/irods_capability_storage_tiering/issues/103

Thanks!

Alan

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CAFaqteZo62t-AN%2BNWDE6O%2BiP-bP%3DCeKewHi%2B7%2BqAU0Ad98Z9_g%40mail.gmail.com.

Reply all

Reply to author

Forward