itrim fails because callback.msiDataObjChkum with the verifyChksum flag

[Robert Verhagen]

Hello,

We have a problem with itrim.

itrim command is not working anymore. It fails in data_obj_checksum, called from pep_api_data_obj_trim_pre. The failure occurs when calling callback.msiDataObjChkum with the verifyChksum flag.

$ ils -L rodsServerLog.2024.04.26
  rods              0 storageResc         4853 2024-04-30.15:25 & rodsServerLog.2024.04.26
    sha2:0RwQjr/uD5ZGld5Xw26jGMbuohjOUtoahHQWRtbH1KA=    generic    /mnt/irods/acc/home/rods/rodsServerLog.2024.04.26
  rods              1 compute2Resc         4853 2024-04-30.15:26 & rodsServerLog.2024.04.26
    sha2:0RwQjr/uD5ZGld5Xw26jGMbuohjOUtoahHQWRtbH1KA=    generic    /mnt/irodstier1/irods/acc/home/rods/rodsServerLog.2024.04.26


$ itrim -M -S compute2Resc -N 1 rodsServerLog.2024.04.26
remote addresses: 127.0.0.1 ERROR: trimUtil: trim error for /rivmZone_acc_01/home/rods/rodsServerLog.2024.04.26.  status = -154000 SYS_INTERNAL_ERR
Total size trimmed = 0.000 MB. Number of files trimmed = 0.
ERROR: [-]  /repos/irods/lib/core/src/rcConnect.cpp:279:int rcDisconnect(rcComm_t *) :  status [SSL_SHUTDOWN_ERROR]  errno [] -- message [failed to call 'client stop']
  [-]  /repos/irods/lib/core/src/sockComm.cpp:98:irods::error sockClientStop(irods::network_object_ptr, rodsEnv *) :  status [SSL_SHUTDOWN_ERROR]  errno [] -- message [failed to call 'client stop']
    [-]  /repos/irods/plugins/network/ssl/libssl.cpp:645:irods::error ssl_client_stop(irods::plugin_context &, rodsEnv *) :  status [SSL_SHUTDOWN_ERROR]  errno [] -- message [error shutting down the SSL connection]

We get the same error with irods 4.2.11 and also with 4.3.2.

Does anyone recognize this problem?

Regards,

Robert.

[Kory Draughn]

Hi Robert,

Based on what you've said, do you have policy attached to the trim PEPs which invoke checksum operations? If so, please post it and explain the motivation behind it.

Kory Draughn

Chief Technologist

iRODS Consortium

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/5af1c699-2f3d-4bcc-80fe-ae7a4c94c7edn%40googlegroups.com.

[Robert Verhagen]

Dear Kory,

We only want to trim if the checksums are equal.

$ ils -L aFile
  rods              0 storageResc            0 2024-06-26.16:41 & aFile
    sha2:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=    generic    /mnt/irods/acc/home/rods/aFile
  rods              2 compute2Resc            0 2024-06-27.09:51 & aFile
    sha2:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=    generic    /mnt/irodstier1/irods/acc/home/rods/aFile

itrim -N 1 -S compute2Resc aFile
remote addresses: 131.224.35.44 ERROR: trimUtil: trim error for /Zone_acc_01/home/rods/aFile.  status = -154000 SYS_INTERNAL_ERR

Total size trimmed = 0.000 MB. Number of files trimmed = 0.

(should have been trimmed because the checksums are equal)

In pep_api_data_obj_trim_pre we call data_obj_checksum

def data_obj_checksum(callback, obj, **options):
    """Wrapper for msiDataObjChksum()
   
    https://docs.irods.org/4.2.6/doxygen/reDataObjOpr_8cpp.html#aae61683c7a9ab9cc59cb2125426387ef

    Args:
        object: string path to data object.
        **options: optional keyword arguments passed on to
            msiDataObjeChksum. Valid options are: ChksumAll,
            verifyChksum, forceChksum, replNum.
    Returns:
        (possibly new) checksum as string.
    """
    # Create option string in iRODS special msKeyValStr format.
    known_options = {"ChksumAll", "verifyChksum", "forceChksum", "replNum"}
    kv_args = []
    for key in set(options.keys()).intersection(known_options):
        kv_args.append("{}={}".format(key, options[key]))
    log.debug("kv_args: %s" % kv_args)

    # Call msi with dummy string, as output is retrieved via return value.
    log.debug("obj: {}  / kv_args: {} ".format(obj, kv_args))
    result = callback.msiDataObjChksum(obj, "++++".join(kv_args), "dummy_string")
    return result["arguments"][2]

The logging:

2024-06-28 15:29:59,472|irods_rules.rodsutils DEBUG:101128 PEP invocation:pep_api_data_obj_trim_pre, rule_args=['api_instance', '<irods_types.PluginContext object at 0x7fa402838e18>', '<irods_types.DataObjInp object at 0x220f2a0>']
2024-06-28 15:29:59,472|irods_rules.policy DEBUG:101128 checksum_before_trim /Zone_acc_01/home/rods/aFile
2024-06-28 15:29:59,477|irods_rules.rodsutils DEBUG:101128 obj: /Zone_acc_01/home/rods/aFile  / kv_args: ['verifyChksum=', 'replNum=2'] 

Do we do something wrong?

Previous versions of irods worked with this code...

Regards,

Robert.

Op donderdag 27 juni 2024 om 14:37:39 UTC+2 schreef korydraughn.renci:

[e.v.wi...@gmail.com]

This seems to be related to the use of the verifyChksum keyword with msiDataObjChksum. I tried the following on an irods server (4.3.2) without any custom rules, and with the PREP installed:

-bash-4.2$ touch testFile
-bash-4.2$ iput -f testFile
-bash-4.2$ ichksum -f testFile
    testFile    sha2:nEJkXahn9eOAI1Fc8Hnz/zgGryMsS4vQmcUsk2oumNY=

-bash-4.2$ irule -r irods_rule_engine_plugin-python-instance -F /var/lib/irods/scripts/irods/test/python_rules/rulemsiDataObjChksum.r "*dataObject='/tempZone/home/rods/testFile'" "*Flags='forceChksum='"
Collection is /tempZone/home/rods and file is testFile
Saved checksum for file testFile is sha2:nEJkXahn9eOAI1Fc8Hnz/zgGryMsS4vQmcUsk2oumNY=, new checksum is sha2:nEJkXahn9eOAI1Fc8Hnz/zgGryMsS4vQmcUsk2oumNY=
-bash-4.2$ irule -r irods_rule_engine_plugin-python-instance -F /var/lib/irods/scripts/irods/test/python_rules/rulemsiDataObjChksum.r "*dataObject='/tempZone/home/rods/testFile'" "*Flags='verifyChksum='"
remote addresses: 127.0.0.1 ERROR: rcExecMyRule error.  status = -154000 SYS_INTERNAL_ERR

This is the error in syslog:

{"log_category":"server","log_level":"critical","log_message":" 0# stacktrace_signal_handler in /lib/libirods_server.so.4.3.2\n 1# 0x00007F5913E29630 in /lib64/libpthread.so.0\n 2# 0x00007F591387E8C1 in /lib64/libc.so.6\n 3# irods::convertFromMsParam(boost::any&, MsParam*) in /lib/libirods_server.so.4.3.2\n 4# irods::default_microservice_manager<RuleExecInfo*>::exec_microservice_adapter(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, RuleExecInfo*, std::__1::list<boost::any, std::__1::allocator<boost::any> >&) in /lib/libirods_server.so.4.3.2\n 5# irods::error irods::default_microservice_manager<RuleExecInfo*>::exec_microservice<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, RuleExecInfo*, irods::unpack&&) in /lib/libirods_server.so.4.3.2\n 6# std::__1::__function::__func<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&), std::__1::allocator<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&)>, irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&) in /lib/libirods_server.so.4.3.2\n 7# irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call<std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&)::'lambda'()::operator()() const in /lib/libirods_server.so.4.3.2\n 8# irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call<std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, RuleExecInfo*&, irods::unpack&&) in /lib/libirods_server.so.4.3.2\n 9# irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const in /lib/libirods_server.so.4.3.2\n10# irods::error irods::control<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), std::__1::tuple<>, irods::unpack>(std::__1::list<irods::re_pack_inp<std::__1::tuple<> >, std::__1::allocator<irods::re_pack_inp<std::__1::tuple<> > > >&, irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) in /lib/libirods_server.so.4.3.2\n11# 0x00007F590DEEE4E8\n12# 0x00007F590DEE678D\n13# 0x00007F590DEEDDEC\n14# 0x00007F590DC16773\n15# 0x00007F590DC18165\n16# 0x00007F590DC1E12A\n17# 0x00007F590DC17E06\n18# 0x00007F590D772E90\n19# 0x00007F590D773021\n20# 0x00007F590D772C93\n21# 0x00007F590D7C64BD\n22# 0x00007F590D772E90\n23# 0x00007F590D81B65C\n24# 0x00007F590D80FBA7\n25# 0x00007F590D81B8D2\n26# 0x00007F590D78AB47\n27# 0x00007F590D772C93\n28# 0x00007F590D773663\n29# 0x00007F590DEE8F62\n30# 0x00007F590DEE2C74\n31# 0x00007F590DF2CDCD\n32# irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule_text(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::callback) in /lib/libirods_server.so.4.3.2\n33# irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule_text(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in /lib/libirods_server.so.4.3.2\n34# rsExecMyRule(RsComm*, ExecMyRuleInp*, MsParamArray**) in /lib/libirods_server.so.4.3.2\n35# irods::api_call_adaptor<ExecMyRuleInp*, MsParamArray**>::operator()(irods::plugin_context&, RsComm*, ExecMyRuleInp*, MsParamArray**) in /lib/libirods_server.so.4.3.2\n36# std::__1::__function::__func<irods::api_call_adaptor<ExecMyRuleInp*, MsParamArray**>, std::__1::allocator<irods::api_call_adaptor<ExecMyRuleInp*, MsParamArray**> >, irods::error (irods::plugin_context&, RsComm*, ExecMyRuleInp*, MsParamArray**)>::operator()(irods::plugin_context&, RsComm*&&, ExecMyRuleInp*&&, MsParamArray**&&) in /lib/libirods_server.so.4.3.2\n37# int irods::api_entry::call_handler<ExecMyRuleInp*, MsParamArray**>(RsComm*, ExecMyRuleInp*, MsParamArray**) in /lib/libirods_server.so.4.3.2\n38# rsApiHandler(RsComm*, int, BytesBuf*, BytesBuf*) in /lib/libirods_server.so.4.3.2\n39# readAndProcClientMsg(RsComm*, int) in /lib/libirods_server.so.4.3.2\n40# agentMain(RsComm*) in /lib/libirods_server.so.4.3.2\n41# runIrodsAgentFactory(sockaddr_un) in /lib/libirods_server.so.4.3.2\n42# main::$_5::operator()() const in /usr/sbin/irodsServer\n43# main in /usr/sbin/irodsServer\n44# __libc_start_main in /lib64/libc.so.6\n45# _start in /usr/sbin/irodsServer\n","server_host":"SERVERNAME","server_pid":8824,"server_timestamp":"2024-07-01T09:47:44.033Z","server_type":"server","server_zone":"tempZone","stacktrace_agent_pid":"10718","stacktrace_timestamp_epoch_milliseconds":"106","stacktrace_timestamp_epoch_seconds":"1719827263","stacktrace_timestamp_utc":"2024-07-01T09:47:43.106Z"}

 

Regards,

Erwin van Wieringen

Op vrijdag 28 juni 2024 om 15:59:56 UTC+2 schreef Robert Verhagen:

[Kory Draughn]

Robert,

I was able to reproduce what you're seeing using tip of 4-3-stable.

I've opened the following issue for this.

• https://github.com/irods/irods/issues/7859

This will be resolved with the release of iRODS 4.3.3.

Kory Draughn

Chief Technologist

iRODS Consortium

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/c276468a-a27b-4d98-aba2-f097f73da50dn%40googlegroups.com.