Storage tiering combined with access rights and groups causing problems.

[Jan de Graaf]

Hi,

We seem to have stumbled on a new issue with the storage tiering plugin.
Basically, we got the tiering plugin working after working with Alan on it at the UGM2024.
But we refined our access rights policies and introduced somewhat more granular access to the data in iRODS. On a projectfolder we add groups (Owner, RW, RO) and then we make users members of the appropriate group. Default RBAC policies.

But when we add groups the storage tiering stops working. No data is being tiered. When we remove the groups from the folders the tiering starts working again.

The log files of irods is throwing a lot of errors on the tiering plugin when groups are added.

 {"log_category":"legacy","log_level":"error","log_message":"iRODS Exception:\n    file: /irods_plugin_source/storage_tiering.cpp\n    function: void irods::storage_tiering::migrate_violating_data_objects(rcComm_t *, const std::string &, const std::string &, const std::string &, const std::string &)\n    line: 674\n    code: -35000 (SYS_INVALID_OPR_TYPE)\n    message:\n        scheduling failed for [9] objects for query [SELECT DATA_NAME, COLL_NAME, USER_NAME, USER_ZONE, DATA_REPL_NUM WHERE META_DATA_ATTR_NAME = 'irods::access_time' AND META_DATA_ATTR_VALUE < '1719905947' AND META_DATA_ATTR_UNITS <> 'irods::storage_tiering::migration_scheduled' AND DATA_RESC_ID IN ('35069',)]\nstack trace:\n--------------\n 0# irods::stacktrace::dump() const in /lib/libirods_common.so.4.3.1\n 1# irods::exception::assemble_full_display_what() const in /lib/libirods_common.so.4.3.1\n 2# irods::exception::what() const in /lib/libirods_common.so.4.3.1\n 3# irods::log(irods::exception const&) in /lib/libirods_common.so.4.3.1\n 4# irods::storage_tiering::migrate_violating_data_objects(RcComm*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-unified_storage_tiering.so\n 5# irods::storage_tiering::apply_policy_for_tier_group(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-unified_storage_tiering.so\n 6# exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback) in /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-unified_storage_tiering.so\n 7# std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*&&, irods::callback&&) in /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-unified_storage_tiering.so\n 8# irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback) in /lib/libirods_server.so.4.3.1\n 9# irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule_expression(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*) in /lib/libirods_server.so.4.3.1\n10# rsExecRuleExpression(RsComm*, ExecRuleExpression*) in /lib/libirods_server.so.4.3.1\n11# irods::api_call_adaptor<ExecRuleExpression*>::operator()(irods::plugin_context&, RsComm*, ExecRuleExpression*) in /lib/libirods_server.so.4.3.1\n12# std::__1::__function::__func<irods::api_call_adaptor<ExecRuleExpression*>, std::__1::allocator<irods::api_call_adaptor<ExecRuleExpression*> >, irods::error (irods::plugin_context&, RsComm*, ExecRuleExpression*)>::operator()(irods::plugin_context&, RsComm*&&, ExecRuleExpression*&&) in /lib/libirods_server.so.4.3.1\n13# int irods::api_entry::call_handler<ExecRuleExpression*>(RsComm*, ExecRuleExpression*) in /lib/libirods_server.so.4.3.1\n14# rsApiHandler(RsComm*, int, BytesBuf*, BytesBuf*) in /lib/libirods_server.so.4.3.1\n15# readAndProcClientMsg(RsComm*, int) in /lib/libirods_server.so.4.3.1\n16# agentMain(RsComm*) in /lib/libirods_server.so.4.3.1\n17# runIrodsAgentFactory(sockaddr_un) in /lib/libirods_server.so.4.3.1\n18# main::$_5::operator()() const at rodsServer.cpp:?\n19# main in /usr/sbin/irodsServer\n20# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6\n21# _start in /usr/sbin/irodsServer\n\n","request_api_name":"EXEC_RULE_EXPRESSION_AN","request_api_number":1206,"request_api_version":"d","request_client_user":"rods","request_host":"172.31.32.83","request_proxy_user":"rods","request_release_version":"rods4.3.1","server_host":"p-irods-001","server_pid":596725,"server_timestamp":"2024-07-02T07:39:09.991Z","server_type":"agent","server_zone":"nkiImaging"}
 {"log_category":"legacy","log_level":"error","log_message":"data movement scheduling failed - [-808000]::[iRODS Exception:\n    file: /irods_plugin_source/storage_tiering.cpp\n    function: std::string irods::storage_tiering::get_metadata_for_data_object(rcComm_t *, const std::string &, const std::string &)\n    line: 95\n    code: -808000 (CAT_NO_ROWS_FOUND)\n    message:\n        no results found for object [/nkiImaging/home/bioimaging/2024/Backup_CSLM/BIF/jg_bif/2024-07-01/foci_00.tif] with attribute [irods::access_time]\nstack trace:\n--------------\n 0# irods::stacktrace::dump() const in /lib/libirods_common.so.4.3.1\n 1# irods::exception::assemble_full_display_what() const in /lib/libirods_common.so.4.3.1\n 2# irods::exception::what() const in /lib/libirods_common.so.4.3.1\n 3# irods::query_processor<RcComm>::execute(irods::thread_pool&, RcComm&)::'lambda'()::operator()() in /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-unified_storage_tiering.so\n 4# boost::asio::detail::executor_op<boost::asio::detail::binder0<irods::query_processor<RcComm>::execute(irods::thread_pool&, RcComm&)::'lambda'()>, std::__1::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) in /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-unified_storage_tiering.so\n 5# boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) in /lib/libirods_server.so.4.3.1\n 6# boost::asio::detail::scheduler::run(boost::system::error_code&) in /lib/libirods_server.so.4.3.1\n 7# boost::asio::detail::posix_thread::func<boost::asio::thread_pool::thread_function>::run() in /lib/libirods_server.so.4.3.1\n 8# boost_asio_detail_posix_thread_function in /lib/libirods_server.so.4.3.1\n 9# 0x00007F545D25B609 in /lib/x86_64-linux-gnu/libpthread.so.0\n10# clone in /lib/x86_64-linux-gnu/libc.so.6\n\n]","request_api_name":"EXEC_RULE_EXPRESSION_AN","request_api_number":1206,"request_api_version":"d","request_client_user":"rods","request_host":"172.31.32.83","request_proxy_user":"rods","request_release_version":"rods4.3.1","server_host":"p-irods-001","server_pid":596725,"server_timestamp":"2024-07-02T07:39:11.453Z","server_type":"agent","server_zone":"nkiImaging"}

The one thing that caught my attention was the iQuery,

With no groups on the folders this is the output:

 iquest "SELECT DATA_NAME, COLL_NAME, USER_NAME, USER_ZONE, DATA_REPL_NUM WHERE META_DATA_ATTR_NAME = 'irods::access_time' AND META_DATA_ATTR_VALUE < '1719905947' AND META_DATA_ATTR_UNITS <> 'irods::storage_tiering::migration_scheduled' AND DATA_RESC_ID IN ('35069',)"
DATA_NAME = foci_00.tif
COLL_NAME = /nkiImaging/home/bioimaging/2024/Backup_CSLM/BIF/jg_bif/2024-07-01
USER_NAME = rods
USER_ZONE = nkiImaging
DATA_REPL_NUM = 0
------------------------------------------------------------

With a group added:

iquest "SELECT DATA_NAME, COLL_NAME, USER_NAME, USER_ZONE, DATA_REPL_NUM WHERE META_DATA_ATTR_NAME = 'irods::access_time' AND META_DATA_ATTR_VALUE < '1719905947' AND META_DATA_ATTR_UNITS <> 'irods::storage_tiering::migration_scheduled' AND DATA_RESC_ID IN ('35069',)"
DATA_NAME = foci_00.tif
COLL_NAME = /nkiImaging/home/bioimaging/2024/Backup_CSLM/BIF/jg_bif/2024-07-01
USER_NAME = grp_bioimaging
USER_ZONE = nkiImaging
DATA_REPL_NUM = 0
------------------------------------------------------------
DATA_NAME = foci_00.tif
COLL_NAME = /nkiImaging/home/bioimaging/2024/Backup_CSLM/BIF/jg_bif/2024-07-01
USER_NAME = rods
USER_ZONE = nkiImaging
DATA_REPL_NUM = 0

Do not know if the multiple rows per file are causing the issues? But it's a very reproduceable problem on our side.

Any ideas how to fix this?

Best,

Jan de Graaf