Rods Log on Globus Transfers.

40 views
Skip to first unread message

Sanju Timsina

unread,
May 9, 2022, 10:10:15 AM5/9/22
to iRODS-Chat
Hi, 

Please find the attached log while transferring files from the Globus. We are getting SQL errors and the RE_RUNTIME_ERROR message.  

Is this iRODS server configuration issue or a database issue or some misconfiguration on Globus end? 

Any help on this.

Thank you,
Sanju
rodsLog.05.09.docx

Terrell Russell

unread,
May 9, 2022, 11:01:46 AM5/9/22
to irod...@googlegroups.com
Okay, so the logical quotas cannot write some metadata...

because it's already in there?

May 9 09:36:12 pid:175706 NOTICE: SQL Error message: [MySQL][ODBC 5.2(w)
Driver][mysqld-10.5.10-MariaDB]Duplicate entry [8426287-8425621] for key
[idx_objt_metamap1]


irods::handler::logical_quotas_count_total_size_in_bytes(const std::string &, const
irods::instance_configuration_map &, std::list<boost::any> &, MsParamArray *, irods::callback
&) : status [RE_RUNTIME_ERROR] errno [] -- message [cannot set metadata: Unknown error -809000]

$ ierror -809000
irods error: -809000 CATALOG_ALREADY_HAS_ITEM_BY_THAT_NAME

We'll look a little deeper - shouldn't ever get a duplicate key entry when 'setting' metadata, it should overwrite cleanly if it already exists.

Terrell


 

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/6abc3975-63fa-4355-8ade-ee757d735de5n%40googlegroups.com.

James, Justin Kyle

unread,
May 10, 2022, 1:08:37 PM5/10/22
to irod...@googlegroups.com
I set up logical quotas (with a limit on the number of files in /<zone>/home/rods) and uploaded and downloaded files to that collection using globus-url-copy and didn't see the error you are getting.  I also saw the logical quotas metadata updating as expected.

Can you iput/iget files to the same location?

Also, what does "imeta ls -C" on the collection give you?


From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Terrell Russell <terrell...@gmail.com>
Sent: Monday, May 9, 2022 11:01 AM
To: irod...@googlegroups.com <irod...@googlegroups.com>
Subject: Re: [iROD-Chat:20426] Rods Log on Globus Transfers.
 

Sanju Timsina

unread,
May 11, 2022, 11:58:43 AM5/11/22
to iRODS-Chat
Hi James,

iput/iget works fine.

 I do not see these errors when transferring through the Globus now.  Earlier the gcs_manager was failing repeatedly, which may have caused the issue. I will check and update if something unpleasant pops up.


The output of imeta ls -C  is 

$ imeta ls -C /arctic/work/stimsina

AVUs defined for collection /arctic/work/stimsina:

attribute: irods::logical_quotas::maximum_size_in_bytes

value: 50000000000

units: 

----

attribute: irods::logical_quotas::total_number_of_data_objects

value: 42

units: 

----

attribute: irods::logical_quotas::total_size_in_bytes

value: 502847036

units: 


Thank you,
Sanju

Sanju Timsina

unread,
May 13, 2022, 9:58:16 AM5/13/22
to iRODS-Chat
Hi,

I am still getting these logs when transferred through Globus

:14 pid:134397 remote addresses: <database-ip>, <resource-server-ip>, <resource-server-ip> ERROR: [-]       /repos/irods/server/core/src/rodsAgent.cpp:566:int runIrodsAgentFactory(sockaddr_un) :  status [SSL_SHUTDOWN_ERROR]  errno [] -- message [failed to call 'agent stop']
        [-]     /repos/irods/lib/core/src/sockComm.cpp:160:irods::error sockAgentStop(irods::network_object_ptr) :  status [SSL_SHUTDOWN_ERROR]  errno [] -- message [failed to call 'agent stop']
                [-]     /repos/irods/plugins/network/ssl/libssl.cpp:952:irods::error ssl_agent_stop(irods::plugin_context &) :  status [SSL_SHUTDOWN_ERROR]  errno [] -- message [error completing shutdown of SSL connection]

  ERROR: Agent process [134397] exited with status [64]
NOTICE: bindVar[1]=8434207
NOTICE: bindVar[2]=31170
NOTICE: bindVar[3]=01652448075
NOTICE: bindVar[4]=01652448075
NOTICE: _cllExecSqlNoResult: SQLExecDirect error: -1 sql:insert into R_OBJT_METAMAP (object_id, meta_id, create_ts, modify_ts) values (?, ?, ?, ?)
NOTICE: SQLSTATE: S1010
NOTICE: SQLCODE: 0
NOTICE: SQL Error message: [unixODBC][Driver Manager]Function sequence error
NOTICE: SQLSTATE: 23000
NOTICE: SQLCODE: 1062
NOTICE: SQL Error message: [MySQL][ODBC 5.2(w) Driver][mysqld-10.5.10-MariaDB]Duplicate entry '8434207-31170' for key 'idx_objt_metamap1'
NOTICE: chlAddAVUMetadata cmlExecuteNoAnswerSql insert failure -809000
NOTICE: bindVar[1]=8434207
NOTICE: bindVar[2]=31170
NOTICE: bindVar[3]=01652448075
NOTICE: bindVar[4]=01652448075
NOTICE: _cllExecSqlNoResult: SQLExecDirect error: -1 sql:insert into R_OBJT_METAMAP (object_id, meta_id, create_ts, modify_ts) values (?, ?, ?, ?)
NOTICE: SQLSTATE: S1010
NOTICE: SQLCODE: 0
NOTICE: SQL Error message: [unixODBC][Driver Manager]Function sequence error
NOTICE: SQLSTATE: 23000
NOTICE: SQLCODE: 1062
SQL Error message: [MySQL][ODBC 5.2(w) Driver][mysqld-10.5.10-MariaDB]Duplicate entry '8434207-31170' for key 'idx_objt_metamap1'
chlAddAVUMetadata cmlExecuteNoAnswerSql insert failure -809000
chlAddAVUMetadata cmlExecuteNoAnswerSql(rollback) succeeded
rsModAVUMetadata: rcModAVUMetadata failed
chlAddAVUMetadata cmlExecuteNoAnswerSql(rollback) succeeded
remote addresses: <database-ip>, <resource-server-ip>, <globus-ip>ERROR: cannot set metadata: Unknown error -809000
NOTICE: rsModAVUMetadata: rcModAVUMetadata failed
remote addresses: <database-ip>, <resource-server-ip>, <globus-ip>ERROR: cannot set metadata: Unknown error -809000
remote addresses: <database-ip>, <resource-server-ip>, <globus-ip>ERROR: iRODS Exception:
    file: ../src/handler.cpp
    function: auto irods::handler::pep_api_replica_close::post(const std::string &, const irods::instance_configuration_map &, std::list<boost::any> &, MsParamArray *, irods::callback &)::(anonymous class)::operator()(type-parameter-0-0 &, const type-parameter-0-1 &) const
    line: 1612
    code: -1205000 (RE_RUNTIME_ERROR)
    message:
        [-]     ../src/handler.cpp:616:irods::error irods::handler::logical_quotas_count_total_number_of_data_objects(const std::string &, const irods::instance_configuration_map &, std::list<boost::any> &, MsParamArray *, irods::callback &) :  status [RE_RUNTIME_ERROR]  errno [] -- message [cannot set metadata: Unknown error -809000]




Thank you,
Sanju

James, Justin Kyle

unread,
May 13, 2022, 11:10:30 AM5/13/22
to iRODS-Chat
Do you still have logical quotas enabled?  Can we disable that for a moment and do a transfer of a new file?

That sequence error is interesting.  That is saying that the object_id and meta_id trying to be inserted is not unique.  That error you see occurs if someone tries to add the exact same metadata twice on the same object.

It certainly seems that the logical quotas metadata is the culprit as there is no other metadata on that object.  I haven't been able to reproduce a problem with the interaction between Globus and logical quotas but I am not dismissing the possibility that a timing issue between the two is causing metadata to be written twice.  

Also, do you have access to the database itself?  Could you run the following query in the database and report the results?

select meta_attr_name, meta_attr_value, meta_attr_unit from r_meta_main where meta_id = 31170;



From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Sanju Timsina <timsina...@gmail.com>
Sent: Friday, May 13, 2022 9:58 AM
To: iRODS-Chat <irod...@googlegroups.com>
Subject: Re: [iROD-Chat:20454] Rods Log on Globus Transfers.
 

Sanju Timsina

unread,
May 13, 2022, 12:58:44 PM5/13/22
to iRODS-Chat
Hi James,

Could it be because there was error on the metadata on that particular collection

The metadata it was showing 

attribute: irods::logical_quotas::maximum_size_in_bytes value: 50000000000 units: 
 ---- attribute: irods::logical_quotas::total_size_in_bytes value: 0 units: 
 ---- attribute: irods::logical_quotas::total_number_of_data_objects value: 261 units:

The metadata it should be

attribute: irods::logical_quotas::maximum_size_in_bytes value: 50000000000 units: 
 ---- attribute: irods::logical_quotas::total_size_in_bytes value: 6518378 units: 
 ---- attribute: irods::logical_quotas::total_number_of_data_objects value: 283 units: 


Please find the output of the above mentioned query:

+-----------------------------------------------------+-----------------+----------------+

| meta_attr_name                                      | meta_attr_value | meta_attr_unit |

+-----------------------------------------------------+-----------------+----------------+

| irods::logical_quotas::total_number_of_data_objects | 1               |                |

+-----------------------------------------------------+-----------------+----------------+


Thank you,
Sanju
Reply all
Reply to author
Forward
0 new messages