s3 resource plugin help

[joris luijsterburg]

All,

I need a bit of help in attaching a new s3 provider. We are running local s3 objectstores within our facility, and want to attach those to iRODS. We already have one running, and are now attaching a second one. The url is a tad different and I am wondering if that is causing some issues here. Relevant configuration of the working s3 resource is as follows:

location: local.irodsmachine.com
vault: /bucket1
context: S3_DEFAULT_HOSTNAME=s3.local.objectstore.com;

Now I have a new objectstore, where the url is new.local.store.com/s3. Using python and boto3 I was able to send files and access the object store. However, with iRODS I am not succeeding yet. I tried two different setups for the resource, both did not work, with different errors when trying an iput. Do you have an advice on how to continue? Can the s3 resource plugin handle the `s3` prefix before the bucketname, or am I doing something else wrong?


location: local.irodsmachine.com
vault: /bucket2
context: S3_DEFAULT_HOSTNAME=new.local.store.com/s3;...;...;

Error:
log_message: /irods_plugin_source/s3_transport/src/s3_transport.cpp:247 [store_and_log_status] [[138014195967680]]  libs3_types::status: [XmlParseFailure] - 31
log_message: /irods_plugin_source/s3_transport/src/s3_transport.cpp:250 [store_and_log_status] [[138014195967680]]  S3Host: new.local.store.com/s3
log_message: /irods_plugin_source/s3_transport/src/s3_transport.cpp:254 [store_and_log_status] [[138014195967680]]  Function: s3_upload::callback_for_write_to_s3_base::on_response_completion
log_message: [-]        /irods_source/server/api/src/rsFileClose.cpp:120:int _rsFileClose(rsComm_t *, fileCloseInp_t *) :  status [S3_PUT_ERROR]  errno [] -- message [fileClose failed for [3]]
        [-]     /irods_source/server/drivers/src/fileDriver.cpp:167:irods::error fileClose(rsComm_t *, irods::first_class_object_ptr) :  status [S3_PUT_ERROR]  errno [] -- message [failed to call 'close']
                [-]     /irods_plugin_source/s3_transport/include/irods/private/s3_transport/s3_transport.hpp:2306:irods::experimental::io::s3_transport::error_codes irods::experimental::io::s3_transport::s3_transport<char>::s3_upload_file(bool) [CharT = char] :  status [S3_PUT_ERROR]  errno [] -- message [failed in S3_put_object]


log_message: Failed to close file object [error_code=-702000].
log_message: [close_replica_without_catalog_update:288] - error closing replica; ec:[-702000]
log_message: [single_buffer_put:337] - error closing replica; ec:[-702000]


location: local.irodsmachine.com
vault: /s3/bucket2
context: S3_DEFAULT_HOSTNAME=new.local.store.com;...;...;

Error:

log_message: /irods_plugin_source/s3_transport/src/s3_transport.cpp:247 [store_and_log_status] [[138014195967680]]  libs3_types::status: [InvalidBucketNameTooShort] - 8
log_message: /irods_plugin_source/s3_transport/src/s3_transport.cpp:250 [store_and_log_status] [[138014195967680]]  S3Host: new.local.store.com
log_message: /irods_plugin_source/s3_transport/src/s3_transport.cpp:254 [store_and_log_status] [[138014195967680]]  Function: s3_upload::callback_for_write_to_s3_base::on_response_completion
log_message: [-]        /irods_source/server/api/src/rsFileClose.cpp:120:int _rsFileClose(rsComm_t *, fileCloseInp_t *) :  status [S3_PUT_ERROR]  errno [] -- message [fileClose failed for [3]]
        [-]     /irods_source/server/drivers/src/fileDriver.cpp:167:irods::error fileClose(rsComm_t *, irods::first_class_object_ptr) :  status [S3_PUT_ERROR]  errno [] -- message [failed to call 'close']
                [-]     /irods_plugin_source/s3_transport/include/irods/private/s3_transport/s3_transport.hpp:2306:irods::experimental::io::s3_transport::error_codes irods::experimental::io::s3_transport::s3_transport<char>::s3_upload_file(bool) [CharT = char] :  status [S3_PUT_ERROR]  errno [] -- message [failed in S3_put_object]


log_message: Failed to close file object [error_code=-702000].
log_message: [close_replica_without_catalog_update:288] - error closing replica; ec:[-702000]
log_message: [single_buffer_put:337] - error closing replica; ec:[-702000]


Kind regards,

Joris

[James, Justin Kyle]

Some questions/comments:

1. Can you send the entire context string?  Maybe I will see something in it.
2. Is this S3 provider from a different vendor than the original?
3. What was the size of the file you were uploading?
4. Can you increase the logging level for the S3 resource and provide a full log of the put?

---

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of joris luijsterburg <jlu...@gmail.com>
Sent: Wednesday, June 24, 2026 3:16 AM
To: iRODS-Chat <irod...@googlegroups.com>
Subject: [iROD-Chat:22562] s3 resource plugin help

 

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/irod-chat/e34204f3-4177-45ce-908b-9e3e0b0b13aen%40googlegroups.com.

[joris luijsterburg]

Justin,

see below:

1. S3_DEFAULT_HOSTNAME=new.local.store.com/s3;S3_REGIONNAME=us-east-1;S3_AUTH_FILE=/var/lib/irods/OA2test1.s3.keypair;S3_RETRY_COUNT=2;S3_WAIT_TIME_SECONDS=3;S3_PROTO=HTTPS;ARCHIVE_NAMING_POLICY=consistent;HOST_MODE=cacheless_attached;S3_CACHE_DIR=/vault/cache;S3_MPU_CHUNK=125
Note, the rest of the string is exactly the same as in the working resource, except for of course hostname and authentication.


2. Same vendor, both fujifilm, new version
3. 527 bytes
4.

iput -fPV -R OA2test1 temp_metrics_wrapper.sh file
0/1 -  0.00% of files done   0.000/0.001 MB -  0.00% of file sizes done
Processing temp_metrics_wrapper.sh - 0.001 MB   2026-06-25.14:15:52
remote addresses: 10.126.130.13 ERROR: putUtil: put error for /RDMtest/home/irods/file, status = -702000 status = -702000 S3_PUT_ERROR
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:2257 (s3_resolve_resc_hier_operation) [[138014856735232]] _opr=CREATE _curr_host=scomp1824 shmem_size=812
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:2278 (s3_resolve_resc_hier_operation) [[138014856735232]] data_size_str = 0x1de3f7a0
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:2297 (s3_resolve_resc_hier_operation) [[138014856735232]] num_threads_str = 0x0
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:2343 (s3_resolve_resc_hier_operation) [[138014856735232]] get_resource_name=OA2test1
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:516 (translate_open_mode_posix_to_stream)  [[138014856735232]] call_from=s3_file_create_operation O_WRONLY=true, O_RDWR=false, O_RDONLY=false, O_TRUNC=true, O_CREAT=true, O_APPEND=false
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:541 (translate_open_mode_posix_to_stream) [[138014856735232]] translated open mode is [app=false][binary=false][in=false][out=true][trunc=true][ate=false]
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:794 (s3_file_create_operation) [[138014856735232]] physical_path = /bucket2/irods/9/4/file.1782396952
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:1000 (s3_file_write_operation) [[138014856735232]]
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:618 (make_dstream) [[138014856735232]] [physical_path=/bucket2/irods/9/4/file.1782396952][bucket_name=bucket2][fd=3]
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:216 (get_number_of_threads_data_size_and_opr_type) [[138014856735232]] ------------- L1desc ---------------
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:224 (get_number_of_threads_data_size_and_opr_type) [[138014856735232]] [3][objPath=/RDMtest/home/irods/file][filePath=/bucket2/irods/9/4/file.1782396952][oprType=1][requested_number_of_threads=0][dataSize=527][dataObjInfo->dataSize=0][srcL1descInx=0]
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:230 (get_number_of_threads_data_size_and_opr_type) [[138014856735232]] ------------------------------------
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:237 (get_number_of_threads_data_size_and_opr_type) [[138014856735232]] number_of_threads in shmem = 0
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:347 (operator()) [[138014856735232]] num_threads_str = 0x0
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:383 (get_number_of_threads_data_size_and_opr_type) [[138014856735232]] number_of_threads set to 1
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:630 (make_dstream) [[138014856735232]] oprType set to 1
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:631 (make_dstream) [[138014856735232]] data_size set to 527
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:632 (make_dstream) [[138014856735232]] number_of_threads=1
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:695 (make_dstream) [[138014856735232]] [put_repl_flag=true][object_size=527][multipart_enabled=true][minimum_part_size=131072000]
log_message: /irods_plugin_source/s3_transport/include/irods/private/s3_transport/s3_transport.hpp:1317 (open_impl) [[138014856735232]] [_mode & in = false][_mode & out = true][_mode & trunc = true][_mode & app = false][_mode & ate = false][_mode & binary = false]
log_message: /irods_plugin_source/s3_transport/include/irods/private/s3_transport/s3_transport.hpp:1354 (open_impl) [[138014856735232]] [object_key_ = irods/9/4/file.1782396952][use_cache_ = false][download_to_cache_ = false]
log_message: /irods_plugin_source/s3_transport/include/irods/private/s3_transport/s3_transport.hpp:1398 (operator()) [[138014856735232]] number_of_client_transfer_threads = 1
log_message: /irods_plugin_source/s3_transport/include/irods/private/s3_transport/s3_transport.hpp:1404 (operator()) [[138014856735232]] open file_open_counter = 1
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:1029 (operator()) [[138014856735232]] number_of_threads in shared memory - 1
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:1042 (s3_file_write_operation) [[138014856735232]] read number_of_threads of 1
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:1053 (s3_file_write_operation) [[138014856735232]] calling dstream_ptr->write of length 527
log_message: /irods_plugin_source/s3_transport/include/irods/private/s3_transport/s3_transport.hpp:2266 (s3_upload_file) [[138014195967680]] S3_put_object(ctx, irods/9/4/file.1782396952, 527, put_props, 0, &putObjectHandler, &data)
log_message: /irods_plugin_source/s3_resource/src/s3_operations.cpp:1085 (s3_file_close_operation) [[138014856735232]] physical_path = /bucket2/irods/9/4/file.1782396952
log_message: /irods_plugin_source/s3_transport/include/irods/private/s3_transport/s3_transport.hpp:384 (close) [[138014856735232]] fd_=3, is_open=true use_cache_=false

log_message: /irods_plugin_source/s3_transport/src/s3_transport.cpp:247 [store_and_log_status] [[138014195967680]]  libs3_types::status: [XmlParseFailure] - 31
log_message: /irods_plugin_source/s3_transport/src/s3_transport.cpp:250 [store_and_log_status] [[138014195967680]]  S3Host: new.local.store.com/s3
log_message: /irods_plugin_source/s3_transport/src/s3_transport.cpp:254 [store_and_log_status] [[138014195967680]]  Function: s3_upload::callback_for_write_to_s3_base::on_response_completion

log_message: /irods_plugin_source/s3_transport/include/irods/private/s3_transport/s3_transport.hpp:2274 (s3_upload_file) [[138014195967680]] S3_put_object returned [status=XmlParseFailure].
log_message: /irods_plugin_source/s3_transport/include/irods/private/s3_transport/s3_transport.hpp:431 (operator()) [[138014856735232]] close BEFORE decrement file_open_counter = 1
log_message: /irods_plugin_source/s3_transport/include/irods/private/s3_transport/s3_transport.hpp:438 (operator()) [[138014856735232]] close AFTER decrement file_open_counter = 0
log_message: /irods_plugin_source/s3_transport/include/irods/private/s3_transport/s3_transport.hpp:448 (operator()) [[138014856735232]] [last_file_to_close=true][know_number_of_threads=true][threads_remaining_to_close=0]

log_message: [-]        /irods_source/server/api/src/rsFileClose.cpp:120:int _rsFileClose(rsComm_t *, fileCloseInp_t *) :  status [S3_PUT_ERROR]  errno [] -- message [fileClose failed for [3]]
        [-]     /irods_source/server/drivers/src/fileDriver.cpp:167:irods::error fileClose(rsComm_t *, irods::first_class_object_ptr) :  status [S3_PUT_ERROR]  errno [] -- message [failed to call 'close']
                [-]     /irods_plugin_source/s3_transport/include/irods/private/s3_transport/s3_transport.hpp:2306:irods::experimental::io::s3_transport::error_codes irods::experimental::io::s3_transport::s3_transport<char>::s3_upload_file(bool) [CharT = char] :  status [S3_PUT_ERROR]  errno [] -- message [failed in S3_put_object]


log_message: Failed to close file object [error_code=-702000].
log_message: [close_replica_without_catalog_update:288] - error closing replica; ec:[-702000]
log_message: [single_buffer_put:337] - error closing replica; ec:[-702000]

[James, Justin Kyle]

I don't see anything obvious.  What version of iRODS and the S3 plugin are you running?

I may have to rebuild the plugin with curl trace logging so we can see the output to determine what is wrong with the XML.

If possible before I do that, could you try to capture the traffic with tcpdump or wireshark?  You might have to force HTTP to do that.

Also, are we sure that the server is using TLS?  I would think the error would be different if not but worth asking.

---

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of joris luijsterburg <jlu...@gmail.com>

Sent: Thursday, June 25, 2026 10:33 AM
To: iRODS-Chat <irod...@googlegroups.com>
Subject: Re: [iROD-Chat:22564] s3 resource plugin help

 

To view this discussion visit https://groups.google.com/d/msgid/irod-chat/6de19f25-1320-438d-b7bb-e929cef62a94n%40googlegroups.com.

[Mikael Loaec]

Hi,

If I check the documentation here: https://github.com/irods/irods_resource_plugin_s3/blob/main/README.md

iadmin mkresc s3resc s3 $(hostname):/s3-irods-bucket-name/prefix/in/bucket "S3_DEFAULT_HOSTNAME=s3.us-east-1.amazonaws.com;S3_AUTH_FILE=/var/lib/irods/s3.keypair;S3_REGIONNAME=us-east-1;S3_RETRY_COUNT=1;S3_WAIT_TIME_SECONDS=3;S

3_PROTO=HTTP;ARCHIVE_NAMING_POLICY=consistent;HOST_MODE=cacheless_attached"

Perhaps I am wring but your  S3_DEFAULT_HOSTNAME should be new.local.store.com not new.local.store.com/s3

Regards,

Mikael

[James, Justin Kyle]

I was wondering if that was it.  In general hostnames should not have a slash in them.

---

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Mikael Loaec <mikael...@fgtech.fr>
Sent: Thursday, June 25, 2026 11:32 AM
To: iRODS-Chat <irod...@googlegroups.com>
Subject: Re: [iROD-Chat:22569] s3 resource plugin help

 

You don't often get email from mikael...@fgtech.fr. Learn why this is important

To view this discussion visit https://groups.google.com/d/msgid/irod-chat/407eca8c-a990-44fc-b9c6-b69e2ca01e9an%40googlegroups.com.

[joris luijsterburg]

That is indeed my question. The s3 host is currently provided including the /s3 suffix after the hostname. I will try to do some tcpdump or wireshark, but we need  to do some networkconfig before I can try that out. I'll let you know!

[James, Justin Kyle]

Could you try an entry in the /etc/hosts file that allows you to use a different hostname without a slash?

I'm not 100% convinced that the slash is the issue because the error is an XML parsing error but those errors are generated in libs3 and it could be a misleading error.

I just tried the opposite:  I put an entry in /etc/hosts for hostname "slash/" and put "http://slash/:9000" as the S3_DEFAULT_HOSTNAME but if failed to resolve the host.  I can "telnet slash/ 9000" and reach it.  So I'm not sure what is causing that.  Either way, it is worth trying the /etc/hosts trick.

---

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of joris luijsterburg <jlu...@gmail.com>

Sent: Friday, June 26, 2026 4:53 AM
To: iRODS-Chat <irod...@googlegroups.com>
Subject: Re: [iROD-Chat:22573] s3 resource plugin help

 

To view this discussion visit https://groups.google.com/d/msgid/irod-chat/4dc65963-5571-4064-908a-2a5db80c0640n%40googlegroups.com.