dcm4chee-arc-psql:5.31.1: Move files from FS1 storage to S3 storage after X days

[Marius S]

Hello,

I have docker version of dcm4chee-arc-psql:5.31.1

I would like to test the following workflow: When new studies are added to the DCM4CHEE studies are stored in local fs1 storage for 1 day and after 1 day it should be  moved to S3 storage.

Yesterday (20240124) I did the setup and uploaded one dicom file to DCM4CHEE and when I expand study tags I see that it was stored in fs1 and it has the Study expiration date and Exporter ID:
Study Expiration Date (7777,1023): 20240125
Storate IDs of Study (7777,1028): fs1
Study Expiration Exporter ID (7777,102C): Foward To S3

However today the file has not been moved to S3.
Also there is no scheduled/completed/failed tasks in Monitoring > Queues and Monitoring > Export. Queues and Export are empty.

What configuration should be done to move files from fs1 to S3 storage after 1 day?

From the docs and other topics what I did so far:

1. Created new storage for S3
dcm4chee-arc > Device Extensions > Archive Device > Storage Descriptor > + New Storage Descriptor

Storage ID: amazon-s3
Storage URI: jclouds:jclouds:aws-s3:https://<bucket-name>.us-east-1.amazonaws.com

Instance Availability: NEARLINE
Storage Duration: PERMANENT

Retrieve Cache Storage ID: fs1
Storage Property: ....

2. Edited fs1 storage and changed the settings:
dcm4chee-arc > Device Extensions > Archive Device > Storage Descriptor > fs1

Instance Availability: ONLINE
Storage Duration: CACHE

Delete Studies Received Before: P1D

3. In dcm4chee-arc > DCM4CHEE > Network AE Extension > Archive Network AE > Object storage ID I have left only fs1 checked, so the received/uploaded studies will be stored in fs1.

4. Created Exporter Descriptor

dcm4chee-arc > Device Extensions > Archive Device > Exporter Descriptor > + New Exporter Descriptor

Exporter ID: Forward To S3

Exporter URI: storage:amazon-s3 (Is this a correct URI format?)

Queue Name: Export Tasks (Export)

Delete Study From Storage ID: fs1

5. Created Exporter Rule

dcm4chee-arc > Device Extensions > Archive Device > Exporter Rule> + New Exporter Rule

Name: Forward to S3 Rule

Export Entity: Study

Exporter ID: Forward To S3

Is there anything else what need to be configured?
Where I have made a mistake in configuration?

PS. S3 storage works fine, before this test I was able to set amazon-s3 as "primary" storage, uploaded Dicom file and it was successfully stored in S3.

Thanks.

[Marius S]

Forgot one more configuration I have added:

6. Study Retention Policy

dcm4chee-arc > Device Extensions > Archive Device > Study Retention Policy > + New Study Retention Policy

Name: Study Move to S3

Study Retention Period: P1D

Export expired Study: Forward To S3

[Vrinda Nayak]

Check if you have configured Reject Expired Studies Polling Interval

[Marius S]

Hello Vrinda, thanks for information!
I think by default for Reject Expired Studies Polling Interval was set to 1 day, I changed to 2minutes (PT2M) and it seems it started triggering Export Descriptor.

However I think my configuration is still not correct for what I am trying to achieve.

I have this test docker DCM4CHEE environment on my computer on virtual machine.

During the weekend my computer was turned off and few studies had to be sent from FS1 to S3, but when I connected to my computer today it seems that the first thing was triggered dcm4chee-arc > Device Extensions > Archive Device > Storage Descriptor > fs1 Delete Studies Received Before: P1D

Because after some time I saw Export tasks with status SCHEDULED FOR RETRY X. When I checked FS1 storage /var/local/dcm4chee-arc/wildfly/data/fs1/2024/01/26 there was no files, also in logs I had this:

2024-01-29 08:20:30,154 WARN  [org.dcm4che.arc.export.storage.StorageExporter] (EE-ManagedExecutorService-default-Thread-745) Failed to copy Instance[iuid=1.3.6.1.4.1.44316.6.102.3.2023121865333482.557058889324992826769,cuid=1.2.840.10008.5.1.4.1.1.4] to Storage[id=amazon-s3, uri=jclouds:aws-s3:https://xxxxxxxxxxx.s3.us-east-1.amazonaws.com]:
: java.io.IOException: Failed to find location of Instance[iuid=1.3.6.1.4.1.44316.6.102.3.2023121865333482.557058889324992826769,cuid=1.2.840.10008.5.1.4.1.1.4]

So in this scenario I ended up with deleted files and there was no copy in S3.

So my question what settings need to be set to ensure that files from FS1 would get Deleted ONLY if they were successfully exported to S3 ?

Should I remove Delete Studies Received Before: P1D from FS1 Storage Descriptor, because it duplicates with Study Retention Period: P1D and Exporter Descriptor setting Delete Study From Storage ID: fs1 ?

PS. I already see that from my list Step 5. Created Exporter Rule is not needed, because if I have this rule, once I upload a DICOM file to DCM4CHEE it automatically puts it in FS1 and send to S3 immediately. So I removed Exporter Rule because I want to move files from FS1 to S3 after 1 day.

The workflow I am trying to achieve:

1. DCH4CHEE receives DICOM images
2. DCH4CHEE Stores DICOM images in FS1

3. After X days DICOM images moved from FS1 to S3 (need to ensure that it is actually moved to S3 before deleting from FS1)

Thanks,

M.

[Vrinda Nayak]

I don't understand the point of copying objects from fs1 to s3 only after 1 day and setting study retention period of 1 day. What is the logic for it? 

You might as well directly export the objects from fs1 to s3 as soon as you receive it. Objects will be deleted from fs1 CACHE only after they are exported to s3 if you have configured Export Storage ID = amazon-s3 on fs1 storage configuration. You don't need a Study Retention Policy at all. 

[Marius S]

Hello Vrinda,

1 day is only for testing, so I could see that everything works. In prod environment DICOM files will be stored in FS1 for ~1month and only then DICOM files should be exported to S3.

There will be no need to export DICOM files from FS1 to S3 as soon as I will receive it.

DICOM files will exist only in FS1 or S3, it will not be duplicated.

Export Storage ID setting on FS1 storage has this description: Constrains deletion of Studies, additionally to configured deleter thresholds and/or deletion retention period constraints, from the Storage System to Studies whose objects are also accessible from the specified other storages.

So as I understood this setting it would delete DICOM files from FS1 only if the same files would exist in S3, but in this case files are uploaded only to FS1 and will be be exporter to S3 after X days.

Do you suggest to use only the following settings?

1. Created new storage for S3
dcm4chee-arc > Device Extensions > Archive Device > Storage Descriptor > + New Storage Descriptor

Storage ID: amazon-s3
Storage URI: jclouds:jclouds:aws-s3:https://<bucket-name>.us-east-1.amazonaws.com

Instance Availability: NEARLINE
Storage Duration: PERMANENT

Storage Property: ....

2. Edited fs1 storage and changed the settings:
dcm4chee-arc > Device Extensions > Archive Device > Storage Descriptor > fs1

Instance Availability: ONLINE
Storage Duration: CACHE

Delete Studies Received Before: P1D

Export Storage ID: amazon-s3 

Thanks,

M.

[Vrinda Nayak]

Configuring just a Study Retention Policy should be enough for the use case you mentioned. Also, ensure Reject Expired Studies Polling Interval value is higher than Task Polling Interval. (Note : Archive's default configuration already handles this - dcmRejectExpiredStudiesPollingInterval: P1D / dcmTaskPollingInterval: PT1M)

However there is a bug which has been fixed and will available in archive version 5.32.0 (not yet released).

[Marius S]

Hello Vrinda Nayak,

I am now using 5.32.1 version (also tested 5.33.1) which you mentioned had this bug fixed. However I am experiencing strange behavior from 5.32 version.

I have set Reject Entity for Data Retention Expiry: false in my Exporter Descriptor but it ignores this parameter and still rejects the study which was copied to nearline storage.

1. I have two storages configured: fs1 and nearline.
2. I have created Queue: Nearline Storage Export

3. I have Created Exporter Descriptor

dcm4chee-arc > Device Extensions > Archive Device > Exporter Descriptor > + New Exporter Descriptor

Exporter ID: CopyToNearlineStorage

Exporter URI: storage:nearline

Queue Name: Nearline Storage Export

Delete Study From Storage ID: fs1

Reject Entity for Data Retention Expiry: false

4. I have created Study Retention Policy

dcm4chee-arc > Device Extensions > Archive Device > Study Retention Policy > + New Study Retention Policy

Name: StudyRetPolicyCopyToNearline

Study Retention Period: P30D

Export expired Study: CopyToNearlineStorage

5. For testing purpose I set: Reject Expired Studies Polling Interval: PT5M, Purge Storage Polling Interval: PT2M

6. I send a study to dcm4chee and change the Study expired date to the past to trigger moving files from fs1 to nearline.

However when files are moved from fs1 to nearline, the Study Expiration State is set to EXPORT_SCHEDULED it seems dcm4chee ignores the Reject Entity for Data Retention Expiry: false setting and still rejects the study from nearline:

.....

2025-06-05 10:36:44,027 INFO  [org.dcm4chee.arc.delete.impl.RejectExpiredStudiesScheduler] (EE-ManagedScheduledExecutorService-default-Thread-5) start RejectExpiredStudiesScheduler.execute()
2025-06-05 10:36:44,062 INFO  [org.dcm4chee.arc.delete.impl.RejectionServiceImpl] (EE-ManagedScheduledExecutorService-default-Thread-5) Start rejection of 3 instances of Study[UID=1.3.6.1.4.1.44316.3.300.1.20250605102645.890.9358283148989483356], Series[UID=null], SOPInstance[UID=null].
2025-06-05 10:36:44,099 INFO  [org.dcm4chee.arc.store.impl.StoreServiceEJB] (EE-ManagedScheduledExecutorService-default-Thread-5) DCM4CHEE: RejectedInstance[pk=10, time=Thu Jun 05 10:36:44 EEST 2025, code=(113039, DCM, "Data Retention Policy Expired")] of Instance[uid=1.3.6.1.4.1.44316.3.300.3.20250605102646.3.399161297018737999702, class=1.2.840.10008.5.1.4.1.1.7] of Series[uid=1.3.6.1.4.1.44316.3.300.2.20250605102645.1.503779913508766831919] of Study[uid=1.3.6.1.4.1.44316.3.300.1.20250605102645.890.9358283148989483356]

....

Could you please let me know why it is being rejected from nearline storage even if I have set Reject Entity for Data Retention Expiry: false in Exporter descriptor?

How to solve this problem, so that the files moved from fs1 to nearline would not be rejected?

Thanks,

Marius.

[Vrinda Nayak]

https://github.com/dcm4che/dcm4chee-arc-light/discussions/4747