Handling C-MOVE requests with a high-latency storage (REST/Lua/Plugin)
55 views
Skip to first unread message
Knut
Jan 18, 2019, 5:25:27 AM1/18/19
to Orthanc Users
Hi all,
I am currently experimenting with the StorageArea Plugin and REST Api to combine Orthanc with high-latency storages (such as Tape or aws Glacier)
Handling C-Store doesn't seem to be a very huge issue as long as you keep the JSON header files locally and use a local cache before moving the DICOMS to the final storage.
The other way around naturally is a bit more nitty-gritty, since it may take hours to provide the requested files on a local cache to serve the StorageRead function in the Storage Plugin.
So 2 questions:
1/ is there a certain timeout when the Orthanc core calls the Storage Area plugin (StorageRead), maybe even a general issue with blocking the Application as a whole?
2/ If yes/alternatively: is there a general possibility (REST, Plugin), to externally link into the general C-Move -> C-Store loop?
Concerning 2/ I am thinking about a workflow like:
onC-MOVErequest: event providing the Moveoriginator and the original request to trigger an external storage handler
As soon as the requested files are available on a local storage: trigger the final C-Store via REST, including the Move Originator etc., to provide the correct context)
Thanks for the help and the great software. Just discovered Orthanc some one month ago and already love it!
I am currently experimenting with the StorageArea Plugin and REST Api to combine Orthanc with high-latency storages (such as Tape or aws Glacier)
Handling C-Store doesn't seem to be a very huge issue as long as you keep the JSON header files locally and use a local cache before moving the DICOMS to the final storage.
The other way around naturally is a bit more nitty-gritty, since it may take hours to provide the requested files on a local cache to serve the StorageRead function in the Storage Plugin.
So 2 questions:
1/ is there a certain timeout when the Orthanc core calls the Storage Area plugin (StorageRead), maybe even a general issue with blocking the Application as a whole?
2/ If yes/alternatively: is there a general possibility (REST, Plugin), to externally link into the general C-Move -> C-Store loop?
Concerning 2/ I am thinking about a workflow like:
onC-MOVErequest: event providing the Moveoriginator and the original request to trigger an external storage handler
As soon as the requested files are available on a local storage: trigger the final C-Store via REST, including the Move Originator etc., to provide the correct context)
Thanks for the help and the great software. Just discovered Orthanc some one month ago and already love it!
Knut
Jan 18, 2019, 7:31:58 AM1/18/19
to Orthanc Users
Maybe just an additional remark:
I know about the OrthancPluginMoveCallback but as far as I understand it the driver function will be called by Orthanc directly after the callback function has been performed, being responsible for the respective C-Stores.
The scenario I describe in 2/ would mean to stop the Orthanc involvement after the Query part and trigger the C-Store (adding parameters for Moveoriginator and ID) e.g. via REST when the DIcom files have been retrieved from the external storage to the local cache.
I know about the OrthancPluginMoveCallback but as far as I understand it the driver function will be called by Orthanc directly after the callback function has been performed, being responsible for the respective C-Stores.
The scenario I describe in 2/ would mean to stop the Orthanc involvement after the Query part and trigger the C-Store (adding parameters for Moveoriginator and ID) e.g. via REST when the DIcom files have been retrieved from the external storage to the local cache.
Sébastien Jodogne
Jan 19, 2019, 6:05:22 AM1/19/19
to Orthanc Users
Hello,
Orthanc is built upon the assumption that it can read from its storage area in a "reasonable time" (i.e. below HTTP timeouts in the REST API, and below TCP timeouts if using the DICOM network protocol). This is fine for "regular" blob storage ("Standard" and "Standard-IA" on S3), not for the S3 Glacier.
In other words, you cannot combine the "real-time" requirements of a DICOM modality/PACS/VNA, with the several hours that might be needed to read from the Glacier. You are simply considering an use case for which Glacier is not designed for:
You must re-think your workflow, and carefully ask yourself: How could e.g. a DICOM viewer ask to retrieve an archived DICOM study from the Glacier, then be warned when this study is retrieved from Glacier, possibly a few hours after the initial request? This is not possible with the DICOM protocol.
In either case, the solution will most probably be to use Orthanc as a local cache, combined with an Orthanc plugin that will be in charge from retrieving a queue of studies from the Glacier using background threads. The content of this queue could be specified by REST calls that extend the REST API of Orthanc.
HTH,
Sébastien-
Sébastien Jodogne
Jan 20, 2019, 3:46:02 AM1/20/19
to Orthanc Users
Just a re-think about my previous answer: It could indeed be envisioned to have a workflow where the C-FIND/C-MOVE requests are transparently separated from the C-STORE answers emerging after the retrieval from the Glacier (possibly few hours after the C-MOVE).
This would however require to have access to the C++ class "Orthanc::DicomUserConnection()" that is implemented by the Orthanc core, in the Orthanc plugin SDK.
We may consider this for inclusion in the long-term, but this is clearly not one of our priorities.
Knut
Jan 20, 2019, 7:49:07 AM1/20/19
to Orthanc Users
Hi Sébastien,
yup, that's what I thought.
It would be quite interesting since it would open the door to use Orthanc as a Dicom front end for object storage in an environment targetted for longterm archiving
(mainly storing images with only a few retrieves)
Sébastien Jodogne
Jan 20, 2019, 8:04:33 AM1/20/19
to Orthanc Users
OK, this is added to our long-term roadmap:
We will consider this development as soon as we find funding from the industry.
Reply all
Reply to author
Forward
0 new messages