JPEG2000 compression of images in Orthanc

[Marc Delpont]

Hi,

is there a simple way to compress an image stored in orthanc, with lossless jpeg2000 compression, before rerouting it to peer ?

I'm thinking of using dcmcjp2k utility in the LUA function OnStoredInstance, but the command expects the full dicom file and creates another dicom file. Any idea ?

Thanks

Marc

[Sébastien Jodogne]

Hello,

You can achieve this goal in a clean way by creating an Orthanc plugin that would link against GDCM and compress each incoming file in the "OrthancPluginRegisterOnChangeCallback()".

However, the same functionality can be achieved through Lua scripting in a more hacky style:

https://bitbucket.org/sjodogne/orthanc/src/default/Resources/Samples/Lua/AutomatedJpeg2kCompression.lua

The trick is to make a system call to the standard command-line utility "dcmconv" from GDCM. Note that because of the test on the origin of the request to avoid infinite loops, this script will not work as such on Orthanc <= 0.9.3.

Sébastien-

[Simon Rascovsky MD, M.Sc]

The Lua way works OK, but it seems to be quite a roundabout way to do this, going outside orthanc and having 2 instances temporarily on disk, etc.

I'm afraid I don't understand the orthanc code enough to develop a plugin, but to begin to wrap my head around it, I was wondering a few things that Sebastian may be able to clarify for me:

1. Why do you suggest the OnChangeCallback instead of the OnStoredInstanceCallback ?
2. The basic need is for compression to be performed before storage. Is there a way to 'intercept' the incoming instance before it gets stored and assigned an uuid?
3. I'm concerned what effect compressing with the OnChangeCallback would have on the changes feed. I believe that if I modify compression on a file, the DICOM standard mandates it is given a new UID, but I'm not sure I would be able to filter out these modified instances from the changes feed as it doesn't currently support the instances 'origin'.

In short, I would appreciate some guidance on how to approach this. The [receive] => [recompress] => [store & index] => [forward] pipeline is a very common use case, and one that would be very useful for orthanc to support, natively or as a plugin.

Thanks!

Simon

[Sébastien Jodogne]

Hello,

Thanks for bringing this problem to my attention.

I have just added the missing function "OrthancPluginGetInstanceOrigin()" to the SDK:

https://bitbucket.org/sjodogne/orthanc/commits/2abfdca9b915735155a6cb616e6894a88a71dd69

The use case you mention can now be solved as follows:

1. Register a callback with "OrthancPluginRegisterOnStoredInstanceCallback()" that is called whenever a DICOM instance is received by Orthanc.
2. Inside this callback, use "OrthancPluginGetInstanceOrigin()" to ignore instances stored by plugins (this avoids an infinite loop because of step 4 below).
3. Apply the operations of your pipeline to the received DICOM instance.
4. Upload the result of your pipeline through "OrthancPluginRestApiPost()" against the "/instances" URI.
5. Remove the original DICOM instance through "OrthancPluginRestApiDelete()" against the "/instances/{id}" URI.

A sample plugin for JPEG2k conversion (using the command-line tool "gdcmconv") is also available:

https://bitbucket.org/sjodogne/orthanc/src/default/Plugins/Samples/AutomatedJpeg2kCompression/Plugin.cpp

Regards,

Sébastien-

[Simon Rascovsky MD, M.Sc]

Thank you Sebastian for your clear explanation. I think I'm starting to become more comfortable with the plugin framework to understand it.

I also tried the plugin sample, which works well. While trying it I confirmed a concern I had which I wanted to share with you:

The plugin in fact does create a new instance with a newly generated SOPInstanceUID from dcmodify. There is an edge case (actually not so edge when studies are pushed manually from the modality to an archive) in which the technologist will re-send the images when there is doubt about whether the study arrived complete or not.

The problem here is that Orthanc will accept the instances as new instances for the same series, thus duplicating the number of images in the series.

Have you come across this use case? I honestly don't know what would be the best approach for solving this, and your thoughts would be most appreciated.

Simon

[Sébastien Jodogne]

Well, this plugin is only a sample... It must of course be adapted and improved to become more robust.

Here are 4 different possibilities to deal with the re-pushing of same images, from the simplest to the most complicated approach:

• Simply do not regenerate the "SOPInstanceUID" (i.e. avoid the second "system()" call). This is however not desirable in practice, as it breaks down the global uniqueness assumption of DICOM identifiers.
• Retrieve the metadata "IndexInSeries" of the newly received instance, and compare it to the "IndexInSeries" that are already present in the parent series. If this index is already present in the series, do not convert it to JPEG2k.
• Store the "SOPInstanceUID" of the original instance as a custom (user-defined) metadata to the compressed instance. Before compressing an incoming image, make sure that the parent series does not contain a JPEG2k-compressed instance whose custom metadata has the same UID as the incoming instance.
• In a separate database (e.g. SQLite), keep a map between the "SOPInstanceUID" of the original instance and that of the compress instance. The database consistency can be e.g. obtained by monitoring changes with the plugin.

Personally, I would favor the third option, maybe with some memory caching to reduce search time.

Please also note that in a production plugin, you should not make "system()" calls but instead link directly to GDCM (or another library).

Sébastien-

[Simon Rascovsky MD, M.Sc]

Once again thank you for laying out so clearly.

FYI David Clunie commented on the issue on the comp.protocols.dicom google group here. Contrary to what I thought, It seems keeping the original SOPInstanceUID with lossless compression is the correct thing to do. I'm probably going to go that way, deleting the original instance and uploading the compressed instance with the original UID.

As for the OrthancPluginGetInstanceOrigin() function, it enables filtering new instances by origin, as the origin['RequestOrigin'] parameter in the Lua OnStoredInstance function. What seems to be missing to have equal functionality across all interfaces, is a way to filter by origin via the REST API. Perhaps an 'origin' object in the changes feed?

Thanks for your help.

Simon

[Sébastien Jodogne]

Thanks for sharing this very interesting information from David Clunie!

The origin of an instance is now available from the REST API, as a metadata attached to the instances, e.g.:

# curl http://localhost:8042/instances/60c98799-c2204ff9-ab177b91-db27764f-a815b614/metadata/Origin

This brings equal functionality across all interfaces.

Sébastien-

[Simon Rascovsky MD, M.Sc]

Thank you Sebastian!

What are your thoughts on adding this object to the changes feed? This would allow us to avoid making one API call for each instance in order to filter processed images based on origin...

Not a big deal, but would probably lighten the load on the server.

Cheers!

Simon

[Sébastien Jodogne]

Thanks for this suggestion!

However, I do not feel the change feed is the proper place to store this information, as it is available elsewhere (in metadata) to serve other kind of purposes. I prefer to keep each information in an unique place to enforce data consistency, and to reduce the amount of information to be stored by the database.

As the embedded Web server of Orthanc uses a thread pool, making 2 successive REST requests does not put a much higher load on the server. As you also notice, the possible problem is rather an increased response time, but only for the users of the REST API (plugins and Lua have direct access to the API, bypassing the full network stack). In such a case, the HTTP clients should make their requests in parallel to optimize their running time.

Sébastien-

[Simon Rascovsky MD, M.Sc]

Got it, thanks!