Using Orthanc to index a directory and serve files from it

[Peter Somlo]

Hi,

it's been a while since I posted to the group, hope everything is going well.

I have the following scenario: in a directory, I have a thousands of
studies stored as dicom files, is it possible to tell Orthanc to scan
and index those files (generate an SQLite file containing the index),
and use the existing dicom files as the storage without moving them?
I would like to get quick, readonly, access to plain files through the
DICOM protocol. The structure of the folders is different than the one
used by Orthanc when it stores the filesfor itself.
Is there a quick way of achieving it, or do I have to send them to
Orthanc with a DICOM Store command or through the REST API?

Best regards,

peter

[Sébastien Jodogne]

Hi Peter,

Your question is closely related to the following task on the roadmap to 1.0.0 release:

https://trello.com/c/zEcG86ra

The idea is to have a switch in the configuration file to prevent Orthanc from writing the DICOM files on the disk, while preserving their indexing and the generation of a JSON summary of their DICOM tags. Initially, such a feature was thought to be very helpful to help and bridge the gap between EMR and WADO servers [1], but it happens that it would also solve your use case.

In your case, you would prevent Orthanc from duplicating the DICOM file in its own database. Whenever you would send an instance to Orthanc using the REST API, you would associate it with a custom metadata containing the path to the original DICOM file. In a second time, after doing a search with Orthanc to locate an instance of interest, you would be able to retrieve the DICOM file by inspecting the latter metadata.

To summarize, there will soon be a way to achieve your requirement (say, around September 2014). In the meantime, you are invited to use DICOM C-Store or the REST API.

HTH,

Sébastien-

[1] http://www.codeproject.com/Articles/797118/Implementing-a-WADO-Server-using-Orthanc

[Sébastien Jodogne]

Hello Peter,

It is now possible to use Orthanc in a "index-only" mode by turning the "StoreDicom" option to "false" in the configuration file.

When this mode is enabled, Orthanc does not store the full DICOM file: It updates its index and only stores the JSON summary of the DICOM file.

You can assign a metadata to each DICOM instance (using the REST API) to store the location of the original DICOM file on your drive.

HTH,

Sébastien-

[Chris Hafey]

Wow, very nice!  One trick you can do with this is have the images sent using the JPIP transfer syntax.  When this is done, a URL to the JPIP stream is sent instead of the pixel data. This way you can build/populate Orthanc with metadata without the overhead of the pixel data!  This requires that the PACS sending data in this way (not many do, but some will)

[jan.van...@gmail.com]

Hi Sebastien,
Thanks a lot for that great piece of work.
Indexing files in-place is a great feature.
In your message you indicate 'using the REST API, you would associate it with a custom metadata containing the path to the original DICOM file.'.
Would you have a link that could indicate how to perform that?
Thanks a lot
jan

[Sébastien Jodogne]

Hello,

You can find examples of setting/reading/clearing a metadata using the REST API in the integration tests of Orthanc:

https://bitbucket.org/sjodogne/orthanc-tests/src/5920954287dd602d065314282753d3d41b76475e/Tests/Tests.py?at=default&fileviewer=file-view-default#Tests.py-910

These examples are written in Python, but they can obviously be ported to other languages.

HTH,

Sébastien-

[jan.van...@gmail.com]

Thanks a lot,
I will carefully read the code. My initial goal was to also to index a directory and keep dicom file in place. I will give it a try!
Thank again
Jan

[nmhaig...@gmail.com]

Am Donnerstag, 9. Juni 2016 14:10:32 UTC+2 schrieb jan.van...@gmail.com:
> On Wednesday, June 8, 2016 at 10:36:58 AM UTC+2, Sébastien Jodogne wrote:
> > Hello,
> >
> >
> > You can find examples of setting/reading/clearing a metadata using the REST API in the integration tests of Orthanc:
> > https://bitbucket.org/sjodogne/orthanc-tests/src/5920954287dd602d065314282753d3d41b76475e/Tests/Tests.py?at=default&fileviewer=file-view-default#Tests.py-910
> >
> >
> > These examples are written in Python, but they can obviously be ported to other languages.
> >
> >
> > HTH,
> > Sébastien-
> >
> >
> > On Thursday, June 2, 2016 at 11:19:27 PM UTC+2, jan.van...@gmail.com wrote:Hi Sebastien,
> >
> > Thanks a lot for that great piece of work.
> >
> > Indexing files in-place is a great feature.
> >
> > In your message you indicate 'using the REST API, you would associate it with a custom metadata containing the path to the original DICOM file.'.
> >
> > Would you have a link that could indicate how to perform that?
> >
> > Thanks a lot
> >
> > jan
> >
> >
> >
> >
> >
> > On Tuesday, September 9, 2014 at 4:58:50 PM UTC+2, Chris Hafey wrote:
> >
> > > Wow, very nice!  One trick you can do with this is have the images sent using the JPIP transfer syntax.  When this is done, a URL to the JPIP stream is sent instead of the pixel data. This way you can build/populate Orthanc with metadata without the overhead of the pixel data!  This requires that the PACS sending data in this way (not many do, but some will)

> >
> > > On Tuesday, September 9, 2014 9:55:08 AM UTC-5, Sébastien Jodogne wrote:
> >
> > > Hello Peter,
> >

> > > It is now possible to use Orthanc in a "index-only" mode by turning the "StoreDicom" option to "false" in the configuration file.
> >

> > > When this mode is enabled, Orthanc does not store the full DICOM file: It updates its index and only stores the JSON summary of the DICOM file.
> >

> >
> > > You can assign a metadata to each DICOM instance (using the REST API) to store the location of the original DICOM file on your drive.
> >
> > > HTH,
> >
> > > Sébastien-

> > > it's been a while since I posted to the group, hope everything is going well.
> >

> > > I have the following scenario: in a directory, I have a thousands of

> > > studies stored as dicom files, is it possible to tell Orthanc to scan

> > > and index those files (generate an SQLite file containing the index),

> > > and use the existing dicom files as the storage without moving them?

> > > I would like to get quick, readonly, access to plain files through the

> > > DICOM protocol. The structure of the folders is different than the one

> > > used by Orthanc when it stores the filesfor itself.
> >
> > > Is there a quick way of achieving it, or do I have to send them to
> Thanks a lot,
> I will carefully read the code. My initial goal was to also to index a directory and keep dicom file in place. I will give it a try!
> Thank again
> Jan
>
> > > Orthanc with a DICOM Store command or through the REST API?
> >

did you have any success in doing so?
if yes could you be so kind and publish your findings here?

i have the exact same problem (don´t wanna duplicate 2TB of dicom images)
but if i turn off the StoreDicom switch i cannot see the images with our dicomviewer

i don´t know how to store the path of the images within the dicomdata, so that external dicomviewers (radiant) cann "see" them ....

[jan.van...@gmail.com]

Not yet,
I'll keep you posted as soon as i succeed
jan

[bgold...@enlitic.com]

Hi all,

We're attempting to use Orthanc in a similar fashion. We have a large repository of DICOM files on a storage array we use for R&D purposes. We'd like to index and reference them in Orthanc for access from our DICOM viewers.

What is the metadata field that needs to be set?

Are there any thoughts about making this possible with a single API call in the future? It seems brittle to send the DICOM file to be processed in one request, and then have to followup with a second request to set the file path.

Thanks for all the great work!
Benjamin

[Sébastien Jodogne]

Hello,

This thread has become quite complex over time, because of the discussion of metadata.

Let me summarize what I understand from Benjamin and Peter's question. You wish to have Orthanc act as a real DICOM server (i.e. able to receive/send DICOM files), but you want the DICOM files stored by Orthanc to be accessible as a human-readable filesystem. I see at least four different approaches to solving this problem:

1. Simply replicate the DICOM files somewhere else, e.g. with a Python (cf. the "AutoClassify.py" sample in the source distribution of Orthanc) or a Lua script (cf. the "WriteToDisk.lua" sample).
2. As discussed above, set the "StoreDicom" configuration option to "false", so that Orthanc will not store the DICOM files after receiving them. In complement, implement a Lua script to write incoming instances somewhere on a filesystem, and to keep track of the actual file location thanks to some metadata.
   
3. Alter Orthanc in some way so that it stores the DICOM files elsewhere. It is very important to read the following FAQ in that respect: https://orthanc.chu.ulg.ac.be/book/faq/orthanc-storage.html#storage-folder
4. Publish the content of Orthanc as a virtual filesystem: https://en.wikipedia.org/wiki/Virtual_file_system

The first solution is clearly the simplest, but will double the size of the storage dedicated to DICOM. Depending on the size of your setup, this might still be a pragmatic solution.

The second solution has the advantage of limiting the size of the storage, at the price of a slightly higher complexity. However, it will not allow Orthanc to retrieve the raw DICOM files, which will prevent further transfers to other modalities. For this reason, I don't think this solution is suitable for Benjamin who needs to send the DICOM files to third-party viewers (but this restriction is maybe OK for Peter).

The third solution involves creating a C/C++ plugin that implements a custom storage area:

https://orthanc.chu.ulg.ac.be/book/developers/creating-plugins.html

Such a custom storage area can be registered into Orthanc through the "OrthancPluginRegisterStorageArea()" function of the Orthanc plugin SDK. The custom storage area must provide 3 functions to create/read/delete each attachment (as the notion of "attachments" encompasses DICOM files). The plugin can use the "type" argument to determine whether it is dealing with a DICOM file ("type == OrthancPluginContentType_Dicom"), or another type of attachment. The function "OrthancPluginDicomBufferToJson()" could be used to extract information about incoming files in order to determine their final location. In addition, the plugin will need to track in a separate database the mapping between the UUIDs generated by Orthanc and their actual location on the filesystem. To avoid any unnecessary complexity, make sure to disable "StorageCompression" in the configuration of Orthanc.

Note that if you are on a UNIX-based system (e.g. Linux), you could take advantage of symbolic links on the filesystem to avoid managing a separate database.

The fourth solution is certainly the most elegant, while being the most complex to implement. The idea would be to publish the DICOM files contained inside Orthanc, either as a read-only FTP server (which is possible either by creating a plugin, or possibly by using a framework such as pyftpdlib to create a custom FTP server on the top of the REST API of Orthanc), or as a FUSE filesystem (that would use REST API of Orthanc). The FUSE filesystem could in turn be published to Windows/Mac environments thanks to Samba.

HTH,

Sébastien-

PS: You know, creating Orthanc plugins is really fun ;)

On Monday, July 18, 2016 at 9:59:37 PM UTC+2, bgold...@enlitic.com wrote:

Hi all,

We're attempting to use Orthanc in a similar fashion. We have a large repository of DICOM files on a storage array we use for R&D purposes. We'd like to index and reference them in Orthanc for access from our DICOM viewers.

[jan van der mor]

Dear Sebastien,

Thx a lot for your answer. Solution 1 looks attractive.

I look at the autoclassify.py. Solutions are there, but time is the scare resource.

Jan

[Emil Pacurar]

I have a SFTP server where I organize data by original/sorted data: D:\USERS\ORIGINAL\Institution\patient. Sorted data and results from different processing under D:\USERS\PROCESSED\Institution\patient. Where institution and patient changes. More than 15Tb of data.

Any updates in the last version of Orphanc to help with these?

Thanks,

Emil

[Sébastien Jodogne]

Check out the Orthanc Book:

http://book.orthanc-server.com/users/replication.html#direct-access-to-the-filesystem

[Emil Pacurar]

Thanks, I am doing it now. But it is very very slow.

Each dicom file takes abut 1-1.5 seconds to post.

Is there a better version of this? Parallel upload or some other ways?

Thanks again,

Emil

[Alain Mazy]

Note: If you're running on windows and addressing your Orthanc server through "localhost", you might experience a 1 sec delay to resolve the DNS (no joke !) depending on your network configuration.  Try using 127.0.0.1 instead.

--

Alain Mazy
a...@osimis.io / +32 494 31 67 27

	OSIMIS S.A.  Quai Banning 6BE-4000 Liège  www.osimis.io