DicomWeb performance seems slow on the metadata query

1,649 views
Skip to first unread message

Howard Lander

unread,
Oct 3, 2019, 4:19:02 PM10/3/19
to Orthanc Users
Hi All

I am using the latest orthannc docker with a postgres back end.  The postgres files are stored on a local SSD, postgres itself is running in another docker and everything is running on an Amazon VM with 8 CPUS and 32 GB of memory, with no real other load.  I've got about 100 studies loaded: the typical study is composed of 4 series, with each series containing roughly 4 to 5 hundred images. The problem I'm seeing is that calls to get metadata to an individual study seem quite slow. For example, I logged into the orthanc container and ran this query:


and the result was 

real 0m54.636s
user 0m0.009s
sys 0m0.653s

That seems odd to me and leads me to believe I have done something wrong.  I've attached my orthanc.json file. Any thoughts/suggestions would be greatly appreciated.

My docker compose looks like this:

version: '3.6'
services:

   cloudtop:
      image: heliumdatastage/renci-guacamole-vnc:use-all-defaults
      container_name: cloudtop
      volumes:
         - ${ORTHANC_DATA_DIR:-./data}:/usr/local/renci/data:ro
      entrypoint: /init
      ports:
         - 8080:8080

   viewer:
      image: ohif/viewer:v1.1.14.3708
      container_name: viewer
      volumes:
         - ${VIEWER_CONFIG:-./config/renci.js}:/usr/share/nginx/html/app-config.js
         - ${NGINX_CONFIG:-./config/nginx-with-proxy.conf}:/etc/nginx/conf.d/default.conf
      environment:
         - MONGO_URL=mongodb://mongo:27017/ohif
         - APP_CONFIG=/usr/share/nginx/html/app-config.js
      depends_on:
         - mongo
         - orthanc
      ports:
         - "3000:80"

   orthanc:
      image: jodogne/orthanc-plugins:latest
      container_name: orthanc
      depends_on:
         - postgres
      volumes:
         - ${ORTHANC_CONFIG:-./config/orthanc.json}:/etc/orthanc/orthanc.json:ro
      ports:
         - "8042:8042"
         - "4242:4242"

   mongo:
      image: mongo:latest
      container_name: mongo
      ports:
         - "27017:27017"
      volumes:
        - ${MONGO_DATA:-./mongo_data}:/data/db

   postgres:
      image: postgres:11
      container_name: postgres
      volumes:
        - ${POSTGRES_DATA_MNT:-/orthadb/postgresql/data/pgdata}:${PGDATA:-/var/lib/postgresql/data}
      environment:
        - POSTGRES_USER=${POSTGRES_USER:-NOTMYREALUSER}
        - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-NOTMYREALPASSWORD}
        - PGDATA=${PGDATA:-/var/lib/postgresql/data}
        - POSTGRES_DB=orthanc
      ports:
         - "5432:5432"

Thanks
Howard
orthanc.json

Chris McGee

unread,
Oct 3, 2019, 5:08:23 PM10/3/19
to Howard Lander, Orthanc Users
Yes, this is happening because the metadata needs more information than what is in the orthanc index database. The dicomweb plugin has to fetch each of the dicom files in the study and read out all of the tags. Over a slow filesystem that gets really slow.

Chris
--
You received this message because you are subscribed to the Google Groups "Orthanc Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orthanc-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/orthanc-users/8baf94a2-85e5-4bf2-a2da-01e6e6497c5b%40googlegroups.com.
<orthanc.json>

Lander, Howard Michael

unread,
Oct 3, 2019, 6:37:46 PM10/3/19
to Chris McGee, Orthanc Users
Hi Chris.

Thanks much for the reply. The postgres filesystem is on an SSD. Is there some faster way of going about this? I tried to configure postgres to load everything into memory, but it didn't seem to help.

Howard
From: Chris McGee <cmc...@macadamian.com>
Sent: Thursday, October 3, 2019 5:08:17 PM
To: Lander, Howard Michael <how...@renci.org>
Cc: Orthanc Users <orthan...@googlegroups.com>
Subject: Re: DicomWeb performance seems slow on the metadata query
 

Luiz Eduardo Guida Valmont

unread,
Oct 4, 2019, 8:24:57 AM10/4/19
to Lander, Howard Michael, Chris McGee, Orthanc Users
Hi, friends!


I'm experiencing something related to this issue. What I came understand is that the size of the index affects dicom-web's performance. If we were to plot the total dicom-web request time as measured by the size of the associated index, it would look like a log function: a steep climb right at the start, followed by a plateau going into infinity.

Sure, there could be many more things affecting (I reckon there are plenty). One of them is what Chris pointed out.

Best,
Luiz


Chris McGee

unread,
Oct 4, 2019, 8:30:22 AM10/4/19
to Lander, Howard Michael, Orthanc Users
Hi Howard,

In my case studies rarely change once they are added. I put a caching proxy in front of orthanc and it did the trick after the first slow access. There's alot of metadata in a MRI study. I've seen 10's of MB of JSON metadata in some of our studies, which seems impressive. I suppose even with an SSD, reading out all of those files to get that volume of metadata each time is still slow.

At some point in the future, I may need to investigate what metadata is really needed in my case and see if there is a way to convince Orthanc to store those in its index DB and also the dicomweb plugin to only serve those out. I've seen some interesting code comments in the dicomweb plugin in this area suggesting that there could be a configuration parameter some day to adjust the cost vs. spec compliance tradeoff of the output.

Cheers,
Chris

Sébastien Jodogne

unread,
Oct 6, 2019, 5:04:46 AM10/6/19
to Orthanc Users
Hello,

I second Chris' answer: Retrieving DICOMweb metadata requires to access all the DICOM files of the study/series of interest on the filesystem, which has a high cost.

However, the DICOMweb calls to "/metadata" have recently been optimized in the mainline of the plugin, so you should give a try to the latest development version of the code (or wait for a new official release), which should already speed up performance:

Furthermore, as Chris suggests, we are considering to implement two complementary strategies to improve the performance of calls to ".../metadata":
  • Provide an option to only retrieve the so-called "main DICOM tags" that are stored within the Orthanc index: This will only use the SQL database (no DICOM file will be accessed), hereby greatly reducing the reading time on the filesystem. Unfortunately, the retrieved information will be basic, and will not be sufficient for some applications.
  • Implement a caching system where the DICOMweb JSON is computed only once (either on store or on first access), and stored as an attachment to the DICOM files. This avoids the cost of parsing large DICOM files multiple times, but the implementation is complex, and cache coherency between multiple versions of Orthanc might be problematic.
At the time of writing, it is not clear whether either of these strategies (or both of them) will finally get implemented. You can follow this development in the function "WriteInstanceMetadata()" within the following file:

HTH,
Sébastien-


On Friday, October 4, 2019 at 2:30:22 PM UTC+2, Chris McGee wrote:
Hi Howard,

In my case studies rarely change once they are added. I put a caching proxy in front of orthanc and it did the trick after the first slow access. There's alot of metadata in a MRI study. I've seen 10's of MB of JSON metadata in some of our studies, which seems impressive. I suppose even with an SSD, reading out all of those files to get that volume of metadata each time is still slow.

At some point in the future, I may need to investigate what metadata is really needed in my case and see if there is a way to convince Orthanc to store those in its index DB and also the dicomweb plugin to only serve those out. I've seen some interesting code comments in the dicomweb plugin in this area suggesting that there could be a configuration parameter some day to adjust the cost vs. spec compliance tradeoff of the output.

Cheers,
Chris

On Oct 3, 2019, at 6:37 PM, Lander, Howard Michael <how...@renci.org> wrote:

Hi Chris.

Thanks much for the reply. The postgres filesystem is on an SSD. Is there some faster way of going about this? I tried to configure postgres to load everything into memory, but it didn't seem to help.

Howard
From: Chris McGee <cmc...@macadamian.com>
Sent: Thursday, October 3, 2019 5:08:17 PM
To: Lander, Howard Michael <how...@renci.org>
Cc: Orthanc Users <orthanc-users@googlegroups.com>
To unsubscribe from this group and stop receiving emails from it, send an email to orthanc-users+unsubscribe@googlegroups.com.
<orthanc.json>

Lander, Howard Michael

unread,
Oct 7, 2019, 1:10:10 PM10/7/19
to orthan...@googlegroups.com
Hi Sébastien

Thanks very much for the response.  I've downloaded and built the dev version of the dicom-web plugin.  I'm running orthanc as a docker, so I thought I could just add my version of the dicom-web plugin to the plugin list and map the plugin into the docker.  But I am getting errors like this:

W1007 17:02:17.332577 main.cpp:702] Loading plugin(s) from: /usr/share/orthanc/plugins/renci/libOrthancDicomWeb.so
E1007 17:02:17.344985 PluginsManager.cpp:264] Plugin 'dicom-web' already registered

followed eventually by

E1007 17:02:19.308291 main.cpp:1502] Uncaught exception, stopping now: [Error while using a shared library (plugin)] (code 25)

Do I need to rebuild the docker?  Any other advice on how to get this to work?

For reference, here's the relevant part of orthanc.json

  "Plugins" : [
    "/usr/share/orthanc/plugins","/usr/local/share/orthanc/plugins","/usr/share/orthanc/plugins/renci/libOrthancDicomWeb.so"
  ],

And the relevant portion of my docker compose file:


   orthanc:
      image: jodogne/orthanc-plugins:latest
      container_name: orthanc
      depends_on:
         - postgres
      volumes:
         - ${ORTHANC_CONFIG:-./config/orthanc.json}:/etc/orthanc/orthanc.json:ro
         - ${ORTHANC_CONFIG:-./localPlugins}:/usr/share/orthanc/plugins/renci:ro

      ports:
         - "8042:8042"
         - "4242:4242"

Thanks again
Howard
To unsubscribe from this group and stop receiving emails from it, send an email to orthanc-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/orthanc-users/3317fb96-d2e3-482e-b154-a70b33f36b4b%40googlegroups.com.

--
Howard Lander
Senior Research Software Developer
Renaissance Computing Institute (RENCI)
The University of North Carolina at Chapel Hill
100 Europa Drive
Suite 540
Chapel Hill, NC 27517
919-445-9651

Lander, Howard Michael

unread,
Oct 7, 2019, 2:09:44 PM10/7/19
to orthan...@googlegroups.com
Hi all

So I just build the database plugins, added them to my RENCI directory and got rid of the other plugin directories from orthanc.json, so now the docker is only using the plugins I build. It all works and it is faster, so that's good news.

From reading your previous response, it seems that the metadata call is still reading metadata from files on the disk.  I'll consider moving our files to a faster storage solution. I can't comment much about the 2 possible strategies you suggest for further speedups, other than to state the obvious, which is that if the "main DICOM tags" are sufficient for the downstream apps (in my case the OHIF viewer) then the first strategy could be a real benefit.

Thanks
Howard

Salim Kanoun

unread,
Oct 7, 2019, 6:07:02 PM10/7/19
to Orthanc Users
Dear Sebastien,

Thank you for considering optimizing the retrieve speed of the DicomWeb plugin. I was also looking at these speed issue with OHIF team, that does optimization on their side too.

To speed up the data retrieve OHIF has changed the called API, they were retrieving the full study metadata with /study/UID/metadata that could be very long to get an answer due to the huge number of instances in this single call.

They now splitted it into individual /series/UID/metadata call that are faster.

I noticed something strange, the first call they do is to list the series level with /study/UID/series but this call took 18 seconds to generate on my server (Orthanc + mySQL database in docker + HDD storage).
This 18 seconds seems to indicate that the study/UID/series triggers a DICOM file scan. 
While when I look to the generated answer, I see only one item that seems to be not directly accessible in the "MainDicomTags" which I supposed are stored in Orthanc database, this tag is 0008,0005 Specific Character Set

If my supposition is correct I wonder if it would be possible to store this additional tag in the database to free the study/UID/series  from individual file scanning ?

Best regards,

Salim
Hello,

Sébastien-

Chris

Cc: Orthanc Users <orthan...@googlegroups.com>
To unsubscribe from this group and stop receiving emails from it, send an email to orthan...@googlegroups.com.
<orthanc.json>

Salim Kanoun

unread,
Oct 14, 2019, 7:32:47 AM10/14/19
to Orthanc Users
Just to give some additional "thought" as I realize my post might not be easy to understood.

The /Study/UID/Metadata is very long to generate beauce of the reading of all instances in a study as it can have thousands of instances (multiple MRI sequences, PET/CT acquisition etc..)

So metadata retrieval at the study level probably should be avoided, and preferentially loaded at the serie level. when the images of a series are loaded this is sufficiantly long operation to make the metadata retrieval without huge impact in the loading time (images are much longuer to retrieve than metadata).

On the other hand, when opening the study it is important to see what series are in it, to be able to select the series you want to pay the price to open.

So the trouble is that the Studies/UID/Series still trigger a scan of all DICOMs instance so this call is still having more or less the same timing cost than the full metadata retrieve at study level.

Maybe i'm missing another way to retrieve the serie's details (just the main tags) from DicomWeb without triggering all the instances scan ? 

Best regards,

Salim 

Sébastien Jodogne

unread,
Feb 17, 2020, 11:04:48 AM2/17/20
to Orthanc Users
Dear all,

This is an update to this issue. I have just committed the following changeset:

This changeset introduces two new configuration option, "StudiesMetadata" and "SeriesMetadata" that are documented in the Orthanc Book:

Briefly speaking, those configuration options allow to choose how Orthanc will compute the "metadata" endpoints: either by reading the full DICOM instances from the storage area ("Full" mode), or by using only the main DICOM tags as stored in the index database ("MainDicomTags" mode).

Besides these configuration options, forthcoming 1.1 release of the DICOMweb also divides by two the number of files that are read from the storage area (the "DICOM-as-JSON" attachments are not used anymore by the "metadata" routes).

As far as DICOMweb viewers are concerned, it is recommended to:
  • Set option "StudiesMetadata" to value "MainDicomTags".
  • Set option "SeriesMetadata" to value "Full".
  • First make a request to "/studies/.../metadata" to have an insight about all the instances that are stored in the study. This should be fast, thanks to "StudiesMetadata" being to "MainDicomTags".
  • Then make a request to "/studies/.../series/.../metadata" when it comes to rendering one given series from the study. This will give accurate results, actually the same as in releases <= 1.0 of the DICOMweb plugin.
HTH,
Sébastien-

Salim Kanoun

unread,
Feb 18, 2020, 11:43:10 AM2/18/20
to Orthanc Users
Amazing,

Thanks Sebastien !

Sébastien Jodogne

unread,
Feb 21, 2020, 11:45:09 AM2/21/20
to Orthanc Users
Hello again,

FYI, I have just implemented a third mode of operation that is named "Extrapolate", and that is now documented in the Orthanc Book:

If Extrapolate mode is used, the plugin will read up to 3 DICOM instances at random that belong to the study/series of interest. It will then test whether the majority of these instances share the same value for a predefined subset of DICOM tags. If so, this value is added to the metadata response; otherwise, the tag is not reported. In other words, this mode extrapolates the value of some predefined tags by assuming that these tags should be constant across all the instances of the study/series. This mode is a compromise between MainDicomTags (focus on speed) and Full (focus on accuracy).

The related changeset is the following:

This mode should be useful to people developing DICOMweb viewers, and will in particular be used in the future Stone Web viewer.

Again, this mode will be part of forthcoming 1.1 release of the DICOMweb plugin.

HTH,
Sébastien-
Reply all
Reply to author
Forward
0 new messages