Extremely slow /tools/find performance when specifing parameters other than studydate

[Rana Asim Wajid]

Im using the latest version of Orthanc with PostgreSQL backend. Studies are not being stored in PostreSQL. Other details are given below:

The problem I'm facing is that when I search for studies using /tools/find and specify any other parameter except studydate the search takes ages. for example

{ "Expand": true, "Level": "Studies", "CaseSensitive": false, "Query": { "PatientName": "**" , "PatientID": "**" , "StudyID": "**" , "StudyDate": "20211115-20211117" }}

the above query returns 85 studies in less than a second. but the below queries take any where from 97 to 123 seconds to return around 60 records

{ "Expand": true, "Level": "Studies", "CaseSensitive": false, "Query": { "PatientName": "**" , "PatientID": "**" , "StudyID": "**" , "StudyDate": "20211115-20211117" , "ModalitiesInStudy":"CT" }}

{ "Expand": true, "Level": "Studies", "CaseSensitive": false, "Query": { "PatientName": "**" , "PatientID": "**" , "StudyID": "**" , "StudyDate": "20211115-20211117" , "InstitutionName":"Abby Pet Clinic" }}

I'm at a total loss as to why this is happening.

[Rana Asim Wajid]

Correction: Searching with InstitutionName parameter works fine. It seems that anything outside the MainDicomTags is causing the issue.

[Rana Asim Wajid]

So I just did another test:

{ "Expand": true, "Level": "Studies", "CaseSensitive": false, "Query": { "PatientName": "**" , "PatientID": "**" , "StudyID": "**" , "StudyDate": "20211116-20211117" , "InstitutionName": "AnSurgical Center", "ModalitiesInStudy": "CT" }}

above query returns one study and takes 88.3 seconds

{ "Expand": true, "Level": "Studies", "CaseSensitive": false, "Query": { "PatientName": "**" , "PatientID": "**" , "StudyID": "**" , "StudyDate": "20211116-20211117" , "InstitutionName": "AnSurgical Center"}}

above query returns 4 studies and takes 0.2 seconds

So its definitely the "ModalitiesInStudy" parameter that's causing the issue (contrary to my previous post, I found that "ModalitiesInStudy" is a MainDicomTag at the Series level in Orthanc) 

Any help sorting this issue out would be greatly appreciated.

[Alain Mazy]

Hi Rana,

Would be nice to share your logs in verbose mode for the query that takes a lot of time.  

Then, you can try to configure this option to "Never" but I'm not sure how it will behave wrt DicomModalitiesInStudy:

  // Performance setting to specify how Orthanc accesses the storage
  // area during find operations (C-FIND, /tools/find API route and
  // QIDO-RS in dicom-web). Three modes are available: (1) "Always"
  // allows Orthanc to read the storage area as soon as it needs an
  // information that is not present in its database (slowest mode),
  // (2) "Never" prevents Orthanc from accessing the storage area, and
  // makes it uses exclusively its database (fastest mode), and (3)
  // "Answers" allows Orthanc to read the storage area to generate its
  // answers, but not to filter the DICOM resources (balance between
  // the two modes). By default, the mode is "Always", which
  // corresponds to the behavior of Orthanc <= 1.5.0.
  "StorageAccessOnFind" : "Always",

HTH,

Alain.

--
You received this message because you are subscribed to the Google Groups "Orthanc Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orthanc-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/orthanc-users/df141a0e-f255-4d0d-9d75-c17307ca1fb6n%40googlegroups.com.

[Rana Asim Wajid]

Hi Alain,

Setting "StorageAccessOnFind" to "Never" made no difference.

Below is the log you asked for along with a sample query:

{ "Expand": true, "Level": "Studies", "CaseSensitive": false, "Query": { "PatientName": "**" , "PatientID": "**" , "StudyID": "**" , "StudyDate": "20211116-20211117" , "InstitutionName": "Animal Hospital Surgical Center", "ModalitiesInStudy": "CT" }}


W1117 08:33:17.967003 main.cpp:1942] Orthanc version: 1.9.6
I1117 08:33:17.969922 main.cpp:1974] Architecture: 64-bit, little endian
W1117 08:33:17.972854 OrthancConfiguration.cpp:117] Scanning folder "C:\pacs\Configuration" for configuration files
W1117 08:33:17.978719 OrthancConfiguration.cpp:66] Reading the configuration from: "C:\pacs\Configuration\dicomweb.json"
W1117 08:33:17.984569 OrthancConfiguration.cpp:66] Reading the configuration from: "C:\pacs\Configuration\orthanc.json"
I1117 08:33:17.992388 Toolbox.cpp:1488] Setting up the ICU common data
I1117 08:33:18.193553 Toolbox.cpp:1467] Using locale: "" for case-insensitive comparison of strings
I1117 08:33:18.198438 Toolbox.cpp:1759] OpenSSL version: OpenSSL 1.1.1k 25 Mar 2021
I1117 08:33:18.203325 FromDcmtkBridge.cpp:291] (dicom) Using DCTMK version: 366
I1117 08:33:18.206250 FromDcmtkBridge.cpp:299] (dicom) Loading the embedded dictionaries
I1117 08:33:18.233589 FromDcmtkBridge.cpp:311] (dicom) Loading the embedded dictionary of private tags
I1117 08:33:18.252145 FromDcmtkBridge.cpp:2293] (dicom) Registering JPEG Lossless codecs in DCMTK
I1117 08:33:18.256058 FromDcmtkBridge.cpp:2301] (dicom) Registering JPEG codecs in DCMTK
I1117 08:33:18.259962 FromDcmtkBridge.cpp:2308] (dicom) Registering RLE codecs in DCMTK
I1117 08:33:18.262890 Enumerations.cpp:2315] Default encoding for DICOM was changed to: Latin1
W1117 08:33:18.276574 main.cpp:864] Loading plugin(s) from: C:/pacs/Plugins/
I1117 08:33:18.280474 PluginsManager.cpp:288] (plugins) Scanning folder C:/pacs/Plugins/ for plugins
I1117 08:33:18.284377 PluginsManager.cpp:311] (plugins) Found a shared library: "C:/pacs/Plugins\OrthancDicomWeb.dll"
W1117 08:33:18.291217 PluginsManager.cpp:269] Registering plugin 'dicom-web' (version 1.6)
I1117 08:33:18.297075 PluginsManager.cpp:172] (plugins) The DICOMweb plugin reads the DICOMweb servers from the configuration file
W1117 08:33:18.302935 PluginsManager.cpp:168] URI to the DICOMweb REST API: /dicom-web/
I1117 08:33:18.305865 OrthancPlugins.cpp:2319] (plugins) Plugin has registered a REST callback for chunked streams on: /dicom-web/studies
I1117 08:33:18.311724 OrthancPlugins.cpp:2319] (plugins) Plugin has registered a REST callback for chunked streams on: /dicom-web/studies/([^/]*)
I1117 08:33:18.318558 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/instances
I1117 08:33:18.325390 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/series
I1117 08:33:18.332232 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/instances
I1117 08:33:18.341020 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/metadata
I1117 08:33:18.348831 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/series
I1117 08:33:18.356646 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/series/([^/]*)
I1117 08:33:18.365443 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/series/([^/]*)/instances
I1117 08:33:18.375201 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/series/([^/]*)/instances/([^/]*)
I1117 08:33:18.384961 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/series/([^/]*)/instances/([^/]*)/bulk/(.*)
I1117 08:33:18.395706 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/series/([^/]*)/instances/([^/]*)/metadata
I1117 08:33:18.406451 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/series/([^/]*)/metadata
I1117 08:33:18.416214 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/series/([^/]*)/instances/([^/]*)/frames
I1117 08:33:18.426962 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/series/([^/]*)/instances/([^/]*)/frames/([^/]*)
I1117 08:33:18.437702 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/servers
I1117 08:33:18.444536 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/servers/([^/]*)
I1117 08:33:18.451373 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/servers/([^/]*)/stow
I1117 08:33:18.459188 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/servers/([^/]*)/wado
I1117 08:33:18.467006 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/servers/([^/]*)/get
I1117 08:33:18.474814 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/servers/([^/]*)/retrieve
I1117 08:33:18.482617 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/servers/([^/]*)/qido
I1117 08:33:18.490438 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/servers/([^/]*)/delete
I1117 08:33:18.498245 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/app/libs/(.*)
I1117 08:33:18.506058 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/info
I1117 08:33:18.511918 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/rendered
I1117 08:33:18.519730 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/series/([^/]*)/rendered
I1117 08:33:18.529494 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/series/([^/]*)/instances/([^/]*)/rendered
I1117 08:33:18.539256 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/studies/([^/]*)/series/([^/]*)/instances/([^/]*)/frames/([^/]*)/r
endered
I1117 08:33:18.550978 OrthancPlugins.cpp:2344] (plugins) Plugin has registered an OnChange callback
I1117 08:33:18.554886 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /dicom-web/app/client/(.*)
W1117 08:33:18.562699 PluginsManager.cpp:168] URI to the WADO-URI API: /wado
I1117 08:33:18.565627 OrthancPlugins.cpp:2302] (plugins) Plugin has registered a REST callback without mutual exclusion on: /wado
I1117 08:33:18.570516 PluginsManager.cpp:311] (plugins) Found a shared library: "C:/pacs/Plugins\OrthancGdcm.dll"
W1117 08:33:18.591022 PluginsManager.cpp:269] Registering plugin 'gdcm' (version 1.4)
W1117 08:33:18.595903 PluginsManager.cpp:168] Version of GDCM: 3.0.8
W1117 08:33:18.597853 PluginsManager.cpp:168] GDCM throttling is disabled
I1117 08:33:18.600782 OrthancPlugins.cpp:2417] (plugins) Plugin has registered a callback to decode DICOM images (1 decoder(s) now active)
I1117 08:33:18.606645 OrthancPlugins.cpp:2430] (plugins) Plugin has registered a callback to transcode DICOM images (1 transcoder(s) now active)
I1117 08:33:18.613480 PluginsManager.cpp:311] (plugins) Found a shared library: "C:/pacs/Plugins\OrthancPostgreSQLIndex.dll"
W1117 08:33:18.621302 PluginsManager.cpp:269] Registering plugin 'postgresql-index' (version 4.0)
W1117 08:33:18.627151 PluginsManager.cpp:168] The index plugin will use 1 connection(s) to the database, and will retry up to 10 time(s) in the case of a collision
I1117 08:33:18.634961 OrthancPlugins.cpp:5084] (plugins) Plugin has registered a custom database back-end
I1117 08:33:18.638867 OrthancPluginDatabaseV3.cpp:1071] (plugins) Identifier of this Orthanc server for the global properties of the custom database: "0e94d85b-dcf64572-b08f6263-d822493e-f6dec04f"
I1117 08:33:18.650592 PluginsManager.cpp:311] (plugins) Found a shared library: "C:/pacs/Plugins\OrthancPostgreSQLStorage.dll"
W1117 08:33:18.657432 PluginsManager.cpp:269] Registering plugin 'postgresql-storage' (version 4.0)
W1117 08:33:18.663284 PluginsManager.cpp:168] The PostgreSQL storage area is currently disabled, set "EnableStorage" to "true" in the "PostgreSQL" section of the configuration file of Orthanc
W1117 08:33:18.673049 main.cpp:1610] Using a custom database from plugins
W1117 08:33:18.675979 OrthancInitialization.cpp:457] Storage directory: "E:\PACSIMAGEDATA"
W1117 08:33:18.716018 HttpClient.cpp:1176] HTTPS will use the CA certificates from this file: C:\pacs\Configuration\ca-certificates.crt
I1117 08:33:18.722849 HttpClient.cpp:498] (http) Setting the default timeout for HTTP client connections: 10 seconds
I1117 08:33:18.727736 HttpClient.cpp:482] (http) Setting the default proxy for HTTP client connections:
I1117 08:33:18.732621 DicomAssociationParameters.cpp:375] (dicom) Default timeout for DICOM connections if Orthanc acts as SCU (client): 10 seconds (0 = no timeout)
I1117 08:33:18.740435 ServerIndex.cpp:438] Starting the monitor for stable resources (stable age = 300)
I1117 08:33:18.740435 LuaJobManager.cpp:79] (lua) Lua: DICOM associations will be closed after 5 seconds of inactivity
I1117 08:33:18.749219 LuaScripting.cpp:782] Initializing Lua for the event handler
W1117 08:33:18.753131 LuaContext.cpp:93] Lua says: Lua toolbox installed
I1117 08:33:18.755080 LuaJobManager.cpp:79] (lua) Lua: DICOM associations will be closed after 5 seconds of inactivity
I1117 08:33:18.759963 LuaScripting.cpp:782] Initializing Lua for the event handler
W1117 08:33:18.763872 LuaContext.cpp:93] Lua says: Lua toolbox installed
I1117 08:33:18.765818 ServerContext.cpp:373] Automated transcoding of incoming DICOM instances is disabled
I1117 08:33:18.770709 ServerContext.cpp:380] (dicom) Deidentification of log contents (notably for DIMSE queries) is enabled
I1117 08:33:18.775590 ServerContext.cpp:384] (dicom) Version of DICOM standard used for deidentification is 2021b
I1117 08:33:18.780477 ServerContext.cpp:403] (dicom) Preferred transfer syntax for Orthanc C-STORE SCU: 1.2.840.10008.1.2.1
I1117 08:33:18.786330 DcmtkTranscoder.cpp:75] Quality for lossy transcoding using DCMTK is set to: 90
W1117 08:33:18.790237 ServerContext.cpp:476] Disk compression is enabled
I1117 08:33:18.792187 ServerContext.cpp:1155] Storing MD5 for attachments: no
W1117 08:33:18.795118 ServerIndex.cpp:391] No limit on the number of stored patients
W1117 08:33:18.798049 ServerIndex.cpp:411] No limit on the size of the storage area
W1117 08:33:18.801958 main.cpp:1314] Orthanc Explorer UI is disabled
I1117 08:33:18.886913 DicomServer.cpp:132] (dicom) Setting timeout for DICOM connections if Orthanc acts as SCP (server): 60 seconds (0 = no timeout)
I1117 08:33:18.955274 DicomServer.cpp:420] (dicom) Orthanc SCP will *not* use DICOM TLS
W1117 08:33:18.958204 main.cpp:1249] DICOM server listening with AET EM-PACS on port: 4242
I1117 08:33:18.959183 DicomServer.cpp:63] (dicom) DICOM server started
I1117 08:33:18.962112 HttpServer.cpp:1579] (http) This Orthanc server uses CivetWeb as its embedded HTTP server
I1117 08:33:18.968950 HttpServer.cpp:2068] (http) The embedded HTTP server will use 50 threads
I1117 08:33:18.972855 HttpServer.cpp:1928] (http) HTTP keep alive is disabled
W1117 08:33:18.975780 HttpServer.cpp:1992] HTTP compression is enabled
I1117 08:33:18.977732 HttpServer.cpp:2081] (http) TCP_NODELAY for the HTTP sockets is set to true
I1117 08:33:18.981643 HttpServer.cpp:2101] (http) Request timeout in the HTTP server is set to 30 seconds
I1117 08:33:18.986525 main.cpp:1113] Version of Lua: Lua 5.3
W1117 08:33:18.988478 main.cpp:1124] Remote LUA script execution is disabled
I1117 08:33:18.990431 HttpServer.cpp:2148] (http) Branching WebDAV bucket at: /webdav
I1117 08:33:18.994339 HttpServer.cpp:1624] (http) Starting embedded Web server using Civetweb
I1117 08:33:18.999226 OrthancWebDav.cpp:1705] Starting the WebDAV upload thread
W1117 08:33:19.002154 HttpServer.cpp:1769] HTTP server listening on port: 8042 (HTTPS encryption is disabled, remote access is allowed)
W1117 08:33:19.008011 main.cpp:876] Orthanc has started
I1117 08:33:19.010948 LuaScripting.cpp:841] Starting the Lua engine
I1117 08:33:31.283410 HttpServer.cpp:1238] (http) POST /tools/find
I1117 08:35:15.291348 ServerContext.cpp:1310] Number of candidate resources after fast DB filtering on main DICOM tags: 2
I1117 08:35:15.298184 ServerContext.cpp:1435] Number of matching resources: 2

[Alain Mazy]

Thanks for the logs.  There's indeed room for improvement but, that's not an easy task at all so all I can do now is add it in our TODO.

So, if you can change the way you use the API, I would advise to look for studies without the "ModalitiesInStudy" criteria and then, iterate through all returned studies by yourself to filter the ones that meet your "ModalitiesInStudy=CT" criteria.

To view this discussion on the web visit https://groups.google.com/d/msgid/orthanc-users/66d20478-815e-4713-b016-08cac87679e9n%40googlegroups.com.

[Rana Asim Wajid]

Thanks for the reply. It seems ill have to change the search function like you suggested. Do you have any idea idea why only the "ModalitiesInStudy" field/parameter is causing this issue?

[Alain Mazy]

Yes, the ModalitiesInStudy is actually not a tag but is "computed"; therefore, it needs to access the "Modalities" tags of all series -> if this is not done at the right time, e.g, on the whole DB, it can take a lot of time !  

To view this discussion on the web visit https://groups.google.com/d/msgid/orthanc-users/fbd573d7-5629-4ea6-924b-acbe83a7a848n%40googlegroups.com.

[Rana Asim Wajid]

Oh I see...makes perfect sense now. Thanks again.