openrem_qr error causes orthanc to stop

[h4n ke]

Over the last couple of days I've encountered a handfull events that made Orthanc server crash. Then eventually openrem webpage crashed in a server error 500.

Digging trhough the openrem_qr log, and the orthanc log, it appears the crash happens during a move of files within the openrem_qr process. Below I have some contents of the log files. I think this results in a "loss" of Orthanc port 104, probably after orthanc crash.

Everything is resolved after a server reboot.

I'm starting the processes for different modalities sequentially through windows tasks scheduler.

Does anybody recognize this kind of problem?

Any ideas on the cause/fix?

thank you

W0517 11:36:45.283568 PluginsManager.cpp:211] Unregistering plugin 'wsi' (version 1.1)
E0517 11:36:45.311504 main.cpp:2075] Uncaught exception, stopping now: [The TCP port of the DICOM server is privileged or already in use] (code 2004)
W0517 11:36:45.311504 main.cpp:2106] Orthanc has stopped

[17/May/2023 11:36:23] INFO [remapp.netdicom.qrscu:2271] Requesting move: modality DX, study 143 (of 154) series 1 (of 2). Series contains 1 objects
[17/May/2023 11:39:29] ERROR [remapp.netdicom.qrscu:2101] Move of study 143, series 1: Warning (0xb000) - Sub-operations completed, one or more failures. Sub-ops completed: 0, failed: 1, warning: 0.
[17/May/2023 11:39:50] ERROR [remapp.netdicom.qrscu:2101] Move of study 143, series 1: Failure (0xc000) - Unable to Process. Sub-ops completed: 0, failed: 0, warning: 0.
[17/May/2023 11:39:50] WARNING [remapp.netdicom.qrscu:2178] Query_id 20716981-5a3f-48dc-aa81-10d021fe3204: Move failed twice in succession. Aborting move request

[Ed McDonagh]

Assuming that first log extract is from Orthanc, could you share what happens just before that?

It seems that maybe the move fails because the Orthanc server has crashed, and the Orthanc log extract starts just after the 
openrem_qr log had already thrown an error.

That shouldn't cause the Django or webserver to return a 500 though, so I'm wondering what else is going on? I wonder if WinSW is under stress? Did you edit the number of workers to be no more than the number of CPU cores available to your system? See https://docs.openrem.org/en/latest/install_windows.html#creating-worker-services

Kind regards

Ed

--
You received this message because you are subscribed to the Google Groups "OpenREM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrem+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/82059d0f-df72-4638-adde-c84941358a1dn%40googlegroups.com.

[h4n ke]

Dear Ed, thank you for looking together with me at this issue.

Below I copy/paste the orthanc log.

Regarding the WinSW: during setup i set the number of workers to spawn equal to 2.

2 is the number of cores powershell: Get-WmiObject –class Win32_processor | ft NumberOfCores,NumberOfLogicalProcessors

returns.

Not sure if this number is dynamic however, I'll check this with our it department.

kind regards, Han

below the log:

W0517 11:36:39.763501 main.cpp:2018] Orthanc version: 1.11.3
W0517 11:36:39.764495 OrthancConfiguration.cpp:108] Scanning folder "Configuration" for configuration files
W0517 11:36:39.764495 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\azure.json"
W0517 11:36:39.781500 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\delayed-deletion.json"
W0517 11:36:39.790607 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\dicomweb.json"
W0517 11:36:39.805496 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\housekeeper.json"
W0517 11:36:39.821494 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\indexer.json"
W0517 11:36:39.829494 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\mysql.json"
W0517 11:36:39.845495 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\odbc.json"
W0517 11:36:39.861499 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\orthanc-explorer-2.json"
W0517 11:36:39.873493 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\orthanc.json"
W0517 11:36:39.895498 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\postgresql.json"
W0517 11:36:39.911587 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\serve-folders.json"
W0517 11:36:39.924494 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\stone-webviewer.json"
W0517 11:36:39.939499 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\tcia.json"
W0517 11:36:39.952500 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\transfers.json"
W0517 11:36:39.974498 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\webviewer.json"
W0517 11:36:39.988500 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\worklists.json"
W0517 11:36:40.585514 main.cpp:911] Loading plugin(s) from: C:\Program Files\Orthanc Server\Plugins
W0517 11:36:40.703496 PluginsManager.cpp:261] Registering plugin 'connectivity-checks' (version 1.11.3)
W0517 11:36:40.835493 PluginsManager.cpp:261] Registering plugin 'delayed-deletion' (version 1.11.3)
W0517 11:36:40.838500 PluginsManager.cpp:157] DelayedDeletion - plugin is loaded but disabled (check your "DelayedDeletion.Enable" configuration)
W0517 11:36:40.939513 PluginsManager.cpp:261] Registering plugin 'housekeeper' (version 1.11.3)
W0517 11:36:40.939513 PluginsManager.cpp:157] Housekeeper plugin is initializing
W0517 11:36:40.941491 PluginsManager.cpp:157] Housekeeper plugin is disabled by the configuration file
W0517 11:36:41.049494 PluginsManager.cpp:261] Registering plugin 'stone-webviewer' (version 2.5)
W0517 11:36:41.151493 PluginsManager.cpp:261] Registering plugin 'worklists' (version 1.11.3)
W0517 11:36:41.152496 PluginsManager.cpp:157] Sample worklist plugin is initializing
W0517 11:36:41.153494 PluginsManager.cpp:157] Worklist server is disabled by the configuration file
W0517 11:36:41.257511 PluginsManager.cpp:261] Registering plugin 'authorization' (version 0.4.1)
W0517 11:36:41.258495 PluginsManager.cpp:157] Initializing the authorization plugin
W0517 11:36:41.258495 PluginsManager.cpp:157] No section "Authorization" in the configuration file, the authorization plugin is disabled
W0517 11:36:41.378495 PluginsManager.cpp:261] Registering plugin 'Azure Blob Storage' (version 2.1.2)
W0517 11:36:41.380714 PluginsManager.cpp:157] Azure Blob Storage plugin is initializing
W0517 11:36:41.381501 PluginsManager.cpp:157] Azure Blob Storage: no "AzureBlobStorage" section found in configuration, plugin is disabled
W0517 11:36:41.487494 PluginsManager.cpp:261] Registering plugin 'dicom-web' (version 1.13)
W0517 11:36:41.488502 PluginsManager.cpp:157] URI to the DICOMweb REST API: /dicom-web/
W0517 11:36:41.489611 PluginsManager.cpp:157] DICOMWeb PublicRoot: /dicom-web/
W0517 11:36:41.489611 PluginsManager.cpp:157] URI to the WADO-URI API: /wado
W0517 11:36:41.575494 PluginsManager.cpp:261] Registering plugin 'orthanc-explorer-2' (version 0.6.0)
W0517 11:36:41.576500 PluginsManager.cpp:157] Root URI to the Orthanc-Explorer 2 application: /ui/
W0517 11:36:41.697492 PluginsManager.cpp:261] Registering plugin 'gdcm' (version 1.5)
W0517 11:36:41.698494 PluginsManager.cpp:157] Version of GDCM: 3.0.10
W0517 11:36:41.698494 PluginsManager.cpp:157] GDCM throttling is disabled
W0517 11:36:41.812497 PluginsManager.cpp:261] Registering plugin 'indexer' (version 1.0)
W0517 11:36:41.814497 PluginsManager.cpp:157] OrthancIndexer is disabled
W0517 11:36:41.977495 PluginsManager.cpp:261] Registering plugin 'mysql-index' (version 4.3)
W0517 11:36:41.981497 PluginsManager.cpp:157] The MySQL index is currently disabled, set "EnableIndex" to "true" in the "MySQL" section of the configuration file of Orthanc
W0517 11:36:42.156493 PluginsManager.cpp:261] Registering plugin 'mysql-storage' (version 4.3)
W0517 11:36:42.160493 PluginsManager.cpp:157] The MySQL storage area is currently disabled, set "EnableStorage" to "true" in the "MySQL" section of the configuration file of Orthanc
W0517 11:36:42.409494 PluginsManager.cpp:261] Registering plugin 'neuro' (version 1.0)
W0517 11:36:42.673493 PluginsManager.cpp:261] Registering plugin 'odbc-index' (version 1.1)
W0517 11:36:42.675494 PluginsManager.cpp:157] The ODBC index is currently disabled, set "EnableIndex" to "true" in the "Odbc" section of the configuration file of Orthanc
W0517 11:36:42.828494 PluginsManager.cpp:261] Registering plugin 'odbc-storage' (version 1.1)
W0517 11:36:42.830494 PluginsManager.cpp:157] The ODBC storage area is currently disabled, set "EnableStorage" to "true" in the "Odbc" section of the configuration file of Orthanc
W0517 11:36:43.002938 PluginsManager.cpp:261] Registering plugin 'postgresql-index' (version 4.0)
W0517 11:36:43.005492 PluginsManager.cpp:157] The PostgreSQL index is currently disabled, set "EnableIndex" to "true" in the "PostgreSQL" section of the configuration file of Orthanc
W0517 11:36:43.123495 PluginsManager.cpp:261] Registering plugin 'postgresql-storage' (version 4.0)
W0517 11:36:43.125501 PluginsManager.cpp:157] The PostgreSQL storage area is currently disabled, set "EnableStorage" to "true" in the "PostgreSQL" section of the configuration file of Orthanc
W0517 11:36:43.263499 PluginsManager.cpp:261] Registering plugin 'tcia' (version 1.1)
W0517 11:36:43.366515 PluginsManager.cpp:261] Registering plugin 'transfers' (version 1.4)
W0517 11:36:43.491498 PluginsManager.cpp:261] Registering plugin 'web-viewer' (version 2.8)
W0517 11:36:43.492501 PluginsManager.cpp:157] Initializing the Web viewer
W0517 11:36:43.493501 PluginsManager.cpp:157] Web viewer using 2 threads for the decoding of the DICOM images
W0517 11:36:43.493501 PluginsManager.cpp:157] Storing the cache of the Web viewer in folder: E:\orthanc\storage\WebViewerCache
W0517 11:36:43.503494 PluginsManager.cpp:157] Web viewer using a cache of 100 MB
W0517 11:36:43.615492 PluginsManager.cpp:261] Registering plugin 'wsi' (version 1.1)
W0517 11:36:43.615492 PluginsManager.cpp:157] The whole-slide imaging plugin will use at most 4 threads to transcode the tiles
W0517 11:36:43.756493 PluginsManager.cpp:261] Registering plugin 'osimis-web-viewer' (version 1.4.2.0-9d9eff4)
W0517 11:36:43.756493 PluginsManager.cpp:157] Initializing the Web viewer
W0517 11:36:43.758493 PluginsManager.cpp:157] Using GDCM instead of the DICOM decoder that is built in Orthanc
W0517 11:36:43.858498 PluginsManager.cpp:261] Registering plugin 'serve-folders' (version 1.11.3)
W0517 11:36:43.860492 PluginsManager.cpp:157] ServeFolders: Empty configuration file: No additional folder will be served!
W0517 11:36:43.860492 OrthancInitialization.cpp:406] SQLite index directory: "E:\orthanc\storage"
W0517 11:36:43.865525 OrthancInitialization.cpp:505] Storage directory: "E:\orthanc\storage"
W0517 11:36:43.875496 HttpClient.cpp:1194] HTTPS will use the CA certificates from this file: C:\Program Files\Orthanc Server\Configuration\ca-certificates.crt
W0517 11:36:43.877495 LuaContext.cpp:94] Lua says: Lua toolbox installed
W0517 11:36:43.903496 LuaContext.cpp:94] Lua says: Lua toolbox installed
W0517 11:36:43.905570 ServerContext.cpp:505] Disk compression is disabled
W0517 11:36:43.905570 ServerIndex.cpp:381] No limit on the number of stored patients
W0517 11:36:43.905570 ServerIndex.cpp:401] No limit on the size of the storage area
W0517 11:36:43.905570 ServerIndex.cpp:420] Maximum Storage mode: Recycle
W0517 11:36:43.908497 ServerContext.cpp:242] Reloading the jobs from the last execution of Orthanc
W0517 11:36:44.012509 JobsEngine.cpp:272] The jobs engine has started with 2 threads
W0517 11:36:44.012509 main.cpp:1299] The DICOM port is privileged (104 is below 1024), make sure you run Orthanc as root/administrator
E0517 11:36:44.015503 OrthancException.cpp:61] The TCP port of the DICOM server is privileged or already in use:  (port = 104) cannot create network: TCP Initialization Error: Only one usage of each socket address (protocol/network address/port) is normally permitted.
E0517 11:36:44.019498 ServerContext.cpp:458] INTERNAL ERROR: ServerContext::Stop() should be invoked manually to avoid mess in the destruction order!
W0517 11:36:44.221494 JobsEngine.cpp:313] The jobs engine has stopped
W0517 11:36:44.399494 PluginsManager.cpp:211] Unregistering plugin 'Azure Blob Storage' (version 2.1.2)
W0517 11:36:44.399494 PluginsManager.cpp:157] Azure Blob Storage plugin is finalizing
W0517 11:36:44.406493 PluginsManager.cpp:211] Unregistering plugin 'authorization' (version 0.4.1)
W0517 11:36:44.406493 PluginsManager.cpp:211] Unregistering plugin 'connectivity-checks' (version 1.11.3)
W0517 11:36:44.406493 PluginsManager.cpp:211] Unregistering plugin 'delayed-deletion' (version 1.11.3)
W0517 11:36:44.407494 PluginsManager.cpp:211] Unregistering plugin 'dicom-web' (version 1.13)
W0517 11:36:44.408493 PluginsManager.cpp:211] Unregistering plugin 'gdcm' (version 1.5)
W0517 11:36:44.421493 PluginsManager.cpp:211] Unregistering plugin 'housekeeper' (version 1.11.3)
W0517 11:36:44.421493 PluginsManager.cpp:157] Housekeeper plugin is finalizing
W0517 11:36:44.422493 PluginsManager.cpp:211] Unregistering plugin 'indexer' (version 1.0)
W0517 11:36:44.422493 PluginsManager.cpp:157] Folder indexer plugin is finalizing
W0517 11:36:44.422493 PluginsManager.cpp:211] Unregistering plugin 'mysql-index' (version 4.3)
W0517 11:36:44.422493 PluginsManager.cpp:157] MySQL index is finalizing
W0517 11:36:44.434494 PluginsManager.cpp:211] Unregistering plugin 'mysql-storage' (version 4.3)
W0517 11:36:44.434494 PluginsManager.cpp:157] MySQL storage area is finalizing
W0517 11:36:44.436590 PluginsManager.cpp:211] Unregistering plugin 'neuro' (version 1.0)
W0517 11:36:44.437503 PluginsManager.cpp:211] Unregistering plugin 'odbc-index' (version 1.1)
W0517 11:36:44.437503 PluginsManager.cpp:157] ODBC index is finalizing
W0517 11:36:44.442499 PluginsManager.cpp:211] Unregistering plugin 'odbc-storage' (version 1.1)
W0517 11:36:44.442499 PluginsManager.cpp:157] ODBC storage area is finalizing
W0517 11:36:44.443519 PluginsManager.cpp:211] Unregistering plugin 'orthanc-explorer-2' (version 0.6.0)
W0517 11:36:44.444493 PluginsManager.cpp:211] Unregistering plugin 'osimis-web-viewer' (version 1.4.2.0-9d9eff4)
W0517 11:36:44.444493 PluginsManager.cpp:157] Finalizing the Web viewer
W0517 11:36:44.458505 PluginsManager.cpp:211] Unregistering plugin 'postgresql-index' (version 4.0)
W0517 11:36:44.458505 PluginsManager.cpp:157] PostgreSQL index is finalizing
W0517 11:36:44.462505 PluginsManager.cpp:211] Unregistering plugin 'postgresql-storage' (version 4.0)
W0517 11:36:44.462505 PluginsManager.cpp:157] PostgreSQL storage area is finalizing
W0517 11:36:44.469495 PluginsManager.cpp:211] Unregistering plugin 'serve-folders' (version 1.11.3)
W0517 11:36:44.469495 PluginsManager.cpp:211] Unregistering plugin 'stone-webviewer' (version 2.5)
W0517 11:36:44.476498 PluginsManager.cpp:211] Unregistering plugin 'tcia' (version 1.1)
W0517 11:36:44.476498 PluginsManager.cpp:157] TCIA plugin is finalizing
W0517 11:36:44.477490 PluginsManager.cpp:211] Unregistering plugin 'transfers' (version 1.4)
W0517 11:36:44.477490 PluginsManager.cpp:157] Transfers accelerator plugin is finalizing
W0517 11:36:44.479493 PluginsManager.cpp:211] Unregistering plugin 'web-viewer' (version 2.8)
W0517 11:36:44.479493 PluginsManager.cpp:157] Finalizing the Web viewer
W0517 11:36:45.283568 PluginsManager.cpp:211] Unregistering plugin 'worklists' (version 1.11.3)
W0517 11:36:45.283568 PluginsManager.cpp:157] Sample worklist plugin is finalizing

W0517 11:36:45.283568 PluginsManager.cpp:211] Unregistering plugin 'wsi' (version 1.1)
E0517 11:36:45.311504 main.cpp:2075] Uncaught exception, stopping now: [The TCP port of the DICOM server is privileged or already in use] (code 2004)
W0517 11:36:45.311504 main.cpp:2106] Orthanc has stopped

Op vrijdag 19 mei 2023 om 11:38:11 UTC+2 schreef Ed McDonagh:

[Ed McDonagh]

Hi

I think in that log, the Orthanc server never gets started - it gets as far as claiming the port (104), realises it can't, then aborts.

I am not familiar with the permissions for that on Windows, but I presume you have had this working so it can't be that you are needing administrator privileges - could it be that it was still running (or had crashed without 'releasing' the port)?

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/083aadde-00b5-4080-94c8-24158287870bn%40googlegroups.com.

[h4n ke]

I think the permissions are ok, since it doet does work normally, but has crashed a couple of times. Maybe a release-the-port after crash indeed.

I guess it is important to run the openrem_qr scripts sequentially for different modalities, i.e. start the query on DX only after the query of CT is completely finished?

below i paste the orthanc log file previous to the one i shared before (note the date/timing wrt the log i shared earlier today).

this is a log that starts after a server reboot.

Han

W0517 10:46:11.608302 main.cpp:2018] Orthanc version: 1.11.3
W0517 10:46:11.608302 OrthancConfiguration.cpp:108] Scanning folder "Configuration" for configuration files
W0517 10:46:11.623926 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\azure.json"
W0517 10:46:11.639551 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\delayed-deletion.json"
W0517 10:46:11.639551 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\dicomweb.json"
W0517 10:46:11.639551 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\housekeeper.json"
W0517 10:46:11.639551 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\indexer.json"
W0517 10:46:11.639551 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\mysql.json"
W0517 10:46:11.639551 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\odbc.json"
W0517 10:46:11.639551 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\orthanc-explorer-2.json"
W0517 10:46:11.639551 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\orthanc.json"
W0517 10:46:11.655176 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\postgresql.json"
W0517 10:46:11.655176 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\serve-folders.json"
W0517 10:46:11.655176 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\stone-webviewer.json"
W0517 10:46:11.655176 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\tcia.json"
W0517 10:46:11.655176 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\transfers.json"
W0517 10:46:11.655176 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\webviewer.json"
W0517 10:46:11.655176 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\worklists.json"
W0517 10:46:12.967676 main.cpp:911] Loading plugin(s) from: C:\Program Files\Orthanc Server\Plugins
W0517 10:46:12.998927 PluginsManager.cpp:261] Registering plugin 'connectivity-checks' (version 1.11.3)
W0517 10:46:13.045819 PluginsManager.cpp:261] Registering plugin 'delayed-deletion' (version 1.11.3)
W0517 10:46:13.045819 PluginsManager.cpp:157] DelayedDeletion - plugin is loaded but disabled (check your "DelayedDeletion.Enable" configuration)
W0517 10:46:13.077051 PluginsManager.cpp:261] Registering plugin 'housekeeper' (version 1.11.3)
W0517 10:46:13.077051 PluginsManager.cpp:157] Housekeeper plugin is initializing
W0517 10:46:13.077051 PluginsManager.cpp:157] Housekeeper plugin is disabled by the configuration file
W0517 10:46:13.108302 PluginsManager.cpp:261] Registering plugin 'stone-webviewer' (version 2.5)
W0517 10:46:13.139552 PluginsManager.cpp:261] Registering plugin 'worklists' (version 1.11.3)
W0517 10:46:13.139552 PluginsManager.cpp:157] Sample worklist plugin is initializing
W0517 10:46:13.139552 PluginsManager.cpp:157] Worklist server is disabled by the configuration file
W0517 10:46:13.233325 PluginsManager.cpp:261] Registering plugin 'authorization' (version 0.4.1)
W0517 10:46:13.233325 PluginsManager.cpp:157] Initializing the authorization plugin
W0517 10:46:13.233325 PluginsManager.cpp:157] No section "Authorization" in the configuration file, the authorization plugin is disabled
W0517 10:46:13.311428 PluginsManager.cpp:261] Registering plugin 'Azure Blob Storage' (version 2.1.2)
W0517 10:46:13.311428 PluginsManager.cpp:157] Azure Blob Storage plugin is initializing
W0517 10:46:13.311428 PluginsManager.cpp:157] Azure Blob Storage: no "AzureBlobStorage" section found in configuration, plugin is disabled
W0517 10:46:13.358337 PluginsManager.cpp:261] Registering plugin 'dicom-web' (version 1.13)
W0517 10:46:13.373931 PluginsManager.cpp:157] URI to the DICOMweb REST API: /dicom-web/
W0517 10:46:13.373931 PluginsManager.cpp:157] DICOMWeb PublicRoot: /dicom-web/
W0517 10:46:13.373931 PluginsManager.cpp:157] URI to the WADO-URI API: /wado
W0517 10:46:13.420812 PluginsManager.cpp:261] Registering plugin 'orthanc-explorer-2' (version 0.6.0)
W0517 10:46:13.420812 PluginsManager.cpp:157] Root URI to the Orthanc-Explorer 2 application: /ui/
W0517 10:46:13.702051 PluginsManager.cpp:261] Registering plugin 'gdcm' (version 1.5)
W0517 10:46:13.702051 PluginsManager.cpp:157] Version of GDCM: 3.0.10
W0517 10:46:13.702051 PluginsManager.cpp:157] GDCM throttling is disabled
W0517 10:46:13.733302 PluginsManager.cpp:261] Registering plugin 'indexer' (version 1.0)
W0517 10:46:13.748925 PluginsManager.cpp:157] OrthancIndexer is disabled
W0517 10:46:13.780177 PluginsManager.cpp:261] Registering plugin 'mysql-index' (version 4.3)
W0517 10:46:13.858303 PluginsManager.cpp:157] The MySQL index is currently disabled, set "EnableIndex" to "true" in the "MySQL" section of the configuration file of Orthanc
W0517 10:46:13.889552 PluginsManager.cpp:261] Registering plugin 'mysql-storage' (version 4.3)
W0517 10:46:14.014562 PluginsManager.cpp:157] The MySQL storage area is currently disabled, set "EnableStorage" to "true" in the "MySQL" section of the configuration file of Orthanc
W0517 10:46:14.061425 PluginsManager.cpp:261] Registering plugin 'neuro' (version 1.0)
W0517 10:46:14.108302 PluginsManager.cpp:261] Registering plugin 'odbc-index' (version 1.1)
W0517 10:46:14.108302 PluginsManager.cpp:157] The ODBC index is currently disabled, set "EnableIndex" to "true" in the "Odbc" section of the configuration file of Orthanc
W0517 10:46:14.123937 PluginsManager.cpp:261] Registering plugin 'odbc-storage' (version 1.1)
W0517 10:46:14.139550 PluginsManager.cpp:157] The ODBC storage area is currently disabled, set "EnableStorage" to "true" in the "Odbc" section of the configuration file of Orthanc
W0517 10:46:14.170801 PluginsManager.cpp:261] Registering plugin 'postgresql-index' (version 4.0)
W0517 10:46:14.170801 PluginsManager.cpp:157] The PostgreSQL index is currently disabled, set "EnableIndex" to "true" in the "PostgreSQL" section of the configuration file of Orthanc
W0517 10:46:14.217675 PluginsManager.cpp:261] Registering plugin 'postgresql-storage' (version 4.0)
W0517 10:46:14.217675 PluginsManager.cpp:157] The PostgreSQL storage area is currently disabled, set "EnableStorage" to "true" in the "PostgreSQL" section of the configuration file of Orthanc
W0517 10:46:14.248932 PluginsManager.cpp:261] Registering plugin 'tcia' (version 1.1)
W0517 10:46:14.264548 PluginsManager.cpp:261] Registering plugin 'transfers' (version 1.4)
W0517 10:46:14.327052 PluginsManager.cpp:261] Registering plugin 'web-viewer' (version 2.8)
W0517 10:46:14.327052 PluginsManager.cpp:157] Initializing the Web viewer
W0517 10:46:14.327052 PluginsManager.cpp:157] Web viewer using 2 threads for the decoding of the DICOM images
W0517 10:46:14.327052 PluginsManager.cpp:157] Storing the cache of the Web viewer in folder: E:\orthanc\storage\WebViewerCache
W0517 10:46:14.358299 PluginsManager.cpp:157] Web viewer using a cache of 100 MB
W0517 10:46:14.405194 PluginsManager.cpp:261] Registering plugin 'wsi' (version 1.1)
W0517 10:46:14.405194 PluginsManager.cpp:157] The whole-slide imaging plugin will use at most 4 threads to transcode the tiles
W0517 10:46:14.592692 PluginsManager.cpp:261] Registering plugin 'osimis-web-viewer' (version 1.4.2.0-9d9eff4)
W0517 10:46:14.592692 PluginsManager.cpp:157] Initializing the Web viewer
W0517 10:46:14.592692 PluginsManager.cpp:157] Using GDCM instead of the DICOM decoder that is built in Orthanc
W0517 10:46:14.623926 PluginsManager.cpp:261] Registering plugin 'serve-folders' (version 1.11.3)
W0517 10:46:14.623926 PluginsManager.cpp:157] ServeFolders: Empty configuration file: No additional folder will be served!
W0517 10:46:14.623926 OrthancInitialization.cpp:406] SQLite index directory: "E:\orthanc\storage"
W0517 10:46:14.655184 OrthancInitialization.cpp:505] Storage directory: "E:\orthanc\storage"
W0517 10:46:14.655184 HttpClient.cpp:1194] HTTPS will use the CA certificates from this file: C:\Program Files\Orthanc Server\Configuration\ca-certificates.crt
W0517 10:46:14.670802 LuaContext.cpp:94] Lua says: Lua toolbox installed
W0517 10:46:14.670802 LuaContext.cpp:94] Lua says: Lua toolbox installed
W0517 10:46:14.686431 ServerContext.cpp:505] Disk compression is disabled
W0517 10:46:14.686431 ServerIndex.cpp:381] No limit on the number of stored patients
W0517 10:46:14.686431 ServerIndex.cpp:401] No limit on the size of the storage area
W0517 10:46:14.686431 ServerIndex.cpp:420] Maximum Storage mode: Recycle
W0517 10:46:14.686431 ServerContext.cpp:242] Reloading the jobs from the last execution of Orthanc
W0517 10:46:14.842676 JobsEngine.cpp:272] The jobs engine has started with 2 threads
W0517 10:46:14.842676 main.cpp:1299] The DICOM port is privileged (104 is below 1024), make sure you run Orthanc as root/administrator
W0517 10:46:14.842676 main.cpp:1305] DICOM server listening with AET ORTHANC on port: 104
W0517 10:46:14.842676 HttpServer.cpp:2036] HTTP compression is enabled
W0517 10:46:14.842676 main.cpp:1058] Remote access is allowed but "AuthenticationEnabled" is not in the configuration, automatically enabling HTTP authentication for security
W0517 10:46:14.842676 main.cpp:1172] Remote LUA script execution is disabled
W0517 10:46:14.858303 HttpServer.cpp:1794] HTTP server listening on port: 8042 (HTTPS encryption is disabled, remote access is allowed)
W0517 10:46:14.858303 main.cpp:923] Orthanc has started
E0517 11:36:24.060210 LuaScripting.cpp:455] Lua: Inexistent file
E0517 11:36:24.060210 LuaScripting.cpp:458] Lua: Error in RestApiGet() for URI: /instances/9fc9e1c9-6e85163f-6e97c6dd-de37a831-02cd6f86/file
E0517 11:36:24.063210 OrthancException.cpp:61] Cannot execute a Lua command: [string "line"]:268: bad argument #1 to 'write' (string expected, got nil)
E0517 11:36:24.065211 LuaScripting.cpp:848] Error while processing Lua events: Cannot execute a Lua command
E0517 11:36:24.078208 LuaScripting.cpp:455] Lua: Inexistent file
E0517 11:36:24.078208 LuaScripting.cpp:458] Lua: Error in RestApiGet() for URI: /instances/1445582d-33accd71-16f3832c-82826d36-89b32db2/file
E0517 11:36:24.078208 OrthancException.cpp:61] Cannot execute a Lua command: [string "line"]:268: bad argument #1 to 'write' (string expected, got nil)
E0517 11:36:24.078208 LuaScripting.cpp:848] Error while processing Lua events: Cannot execute a Lua command
E0517 11:36:24.089211 LuaScripting.cpp:455] Lua: Inexistent file
E0517 11:36:24.089211 LuaScripting.cpp:458] Lua: Error in RestApiGet() for URI: /instances/a6c8ef60-2e8d3d35-8419c091-5cd45bb7-8bb9ca6c/file
E0517 11:36:24.089211 OrthancException.cpp:61] Cannot execute a Lua command: [string "line"]:268: bad argument #1 to 'write' (string expected, got nil)
E0517 11:36:24.090210 LuaScripting.cpp:848] Error while processing Lua events: Cannot execute a Lua command
E0517 11:36:24.099209 LuaScripting.cpp:455] Lua: Inexistent file
E0517 11:36:24.099209 LuaScripting.cpp:458] Lua: Error in RestApiGet() for URI: /instances/f70c5a03-7990a56c-189741f9-88d03252-60af2134/file
E0517 11:36:24.099209 OrthancException.cpp:61] Cannot execute a Lua command: [string "line"]:268: bad argument #1 to 'write' (string expected, got nil)
E0517 11:36:24.099209 LuaScripting.cpp:848] Error while processing Lua events: Cannot execute a Lua command
E0517 11:36:24.107211 LuaScripting.cpp:455] Lua: Inexistent file
E0517 11:36:24.107211 LuaScripting.cpp:458] Lua: Error in RestApiGet() for URI: /instances/01910fbd-8ba1c17b-b7efee49-98b7ebf8-a20612e5/file
E0517 11:36:24.107211 OrthancException.cpp:61] Cannot execute a Lua command: [string "line"]:268: bad argument #1 to 'write' (string expected, got nil)
E0517 11:36:24.107211 LuaScripting.cpp:848] Error while processing Lua events: Cannot execute a Lua command
E0517 11:36:24.117209 LuaScripting.cpp:455] Lua: Inexistent file
E0517 11:36:24.117209 LuaScripting.cpp:458] Lua: Error in RestApiGet() for URI: /instances/db15e2ae-14e595ab-06b4730f-deef98f9-468a6d23/file
E0517 11:36:24.117209 OrthancException.cpp:61] Cannot execute a Lua command: [string "line"]:268: bad argument #1 to 'write' (string expected, got nil)
E0517 11:36:24.117209 LuaScripting.cpp:848] Error while processing Lua events: Cannot execute a Lua command

Op vrijdag 19 mei 2023 om 13:03:41 UTC+2 schreef Ed McDonagh:

[dpla...@gmail.com]

Dear Han,

For our OpenREM server I have scheduled PACS queries that run throughout the day. They are spaced out so that they hopefully don't overlap. For example:

00:15 query for DX from previous day

02:00 query for CT from previous day

07:00 query for DX from today with study time 00:00 to 07:00

12:00 query for DX from today with study time 06:30 to 12:00

etc.

Maybe trying to carry out multiple queries at the same time is causing an issue with access to the Orthanc internal database or log file?

Kind regards,

David

[Ed McDonagh]

I see in the new log file that Orthanc is having trouble with the lua script, on this line where it is writing the file temporarily before importing into OpenREM. As it has been working for you, I wonder if there is a particular study that you are picking up that has missing fields or something strange that causes the lua file to not function?

Can you stop all the scheduled queries, then try querying smaller time-frames with single modality types to see if some work and some don't - see if you can isolate where the issue is?

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/2ef7a8af-9d14-4d55-8a44-3d9034363dd4n%40googlegroups.com.

[h4n ke]

Thank you Ed, this makes sense. I'll try to set the queries up in smaller timeframes, by which I'll be able to more easily isolate erroneous studies. 

regards

Han

Op vrijdag 19 mei 2023 om 14:10:27 UTC+2 schreef Ed McDonagh:

[h4n ke]

thank you David, this also makes sense. I'll experiment with this.

regards

Op vrijdag 19 mei 2023 om 13:39:58 UTC+2 schreef dpla...@gmail.com:

[Ed McDonagh]

If you can isolate it, please let us know in case we can improve the script to deal with it gracefully.

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/947b5f2c-1976-4212-8467-b91604c1e15dn%40googlegroups.com.

[h4n ke]

in the last couple of weeks I've been experimenting a little bit. I have the impression I was dealing with a few different error causes but am not sure.

There are a few thing I'm pretty sure about:

- error occurs/starts during Requesting move: modality DX, study xx (of xxx) ... proces in openrem_qr log

- task_admin page processes are "running", but freeze in the interface and I have to terminate task for new jobs to start

- at some times during or previous to the job that started requesting move process orthanc log has memory errors:

E0607 10:38:19.562351 RunnableWorkersPool.cpp:69] Not enough memory to handle some runnable object
E0607 10:52:59.446396 RunnableWorkersPool.cpp:69] Not enough memory to handle some runnable object

- in windows resource monitor it is absolutely clear that plenty of memory is available. The server has some adaptable memory setting that grows by demand. I do see that memory usage increases pretty much when the query starts

- then if I do nothing for about 10 minutes the server error (500) appears and the server is down. 

now I think this particular error is not caused by lua problem, because if i repeat the query after the reboot at some point it just succeeds and al files are parsed.

- also earlier today I've seen about 200+ python.exe processes in my resource monitor list. Not sure if this should be the case. Perhaps python processes are not well closed after an error of some kind.

My main question is: how to solve the Orthanc error  "Not enough memory to handle some runnable object", and any advise is welcome.

when I terminate query and import_dx tasks in task_admin

Op vrijdag 19 mei 2023 om 14:58:08 UTC+2 schreef Ed McDonagh:

[Ed McDonagh]

Can you try two things please? 

1. Start Orthanc in verbose logging mode: https://book.orthanc-server.com/faq/log.html#under-windows to see if there is additional information available?

2. Is it possible to ask for the adaptable memory to be fixed while we troubleshoot the issue?

Kind regards

Ed

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/1e180021-9a57-4bec-b547-7c144695b6cbn%40googlegroups.com.

[h4n ke]

dear Ed,

thank you very much for your assistance. This really helps me with getting OpenREM operational.

I have tried the two things,

on 1: the guide does not let me start orthanc in a verbose logging mode that writes to a log file. somehow the script needs some tweaking. I managed to start it in verbose mode, writing output to the terminal screen.

on 2: ICT department fixed memory on 16Gb and in the more or less 48 hours after that I've ben querying two times an hour on all modalities without any errors. I'm very happy with that.

for now I conclude that adaptable memory was not compatible with my installation on Windows Server and that this was fixed by fixing the memory.

You remember I was in the impression that I'm dealing with a few different type of errors.

Now today I'm querying on DX, for all studies on a given day. Say there are 130 studies.

Then in about 50% of the cases (days) the import_dx goes fine, until somewhere halfway during the "move" process, where I get the following error:

[09/Jun/2023 15:25:31] ERROR [remapp.netdicom.qrscu:2101] Move of study 106, series 2: Warning (0xb000) - Sub-operations completed, one or more failures. Sub-ops completed: 0, failed: 1, warning: 0.

there is no orthanc log/error.

Because I send the command through the terminal I get the traceback  in the terminal;

Traceback (most recent call last):
  File "E:\venv\Lib\site-packages\openrem\scripts\openrem_qr.py", line 12, in <module>
    qrscu_script()
  File "E:\venv\lib\site-packages\openrem\remapp\netdicom\qrscu.py", line 2646, in qrscu_script
    wait_task(b)
  File "E:\venv\lib\site-packages\openrem\remapp\tools\background.py", line 245, in wait_task
    return task(blocking=True)
  File "E:\venv\lib\site-packages\huey\api.py", line 932, in __call__
    return self.get(*args, **kwargs)
  File "E:\venv\lib\site-packages\huey\api.py", line 974, in get
    raise TaskException(result.metadata)
huey.exceptions.TaskException: InterfaceError('connection already closed')

which seems to be related to a port issue?

after this error the openrem log keeps looping;

every 2 or 3 seconds: [09/Jun/2023 15:26:41] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC

now this loop stops with a fresh terminal query, and after that I remove the error task from the system through the web interface.

then with a lower frequency I get a similar error;

[09/Jun/2023 15:52:46] ERROR [remapp.netdicom.qrscu:2101] Move of study 79, series 3: Warning (0xb000) - Sub-operations completed, one or more failures. Sub-ops completed: 0, failed: 1, warning: 0.

with no traceback in the command prompt.

however with an orthanc error log
E0609 15:52:52.749825 OrthancException.cpp:61] The TCP port of the DICOM server is privileged or already in use:  (port = 104) cannot create network: TCP Initialization Error: Only one usage of each socket address (protocol/network address/port) is normally permitted.
E0609 15:52:59.155926 main.cpp:2075] Uncaught exception, stopping now: [The TCP port of the DICOM server is privileged or already in use] (code 2004)
new query:
[09/Jun/2023 15:57:59] ERROR [remapp.netdicom.qrscu:2588] Query-retrieve aborted: DICOM nodes not ready. QR SCP echo is Success, Store SCP echo is Association aborted or never connected
W0609 15:52:59.155926 main.cpp:2106] Orthanc has stopped

then on the web interface:

Orthanc PACS
Error: Server is down - see status

and after windows server reboot all is fine again.

Now somehow this does not happen (as far as i understand now) with the short queries of 1 hour or such. And then only when querying on DX for a whole day, which is usually a large number of files and hence a relatively long process (couple of minutes).

It would be nice for our hospital to be able to query all studies from a past week or couple of days without error, hence I'm interested in tweaking settings or such thing to prevent above errors.

best regards

Han

Op woensdag 7 juni 2023 om 12:03:02 UTC+2 schreef Ed McDonagh:

[Ed McDonagh]

Great that the memory issue appears to be resolved.

Do you have your QR logs running at DEBUG level? Looking through the docs, I can see that I need to update this element as I can't find a suitable page to refer you to!

Find this section in your E:\venv\Lib\site-packages\openrem\openremproject\local_settings.py file:

# Set the log level. Options are 'DEBUG', 'INFO', 'WARNING', 'ERROR', and 'CRITICAL', with progressively less logging.

LOGGING['loggers']['remapp']['level'] = 'INFO' # General logs

LOGGING['loggers']['remapp.netdicom.qrscu']['level'] = 'INFO' # Query Retrieve SCU logs

LOGGING['loggers']['remapp.extractors']['level'] = 'INFO' # Extractor logs

and change the remap.netdicom.qrscu logging to DEBUG, and restart everything. This might give us some more clues as to why the move is aborted/fails.

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/d2b311a7-cf12-4c76-bd0a-0d7495c38170n%40googlegroups.com.

[h4n ke]

dear Ed, I really appreciatie your effort and help.

I switched on DEBUG in local_settings.

Now the log file has more detailed information;

[13/Jun/2023 17:25:38] INFO [remapp.netdicom.qrscu:2271] Requesting move: modality DX, study 118 (of 157) series 2 (of 2). Series contains 1 objects
[13/Jun/2023 17:25:38] DEBUG [remapp.netdicom.qrscu:2282] _move_req launched
[13/Jun/2023 17:25:38] DEBUG [remapp.netdicom.qrscu:2297] Series-level move - d is: (0008, 0052) Query/Retrieve Level                CS: 'SERIES'
(0020, 000d) Study Instance UID                  UI: #####deleted by me
(0020, 000e) Series Instance UID                 UI: #####deleted by me

then it stops/hangs, and I get the following traceback on the powershell:

>> H:\>
PS H:\> python 'E:\venv\Lib\site-packages\openrem\scripts\openrem_qr.py' 1 1 -dx -sd 2023-04-12 -e "import"
>>
Running Query

Traceback (most recent call last):
  File "E:\venv\Lib\site-packages\openrem\scripts\openrem_qr.py", line 12, in <module>
    qrscu_script()
  File "E:\venv\lib\site-packages\openrem\remapp\netdicom\qrscu.py", line 2646, in qrscu_script
    wait_task(b)
  File "E:\venv\lib\site-packages\openrem\remapp\tools\background.py", line 245, in wait_task
    return task(blocking=True)
  File "E:\venv\lib\site-packages\huey\api.py", line 932, in __call__
    return self.get(*args, **kwargs)
  File "E:\venv\lib\site-packages\huey\api.py", line 974, in get
    raise TaskException(result.metadata)
huey.exceptions.TaskException: InterfaceError('connection already closed')

after clearing the tasks from the openrem taks page I can start a new job, small jobs give no errors.

Op vrijdag 9 juni 2023 om 17:23:07 UTC+2 schreef Ed McDonagh:

[Ed McDonagh]

If you run the same QR again, does it fall over at the same study/series or not? I'm guessing not?

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/23126a21-f306-487a-a277-e4ebd4fbf6f2n%40googlegroups.com.

[h4n ke]

Dear Ed, David,

i appreciate your help and advice.

I'm still dealing with some problems on my windows installation.  I'm trying to narrow it down, but do not understand it yet.

I think I have enough memory (32Gb) and normal queries per modality of a 2 hour production in my small hospital do run without problem, memory maxes during an import session at 10Gb or something like that, sufficiently below 32Gb.

Now I copy some parts of my openrem_qr.log:

[19/Oct/2023 11:31:13] INFO [remapp.netdicom.qrscu:2271] Requesting move: modality MG, study 3 (of 3) series 10 (of 10).
[19/Oct/2023 11:31:13] DEBUG [remapp.netdicom.qrscu:2282] _move_req launched
[19/Oct/2023 11:31:13] DEBUG [remapp.netdicom.qrscu:2297] Series-level move - d is: (0008, 0052) Query/Retrieve Level                CS: 'SERIES'
(0020, 000d) Study Instance UID                  UI: 1.2.840.113845.11.1000000001865884969.20231016155405.3508747
(0020, 000e) Series Instance UID                 UI: 1.2.840.113845.11.2000000001865884969.20231019110017.2498568
[19/Oct/2023 11:31:14] INFO [remapp.netdicom.qrscu:2305] Query_id 742fc685-0df9-466c-add8-6ea1bdbc7fd8: Move association released
[19/Oct/2023 11:31:14] DEBUG [remapp.netdicom.qrscu:2319] Move complete. 3 studies.  Cumulative sub-ops completed: 342, failed: 0, warning: 0.
[19/Oct/2023 11:31:14] DEBUG [remapp.netdicom.qrscu:2321] Query_id 742fc685-0df9-466c-add8-6ea1bdbc7fd8: Releasing move association
[19/Oct/2023 11:31:14] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:17] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:20] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:23] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:26] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:29] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:32] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:35] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:38] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:41] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:44] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:47] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:50] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:53] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:56] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:31:59] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:32:02] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC
[19/Oct/2023 11:32:35] INFO [remapp.netdicom.qrscu:126] Association aborted or never connected to localhost 104 ORTHANC

it is between 11:32:02 and 11:32.35 that something happens which eventually makes Orthanc PACS crash due to unavailable port.

When Orthanc crashes, I find specific orthanc logfiles. Below I copy the content of the orthanc crash (?) logfile:

W1019 11:32:05.724997 main.cpp:2018] Orthanc version: 1.11.3
W1019 11:32:05.724997 OrthancConfiguration.cpp:108] Scanning folder "Configuration" for configuration files
W1019 11:32:05.724997 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\azure.json"
W1019 11:32:05.756246 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\delayed-deletion.json"
W1019 11:32:05.774723 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\dicomweb.json"
W1019 11:32:05.790671 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\housekeeper.json"
W1019 11:32:05.818745 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\indexer.json"
W1019 11:32:05.834371 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\mysql.json"
W1019 11:32:05.850022 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\odbc.json"
W1019 11:32:05.865621 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\orthanc-explorer-2.json"
W1019 11:32:05.881248 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\orthanc.json"
W1019 11:32:05.896870 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\postgresql.json"
W1019 11:32:05.912497 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\serve-folders.json"
W1019 11:32:05.928124 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\stone-webviewer.json"
W1019 11:32:05.943749 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\tcia.json"
W1019 11:32:05.959373 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\transfers.json"
W1019 11:32:05.974998 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\webviewer.json"
W1019 11:32:05.990622 OrthancConfiguration.cpp:57] Reading the configuration from: "Configuration\worklists.json"
W1019 11:32:06.475008 main.cpp:911] Loading plugin(s) from: C:\Program Files\Orthanc Server\Plugins
W1019 11:32:06.568760 PluginsManager.cpp:261] Registering plugin 'connectivity-checks' (version 1.11.3)
W1019 11:32:06.709385 PluginsManager.cpp:261] Registering plugin 'delayed-deletion' (version 1.11.3)
W1019 11:32:06.709385 PluginsManager.cpp:157] DelayedDeletion - plugin is loaded but disabled (check your "DelayedDeletion.Enable" configuration)
W1019 11:32:06.787513 PluginsManager.cpp:261] Registering plugin 'housekeeper' (version 1.11.3)
W1019 11:32:06.787513 PluginsManager.cpp:157] Housekeeper plugin is initializing
W1019 11:32:06.787513 PluginsManager.cpp:157] Housekeeper plugin is disabled by the configuration file
W1019 11:32:06.896893 PluginsManager.cpp:261] Registering plugin 'stone-webviewer' (version 2.5)
W1019 11:32:06.959392 PluginsManager.cpp:261] Registering plugin 'worklists' (version 1.11.3)
W1019 11:32:06.959392 PluginsManager.cpp:157] Sample worklist plugin is initializing
W1019 11:32:06.975016 PluginsManager.cpp:157] Worklist server is disabled by the configuration file
W1019 11:32:07.053142 PluginsManager.cpp:261] Registering plugin 'authorization' (version 0.4.1)
W1019 11:32:07.053142 PluginsManager.cpp:157] Initializing the authorization plugin
W1019 11:32:07.053142 PluginsManager.cpp:157] No section "Authorization" in the configuration file, the authorization plugin is disabled
W1019 11:32:07.146895 PluginsManager.cpp:261] Registering plugin 'Azure Blob Storage' (version 2.1.2)
W1019 11:32:07.146895 PluginsManager.cpp:157] Azure Blob Storage plugin is initializing
W1019 11:32:07.146895 PluginsManager.cpp:157] Azure Blob Storage: no "AzureBlobStorage" section found in configuration, plugin is disabled
W1019 11:32:07.240645 PluginsManager.cpp:261] Registering plugin 'dicom-web' (version 1.13)
W1019 11:32:07.240645 PluginsManager.cpp:157] URI to the DICOMweb REST API: /dicom-web/
W1019 11:32:07.240645 PluginsManager.cpp:157] DICOMWeb PublicRoot: /dicom-web/
W1019 11:32:07.240645 PluginsManager.cpp:157] URI to the WADO-URI API: /wado
W1019 11:32:07.334394 PluginsManager.cpp:261] Registering plugin 'orthanc-explorer-2' (version 0.6.0)
W1019 11:32:07.334394 PluginsManager.cpp:157] Root URI to the Orthanc-Explorer 2 application: /ui/
W1019 11:32:07.443773 PluginsManager.cpp:261] Registering plugin 'gdcm' (version 1.5)
W1019 11:32:07.443773 PluginsManager.cpp:157] Version of GDCM: 3.0.10
W1019 11:32:07.443773 PluginsManager.cpp:157] GDCM throttling is disabled
W1019 11:32:07.553150 PluginsManager.cpp:261] Registering plugin 'indexer' (version 1.0)
W1019 11:32:07.553150 PluginsManager.cpp:157] OrthancIndexer is disabled
W1019 11:32:07.725029 PluginsManager.cpp:261] Registering plugin 'mysql-index' (version 4.3)
W1019 11:32:07.725029 PluginsManager.cpp:157] The MySQL index is currently disabled, set "EnableIndex" to "true" in the "MySQL" section of the configuration file of Orthanc
W1019 11:32:07.881284 PluginsManager.cpp:261] Registering plugin 'mysql-storage' (version 4.3)
W1019 11:32:07.881284 PluginsManager.cpp:157] The MySQL storage area is currently disabled, set "EnableStorage" to "true" in the "MySQL" section of the configuration file of Orthanc
W1019 11:32:08.068817 PluginsManager.cpp:261] Registering plugin 'neuro' (version 1.0)
W1019 11:32:08.271913 PluginsManager.cpp:261] Registering plugin 'odbc-index' (version 1.1)
W1019 11:32:08.287538 PluginsManager.cpp:157] The ODBC index is currently disabled, set "EnableIndex" to "true" in the "Odbc" section of the configuration file of Orthanc
W1019 11:32:08.428166 PluginsManager.cpp:261] Registering plugin 'odbc-storage' (version 1.1)
W1019 11:32:08.443790 PluginsManager.cpp:157] The ODBC storage area is currently disabled, set "EnableStorage" to "true" in the "Odbc" section of the configuration file of Orthanc
W1019 11:32:08.553269 PluginsManager.cpp:261] Registering plugin 'postgresql-index' (version 4.0)
W1019 11:32:08.553269 PluginsManager.cpp:157] The PostgreSQL index is currently disabled, set "EnableIndex" to "true" in the "PostgreSQL" section of the configuration file of Orthanc
W1019 11:32:08.662548 PluginsManager.cpp:261] Registering plugin 'postgresql-storage' (version 4.0)
W1019 11:32:08.662548 PluginsManager.cpp:157] The PostgreSQL storage area is currently disabled, set "EnableStorage" to "true" in the "PostgreSQL" section of the configuration file of Orthanc
W1019 11:32:08.787549 PluginsManager.cpp:261] Registering plugin 'tcia' (version 1.1)
W1019 11:32:08.881304 PluginsManager.cpp:261] Registering plugin 'transfers' (version 1.4)
W1019 11:32:09.068804 PluginsManager.cpp:261] Registering plugin 'web-viewer' (version 2.8)
W1019 11:32:09.068804 PluginsManager.cpp:157] Initializing the Web viewer
W1019 11:32:09.068804 PluginsManager.cpp:157] Web viewer using 2 threads for the decoding of the DICOM images
W1019 11:32:09.068804 PluginsManager.cpp:157] Storing the cache of the Web viewer in folder: E:\orthanc\storage\WebViewerCache
W1019 11:32:09.068804 PluginsManager.cpp:157] Web viewer using a cache of 100 MB
W1019 11:32:09.178182 PluginsManager.cpp:261] Registering plugin 'wsi' (version 1.1)
W1019 11:32:09.178182 PluginsManager.cpp:157] The whole-slide imaging plugin will use at most 4 threads to transcode the tiles
W1019 11:32:09.334434 PluginsManager.cpp:261] Registering plugin 'osimis-web-viewer' (version 1.4.2.0-9d9eff4)
W1019 11:32:09.334434 PluginsManager.cpp:157] Initializing the Web viewer
W1019 11:32:09.334434 PluginsManager.cpp:157] Using GDCM instead of the DICOM decoder that is built in Orthanc
W1019 11:32:09.443808 PluginsManager.cpp:261] Registering plugin 'serve-folders' (version 1.11.3)
W1019 11:32:09.459810 PluginsManager.cpp:157] ServeFolders: Empty configuration file: No additional folder will be served!
W1019 11:32:09.459810 OrthancInitialization.cpp:406] SQLite index directory: "E:\orthanc\storage"
W1019 11:32:09.459810 OrthancInitialization.cpp:505] Storage directory: "E:\orthanc\storage"
W1019 11:32:09.459810 HttpClient.cpp:1194] HTTPS will use the CA certificates from this file: C:\Program Files\Orthanc Server\Configuration\ca-certificates.crt
W1019 11:32:09.459810 LuaContext.cpp:94] Lua says: Lua toolbox installed
W1019 11:32:09.475057 LuaContext.cpp:94] Lua says: Lua toolbox installed
W1019 11:32:09.475057 ServerContext.cpp:505] Disk compression is disabled
W1019 11:32:09.475057 ServerIndex.cpp:381] No limit on the number of stored patients
W1019 11:32:09.475057 ServerIndex.cpp:401] No limit on the size of the storage area
W1019 11:32:09.475057 ServerIndex.cpp:420] Maximum Storage mode: Recycle
W1019 11:32:09.475057 ServerContext.cpp:242] Reloading the jobs from the last execution of Orthanc
W1019 11:32:09.771942 JobsEngine.cpp:272] The jobs engine has started with 2 threads
W1019 11:32:09.771942 main.cpp:1299] The DICOM port is privileged (104 is below 1024), make sure you run Orthanc as root/administrator
E1019 11:32:09.787563 OrthancException.cpp:61] The TCP port of the DICOM server is privileged or already in use:  (port = 104) cannot create network: TCP Initialization Error: Only one usage of each socket address (protocol/network address/port) is normally permitted.
E1019 11:32:09.787563 ServerContext.cpp:458] INTERNAL ERROR: ServerContext::Stop() should be invoked manually to avoid mess in the destruction order!
W1019 11:32:14.881407 JobsEngine.cpp:313] The jobs engine has stopped
W1019 11:32:14.990780 PluginsManager.cpp:211] Unregistering plugin 'Azure Blob Storage' (version 2.1.2)
W1019 11:32:14.990780 PluginsManager.cpp:157] Azure Blob Storage plugin is finalizing
W1019 11:32:15.006407 PluginsManager.cpp:211] Unregistering plugin 'authorization' (version 0.4.1)
W1019 11:32:15.006407 PluginsManager.cpp:211] Unregistering plugin 'connectivity-checks' (version 1.11.3)
W1019 11:32:15.006407 PluginsManager.cpp:211] Unregistering plugin 'delayed-deletion' (version 1.11.3)
W1019 11:32:15.006407 PluginsManager.cpp:211] Unregistering plugin 'dicom-web' (version 1.13)
W1019 11:32:15.006407 PluginsManager.cpp:211] Unregistering plugin 'gdcm' (version 1.5)
W1019 11:32:15.006407 PluginsManager.cpp:211] Unregistering plugin 'housekeeper' (version 1.11.3)
W1019 11:32:15.006407 PluginsManager.cpp:157] Housekeeper plugin is finalizing
W1019 11:32:15.006407 PluginsManager.cpp:211] Unregistering plugin 'indexer' (version 1.0)
W1019 11:32:15.006407 PluginsManager.cpp:157] Folder indexer plugin is finalizing
W1019 11:32:15.006407 PluginsManager.cpp:211] Unregistering plugin 'mysql-index' (version 4.3)
W1019 11:32:15.006407 PluginsManager.cpp:157] MySQL index is finalizing
W1019 11:32:15.006407 PluginsManager.cpp:211] Unregistering plugin 'mysql-storage' (version 4.3)
W1019 11:32:15.006407 PluginsManager.cpp:157] MySQL storage area is finalizing
W1019 11:32:15.006407 PluginsManager.cpp:211] Unregistering plugin 'neuro' (version 1.0)
W1019 11:32:15.006407 PluginsManager.cpp:211] Unregistering plugin 'odbc-index' (version 1.1)
W1019 11:32:15.006407 PluginsManager.cpp:157] ODBC index is finalizing
W1019 11:32:15.006407 PluginsManager.cpp:211] Unregistering plugin 'odbc-storage' (version 1.1)
W1019 11:32:15.006407 PluginsManager.cpp:157] ODBC storage area is finalizing
W1019 11:32:15.006407 PluginsManager.cpp:211] Unregistering plugin 'orthanc-explorer-2' (version 0.6.0)
W1019 11:32:15.006407 PluginsManager.cpp:211] Unregistering plugin 'osimis-web-viewer' (version 1.4.2.0-9d9eff4)
W1019 11:32:15.006407 PluginsManager.cpp:157] Finalizing the Web viewer
W1019 11:32:15.037657 PluginsManager.cpp:211] Unregistering plugin 'postgresql-index' (version 4.0)
W1019 11:32:15.037657 PluginsManager.cpp:157] PostgreSQL index is finalizing
W1019 11:32:15.037657 PluginsManager.cpp:211] Unregistering plugin 'postgresql-storage' (version 4.0)
W1019 11:32:15.037657 PluginsManager.cpp:157] PostgreSQL storage area is finalizing
W1019 11:32:15.053295 PluginsManager.cpp:211] Unregistering plugin 'serve-folders' (version 1.11.3)
W1019 11:32:15.053295 PluginsManager.cpp:211] Unregistering plugin 'stone-webviewer' (version 2.5)
W1019 11:32:15.053295 PluginsManager.cpp:211] Unregistering plugin 'tcia' (version 1.1)
W1019 11:32:15.053295 PluginsManager.cpp:157] TCIA plugin is finalizing
W1019 11:32:15.053295 PluginsManager.cpp:211] Unregistering plugin 'transfers' (version 1.4)
W1019 11:32:15.053295 PluginsManager.cpp:157] Transfers accelerator plugin is finalizing
W1019 11:32:15.053295 PluginsManager.cpp:211] Unregistering plugin 'web-viewer' (version 2.8)
W1019 11:32:15.053295 PluginsManager.cpp:157] Finalizing the Web viewer
W1019 11:32:16.398012 PluginsManager.cpp:211] Unregistering plugin 'worklists' (version 1.11.3)
W1019 11:32:16.398012 PluginsManager.cpp:157] Sample worklist plugin is finalizing
W1019 11:32:16.398012 PluginsManager.cpp:211] Unregistering plugin 'wsi' (version 1.1)
E1019 11:32:16.490806 main.cpp:2075] Uncaught exception, stopping now: [The TCP port of the DICOM server is privileged or already in use] (code 2004)
W1019 11:32:16.490806 main.cpp:2106] Orthanc has stopped

This is the full logfile. You can see that

- openrem_qr seems to be doing fine until:

[19/Oct/2023 11:32:02] INFO [remapp.netdicom.qrscu:91] Returning Success response from echo to localhost 104 ORTHANC

- orthanc log shows on:

W1019 11:32:05.724997 main.cpp:2018] Orthanc version: 1.11.3
W1019 11:32:05.724997 OrthancConfiguration.cpp:108] Scanning folder "Configuration" for configuration files

Do you have any idea what this is about?

to me it looks as if Orthanc is rebooted between 11:32:02 and 11:32:05? 

And then it cannot continue because the dedicated TCP port (104) is already in use, and somehow the whole orthanc crashes.

Any ideas/help are welcome.

I have not been able to continuously run OpenREM for more than a couple of hours before an orthanc crash.

Op woensdag 14 juni 2023 om 08:13:44 UTC+2 schreef OpenREM:

[Ed McDonagh]

Hi Han, 

Can you review bitbucket issue 1003 and see if it fixes your issue? With such regularity of crashing, it should become clear fairly quickly if this is the fix you need.

Kind regards

Ed

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/65f86cd0-3211-4803-9b93-74a860877429n%40googlegroups.com.

[h4n ke]

Thank you Ed for your fast reply and pointing to  bitbucket issue 1003. I edited the lua file which i found in E:/Orthanc/ and rebooted the server.

Now I enabled the queries on my four modality groups and will monitor the succesrate.

The memory load was already worrying me and my colleagues in our IT department, hope it is fixed in this easy manner.

kind regards,

Han

Op donderdag 19 oktober 2023 om 17:21:16 UTC+2 schreef Ed McDonagh:

[h4n ke]

Dear Ed, the fix in   bitbucket issue 1003 solved the orthanc PACS crashing problem. Now I can run OpenREM continuously. The memory load is not ridiculously high anymore, and the huge, growing, list of pyhton-processes in taskmanager has also gone.

Much appreciated.

Han

Op vrijdag 20 oktober 2023 om 10:33:00 UTC+2 schreef h4n ke:

[Ed McDonagh]

That's great to hear, thanks for the feedback Han!

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/ac9265cc-994f-412e-8a63-b812d8b87f1cn%40googlegroups.com.

[John Cronin]

Hi All,

I'm also having an issue similar to ED with Q/R failing after random times/number of studies imported. I'm running OpenREM through docker on Windows 10.

Would the fix in   bitbucket issue 1003 fix this for my setup or is this fix exclusively for an OpenREM install directly on Windows?

I can't seem to find that "openrem_orthanc_config_windows.lua" in my directory but I'm a newbie to Docker so I'm probably missing something obvious so apologies!

Thanks,

John

[Ed McDonagh]

Hi John

I think the original issue for this thread was related to memory use on Windows. By using Docker, you are actually using Linux in a container and the same fix won't apply.

Does your system run out of memory and crash, or is it limited to the Q/R failing?

Can you share any OpenREM log files (https://docs.openrem.org/en/latest/troubleshooting.html#openrem-log-files) or Docker log files (https://docs.openrem.org/en/latest/troubleshooting.html#general-docker-troubleshooting)?

It might be worth starting a new thread too 😊

Kind regards

Ed