QR issues

[t.c....@gmail.com]

Since the migration from OpenREM 0.10 to 1.0.0b2 I started seeing QR issues.

During a c-move transfer it randomly stops with the error message:
"Move failed twice in succession. Aborting move request"

Instead of skipping this single import and continuing to the next it seems to stop the import altogether:

[25/Mar/2024 02:37:48] DEBUG [remapp.netdicom.qrscu:2244] Mv: study_no 68
[25/Mar/2024 02:37:48] DEBUG [remapp.netdicom.qrscu:2250] Mv: study no 68 series no 1
[25/Mar/2024 02:37:48] INFO [remapp.netdicom.qrscu:2271] Requesting move: modality CT, study 68 (of 470) series 1 (of 1). Series contains 1 objects
[25/Mar/2024 02:37:48] DEBUG [remapp.netdicom.qrscu:2282] _move_req launched
[25/Mar/2024 02:37:48] DEBUG [remapp.netdicom.qrscu:2297] Series-level move - d is: (0008, 0052) Query/Retrieve Level                CS: 'SERIES'
(0020, 000d) Study Instance UID                  UI: 1.2.840.114350.2.408.2.798268.2.224634439.1
(0020, 000e) Series Instance UID                 UI: 1.3.12.2.1107.5.1.4.76043.30000024031806531497300003144
[25/Mar/2024 02:38:37] ERROR [remapp.netdicom.qrscu:2101] Move of study 68, series 1: Failure (0xa702) - Refused: Out of resources, unable to perform sub-operations. Sub-ops comp
leted: 0, failed: 1, warning: 0.
[25/Mar/2024 02:38:39] ERROR [remapp.netdicom.qrscu:2101] Move of study 68, series 1: Failure (0xa702) - Refused: Out of resources, unable to perform sub-operations. Sub-ops comp
leted: 0, failed: 1, warning: 0.
[25/Mar/2024 02:38:39] WARNING [remapp.netdicom.qrscu:2178] Query_id 267c80c7-5c3f-4fa0-9028-8b22e65c7462: Move failed twice in succession. Aborting move request
[25/Mar/2024 02:38:39] INFO [remapp.netdicom.qrscu:2305] Query_id 267c80c7-5c3f-4fa0-9028-8b22e65c7462: Move association released

When retrying the import of this single series manually, using the dcmtk utility "movescu" it keeps failing with the same error message.

When I retry the c-move request but with another target AET it succeeds immediately.

If I retry the original cmove request to Openrem it succeeds as well?!

In our hospital the dicom connections with PACS are proxied with the "Dicom Router" solution from Dicom Systems. Apparently this Dicom Router has some issues with Orthanc. I've never seen these errors with Conquest.

Any idea how I could debug/investigate these errors further?

@developers: would it be possible to proceed with QR even if one retrieval fails?

regards,

Tim de Wit

[Ed McDonagh]

Hi Tim

That is frustrating that the dicom systems software - is it the Rapid DICOM Router? - has issues with Orthanc. One of the reasons we went with Orthanc was that it is well maintained and conformant.

What behaviour would you want to see in this circumstance? We quit at that stage, which seemed reasonable when I coded it. If you move onto the next move, would you expect it to work, or to continue failing?

Kind regards

Ed

--
You received this message because you are subscribed to the Google Groups "OpenREM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrem+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/b1ac7e27-cf08-4e00-936b-9118e0acc8f8n%40googlegroups.com.

[t.c....@gmail.com]

Hi Ed,

Yes that's the one.. I solved one source of the problem already by increasing the DicomScpTimeout from 30 to 300 seconds in the Orthanc config. Apparently Orthanc was choking on huge mammography series of ~3GBq that caused timeout issues (which in itself shouldn't have been a problem if DIMSE packages were sent in time by the SCP, so I'm not sure which side to blame).

After that modification the MG images didn't cause any problems anymore.. but the problem shifted to SR series from CT studies. On several occasions I noticed that around series 81-82 (out of 150 or 200) the association errors happened. Perhaps too many open associations? Associatons not being closed fast enough? Or too many/second? Needs some further testing...

kind regards, Tim

Op dinsdag 26 maart 2024 om 13:48:23 UTC+1 schreef Ed McDonagh:

[t.c....@gmail.com]

Placed Orthac in debug mode and waited until the error occurred. This time it got to study 87 out of 325. 
It was hard to get useful info from the Orthanc logging but I noticed that it contained: 92 x "Association released" and 188 x "Association received".

Perhaps some associations are not released (quick enough)? Now testing with increasing the following setting in the orthanc config (from 100 to 1000):

  // Maximum number of query/retrieve DICOM requests that are
  // maintained by Orthanc. The least recently used requests get
  // deleted as new requests are issued.
  "QueryRetrieveSize" : 100,

Fingers crossed..

regards, Tim

Op woensdag 27 maart 2024 om 17:36:16 UTC+1 schreef t.c....@gmail.com:

[t.c....@gmail.com]

The QueryRetrieveSize setting didn't fix anything unfortunately.

I also had to turn off debug mode since it caused Orthanc itself to crash repeatedly.

Next I did the following.. from openrem_qr.log I extracted a list of 171 studyinstanceuid/seriesinstanceuid combinations of CT SR datasets and used it as input for a quickly written c-move python script (using the same pynetdicom that openrem uses). I had to place a sleep(1) inside the for loop; otherwise Orthanc stopped at some point, didn't respond anymore and caused the c-move requests to timeout. When stopping the script and even waiting for 15min Orthanc was still broken. The Orthanc process was still running, no errors in Orthanc.log and the service still responded on c-echo requests; however the SCP service didn't respond until the service was restarted. With a small delay inside the loop all 171 series were retrieved without problems!

I did some tests with an added "time.sleep(1)" inside OpenREMs qrscu.py and so far I haven't encountered any problems!

I placed the sleep() the line below "for series in study.dicomqrrspseries_set.filter(deleted_flag=False).all():"

Still with QueryRetrieveSize=1000 though so next step is to restore that setting to 100 and keep the sleep(1).

regards, Tim

Op donderdag 28 maart 2024 om 13:39:47 UTC+1 schreef t.c....@gmail.com:

[Ed McDonagh]

Thanks for doing and sharing this Tim, it's really helpful!

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/abe6a897-d56d-48d3-89a3-7d42c3e2ceedn%40googlegroups.com.

[t.c....@gmail.com]

Even though increasing the dimse timeout and inserting a sleep(1) inside the loop did help somewhat, the errors still kept returning (less frequent though). After a while Orthanc would still become unresponsive (c-echo still worked but no response whatsoever on c-store requests) and had to be restarted.

Finally I followed the procedure from:

     https://orthanc.uclouvain.be/book/users/debian-packages.html#replacing-the-package-from-the-service-by-the-lsb-binaries

to manually upgrade the latest Ubuntu Orthanc package (v1.10.0) to the latest available Orthanc version (1.12.3). After that it has been running for half a day now without any errors!

Ed, can you confirm that the Orthanc docker package also uses a version newer than 1.10.0?

regards, Tim

Op donderdag 28 maart 2024 om 19:30:07 UTC+1 schreef Ed McDonagh:

[Ed McDonagh]

Thanks Tim - that's a great help.

I'll check when I'm next near a computer, but I think Docker uses the latest from Orthanc.

I'll look at the options for Linux install versions too.

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/7ddcc05e-93be-47b1-94fc-055c9d41d9aen%40googlegroups.com.

[Ed McDonagh]

That was a really useful prod Tim!

The Orthanc image was a little neglected. All it does is take the Osimis build of Orthanc and add zip/unzip to the image. However, it hadn't been built for three years! And now I find that the repo has changed to orthancteam, and the way I was building the openrem/orthanc images, on Docker Hub, is no longer available.

I have now implemented a new Bitbucket Pipeline to build the orthancteam version of Orthanc, and pushed the resulting image to Docker Hub. This is based on Orthanc 1.12.3.

I'll add building the Orthanc image to the main repo instructions.

Thanks again,

Ed

To view this discussion on the web, visit https://groups.google.com/d/msgid/openrem/CAFoYOo1obLjH3LMOvvJ7SKYtFNPn9MZQ8ayHAtG%2B6Oe5E%3DWt6g%40mail.gmail.com.

[Ed McDonagh]

Regarding Ubuntu installs - it doesn't look like there is a clean easy way to get the latest version. I wonder if it is worth adding replacing with LSB binaries as a troubleshooting step?

How many of the libraries did you install on your system?

Kind regards

Ed

[t.c....@gmail.com]

Might be a good idea indeed... I only replaced the libraries already present after the default install. I used/created the following script for updating:

#!/bin/bash

# downloads: https://orthanc.uclouvain.be/downloads/linux-standard-base/orthanc/index.html

sudo service orthanc stop

sudo mkdir -p old/usr/sbin
sudo mkdir -p old/lib/orthanc
sudo mkdir -p old/usr/share/orthanc/plugins

sudo cp /usr/sbin/Orthanc old/usr/sbin/
sudo cp /lib/orthanc/* old/lib/orthanc/
sudo cp /usr/share/orthanc/plugins/* old/usr/share/orthanc/plugins/

sudo wget https://orthanc.uclouvain.be/downloads/linux-standard-base/orthanc/1.12.3/Orthanc --output-document /usr/sbin/Orthanc
sudo chmod +x /usr/sbin/Orthanc

# remove symlinks
sudo rm -f /usr/share/orthanc/plugins/*.so
sudo wget https://orthanc.uclouvain.be/downloads/linux-standard-base/orthanc/1.12.3/libServeFolders.so --output-document /lib/orthanc/libServeFolders.so.1.12.3
sudo wget https://orthanc.uclouvain.be/downloads/linux-standard-base/orthanc/1.12.3/libModalityWorklists.so --output-document /lib/orthanc/libModalityWorklists.so.1.12.3
#sudo wget https://orthanc.uclouvain.be/downloads/linux-standard-base/orthanc-dicomweb/1.16/libOrthancDicomWeb.so --output-document /lib/orthanc/libOrthancDicomWeb.so.1.12.3

ln -s /lib/orthanc/libServeFolders.so.1.12.3 /usr/share/orthanc/plugins/libServeFolders.so
ln -s /lib/orthanc/libModalityWorklists.so.1.12.3 /usr/share/orthanc/plugins/libModalityWorklists.so

sudo service orthanc restart

regards, Tim

Op zaterdag 6 april 2024 om 22:26:33 UTC+2 schreef Ed McDonagh: