decompressor stage produces broken DICOM files

[Keith Hulsey]

I used the decompressor stage on images compressed with JPEG2000Lossless because the target of the DiconExportService cannot negotiate that transfer syntax. The decompressed file cannot be transmitted as indicated in the logs I copied below. I can display the original image in Microdicom but the decompressed image will not load because it is not a valid DICOM image.

These logs include the pipeline that processed the image and the logs. Below that I included the logs that indicate the CTP version that is running.

    <Pipeline name="Test JPEG2000LossLess send"> <DirectoryImportService class="org.rsna.ctp.stdstages.DirectoryImportService" import="roots/JPEG/DirectoryImportService/import" name="DirectoryImportService" quarantine="quarantines/JPEG/DirectoryImportService" root="roots/JPEG/DirectoryImportService"/> <DicomDecompressor class="org.rsna.ctp.stdstages.DicomDecompressor" name="DicomDecompressor" quarantine="quarantines/JPEG/DicomDecompressor" root="roots/JPEG/DicomDecompressor" script="scripts/JPEG/DicomDecompressor.script"/> <DicomExportService associationTimeout="15" class="org.rsna.ctp.stdstages.DicomExportService" dicomScript="scripts/JPEG/CTPout/SelectForTransfer.script" name="Export to CTP" quarantine="quarantines/JPEG/CTPout/DicomExportService" root="roots/JPEG/CTPout/DicomExportService" url="dicom://CTP_test:CTP_te...@198.215.54.32:9205"/> <DicomExportService associationTimeout="15" class="org.rsna.ctp.stdstages.DicomExportService" dicomScript="scripts/JPEG/CC/SelectForTransfer.script" name="Export to ClearCanvas" quarantine="quarantines/JPEG/CC/DicomExportService" root="roots/JPEG/CC/DicomExportService" url="dicom://E6-230BX-RAD:CTP_te...@E6-230BX-RAD.swmed.org:104"/> <DicomExportService associationTimeout="15" class="org.rsna.ctp.stdstages.DicomExportService" dicomScript="scripts/JPEG/iPACS/SelectForTransfer.script" name="Export to iPACS" quarantine="quarantines/JPEG/iPACS/DicomExportService" root="roots/JPEG/iPACS/DicomExportService" url="dicom://CTP_test:CTPsen...@ipacs.swmed.org:1040"/> <DicomExportService associationTimeout="15" class="org.rsna.ctp.stdstages.DicomExportService" dicomScript="scripts/JPEG/FW/SelectForTransfer.script" name="Export to Flywheel" quarantine="quarantines/JPEG/FW/DicomExportService" root="roots/JPEG/FW/DicomExportService" url="dicom://FW_Connector:CTP_te...@198.215.54.92:30104"/> <DicomFilter class="org.rsna.ctp.stdstages.DicomFilter" name="DicomFilter" quarantine="quarantines/JPEG/DicomFilter" root="roots/JPEG/DicomFilter" script="scripts/JPEG/DicomFilter.script"/> </Pipeline> </Configuration> 16:55:31 INFO [Configuration] Available ImageIO Codecs: R/W BMP R/W DICOM R/W GIF R/W JPEG R JPEG-LOSSLESS R/W JPEG2000 R/W JPG R/W PCX R/W PNG R/W PNM R/W RAW R RLE R/W TIFF R/W WBMP 16:55:31 INFO [DicomStorageSCU] AssociationCloser instantiated with 15 second timeout 16:55:31 INFO [DicomStorageSCU] AssociationCloser instantiated with 15 second timeout 16:55:31 INFO [DicomStorageSCU] AssociationCloser instantiated with 15 second timeout 16:55:32 INFO [DicomStorageSCU] AssociationCloser instantiated with 15 second timeout 16:55:32 INFO [DicomStorageSCU] AssociationCloser instantiated with 15 second timeout 16:55:32 INFO [DicomStorageSCU] AssociationCloser instantiated with 15 second timeout 16:55:32 INFO [DicomStorageSCU] AssociationCloser instantiated with 15 second timeout 16:55:32 INFO [DicomStorageSCU] AssociationCloser instantiated with 15 second timeout 16:55:32 INFO [DicomStorageSCU] AssociationCloser instantiated with 15 second timeout 16:55:32 INFO [HttpServer] HttpServer started on port 1082 [maxThreads=20] 16:55:32 INFO [HttpServer] HttpServer IP adddress: 198.215.54.32 16:55:32 INFO [AbstractExportService] Export to CTP: Exporter Thread: Started 16:55:32 INFO [AbstractExportService] Export to ClearCanvas: Exporter Thread: Started 16:55:32 INFO [AbstractExportService] Export to iPACS: Exporter Thread: Started 16:55:32 INFO [AbstractExportService] Export to Flywheel: Exporter Thread: Started 16:55:32 INFO [AbstractExportService] DicomExportService: Exporter Thread: Started 16:55:32 INFO [AbstractExportService] DicomExportService: Exporter Thread: Started 16:55:32 INFO [DicomStorageSCP] DicomImportService: SCP open on port 9205 16:55:32 INFO [AbstractExportService] DicomExportService: Exporter Thread: Started 16:55:32 INFO [DicomStorageSCP] DicomImportService: SCP open on port 9204 16:55:32 INFO [AbstractExportService] DicomExportService: Exporter Thread: Started 16:55:32 INFO [AbstractExportService] DicomExportService: Exporter Thread: Started 16:55:32 INFO [AbstractExportService] DicomExportService: Exporter Thread: Started 16:55:56 WARN [LoggerLevelServlet] org.rsna.ctp.pipeline.AbstractExportService logger level set to DEBUG 16:56:07 WARN [LoggerLevelServlet] org.rsna.ctp.stdstages.DicomDecompressor logger level set to DEBUG 16:56:14 WARN [LoggerLevelServlet] org.rsna.ctp.stdstages.anonymizer.dicom.DICOMDecompressor logger level set to DEBUG 16:56:21 WARN [LoggerLevelServlet] org.rsna.ctp.stdstages.DicomExportService logger level set to DEBUG 16:56:34 WARN [LoggerLevelServlet] org.rsna.ctp.stdstages.dicom.DicomStorageSCU logger level set to DEBUG 16:58:52 DEBUG [DICOMDecompressor] File length = 29232 16:58:52 DEBUG [DICOMDecompressor] planarConfig = 0 16:58:52 DEBUG [DICOMDecompressor] photometricInt = MONOCHROME2 16:58:52 DEBUG [DICOMDecompressor] numberOfFrames = 1 16:58:52 DEBUG [DICOMDecompressor] rows = 1024 16:58:52 DEBUG [DICOMDecompressor] columns = 1024 16:58:52 DEBUG [DICOMDecompressor] samplesPerPixel = 1 16:58:52 DEBUG [DICOMDecompressor] bytesPerSample = 1 16:58:52 DEBUG [DICOMDecompressor] pixelsVR = OB 16:58:52 DEBUG [DICOMDecompressor] nPixelBytes = 1048576 16:58:52 DEBUG [DICOMDecompressor] pixelBytesLength = 1048576 16:58:52 DEBUG [DICOMDecompressor] Decompressing frame 0 16:58:52 DEBUG [DICOMDecompressor] Number of banks = 1 16:58:52 DEBUG [DICOMDecompressor] Reading bank 0 16:58:52 DEBUG [DICOMDecompressor] Datatype: DataBuffer.TYPE_BYTE 16:58:52 DEBUG [DICOMDecompressor] Buffer length = 1048576 16:58:52 DEBUG [DICOMDecompressor] Done decompressing frame 0 16:58:57 DEBUG [DicomStorageSCU] Exporting QF-664137928732632470 to dicom://FW_Connector:CTP_te...@198.215.54.92:30104 16:58:57 DEBUG [DicomStorageSCU] active is null 16:58:57 DEBUG [DicomStorageSCU] currentTSUID is null 16:58:57 DEBUG [DicomStorageSCU] currentSOPClassUID is null 16:58:57 DEBUG [DicomStorageSCU] currentHost != requestedHost 16:58:57 DEBUG [DicomStorageSCU] currentPort != requestedPort 16:58:57 DEBUG [DicomStorageSCU] currentCalledAET != requestedCalledAET 16:58:57 DEBUG [DicomStorageSCU] currentCallingAET != requestedCallingAET 16:58:57 DEBUG [DicomStorageSCU] ...attempting to open a new association 16:58:57 DEBUG [DicomStorageSCU] ...successfully negotiated transfer syntax for 1.3.46.670589.70.1.9025622816976.20230822141423.1.2 16:58:57 DEBUG [DicomStorageSCU] ......SOPClass: Secondary Capture Image Storage 16:59:27 ERROR [ActiveAssociationImpl] Received A-ABORT source=0 - service-user reason=0 - reason-not-specified org.dcm4che.net.PDUException: Received A-ABORT source=0 - service-user reason=0 - reason-not-specified at org.dcm4cheri.net.DimseReaderImpl.nextPDataTF(DimseReaderImpl.java:200) ~[dcm4che.jar:1.4-JP 2022.03.26] at org.dcm4cheri.net.DimseReaderImpl.nextPDV(DimseReaderImpl.java:168) ~[dcm4che.jar:1.4-JP 2022.03.26] at org.dcm4cheri.net.DimseReaderImpl.read(DimseReaderImpl.java:78) ~[dcm4che.jar:1.4-JP 2022.03.26] at org.dcm4cheri.net.AssociationImpl.read(AssociationImpl.java:313) ~[dcm4che.jar:1.4-JP 2022.03.26] at org.dcm4cheri.net.ActiveAssociationImpl.run(ActiveAssociationImpl.java:219) ~[dcm4che.jar:1.4-JP 2022.03.26] at org.dcm4cheri.util.LF_ThreadPool.join(LF_ThreadPool.java:174) ~[dcm4che.jar:1.4-JP 2022.03.26] at org.dcm4cheri.net.ActiveAssociationImpl.run(ActiveAssociationImpl.java:164) ~[dcm4che.jar:1.4-JP 2022.03.26] at java.lang.Thread.run(Unknown Source) ~[?:1.8.0_281] 16:59:28 DEBUG [DicomStorageSCU] ...closing the open association 16:59:28 DEBUG [DicomStorageSCU] Error processing a DicomObject for transmission java.lang.NullPointerException: null at org.rsna.ctp.stdstages.dicom.DicomStorageSCU.send(DicomStorageSCU.java:337) ~[CTP.jar:?] at org.rsna.ctp.stdstages.DicomExportService.export(DicomExportService.java:85) ~[CTP.jar:?] at org.rsna.ctp.pipeline.AbstractExportService$Exporter.run(AbstractExportService.java:158) ~[CTP.jar:?] 16:59:28 WARN [DicomStorageSCU] java.lang.NullPointerException 16:59:28 WARN [DicomStorageSCU] ...1.3.46.670589.70.1.9025622816976.20230822141423.1.2 16:59:28 WARN [DicomStorageSCU] ...Secondary Capture Image Storage 16:59:28 WARN [AbstractExportService] Export to Flywheel: Unable to export roots/JPEG/FW/DicomExportService/active/QF-664137928732632470

16:55:31 INFO [ClinicalTrialProcessor] Cache cleared 16:55:31 INFO [Configuration] Operating system: Linux 16:55:31 INFO [Configuration] Java vendor: Oracle Corporation 16:55:31 INFO [Configuration] Java version: 1.8.0_281 16:55:31 INFO [Configuration] Java VM vendor: Oracle Corporation 16:55:31 INFO [Configuration] Java VM version: 25.281-b09 16:55:31 INFO [Configuration] Java data model: 32 bits 16:55:31 INFO [Configuration] Java max memory: 932,184,064 bytes 16:55:31 INFO [Configuration] CTP crypto strength: 2147483647 16:55:31 INFO [Configuration] Max crypto strength: 2147483647 16:55:31 INFO [Configuration] ImageIO Tools: 1.2-pre-dr-b04 (installed) 16:55:31 INFO [Configuration] CTP build: 2023.05.23 at 20:27:51 CDT 16:55:31 INFO [Configuration] Util build: 2022.10.13 at 10:05:30 CDT 16:55:31 INFO [Configuration] Start time: 2023.09.14 at 16:55:31 16:55:31 INFO [Configuration] user.dir: /data/CTP_test/CTP 16:55:31 INFO [Configuration] java.ext.dirs: /usr/java/jre1.8.0_281-i586/lib/ext:/usr/java/packages/lib/ext 16:55:31 INFO [Configuration] IP Address: 198.215.54.32 16:55:31 INFO [Configuration] Classpath: file:/data/CTP_test/CTP/libraries/imageio/pixelmed_codec.jar file:/data/CTP_test/CTP/libraries/imageio/dcm4che-imageio-rle-2.0.25.jar file:/data/CTP_test/CTP/libraries/imageio/twelvemonkeys-imageio-core-3.6-SNAPSHOT.jar file:/data/CTP_test/CTP/libraries/imageio/jai_imageio-1.2-pre-dr-b04.jar file:/data/CTP_test/CTP/libraries/imageio/twelvemonkeys-imageio-jpeg-3.6-SNAPSHOT.jar file:/data/CTP_test/CTP/libraries/imageio/twelvemonkeys-common-io-3.6-SNAPSHOT.jar file:/data/CTP_test/CTP/libraries/imageio/twelvemonkeys-imageio-metadata-3.6-SNAPSHOT.jar file:/data/CTP_test/CTP/libraries/imageio/clibwrapper_jiio-1.2-pre-dr-b04.jar file:/data/CTP_test/CTP/libraries/imageio/twelvemonkeys-common-image-3.6-SNAPSHOT.jar file:/data/CTP_test/CTP/libraries/imageio/pixelmed_imageio.jar file:/data/CTP_test/CTP/libraries/imageio/twelvemonkeys-common-lang-3.6-SNAPSHOT.jar file:/data/CTP_test/CTP/libraries/pixelmed_codec.jar file:/data/CTP_test/CTP/libraries/ftp/jsch-0.1.53.jar file:/data/CTP_test/CTP/libraries/ftp/commons-net-3.3.jar file:/data/CTP_test/CTP/libraries/ftp/edtftpj.jar file:/data/CTP_test/CTP/libraries/ftp/commons-vfs2-2.0.jar file:/data/CTP_test/CTP/libraries/ftp/commons-logging-1.2.jar file:/data/CTP_test/CTP/libraries/CTP.jar file:/data/CTP_test/CTP/libraries/jdbm.jar file:/data/CTP_test/CTP/libraries/log4j/log4j-core-2.17.2.jar file:/data/CTP_test/CTP/libraries/log4j/log4j-1.2-api-2.17.2.jar file:/data/CTP_test/CTP/libraries/log4j/log4j-api-2.17.2.jar file:/data/CTP_test/CTP/libraries/email/mail.jar file:/data/CTP_test/CTP/libraries/email/mailapi.jar file:/data/CTP_test/CTP/libraries/email/activation.jar file:/data/CTP_test/CTP/libraries/edtftpj.jar file:/data/CTP_test/CTP/libraries/slf4j-log4j12-1.6.1.jar file:/data/CTP_test/CTP/libraries/getopt.jar file:/data/CTP_test/CTP/libraries/commons-compress-1.0.jar file:/data/CTP_test/CTP/libraries/dcm4che.jar file:/data/CTP_test/CTP/libraries/util.jar file:/data/CTP_test/CTP/libraries/slf4j-api-1.6.1.jar

 

[matthe...@gmail.com]

this issue here is that the DicomDecompressor stage is outputting a DICOM where there is an extra delimiter after the PixelData. Many (but not all) DICOM systems do not like this (ex. Horos, pydicom, Flywheel web uploader)