OpenCV JpegLS SIGSEGV

[Pavel Navrkal]

Hi Nicolas,

I'm trying to compress many small CT IVRLE dcm files to JpegLS TransferSyntax with dcm4che 5.14.1 and OpenCV codecs on Linux. Randomly the java process hangs with SIGSEGV in OpenCV codecs.

The crash is not caused by a particular dicom image but randomly every time on different image.

In the attachment is part of the hs_err file with the stack trace.

Can you look at it, and detect possible memory allocation problems in C code.

Best regards

Pavel

[Nicolas Roduit]

With your description, there is no evidence that the problem is located in the jpeg-ls encoder. 

It seems the image to be encoded become empty.

I need a way to reproduce the error to investigate more.

[navrka...@gmail.com]

Hi Nicolas,
I've tried the latest dcm4che version 5.16.1 with new OpenCV version, but there is still the same problem.
So I have investigating more time to debug it and I found it to be a problem with multiple threads and accessing native data of OpenCV Mat

I've tried to compress and decompress one small MR file in multiple threads. In one thread it seems to work correctly. But in more threads it randomly crash with malloc(), free() or SIGSEGV errors.

I've used jpeg-ls or jpeg2000 compression, both hangs after a few minutes. But most visible it is on jpeg-ls decompression. It hangs after a few seconds in 5 threads.

Without source of native lib I'm not able to debug it deeper.

The small java example and dicom MR file is in attachement. I've run it with -Xmx512m or -Xmx1024m (with more memory it runs longer before crash - perhaps some problem with GC and JNI) on Linux 64bit.

Best regards

Pavel

[Nicolas Roduit]

With the release 5.16.1 I cannot reproduce the crash. If I change to 25 Threads and set Xmx to 64Mb, I get some errors but no crash:

Exception in thread "Thread-0" 09:11:54,576 DEBUG - Decompressor: org.dcm4che3.opencv.NativeImageReader

java.lang.OutOfMemoryError: Java heap space

I can reproduce the VM crash with the version 5.14.1 which have problems to free the native objects.

[navrka...@gmail.com]

I've rechecked the version of dcm4che jars and native libs and all is OK from 5.16.1. I've tested the 5.14.1 version too and there is no difference. Both version crash after a few seconds with 5 threads and 1GB in -Xmx. I've tested with oracle JDK 8 and 11 and openjdk 11. In all jdk it hangs with "free(): invalid pointer":

[Jan Horáček]

Hi Nicolas,

I've faced the similar problem with the codecs. I tried the example above. I created a simple project in Eclipse. When I run the project in debug mode, there's no error. But if I run it, it fails within few seconds. So I compared in on Linux and Windows and it crashes on both:

Linux: java version "1.8.0_202", ldd (Gentoo 2.27-r6 p3) 2.27,

15:55:45.838 [Thread-4] DEBUG org.dcm4che3.imageio.codec.Transcoder - Compressed frame #1 in 41 ms, ratio 3.1085312:1
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fc1766c1161, pid=18255, tid=0x00007fc153dfd700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build 1.8.0_202-b08)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C15:55:45.838 [Thread-1] DEBUG org.dcm4che3.imageio.codec.Transcoder - Compressed frame #1 in 32 ms, ratio 3.1085312:1
15:55:45.839 [Thread-4] DEBUG org.dcm4che3.imageio.codec.Transcoder - Compressor: org.dcm4che3.opencv.NativeJLSImageWriter
15:55:45.839 [Thread-1] DEBUG org.dcm4che3.imageio.codec.Transcoder - Compressor: org.dcm4che3.opencv.NativeJLSImageWriter
  [libc.so.6+0x16a161]

Windows 10: java version "11.0.3"

18:04:36.555 [Thread-4] DEBUG org.dcm4che3.imageio.codec.Transcoder - Decompressed frame #1 in 15 ms, ratio 1:3.1085129
java.lang.Exception: unknown exception
18:04:36.567 [Thread-4] DEBUG org.dcm4che3.imageio.codec.Transcoder - Decompressor: org.dcm4che3.opencv.NativeImageReader
    at org.opencv.imgcodecs.Imgcodecs.dicomJpgMatRead_0(Native Method)
    at org.opencv.imgcodecs.Imgcodecs.dicomJpgMatRead(Imgcodecs.java:195)
    at org.dcm4che3.opencv.NativeImageReader.getNativeImage(NativeImageReader.java:278)
    at org.dcm4che3.opencv.NativeImageReader.read(NativeImageReader.java:251)
    at org.dcm4che3.imageio.codec.Transcoder.decompressFrame(Transcoder.java:588)
    at org.dcm4che3.imageio.codec.Transcoder.decompressPixelData(Transcoder.java:447)
    at org.dcm4che3.imageio.codec.Transcoder.processPixelData(Transcoder.java:428)
    at org.dcm4che3.imageio.codec.Transcoder.access$200(Transcoder.java:70)
    at org.dcm4che3.imageio.codec.Transcoder$1.readValue(Transcoder.java:383)
    at org.dcm4che3.io.DicomInputStream.readAttributes(DicomInputStream.java:534)
    at org.dcm4che3.imageio.codec.Transcoder.transcode(Transcoder.java:362)
    at test.transcode.Transcode.run(Transcode.java:41)
    at java.base/java.lang.Thread.run(Thread.java:834)

Difference between Windows and Linux is that on Linux the system kills the whole process, on Windows it just throws an Exception and the rest continues without any problem.

Kind regards

Jan

Dne středa 24. dubna 2019 9:19:39 UTC+2 Nicolas Roduit napsal(a):

[gunterze]

I could reproduce the crash after 400-1200 decompressions with OpenJDK 11.0.2+9 on Ubuntu 18.10:

# A fatal error has been detected by the Java Runtime Environment:
#

#  SIGSEGV (0xb) at pc=0x00007f997b14ba3e, pid=7937, tid=7960
#
# JRE version: OpenJDK Runtime Environment (11.0.2+9) (build 11.0.2+9-Ubuntu-3ubuntu118.10.3)
# Java VM: OpenJDK 64-Bit Server VM (11.0.2+9-Ubuntu-3ubuntu118.10.3, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libopencv_java.so+0x28fa3e]  cv::Mat::deallocate()+0x3e
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P" (or dumping to /home/gunter/work/dcm4che/dcm4che-imageio-test/core.7937)
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  S U M M A R Y ------------

Command Line: -Djava.library.path=/home/gunter/dcm4che/lib/linux-x86_64 test.transcode.Transcode

but not with OpenJDK 1.8.0_191-8u191-b12 .

[Nicolas Roduit]

I finally managed to reproduce the bug (strangely in debug mode eclipse the issue appears very rarely but always in run mode).

I was a bit worried about how to solve the issue after reading this post. However, the problem is that Java and C++ have different memory management paradigm, so calling Mat.release()  explicitly when the image is not used anymore is enough.

This problem occurs when images are intensively decoded or encoded. The current implementation does a lot of conversions between Mat and BufferedImage, this could be improved by having an implementation without using BufferedImage.

I publish a fix which has been tested to transcode simultaneously 50 images (in 50 threads) from jpeg-ls to jpeg2000 with 128MB (xmx). No crash after more than 10'000 conversions.

[Gunter Zeilinger]

Great. Will release it in 5.16.3 today. Thanks!

gunter

--
You received this message because you are subscribed to the Google Groups "dcm4che" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dcm4che+u...@googlegroups.com.
To post to this group, send email to dcm...@googlegroups.com.
Visit this group at https://groups.google.com/group/dcm4che.
For more options, visit https://groups.google.com/d/optout.

[navrka...@gmail.com]

It looks good at first glance. Thanks.

Pavel

Dne čtvrtek 25. dubna 2019 22:13:20 UTC+2 Nicolas Roduit napsal(a):

[Jan Horáček]

Hi Nicolas,

it still crashes for me unfortunately with different exceptions:

11:46:24.044 [Thread-8] DEBUG org.dcm4che3.imageio.codec.Transcoder - Decompressor: org.dcm4che3.opencv.NativeImageReader
java.lang.IllegalArgumentException: Width (0) and height (0) must be > 0
11:46:24.056 [Thread-8] DEBUG org.dcm4che3.imageio.codec.Transcoder - Decompressed frame #1 in 12 ms, ratio 1:3.1085129

 at java.desktop/java.awt.image.SampleModel.<init>(SampleModel.java:126)
 at java.desktop/java.awt.image.ComponentSampleModel.<init>(ComponentSampleModel.java:140)
 at java.desktop/java.awt.image.PixelInterleavedSampleModel.<init>(PixelInterleavedSampleModel.java:87)
 at java.desktop/java.awt.image.Raster.createInterleavedRaster(Raster.java:642)
 at java.desktop/java.awt.image.Raster.createInterleavedRaster(Raster.java:278)
 at java.desktop/java.awt.image.Raster.createInterleavedRaster(Raster.java:212)
 at java.desktop/java.awt.image.ComponentColorModel.createCompatibleWritableRaster(ComponentColorModel.java:2827)
 at org.weasis.opencv.op.ImageConversion.toBufferedImage(ImageConversion.java:79)
 at org.weasis.opencv.op.ImageConversion.toBufferedImage(ImageConversion.java:116)
 at org.dcm4che3.opencv.NativeImageReader.read(NativeImageReader.java:252)

 at org.dcm4che3.imageio.codec.Transcoder.decompressFrame(Transcoder.java:588)
 at org.dcm4che3.imageio.codec.Transcoder.decompressPixelData(Transcoder.java:447)
 at org.dcm4che3.imageio.codec.Transcoder.processPixelData(Transcoder.java:428)
 at org.dcm4che3.imageio.codec.Transcoder.access$200(Transcoder.java:70)
 at org.dcm4che3.imageio.codec.Transcoder$1.readValue(Transcoder.java:383)
 at org.dcm4che3.io.DicomInputStream.readAttributes(DicomInputStream.java:534)
 at org.dcm4che3.imageio.codec.Transcoder.transcode(Transcoder.java:362)

 at test.transcode.Transcode.run(Transcode.java:46)
 at java.base/java.lang.Thread.run(Thread.java:834)


12:01:04.703 [Thread-3] DEBUG org.dcm4che3.imageio.codec.Transcoder - Decompressor: org.dcm4che3.opencv.NativeImageReader
java.lang.Exception: unknown exception
 at org.opencv.core.Mat.nGetS(Native Method)
 at org.opencv.core.Mat.get(Mat.java:1082)
 at org.weasis.opencv.op.ImageConversion.toBufferedImage(ImageConversion.java:98)
 at org.weasis.opencv.op.ImageConversion.toBufferedImage(ImageConversion.java:116)
 at org.dcm4che3.opencv.NativeImageReader.read(NativeImageReader.java:252)

 at org.dcm4che3.imageio.codec.Transcoder.decompressFrame(Transcoder.java:588)
 at org.dcm4che3.imageio.codec.Transcoder.decompressPixelData(Transcoder.java:447)
 at org.dcm4che3.imageio.codec.Transcoder.processPixelData(Transcoder.java:428)
 at org.dcm4che3.imageio.codec.Transcoder.access$200(Transcoder.java:70)
 at org.dcm4che3.imageio.codec.Transcoder$1.readValue(Transcoder.java:383)
 at org.dcm4che3.io.DicomInputStream.readAttributes(DicomInputStream.java:534)
 at org.dcm4che3.imageio.codec.Transcoder.transcode(Transcoder.java:362)

 at test.transcode.Transcode.run(Transcode.java:46)
 at java.base/java.lang.Thread.run(Thread.java:834)

it's still the same example as above, 10 threads, Windows 10: Oracle java version "11.0.3"

Jan

Dne pátek 26. dubna 2019 9:10:25 UTC+2 navrka...@gmail.com napsal(a):

[navrka...@gmail.com]

After 20 minutes running decompression in 10 threads it crashed on Linux OpenJDK11 on "corrupted double-linked list" :-(

Pavel

Dne pátek 26. dubna 2019 12:20:18 UTC+2 Jan Horáček napsal(a):

[Jan Horáček]

Hi Nicolas,

I also tried the reverse encoding (IVRLE -> JPEG-LS) and it crashes too. The test file is attached. In the example I just changed the destination transfer syntax:

t.setDestinationTransferSyntax(UID.JPEGLSLossless);

and got following error after few secons:

15:13:36.630 [Thread-0] DEBUG org.dcm4che3.imageio.codec.Transcoder - Compressor: org.dcm4che3.opencv.NativeJLSImageWriter
javax.imageio.IIOException: Native JPEG-LS encoding error
 at org.dcm4che3.opencv.NativeJLSImageWriter.write(NativeJLSImageWriter.java:146)
 at org.dcm4che3.imageio.codec.Transcoder.compressFrame(Transcoder.java:620)
 at org.dcm4che3.imageio.codec.Transcoder.compressPixelData(Transcoder.java:488)
 at org.dcm4che3.imageio.codec.Transcoder.processPixelData(Transcoder.java:425)

 at org.dcm4che3.imageio.codec.Transcoder.access$200(Transcoder.java:70)
 at org.dcm4che3.imageio.codec.Transcoder$1.readValue(Transcoder.java:383)
 at org.dcm4che3.io.DicomInputStream.readAttributes(DicomInputStream.java:534)
 at org.dcm4che3.imageio.codec.Transcoder.transcode(Transcoder.java:362)

 at test.transcode.Transcode.run(Transcode.java:48)
 at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: CvException [org.opencv.core.CvException: cv::Exception: OpenCV(4.0.0-dev) C:\jenkins\workspace\opencv\modules\imgcodecs\src\loadsave.cpp:1181: error: (-215:Assertion failed) !image.empty() && image.isContinuous() in function 'cv::dicomJpgWrite'
]
 at org.opencv.imgcodecs.Imgcodecs.dicomJpgWrite_0(Native Method)
 at org.opencv.imgcodecs.Imgcodecs.dicomJpgWrite(Imgcodecs.java:227)
 at org.dcm4che3.opencv.NativeJLSImageWriter.write(NativeJLSImageWriter.java:133)
 ... 9 more

Windows 10: Oracle java version "11.0.3", -Xmx1024m

On Linux (java version "1.8.0_202", ldd (Gentoo 2.27-r6 p3) 2.27) it runs for a few hours without any problem yet.

Jan

Dne pátek 26. dubna 2019 13:24:06 UTC+2 navrka...@gmail.com napsal(a):

[Nicolas Roduit]

With configuration 75 Threads, -Xmx256m, openjdk-11 (build 11.0.3+7-Ubuntu-1ubuntu1) and after 2.5 Millions of conversions in jpeg-ls of ivrie.dcm I didn't manage to crash the VM or get any errors.

If you still have issues, could you provide a test case that I can reproduce?

[Jan Horáček]

Hi Nicolas,

the error occurs for me on Windows. I use just slightly modified example as provided by Pavel and ivrle.dcm image I attached earlier.

Error is:

10:30:01.657 [Thread-3] DEBUG org.dcm4che3.imageio.codec.Transcoder - Compressed frame #1 in 172 ms, ratio 3.1085312:1

javax.imageio.IIOException: Native JPEG-LS encoding error
 at org.dcm4che3.opencv.NativeJLSImageWriter.write(NativeJLSImageWriter.java:146)
 at org.dcm4che3.imageio.codec.Transcoder.compressFrame(Transcoder.java:620)
 at org.dcm4che3.imageio.codec.Transcoder.compressPixelData(Transcoder.java:488)
 at org.dcm4che3.imageio.codec.Transcoder.processPixelData(Transcoder.java:425)
 at org.dcm4che3.imageio.codec.Transcoder.access$200(Transcoder.java:70)
 at org.dcm4che3.imageio.codec.Transcoder$1.readValue(Transcoder.java:383)
 at org.dcm4che3.io.DicomInputStream.readAttributes(DicomInputStream.java:534)
 at org.dcm4che3.imageio.codec.Transcoder.transcode(Transcoder.java:362)
 at test.transcode.Transcode.run(Transcode.java:48)
 at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: CvException [org.opencv.core.CvException: cv::Exception: OpenCV(4.0.0-dev) C:\jenkins\workspace\opencv\modules\imgcodecs\src\loadsave.cpp:1181: error: (-215:Assertion failed) !image.empty() && image.isContinuous() in function 'cv::dicomJpgWrite'
]
 at org.opencv.imgcodecs.Imgcodecs.dicomJpgWrite_0(Native Method)
 at org.opencv.imgcodecs.Imgcodecs.dicomJpgWrite(Imgcodecs.java:227)
 at org.dcm4che3.opencv.NativeJLSImageWriter.write(NativeJLSImageWriter.java:133)
 ... 9 more

it does not matter if I use 10 or 75 threads, 256 or 1024 MB of memory... It always crashes within few minutes. But it never crashes when I run it in debuggin mode.

Kind regards

Jan

Dne středa 1. května 2019 10:37:31 UTC+2 Nicolas Roduit napsal(a):

[Jan Horáček]

Hi Nicolas,

attaching crash report from Linux (conversion JPEG -> IVRLE). 75 threads, -Xmx1024m. But I must mention that in this test it crashed after few hours... I hope it helps.

Kind regards

Jan

Dne čtvrtek 2. května 2019 15:06:20 UTC+2 Jan Horáček napsal(a):