Help needed with JPEG tests
Yves Martel
As some of you know, one of the popular library used to encode and
decode the JPEG lossless images in DICOM streams is the PVRG library.
You may also know that this library has a bug for 16 bits images. At
first the bug was not too noticeable since most images were only 12
bits anyway. But since a couple of years, manufacturers started using
16 bits of resolution in their images.
This means that the images encoded with the PVRG library could only be
displayed with viewers using the same library. Using a different
library would either give erroneous results or simply crash the
program altogether. And for the same reason, viewers using PVRG could
not display images that were encoded with other libraries. (Sample of
images having the PVRG bug can be found in David's sample files. In
the JPLL section the MR1, MR3 and NM1 images have the PVRG bug).
I don't want to criticize the PVRG library, apart from that detail,
it's a good library and I was using it myself (I still use it for
lossy images). The problem is that you need to know in advance if the
images you are reading have the bug or not. If you don't, the library
will probably crash while reading the data.
In my programs I've tried to detect and correct this problem. I
thought I had a good heuristic to detect the problem from the first
pixel in the image, but I just received a few images from a Hitachi
scanner that my algorithm failed to detect has having the bug :(
So I finally decided to byte the bullet and last week I did my own
decoder for the lossless JPEG. In keeping with the philosophy of my
applications, I aim for robustness instead of speed. So now, even if
I don't predict the bug at the first try, my application will not
crash and I can then attempt a second reading with different
settings...
This brings me to the "help needed" part:
I need to get the last few bugs out of my new library.
I've tried it with all the Lossless JPEG images I have, but I'm sure
that there are some images out there that will crash it! Among the
possible problems, I'm not sure I treat the "DRI" (Define Restart
Interval) or the "DNL" (Define Number of Lines) correctly, but none of
the images I have use these marker :(
If you could try it out with different images having JPEG lossless
data in an attempt to make it crash, I would appreciate any feedback
and sample of problem images :)
I placed the latest version of TomoVision (my viewer) on my web site
(it's the 1.4 rev-0 beta version). You can downloaf it at:
http://www.tomovision.com/download/tomovision.htm
Merci,
Yves
David Clunie
Do you have a suggested patch for PVRG code that would stop it
generating this bug ?
Do you have a description of your new detection algorithm ?
I guess I should make some new test images, with and without the bug.
David
Mathieu Malaterre
I tried version from zip file: tomo_14_r0.zip
It seg fault on the very first image I tried (to be honest I knew there
would be few chance it will). The DICOM image is
gdcm-JPEG-LossLessThoravision.dcm
You can get it using cvs, follow instruction from:
http://www.creatis.insa-lyon.fr/Public/Gdcm/Cvs.html
On linux I do:
cvs -d :pserver:anon...@cvs.creatis.insa-lyon.fr:/cvs/public co gdcmData
Good luck, this image is tricky :)
Mathieu
Mathieu Malaterre
> Hi Yves
>
> Do you have a suggested patch for PVRG code that would stop it
> generating this bug ?
>
> Do you have a description of your new detection algorithm ?
>
> I guess I should make some new test images, with and without the bug.
So you acknowledge the DICOM image is buggy but those images should
remain 'DICOM' images ? As long as they will be around people will have
to deal/worry with those.
I would really appreciate if -at least- your baseline was exempt from
images that do not respect properly the JPEG compression. Those image
should clearly be mark as either not DICOM, sort of compatible or close
to DICOM specifications.
Or else we need a Transfer Syntax for the buggy PVRG jpeg.
Regards,
Mathieu
Yves Martel
>Hi Yves
>
>Do you have a suggested patch for PVRG code that would stop it
>generating this bug ?
>
Bonjour David,
I must admit I didn't look to closely at the "encoding" side of the
problem in PVRG, I only use the decoder...
In the losslessDecodeDC() function of decode.c, you have the lines:
if ( s== 16 )
return( 32768 )
So if you have a 16 bit encoded word, it will automatically return the
value 32768 instead of computing the actual value from the next 16
bits of data.
Just commenting these lines solve the problem on the decoder side. I
suspect it might be more complex to fix on the encoding side. You do
have a matching line in EncodeDC():
if (S!=16) fputv(s,coef) ;
But I'm not sure if just removing this test will not have
repercussions in the rest of the code...
>Do you have a description of your new detection algorithm ?
>
Basically, PVRG does not seem to be able to encode actual 16 bit
values. As soon as the difference between a value and it's predictor
is greater than 32768, it is encode as 32768 with just the index in
the DC table (=16) without the actual value.
This means that if you read a stream from PVRG with a normal decoder
an you hit on a value encode as 32768 by PVRG, the normal decoder will
read a supplementary 16 bits from the stream. After that, your
lost...
On the other hand, if you read a non-PVRG stream with the PVRG
library, the library will hit the code for 16 bits and return 32768
without reading the next 16 bits. These 16 bits will be read as a new
pixels and again, your lost.
How do you know if the image is PVRG or not?
We do know that the first predictor for a 16 bit image is:
1 << (Frame_P - Scan_Al - 1)
with:
Frame_P = Frame precision = 16
Sacn_Al = point transform = 0 to 15, but usually 0
So the predictor is usually = 32768
So if we want the first pixel to be between 0 to 1000, we will need
the first value to be between -32768 to -31768. Both will be encoded
as 16 bits values, so the very first pixel in a typical 16 bit image
is a 16 bit value!
I read and decode this first pixel using a non-PVRG algo. I then look
at the resulting value. Is it something that make sense in a medical
image context (Is its value between +- 6000 of the predictor)?
But of course, this does not work all the time :(
In the an Hitachi image I have, the values are signed, and the first
pixel is -2048. This encode to 0xF800. To get this value, you only
need to add 7800 to the predictor, and that's a 15 bit value...
So if after my prediction, if the decoder get lost during reading, I
assume I predicted incorrectly and re-try to decode the stream with
the other algorithm.
And even if the decoder did not get lost, but the dynamic range of the
image look suspicious, I try with the different algorithm to make sure
I do not get better results.
>I guess I should make some new test images, with and without the bug.
>
I have a few MR images of the head of my girl friend from a Siemens
scanner. I do beleive they are 16 bits and they do not use PVRG. Do
you want them? (I've ask and she doesn't mind, I'll anonymize them
anyway...)
Does this help?
Merci,
Yves