Overlay in DataSet CompressedSamples^RG3
Mathieu Malaterre
Since the early beginning of GDCM we have always been 'cleaning up'
the unused bits of the Pixel Data just in case an overlay would be
found.
This is extremely time consuming, and I was running some test to
check if I could simply run the cleanup only when a 0x60xx element was
to be found. It works on all but one gdcmData files we have been
gathering.
the only exception is the Dataset CompressedSamples^RG3, from the
compressed jpeg in WG04, see file
ftp://medical.nema.org/medical/dicom/DataSets/WG04/compsamples_jpeg.tar
-> IMAGES/JPLY/RG3_JPLY
(1.3.6.1.4.1.5962.1.1.11.1.5.20040826185059.5457)
Clearly the DataSet does not declare any Overlay, but still one
cannot simply load the dataset without cleaning up the unused bit.
When loading the image, the IJG (the jpeg lib used in GDCM) is
complaining about
"Invalid SOS parameters for sequential JPEG"
I am wondering if the artefact I am seeing is not simply due to a
poor jpeg encoding in which case I could detect that instead of
looping over the whole image in quest for a potential artefact.
* TomoVision is apparently using the PVRG jpeg lib and has the exact
same artefact as we do (a bright spot in the center right image).
* DicomWorks is not properly displaying the image, but does not show
the artefact.
* IrfanView is simply crashing.
Thanks for comments
-Mathieu
Yves Martel at
Yop, TomoVision use PVRG for lossy JPEG decoding. (I had to write my
own decoder for lossless JPEG because of the bugs found in some 16
bits images... Maybe you should add some of these to your "hall of
shame"?)
Indeed this is a compression problem, the pixel in question is
decompressed as "1024", and this is an overflow. The structure is
unsigned with 10 bits, so the largest valid value should be 1023. If
you remove the "overlay" bits, the pixel falls to "0'.
I did patch this some time ago and I now clip the JPEG lossy
compression results to the max values. Are you using an older version
of TomoVision?
And I also "clean up" any "unused" bits. I don't think you can use
the trick with the 0x60xx elements since I did run acros some older
images that had stuff in the higher bits even without the 0x60xx tags
being present.
Yves
Mathieu Malaterre
On Jan 4, 6:51 pm, Yves Martel <Martel(at)TomoVision.com> wrote:
> Bonjour Mathieu,
>
> Yop, TomoVision use PVRG for lossy JPEG decoding. (I had to write my
> own decoder for lossless JPEG because of the bugs found in some 16
> bits images... Maybe you should add some of these to your "hall of
> shame"?)
Always looking for more sample datasets. Please send me if you have
one laying around, thanks !
> Indeed this is a compression problem, the pixel in question is
> decompressed as "1024", and this is an overflow. The structure is
> unsigned with 10 bits, so the largest valid value should be 1023. If
> you remove the "overlay" bits, the pixel falls to "0'.
>
> I did patch this some time ago and I now clip the JPEG lossy
> compression results to the max values. Are you using an older version
> of TomoVision?
No, I am using the latest version (with support for segmented
palette). You can still clearly see a bright spot (value for invalid
pixel is 0, but image is MONOCHROME1 so pixel appears as very bright).
> And I also "clean up" any "unused" bits. I don't think you can use
> the trick with the 0x60xx elements since I did run acros some older
> images that had stuff in the higher bits even without the 0x60xx tags
> being present.
Any sample for my hall of shame ? Thanks.
Anyway, I'll go ahead and remove the cleanup for now. We have some
other QC downstream, so if we detect too much of those I'll turn it
back on and leave it as an option.
Thanks,
-Mathieu
> On Fri, 4 Jan 2008 06:15:05 -0800 (PST), Mathieu Malaterre
>
Yves Martel at
Oups! You are right, I had a "clip_to_range" function after the
Jasper decoder (I use the Jasper library for JPEG-2000) because it
also has the same problem, but I did not have the function after the
PVRG library (I guess I never noticed the problem before...)
Merci,
Yves
P.S. I'll send you a Siemens image with the 16 bit bug, but I do not
remember where I saw the "overlay" bits used... I think it was in an
old SPI image, but I can't be sure...
Yves
Jean-Pierre Roux
> Bonjour Mathieu,
>
> Oups! You are right, I had a "clip_to_range" function after the
> Jasper decoder (I use the Jasper library for JPEG-2000) because it
> also has the same problem, but I did not have the function after the
> PVRG library (I guess I never noticed the problem before...)
>
> Merci,
> Yves
>
> P.S. I'll send you a Siemens image with the 16 bit bug, but I do not
> remember where I saw the "overlay" bits used... I think it was in an
> old SPI image, but I can't be sure...
Bonjour, Yves, Mathieu
I had troubles with 'unused bits' and no overlays on old MR Siemens
ACR-NEMA images ... more than 10 years ago.
I didn't keep any sample of this images (and I was wrong!)
If your 'old SPI images' are ACR-NEMA, and not Dicom, I think it will be
easy for time-consuming trackers to check the type, and clean up the
unused bits only when necessary (ACR-NEMA or DICOM+JpegLossy) ?
Jean-Pierree Roux
Yves Martel at
Yes, SPI is based either NEMA-1 or NEMA-2 (depending on the revision
of the software). It was used by Siemens (and Philips) in the
pre-DICOM days.
I guess you are right, but I really don't see where cleaning up the
unused bits take that much time... After all it is only 2 shift
operations, so for a 512x512 image, it should not take more than a
fraction of a second.
Personally, it is par of my pixel querry function:
...
case DATA_TYPE_SHORT :
// - get rid of the overlays and do the sign extension -
ss_val = ((signed short *) data)[k] ;
ss_val <<= 15-ima->g2.pix_high_bit ;
ss_val >>= 15-ima->g2.pix_high_bit ;
break ;
case DATA_TYPE_USHORT :
// --- get rid of the overlays ---
us_val = ((unsigned short *) data)[k] ;
us_val <<= 15-ima->g2.pix_high_bit ;
us_val >>= 15-ima->g2.pix_high_bit ;
break ;
...
Yves
Mathieu Malaterre
See my comments interlaced.
On Jan 9, 3:27 am, Yves Martel <Martel(at)TomoVision.com> wrote:
> Bonjour Jean-Pierre (and the others),
>
> Yes, SPI is based either NEMA-1 or NEMA-2 (depending on the revision
> of the software). It was used by Siemens (and Philips) in the
> pre-DICOM days.
>
> I guess you are right, but I really don't see where cleaning up the
> unused bits take that much time... After all it is only 2 shift
> operations, so for a 512x512 image, it should not take more than a
> fraction of a second.
Completely agree, well the issue is that we are not dealing with only
one image. For particular protocol we have to load a 4D dataset:
256x256x24x99, and 6 3D dataset: 256x256x24. Now add some derived
images on top of that for vizualization. Time lost during data loading
is money lost.
...
-Mathieu