request help reading a fits file

[Fred Walter]

Hi all,

I've run across a type of fits file I've never seen before. Perhaps one of you can help me with it.

It is a fits binary table file containing variable length compressed data. I can read it using mrdfits() as follows:

IDL>a=mrdfits(file,1,h)  
MRDFITS: Binary table.  4 columns by  8000 rows.
MRDFITS: Uses variable length arrays
MRDFITS:  2 length column[s] added
IDL>help,a,/st
** Structure <26cf698>, 6 tags, length=8952, data length=8948, refs=1:
   COMPRESSED_DATA BYTE      Array[8540]
   L0_COMPRESSED_DATA                  LONG                 0
   GZIP_COMPRESSED_DATA                   BYTE      Array[384]
   L1_GZIP_COMPRESSED_DATA                   LONG               384
   ZSCALE          DOUBLE           0.0000000
   ZZERO           DOUBLE           0.0000000

The header starts out:

BITPIX  =                    8 / array data type                                
NAXIS   =                    2 / number of array dimensions                    
NAXIS1  =                   32 / width of table in bytes                        
NAXIS2  =                 8000 / number of rows in table                        
PCOUNT  =             53205145 / number of group parameters                    
GCOUNT  =                    1 / number of groups                              
TFIELDS =                    4 / number of fields in each row                  
TTYPE1  = 'COMPRESSED_DATA'    / label for field 1                              
TFORM1  = '1PB(8540)'          / data format of field: variable length array    
TTYPE2  = 'GZIP_COMPRESSED_DATA' / label for field 2                            
TFORM2  = '1PB(384)'           / data format of field: variable length array    
TTYPE3  = 'ZSCALE  '           / label for field 3                              
TFORM3  = '1D      '           / data format of field: 8-byte DOUBLE            
TTYPE4  = 'ZZERO   '           / label for field 4                              
TFORM4  = '1D      '           / data format of field: 8-byte DOUBLE            
ZIMAGE  =                    T / extension contains compressed image            
ZSIMPLE =                    T / conforms to FITS standard                      
ZTENSION= 'IMAGE   '           / binary table extension                         

ZBITPIX =                  -64 / array data type                                
ZNAXIS  =                    2 / number of array dimensions                    
ZNAXIS1 =                10000 / length of original image axis                  
ZNAXIS2 =                 8000 / length of original image axis                  
ZPCOUNT =                    0 / number of group parameters                    
ZGCOUNT =                    1 / number of groups                              
ZTILE1  =                10000 / size of tiles to be compressed                
ZTILE2  =                    1 / size of tiles to be compressed                
ZCMPTYPE= 'RICE_1  '           / compression algorithm                          
ZNAME1  = 'BLOCKSIZE'          / compression block size                        
ZVAL1   =                   32 / pixels per block        

...

My question is: how do I deal with the COMPRESSED_DATA tag? I expect it to be a 10000x8000 floating point array. 

Thanks,

Fred

[Fred Walter]

To clarify, the 8000 rows are in the 8000 elements of the structure. My problem is not knowing how to uncompress the data. Running MRDFITS with the /pointer keyword gives me pointers to the compressed data. 

Thanks.

[Wayne Landsman]

Fred,

        This file is compressed using Rice compression, which has several advantages over, say, using gzip for compression.      The HEASARC group has a 'funpack' executable available

https://heasarc.gsfc.nasa.gov/fitsio/fpack/ 

for decompressing the FITS image, and mrdfits.pro is supposed to spawn to this command if it encounters a Rice compressed file.

      You need to first have funpack installed, so that 'funpack' is recognized at the command line.      The HEASARC group used to make available a standalone funpack binary, but now I believe that you have to download the entire CFITSIO library.   (But you only have to do this once).

--Wayne     

On Thursday, February 10, 2022 at 11:39:59 PM UTC-5 Fred Walter wrote: