Non-ascii characters in fits headers

287 views

Skip to first unread message

Kamen Kozarev

unread,

Nov 23, 2016, 7:14:14 AM11/23/16

to SunPy

Hello,

I have come into the problem of astropy.io.fits quitting when I try to read the header of a fits file, if it contains non-ascii characters.

It doesn't crap on loading the file with pyfits directly, but when I try to access an element of the header with non-ascii chars. This also happens when I try to create a map directly. As far as I know people are aware of this, but no fix has been implemented: https://github.com/astropy/astropy/pull/2887

I realize this isn't strictly a Sunpy bug, but I was wondering if people might suggest a workaround...

Here's some code:

from astropy.io import fits as pyfits

hd=pyfits.open(dir+outfile)

index=hd[0].header

hd.close()

In [18]: index['ORIGIN']

---------------------------------------------------------------------------

VerifyError Traceback (most recent call last)

<ipython-input-18-afeb89606855> in <module>()

----> 1 index['ORIGIN']

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/header.pyc in __getitem__(self, key)

155 # the raw value, not the parsed out float value

156 return card.rawvalue

--> 157 return card.value

158

159 def __setitem__(self, key, value):

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/card.pyc in value(self)

542 value = self._value

543 elif self._valuestring is not None or self._image:

--> 544 self._value = self._parse_value()

545 value = self._value

546 else:

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/card.pyc in _parse_value(self)

1012 if m is None:

1013 raise VerifyError("Unparsable card (%s), fix it first with "

-> 1014 ".verify('fix')." % self.keyword)

1015

1016 if m.group('bool') is not None:

VerifyError: Unparsable card (ORIGIN), fix it first with .verify('fix').

In [22]: index

Out[22]:

SIMPLE = T / Written by IDL: Tue Nov 22 18:44:32 2016

BITPIX = -32 /Real*4 (floating point)

NAXIS = 2 /

NAXIS1 = 1352 /

NAXIS2 = 1353 /

WCSNAME = 'Helioprojective-Cartesian' /

CRPIX1 = 676.500000000 /

CRVAL1 = 1970.00000000 /

CTYPE1 = 'SOLARX ' /

CUNIT1 = 'arcsec ' /

CDELT1 = 20.0000000000 /

CRPIX2 = 677.000000000 /

CRVAL2 = -860.000000000 /

CTYPE2 = 'SOLARY ' /

CUNIT2 = 'arcsec ' /

LVL_NUM = 1 /

R_SUN = 48.3404458235 /

WAVEUNIT= 'm ' /

WAVELNTH= 3.78717100730 /

RESTWAVE= 3.78717100730 /

FRQUNIT = 'Hz ' /

DELTAFRQ= 280000.004025 /

RESTFRQ = 79160000.2805 /

OBSID = '1130642640' /

CHANNEL = '62-62 ' /

DSK_3SIG= 0.00000000000 /

DSK_FRAC= 'NAN ' /

DSK_RMS = 'NAN ' /

DSK_SNR = 'NAN ' /

SNR = 'NAN ' /

RMS = 'NAN ' /

DATARMS = 'NAN ' /

DATATOT = 'NAN ' /

DATAMEDN= -0.00254863000000 /

DATAMEAN= 'NAN ' /

DATAMAX = 'NAN ' /

DATAMIN = 'NAN ' /

EQUINOX = 2000.00000000 /

BUNIT = 'Jy/beam ' /

BZERO = 0.00000000000 /

BSCALE = 1.00000000000 /

CTYPE3 = 'FREQ ' /

CRPIX3 = 1.00000000000 /

CRVAL3 = 79160000.2805 /

CDELT3 = 280000.004025 /

CUNIT3 = 'Hz ' /

CTYPE4 = 'STOKES ' /

CRPIX4 = 1.00000000000 /

CRVAL4 = -5.00000000000 /

CDELT4 = 1.00000000000 /

CUNIT4 = ' ' /

SPECSYS = 'TOPOCENT' /

CDELT2 = 20.0000000000 /

CROTA2 = 0.00000000000 /

DATE-OBS= '2015-11-04T03:24:31.000' /

DSUN_OBS= 148382214544. /

SOLAR_B0= 4.10936099847 /

HGLN_OBS= 0.00000000000 /

HGLT_OBS= 4.10936099847 /

XCEN = 1970.00000000 /

YCEN = -860.000000000 /

BMAJ = 0.130909686618 /

BMIN = 0.0966836632623 /

BPA = 13.7469234467 /

BTYPE = 'Intensity' /

OBJECT = 'Sun ' /

PC01_01 = 1.00000000000 /

PC02_01 = 0.00000000000 /

PC03_01 = 0.00000000000 /

PC04_01 = 0.00000000000 /

PC01_02 = 0.00000000000 /

PC02_02 = 1.00000000000 /

PC03_02 = 0.00000000000 /

PC04_02 = 0.00000000000 /

PC01_03 = 0.00000000000 /

PC02_03 = 0.00000000000 /

PC03_03 = 1.00000000000 /

PC04_03 = 0.00000000000 /

PC01_04 = 0.00000000000 /

PC02_04 = 0.00000000000 /

PC03_04 = 0.00000000000 /

PC04_04 = 1.00000000000 /

ALTRVAL = -8247.30406530 /

ALTRPIX = 1.00000000000 /

VELREF = 259 /

TELESCOP= 'MWA ' /

OBSERVER= 'tfranzen' /

TIMESYS = 'UTC ' /

OBSRA = 218.123471436 /

OBSDEC = -15.1774288353 /

DATE = '2016-10-18T12:43:13.061000' /

ORIGIN = 'CASA 4.7.0-REL (r38335)

' /

DATE_OBS= '2015-11-04T03:24:31.000' /

COMMENT casacore non-standard usage: 4 LSD, 5 GEO, 6 SOU, 7 GAL

COMMENT FITSHEAD2STRUCT

HISTORY MWA_PREP.PRO V0.2 run at: 22-Nov-2016 18:44:32

HISTORY FITSHEAD2STRUCT run at: Tue Nov 22 18:44:32 2016

In [37]: index[-6].decode('utf-8').encode('ascii', 'ignore')

---------------------------------------------------------------------------

VerifyError Traceback (most recent call last)

<ipython-input-37-a498a88d2da2> in <module>()

----> 1 index[-6].decode('utf-8').encode('ascii', 'ignore')

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/header.pyc in __getitem__(self, key)

155 # the raw value, not the parsed out float value

156 return card.rawvalue

--> 157 return card.value

158

159 def __setitem__(self, key, value):

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/card.pyc in value(self)

542 value = self._value

543 elif self._valuestring is not None or self._image:

--> 544 self._value = self._parse_value()

545 value = self._value

546 else:

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/card.pyc in _parse_value(self)

1012 if m is None:

1013 raise VerifyError("Unparsable card (%s), fix it first with "

-> 1014 ".verify('fix')." % self.keyword)

1015

1016 if m.group('bool') is not None:

VerifyError: Unparsable card (ORIGIN), fix it first with .verify('fix').

In [38]: index[-6].verify('fix').decode('utf-8').encode('ascii', 'ignore')

---------------------------------------------------------------------------

VerifyError Traceback (most recent call last)

<ipython-input-38-69786e7e061a> in <module>()

----> 1 index[-6].verify('fix').decode('utf-8').encode('ascii', 'ignore')

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/header.pyc in __getitem__(self, key)

155 # the raw value, not the parsed out float value

156 return card.rawvalue

--> 157 return card.value

158

159 def __setitem__(self, key, value):

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/card.pyc in value(self)

542 value = self._value

543 elif self._valuestring is not None or self._image:

--> 544 self._value = self._parse_value()

545 value = self._value

546 else:

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/card.pyc in _parse_value(self)

1012 if m is None:

1013 raise VerifyError("Unparsable card (%s), fix it first with "

-> 1014 ".verify('fix')." % self.keyword)

1015

1016 if m.group('bool') is not None:

VerifyError: Unparsable card (ORIGIN), fix it first with .verify('fix').

In [40]: mwamap=sunpy.map.Map(dir+outfile)

---------------------------------------------------------------------------

ValueError Traceback (most recent call last)

<ipython-input-40-709c619181b0> in <module>()

----> 1 mwamap=sunpy.map.Map(dir+outfile)

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/sunpy/map/map_factory.py in __call__(self, *args, **kwargs)

259 silence_errors = kwargs.pop('silence_errors', False)

260

--> 261 data_header_pairs, already_maps = self._parse_args(*args, **kwargs)

262

263 new_maps = list()

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/sunpy/map/map_factory.py in _parse_args(self, *args, **kwargs)

183 os.path.isfile(os.path.expanduser(arg))):

184 path = os.path.expanduser(arg)

--> 185 pairs = self._read_file(path, **kwargs)

186 data_header_pairs += pairs

187

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/sunpy/map/map_factory.py in _read_file(self, fname, **kwargs)

103 # File gets read here. This needs to be generic enough to seamlessly

104 #call a fits file or a jpeg2k file, etc

--> 105 pairs = read_file(fname, **kwargs)

106

107 new_pairs = []

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/sunpy/io/file_tools.py in read_file(filepath, filetype, **kwargs)

78 for extension, readername in _known_extensions.items():

79 if filepath.endswith(extension) or filetype in extension:

---> 80 return _readers[readername].read(filepath, **kwargs)

82 # If filetype is not apparent from extension, attempt to detect

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/sunpy/io/fits.py in read(filepath, hdus, memmap, **kwargs)

77 hdulist = [hdulist[i] for i in hdus]

78 try:

---> 79 hdulist.verify('silentfix+warn')

81 headers = get_header(hdulist)

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/verify.pyc in verify(self, option)

72 return

---> 74 errs = self._verify(opt)

76 # Break the verify option into separate options related to reporting of

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.pyc in _verify(self, option)

923

924 else:

--> 925 result = hdu._verify(option)

926 if result:

927 errs.append(result)

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/hdu/image.pyc in _verify(self, option)

1019

1020 def _verify(self, option='warn'):

-> 1021 errs = super(PrimaryHDU, self)._verify(option=option)

1022

1023 # Verify location and value of mandatory keywords.

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/hdu/image.pyc in _verify(self, option)

563 self._verify_blank()

564

--> 565 return super(_ImageBaseHDU, self)._verify(option)

566

567 def _verify_blank(self):

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/hdu/base.pyc in _verify(self, option)

1210 # verify each card

1211 for card in self._header.cards:

-> 1212 errs.append(card._verify(option))

1213

1214 return errs

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/card.pyc in _verify(self, option)

1390 'string: %r).' % (self.keyword, valuecomment),

1391 fix_text=fix_text,

-> 1392 fix=self._fix_value))

1393

1394 # verify the comment (string), it is never fixable

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/verify.pyc in run_option(self, option, err_text, fix_text, fix, fixable)

46 else:

47 if fix:

---> 48 fix()

49 text += ' ' + fix_text

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/card.pyc in _fix_value(self)

1129 self.comment = comment.strip()

1130 except (ValueError, IndexError):

-> 1131 self.value = valuecomment

1132 self._valuestring = self._value

1133 return

/Users/kkozarev/anaconda2/lib/python2.7/site-packages/astropy/io/fits/card.pyc in value(self, value)

601 'FITS header values must contain standard printable ASCII '

602 'characters; %r contains characters/bytes that do not '

--> 603 'represent printable characters in ASCII.' % value)

604 elif isinstance(value, np.bool_):

605 value = bool(value)

ValueError: FITS header values must contain standard printable ASCII characters; "'CASA 4.7.0-REL (r38335)\n' /" contains characters/bytes that do not represent printable characters in ASCII.

Reply all

Reply to author

Forward

0 new messages