Saving table to FITS and missing value

[Yannick Roehlly]

Dear all,

I noticed that saving a table has different behaviours concerning missing
values according to the format of the saved file. For VO-table, a missing
value results in an empty cell. For FITS, it is replaced by the "fill value"
(default to 1.e+20 for floats apparently).

Is this a requirement of the FITS format ?

Yannick

[Thomas Robitaille]

Hi Yannick,

FITS has no good way of representing missing values for non-integer
columns, so at the moment we just return the filled value of the
underlying Numpy masked array. The default fill value set in Numpy is
1.e20 but we could consider changing it to NaN if needed. NaN isn't
strictly a missing value, but it might be better than 1e20?

VO tables have better support for missing values, hence why we return
empty cells.

Cheers,
Tom

> --
> You received this message because you are subscribed to the Google Groups "astropy-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to astropy-dev...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Mark Taylor]

The FITS standard (sec 6.3) says:

The BLANK keyword should not be used when BITPIX = -32 or -64;
rather, the IEEE NaN should be used to represent an undefined value.

which I read to mean that the "missing value" functionality FITS
provides for integers should be replaced with IEEE NaNs when you're
dealing with floats. It is true that NaN is not strictly the same
as missing, but it would seem like a better choice here than (e.g.) 1e20.

We've had detailed conversations in the IVOA in relation to VOTable
about the relation between missing and NaN values; I don't want to
rehearse them here, but the headline conclusion seems to be that
most (though not all) astronomy applications don't care too much
about the distinction.

When I (STIL/TOPCAT) translate between VOTable and FITS table.
I basically represent null/missing floating point values as NaN.
Very few people have complained about this behaviour.

Mark

--
Mark Taylor Astronomical Programmer Physics, Bristol University, UK
m.b.t...@bris.ac.uk +44-117-9288776 http://www.star.bris.ac.uk/~mbt/

[Thomas Robitaille]

I have opened an issue for this:

https://github.com/astropy/astropy/issues/2186

One question is whether we should fix this for FITS output only, or
whether this should be fixed in astropy.table so that masked float
columns always have a fill value of NaN by default. Maybe for now we
should just do it for FITS output, though at that point it's not easy
to know if the fill value is the default one or was specified by the
user - unless we assume a value of 1.e20 always means the user left
the default.

Cheers,
Tom

[Yannick Roehlly]

Salut Thomas,

Le lundi 10 mars 2014, 14:36:36 Thomas Robitaille a écrit :
> The default fill value set in Numpy is
> 1.e20 but we could consider changing it to NaN if needed. NaN isn't
> strictly a missing value, but it might be better than 1e20?

I don't think there is a best default behaviour. We could also choose -99 for
the missing value but there are some case where it won't be good.

I think that the best is to make users aware of the problem when saving to
FITS; maybe with a warning like "Fill value was used for missing value" (we
can't even tell which fill value as the user can give some per column fill
value).

Cheers,

Yannick