Attempt at numpy/Numeric support
Chris
I've uploaded a tarball with a numerix layer for supporting both
Numeric and numpy. I haven't really tested it. The usage is
transparent to the end user. If numpy is available it is used. If
not then Numeric is used. If neither is available an exception is
raised. Once you have some test cases available I can debug any
problems you might have.
Goodnight,
Chris
jeremy....@gmail.com
Sorry I'm a little late in replying... while looking over your patch I
realized how fragile the solution I suggested was (mostly because
fitsimage uses many more Numeric functions than I remembered). This
is sort of annoying because, for example, if we add a sqrt scaling
then we'd have to modify the numpy and Numeric modules. I thought
about having a smart module or class that would forward on functions
it didn't understand to the proper array module, but this is also not
so great. In addition, I would prefer to define this code inside of
fitsimage/fitslib to reduce the number of modules we expose to other
progammers. How does pyfits handle this (or do they require numpy
now)? I might be persuaded to drop Numeric support if pyfits already
requires numpy.
Speaking of pyfits, how much work do you think removing the blind
except clauses would be (except: pass)? I wouldn't mind switching to
pyfits if it raised exceptions for parse errors (stating the file
isn't FITS or that it's corrupted) and cleaned up the loose exception
handling (the performance issues could be dealt with later). For me,
it's very important that something like fits2png.py give very specific
and reasonable errors for users. If you open a non-FITS file in
pyfits currently and read the first header, you get an error that
looks like:
Warning: Required keywords missing when trying to read HDU #1.
There may be extra bytes after the last HDU or the file is
corrupted.
Traceback (most recent call last):
File "./pyfits_open.py", line 17, in ?
main(sys.argv)
File "./pyfits_open.py", line 13, in main
header = fits[0].header
File "/usr/lib/python2.4/site-packages/pyfits/NP_pyfits.py", line
4619, in __getitem__
_item = super(HDUList, self).__getitem__(key)
IndexError: list index out of range
Actually, testing fitslib shows that it raises an EOFError for most
files because it's not 2880 byte aligned which isn't much better...
the header parser definitely checks for the SIMPLE keyword and raises
a ValueError (I think) if not found saying it's not a FITS file, but
this is after the HDU parser looks through the file. Anyway, the
point is that this error should clearly state "This is not a FITS
file" or "This appears to be a corrupt FITS file" somewhere.
So, how hard would getting those 2 changes (remove blind excepts and
raising exceptions for parse errors) back into pyfits?
It will probably be Friday before I get around to working on some test
cases, but feel free to jump in. I'm just going to make a 20x20 FITS
image and set the values according to a simple, repeatable function,
then test data and header output.
Jeremy
Chris
I guess I undersold it when I mentioned that you might not want to
complicate your code supporting both numpy and Numeric. I actually
came up with a clever way of support both numarray and numpy that only
required you to overload the numarray syntax when needed. Basically
you would import one array package or the other into the same
namespace. In the case of numarray you would subclass the ndarray
object and use properties to simulate the numpy syntax when
necessary. However, this solution has problems as soon as you pass
your array to 3rd party libraries. In one case you would have
numpy.ndarrays and the other you would have your subclass.ndarray
object. You could turn around and make that work by looking at the
context frames and doing conversions through the array interface when
necessary. Unfortunately this solution broke down as soon as someone
imported both array packages during interactive use.
PyFITS currently supports both numarray and numpy. This was achieved
by branching the code. We ended up making completely separate numarray
and numpy branches. The '__init__.py' file in the pyfits module is
used to decide which version to import. This branching was necessary
because of how tightly coupled pyfits was to the numarray internals.
However, we have announced that we will no longer make enhancements or
bug fixes to the numarray version. We have also transitioned all of
our other Python applications from numarray to numpy. None of our
code requires numarray any longer and no STScI people are supporting
numarray development.
I will need to talk to Perry and the other pyfits developers about the
changes you propose but I don't see it being a major problem. I would
need to take a closer look at the code to see how much work would
actually be involved. I don't see it being a major project.
Chris
Jeremy Brewer
First, I think we'll just transition to numpy by itself as it's less
work and a cleaner design.
I haven't made any progress on a Python FITS test since I've been
working on getting the C++ unit tests in place, but there will be some
test data checked in probably tomorrow (but C++ test oriented). Since
I'm really short on time this week, let me just ask what is currently
holding you back? If I can get one thing done this week, what should
it be?
If you need something to do, here's how I think we should handle the
pyfits transition:
- Write a function in fitslib called fits_simple_verify that takes a
file as input and does 2 things:
1. Read the first 6 characters and verify that they are SIMPLE,
otherwise raise a ValueError indicating it's not a FITS file
2. Check that the file is 2880 byte aligned and raise a ValueError
saying the FITS file is corrupted otherwise
- Leave the rest of fitslib in place so we can fix scipts as we go
without breaking anything
- Replace fitslib.Fits with pyfits.open and call the function above
before every instance of pyfits.open so that we get nice error
messages
An alternative approach would be to change the Fits class so that it
uses the pyfits code + mixins or inheritance to achieve the same API
as my Fits class but using pyfits underneath. But since pyfits should
add my extra methods soon I don't think this is needed.
The porting should be really straightfoward with 2 very minor
exceptions:
1. I probably use the get() Header method somewhere
2. I use the parse_card() function in wcs2fits.py, but you can just
replace the Fits class and still use fitslib for this
Is there anything you'd like to see done on the C++ side of things?
Jeremy
Chris
Don't worry, I never need anything to do. This is just a night time
project for me. I'm actually working on image distortion solutions in
my "real" job.
I can handle the numpy conversion for you. I'll work on some tests as
the same time. Do you prefer unittest, doctest, nose, or py.test? Do
you want me to check in the unittests somewhere or just upload them to
the group page?
I'm inclined to work on your alternative suggestion for adding pyfits
support and just add the native pyfits api under the hood in your Fits
class. This will help me identify additional features that will need
to be added to pyfits itself. I think that is the better interim step
for both projects.
I haven't looked much at the C++ code other than to get it working so
I have no comments right now.
Chris
Jeremy Brewer
one included in the standard library. Looking over the documentation
on the Python site, I'd say that doctest looks really cool since I'm a
big fan of including example usage code in documentation. If you feel
strongly about a certain testing framework, feel free to convince me
it's the greatest thing ever.
Going with the alternate approach of making fitslib use pyfits is
fine, but I would really like that simple verification function.
As for where to submit patches, for now just upload it to the group
discussion page. If you keep making patches then I'll make you an
actual member so you can checkout / commit on your own.
Jeremy