Creating a Debian package

[Ole Streicher]

Hi,

I am planning to create a Debian package for astrometry.net. The Ubuntu package that is available is a bit outdated yet (0.46), and for Debian it needs some substantial polishing.

However, I have some questions, and I would like to discuss the changes that I am going to make for the package

When I do the default installation (with extras), I get a shared library, a few static libraries and header files. However, the binaries seem to be linked statically. So, should the libastrometry.so file distributed Are (or will) there be any external programs that use this library? Does this library have a somehow fixed and defined API? If yes, I would like to attach a SONAME to it (libastrometry.so.1.0.0 or so), and distribute it as an extra package libastrometry1. The header files and the static libs could be packaged into a package libastrometry-dev if there is the chance of the development of external programs that use them.

The package contains quite many executables, where some of them seem to be duplicating other programs -- fitscopy is already distributed with cfitsio, for example. Do you have a list of the programs that come from somewhere else? I would leave them out and replace them with dependencies to the original packages.

Are all executables meant to be run by the end user (and therefore should be put into the PATH)? Or are some of them just used by other programs? Then I could I move them to another directory (/usr/lib/astrometry.net/bin/) and just ensure that the correct PATH is set before they are called.

For all executables that are to be called by the user, I would need a Unix manpage. I guess that you don't have them yet, right? Then, I would create them myself and create a pull request on github so that they can be included there. The other solution here would be to change the programs themself so that they produce a help output that can be parsed with help2man. That is basically consistently adding --help and --version options to all executables, and making the executables more verbose about what they actually do: Many of them have a short description in doc/readme.rst, but there are some which don't have any description: astrometry-engine or fit-wcs are examples here. This work has to be done only for the user callable executables, however (that's why I am looking to keep their number small).

The package contains also a Python package, which I would put into an extra Debian package python-astrometry. However, for this I would need to create the packages for all supported Python versions (2.6, 2.7). Does it work for Python3 as well? There are also some Python scripts in /usr/bin/; are they just for convienience? In that case, I would remove them, since otherwise, I would again need a manpage for them (and they should provide some help output...).

Last point would be the index files: they are going to be quite large, and in the moment, Debian is not well suited yet to handle >>1GB data files (they are mirrored all over the world, which makes no sense for a package that is of interest for just a few percent of the users) -- we have, however, an ongoing discussion about this. Until we have a good solution here, I would package only the smaller files -- (4208-4219; ~120 MB). Would this make sense?

Finally, I need to make some changes in the directory structure, which has to follow the FHS:

• Configuration goes to /etc/astrometry.cfg
• Examples go to /usr/share/doc/astrometry.net/examples/
• Index data go to /usr/share/astrometry.net/data/
• /usr/ups/ will be also moved to /usr/share/astrometry.net/ups/
• The static and shared libs (if installed; see above) goes to /usr/lib/x86_64-linux-gnu/ to make the package multi-arch aware,
• The Python package goes to /usr/lib/python2.6/ and /usr/lib/python2.7/ (and similar for Python3, if the package is usable for that).

Does this make sense to you?

To summarize the package structure: I will create the following packages:

• astrometry.net,
• libastrometry1 (if useful, see above),
  
• libastrometry-dev (if useful, see above),
  
• python-astrometry,
• python3-astrometry  (if possible),
  
• astrometry-data-2mass (created independently).
  

Sorry for the lengths mail; however astrometry.net is not really easy to package.

Best regards

Ole

[Dustin Lang]

Hi Ole,

Thanks for your email.  It would be great to have it available in Debian!

I do not know of any program that use the libastrometry library, and as you saw, the Astrometry.net programs are statically linked.

Indeed, 'fitscopy' and a few other programs come from cfitsio.  They are distributed in cfitsio, but not built and installed by default (at least, that was the case in the past; this may have changed?).  I agree that these should just be dependencies, and that makes sense for a Debian install.

(That list of programs: tablist modhead fitscopy tabmerge liststruc listhead imcopy imarith imstat)

The main program to be run by users is 'solve-field'.  Basically everything else is either run by that, or "convenience" programs that could be useful for some people...

* an-fitstopnm, an-pnmtofits -- PNM <-> FITS image conversion.  Netpbm has this, but it was broken in some popular release, resulting in many bug reports, so I use these instead

* astrometry-engine -- this does the actual work of identifying where the list of stars is on the sky -- solve-field runs it.  One could imagine a situation where a user want to run this directly, but it wouldn't be common.

* deg2hms.py -- Hours:Minutes:Seconds to decimal degrees.  Handy script.
* hmstodeg.py -- convert the other way.  Nice inconsistent naming.
* get-healpix -- "HEALPix" projection to RA,Dec; obscurely handy.

* downsample-fits -- scale down a FITS image.  Handy.

* fit-wcs -- given a list of x,y coordinates and matching RA,Dec coordinates, fit a transformation and produce a WCS file.  Again, useful for some users.

* fits-column-merge, fits-flip-endian, fitsgetext -- Handy utils for working with FITS tables & images.
* merge-columns.py -- redundant with fits-column-merge?  Or more flexible?

* fits2fits.py -- "Clean" FITS files.  I think this is pretty much broken at the moment, but useful in principle; called by solve-field.

* hpsplit, build-astrometry-index -- used for *building* index files that will later be used by solve-field.  Used by some people.

* image2pnm.py -- convert images in various formats to PNM; called by solve-field

* image2xy -- convert FITS image to list of detected stars; possibly useful for some users / workflows.

* new-wcs -- add a WCS header to an image file.  Handy util.

* pad-file -- pad a FITS file out to the correct size.  Handy-ish util.

* plot-constellations, plotxy, plotquad -- draw annotations on astronomical images, called by solve-field

* plotann.py -- draw annotations on astronomical images, better than plot-constellations, used by some people.

* query-starkd -- search index files for stars near a given point on the sky; handy util.

* removelines.py -- find & remove lines of stars from images; called by solve-field

* tabsort -- sort a FITS table by a given column name; handy util.

* text2fits.py -- convert CSV and other files to FITS tables; handy util.

* uniformize.py -- select a spatially uniform subset of stars; called by solve-filed

* wcs-pv2sip -- convert TPV WCS headers to SIP distortion headers; handy util.

* wcs-rd2xy, wcs-xy2rd -- convert between RA,Dec and x,y positions via WCS headers; handy util.



You are correct that we have no man pages.

The python code is not (yet!) python3 compatible.

You can drop the 'ups' directory; this was for the convenience of a former employer :)

Your proposal for handling the index files seems reasonable; the only concern I would have is that then "out of the box" it only works for a small number of images.  If there is some easy way of indicating that the user probably wants to download more index files, that would be nice.


Thanks again for your note and effort here, and apologies for the mess.  This is ten years of accumulated research code :)  And if we knew what we were doing when we started, it wouldn't be "research"!

cheers,
--dustin

[Ole Streicher]

Hi Dustin,

thank you for integrating the manpages into your source. This will make
future Debian releases simpler, since I don't have to add them myself.

Here are a few more thoughts while creating the Debian package. The major
problem I have now are some copyright glitches: First, many files are GPL-2
only (see util/bl-sort.c as an example), which does conflict with the GPL-3
of gsl-an/*. Could you explicitely release your files under GPL-2+ ("version
2 or later")? This would need to put this statement into the header files. I
can prepare a pull request for that if you like.

Then, the ngc2000 catalogs are not free in the Debian sense [1], since they
are for non-commercial use only. Are they required to run astrometry.net?
Otherwise, I would like just to drop them from the Debian package. If they
are really needed I would have to create another, non-free package with
these catalogues.

Am 27.07.2015 um 20:15 schrieb Dustin Lang:
> I do not know of any program that use the libastrometry library, and as you
> saw, the Astrometry.net programs are statically linked.

In the google archive, I found one or two people who use the shared lib, so
it may worth to provide the lib and the headers. Is there a reason that you
link statically instead of using the shared library?

And how stable is the API of the library?

> Indeed, 'fitscopy' and a few other programs come from cfitsio. They are
> distributed in cfitsio, but not built and installed by default (at least,
> that was the case in the past; this may have changed?). I agree that these
> should just be dependencies, and that makes sense for a Debian install.

I contacted the maintainer of cfitsio, and it seems that most of these
programs are not available anymore in cfitsio [2]. So, they will kept from
your codebase: tablist, modhead, liststruc, imarith, imstat, listhead,
tabmerge. I will just remove fitscopy and imcopy.

> Your proposal for handling the index files seems reasonable; the only
> concern I would have is that then "out of the box" it only works for a small
> number of images. If there is some easy way of indicating that the user
> probably wants to download more index files, that would be nice.

I would provide a README describing where to get the files. Another way
would be to create "downloader" packages for the remaining. However, this
would somehow require that the files and locations will remain stable over
the next few years.

Best regards

Ole

[1] https://www.debian.org/social_contract#guidelines
[2] https://bugs.debian.org/793851

[Dustin Lang]

Well, whoops.  We originally imported GSL when it was under GPL v2 (gsl-1.9).  We then updated to gsl-1.11, 1.14, and 1.16, which are under GPL v3.  That was an oversight.

Personally, I deeply dislike the "or any later version" option.  Who knows what nutbar thing the FSF might do in the future?  It's crazy to give up one's rights like that.

I'm also not really fond of GPLv3.  Argh.

The ngc2000 catalog is used to create the annotated image, in the program 'plotconstellations'.

I can see how to make the NGC2000 catalog optional:

1. switch from plotconstellations to plotann.py -- this would be a good thing to do anyway, but will take a bit of work.

2. modify plotannotations.c (used by plotann.py) so that the NGC objects are read from a FITS table, rather than having the fulltext being linked in.


I link the programs statically because (1) historical reasons; and (2) simplicity; we never get complaints about libraries installed in weird places, LD_LIBRARY_PATH, etc.

The whole project doesn't change much these days, so the API is stable/stagnant.

The index file locations have been stable for multiple years, I believe, and are likely to stay that way.

cheers,
--dustin

[Ole Streicher]

Hi Dustin,

Dustin Lang <dstn...@gmail.com> writes:
> Personally, I deeply dislike the "or any later version" option. Who
> knows what nutbar thing the FSF might do in the future? It's crazy to
> give up one's rights like that.
>
> I'm also not really fond of GPLv3. Argh.

I understand your doubts. However, this makes it difficult to keep the
program compatible with current and future libraries -- as you see on
the GSL case. Legally, you cannot link to the (an-)gsl, since GSL is
GPLv3 now and they consider linking as "dervived software", so that the
program as a whole whole must be then under GPLv3. This is currently
impossible, since the GPLv3 is not an option for your code yet.

In principle, there are two possible ways out of this: First (this is
that pragmatic one) is just to extend your license to "... or any later
version"; which requires some trust into the FSF. The other way would be
to put the code under a different license, which is compatible to the
GPL -- BSD could be an option here. Jake VanderPlas wrote an article
about this on astrobetter:

http://www.astrobetter.com/the-whys-and-hows-of-licensing-scientific-code/

A larger project that did such a relicensing is the "yt" project. They
also have a blog post about the why and hows:

http://blog.yt-project.org/post/Relicensing.html

However, even changing the license to BSD would have no impact of the
license of the program as a whole, because it is linked to a GPLv3
library. The only way to avoid this would be to stay away from GPLv3
code (which seems a bit unrealisting to me).

> The ngc2000 catalog is used to create the annotated image, in the program
> 'plotconstellations'.

> I can see how to make the NGC2000 catalog optional: [...]

I have contacted the publisher of the NGC2000.0 catalogue in order to
discuss whether they can remove the non-commercial clause.

> I link the programs statically because (1) historical reasons; and (2)
> simplicity; we never get complaints about libraries installed in weird
> places, LD_LIBRARY_PATH, etc.

Would you mind if I would change that for the Debian package? In Debian,
the library location is well-defined, and we try to avoid static linking
if possible. My current patch for this is, however, a bit Debian
specific, so it is not worth to be included in the common astrometry.net
codebase.

> The whole project doesn't change much these days, so the API is
> stable/stagnant.

Good to hear. I'll then create the packages as proposed (astrometry.net,
libastrometry, libastrometry-dev, python-astrometry).

> The index file locations have been stable for multiple years, I believe, and
> are likely to stay that way.

.... and the data downloaders (for the larger files). One question here:
Do the larger databases require the smaller ones, or are they
independent?



Great! I think the problems are now squeezed down to the licensing
issue(s).

For the licensing, I also want to make a big compliment to you: It
greatly helped that you have a CREDITS file with all the important
information in. Thank you very much for being that careful there!

Best regards

Ole