Debian/Ubuntu packaging of casacore

[Ole Streicher]

Dear casacore developers

(Cc: Gijs Molenaar: I found your Ubuntu package at Launchpad,
and Malte Marquarding: some Debian packaging in casacore)

I am currently packaging "casacore" for Debian [1], from where it will
find its way to Ubuntu. Eventually I hope to use this package as a base
for CASA [2].

Because the next Debian release (7.0 "Wheezy") is already frozen and
does not accept new packages, it will be put into the "testing" tree to
appear on the 8.0 "Jessie" release. Similarly, it will not appear in the
12.10 "Quantal Quetzal" Ubuntu release but hopefully in the following
13.04 release.

In the packaging process, a number of questions raised:

* Is there a mailing list to discuss the package building? What should
be set as the contact e-mail in the package?

* What is the relation between casacore and the CASA software package?
Can I build CASA (3.4) on top of the distributed casacore (1.5)
libraries? Are both developed independently of each other?

* What would be an optimal set of optional external packages? Is it
wise to build on top of all possible external dependencies (hdf5,
readline, fftw)? Also, would you recommend to enable or disable
threading?

* The libraries do not set a SONAME, making it difficult to handle
binary incompabilities over a longer time scale. Debian requires to
set this () Do you plan to introduce a SONAME attribute into the
shared libraries and/or what would you recommend to set here? [3]

* I plan to package every shared library into one individual package,
and to provide additional packages with the header files
(casacore-dev), the API documentation (casacore-doc) and the binary
tools (casacore-tools). However, for some of the libraries I couldn't
find a description: meas, mirlib, scimath_f, Could you shortly
explain their meaning?

* Many unit tests require external data files which seems to be not
fixed ("updated weekly") [4]. Is it useful to restrict the unit tests
to the ones that do not depend on these data? And is there a flag
available to apply this restriction?

* Debian is ported to a large number of architectures. Possible CPU
types include Alpha, MIPS, S390 and SPARC; aside from Linux, the
kernel may be kfreeBSD or HURD. Do you have experiences with these
architectures which are known to work/not work?

* Do you have any other comment on the packaging?

I would be glad to cooperate for the packaging process.

Best regards

Ole

[1] Intent to package announcement: <http://bugs.debian.org/686924>
[2] CASA homepage <http://casa.nrao.edu/>
[3] Debian Policy Manual - Shared libraries
<http://www.debian.org/doc/debian-policy/ch-sharedlibs.html>
[4] Weekly "measures" meta-data
<ftp://ftp.atnf.csiro.au/pub/software/measures_data/measures_data.tar.bz2>

[Malte Marquarding]

Hi Ole,

I am one of the maintainers of casacore and will try to answer your questions in-line. Ger might have something to say as well.

Cheers,

Malte.

On Tuesday, 11 September 2012 21:57:27 UTC+10, Ole Streicher wrote:

Dear casacore developers

(Cc: Gijs Molenaar: I found your Ubuntu package at Launchpad,
and Malte Marquarding: some Debian packaging in casacore)

I am currently packaging "casacore" for Debian [1], from where it will
find its way to Ubuntu. Eventually I hope to use this package as a base
for CASA [2].

Fantastic.  Here at CASS (wcslib comnes from here) we are trying to debianize most of our software as well. I hadn't gotten to casacore yet, so as a quick fix I am using cpack.

Unfortunately, CASA has its own version of "casacore". We occasionally  merge in changes from CASA into casacore but it has diverged a fair bit.

Because the next Debian release (7.0 "Wheezy") is already frozen and
does not accept new packages, it will be put into the "testing" tree to
appear on the 8.0 "Jessie" release. Similarly, it will not appear in the
12.10 "Quantal Quetzal" Ubuntu release but hopefully in the following
13.04 release.

In the packaging process, a number of questions raised:

* Is there a mailing list to discuss the package building? What should
  be set as the contact e-mail in the package?

Both Ger and my official addresses are fine. I will send them to you outside this list.

There is casacore-devel which could be used even though most discussions are between Ger and myself. 

* What is the relation between casacore and the CASA software package?
  Can I build CASA (3.4) on top of the distributed casacore (1.5)
  libraries? Are both developed independently of each other?

See above.

 

* What would be an optimal set of optional external packages? Is it
  wise to build on top of all possible external dependencies (hdf5,
  readline, fftw)? Also, would you recommend to enable or disable
  threading?

I guess readline could be non-optional on debian based system. hdf5, fftw should remain optional. By default threading should be disabled, but Ger should comment on this.

* The libraries do not set a SONAME, making it difficult to handle
  binary incompabilities over a longer time scale. Debian requires to
  set this () Do you plan to introduce a SONAME attribute into the
  shared libraries and/or what would you recommend to set here? [3]

Sure I will add this. I can also give you svn access permissions, so the packaging could go directly into the source here.

We can create a branch for this. 

* I plan to package every shared library into one individual package,
  and to provide additional packages with the header files
  (casacore-dev), the API documentation (casacore-doc) and the binary
  tools (casacore-tools). However, for some of the libraries I couldn't
  find a description: meas, mirlib, scimath_f, Could you shortly
  explain their meaning?

mirlib: read miriad images, so should be part of casa_images

meas: need to ask Ger

scimath_f: fortran extensions which should go with casa_scimath

(measures_f is only used in tests)

Do you plan to separate module executables out? Would that be casacore-bin and depend on all of them.

 

 

* Many unit tests require external data files which seems to be not
  fixed ("updated weekly") [4]. Is it useful to restrict the unit tests
  to the ones that do not depend on these data? And is there a flag
  available to apply this restriction?

There aren't any flags. These are not "unit" test per se. The measures metadata files usually outdate every six months (leap seconds and Earth Orientation parameters.

It should  be provided as as separate package casacore depends on, and possibly provide a cron job to update itself (not weekly).

* Debian is ported to a large number of architectures. Possible CPU
  types include Alpha, MIPS, S390 and SPARC; aside from Linux, the
  kernel may be kfreeBSD or HURD. Do you have experiences with these
  architectures which are known to work/not work?

FreeBSD itself is working, SPARC/ALPHA used to but have not be used in quite some time. Look at casa/aipsdef.h

 

* Do you have any other comment on the packaging?

As mentioned CASS is using debian so this would be very much appreciated. ASTRON however doesn't use it.

[Ole Streicher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Gijs,

some short comments:

On 13.09.2012 11:51, Gijs Molenaar wrote:
> On 09/13/2012 02:26 AM, Malte Marquarding wrote: [...] I'm not sure
> if you can compile packages with 'optional' dependencies, as far as
> I know they should be turned off or on. If we are going to use
> these packages we probably will need HDF5 support. What would be
> the advantage of disabling these features?

Did you have success by enabling HDF5 on Debian or Ubuntu? Did you use
the serial version here or some other?

>> Do you plan to separate module executables out? Would that be
>> casacore-bin and depend on all of them.
>

> Your naming schema doesn't comply to the Debian standard. library
> packages should be named like libcasacore0, libcasacore-dev, and
> cacascore for the executables for example.

I plan to build:
- - casacore-dev - development package
- - casacore-doc - API (doxygen) documentation
- - casacore-tools - executables
- - libcasa_<module>1 - library package for <module>, if SOVERSION is 1
(one package per library)

> you probably don't need the unit tests for packaging, these should
> only be used during development. Still, you could use them for
> checking your package validity, but overall they are not used by
> end users.

Since I am not sure about the results on all the Debian packages, I
would like to include the unit tests into the build process. But I
agree that it makes no sense to provide the measurement_data as Debian
package makes no sense unless they are needed by the end user. Are they?

>> I would be glad to cooperate for the packaging process.
>

> me too, if I have time.

Great, thank you.

Best

Ole


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQIcBAEBAgAGBQJQUa+yAAoJEHEVr9B3ENz33mwQAIs/EgClB/KEBwXxCgEYMF+0
sx8/krJMuJz7odbxzi2bTdNNXAG9IM5ppwQa2RVv9oPLZDznsSSWceFiw0knNVtZ
UFhemC0U3F1Qby2P4UAF7jS2jiIDDBY1LsFzO58xh8BE/f7puKFQn9PZoi4dOV4p
/KvCOjUbn6X04PpqZXYEA3Dw1hY4kwGgJ454gJMw1G4SgtlvKbv16TLgqXpmwglF
xqFUzxGOFJQRzLfYv+l+xEpbrYHV1EIIEbG4Os75vxPxXySTf3YgDtcAnUoyjIgT
hEjh4OsGfZY6D8pfSEYYMUQeOF31xTZOnh8qudJZNtjlOgC+HatFm/vEfC2fLdnx
a9uurG8JCdQJuAI0A7Pqm3k5kev4YZMOjNu2MbBvmlUBaa8u+LQarYE9Yc94Ldhn
a5S4SpB++kPjAcGmKmLIri1uEdypv45jTlFAxv+K7McYrUa7ofofDddLaaEI0Zu2
qX+VN5wtKD29GWpFg9MPp/0XQQexrr9UD6cWBQLsjuvmoVh0c7vOfeN/k1utZ5Nx
pmFkb4J62PfxCbm1XUNgX/rGr4JvJBUnPLYbHEtVLrKa3ZqgzNlewwAospjP5hwM
LBurimBxQE//POqdynH6DE84PsF+yE8ipDVt0EUYRJ0SLYdGFsk3M0bnwZS29PxV
qgWYaSP+aWO075IVmOJo
=zOPU
-----END PGP SIGNATURE-----

[Ole Streicher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 13.09.2012 12:31, Gijs Molenaar wrote:

> On 09/13/2012 12:04 PM, Ole Streicher wrote:
>> Did you have success by enabling HDF5 on Debian or Ubuntu? Did
>> you use the serial version here or some other?
>

> I didn't try it yet, I only know we might use it some day :)
>
> I'm working on the LOFAR project and some people are working on a
> data storage standard based on HDF5 and produced by casacore.

I will have a look what exactly failed (probably I just used the wrong
hdf5 library). However, could you ask whether LOFAR uses the standard
hdf5 libraries provided by Ubuntu or Debian? Could you check which
hdf5 package is installed? And/or did they patch the casacore souorce?

>> I plan to build: - casacore-dev - development package -
>> casacore-doc - API (doxygen) documentation - casacore-tools -
>> executables - libcasa_<module>1 - library package for <module>,

>> if SOVERSION is 1 (one package per library)
>

> I thought casacore-tools should just be named casacore, but
> probably I'm wrong.

I would call it "casacore" if it would serve as the main package. But
the executables are quite simple and probably more an implementation
example, so I think that "tools" is more appropriate here, especially
as they are almost undocumented. If they are just examples, I would
even drop the -tools package completely and put its source code as
examples into the documentation of casacore-dev.

>>> you probably don't need the unit tests for packaging, these
>>> should only be used during development. Still, you could use
>>> them for checking your package validity, but overall they are
>>> not used by end users.
>

> Ok one note here, I'm not an astronomer (also), so I'm not sure,
> but I believe astronomers actually do use this data so it would be
> nice to have it available in an easy way. But as from what I
> understand this data changes from time to time so I don't know if
> packaging would be feasible since then it would lag behind. Maybe
> some mechanism like how virus scanners get their definitions is
> more appropriate? If I remember correctly there is something like a
> volatile repository.

It seems to change weekly, however it may be enough to update about
twice per year, so updating via package may be reasonable.

Cheers

Ole

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQIcBAEBAgAGBQJQUcdVAAoJEHEVr9B3ENz3gccP/jQHsCZa+mIv0t8kQmOp7tMC
C/rOlww4DfSg90fmkeecezoKX+AsJ6Bmfqs+H7fJx4FZSmdXPfcEonfQ7YztSX+Q
ZxB1lyn+YNqzb2bxq9XDARevvBiqp1o7+NVLET8rKRp8c5re5F5cRCP/9fkT1Upn
0aCZ8sQgpd4ayzn/JSeoGK/lwHzQq/zwHe9rhqGtvjsTsN8POD8P7kyYLgus184X
WB+mOJmJx7TNAbMBtfbv7XZzs4vFfOn/kr1b+KhPEZH9x63p+Kb6JW0mgeLIRmFq
gyl3rbwpuNtjLuwgVT/lTGlXxk01XIm5YS67W7SwapXW9GyGja0f3W/RLMsVX2k+
lOgdd5bdM5pNVBzrLIAz696AJ7GGWMJFVYkDPggUaJbBBWy4dPfExOC+XN65zYst
MM45qq5LtkHExWr4nR/g5RtSZEWptUWxZo8LIDC9EDQFT4PPSgCsoHhx4PleQh2U
RQ+rPjnxFJ3CDTkPK2R/+uEIdEbyh0mJK3MGtpn83wfvbU6fbuu/wcR6SyY2NexT
HZjHetTwtt/QIK9ABoCVP6iK1wN2dn58pSTjORhzjKzXV5QsN73uhiPYhO8eh6Ny
v0+tTBdjhfRU6LWF1j1MIvq8rxiLsYTWFmd185RT34aObqVg/pNR0GDTXmsy5Smj
CGiqp6Ns4fy8nFsOQjAd
=eO2w
-----END PGP SIGNATURE-----