Correct procedure for an experimental package

[Charles Bouillaguet]

Hi all,

I (amongst with others) have written a library that solves boolean
equations in some way. I wanted to include it as an experimental spkg. I
thus wrote a cython interface to Sage. To be usable from inside Sage,
both the library itself and the cython interface are necessary. What is
the proper way to do this ?

I first thought that the Right Way (r) was to conditionally include the
cython part to the library when the spkg is installed, by writing
something like this in module_list.py :

if is_package_installed('stuff'):
ext_modules.extend([ blablabla ])


This works fine, except that it forces potential users to not only
install an spkg but also run sage -b, and it apparently makes it
impossible to update the reference manual: if I add a line

sage/libs/stuff

to libs.rst, then building the reference manual fails when the spkg is
not installed.

What should I do ? Unconditionally include the cython interface to the
sage library, and mark all doctests as optional ?

Cheers,
--
Charles Bouillaguet

[Burcin Erocal]

Hi Charles,

Can you compile the Cython interface without the header files installed
by your package?


I assume the Cython interface needs to be changed whenever you update
your library. Which Sage version is used with an experimental package
cannot be controlled, since Sage just grabs the latest version available
at install time. Hence, if you keep the interface separate from
the spkg, you will run into many incompatibility issues.

Wouldn't it be easier to include the Cython interface in the package?
AFAIK, Cython's build system improved significantly and there is no
reason to use Sage's build system for a Cython module.


Cheers,
Burcin

[Charles Bouillaguet]

On 12/20/2012 01:02 PM, Burcin Erocal wrote:
>> What should I do ? Unconditionally include the cython interface to
>> the sage library, and mark all doctests as optional ?
>
> Can you compile the Cython interface without the header files installed
> by your package?

No

> I assume the Cython interface needs to be changed whenever you update
> your library.

Yes, but changing spkg versions requires a ticket, on which the sage
library can also be changed.

> Which Sage version is used with an experimental package
> cannot be controlled, since Sage just grabs the latest version available
> at install time. Hence, if you keep the interface separate from
> the spkg, you will run into many incompatibility issues.

OK. But then how do the other spkgs do ?

> Wouldn't it be easier to include the Cython interface in the package?
> AFAIK, Cython's build system improved significantly and there is no
> reason to use Sage's build system for a Cython module.

Is it possible to patch the sage library when installing an spkg ? What
if the patch fails because the sage library was updated since the last
spkg updatye ??? This seems like asking for trouble, and incompatibilities.

I still don't understand how it is currently done.

Charles

[Martin Albrecht]

> Wouldn't it be easier to include the Cython interface in the package?
> AFAIK, Cython's build system improved significantly and there is no
> reason to use Sage's build system for a Cython module.

Hi Burcin,

my understanding is that Charles was more specifically asking about including
his experimental package in the reference manual. My understanding is that
this is currently not possible.

Cheers,
Martin

--
name: Martin Albrecht
_pgp: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x8EF0DC99
_otr: 47F43D1A 5D68C36F 468BAEBA 640E8856 D7951CCF
_www: http://martinralbrecht.wordpress.com/
_jab: martinr...@jabber.ccc.de

[Burcin Erocal]

I am not suggesting to patch the Sage library when installing your
package. You can put the interface in a separate Cython module which is
built by the usual Python/Cython setup.py magic and installed in the
system python module directory. Then people will be able to import your
module from the Sage command line as usual.

> I still don't understand how it is currently done.

I don't think there is a well established procedure for this. There is
code in the Sage library that depends on optional packages. This works
just like you described. modules_list.py already includes some examples.
In this case, all doctests will need to be marked optional.


Given the fact that there is no way to guarantee that the wrapper
code in the Sage library corresponds to the library version
available in the package, I don't see how the current practice is
better than bundling the wrappers in the package.

I can see the benefits of having the wrappers "included in Sage" from a
reputation/review point of view. However, this is not on par with the
rest of the Sage library code, since this code is hidden behind
optional tags and not tested by Sage users or developers regularly.

#13540 improves the situation somewhat. But the way optional
packages/modules are handled still leaves something to be desired.
Especially if we want to promote creating separate specialized packages
as a way of distributing research code:

https://groups.google.com/d/topic/sage-devel/fcxNrQqVSz0/discussion


Cheers,
Burcin

[Charles Bouillaguet]

> I am not suggesting to patch the Sage library when installing your
> package. You can put the interface in a separate Cython module which is
> built by the usual Python/Cython setup.py magic and installed in the
> system python module directory. Then people will be able to import your
> module from the Sage command line as usual.

The python interface **uses** the sage library (matrices, modules, polynomials, term orders, etc. etc. etc.). It does not make sense outside of sage. It is a binding between sage and my piece of C-code.

I actrually followed the same pattern I found there : http://trac.sagemath.org/sage_trac/ticket/418

> I don't think there is a well established procedure for this. There is
> code in the Sage library that depends on optional packages. This works
> just like you described. modules_list.py already includes some examples.
> In this case, all doctests will need to be marked optional.

Strictly speaking, they don't, because the code is not even considered if the spkg is not there.

> I can see the benefits of having the wrappers "included in Sage" from a
> reputation/review point of view.

You are right, but again I followed what I thought was the right way to go (http://trac.sagemath.org/sage_trac/ticket/418). I certainly agree that it is a bit weird to have a lot of "inactive" code on everyone's installation.

Again, what should I do :) ?

Charles

[Jeroen Demeyer]

On 2012-12-20 14:10, Charles Bouillaguet wrote:
> Again, what should I do :) ?

The question is really: what to do with *documentation* of
optional/experimental packages, should it be amongst the Sage
documentation? If YES, then indeed all doctests need to be marked optional.

[Burcin Erocal]

Hi Martin,

On Thu, 20 Dec 2012 13:15:51 +0100
Martin Albrecht <martinr...@googlemail.com> wrote:

> > Wouldn't it be easier to include the Cython interface in the
> > package? AFAIK, Cython's build system improved significantly and
> > there is no reason to use Sage's build system for a Cython module.
>

> my understanding is that Charles was more specifically asking about
> including his experimental package in the reference manual. My
> understanding is that this is currently not possible.

Aaah.. thanks for the pointer. When I read his message, having to rerun
"sage -b" to get the relevant code seemed more problematic than not
being able to access the documentation. I guess I am used to reading
the code for documentation. I don't mind if something is not in the
reference manual.


This is orthogonal to my main point though: I think wrappers for
optional packages should live in the package and not the Sage library.

I know this goes against the popular general philosophy of "Sage
development can be streamlined by bringing everything under one roof."
I believe the Sage project would benefit more if we stop trying to make
research code and a stable library interface fit into the same package
and encourage people to build on Sage while we concentrate on creating
a rock solid python library with an intuitive interface for generally
useful mathematics functionality.


Cheers,
Burcin

[Burcin Erocal]

On Thu, 20 Dec 2012 14:10:55 +0100
Charles Bouillaguet <charles.b...@gmail.com> wrote:

> > I am not suggesting to patch the Sage library when installing your
> > package. You can put the interface in a separate Cython module
> > which is built by the usual Python/Cython setup.py magic and
> > installed in the system python module directory. Then people will
> > be able to import your module from the Sage command line as usual.
>
> The python interface **uses** the sage library (matrices, modules,
> polynomials, term orders, etc. etc. etc.). It does not make sense
> outside of sage. It is a binding between sage and my piece of C-code.

Some examples of Python/Cython packages that depend on Sage:

http://sage.math.washington.edu/home/SimonKing/Cohomology/

https://github.com/williamstein/psage

Sage is supposed to be a Python library for mathematics among other
things (software distribution, notebook interface, interface to maths
software). It is natural to use it as a library. :)

> I actrually followed the same pattern I found there :
> http://trac.sagemath.org/sage_trac/ticket/418

I am aware that you followed an often used pattern for development. I
am not writing to criticize you in any way. I am merely aiming to
point out the deficiencies of this pattern.

> > I don't think there is a well established procedure for this. There
> > is code in the Sage library that depends on optional packages. This
> > works just like you described. modules_list.py already includes
> > some examples. In this case, all doctests will need to be marked
> > optional.
>
> Strictly speaking, they don't, because the code is not even
> considered if the spkg is not there.

??? Won't it be considered if you run doctests on the full Sage library
with

sage -testall



Cheers,
Burcin

[Simon King]

Hi All,

On 2012-12-20, Burcin Erocal <bur...@erocal.org> wrote:
> On Thu, 20 Dec 2012 14:10:55 +0100

> Some examples of Python/Cython packages that depend on Sage:
>
> http://sage.math.washington.edu/home/SimonKing/Cohomology/

As author of this package, I can confirm that it works easily to create
Cython modules in an optional package, linking against Sage's pxd headers.

And of course, if you install any python module (say, using
easy_install) in a Sage shell, then you can import things in a Sage
session. In the cohomology package, it simply is "from
pGroupCohomology.cochain import ...".

Concerning documentation, note the environment variable
SAGE_SPKG_INSTALL_DOCS. If it is "yes", an optional spkg is supposed to build
documentation and put it into
SAGE_ROOT/local/share/doc/<package_import_name>/html. Not all optional spkgs do
that, but IIRC I was told that this is how things *should* work.

But I think the OP's question was different. If I understand correctly,
the problem is that the experimental/optional package offers additional
methods that would be useful in *existing* methods from the Sage
library. Say, you have an spkg that computes Gröbner bases much easier
than Singular in some cases, then you want to modify the
groebner_basis() method so that the algorithm from the spkg can be
selected. And I don't see how that would be possible without modifying
the Sage library.

Best regards,
Simon