interrupt.pyx on PyPi?

[Martin R. Albrecht]

Hi all,

In a thread over at [sage-support] William wrote:
> I've suggested numerous times that we should massively refactor the
> sage library to be a bunch of smaller Python libraries, develop them
> say on github (?), use Pypi and pip. If people would realize how
> important it is that we revamp how Sage development is done in a much
> less monolithic way, and better using existing tools, then I would be
> happy and enjoy watching as people do the revamp (e.g., like happened
> with switching from Mercurial to Git, which I didn't do much on, but
> definitely supported).

I don’t mean to restart a general strategy/development discussion here,
but perhaps turning Sage into a bunch of smaller Python libraries is
something which can be accomplished step-by-step.

In particular, I am *very* interested in turning Sage’s interrupt
handling into something that can be easily installed from PyPI. For
those who don’t know what Sage’s interrupt handling does: it allows that
you can press Ctrl-C to interrupt long running C code and that crashes
in a library do not necessarily crash your Python shell. It is a
difference between night and day to interface with external C/C++ code
From Python without and with it.

Having this code available outside of Sage enables, for example, to turn
some of our C/C++ library interfaces into stand-alone libraries[^1], like
I’ve done with our fplll interface at

https://github.com/malb/fpylll

Sage’s interrupt handling lives at

https://github.com/sagemath/sage/tree/master/src/sage/ext/interrupt

I think the technical challenges are manageable. I’m using a copy of it
separately in

https://github.com/malb/fpylll/tree/master/src/fpylll/interrupt

without issues. Making this code independent would mean one big patch
to the library, though, where we replace all

include 'sage/ext/interrupt.pxi'

with an appropriate new line depending on where we’d install the file.

My question is: am I overlooking some technical challenge or another
reason why making this code independent could be a problem?

I’m happy to put some work into this, in case you’re wondering.

Cheers,
Martin

[^1]: Okay, there’s also the small issue of how to do type conversion
Sage ←→ library in a nice modular yet fast way, but one step at a time.

--

_pgp: https://keybase.io/martinralbrecht
_www: https://martinralbrecht.wordpress.com
_jab: martinr...@jabber.ccc.de
_otr: 47F43D1A 5D68C36F 468BAEBA 640E8856 D7951CCF

[Jeroen Demeyer]

On 2016-02-01 10:40, 'Martin R. Albrecht' via sage-devel wrote:
> In particular, I am *very* interested in turning Sage’s interrupt
> handling into something that can be easily installed from PyPI. For
> those who don’t know what Sage’s interrupt handling does: it allows that
> you can press Ctrl-C to interrupt long running C code and that crashes
> in a library do not necessarily crash your Python shell.

First of all, do you assume that the code you want to interrupt is
Cython? Or do you want something which could work more generally?

I guess it could be separated from Sage. It might be a good project for
some Sage Days workshop.

One non-trivial thing is that the Sage interrupt framework also refers
directly to PARI. So this would need to be abstracted.

And of course it will only work on POSIX systems.


Jeroen.

[Martin R. Albrecht]

Hi Jeroen,

Jeroen Demeyer writes:
> On 2016-02-01 10:40, 'Martin R. Albrecht' via sage-devel wrote:
>> In particular, I am *very* interested in turning Sage’s interrupt
>> handling into something that can be easily installed from PyPI. For
>> those who don’t know what Sage’s interrupt handling does: it allows that
>> you can press Ctrl-C to interrupt long running C code and that crashes
>> in a library do not necessarily crash your Python shell.
>
> First of all, do you assume that the code you want to interrupt is
> Cython? Or do you want something which could work more generally?

Cython-only is enough for me.

> I guess it could be separated from Sage. It might be a good project for
> some Sage Days workshop.

+1, I don’t think I’ll attend any anytime soon. Let’s see if I can find
some time to work on this in the mean time. I’ll wait a bit more if
there are objects here and then open a ticket, where we can move
technical discussions.

> One non-trivial thing is that the Sage interrupt framework also refers
> directly to PARI. So this would need to be abstracted.

Ah, I forgot about removing that one for my hack. Good point.

> And of course it will only work on POSIX systems.

I guess we could provide some noop for non-POSIX systems?

Cheers,
Martin

[Volker Braun]

On Monday, February 1, 2016 at 10:40:42 AM UTC+1, Martin Albrecht wrote:

but perhaps turning Sage into a bunch of smaller Python libraries is
something which can be accomplished step-by-step.

The first order of business should then be to modularize the doctest framework, otherwise you end up with a bunch of packages that have no working testsuite. Which nobody in their right mind would want to base their work on.

Also, namespace packages (i.e. multiple packages sharing sage.*) are easier in Python >= 3.3 thanks to PEP420.

Splitting of non-mathematical tools would be definitely an advantage since they would be useful in other projects. And some fairly isolated interfaces with third-party programs could be factored out, too. But I don't think there is much value in splitting up most of the core functionality; If the doctests can't be run independently then modules are irreducible. And there aren't many doctests that you can run without libgmp or libsingular, for example. 

Also, the semver coolaid doesn't really work if there is nobody running e2e tests. Which really is admitting defeat, if I can't develop packages independently then they are not independent. At least in npm one can work around broken semver by installing specific versions in parallel, but pip does not support that. On the plus side, one can still install install pip packages without pages of warnings about unsupported/abandoned versions.... 

[Jeroen Demeyer]

One more thing: I think that this "interrupt" project would really
benefit from autoconf. I know that Python + autoconf is not a common
combination, but there are some non-trivial system-specific things which
are best handled by autoconf. Moreover, I have always wanted to add
support for handling stack overflows using sigaltstack(), possibly also
using getcontext()/setcontext(), which might again require autoconf checks.

[Volker Braun]

The system-specific part could be a separate C library "libinterrupt" that the python package depends on. That is how many other Python packages depend on system-specific libraries...

[Jeroen Demeyer]

On 2016-02-02 09:51, Volker Braun wrote:
> The system-specific part could be a separate C library "libinterrupt"
> that the python package depends on. That is how many other Python
> packages depend on system-specific libraries...

That just adds an extra step, I don't see why this is a good idea. I can
just compile C code directly in the Cython extension, that's not the
problem.

[Martin R. Albrecht]

Hi Volker,

I’m not sure I follow. Many Python packages have tests, pick your
favourite Python testing framework. Why should this be a problem here?
In addition, Sage can/should run its own doctests to check there’s no
mismatch.

Cheers,
Martin

[Martin R. Albrecht]

I guess this ends my plan to just

pip install -r requirements.txt

it. But it makes sense.

This is now http://trac.sagemath.org/ticket/20002

libcynterupt?

Cheers,
Martin

Volker Braun writes:

[Jeroen Demeyer]

On 2016-02-02 10:16, 'Martin R. Albrecht' via sage-devel wrote:
> I guess this ends my plan to just
>
> pip install -r requirements.txt

I'm not sure it does. I have no idea what pip install $FOO actually
does. If it just runs setup.py, we can still run ./configure from setup.py.

> libcynterupt?
Bikeshedding time! For of all, I would certainly drop the "lib".

Then we should consider if we want to refer to "interrupt" or to
"signal". Arguments for "signal" are that the interrupt/signal framework
handles more signals than just SIGINT and that the macros are also
called sig_on()/sig_off(). So what about "cysignals"?

[Jean-Pierre Flori]

"cygnals"?

[Volker Braun]

Right now all tests for interrupts use "sage:" markers and customized magic comments. So its not easy to run the existing doctests under $favorite_testing_framework.

[Volker Braun]

On Tuesday, February 2, 2016 at 9:59:11 AM UTC+1, Jeroen Demeyer wrote:

That just adds an extra step, I don't see why this is a good idea.

* makes it easier for distros, e.g. passing custom arguments to ./configure

* easier and faster venv installations if you don't have to recompile the c-level stuff every time 

[Martin R. Albrecht]

Hi Jeroen,

my concern is mainly of convention: I don’t think people expect pip to
install shared libraries. Also, virtual environments tend not to set
LD_LIBRARY_PATH but many people use pip with virtualenvs.

Cheers,
Martin

[Jeroen Demeyer]

On 2016-02-02 11:04, Jean-Pierre Flori wrote:
> "cygnals"?
Sounds too much like Cygwin.

[Jeroen Demeyer]

On 2016-02-02 11:12, Volker Braun wrote:
> Right now all tests for interrupts use "sage:" markers and customized
> magic comments. So its not easy to run the existing doctests under
> $favorite_testing_framework.

At least for the interrupt tests, this shouldn't be a big issue. For
other parts of Sage, that's a different story.

[Jeroen Demeyer]

On 2016-02-02 11:33, 'Martin R. Albrecht' via sage-devel wrote:
> Hi Jeroen,
>
> my concern is mainly of convention: I don’t think people expect pip to
> install shared libraries.

I am not convinced that this "cysignals" package should contain a shared
library. In Sage, it's a Python module(*), not a shared library. I know
that Volker argues for a shared library, but I think that's just
needlessly complicating matters.

(*) To be more precise, it's a Python module with some Cython files
(.pxd and .pxi) and C headers (.h). It will be an interesting exercise
to figure out how to install this properly :-)

[Jeroen Demeyer]

On 2016-02-02 11:16, Volker Braun wrote:
> * makes it easier for distros, e.g. passing custom arguments to ./configure

Why is it easier to pass custom arguments to ./configure if there are 2
packages instead of 1 package?

> * easier and faster venv installations if you don't have to recompile
> the c-level stuff every time

I don't get what do you mean. Would you install the library outside the
venv and the Python package inside the venv? And besides, it's a small
package, the compilation time does not matter much.

Adding a library as separate package really looks like a solution in
search of problem.

[Martin R. Albrecht]

Jean-Pierre Flori writes:
> "cygnals"?

+1

[Volker Braun]

On Tuesday, February 2, 2016 at 11:44:44 AM UTC+1, Jeroen Demeyer wrote:

> * makes it easier for distros, e.g. passing custom arguments to ./configure
Why is it easier to pass custom arguments to ./configure if there are 2
packages instead of 1 package?

Because only one of them has a ./configure, the other only has setup.py.

Its just a version of separation of concerns. Abstract the system-specifics into a c shared library and use that api from python. One might want to use it without Cython, e.g. plain CPython or boost::python

Compilation is fast as there isn't that much code, but running configure checking for all kinds of signal handler quirks may be less so.

Anyways just a thought.

[Jeroen Demeyer]

On 2016-02-02 12:00, Volker Braun wrote:
> One might want to use it without Cython, e.g. plain CPython or boost::python

Well, the Cython and C parts are quite intertwined, so I do not intend
to support anything but Cython.

[Martin R. Albrecht]

Hi all,

work on this has started:

https://github.com/malb/signal.pyx

http://trac.sagemath.org/ticket/20002

So far, it’s pretty standard setup.py stuff, i.e. no autoconf.

Help welcome:

https://github.com/malb/signal.pyx/issues

Cheers,
Martin

[William Stein]

Hi,

Ondrej Certik asks "Any chance to BSD license it [signal.pyx]?"

I wrote the first version of the code and would be for that. It's
core infrastructure and if we BSD it, then the chances it'll get used
widely and be better quality go up. So I'm +1. I think we would
mainly need Jereon Demeyer and Martin to also be +1 for BSD licensing
and could do it, though searching the git blame would be a good idea.

William

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



--
William (http://wstein.org)

[Jeroen Demeyer]

See https://github.com/malb/signal.pyx/issues/2

[Jeroen Demeyer]

On 2016-02-01 10:40, 'Martin R. Albrecht' via sage-devel wrote:
> In particular, I am *very* interested in turning Sage’s interrupt
> handling into something that can be easily installed from PyPI.

This is now ready. The upstream package is hosted at
https://github.com/sagemath/cysignals and the Sage ticket is
http://trac.sagemath.org/ticket/20002

After the work which has been done, it would be very nice if somebody
could review the Sage ticket #20002. Note that a review would mostly
consist of checking the build and packaging. The "interrupt" part was
just moved from Sage to an external package, so that doesn't need to be
reviewed.

Jeroen.