Why is struct_signals.h included, and why is it not found?

[Simon King]

Hi!

Once more I got the impression that "internal changes that don't require a
deprecation" broke the software that is an important part of my
research.

When trying to build the old-style version of my modular group
cohomology spkg, I now get:
pGroupCohomology/resolution.c:279:28: fatal error: struct_signals.h: No
such file or directory
#include "struct_signals.h"

So, first question:
- Why is it trying to include struct_signals.h? In the non-published spkg
version on my laptop, there is
include "cysignals/signals.pxi"
Is this why Cythonization results in the inclusion of struct_signals.h?
Anyway, the string "struct_signals" does not appear in the spkg's
sources.

Second question:
- Why is struct_signals.h not found? It is in fact in
$SAGE_LOCAL/lib/python2.7/site-packages/cysignals/struct_signals.h

Hopefully I can get the software running again, and soon -- a relevant
workshop in Oberwolfach is starting Monday.

Best regards,
Simon

PS: I still try to break the SPKG into modules, rewriting everything
based on upstream sources that have thoroughly changed their API. The
attempt to switch from a self-contained old style package to modular
new style packages has been a desaster for my productivity.

[Vincent Delecroix]

Hi Simon,

Are the sources available somewhere else than on your computer?

[Vincent Delecroix]

trying to install version 2.1.4.p0 available from the mirrors I got a
lot of errors (unrelated to cysignals)

../mtx2.2.4/src/zzz.c:110:21: error: expected identifier or '(' before
'[' token
static BYTE restrict[4][256]; /* Restrictions to subfields */

../mtx2.2.4/src/zzz.c:382:10: error: expected expression before 'restrict'
fread(restrict,256,4,fd) != 4

../mtx2.2.4/src/zzz.c:382:4: error: too few arguments to function 'fread'
fread(restrict,256,4,fd) != 4

../mtx2.2.4/src/zzz.c:454:12: error: expected expression before 'restrict'
return restrict[i][a];

On 30/07/16 19:09, Simon King wrote:

[Simon King]

Hi Vincent,

On 2016-07-30, Vincent Delecroix <20100.d...@gmail.com> wrote:
> Are the sources available somewhere else than on your computer?

Sure --- but some changes with respect to the currently published
sources are needed anyway, that's why I asked more generally why the
inclusion of the header is generated by Cython and why it isn't found.

The documentation is at
http://users.minet.uni-jena.de/cohomology/documentation/
and the package is at
http://users.minet.uni-jena.de/cohomology/p_group_cohomology-2.1.5.spkg

The problem appears when building src/pGroupCohomology/resolution.pyx

With respect to the currently published version, I changed
include "interrupt.pxi"
into
include "cysignals/signals.pxi"
Both versions didn't work, but I think I recall that "interrupt.pxi" has
vanished, whereas "cysignals/signals.pxi" is the thing that currently is
in use and is supposed to work.

Note that I need gcc 4.9.3 to come to the point of building this
cython module. With gcc 5.2.1, an error occurs when building a third
party package included in the spkg, but probably I have no time to try
and sort out compiler problems.

Best regards,
Simon

[Vincent Delecroix]

With

$ export CC="gcc-4.9"
$ sage -p p_group_cohomology-2.1.5.spkg

I confirm at least that I got the very same error as you do!

[leif]

Vincent Delecroix wrote:
> Hi Simon,
>
> Are the sources available somewhere else than on your computer?

And/or could you post the log?

Which version of Sage are you currently using, and which version(s) will
be used at the workshop?


-leif

[Vincent Delecroix]

At least, adding explicitely to the include_dirs in setup.py the line

os.path.join(SAGE_LOCAL, "lib/python2.7/site-packages/cysignals/")

the error disappear (I guess this is a Sage part bug). But there is
another incompatible API change on the recent versions. All of _add_,
_mul_ etc is now untyped

in other words

cpdef ModuleElement _sub_(self, ModuleElement other):

should now be

cpdef _add_(self, right)

Vincent

[Simon King]

Hi Leif,

On 2016-07-30, leif <not.r...@online.de> wrote:
> And/or could you post the log?

http://users.minet.uni-jena.de/~king/p_group_cohomology-2.1.5.log

> Which version of Sage are you currently using,

It is sage-7.3.beta8, which I have built using Sage's gcc-4.9.3 package,
after finding that my system-wide gcc-5.2.1 didn't succeed with my spkg.

> and which version(s) will
> be used at the workshop?

I don't know whether some of the participants are Sage users. The topic
is "computational group theory".

Best regards,
Simon

[leif]

Vincent Delecroix wrote:
> With
>
> $ export CC="gcc-4.9"
> $ sage -p p_group_cohomology-2.1.5.spkg
>
> I confirm at least that I got the very same error as you do!

I guess that refers to 'restrict' in MeatAxe; it's a reserved word in
C99, so you should either patch your MeatAxe sources or compile with
'-std=gnu89' (or '-std=c89') in CFLAGS (or CC), at least when using
newer versions of GCC that default to more recent standards.


-leif

[leif]

Probably replied to the wrong post:

To compile MeatAxe (with more recent versions of GCC), put '-std=c89'
(or '-std=gnu89') into CFLAGS, as 'restrict' is a keyword since C99.


-leif

[Simon King]

Hi Vincent,

On 2016-07-31, Vincent Delecroix <20100.d...@gmail.com> wrote:
> At least, adding explicitely to the include_dirs in setup.py the line
>
> os.path.join(SAGE_LOCAL, "lib/python2.7/site-packages/cysignals/")
>
> the error disappear (I guess this is a Sage part bug). But there is
> another incompatible API change on the recent versions. All of _add_,
> _mul_ etc is now untyped

With these changes, it seems that the package builds! Thank you very
much. As soon as I will have a better bandwidth, I'll publish the new
old-style spkg version 2.1.6. That way, I will at least be able to point
people to something that works...

Best regards,
Simon

[Simon King]

Hi Leif,

On 2016-07-31, leif <not.r...@online.de> wrote:
> Probably replied to the wrong post:
>
> To compile MeatAxe (with more recent versions of GCC), put '-std=c89'
> (or '-std=gnu89') into CFLAGS, as 'restrict' is a keyword since C99.

I'll keep that in mind. The dead old MeatAxe sources are in the package
only because the new version has totally changed it's API, and so far I
haven't been able to re-write another third-party source on top of the
current MeatAxe version (That's the main stumbling block in creating a
new style version of my spkg).

Best regards,
Simon

[Simon King]

Hi again,

On 2016-07-31, Simon King <simon...@uni-koeln.de> wrote:
> With these changes, it seems that the package builds!

Spoke too soon. It does build. But it can not be used: Accourding to
setup.py, the package builds an extension module pGroupCohomology.cohomology.
But after successfully building the package, the extension module
cannot be imported. That's yet another problem that I haven't had
before.

I have updated
users.minet.uni-jena.de/~king/p_group_cohomology-2.1.5.log

How could that new problem happen/be fixed?

Best regards,
Simon

[Vincent Delecroix]

Be careful that dropping the type from _add_, _sub_ is recent (ticket
#20740). Not included in 7.2.

[Vincent Delecroix]

Apparently there is something weird happening with the cysignals
deprecation (occurring when interrupt.pxi is invoked) when importing the
module. For example you can test the following (that *does* work)

$ cd ${SAGE_LOCAL}/lib/python2.7/site-packages/pGroupCohomology
$ sage -q
sage: import cohomology # warning but fine!!
<WARNING ABOUT interrupt.pxi>
sage:

In order to make it work, you can change *all* occurrences of

include "interrupt.pxi"

from your code with

include "cysignals/signals.pxi"

I did it and it worked (for the import part).

Vincent

[leif]

Well, then conditionally apply a patch regarding the API change to the
sources, based on the Sage version, in spkg-install.


For me the following works with Sage 7.3.beta9, GCC 6.1.0, and the
attached patch to the spkg (now .p1):

$ ./sage -p tmp/p_group_cohomology-2.1.5.p1.spkg
...
$ make build
...
$ ./sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 7.3.beta9, Release Date: 2016-07-22 │
│ Type "notebook()" for the browser-based notebook interface. │
│ Type "help()" for help. │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable. ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Forcing sage-location, probably because a new package was installed.
Updating various hardcoded paths...
(Please wait at most a few minutes.)
DO NOT INTERRUPT THIS.
Done updating paths.
sage: import pGroupCohomology
sage:


-leif

[Jeroen Demeyer]

On 2016-07-31 01:09, Simon King wrote:
> When trying to build the old-style version of my modular group
> cohomology spkg, I now get:
> pGroupCohomology/resolution.c:279:28: fatal error: struct_signals.h: No
> such file or directory
> #include "struct_signals.h"

The problem is this line in setup.py:

from Cython.Distutils import build_ext

This "build_ext" command is an old and semi-broken way to compile Cython
files. In fact, newer versions of Cython have an improved "build_ext"
command and give the warning

/usr/local/src/sage-git/local/src/cython/Cython/Distutils/build_ext.py:20:
UserWarning: Cython.Distutils.build_ext does not properly handle
dependencies and is deprectated. Use Cython.Build.build_ext instead.

The right way to compile Cython files is using cythonize(). But
unfortunately I cannot find the Cython docs, so I cannot point you to
the documentation for that.

[Simon King]

Hi Vincent,

On 2016-07-31, Vincent Delecroix <20100.d...@gmail.com> wrote:

> $ sage -q

What das sage -q do?

> sage: import cohomology # warning but fine!!
><WARNING ABOUT interrupt.pxi>

I haven't gotten a warning but a plain import error.

> In order to make it work, you can change *all* occurrences of
>
> include "interrupt.pxi"

I did that before posting my question. Later, I tried rebuilding the
package, and in contrast to the previous attempt I also did "make"
(which is suggested to do after installing a package). Perhaps that did
the trick: It now works. Great relief!!

Thanks to all who helped me!
Best regardds,
Simon

[Simon King]

Hi Jeroen,

On 2016-07-31, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> The problem is this line in setup.py:
>
> from Cython.Distutils import build_ext
>
> This "build_ext" command is an old and semi-broken way to compile Cython
> files. In fact, newer versions of Cython have an improved "build_ext"
> command and give the warning
>
> /usr/local/src/sage-git/local/src/cython/Cython/Distutils/build_ext.py:20:
> UserWarning: Cython.Distutils.build_ext does not properly handle
> dependencies and is deprectated. Use Cython.Build.build_ext instead.
>
> The right way to compile Cython files is using cythonize(). But
> unfortunately I cannot find the Cython docs, so I cannot point you to
> the documentation for that.

The pages
http://docs.cython.org/en/latest/src/quickstart/build.html
and
http://docs.cython.org/en/latest/src/reference/compilation.html
tell me to use build_ext. So far I haven't been able to find a
cythonize() function.

Best regards,
Simon

[Dima Pasechnik]

On Sunday, July 31, 2016 at 1:18:11 PM UTC+1, Simon King wrote:

Hi Vincent,

On 2016-07-31, Vincent Delecroix <20100.d...@gmail.com> wrote:
> $ sage -q

What das sage -q do? 

Das ist "Keine Sage banner" Option :-)

$ sage -advanced | less

...

  -q                  -- quiet; start with no banner 

...

[Simon King]

Hi Jeroen,

On 2016-07-31, Simon King <simon...@uni-koeln.de> wrote:

> On 2016-07-31, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
>> The problem is this line in setup.py:
>>
>> from Cython.Distutils import build_ext
>>
>> This "build_ext" command is an old and semi-broken way to compile Cython
>> files. In fact, newer versions of Cython have an improved "build_ext"
>> command and give the warning
>>
>> /usr/local/src/sage-git/local/src/cython/Cython/Distutils/build_ext.py:20:
>> UserWarning: Cython.Distutils.build_ext does not properly handle
>> dependencies and is deprectated. Use Cython.Build.build_ext instead.

When I replace
from Cython.Distutils import build_ext
by
from Cython Build import build_ext
I am told that it cannot import name build_ext.

So, that doesn't seem to work, at least not with the Cython version that
is part of Sage.

Best regards,
Simon

[leif]

sage: from Cython.Build import cythonize
sage: cythonize?
Signature: cythonize(module_list, exclude=None, nthreads=0,
aliases=None, quiet=False, force=False, language=None,
exclude_failures=False, **options)
Docstring:
Compile a set of source modules into C/C++ files and return a list
of distutils Extension objects for them.

As module list, pass either a glob pattern, a list of glob patterns
or a list of Extension objects. The latter allows you to configure
the extensions separately through the normal distutils options.

When using glob patterns, you can exclude certain module names
explicitly by passing them into the 'exclude' option.

To globally enable C++ mode, you can pass language='c++'.
Otherwise, this will be determined at a per-file level based on
compiler directives. This affects only modules found based on file
names. Extension instances passed into cythonize() will not be
changed.

For parallel compilation, set the 'nthreads' option to the number
of concurrent builds.

For a broad 'try to compile' mode that ignores compilation failures
and simply excludes the failed extensions, pass
'exclude_failures=True'. Note that this only really makes sense for
compiling .py files which can also be used without compilation.

Additional compilation options can be passed as keyword arguments.
Init docstring: x.__init__(...) initializes x; see help(type(x)) for
signature
File:
/data/Sage/release/devel/sage-7.3.beta9-gcc-6.1.0/local/lib/python2.7/site-packages/Cython-0.24.1-py2.7-linux-x86_64.egg/Cython/Build/Dependencies.py
Type: function


-leif

[Jeroen Demeyer]

On 2016-07-31 22:01, Simon King wrote:
> The pages
> http://docs.cython.org/en/latest/src/quickstart/build.html
> and
> http://docs.cython.org/en/latest/src/reference/compilation.html
> tell me to use build_ext.

Don't believe those pages. That's a documentation bug.

[Jeroen Demeyer]

On 2016-07-31 22:21, Simon King wrote:
> So, that doesn't seem to work, at least not with the Cython version that
> is part of Sage.

Yes, this new build_ext is not yet in any released version, the current
recommendation is to use cythonize().

[Simon King]

Hi Leif,

Am Montag, 1. August 2016 00:31:58 UTC+2 schrieb leif:

sage: from Cython.Build import cythonize
sage: cythonize?

Thanks! Unfortunately it doesn't tell how to actually use cythonise in setup.py. I tried to replace the line
    cmdclass = {'build_ext': build_ext}
by
    cmdclass = {'cythonize': cythonize}
but to no avail.

So, what should I do?
Cheers,
Simon

[leif]

> but to no avail..

? As in the description / docstring; cythonize() *takes* a module_list
and also *returns* a (modified) module_list.


Instead of importing build_ext from Cython, do ("as usual"):

from distutils.command.build_ext import build_ext


Then just change your setup():

setup(
...
ext_modules = cythonize(
... # the module list you already have
),
...

keeping cmd_class as is (it's now just plain disutils' build_ext, no
longer Cython's).


-leif

[Simon King]

Hi Leif,

On 2016-08-01, leif <not.r...@online.de> wrote:

> Simon King wrote:
> ? As in the description / docstring; cythonize() *takes* a module_list
> and also *returns* a (modified) module_list.

Aha! This is what I actually didn't get from the description.

> Instead of importing build_ext from Cython, do ("as usual"):
>
> from distutils.command.build_ext import build_ext
>
>
> Then just change your setup():
>
> setup(
> ...
> ext_modules = cythonize(
> ... # the module list you already have
> ),
> ...
>
> keeping cmd_class as is (it's now just plain disutils' build_ext, no
> longer Cython's).

Good, that seems to work fine!

Thanks a lot for your help and your patience!
Simon

[Jeroen Demeyer]

The Cython documentation has been fixed:

http://docs.cython.org/en/latest/src/reference/compilation.html#compiling-with-distutils