Problems building farey_symbol.pyx on Cygwin

[Jean-Pierre Flori]

Dear all,

Still trying to build Sage on Cygwin, I am now stuck with farey_symbol.pyx (in the Sage spkg).
Quite strangely, at linking time I get undefined references in farey.so to convert_to_* functions wich are available in farey_symbol.so (after fixing the undefined references to gmp functions IIRC).
I tried to tweak the order of the so files on the command lines but did not get more lucky.
I also thought it could be a function name mamgling issue, but trying to changing extern C to extern was not helpful either.

Could anyone give it a try? (Here is Cygwin 1.7.15 on Windows 7 64 bits, gcc/g++ 4.5.3)
That would be nice to confirm that it's not my installation which is completely broken.
I guess one could try this without building all spkg's (not sure what the dependencies really are), by running Cython on farey_symbol.pyx and then g++ on the three farey.cpp/farey_symbol.cpp/sl2z.cpp and then linking them with g++.

Best,
JP

[kcrisman]

On Wednesday, July 18, 2012 8:54:18 AM UTC-4, Jean-Pierre Flori wrote:

Dear all,

Still trying to build Sage on Cygwin, I am now stuck with farey_symbol.pyx (in the Sage spkg).

Keep it up!

 

Quite strangely, at linking time I get undefined references in farey.so to convert_to_* functions wich are available in farey_symbol.so (after fixing the undefined references to gmp functions IIRC).
I tried to tweak the order of the so files on the command lines but did not get more lucky.
I also thought it could be a function name mamgling issue, but trying to changing extern C to extern was not helpful either.

Could anyone give it a try? (Here is Cygwin 1.7.15 on Windows 7 64 bits, gcc/g++ 4.5.3)
That would be nice to confirm that it's not my installation which is completely broken.
I guess one could try this without building all spkg's (not sure what the dependencies really are), by running Cython on farey_symbol.pyx and then g++ on the three farey.cpp/farey_symbol.cpp/sl2z.cpp and then linking them with g++.

Are you sure that all dependencies for this are really in module_list.py?  Cygwin seems to be much stricter about this - you can check out http://trac.sagemath.org/sage_trac/wiki/CygwinPort for older tickets where this was the fix.  Just a guess; obviously you actually understand these things, as opposed to my own halting attempts :) 

[Jean-Pierre Flori]

In fact I somehow solved the problem.
I think there is some black magic dll import/export stuff missing to let ld link the files together on Cygwin.

As a first solution, adding -Wl,--enable-auto-import to extra_compiler_flags in module_list.py seems to be sufficient to fix the problem.
This let the farey stuff compile.
Hopefully this will be the last step before Sage completely compiles, with obviously no guarantee that it actually starts afterward :)

A more proper fix could be to let Cython add _declspec(_dllimport) (approximate spelling...) to the produced header file, not sure about that.

By the way, i guess the farey lines are a missing proper "depends" section including farey.hpp and sl2z.hpp, but that's not the source of my problems.

[Dima Pasechnik]

On Thursday, 19 July 2012 07:35:15 UTC+8, Jean-Pierre Flori wrote:

In fact I somehow solved the problem.
I think there is some black magic dll import/export stuff missing to let ld link the files together on Cygwin.

As a first solution, adding -Wl,--enable-auto-import to extra_compiler_flags in module_list.py seems to be sufficient to fix the problem.

As far as I recall, we did something like this to a number of packages already last year...

 

This let the farey stuff compile.
Hopefully this will be the last step before Sage completely compiles, with obviously no guarantee that it actually starts afterward :)

A more proper fix could be to let Cython add _declspec(_dllimport) (approximate spelling...) to the produced header file, not sure about that.

it could be the case. IMHO Cython is not tested on Cygwin, although I might be wrong there.

Last time I checked (cygwin version 1.7.10 ?) Sage's Cython 

building dynamic modules, even very simple ones, did not work well due to Cygwin

fork problems. 

Even now if I try easy_install cython on Cygwin, I get an error indicating that it has no idea about Cygwin, basically, complaining about absence of Microsoft's C compiler.

Dima

[Dima Pasechnik]

On Thursday, 19 July 2012 13:26:47 UTC+8, Dima Pasechnik wrote:

On Thursday, 19 July 2012 07:35:15 UTC+8, Jean-Pierre Flori wrote:

In fact I somehow solved the problem.
I think there is some black magic dll import/export stuff missing to let ld link the files together on Cygwin.

As a first solution, adding -Wl,--enable-auto-import to extra_compiler_flags in module_list.py seems to be sufficient to fix the problem.

As far as I recall, we did something like this to a number of packages already last year...

 

This let the farey stuff compile.
Hopefully this will be the last step before Sage completely compiles, with obviously no guarantee that it actually starts afterward :)

A more proper fix could be to let Cython add _declspec(_dllimport) (approximate spelling...) to the produced header file, not sure about that.

it could be the case. IMHO Cython is not tested on Cygwin, although I might be wrong there.

Last time I checked (cygwin version 1.7.10 ?) Sage's Cython 

building dynamic modules, even very simple ones, did not work well due to Cygwin

fork problems. 

Even now if I try easy_install cython on Cygwin, I get an error indicating that it has no idea about Cygwin, basically, complaining about absence of Microsoft's C compiler.

however, installing Cython 0.16 from source works on Cygwin 1.7.15, and is able to build some (trivial) dynamic extensions,

so this looks much better now than last year.

Can you post somewhere your Cygwin-specific patches to  Sage (which version, exactly?) you produced so far?

With them in hand, I'll give it a try.

Dima

[Jean-Pierre Flori]

I'll do when I'm finished, for the moment, I'll just keep track of the progress on the wiki page.
Hopefully there is only one problematic file left: expression.pyx.
I've looked into this all night long but failed to solve the problem.

g++ complains about a template instantiation (parse error, blah, cannot find function, blah) using the infinity class (from pynac).
More precisely functions like "bool is_a<infinity>(const &ex)", "ex_to<infinity>" and similar ones.
The link I posted on the wiki page does not seem useful after all, although the problem shall dwell in this part of the code.
What's strange is that the same functions are used with different classes, even earlier in expression.cpp and are not problematic.
Looking at infinity.c/h and e.g. relational.h/c, and what's about them in basic.h/c and ex.h/c I can not spot an obvious difference.
As I'm not really an expert of templates I hope I have missed something obvious.

I'll try to cc Burcin about that, as he's surely the more acquainted with pynac source code and organization.

Best,
JP

[Burcin Erocal]

Hi Jean-Pierre,

The infinity class is a recent addition to pynac by Volker. It is
possible that we missed something there, but I cannot spot any obvious
problems just by looking at the (pynac) code. Can you post the error
messages?


Cheers,
Burcin

[Jean-Pierre Flori]

Here it is:

Executing 9 commands (using 1 thread)
gcc -fno-strict-aliasing -fwrapv -I/usr/include/ncurses -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/home/jp/sage-5.1.rc1/local/include -I/home/jp/sage-5.1.rc1/local/include/csage -I/home/jp/sage-5.1.rc1/devel/sage/sage/ext -I/home/jp/sage-5.1.rc1/local/include/python2.7 -c sage/symbolic/expression.cpp -o build/temp.cygwin-1.7.16-i686-2.7/sage/symbolic/expression.o -Wl,--enable-auto-import -w
cc1plus: attention : l'option de la ligne de commande "-Wstrict-prototypes" est valide pour Ada/C/ObjC mais pas pour C++
sage/symbolic/expression.cpp: In function ‘PyObject* __pyx_f_4sage_8symbolic_10expression_10Expression_pyobject(__pyx_obj_4sage_8symbolic_10expression_Expression*, int)’:
sage/symbolic/expression.cpp:3047:15: erreur: parse error in template argument list
sage/symbolic/expression.cpp:3047:49: erreur: no matching function for call to ‘is_a(GiNaC::ex&)’
sage/symbolic/expression.cpp:3057:17: erreur: parse error in template argument list
sage/symbolic/expression.cpp:3057:52: erreur: no matching function for call to ‘ex_to(GiNaC::ex&)’
sage/symbolic/expression.cpp:3076:17: erreur: parse error in template argument list
sage/symbolic/expression.cpp:3076:52: erreur: no matching function for call to ‘ex_to(GiNaC::ex&)’
sage/symbolic/expression.cpp:3095:17: erreur: parse error in template argument list
sage/symbolic/expression.cpp:3095:52: erreur: no matching function for call to ‘ex_to(GiNaC::ex&)’
sage/symbolic/expression.cpp: In function ‘int __pyx_f_4sage_8symbolic_10expression_10Expression_is_infinity(__pyx_obj_4sage_8symbolic_10expression_Expression*, int)’:
sage/symbolic/expression.cpp:9117:13: erreur: parse error in template argument list
sage/symbolic/expression.cpp:9117:47: erreur: no matching function for call to ‘is_a(GiNaC::ex&)’
sage/symbolic/expression.cpp: In function ‘int __pyx_pf_4sage_8symbolic_10expression_10Expression_52__nonzero__(PyObject*)’:
sage/symbolic/expression.cpp:9514:17: erreur: parse error in template argument list
sage/symbolic/expression.cpp:9514:41: erreur: no matching function for call to ‘is_a(GiNaC::ex&)’
sage/symbolic/expression.cpp:9517:19: erreur: parse error in template argument list
sage/symbolic/expression.cpp:9517:43: erreur: no matching function for call to ‘is_a(GiNaC::ex&)’
error: command 'gcc' failed with exit status 1

Best,
JP

[Jean-Pierre Flori]

FYI, it seems that somehow the infinity class is not bound to GiNaC::infinity on the contrary to what happens to all the other pynac classes.
Not sure why, but maybe there is a name clash with sage.rings.infinity or the infinity variable declared therein.
At least, changing the <infinity> to <GiNaC::infinity> in the cpp file produced by Cython (that's bad, but the quickest way to go on compiling Sage) solved the problem.

After that, I got:
 * undefined references in stl_vectors solved by adding the correct library dependencies in modules_list
 * undefined references in wrapper_rdf because of a functions defined in interp_rdf and seemingly correctly shown by nm in both files. Changing the order of the files in the linking command, and adding -no-undefined flag, did not solve the problem yet.

[Jean-Pierre Flori]

Ok, I solved the (last?) problem with gen_interpreter.py.
It seems that there are incorrect __imp_ prefix added to the symbols of the interp_* functions when wrapper_*.o is built.
Because of that, the functions are looked to be imported from some .dll rather .o file.
Indeed, building an independent .dll for interp_* and then linking wrapper_* to it to produce wrapper_*.dll works.

To only have one wrapper_*.dll I've added an header file for interp_*.c and included it as cdef extern from "interp_*.h" in wrapper_*.pyx rather than without the from part as before.
This seems functional, but needs non-trivial modifications to gen_interpreter.py.

Apart from that, one of the modules built by gen_interpreter.py linked to the mc library which is not available on my system (also had that problem for some module build in module_list.py).

I've relaunched make and let's hope it now finishes.

[Jean-Pierre Flori]

So the sage spkg built successfully, but there are other packaes left.
Gap fails with:
Error creating dummy 'gap.exe' and/or a link from 'gap' to it.
during configuration.

[Dima Pasechnik]

On Thursday, 26 July 2012 01:43:56 UTC+8, Jean-Pierre Flori wrote:

So the sage spkg built successfully, but there are other packaes left.
Gap fails with:
Error creating dummy 'gap.exe' and/or a link from 'gap' to it.
during configuration.

maybe the hack from 

http://trac.sagemath.org/sage_trac/attachment/ticket/7041/gap-4.4.12.p6-p7.diff

is no longer needed?

Or is it a symlink creation problem? (permissions, etc...)

Dima

[Jean-Pierre Flori]

maybe the hack from 

http://trac.sagemath.org/sage_trac/attachment/ticket/7041/gap-4.4.12.p6-p7.diff

is no longer needed?

It seems that gap is indeed automagicall y resolved to gap.exe.
So on my version at least it is not needed anymore.

[Jean-Pierre Flori]

Now stuck at the end of the Maxima spkg installation.
Some forks (cannot commit memory) problems.
I've rebased all my system but this still errors out.
I'll just skip that spkg now by touching spkg/installed/maxima... and retry later.
In the worst case, it should be possible to let the install go until the forking problem, rebase then, finish the installation and trick Sage to think this was completely correctly installed by touching spkg/installed/maxima...

[kcrisman]

You may want to try building the absolute latest ECL and Maxima (on
ECL) from scratch on your Cygwin and see if anything is different.
Juanjo does want ECL to work and build things like Maxima on Cygwin.
Weirdly, the whole forking stuff should no longer have been an issue,
there were previous upgrades that should have dealt with that. See
#11884, and possibly #11502 (but probably unrelated).

[Jean-Pierre Flori]

Just in case this is useful to you, here is the tail of the log:
make[4] : on quitte le répertoire « /home/jp/sage-5.1.rc1/spkg/build/maxima-5.26.0.p0/src/interfaces »
make[3] : on quitte le répertoire « /home/jp/sage-5.1.rc1/spkg/build/maxima-5.26.0.p0/src/interfaces »
Making install in share
make[3] : on entre dans le répertoire « /home/jp/sage-5.1.rc1/spkg/build/maxima-5.26.0.p0/src/share »
make[4] : on entre dans le répertoire « /home/jp/sage-5.1.rc1/spkg/build/maxima-5.26.0.p0/src/share »
make[4]: Rien à faire pour « install-exec-am ».
/home/jp/sage-5.1.rc1/spkg/build/maxima-5.26.0.p0/src/install-sh -d /home/jp/sage-5.1.rc1/local/share/maxima/5.26.0/share
      1 [main] sh 434132 child_info_fork::abort: can't commit memory for stack 0x289000(94208), Win32 error 487
/bin/sh: fork: retry: Resource temporarily unavailable
      1 [main] sh 434696 child_info_fork::abort: can't commit memory for stack 0x289000(94208), Win32 error 487
/bin/sh: fork: retry: Resource temporarily unavailable
      1 [main] sh 434304 child_info_fork::abort: can't commit memory for stack 0x289000(94208), Win32 error 487
/bin/sh: fork: retry: Resource temporarily unavailable
      1 [main] sh 433424 child_info_fork::abort: can't commit memory for stack 0x289000(94208), Win32 error 487
/bin/sh: fork: retry: Resource temporarily unavailable
      2 [main] sh 434736 child_info_fork::abort: can't commit memory for stack 0x289000(94208), Win32 error 487
/bin/sh: fork: Resource temporarily unavailable
Makefile:1578: recipe for target `install-datafiles' failed
make[4]: *** [install-datafiles] Error 254
make[4] : on quitte le répertoire « /home/jp/sage-5.1.rc1/spkg/build/maxima-5.26.0.p0/src/share »
Makefile:1494: recipe for target `install-am' failed
make[3]: *** [install-am] Error 2
make[3] : on quitte le répertoire « /home/jp/sage-5.1.rc1/spkg/build/maxima-5.26.0.p0/src/share »
Makefile:320: recipe for target `install-recursive' failed
make[2]: *** [install-recursive] Error 1
make[2] : on quitte le répertoire « /home/jp/sage-5.1.rc1/spkg/build/maxima-5.26.0.p0/src »
***********************************************************
Error: Failed to install Maxima.
***********************************************************

Not much time to investigate further right now.

[Dima Pasechnik]

any BLODA on your machine, by any chance?

http://cygwin.com/faq/faq.using.html#faq.using.bloda

[Jean-Pierre Flori]

On Thursday, July 26, 2012 4:30:20 PM UTC+2, Dima Pasechnik wrote:

any BLODA on your machine, by any chance?

http://cygwin.com/faq/faq.using.html#faq.using.bloda

Nothing on the list there at least.
I also uninstalled some Lenovo stuff but Cygwin and MinGW are still quite slow.

Rebasing between the build and install stuff was enough to properly make Maxima.
Afterwards I had no further issue, except that Sage does not launch properly.
More precisely, the principal software interfaces work ok (singular, python, ipython, maxima, ecl, gap, sh), and can compute 1+1.
But somehow when launching the real sage interface ipython seems unable to locate the dll of the cython compiled module (although they are right in plac ein local/lib/python2.7/site-packages/sage/...).
Strange.
The main failure is about importing current_randtest in sage;misc.randtest , but I guess this is because it is the first dll which gets loaded.
Then the prompt is kind of broken, but has apparently ipython working ok (it can compute 1+1).

[Dima Pasechnik]

On Friday, 27 July 2012 05:59:43 UTC+8, Jean-Pierre Flori wrote:

On Thursday, July 26, 2012 4:30:20 PM UTC+2, Dima Pasechnik wrote:

any BLODA on your machine, by any chance?

http://cygwin.com/faq/faq.using.html#faq.using.bloda

Nothing on the list there at least.
I also uninstalled some Lenovo stuff but Cygwin and MinGW are still quite slow.

Rebasing between the build and install stuff was enough to properly make Maxima.
Afterwards I had no further issue, except that Sage does not launch properly.
More precisely, the principal software interfaces work ok (singular, python, ipython, maxima, ecl, gap, sh), and can compute 1+1.
But somehow when launching the real sage interface ipython seems unable to locate the dll of the cython compiled module (although they are right in plac ein local/lib/python2.7/site-packages/sage/...).

maybe something needs to be symlinked in local/bin rather than in local/lib ?

or try to add these local/lib/python2.7/site-packages/sage/ to PATH

(or to LD_LIBRARY_PATH?!)

 

Strange.

well, that's a typical for Cygwin mixup, when dll's are "libs", but actually for windows they are executables...

I recall seeing something like this...

[Jean-Pierre Flori]

well, that's a typical for Cygwin mixup, when dll's are "libs", but actually for windows they are executables...

I recall seeing something like this...

 

The main failure is about importing current_randtest in sage;misc.randtest , but I guess this is because it is the first dll which gets loaded.
Then the prompt is kind of broken, but has apparently ipython working ok (it can compute 1+1).

Just found http://trac.sagemath.org/sage_trac/ticket/12104

[Jean-Pierre Flori]

As in #12104, libntl.dll could not be found.
Is it because it was moved rather than copied to libntl.dll.a in #9050?

[Jean-Pierre Flori]

On Friday, July 27, 2012 3:19:38 PM UTC+2, Jean-Pierre Flori wrote:

As in #12104, libntl.dll could not be found.
Is it because it was moved rather than copied to libntl.dll.a in #9050?

And I can now start Sage!
I don't seem to suffer from #11551.

[Jean-Pierre Flori]

Tada!

[Charles Bouillaguet]

I'm not using Windows, but I share your excitement :)

Charles

--
--
To post to this group, send an email to sage-...@googlegroups.com
To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
 
 
 

[Jean-Pierre Flori]

On Friday, July 27, 2012 3:40:48 PM UTC+2, Charles Bouillaguet wrote:

I'm not using Windows, but I share your excitement :)

Charles

In fact I don't either :)

By the way I could not stop the notebook server.

[Dima Pasechnik]

On Friday, 27 July 2012 21:39:32 UTC+8, Jean-Pierre Flori wrote:

Tada!

not bad! So that startup crash where we were stuck last year apparently went away (Cygwin upgrade? Sage updates? Favourable

disposition  of planets? :–)

[Dima Pasechnik]

do you mean it doesn't get your Ctrl-C?

This is very minor, I think.

Try running tests :–)

 

[kcrisman]

Tada!

Wow!  And it computes?  The 1+1 in Ipython would be an int, after all; is 1/2 = 0 or 1/2?

 

not bad! So that startup crash where we were stuck last year apparently went away (Cygwin upgrade? Sage updates? Favourable

disposition  of planets? :–)

Almost certainly the latter. 

[Andrea Lazzarotto]

2012/7/27 kcrisman <kcri...@gmail.com>

Almost certainly the latter. 

You added an "almost" which is not necessary since we're talking about Windows. :D

--
Andrea Lazzarotto - http://andrealazzarotto.com

[Jean-Pierre Flori]

On Friday, July 27, 2012 4:15:10 PM UTC+2, kcrisman wrote:

Tada!

Wow!  And it computes?  The 1+1 in Ipython would be an int, after all; is 1/2 = 0 or 1/2?

 

I was able to compute the cardinality on an elliptic curve over F_2003 and over F_p where p was next_prime(2^100) which should use PARI.
I kind of afraid of what a make ptestlong could output.

About the NTL stuff, it might be solved by using the stuff provided by NTL itself rather than what we do in the spkg.
NTL uses libtool for shared library, and I get no problem in Cygwin with that (or just little problems).

[kcrisman]

Cygwin is a very special case - people would not ordinarily be building from scratch - so we can ask for additional prereqs in that case anyway.

Please please please start opening tickets for the things that you discovered (I am glad you did everything at http://trac.sagemath.org/sage_trac/wiki/CygwinPort too) so we can actually get some of them merged in Sage as spkgs etc.  I will not be able to test them immediately but even solving half of them would be fantastic.

Also, did your Python build have any problems?  I not only often had to rebase, but also there were some "bits" that never built - mostly harmless, but  sage-gdb didn't work, which was bad.  See earlier on the Cygwin port page :)

[Jean-Pierre Flori]

Also, did your Python build have any problems?  I not only often had to rebase, but also there were some "bits" that never built - mostly harmless, but  sage-gdb didn't work, which was bad.  See earlier on the Cygwin port page :)

I had some problems with Python mentioned on the Cygwin port page, but the spkg finally finished installing without anything too hackish.
I did not really pay attention to the log at that point.

And indeed, sage -gdb fails  with:
Reading symbols from /home/jp/sage-5.1.rc1/local/bin/python...done.
/home/jp/sage-5.1.rc1/local/bin/sage-gdb-commands:1: Error in sourced command file:
Error creating process /home/jp/sage-5.1.rc1/local/bin/python, (error 87).
(gdb)

[Dima Pasechnik]

On Friday, 27 July 2012 22:33:55 UTC+8, Jean-Pierre Flori wrote:

On Friday, July 27, 2012 4:15:10 PM UTC+2, kcrisman wrote:

Tada!

Wow!  And it computes?  The 1+1 in Ipython would be an int, after all; is 1/2 = 0 or 1/2?

 

I was able to compute the cardinality on an elliptic curve over F_2003 and over F_p where p was next_prime(2^100) which should use PARI.
I kind of afraid of what a make ptestlong could output.

the crucial thing to see is whether we will see fork() failures.

If not, we are in business. If yes, one might need to rebase all AND the dlls that were built with Sage,

and try again...

[Dima Pasechnik]

On Friday, 27 July 2012 23:07:34 UTC+8, Dima Pasechnik wrote:

On Friday, 27 July 2012 22:33:55 UTC+8, Jean-Pierre Flori wrote:

On Friday, July 27, 2012 4:15:10 PM UTC+2, kcrisman wrote:

Tada!

Wow!  And it computes?  The 1+1 in Ipython would be an int, after all; is 1/2 = 0 or 1/2?

 

I was able to compute the cardinality on an elliptic curve over F_2003 and over F_p where p was next_prime(2^100) which should use PARI.
I kind of afraid of what a make ptestlong could output.

the crucial thing to see is whether we will see fork() failures.

If not, we are in business. If yes, one might need to rebase all AND the dlls that were built with Sage,

and try again...

I started doing the same on my system, and the 1st obstacle I ran at was a fork() failure during the building of Python spkg. Well, this is OK, still. Probably can be fought with using rebase/rebaseall.

What definitely has to be fixed, though, is blatant copying of Cygwin system dlls into SAGE_LOCAL/bin and SAGE_LOCAL/lib.

Having two copies of a same dll is a huge no-no.

Also, it's totally unneeded. E.g. this is done by readline and sqllight spkgs.

It's easy to fix this along the fix in the patch spkg, where we just 

do not build readline on cygwin.