Building Sage 7.2 from source: Round 2

[paulmasson]

Running OS X El Capitan on a year-old Macbook.

Deleted my entire Sage directory. Did a clean clone from github.com/sagemath/sage.git. Immediately changed to Sage directory and ran make. Same error as previously:

error: command 'gcc' failed with exit status 1

/Users/Masson/downloads/github/sage/src/build/cythonized/sage/combinat/partitions.cpp:323:26: fatal error: partitions_c.h: No such file or directory

 #include "partitions_c.h"

                          ^

compilation terminated.

make[3]: *** [sage] Error 1

Is there a problem building from the github copy?

[Dima Pasechnik]

please show the output of

git remote -v

git status

and

the top commit (the top part of the output of )

git log

Do you have homebrew or other 3rd party environment like this installed?

Are your Xcode (tools) up to date?

[paulmasson]

Turns out this issue has nothing to do with the Github copy. I downloaded a 7.2 tarball and had the same issue a total of nine times. Here are the successive error messages:

/Users/Masson/desktop/sage/src/build/cythonized/sage/combinat/partitions.cpp:323:26: fatal error: partitions_c.h: No such file or directory

 #include "partitions_c.h"

/Users/Masson/desktop/sage/src/build/cythonized/sage/combinat/matrices/dancing_links.cpp:319:29: fatal error: dancing_links_c.h: No such file or directory

 #include "dancing_links_c.h"

/Users/Masson/desktop/sage/src/build/cythonized/sage/graphs/base/boost_graph.cpp:314:31: fatal error: boost_interface.cpp: No such file or directory

 #include "boost_interface.cpp"

/Users/Masson/desktop/sage/src/build/cythonized/sage/libs/ecl.c:324:20: fatal error: eclsig.h: No such file or directory

 #include "eclsig.h"

/Users/Masson/desktop/sage/src/build/cythonized/sage/libs/ppl.cpp:336:23: fatal error: ppl_shim.hh: No such file or directory

 #include "ppl_shim.hh"

/Users/Masson/desktop/sage/src/build/cythonized/sage/libs/eclib/mwrank.cpp:356:20: fatal error: wrap.cpp: No such file or directory

 #include "wrap.cpp"

/Users/Masson/desktop/sage/src/build/cythonized/sage/matroids/matroid.c:296:22: fatal error: minorfix.h: No such file or directory

 #include "minorfix.h"

/Users/Masson/desktop/sage/src/build/cythonized/sage/misc/darwin_utilities.c:290:33: fatal error: darwin_memory_usage.h: No such file or directory

 #include "darwin_memory_usage.h"

/Users/Masson/desktop/sage/src/build/cythonized/sage/symbolic/pynac.cpp:334:22: fatal error: pynac_cc.h: No such file or directory

 #include "pynac_cc.h"

After each failed build I manually copied each missing file to the appropriate directory and ran 'make' again. I now have a running build of 7.2. Yippee!

A friend has suggested that the errors might be caused by bad line endings that only show up on a Mac. Any other explanations for such apparently random behavior?

[Volker Braun]

These are copied during the build process:

$ egrep -r partitions_c.h logs/install.log 

copying /home/vbraun/Sage/git-develop/src/build/cythonized/sage/combinat/partitions_c.h -> /home/vbraun/Sage/git-develop/local/lib/python2.7/site-packages/sage/combinat

Can you post your logs/pkgs/sagelib-7.3.beta2.log and whats the output of 

sage: from sage_setup.find import find_extra_files

sage: from sage.env import SAGE_SRC, SAGE_CYTHONIZED

sage: find_extra_files(["sage.combinat"], SAGE_SRC, SAGE_CYTHONIZED, ".")

[('./sage/combinat',

  ['/home/vbraun/Code/sage/src/sage/combinat/combinat_cython.pxd',

   '/home/vbraun/Code/sage/src/sage/combinat/enumeration_mod_permgroup.pxd',

   '/home/vbraun/Code/sage/src/sage/combinat/permutation_cython.pxd',

   '/home/vbraun/Code/sage/src/build/cythonized/sage/combinat/partitions_c.h'])]

[Justin C. Walker]

I haven’t followed this thread, but it seems quite strange to me.

> On May 29, 2016, at 15:02 , paulmasson <paulm...@comcast.net> wrote:
>
> Turns out this issue has nothing to do with the Github copy. I downloaded a 7.2 tarball and had the same issue a total of nine times. Here are the successive error messages:

I have not checked every one of these errors, but in the cases I have checked, the missing files are present in a fresh unpacking of the tarball.

Maybe I missed something earlier in the discussion, but have you checked whether they are present before you start a build?

Justin

--
Justin C. Walker
Curmudgeon-at-large
--
Network, n., Difference between work
charged for and work done

[paulmasson]

Justin, the files are present in src/sage before the build. They just don't get copied to src/build/cythonized/sage/ during the build for whatever reason.

[paulmasson]

Volker, the line you have in logs/install.log does not exist for me when the build halts. Just this:

$ egrep -r partitions_c.h logs/install.log

logs/install.log:/Users/Masson/downloads/github/sage/src/build/cythonized/sage/combinat/partitions.cpp:323:26: fatal error: partitions_c.h: No such file or directory

logs/install.log: #include "partitions_c.h"

I'll save a copy of the install log at the point of halting if you want that as well.

There is no sagelib7.3-beta*.log when the build halts. Attaching sagelib-7.2.log instead if that helps.

When the build halts I cannot run Sage to give you the other output. In the working version of 7.2, the output of the indicated commands is exactly the same as yours, but that is because I manually made sure the missing files were put in place.

These nine files are just not getting copied during the build on my machine.

[Dima Pasechnik]

Certainly, there is something unusual about your system/setup; that's why I asked whether you have anything like homebrew (and, perhaps, an antivirus?) installed. Have you?

As well, does /Users/Masson/ live on some non-standard file system? 

Thanks,

Dima

[Volker Braun]

You can run the script with your system python directly in your Sage directory, e.g.:

$ python

Python 2.7.11 (default, Mar 31 2016, 20:46:51) 

[GCC 5.3.1 20151207 (Red Hat 5.3.1-2)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> import sys

>>> sys.path.append('src')

>>> from sage_setup.find import find_extra_files

>>> find_extra_files(["sage.combinat"], 'src', 'src/build/cythonized', '.')

[('./sage/combinat', ['src/sage/combinat/combinat_cython.pxd', 'src/sage/combinat/enumeration_mod_permgroup.pxd', 'src/sage/combinat/permutation_cython.pxd', 'src/build/cythonized/sage/combinat/partitions_c.h'])]

[Volker Braun]

Extra logging at http://trac.sagemath.org/ticket/20712 (needs review)

[paulmasson]

Dima,  I'm not running any package managers or antivirus software on my Macbook. The only third-party software that might be relevant is the Github desktop client.

My installation of OS X is very standard. No additional partitions or virtual environments. Straight from the box setup.

[paulmasson]

Volker:

MacBook-Retina:sage Masson$ python

Python 2.7.10 (default, Oct 23 2015, 19:19:21) 

[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import sys

>>> sys.path.append('src')

>>> from sage_setup.find import find_extra_files

>>> find_extra_files(["sage.combinat"], 'src', 'src/build/cythonized', '.')

[('./sage/combinat', ['src/sage/combinat/combinat_cython.pxd', 'src/sage/combinat/enumeration_mod_permgroup.pxd', 'src/sage/combinat/permutation_cython.pxd'])]

[Volker Braun]

So it seems that cythonize doesn't pick up the dependency for whatever reason. Can you post your generated cython output, that is, src/build/cythonized/sage/combinat/partitions.cpp? Maybe there are more differences that'll tell us whats going on.

[paulmasson]

[paulmasson]

I have a partial answer as to what is causing these errors. Before starting 'make' I switched to the Sage directory using 'cd desktop/sage', as you can see by the small 'd' in 'desktop'. On a Mac that directory is most properly 'Desktop' with a capital 'D'. I revisited the build that halted by switching to the Sage directory with 'cd Desktop/sage' and again running 'make', and now everything copies and builds just fine.

That leaves open the question of why the build process is allowed to mostly work with the slightly wrong directory name, and how to flag the error before building starts.

[Dima Pasechnik]

On Tuesday, May 31, 2016 at 7:37:09 PM UTC+1, paulmasson wrote:

I have a partial answer as to what is causing these errors. Before starting 'make' I switched to the Sage directory using 'cd desktop/sage', as you can see by the small 'd' in 'desktop'. On a Mac that directory is most properly 'Desktop' with a capital 'D'. I revisited the build that halted by switching to the Sage directory with 'cd Desktop/sage' and again running 'make', and now everything copies and builds just fine.

oy gevalt :-( Well, OSX case-insensitive file systems (legacy of OSY with Y<10) lend you a very helping hand to shoot yourself in the foot.

That leaves open the question of why the build process is allowed to mostly work with the slightly wrong directory name, and how to flag the error before building starts.

well, it's because of inability to distinguish right from wrong, apparently, due to oh so helpful case-insensitivity "feature".

[Dima Pasechnik]

in more detail, build scrips routinely use `pwd` to figure out the current directory. Now, watch this (on OSX):

nash:~ dima$ cd desktop

nash:desktop dima$ pwd

/Users/dima/desktop

nash:desktop dima$ cd    

nash:~ dima$ cd Desktop 

nash:Desktop dima$ pwd

/Users/dima/Desktop 

No wonder these "slightly wrong" paths get into configurations, and then some tools, which are case-sensitive, choke on them...

[Volker Braun]

Yes, there is not much we can do about the case-sensitivity "feature" in OSX. Probably the best course of action would be to raise an error as soon as possible instead of attempting to build.

[Dima Pasechnik]

On Wednesday, June 1, 2016 at 12:26:31 AM UTC+1, Volker Braun wrote:

Yes, there is not much we can do about the case-sensitivity "feature" in OSX. Probably the best course of action would be to raise an error as soon as possible instead of attempting to build.

I just asked : http://apple.stackexchange.com/questions/240245/checking-path-capitalization-of-pwd

We'll see, if there will be no reply, we'd work out ourselves a quick check that we can do in the top makefile or some more appropriate way.

[Dima Pasechnik]

On Wednesday, June 1, 2016 at 8:10:26 AM UTC+1, Dima Pasechnik wrote:

On Wednesday, June 1, 2016 at 12:26:31 AM UTC+1, Volker Braun wrote:

Yes, there is not much we can do about the case-sensitivity "feature" in OSX. Probably the best course of action would be to raise an error as soon as possible instead of attempting to build.

I just asked : http://apple.stackexchange.com/questions/240245/checking-path-capitalization-of-pwd

We'll see, if there will be no reply, we'd work out ourselves a quick check that we can do in the top makefile or some more appropriate way.

OK, so they suggest to use `/bin/pwd -P` as a way to get the correct capitalisation, and it does work.
So, perhaps adding comparison of this output with the one from `pwd` is a good idea, to insert somewhere
as a sanity check.


Sanitising all the Sage scripts to use this instead of plain pwd is perhaps too much trouble...