Re: Cleaner build process for my spkg

[William Stein]

On Wed, Aug 26, 2009 at 3:41 PM, Simon King<simon...@nuigalway.ie> wrote:
> Dear William,
>
> John Palmieri reported a problem building our cohomology on an Intel
> Mac. The error message said that the resource libmtx.a could not be
> accessed.
>
> Meanwhile I changed the build process for my package, in the hope that
> it makes things more stable.
>
> Does this imply that a thorough build test now became necessary?

That seems to. Did you fix his problem?

>
> By the way, as you suggested, I relocated the install-wide cohomology
> database from SAGE_LOCAL to SAGE_DATA. Probably I am the only Sage user
> who has cohomology data in SAGE_LOCAL, so, I guess that a warning
> message "please move your old data from
> SAGE_ROOT/local/pGroupCohomology/db to SAGE_DATA/pGroupCohomology/" is
> not needed.

Yes, that sounds fine to me.

William

[Simon King]

Hi William,

On Aug 26, 11:47 pm, William Stein <wst...@gmail.com> wrote:
[...]

> > Does this imply that a thorough build test now became necessary?
>
> That seems to.  Did you fix his problem?

I don't know. The point is that I have no access to any Mac, and I
have never seen this error message on any other machine. But John's
report pointed me to some oddity in my building the C-MeatAxe library,
and now it seems smoother.

So, if someone could test it for me, I'd appreciate.

By the way, this is about ticket #6750.

Best regards,
Simon

[Minh Nguyen]

Hi Simon,

On Thu, Aug 27, 2009 at 8:58 AM, Simon King<simon...@nuigalway.ie> wrote:

<SNIP>

> I don't know. The point is that I have no access to any Mac, and I
> have never seen this error message on any other machine. But John's
> report pointed me to some oddity in my building the C-MeatAxe library,
> and now it seems smoother.
>
> So, if someone could test it for me, I'd appreciate.

This harks back to the build problem with cliquer. I can test spkg's
under any Linux boxes that I have access to. But not Mac boxes, which
is a shame as many people are using OS X.

--
Regards
Minh Van Nguyen

[Simon King]

Hi Minh, and Hi Mac Experts,

On Aug 27, 12:34 am, Minh Nguyen <nguyenmi...@gmail.com> wrote:
> This harks back to the build problem with cliquer. I can test spkg's
> under any Linux boxes that I have access to. But not Mac boxes, which
> is a shame as many people are using OS X.

Now I feel less lonely. I thought I am the only sage developer without
a Mac :-)

But seriously, I would appreciate if some Mac expert could try and
help me to hunt down the build problem on Intel Mac!

The reasons why I am puzzled:

* I understood that William did a successful test on many platforms,
presumably including Intel Mac, of a previous package version that now
is an optional package, installable by
sage: install_package('p_group_cohomology')
AFAIK, I did not change the part that now fails to build.

* The error reported by John Palmieri was
{{{
ar rv /Applications/sage_builds/sage-4.1.1-binary/spkg/build/
p_group_cohomology-1.1/src/lib/libmtx.a os.o
ar rv /Applications/sage_builds/sage-4.1.1-binary/spkg/build/
p_group_cohomology-1.1/src/lib/libmtx.a profile.o
ar: /Applications/sage_builds/sage-4.1.1-binary/spkg/build/
p_group_cohomology-1.1/src/lib/libmtx.a: Resource temporarily
unavailable
make[2]: *** [/Applications/sage_builds/sage-4.1.1-binary/spkg/build/
p_group_cohomology-1.1/src/lib/libmtx.a(profile.o)] Error 1
make[2]: *** Waiting for unfinished jobs....
a - os.o
}}}

What does it mean, "Resource temporarily unavailable"? Does the Mac
try a parallel build, so that two different parts of the build process
try to write into libmtx.a at the same time? Is it a write permission
issue?

Best regards,
Simon

[John H Palmieri]

I now think it's a parallel build issue. It turns out that I have
"$MAKE" set to "make -j2", and when I unset this, the build process
gets past this point. It now claims to have been installed
successfully, so now I'll test it out.

Should the use of "make -jN" be checked for in the spkg-install file
in this case, since it seems to lead to problems?

John

[Simon King]

Hi John,

On Aug 27, 4:15 pm, John H Palmieri <jhpalmier...@gmail.com> wrote:
[...]

> Should the use of "make -jN" be checked for in the spkg-install file
> in this case, since it seems to lead to problems?

Perhaps. But frankly I don't know how one does those things. I am
really not good at writing shell scripts.

And what should be done if it is set? Can one temporarily unset it
during the build? How?

Cheers,
Simon

[William Stein]

Yes, this is done in many spkg's actually.

William

[Simon King]

Hi William,

On Aug 27, 6:46 pm, William Stein <wst...@gmail.com> wrote:
> > Should the use of "make -jN" be checked for in the spkg-install file
> > in this case, since it seems to lead to problems?
>
> Yes, this is done in many spkg's actually.

Can you name one of them, so that I can learn from reading?

By the way, it seems that there is also an issue with 64 versus 32 bit
architecture on Intel Mac. John pointed out to me how this might be
solved. I hope it works.

Cheers,
Simon

[William Stein]

On Thu, Aug 27, 2009 at 10:56 AM, Simon King<simon...@nuigalway.ie> wrote:
>
> Hi William,
>
> On Aug 27, 6:46 pm, William Stein <wst...@gmail.com> wrote:
>> > Should the use of "make -jN" be checked for in the spkg-install file
>> > in this case, since it seems to lead to problems?
>>
>> Yes, this is done in many spkg's actually.
>
> Can you name one of them, so that I can learn from reading?

I think "python" is one of them.

You won't like what you see -- basically for those packages I think we
use "make" instead of $MAKE. That's it. It sort of defeats the
whole purpose of $MAKE..., but it works.

>
> By the way, it seems that there is also an issue with 64 versus 32 bit
> architecture on Intel Mac. John pointed out to me how this might be
> solved. I hope it works.
>
> Cheers,
> Simon
>
> >
>

--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

[Simon King]

Hi William!

On Aug 27, 7:01 pm, William Stein <wst...@gmail.com> wrote:
[...]

> I think "python" is one of them.
>
> You won't like what you see -- basically for those packages I think we
> use "make" instead of $MAKE.  That's it.    It sort of defeats the
> whole purpose of $MAKE..., but it works.

Great, thank you!

And since I am not "spoiled" by good style, I doubt that I will miss
$MAKE. But indeed, $MAKE is used in the Makefile of the C-MeatAxe, and
therefore defining MAKE=make in spkg-install could help.

Best regards,
Simon

[Simon King]

On Aug 27, 8:05 pm, Simon King <simon.k...@nuigalway.ie> wrote:
> And since I am not "spoiled" by good style, I doubt that I will miss
> $MAKE. But indeed, $MAKE is used in the Makefile of the C-MeatAxe, and
> therefore defining MAKE=make in spkg-install could help.

For the record: John Palmieri reported that the spkg now builds fine
on Intel Mac, both 32 and 64 bit, and whether "MAKE" is set to "make -
j2" or not. Failing doc tests are apparently due to using unsorted
dictionaries or sets, but this should be easy to fix.

Thank you very much for all your hints!

But back to my original question to William:
Does it suffice if the Intel Mac problem reported by John is fixed and
the spkg still builds on sage.math, or do you think the package should
(after fixing the set/dictionary doc tests) be tested on your whole
build park? Or would you leave this decision to the reviewers?

Best regards,
Simon

[Minh Nguyen]

Hi Simon,

On Fri, Aug 28, 2009 at 8:48 PM, Simon King<simon...@nuigalway.ie> wrote:

<SNIP>

> For the record: John Palmieri reported that the spkg now builds fine

> on Intel Mac, both 32 and 64 bit, and whether "MAKE" is set to "make -
> j2" or not.

Cool! Good to know.

> Failing doc tests are apparently due to using unsorted
> dictionaries or sets, but this should be easy to fix.

I suspect those failed doctests should be easy to fix. Don't take my
words for it, since I don't have access to a Mac.

> Thank you very much for all your hints!
>
> But back to my original question to William:
> Does it suffice if the Intel Mac problem reported by John is fixed and
> the spkg still builds on sage.math, or do you think the package should
> (after fixing the set/dictionary doc tests) be tested on your whole
> build park? Or would you leave this decision to the reviewers?

The top-level README.txt claims that Sage is regularly tested on the
following platforms:

PROCESSOR OPERATING SYSTEM
x86 32-bit Linux -- Debian, Ubuntu, CentOS (=Redhat),
Fedora Core, OpenSuse, Mandriva
x86_64 64-bit Linux -- Debian, Ubuntu, CentOS (=Redhat),
Fedora Core, OpenSuse, Mandriva
ia64 itanium2 64-bit Linux -- Redhat, Suse
x86 Apple Mac OS X 10.5.x
ppc Apple Mac OS X 10.5.x

So it really depends on one's access to any of these platforms. All a
reviewer can say is that something works on those platforms that they
have access to. If a patch is specific to Mac OS X, then the person
can review the code. But to actually test the code in practice,
someone needs to take it on a test drive on a Mac box. Testing on all
supported platforms should happen before announcing the stable
release, as doing so catches any platform-specific problems. In the
end, it boils down to people and the release manager(s) having access
to a build farm or different platforms for testing. I don't have
access to a Mac box, so I rely on other people to test Mac specific
code.

[William Stein]

On Sun, Aug 30, 2009 at 6:01 AM, Simon King <simon...@nuigalway.ie> wrote:

Hi William,

PS: Although database_gap...spkg was possible to install, the
installation of our cohomology package failed -- because Sage (for
whatever reason) believes that cliquer needs to be available. See below.

Should I follow the advice and inform sage-devel, or is this a known
problem?

This is a known problem.  You can get around it by commenting out the line that imports cliquer in

   devel/sage/sage/graphs/all.py

That's what I did for the OS X 64-bit binary that is posted.

William
 

Sheers,
Simon

Traceback (most recent call last):
 File "setup.py", line 21, in <module>
   from sage.all import SAGE_ROOT
 File
"/scratch/king/SAGE64/sage-4.1.1/local/lib/python2.6/site-packages/sage/all.py", line 85, in <module>
   from sage.graphs.all     import *
 File
"/scratch/king/SAGE64/sage-4.1.1/local/lib/python2.6/site-packages/sage/graphs/all.py", line 9, in <module>
   from sage.graphs.cliquer import *

ImportError:
dlopen(/scratch/king/SAGE64/sage-4.1.1/local/lib/python2.6/site-packages/sage/graphs/cliquer.so, 2): Symbol not found: _graph_new
 Referenced
from: /scratch/king/SAGE64/sage-4.1.1/local/lib/python2.6/site-packages/sage/graphs/cliquer.so
 Expected in: dynamic lookup

real    0m18.212s
user    0m8.624s
sys     0m3.819s
sage: An error occurred while installing p_group_cohomology-1.1
Please email sage-devel http://groups.google.com/group/sage-devel

[Minh Nguyen]

On Mon, Aug 31, 2009 at 1:45 AM, William Stein<wst...@gmail.com> wrote:

<SNIP>

> This is a known problem. You can get around it by commenting out the line
> that imports cliquer in
>
> devel/sage/sage/graphs/all.py
>
> That's what I did for the OS X 64-bit binary that is posted.

After some experimentation and reading, I got Sage 4.1.1 to build in
64-bit mode under OS X 10.5.8. I used Michael Abshoff's custom-built
Fortran spkg as documented at

http://wiki.sagemath.org/osx64

I have written up the steps. You can see them at

http://mvngu.wordpress.com/2009/09/02/compile-sage-4-1-in-64-bit-mode-on-os-x-10-5-8/

[Simon King]

Hi Minh,

On Sep 2, 8:41 am, Minh Nguyen <nguyenmi...@gmail.com> wrote:
[...]

> After some experimentation and reading, I got Sage 4.1.1 to build in
> 64-bit mode under OS X 10.5.8. I used Michael Abshoff's custom-built
> Fortran spkg as documented at
>
> http://wiki.sagemath.org/osx64
>
> I have written up the steps. You can see them at
>

> http://mvngu.wordpress.com/2009/09/02/compile-sage-4-1-in-64-bit-mode...

There is a slightly different way, that was explained to me by William
off list. There is no need to download the extra fortran package
manually, so, perhaps it is very slightly easier.

1. Get the sources, untar them.
2. $ export SAGE64=yes
$ cd sage-4.1/
$ make
3. Wait until it fails.
4. $ ./sage -i fortran-OSX64-20090120
First it was a bit surprising to me, but "./sage -i" works even
though the build process failed.
5. $ make
This time it should work
6. Comment out the line that imports cliquer in SAGE_ROOT/devel/sage/
sage/graphs/all.py

Now ./sage works.

Cheers,
Simon