Re: [sage-release] release management: merging tickets

[Robert Miller]

On Sun, Feb 7, 2010 at 4:42 AM, Minh Nguyen <nguye...@gmail.com> wrote:
> Before starting on merging tickets for a new milestone, I would
> compile the source tarball of the current stable release of Sage under
> /scratch/. In my case, it's under my /scratch home directory
> /scratch/mvngu/release/ on sage.math. As you know, the idea is to
> avoid working under a directory that resides on an NFS mounted disk.

It is also a good idea to set your DOT_SAGE environment variable to
something outside NFS filesystems (the default location is ~/.sage,
which is NFS mounted).

--
Robert L. Miller
http://www.rlmiller.org/

[Pat LeSmithe]

Thanks for all of your advice! Some questions:

* Can we run tests on the build farm?

* Has anyone investigated building multiple spkgs in parallel, as long
as they're independent? I tried this last night, and it didn't simply
work. I'm not sure yet whether the problems are surmountable. It would
be great to learn about prior experience.

[Pat LeSmithe]

Also:

* Should we install ccache on the cluster?

[Nick Alexander]

>> * Has anyone investigated building multiple spkgs in parallel, as
>> long
>> as they're independent? I tried this last night, and it didn't
>> simply
>> work. I'm not sure yet whether the problems are surmountable. It
>> would
>> be great to learn about prior experience.
>

> I don't know. I haven't had time to investigate that issue.

I believe that parallel spkg building was implemented and found to be
terribly buggy. Lots of packages did not play nice, IIRC: atlas and
gsl seem likely. Craig Citro might know more?

Nick

[Robert Miller]

On Mon, Feb 8, 2010 at 6:17 PM, Pat LeSmithe <qed...@gmail.com> wrote:
>  * Has anyone investigated building multiple spkgs in parallel, as long
> as they're independent?  I tried this last night, and it didn't simply
> work.  I'm not sure yet whether the problems are surmountable.  It would
> be great to learn about prior experience.

There is also this thread:

http://groups.google.com/group/sage-devel/browse_thread/thread/9e849c199b9cdec6/

[Pat LeSmithe]

Thanks for the link. I think I have parallel inter/intra-spkg builds
working with make. I'll try to make a ticket soon.

I've updated

http://www.sagenb.org/home/pub/1530/

with a simple estimate of the parallelism

http://www.cprogramming.com/parallelism.html

which I hope is accurate within the simplified execution model, at least.

The data are from a completely serial build of Sage 4.3.2 on sage.math.
Total and top 10 spkg build times:

total 8070.0
sage 1996.0
python 701.0
atlas 685.0
singular 526.0
r 349.0
symmetrica 311.0
ntl 293.0
scipy 271.0
linbox 253.0
eclib 235.0

The critical path:

none -> libgpg_error -> libgcrypt -> opencdk -> gnutls -> python ->
atlas -> gsl -> linbox -> sage -> gap -> all

[Pat LeSmithe]

On 02/11/2010 04:44 AM, Pat LeSmithe wrote:
> On 02/09/2010 07:39 AM, Robert Miller wrote:
>> On Mon, Feb 8, 2010 at 6:17 PM, Pat LeSmithe <qed...@gmail.com> wrote:
>>> * Has anyone investigated building multiple spkgs in parallel, as long
>>> as they're independent? I tried this last night, and it didn't simply
>>> work. I'm not sure yet whether the problems are surmountable. It would
>>> be great to learn about prior experience.
>>
>> There is also this thread:
>>
>> http://groups.google.com/group/sage-devel/browse_thread/thread/9e849c199b9cdec6/
>
> Thanks for the link. I think I have parallel inter/intra-spkg builds
> working with make. I'll try to make a ticket soon.

Please see

http://trac.sagemath.org/sage_trac/ticket/8306

for updated files and some notes.

This seems to work well on the Sage cluster. With a warm compiler cache

http://ccache.samba.org/

building Sage sans docs takes 15-20 minutes with

export MAKE="make -j20"

on an otherwise mostly idle sage.math. The long doctests pass (after I
build the docs). Feedback about other systems is welcome!

Possible tip: "sage -b" works in parallel with "make -jX" (no space) but
not with "make -j X".

[William Stein]

On Thu, Feb 25, 2010 at 10:07 AM, Pat LeSmithe <qed...@gmail.com> wrote:
> On 02/11/2010 04:44 AM, Pat LeSmithe wrote:
>> On 02/09/2010 07:39 AM, Robert Miller wrote:
>>> On Mon, Feb 8, 2010 at 6:17 PM, Pat LeSmithe <qed...@gmail.com> wrote:
>>>>  * Has anyone investigated building multiple spkgs in parallel, as long
>>>> as they're independent?  I tried this last night, and it didn't simply
>>>> work.  I'm not sure yet whether the problems are surmountable.  It would
>>>> be great to learn about prior experience.
>>>
>>> There is also this thread:
>>>
>>> http://groups.google.com/group/sage-devel/browse_thread/thread/9e849c199b9cdec6/
>>
>> Thanks for the link.  I think I have parallel inter/intra-spkg builds
>> working with make.  I'll try to make a ticket soon.
>
> Please see
>
> http://trac.sagemath.org/sage_trac/ticket/8306
>
> for updated files and some notes.
>
> This seems to work well on the Sage cluster.  With a warm compiler cache
>
> http://ccache.samba.org/
>
> building Sage sans docs takes 15-20 minutes with

Holy crap! WOW! That is really, really impressive.

William

>
> export MAKE="make -j20"
>
> on an otherwise mostly idle sage.math.  The long doctests pass (after I
> build the docs).  Feedback about other systems is welcome!
>
> Possible tip: "sage -b" works in parallel with "make -jX" (no space) but
> not with "make -j X".
>

> --
> You received this message because you are subscribed to the Google Groups "sage-release" group.
> To post to this group, send email to sage-r...@googlegroups.com.
> To unsubscribe from this group, send email to sage-release...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/sage-release?hl=en.
>
>

--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org