building sage in parallel in parallel

[William Stein]

Hi,

I just tried the following.

1. Take a clean Sage-4.0.2 tarball and edit SAGE_ROOT/spkg/install
replacing the line

time make -f standard/deps $1

by

time make -j20 -f standard/deps $1

2. make. This builds a whole bunch of different spkg's in parallel.
At one point spkg/install
gets overwritten. At another point something fails and the build fails.

3. Changed spkg/install again to use "-j20". Built some more by typing "make".

4. Again it fails eventually, after building most of sage pretty
quickly. Finally, I change
spkg/install back to the default (no -j20), and type "make". The
build finishes and Sage works.

This may be of interest to people who are interested in doing more
efficient Sage builds.

William

--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

[jyr]

Below is a python script that implements the above and sets
environment variables for parallel compiles. Just save the script to
makesage.py (say) in the sage root dir and run it via

python makesage.py -j8

It reduced the sage-4.0.2 compile to 1hour 51 minutes on a dual quad
AMD 2GHz machine.

--------------------------------------------------------------------------------------

import os,sys
import getopt


def replace_install(sj):
fpin=open('spkg/install',"r")
sinstall=fpin.readlines()
fpin.close()

for iline,sline in enumerate(sinstall):
if 'time make' in sline and '-f standard/deps $1' in sline:
sinstall[iline]='time make '+sj+' -f standard/deps $1\n'
sinstall="".join(sinstall)

fpou=open('spkg/install',"w")
fpou.write(sinstall)
fpou.close()


def main(argv):
sj='-j1'
try:
opts,args=getopt.getopt(argv[1:],"j:")
except getopt.GetoptError:
#print_usage()
sys.exit(1)
for oo,aa in opts:
if oo in ("-j"):
sj="-j"+aa

smake='make '+sj
os.environ['MAKE']=smake

replace_install(sj)

os.system('export MAKE="%s"'%(smake)+' && make')

replace_install(sj)

os.system('export MAKE="%s"'%(smake)+' && make')

replace_install('')

os.system('export MAKE="%s"'%(smake)+' && make')



if __name__ == '__main__':
main(sys.argv)

[William Stein]

On Mon, Jun 22, 2009 at 4:29 PM, jyr<jyr...@googlemail.com> wrote:
>
> Below is a python script that implements the above and sets
> environment variables for parallel compiles. Just save the script to
> makesage.py (say) in the sage root dir and run it via
>
> python makesage.py -j8
>
> It reduced the sage-4.0.2 compile to 1hour 51 minutes on a dual quad
> AMD 2GHz machine.

Thanks. Any thoughts about how to *fix* the build of Sage so that it works
the first time instead of having to try 3 times. But again, thanks!