bug in cloning? (updating modification times for the documentation)

[John H Palmieri]

With Sage 4.2 on my machine running OS X 10.6, when I clone, I get
this:

Updating modification times for the documentation...
sage -docbuild --update-mtimes reference html
sphinx-build -b html -d /Applications/sage/devel/sage/doc/output/
doctrees/en/reference /Applications/sage/devel/sage/doc/en/
reference /Applications/sage/devel/sage/doc/output/html/en/reference

At this point, it hangs. Has anyone else seen this?

John

[Georg S. Weber]

Hi,

cloning in Sage-4.2 sometimes does take a looooong time. I'm using OS
X 10.4 on an Intel Mac. I remember Ctrl-C'ing several times to
interrupt this on several occasions, but I am not sure whether it
really did hang, or whether I was too impatient.

Cheers,
Georg

[William Stein]

I can replicate this. For me with 4.2 on OS X 10.6. On my Macbook
Air laptop we have:

d-69-91-158-18:~ wstein$ time sage -clone testclone
Now cloning the current Sage library branch...
hg clone sage sage-testclone
updating working directory
1963 files updated, 0 files merged, 0 files removed, 0 files unresolved
Copying over all Cython auto-generated .c, .cpp and .h files...
Copying over build directory...
Copying over all auto-generated reference manual .rst files...
Copying over documentation output...
Building sage-testclone...
sage -b testclone

----------------------------------------------------------
sage: Building and installing modified Sage library files.

Installing c_lib
gcc -o src/convert.os -c -fPIC -I/Users/wstein/s/local/include
-I/Users/wstein/s/local/include/python2.6
-I/Users/wstein/s/local/include/NTL -Iinclude src/convert.c
...

Updating modification times for the documentation...
sage -docbuild --update-mtimes reference html
sphinx-build -b html -d

/Users/wstein/s/devel/sage/doc/output/doctrees/en/reference
/Users/wstein/s/devel/sage/doc/en/reference
/Users/wstein/s/devel/sage/doc/output/html/en/reference
...

Hangs with *0 activity* after a while. I never noticed this before,
because I always hit control-c the second I see anything about Sphinx
when cloning...

Why do we do *anything* with Sphinx after cloning? Shouldn't we just
copy everything over?

Anyway, I think you've found a serious bug.

William

[John H Palmieri]

On Oct 26, 3:54 pm, William Stein <wst...@gmail.com> wrote:

Because if we just copy everything over, Sphinx gets confused (for
reasons I don't understand) and thinks that everything needs to be
rebuilt. So if you clone and change one file in the Sage library,
then if you ask to build the reference manual, it rebuilds the whole
thing from scratch. This new business with updating the modification
times was supposed to help with that, so that it would only build the
parts that had changed.

John

[William Stein]

Did you try copying with "cp -p"?

William

>  So if you clone and change one file in the Sage library,
> then if you ask to build the reference manual, it rebuilds the whole
> thing from scratch.  This new business with updating the modification
> times was supposed to help with that, so that it would only build the
> parts that had changed.
>
>  John
>
> >
>

--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

[John H Palmieri]

On Oct 26, 4:52 pm, William Stein <wst...@gmail.com> wrote:

> On Mon, Oct 26, 2009 at 4:41 PM, John H Palmieri <jhpalmier...@gmail.com> wrote:
>
> >> Why do we do *anything* with Sphinx after cloning?  Shouldn't we just
> >> copy everything over?
>
> > Because if we just copy everything over,
> > Sphinx gets confused (for
> > reasons I don't understand) and thinks that everything needs to be
> > rebuilt.
>
> Did you try copying with "cp -p"?

We tried all sorts of things: see the discussion at <http://
trac.sagemath.org/sage_trac/ticket/6187>.

John

[Pat LeSmithe]

Do we know why the process hangs or appears to hang? What happens with

# cmd = 'sage -docbuild%s --update-mtimes reference html'%jsmath
cmd = 'sage -docbuild%s --update-mtimes reference html -v3'%jsmath
print cmd
proc = subprocess.Popen([cmd], shell=True)
# proc = subprocess.Popen([cmd], stdout=subprocess.PIPE, shell=True)

in sage-clone? Is Sphinx simply rebuilding the reference manual?

(Or is the warning at

http://docs.python.org/library/subprocess.html#popen-objects

relevant?)

[John H Palmieri]

On Oct 26, 6:15 pm, Pat LeSmithe <qed...@gmail.com> wrote:
> On 10/26/2009 05:16 PM, John H Palmieri wrote:
>
> > On Oct 26, 4:52 pm, William Stein <wst...@gmail.com> wrote:
> >> On Mon, Oct 26, 2009 at 4:41 PM, John H Palmieri <jhpalmier...@gmail.com> wrote:
>
> >>>> Why do we do *anything* with Sphinx after cloning?  Shouldn't we just
> >>>> copy everything over?
>
> >>> Because if we just copy everything over,
> >>> Sphinx gets confused (for
> >>> reasons I don't understand) and thinks that everything needs to be
> >>> rebuilt.
>
> >> Did you try copying with "cp -p"?
>
> > We tried all sorts of things: see the discussion at <http://
> > trac.sagemath.org/sage_trac/ticket/6187>.
>
> Do we know why the process hangs or appears to hang?  What happens with
>
> #    cmd = 'sage -docbuild%s --update-mtimes reference html'%jsmath
>     cmd = 'sage -docbuild%s --update-mtimes reference html -v3'%jsmath
>     print cmd
>     proc = subprocess.Popen([cmd], shell=True)
> #    proc = subprocess.Popen([cmd], stdout=subprocess.PIPE, shell=True)

If I do this, it sort of rebuilds the reference manual: it at least
writes its output again. I get:

sage -docbuild --update-mtimes reference html -v3
builder #349 _wrapper() Checking for .rst file mtimes to update...
builder #416 get_sphinx_environment() Opened Sphinx environment: /
Applications/sage/devel/sage/doc/output/doctrees/en/reference/
environment.pickle
builder #432 update_mtimes() Updated 637 .rst file mtimes
builder #438 update_mtimes() Saved Sphinx environment: /Applications/
sage/devel/sage/doc/output/doctrees/en/reference/environment.pickle
builder #353 _wrapper() Updating .rst files with modified modules...
builder #416 get_sphinx_environment() Opened Sphinx environment: /
Applications/sage/devel/sage/doc/output/doctrees/en/reference/
environment.pickle
builder #451 get_modified_modules() Sphinx found 0 modified modules
builder #358 _wrapper() Writing .rst files for newly-included
modules...
builder #390 get_cache() Loaded .rst file cache: /Applications/sage/
devel/sage/doc/output/doctrees/en/reference/reference.pickle
builder #502 get_newly_included_modules() Found 0 newly included
modules
builder #401 save_cache() Saved .rst file cache: /Applications/sage/
devel/sage/doc/output/doctrees/en/reference/reference.pickle
builder #119 f() sphinx-build -b html -d /Applications/sage/devel/sage/
doc/output/doctrees/en/reference /Applications/sage/devel/sage/doc/
en/reference /Applications/sage/devel/sage/doc/output/html/en/
reference
Running Sphinx v0.6.3
loading pickled environment... done
building [html]: targets for 637 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
preparing documents... done
writing output... [100%] structure
writing additional files... genindex modindex search
copying static files... done
dumping search index... done
dumping object inventory... done
build succeeded.
builder #122 f() Build finished. The built documents can be found in /
Applications/sage/devel/sage/doc/output/html/en/reference

and the clone finishes in under five minutes.

> in sage-clone?  Is Sphinx simply rebuilding the reference manual?

I don't think so: today I forgot about it, and it ran for more than an
hour without finishing. The reference manual doesn't take that long
on that machine.

> (Or is the warning at
>
> http://docs.python.org/library/subprocess.html#popen-objects
>
> relevant?)

I wonder if it is. If I do this change:

# cmd = 'sage -docbuild%s --update-mtimes reference html'%jsmath
cmd = 'sage -docbuild%s --update-mtimes reference html -
v3'%jsmath
print cmd

proc = subprocess.Popen([cmd], stdout=subprocess.PIPE,
shell=True)

(making the first of your changes and not the second), I get:

Updating modification times for the documentation...

sage -docbuild --update-mtimes reference html -v3
builder #349 _wrapper() Checking for .rst file mtimes to update...
builder #416 get_sphinx_environment() Opened Sphinx environment: /
Applications/sage/devel/sage/doc/output/doctrees/en/reference/
environment.pickle
builder #432 update_mtimes() Updated 637 .rst file mtimes
builder #438 update_mtimes() Saved Sphinx environment: /Applications/
sage/devel/sage/doc/output/doctrees/en/reference/environment.pickle
builder #353 _wrapper() Updating .rst files with modified modules...
builder #416 get_sphinx_environment() Opened Sphinx environment: /
Applications/sage/devel/sage/doc/output/doctrees/en/reference/
environment.pickle
builder #451 get_modified_modules() Sphinx found 0 modified modules
builder #358 _wrapper() Writing .rst files for newly-included
modules...
builder #390 get_cache() Loaded .rst file cache: /Applications/sage/
devel/sage/doc/output/doctrees/en/reference/reference.pickle
builder #502 get_newly_included_modules() Found 0 newly included
modules
builder #401 save_cache() Saved .rst file cache: /Applications/sage/
devel/sage/doc/output/doctrees/en/reference/reference.pickle
builder #119 f() sphinx-build -b html -d /Applications/sage/devel/sage/
doc/output/doctrees/en/reference /Applications/sage/devel/sage/doc/
en/reference /Applications/sage/devel/sage/doc/output/html/en/
reference

and then it hangs.

John