Problems building docs on sage-5.8.beta4

[Montgomery-Smith, Stephen]

Building on FreeBSD, the doc creation process freezes after this point:

[history_a] loading pickled environment... not yet created
[history_a] building [inventory]: targets for 1 source files that are
out of date
[history_a] updating environment: 1 added, 0 changed, 0 removed
[history_a] reading sources... [100%] index
[history_a] loading cross citations...
[history_a] pickling environment... done
[history_a] checking consistency... done
[history_a] preparing documents... done
[history_a] writing output... [100%] index
[history_a] dumping object inventory... done
[history_a] build succeeded.

[Volker Braun]

can you strace/debug the process and see what its hanging at?

[Montgomery-Smith, Stephen]

So I ran the make process again. After getting to the point of freezing, using top, I could see that there were a couple of invocations of python2.7. FreeBSD uses truss in place of strace, and using "truss -p pid" where pid was the pid of one of the invocations of python2.7, I could see lines like this pouring out of truss:

syscall(0x0,0x0,0x0,0x7fffffffc6c0,0x3fa99999,0x838541cc8) = 116 (0x74)
-- UNKNOWN SYSCALL -14656 --
syscall(0x0,0x0,0x0,0x7fffffffc6c0,0x3fa99999,0x838541cb0) = 116 (0x74)
-- UNKNOWN SYSCALL -14656 --
syscall(0x0,0x0,0x0,0x7fffffbfe430,0x3fb99999,0x801443e18) = 7 (0x7)
-- UNKNOWN SYSCALL 22131 --

I have no idea what this means.

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Volker Braun]

Is it actually frozen or is your computer just slow? ;-)  In any case your dtrace seems to be out of date. The new docbuilder uses the python multiprocessing module. Is your FreeBSD new enough to support that? Google search suggests that there were some semaphore problems up until a few years ago.

[leif]

Montgomery-Smith, Stephen wrote:
> So I ran the make process again. After getting to the point of freezing, using top, I could see that there were a couple of invocations of python2.7. FreeBSD uses truss in place of strace, and using "truss -p pid" where pid was the pid of one of the invocations of python2.7, I could see lines like this pouring out of truss:
>
> syscall(0x0,0x0,0x0,0x7fffffffc6c0,0x3fa99999,0x838541cc8) = 116 (0x74)
> -- UNKNOWN SYSCALL -14656 --
> syscall(0x0,0x0,0x0,0x7fffffffc6c0,0x3fa99999,0x838541cb0) = 116 (0x74)
> -- UNKNOWN SYSCALL -14656 --

On Linux at least, these would have returned ESTALE (stale NFS file handle).

> syscall(0x0,0x0,0x0,0x7fffffbfe430,0x3fb99999,0x801443e18) = 7 (0x7)
> -- UNKNOWN SYSCALL 22131 --

This should be E2BIG (argument list too long)... :-)


-leif

--
() The ASCII Ribbon Campaign
/\ Help Cure HTML E-Mail

[leif]

Volker Braun wrote:
> Is it actually frozen or is your computer just slow? ;-) In any case
> your dtrace seems to be out of date. The new docbuilder uses the python
> multiprocessing module. Is your FreeBSD new enough to support that?
> Google search suggests that there were some semaphore problems up until
> a few years ago.

Well, building the Sage library apparently succeeded, which also uses
the multiprocessing module (even with SAGE_NUM_THREADS=1 or MAKE="make
-j1" I think).

[Montgomery-Smith, Stephen]

I think it may well be related to the semaphore issue. But if that is the case, it might take more time to fix than I have right now. So for a while, I guess the FreeBSD port of sage will stay with 5.7.

In the mean time, if someone tells which part of the python code does the document building, then I can try to narrow down what triggers the problem.

I did copy the fixes from the python27 FreeBSD port to the python that comes with sage, and it was supposed to have fixed the semaphore issue. The version of FreeBSD I am using should be recent enough. But maybe they didn't quite fix it right.

Another reason I might put this off a while is because my current smtp server is not being accepted by google groups or gmail. All this frustration adds up to make me want to put this off a while.

[leif]

Montgomery-Smith, Stephen wrote:
> Another reason I might put this off a while is because my current smtp server is not being accepted by google groups or gmail. All this frustration adds up to make me want to put this off a while.

Well, there's also news.gmane.org, at least for sage-{devel,support}... ;-)

[John H Palmieri]

On Sunday, March 10, 2013 11:25:35 AM UTC-7, Stephen Montgomery-Smith wrote:

I think it may well be related to the semaphore issue.  But if that is the case, it might take more time to fix than I have right now.  So for a while, I guess the FreeBSD port of sage will stay with 5.7.

In the mean time, if someone tells which part of the python code does the document building, then I can try to narrow down what triggers the problem.

The entry point is SAGE_ROOT/devel/sage/doc/common/builder.py.

--
John

[Stephen Montgomery-Smith]

Building on FreeBSD, the doc creation process freezes after this point:

....

[history_a] writing output... [100%] index
[history_a] dumping object inventory... done
[history_a] build succeeded.

P.S. I sent this email twice before. Perhaps google flagged it as spam.

[Stephen Montgomery-Smith]

Building on FreeBSD, the doc creation process freezes after this point:

[history_a] loading pickled environment... not yet created
[history_a] building [inventory]: targets for 1 source files that are
out of date
[history_a] updating environment: 1 added, 0 changed, 0 removed
[history_a] reading sources... [100%] index
[history_a] loading cross citations...
[history_a] pickling environment... done
[history_a] checking consistency... done
[history_a] preparing documents... done

[history_a] writing output... [100%] index
[history_a] dumping object inventory... done
[history_a] build succeeded.

P.S. I sent this email earlier. Perhaps google flagged it as spam.

[Stephen Montgomery-Smith]

[Stephen Montgomery-Smith]

OK, I have done a lot of work trying to figure this out. What I
discovered is that some of the doc building simply isn't finishing. For
example:

[calculus ] loading pickled environment... not yet created
[calculus ] building [inventory]: targets for 20 source files that are
out of date
[calculus ] updating environment: 20 added, 0 changed, 0 removed
[calculus ] reading sources... [ 5%] index
[calculus ] reading sources... [ 10%] sage/calculus/calculus
[categorie] loading pickled environment... not yet created

You see, it starts building for calculus, and then it just stops. Then
categories starts building. This is all with NUMBERTHREADS set to 1.

If the building of calculus just stops, but the number of threads is 1,
why then doesn't everything just stop? Instead it seems to stop on a
pool_join command in builder.py?

I'll give more details to anyone who is interested. In the meantime, if
anyone has any ideas, I would appreciate hearing them.

Thanks, Stephen

[Stephen Montgomery-Smith]

On 03/17/2013 02:48 PM, Stephen Montgomery-Smith wrote:
> On 03/09/2013 06:40 PM, Montgomery-Smith, Stephen wrote:

>> Building on FreeBSD, the doc creation process freezes at some point ....

I have done more investigation. The following patch fixes the problem,
so it is definitely a problem to do with multithreading. Anyway, so
that I can get sage-5.8 in the FreeBSD port, I will include this patch.
I provide it for your information only, as I don't think any other OS
will need it.

[kcrisman]

On Mar 18, 9:43 pm, Stephen Montgomery-Smith <step...@missouri.edu>
wrote:

>  spkg-patch-sage_-_doc_common_builder.py
> 2KViewDownload

Well, but should we incorporate it as well?

[John H Palmieri]

Should we include this patch with a suitable block

  if using FreeBSD:
      use your version
  else:
      use multithreading

Is there any good way of testing whether multithreading is broken, instead of just testing whether it's FreeBSD?

--
John

[Dima Pasechnik]

On 2013-03-19, John H Palmieri <jhpalm...@gmail.com> wrote:
> ------=_Part_744_6416061.1363663809941
> Content-Type: text/plain; charset=ISO-8859-1

>
>
>
> On Monday, March 18, 2013 6:43:03 PM UTC-7, Stephen Montgomery-Smith wrote:
>>
>> On 03/17/2013 02:48 PM, Stephen Montgomery-Smith wrote:
>> > On 03/09/2013 06:40 PM, Montgomery-Smith, Stephen wrote:
>> >> Building on FreeBSD, the doc creation process freezes at some point
>> ....
>>
>> I have done more investigation. The following patch fixes the problem,
>> so it is definitely a problem to do with multithreading. Anyway, so
>> that I can get sage-5.8 in the FreeBSD port, I will include this patch.
>> I provide it for your information only, as I don't think any other OS
>> will need it.
>>
>
> Should we include this patch with a suitable block
>
> if using FreeBSD:
> use your version
> else:
> use multithreading

Indeed, that's the way it should be. This can be done inside Python,
checking out on which OS you are, so I don't see why this should not get
into the main Sage codebase.

>
> Is there any good way of testing whether multithreading is broken, instead
> of just testing whether it's FreeBSD?

I imagine this might be possible too...

Dima

>
> --
> John
>

[Jeroen Demeyer]

On 2013-03-19 04:30, John H Palmieri wrote:
> Is there any good way of testing whether multithreading is broken

That would of course be ideal, but we should first find out exactly
why/how it's broken on FreeBSD.

[leif]

Yep. As said before, apparently the Sage library /did/ build (which
*always* uses multiprocessing, even if SAGE_NUM_THREADS=1, i.e., even if
it uses just one thread, AFAIK).

I also wouldn't special-case on FreeBSD in the docbuilder, but use
SAGE_DOCBUILD_NUM_THREADS (say) there instead, and set it to 1 if
necessary elsewhere, otherwise let it default to SAGE_NUM_THREADS or
whatever. (The user should be able to set it as well.)

[Stephen Montgomery-Smith]

I should add that the doc building for FreeBSD breaks, even if the
number of threads is 1. It is the use of Pool that seems to be the problem.

I don't think this problem can be tested by seeing if threading works,
because I don't know a reliable test. I write small test programs, and
they seem to work just fine.

For all I know, it works if the build is on a 386, and not on a 64 bit
machine.

The breakage is very mysterious, and is probably some horribly difficult
to test race condition.

[leif]

Stephen Montgomery-Smith wrote:
> On 03/19/2013 06:07 AM, leif wrote:
>> Jeroen Demeyer wrote:
>>> On 2013-03-19 04:30, John H Palmieri wrote:
>>>> Is there any good way of testing whether multithreading is broken
>>> That would of course be ideal, but we should first find out exactly
>>> why/how it's broken on FreeBSD.
>>
>> Yep. As said before, apparently the Sage library /did/ build (which
>> *always* uses multiprocessing, even if SAGE_NUM_THREADS=1, i.e., even if
>> it uses just one thread, AFAIK).
>>
>> I also wouldn't special-case on FreeBSD in the docbuilder, but use
>> SAGE_DOCBUILD_NUM_THREADS (say) there instead, and set it to 1 if
>> necessary elsewhere, otherwise let it default to SAGE_NUM_THREADS or
>> whatever. (The user should be able to set it as well.)
>
> I should add that the doc building for FreeBSD breaks, even if the
> number of threads is 1. It is the use of Pool that seems to be the problem.

I of course meant Sage must not use multiprocessing if *_THREADS==1.
(This applies to docbuilding as well as to building the Sage library.)

> I don't think this problem can be tested by seeing if threading works,
> because I don't know a reliable test. I write small test programs, and
> they seem to work just fine.

:-) Keep on trying...


-leif

> For all I know, it works if the build is on a 386, and not on a 64 bit
> machine.
>
> The breakage is very mysterious, and is probably some horribly difficult
> to test race condition.

[Julien Puydt]

Le 10/03/2013 01:40, Montgomery-Smith, Stephen a �crit :

After reviewing this thread and the problem I had on debian with sage
5.8 crossed with system packages, I think it's the same thing.

So let's see what is in common between FBSD and debian ; if the problem
is the use of the Pool class in doc/common/builder.py, and if that class
is really the one coming from "from multiprocessing import Pool", then
debian currently ships python 2.7.3... which is the same as sage's. :-/

Which python does FBSD use?
What can we search for next?
Why isn't sage affected by the bug?

Snark on #sagemath

[kcrisman]

My understanding is that you did eventually fix this, correct?  I assume so from 

http://www.freebsd.org/cgi/cvsweb.cgi/ports/math/sage/files/spkg-patch-sage_-_doc_common_builder.py?rev=1.1;content-type=text%2Fplain

Did anyone ever open a ticket for this?  I can but don't want to duplicate. 

[Jean-Pierre Flori]

IIRC its not a fix but a workaround: stephen disabled parallel building of the docs.

[Jean-Pierre Flori]

FYI same problems on a raspberyr pi (running raspbian):
{{{
Build finished. The built documents can be found in /home/pi/sage-5.10/devel/sage/doc/output/html/de/tutorial
Traceback (most recent call last):
  File "/home/pi/sage-5.10/devel/sage/doc/common/builder.py", line 1452, in <module>
    getattr(get_builder(name), type)()
  File "/home/pi/sage-5.10/devel/sage/doc/common/builder.py", line 281, in _wrapper
    pool.map_async(build_other_doc, L, 1).get(99999)
  File "/home/pi/sage-5.10/local/lib/python/multiprocessing/pool.py", line 550, in get
    raise TimeoutError
multiprocessing.TimeoutError
make: *** [doc-html] Error 1