- Alex
<snip>
> Could you please test if you can replicate the overhead? Because, if I
> am the only person experiencing it, it would hardly be worth a ticket.
>
> Cheers,
> Simon
I don't see this issue on my OpenSolaris machine
sage: def test(n):
....: st = singular.cputime()
....: ct = cputime()
....: wt = walltime()
....: for i in range(n):
....: a = singular.eval('def a%d=%d'%(i,i))
....: print "Wall time:", walltime(wt)
....: print "Total CPU:", cputime(ct)+singular.cputime(st)
....:
sage: sage: test(1000)
Wall time: 0.151273012161
Total CPU: 0.170581
I note we are not running the latest version of pexpect. We are
running 2.0, but the latest is 2.3
http://www.noah.org/wiki/pexpect#Download_and_Installation
I would be tempted to try updating the .spkg to the latest pexpect and
see if that fixes it.
Even if you are currently the only one experiencing this, it is worth
creating a ticket for it. But put as much information as possible
about your system. Are the file systems local or on an NFS system? If
on NFS, can you find out what the server is?
It could be others will get this, but just don't notice is because
they don't use Singular, or have just accepted the slowness, rather
than questioning it, as you are doing.
Do you get any warnings when the pexpect module is built?
If you run SAGE_CHECK=yes, what failures do you get? (Everyone seems
to get some failures.) I would add that to a trac ticket, as it might
be useful.
I wonder if there's any chance of getting the Singular developers to
provide a C API, which means we could despense with using pexect for
this.
Dave
<snip>
> Could you please test if you can replicate the overhead? Because, if I
> am the only person experiencing it, it would hardly be worth a ticket.
>
> Cheers,
> Simon
I don't know an alful lot about the doctests, but I wonder if it's
possible to create one that would detect the problem you see? You are
using 40 seconds wall time, I'm using 0.15. If the number 'n' could be
adjusted so that it would not take more than 1 second on even the
slowest hardware, then we could check that the total wall time used is
less than a second.
I somewhat doubt you are the only one seeing this to be honest, but
without some sort of test, we will never know.
Dave
I've no idea, but I'd certainly give it a try myself.
>> Do you get any warnings when the pexpect module is built?
>
> No. But since pexpect seems to be pure Python, I think there can't
> compiler problems be involved.
>> If you run SAGE_CHECK=yes, what failures do you get? (Everyone seems
>> to get some failures.)
>
> I did not compile with SAGE_CHECK=yes. Would "make ptestall" suffice?
No, you would have to run
$ export SAGE_CHECK=yes
$ ./sage -f python-$version
That will list any failures. Conceivably pexpect might use a different
method depending on what bits of python work properly.
>> I wonder if there's any chance of getting the Singular developers to
>> provide a C API, which means we could despense with using pexect for
>> this.
>
> I don't think that Singular is to blame. I observe a similar overhead
> for the communication with Gap. And if I'm not mistaken, the C API is
> used in libsingular.
Just in general, it would be good if pexpect could be avoided whenever
possible. I believe for example someone on the Sage project is working
on one for Maxima.
> Cheers,
> Simon
Distutils is for building other python modules
http://docs.python.org/library/distutils.html
so conceivably a failure on that could cause an issue with a module
like pexpect.
But I think others have had that fail too. test_zlib has failed for
many people. I don't think I've ever seen anyone on the sage list
report a failure of test_urllib, but the name does not really suggest
it will be the cause.
3 failues is not an excessive number. Most people seem to get a few.
> 39 tests skipped:
> test_aepack test_al test_applesingle test_bsddb test_bsddb185
> test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk
> test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
> test_dbm test_dl test_gdbm test_gl test_imageop test_imgfile
> test_kqueue test_linuxaudiodev test_macos test_macostools
> test_normalization test_ossaudiodev test_pep277 test_py3kwarn
> test_scriptpackages test_smtpnet test_socketserver test_startfile
> test_sunaudiodev test_timeout test_urllib2net test_urllibnet
> test_winreg test_winsound test_zipfile64
The reasons for skippling some of them are obvious, like test_sunaudiodev.
> 3 skips unexpected on linux2:
> test_dbm test_gdbm test_bsddb
I don't know if these have any significance.
You might get more joy asking on the python forums, or perhaps a
pexpect mailing list, forum, or similar. The pexpect author should
have more idea than any of us.
> test test_distutils failed -- errors occurred; run in verbose mode for
> details
>
>
> Does that seem related to select.select being slow?
> Cheers,
> Simon
Dave
On Thu, 18 Nov 2010 at 12:09AM -0800, Simon King wrote:
> When I define
> {{{
> def test(n):
> st = singular.cputime()
> ct = cputime()
> wt = walltime()
> for i in range(n):
> a = singular.eval('def a%d=%d'%(i,i))
> print "Wall time:", walltime(wt)
> print "Total CPU:", cputime(ct)+singular.cputime(st)
> }}}
> then I get
> {{{
> sage: test(1000)
> Wall time: 40.0399508476
> Total CPU: 0.0
> }}}
I get:
sage: test(1000)
Wall time: 40.4516711235
Total CPU: 0.37
> When I replace "singular.eval..." by "singular._sendstr('def a%d=%d;
> \n'%(i,i))" in the above function, I get
> {{{
> sage: test(1000)
> Wall time: 0.015585899353
> Total CPU: 0.04
> sage: singular('a100')
> 100
> }}}
With the same change, I get:
sage: test(1000)
Wall time: 0.00672912597656
Total CPU: 0.02
> But when I replace "singular.eval..." by "singular._synchronize()", I
> get
> {{{
> sage: test(1000)
> Wall time: 19.9999539852
> Total CPU: 0.0
> }}}
Here, I get:
sage: test(1000)
Wall time: 20.219892025
Total CPU: 0.13
So your machine is not alone.
On sagenb.kaist.ac.kr, an 8-core Xeon machine running Ubuntu 10.04.1, I get:
sage: test(1000) # original singular.eval
Wall time: 40.0399620533
Total CPU: 0.4
sage: test(1000) # singular._sendstr
Wall time: 0.0119400024414
Total CPU: 0.04
sage: test(1000) # singular._synchronize
Wall time: 19.9999010563
Total CPU: 0.15
So your machine is *definitely* not alone.
Dan
--
--- Dan Drake
----- http://mathsci.kaist.ac.kr/~drake
-------
For what it's worth, I ran your tests on my Macbook. It's an Intel
Macbook from 2006 or 2007, running OS X 10.5 and Sage 4.5. I got:
sage: test1(1000) # the basic "eval()" test
Wall time: 0.275583028793
Total CPU: 0.327884
sage: test3(1000) # the "synchronize()" test
Wall time: 0.106857776642
Total CPU: 0.141513
The _sendstr() test would work when I ran it a few times, but eventually it
would hang. I'm guessing that the interface got de-sync'ed.
I can check this on an Arch VM tomorrow.
Could you try to download, compile and run a small test program on a
problematic machine? It times how fast a pseudo-terminal responds, which might
be the problem judging by a few quick tests I ran.
wget http://www.usecode.org/misc/timeptmx.c
gcc -o timeptmx timeptmx.c
strace -o timeptmx.log -f -ttt ./timeptmx
grep aaa timeptmx.log
That should output something like this:
16095 1290175675.065705 write(3, "aaa", 3) = 3
16096 1290175675.065749 <... read resumed> "aaa", 256) = 3
The difference between these two timestamps seems to determine how fast pexpect
responds. In this case it's fast (1290175675.065705 to 1290175675.065749 is
only 44 microseconds), but I've seen 1.8ms on other machines with newer
kernels.
Something similar to this is run twice for every singular.eval() call, and
seems to be the major factor in execution (wall) time.
-Willem Jan
If there is no trac ticket for this, which i believe is the case, then
one should be created.
This could be real mess if updates of the kernal are going to cause
this slowdown.
it must be worth trying the latst pexpect source code - we are several
revisions out of date.
Dave
drake@klee:/scratch/download$ grep aaa timeptmx.log
508 1290216318.809968 write(3, "aaa", 3) = 3
509 1290216318.837492 <... read resumed> "aaa", 256) = 3
So, about .027 seconds. That's on my 64-bit Ubuntu 10.10 machine, Core 2
Quad, with a current kernel:
Linux klee 2.6.35-22-generic #35-Ubuntu SMP Sat Oct 16 20:45:36 UTC 2010
x86_64 GNU/Linux
On the same machine, inside an Arch virtual machine running in
VirtualBox: (I don't have cut and paste enabled for the VM, so I'm
manually typing this:
936 129...5.908400 write(3, "aaa"...
937 129...5.928973 >... read resumed...
So, the timing is faster, but still a couple orders of magnitude behind
your timings. Yikes.
I doubt it will help, but it is remotely possible, and to do test would probably
take you less than 10 minutes - no need to bother with mercurial, patches,
review or anything else. Just try it and see if it works.
Dave
Francois
in the first case:
Wall time: 16.0200681686
Total CPU: 0.862045
in the second one:
Wall time: 0.0230610370636
Total CPU: 0.042001
in the third one:
Wall time: 8.00897812843
Total CPU: 0.300014
and my config:
Linux neo 2.6.32-25-generic #45-Ubuntu SMP Sat Oct 16 19:48:22 UTC 2010
i686 GNU/Linux
j_schn14@neo:~$ sage -version
| Sage Version 4.6, Release Date: 2010-10-30
hope this helps.
greatz Johannes
Thanks for all your efforts about the performance of our interface.
For Singular, there exists the libsingular interface.
I would appreciate every help making it the perfect Singular interface, instead of optimizing
the generic pexpect interface.
Cheers,
Michael
>
> - Alex
>
> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
-------------------------------------------
Dr. rer. nat. Michael Brickenstein
Mathematisches Forschungsinstitut Oberwolfach gGmbH
Schwarzwaldstr. 9 - 11
77709 Oberwolfach
Tel.: 07834/979-31
Fax: 07834/979-38
That's not clear to me. Maybe the full singular command prompt could
be used via a library interface? After all, it is just a C++ program.
The C interface to GAP that I wrote just involves rewriting the
GAP read-eval-print loop (the command line interface) and turning it
into C library calls. I literally just stick some data in where the
read would put it, then tell GAP that it just read something in, so go
evaluate it, etc. I mentioned this to Martin Albrecht (author of
libsingular), and he said it would be more difficult to do that with
Singular. But that doesn't mean it is impossible. It might be better
to write something like this and not change any of Sage, than to have
to rewrite a lot of Sage.
-- William
pexpect (and other expect variants) were written specifically to avoid
deadlocks which you are most likely seeing with Singular. For more
info on this, check out
http://effbot.org/pyfaq/how-do-i-run-a-subprocess-with-pipes-connected-to-both-input-and-output.htm
--Mike
pexpect (and other expect variants) were written specifically to avoid
deadlocks which you are most likely seeing with Singular. For more
info on this, check out
http://effbot.org/pyfaq/how-do-i-run-a-subprocess-with-pipes-connected-to-both-input-and-output.htm
--Mike
> Loading the library. Please be patient, this may take a while.
> Got:
> GAP4, Version: 4.4.12 of 17-Dec-2008, x86_64-unknown-linux-gnu-gcc
> Got:
> gap>
> p.terminate()
> returncode=-15
> }}}
>
>
> So it seems singular needs a tty.
>
> - Alex
sage -singular --help
Singular version 3-1-0 -- a CAS for polynomial computations. Usage:
Singular-3-1-0 [options] [file1 [file2 ...]]
Options:
...
-t --no-tty Do not redefine the terminal characteristics
...
Cheers,
Michael
>
> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
> <test_Popen.py>