Any information about reason of needing to use ancient ipython and pexpect in sagemath ?

[Paulo César Pereira de Andrade]

I opened a ticket in fedora to bundle cython, ipython and pexpect
for the hopefully very close, fedora sagemath package, see
https://fedorahosted.org/fpc/ticket/238

cython hopefully should not be an issue, and I said it should
require roughly 6 months to be able to use system ipython.

About pexpect I am not sure. I recall some comments about
newer versions being too slow. For the sake of using system
packages, I would go with it slower, but it actually does not
work... I did not debug it much recently, but almost 4 years
ago when I packaged sagemath in Mandriva I gave up
debugging, but for somebody that knows the code, it should
not be too hard to at least tell why it cannot be used, and I
can use that comment as the reason of needing to bundle
pexpect :-)

Thanks,
Paulo

[Jason Grout]

On 12/21/12 2:55 PM, Paulo C�sar Pereira de Andrade wrote:
> I opened a ticket in fedora to bundle cython, ipython and pexpect
> for the hopefully very close, fedora sagemath package, see
> https://fedorahosted.org/fpc/ticket/238
>
> cython hopefully should not be an issue, and I said it should
> require roughly 6 months to be able to use system ipython.

The IPython update ticket is at
http://trac.sagemath.org/sage_trac/ticket/12719

Thanks,

Jason

[Francois Bissey]

Yes pexpect. Same deal in sage-on-gentoo. I could add one of the patch from the current spkg on top of the latest version and it would work everywhere I could think of except the notebook. Plot would not be rendered. I don't remember trying with the new notebook.

I would take slower too. On the other hand I cannot think of something that absolutely needs the latest version of pexpect.

Francois

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To post to this group, send email to sage-...@googlegroups.com.
> To unsubscribe from this group, send email to sage-devel+...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel?hl=en.
>
>

This email may be confidential and subject to legal privilege, it may
not reflect the views of the University of Canterbury, and it is not
guaranteed to be virus free. If you are not an intended recipient,
please notify the sender immediately and erase all copies of the message
and any attachments.

Please refer to http://www.canterbury.ac.nz/emaildisclaimer for more
information.

[William Stein]

On Fri, Dec 21, 2012 at 4:43 PM, Francois Bissey <francoi...@canterbury.ac.nz> wrote:

Yes pexpect. Same deal in sage-on-gentoo. I could add one of the patch from the current spkg on top of the latest version and it would work everywhere I could think of except the notebook. Plot would not be rendered. I don't remember trying with the new notebook.

I would take slower too.

That is possibly naive.  When I evaluated updated pexpect last, "slower" was dramatically slower.    I didn't want want every ptty call to maxima, octave, R, pari, etc., to be 100 times slower...

Please at least do a test like this:

sage: timeit('gp("2+2")')

before and after switching.  

I had the impression that maybe the pexpect people had added a huge timeout or something to hack around race conditions they didn't understand, but I didn't look into it in detail (this was back in maybe 2007, and pexpect in sage worked fine).  Their change may be fine for most applications of pexpect, e.g., scripting an ssh session.  However, for Sage, where basic arithmetic may involve a call to pexpect, it is a very bad idea to throw in a 100ms timeout (say), since the net impact could be to slow down Sage by a factor of 200 for certain computations. 

--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

[Julien Puydt]

Le 22/12/2012 17:12, William Stein a �crit :

> On Fri, Dec 21, 2012 at 4:43 PM, Francois Bissey
> <francoi...@canterbury.ac.nz

> <mailto:francoi...@canterbury.ac.nz>> wrote:
>
> Yes pexpect. Same deal in sage-on-gentoo. I could add one of the
> patch from the current spkg on top of the latest version and it
> would work everywhere I could think of except the notebook. Plot
> would not be rendered. I don't remember trying with the new notebook.
>
> I would take slower too.
>
>
> That is possibly naive. When I evaluated updated pexpect last, "slower"
> was dramatically slower. I didn't want want every ptty call to
> maxima, octave, R, pari, etc., to be 100 times slower...
>
> Please at least do a test like this:
>
> sage: timeit('gp("2+2")')
>
> before and after switching.
>
> I had the impression that maybe the pexpect people had added a huge
> timeout or something to hack around race conditions they didn't
> understand, but I didn't look into it in detail (this was back in maybe
> 2007, and pexpect in sage worked fine). Their change may be fine for
> most applications of pexpect, e.g., scripting an ssh session. However,
> for Sage, where basic arithmetic may involve a call to pexpect, it is a
> very bad idea to throw in a 100ms timeout (say), since the net impact
> could be to slow down Sage by a factor of 200 for certain computations.

Did you report and discuss the issue with upstream?

Snark on #sagemath

[Francois Bissey]

Well there was a big commit to change the handling of the license 2
months ago but the last commit before was 4 years ago I think.
So for all purpose it looks abandoned. May be the author is waiting
for someone interested?

Francois

[Jeroen Demeyer]

On 2012-12-24 01:30, Francois Bissey wrote:
> Well there was a big commit to change the handling of the license 2
> months ago but the last commit before was 4 years ago I think.
> So for all purpose it looks abandoned. May be the author is waiting
> for someone interested?

pexpect is quite horribly coded IMHO. I think it would be best to
rewrite it essentially from scratch, in Cython. In my dreams, I would do
this, but I don't have the time, nor is it a high priority.

[Volker Braun]

Pexpect is a nice trick if you want to get off the ground quickly in interfacing another program, but the strategic goal should be to turn it into a shared library or develop a RPC call library that we can plumb into the source code. Your average scientific code is just a scanf/printf loop so it would be easy to replace those calls with our own.

I don't think that pexpect can be significantly better implemented, all the trouble comes from possible timing or buffering issues. At the core its a very simple process, its only difficult to make it work well on a large number of platforms.

[William Stein]

On Wed, Dec 26, 2012 at 9:42 AM, Volker Braun <vbrau...@gmail.com> wrote:

Pexpect is a nice trick if you want to get off the ground quickly in interfacing another program, but the strategic goal should be to turn it into a shared library or develop a RPC call library that we can plumb into the source code.

I just want to clarify what I think Volker is saying here.  He's say: "the strategic goal for any component of Sage that uses pexpect (e.g., Maxima, Gap, Pari, Singular, etc.) is to instead using a shared library interface or ...".   

This isn't an option in some cases in which we use pexpect, e.g., probably not for KASH, since KASH is closed source and mostly abandoned.  

Also, there is one possible advantage to pexpect over a shared library: you can easily control run multiple separate processes using pexpect, which can be running on possibly different computers.  (Of course, you could do something similar with multiple Python processes, so this is a dubious advantage.) 

 

Your average scientific code is just a scanf/printf loop so it would be easy to replace those calls with our own.

I don't think that pexpect can be significantly better implemented, all the trouble comes from possible timing or buffering issues. At the core its a very simple process, its only difficult to make it work well on a large number of platforms.

Having used it a lot, I tend to agree.  Over the years, our use of pexpect has been debugged in various ways.  There have been numerous very subtle timing and memory allocation bugs in our *usage* of pexpect that got fixed over the years (I'm thinking mainly of things that Carl Witty and Mike Hansen tracked down).   Rewriting the pexpect code itself in Cython would make little difference, since it is already mostly using existing C code from the Python library, for forking and for regular expression detection. 

Having written a lot of async javascript code lately, making a version of the pexpect interfaces that is asynchronous seems kind of interesting, e.g., with callbacks (or whatever).  This would be something that would probably only fit into the world of Twisted.  

On Wednesday, December 26, 2012 5:13:45 PM UTC, Jeroen Demeyer wrote:

I think it would be best to rewrite it essentially from scratch, in Cython.

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.
To post to this group, send email to sage-...@googlegroups.com.
To unsubscribe from this group, send email to sage-devel+...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel?hl=en.