Strange error using R pexpect interface?

[kcrisman]

I'm using R matrices to use an R program and then do things with it in
Sage. For some reason Sage doesn't get the "right" answer for
matrices above a certain size.

The first one is right (it gives the space that is in the returned
string) while the second one makes no sense; ZZ='' is what actually
comes back. But there is no real reason for this - what's special
about the length? - and doing these in Sage's r_console() gives normal
results for the matrix.

So I feel like pexpect must be doing something naughty. Does anyone
have any ideas what might be going on so I can use more data?

Thanks,
- kcrisman


sage: ZZ = r.eval('matrix(c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 2, 2, 2, 2, 2, 2, 2,
2, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 1, 1, 1,
1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 4, 4, 4, 4, 4, 4, 4, 4, 4, 1,
1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 4, 4, 4, 4, 2, 2, 2, 2, 4, 4, 4, 4, 2, 2, 2, 2, 3, 3, 3,
3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 1, 1, 1, 1, 4, 4, 4, 4, 1, 1, 1, 1,
3, 3, 3, 3, 2, 2, 2, 2, 4, 4, 4, 4, 1, 1, 1, 1, 4, 4, 4, 4, 1, 1, 1,
1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 1, 1, 1, 1, 3, 3, 3, 3, 1,
1, 1, 1, 2, 2, 2, 2, 4, 4, 4, 4, 3, 3, 3, 3, 4, 4, 4, 4, 2, 2, 2, 2,
3, 3, 3, 3, 2, 2, 2, 2, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 4, 4, 4, 4,
1),ncol=4)'); ZZ[1]
' '
sage: ZZ = r.eval('matrix(c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 2, 2, 2, 2, 2, 2, 2,
2, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 1, 1, 1,
1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 4, 4, 4, 4, 4, 4, 4, 4, 4, 1,
1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 4, 4, 4, 4, 2, 2, 2, 2, 4, 4, 4, 4, 2, 2, 2, 2, 3, 3, 3,
3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 1, 1, 1, 1, 4, 4, 4, 4, 1, 1, 1, 1,
3, 3, 3, 3, 2, 2, 2, 2, 4, 4, 4, 4, 1, 1, 1, 1, 4, 4, 4, 4, 1, 1, 1,
1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 1, 1, 1, 1, 3, 3, 3, 3, 1,
1, 1, 1, 2, 2, 2, 2, 4, 4, 4, 4, 3, 3, 3, 3, 4, 4, 4, 4, 2, 2, 2, 2,
3, 3, 3, 3, 2, 2, 2, 2, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 4, 4, 4, 4, 1,
1, 1, 1, 3),ncol=4)'); ZZ[1]

[John H Palmieri]

On Monday, June 6, 2011 10:54:57 AM UTC-7, kcrisman wrote:

I'm using R matrices to use an R program and then do things with it in
Sage.  For some reason Sage doesn't get the "right" answer for
matrices above a certain size.

The first one is right (it gives the space that is in the returned
string) while the second one makes no sense; ZZ='' is what actually
comes back.  But there is no real reason for this - what's special
about the length? - and doing these in Sage's r_console() gives normal
results for the matrix.

So I feel like pexpect must be doing something naughty.  Does anyone
have any ideas what might be going on so I can use more data?

I tried this experiment: I added spaces to the first string to be evaluated.  When the string has length <= 1024, it seems to work, and when the string has length > 1024, it doesn't.  For example:

    sage: s = 'matrix(c(1, 1, 1, 1,' + ' '*987 + '2,2,2,2), ncol=4)'
    sage: len(s)
    1024
    sage: r.eval(s)
    '     [,1] [,2] [,3] [,4]\n[1,]    1    1    2    2\n[2,]    1    1    2    2'
    sage: s = 'matrix(c(1, 1, 1, 1,' + ' '*988 + '2,2,2,2), ncol=4)'
    sage: len(s)
    1025
    sage: r.eval(s)
    ''

I don't know why, but maybe that can help you track it down.

--
John

[kcrisman]

>
> > So I feel like pexpect must be doing something naughty.  Does anyone
> > have any ideas what might be going on so I can use more data?
>
> I tried this experiment: I added spaces to the first string to be
> evaluated.  When the string has length <= 1024, it seems to work, and when
> the string has length > 1024, it doesn't.  For example:
>
>     sage: s = 'matrix(c(1, 1, 1, 1,' + ' '*987 + '2,2,2,2), ncol=4)'
>     sage: len(s)
>     1024
>     sage: r.eval(s)
>     '     [,1] [,2] [,3] [,4]\n[1,]    1    1    2    2\n[2,]    1    1    
> 2    2'
>     sage: s = 'matrix(c(1, 1, 1, 1,' + ' '*988 + '2,2,2,2), ncol=4)'
>     sage: len(s)
>     1025
>     sage: r.eval(s)
>     ''
>
> I don't know why, but maybe that can help you track it down.

Thanks - that definitely helps, since pexpect is actually passing
strings.

At http://pexpect.sourceforge.net/pexpect.html I see
"The maxread attribute sets the read buffer size. This is maximum
number
of bytes that Pexpect will try to read from a TTY at one time. Setting
the maxread size to 1 will turn off buffering. Setting the maxread
value higher may help performance in cases where large amounts of
output are read back from the child. "

Can a pexpect expert help me here? It seems like we have

maxread=100000

for R (and most other interfaces), which would seem to be plenty of
bytes for a 1000-character string, if that is what maxread controls.

Doing it in R "by hand" allows me arbitrarily long things (well, 'by
hand' arbitrary) so that is definitely not the bottleneck. And I
don't see this problem in Maxima (I am using an older version of Sage
without the library interface for Maxima/ECL, so this is a relevant
data point).

Thanks to anyone!

- kcrisman

[John H Palmieri]

On Monday, June 6, 2011 12:55:49 PM UTC-7, kcrisman wrote:

>
> > So I feel like pexpect must be doing something naughty.  Does anyone
> > have any ideas what might be going on so I can use more data?
>
> I tried this experiment: I added spaces to the first string to be
> evaluated.  When the string has length <= 1024, it seems to work, and when
> the string has length > 1024, it doesn't.  For example:
>
>     sage: s = 'matrix(c(1, 1, 1, 1,' + ' '*987 + '2,2,2,2), ncol=4)'
>     sage: len(s)
>     1024
>     sage: r.eval(s)
>     '     [,1] [,2] [,3] [,4]\n[1,]    1    1    2    2\n[2,]    1    1    
> 2    2'
>     sage: s = 'matrix(c(1, 1, 1, 1,' + ' '*988 + '2,2,2,2), ncol=4)'
>     sage: len(s)
>     1025
>     sage: r.eval(s)
>     ''
>
> I don't know why, but maybe that can help you track it down.

Thanks - that definitely helps, since pexpect is actually passing
strings.

Well, searching for "1024" in r.py results in two hits, and I think the relevant one is

                  # If an input is longer than this number of characters, then
                  # try to switch to outputting to a file.
                  eval_using_file_cutoff=1024)

If you input a string longer than 1024 characters, it writes it to a file and then tries to read that file.  It looks to me as though the method _read_in_file_command in r.py isn't doing the right thing: the correct string is getting written to the correct file, but then it's not getting imported properly.  That is, in expect.py,  it's executing this code:

        try:
            s = self._eval_line(self._read_in_file_command(tmp_to_use), allow_use_file=False)

and it's returning an empty string, even though tmp_to_use is set to the correct file name and that file has the correct contents.  I don't know R syntax, so I don't know what's going wrong.

--
John

[kcrisman]

> > > I don't know why, but maybe that can help you track it down.
>
> > Thanks - that definitely helps, since pexpect is actually passing
> > strings.
>
> Well, searching for "1024" in r.py results in two hits, and I think the

I don't know why that didn't occur to me.

> relevant one is

The other one seems to be a dud left over from interfaces/gp.py,
incidentally.

>
>                   # If an input is longer than this number of characters,
> then
>                   # try to switch to outputting to a file.
>                   eval_using_file_cutoff=1024)
>
> If you input a string longer than 1024 characters, it writes it to a file
> and then tries to read that file.  It looks to me as though the method
> _read_in_file_command in r.py isn't doing the right thing: the correct
> string is getting written to the correct file, but then it's not getting
> imported properly.  That is, in expect.py,  it's executing this code:
>
>         try:
>             s = self._eval_line(self._read_in_file_command(tmp_to_use),
> allow_use_file=False)
>
> and it's returning an empty string, even though tmp_to_use is set to the
> correct file name and that file has the correct contents.  I don't know R
> syntax, so I don't know what's going wrong.

Yeah, I'm investigating this, and somehow we are using source and file
not quite correctly. I've been trying a number of variations in the R
command line of Sage, and only can get something if I ask for verbose
output.


> source(file=file("/Users/.../.sage//temp/.../48141//interface//tmp48141",open="r"),verbose=TRUE)
'envir' chosen:<environment: R_GlobalEnv>
--> parsed 1 expressions; now eval(.)ing them:

>>>> eval(expression_nr. 1 )
=================

> c(1, 2, 3)
curr.fun: symbol c
[1] 1 2 3
.. after ‘expression(c(1, 2, 3))’


Hopefully I can track it down relatively quickly. Thank you very
much for your help on this.

- kcrisman

[kcrisman]

This is now http://trac.sagemath.org/sage_trac/ticket/11436.