Error in sage-5.0 + R: sage notebook does not show R parsing errors if length(input)>=1024 characters

[Maciek Sykulski]

Hi All,

I've compiled from source sage-5.0 on Gentoo on the following machine:
Linux 3.2.6 #3 SMP Thu Feb 16 17:40:05 CET 2012 x86_64 Intel(R)
Xeon(R) CPU X5690 @ 3.47GHz GenuineIntel GNU/Linux

Everything seems to work fine except for this:
When an input cell with some R code contains a syntax error and the
input is longer than 1023 characters, the cell evaluates without
printing any message at all.

This is exemplified below (evalute in R enironment):
----------------
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#234567890123456789012
]
----------------
The above code when evaluated, as it should, prints :
Error: unexpected ']' in "]"

However, the below code with one more letter inside comments evaluates
and nothing is printed at all:
----------------
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
#2345678901234567890123
]
----------------

Does anyone have a clue where in the source code this bug hides?
It's a serious bug, it makes developing R code rather difficult.

Thanks,
MS

[Nils Bruin]

On May 20, 7:12 am, Maciek Sykulski <macie...@gmail.com> wrote:
> Hi All,
>
> I've compiled from source sage-5.0 on Gentoo on the following machine:
> Linux 3.2.6 #3 SMP Thu Feb 16 17:40:05 CET 2012 x86_64 Intel(R)
> Xeon(R) CPU X5690 @ 3.47GHz GenuineIntel GNU/Linux
>
> Everything seems to work fine except for this:
> When an input cell with some R code contains a syntax error and the
> input is longer than 1023 characters, the cell evaluates without
> printing any message at all.

Confirmed. It's not confined to the notebook. Example:

sage: Ri=R() #make an R interpreter
sage: Ri.eval("#1"**3+"\n]")
'\nError: unexpected \']\' in "]"'
sage: Ri.eval("#1"**1000+"\n]")
''

I don't think this bug is new in 5.0, though. I also observe it in
4.8. The difference in behaviour probably comes from the fact that
sage's "expect" interfaces (this is used for R) prefer to communicate
long input via files rather than via the child's stdin. It reminds me
of

http://trac.sagemath.org/sage_trac/ticket/11401

which was merged in 4.7.2. Note that the API of Expect.eval got
changed a little (the parameters allow_use_file and split_lines got
changed). It's quite likely that the default settings for these
parameters don't work well for the R interface. In that case, the
right solution is to make sure that these defaults get overridden in
the R interface, which probably inherits from expect.

When I made the change I tried to ensure other interfaces weren't
adversely affected, but apparently this particular issue is not
doctested. The file of interest is:

sage/interfaces/r.py

The eval method there explicitly calls Expect.eval, so putting in the
right defaults should be straightforward. Setting split_lines=True
should get you back the original behaviour, but note that for the R
interface too, this might well cause grammatical input be delivered to
R in an ungrammatical way. Some more testing will be required.

[Nils Bruin]

As it turns out, experimentation is very easy:

sage: Ri=R()
sage: Ri.eval(("#1"**100+"\n")**10+"]")
''
sage: Ri.eval(("#1"**100+"\n")**10+"]",allow_use_file=False)
'\n\n\n\n\n\n\n\n\n\nError: unexpected \']\' in "]"'
sage: Ri.eval(("#1"**100+"\n")**10+"]",split_lines=True)
'\n\n\n\n\n\n\n\n\n\nError: unexpected \']\' in "]"'

As you see, disallowing using files or unconditionally splitting lines
both makes the problem go away. However, with split_lines=True, the
system really just splits lines and decides for each of those whether
to send them via file or directly to STDIN. The small line lengths
really are just avoiding file transfer. If we increase line length,
odd things happen:

sage: Ri.eval(("#1"**1000+"\n")**10+"]",split_lines=True)
'\n\n\n\nfile=file("...",open="r")\nsource(file)\n
\nfile=file("...",open="r")\nsource(file)\n\nfile=file("...",open="r")
\nsource(file)\n\n'

we see things going wrong. The split_lines=True was the old behaviour.
All these examples work reasonably well in 4.7.1 with the patch from
#11401 in place, so it may be something else after all (R did get
upgraded since then)

[Maciek Sykulski]

Thanks Nils.
It turns out the problem concerns any text output from R, not only
error messages.
In each case R code is being evaluated (ex. values of variables
change),
however no sign of feedback from R if the input is too long.

sage: Ri.eval(('#1')**100+"\n print(1)")
'\n[1] 1'
sage: Ri.eval(('#1')**1000+"\n print(1)")
''

For now, I'll stick with this change in local/lib/python/site-packages/
sage/interfaces/r.py
line 1045 from:
< return Expect.eval(self, code, synchronize=synchronize,
*args, **kwds)
to:
> return Expect.eval(self, code, synchronize=synchronize, allow_use_file=False, *args, **kwds)

By the way, is this mixing of positional (*args) and named arguments
done properly here?
Wouldn't a problem arise if someone passes unnamed *arg to eval, and
it's placed after named ones?

[kcrisman]

On May 20, 5:38 pm, Maciek Sykulski <macie...@gmail.com> wrote:
> Thanks Nils.
> It turns out the problem concerns any text output from R, not only
> error messages.
> In each case R code is being evaluated (ex. values of variables
> change),
> however no sign of feedback from R if the input is too long.

A related ticket is http://trac.sagemath.org/sage_trac/ticket/7681.
We currently have (unless it was changed in one of the updates? but I
don't think so)

# don't abort on errors, just raise them!
# necessary for non-interactive execution
self.eval('options(error = expression(NULL))')

but this doesn't actually raise an error. So this might be part of
your problem too.

- kcrisman

[kcrisman]

On May 20, 5:38 pm, Maciek Sykulski <macie...@gmail.com> wrote:

> Thanks Nils.
> It turns out the problem concerns any text output from R, not only
> error messages.
> In each case R code is being evaluated (ex. values of variables
> change),
> however no sign of feedback from R if the input is too long.
>
> sage: Ri.eval(('#1')**100+"\n print(1)")
> '\n[1] 1'
> sage: Ri.eval(('#1')**1000+"\n print(1)")
> ''
>
> For now, I'll stick with this change in local/lib/python/site-packages/
> sage/interfaces/r.py
> line 1045 from:
> <        return Expect.eval(self, code, synchronize=synchronize,
> *args, **kwds)
> to:
>
> >        return Expect.eval(self, code, synchronize=synchronize, allow_use_file=False, *args, **kwds)

Update:

Apparently I solved this a year ago, and forgot. Try the solution at
http://trac.sagemath.org/sage_trac/ticket/11436 and see if it is what
you are looking for.

[Maciek Sykulski]

The old patch somewhat works.
http://trac.sagemath.org/sage_trac/attachment/ticket/11436/trac_11436.patch

It didn't apply automatically because the source in sage-5.0
looks a bit different now:
---------
def _read_in_file_command(self, filename):
r"""
Return the R command (as a string) to read in a file named
filename into the R interpreter.

EXAMPLES::

sage: r._read_in_file_command('file.txt')
'file=file("file.txt",open="r")\nsource(file)'
"""
return 'file=file("%s",open="r")\nsource(file)'%filename
---------

Changing to
> return 'file=file("%s",open="r")\nsource(file,print.eval=TRUE)'%filename
doesn't do the trick, however reverting back to the old way, does:
> return 'source("%s",print.eval=TRUE)'%filename

The error message in the notebook prints full %filename, however.

- MS

> Apparently I solved this a year ago, and forgot.  Try the solution athttp://trac.sagemath.org/sage_trac/ticket/11436and see if it is what
> you are looking for.

[kcrisman]

On May 21, 5:08 pm, Maciek Sykulski <macie...@gmail.com> wrote:
> The old patch somewhat works.http://trac.sagemath.org/sage_trac/attachment/ticket/11436/trac_11436...

>
> It didn't apply automatically because the source in sage-5.0
> looks a bit different now:
> ---------
>     def _read_in_file_command(self, filename):
>         r"""
>         Return the R command (as a string) to read in a file named
>         filename into the R interpreter.
>
>         EXAMPLES::
>
>             sage: r._read_in_file_command('file.txt')
>             'file=file("file.txt",open="r")\nsource(file)'
>         """
>         return 'file=file("%s",open="r")\nsource(file)'%filename
> ---------
>
> Changing to>        return 'file=file("%s",open="r")\nsource(file,print.eval=TRUE)'%filename
>
> doesn't do the trick, however reverting back to the old way, does:
>
> >        return 'source("%s",print.eval=TRUE)'%filename
>
> The error message in the notebook prints full %filename, however.
>

I bet this is debuggable without changing to the old syntax (which
change I assume was made for some good reason in one of the more
recent R fixes). Can you put some of your experiments on that
ticket? (Yes, this is a not-so-subtle ploy to get you to acquire a
Trac account and lure you into Sage development!)

Thanks for helping look at this,
- kcrisman