Start a Sage session from a session of its own Python interpreter ?

[Emmanuel Charpentier]

Motivation : see this ask.sagemath.org question.

Tl;dr : I want to call sage from the R reticulate package in order to create mixed text/R/Sage documents. 

• I am aware that Sagetex allows this, at least when using Sage's R in a \LaTeX document.
  
• But this solution doesn't extend to noweb Markdown documents, which are more and more in demand (Web pages, ebooks, other interactive gadgets).
  
• I have checked that the \LaTeX--> Markdown/whatever conversion currently doable with pandoc or similar tools is cumbersome to the extreme and/or loses a lot of information.

Done so far : when used as the "Python" interpreter, Sage starts a Python session of its own interpreter (i. e. doesn't start the usual IPython session). The various environment variables are correctly defined.

Thanks to Thierry Monteil, I have been able to create this Sage IPython session, with correct initialization (preparsing, imports, etc...). I have checked that this session can access  R objects, and that the R session can access objects created in Sage.

But this is insufficient :

• The Sage session does not (re-)starts automatically : one has to explicitly call IPython.embed(). Not a problem when tran manually ; problematic for the intended use (creating Sage code chunks in a noweb document).
  
• To get back to R, you have to exit twice : from the IPython session then from the Python session. Again not a problem in interactive use, again a serious problem for the intended use.

So the question is : how can one *replace* the Python REPL  with Sage's ? A serious look at $SAGE_ROOT/src/sage/repl/ipython_extension.py wasn't specially enlightening...

Any ideas ?

[ Note that this answer is necessary to the "right" function, but not sufficient : the name of the Python object interfacing Python to the *calling* R session is, unfortunately "r". Which is the standard name we have picked for our *called* R interpreter... So some r-handling will be necessary from reticulate's side. Keeping the distinction between those may be necessary (e. g. : reusing old code...).

But I feel that asking for a patch has better chances if we "do our homework first", by solving *our* side of the problem *before* asking for help... ]

[Jeroen Demeyer]

On 2018-08-13 19:07, Emmanuel Charpentier wrote:
> So the question is : how can one *replace* the Python REPL with Sage's ?

This seems like an instance of the XY problem: I don't think that
"replacing" REPLs is the correct way to deal with this. Can't you
configure/patch reticulate to use Sage instead instead of Python?

[Emmanuel Charpentier]

Dear Jeroen,

That's exactly what I'm trying to do. 

Before delving in reticulate surgery, I'm trying to use what it offers. I notice :

• use_python : path of the python binary. I set that to the path of my main sage *script* which sets correctly the various environment variables...
  
• repl_python : invokes the repl. Accepts an optional argument which is the name of a module imported before the REPL is launched.
  

So my tentative plan is either :

• write a module that imports IPython and launches  IPython.start_ipython(argv=['-c' '%load ext sage'])
  

This can be emulated by running this interactively with  py_run_string, and gives *partal* results, as illustrated below in a session of Sage's R interpreter (with comments) :

> library(reticulate)

> use_python("/usr/local/sage/sage", required=TRUE)

# No error : this has been accepted as a Python interpreter

> py_run_string("import IPython ; IPython.start_ipython(argv=['-c', '%load_ext sage'])")

# Again no error. And, by the way :

> py_run_string("print arctan(x).diff(x)")

1/(x^2 + 1)

# Sage is present, and gives sensible results to sensible inputs.

# Now, can we communicate between Sage and R ?

> py_run_string("foo=arctan(x).integrate(x)")

> py$foo

x*arctan(x) - 1/2*log(x^2 + 1)

We can fetch in R the value of a Sage variable

> paste("*** ",py_to_r(py$latex(py$foo))," ***")

[1] "***  x \\arctan\\left(x\\right) - \\frac{1}{2} \\, \\log\\left(x^{2} + 1\\right)  ***"

This result can be "sensibly" converted to R and used here (as a string. It would be interesting to try to create an R algebraic expression... But this supposes that we clear the Sage/reticulate nameclash. Later...

But there's a (large) fly in that ointment :

> py_run_string("print 2^3")

1

> py_run_string("print 2**3")

8

The  preparser is out to lunch.. And this has consequences...

> py_run_string("print (x^2).diff(x)")

Error in py_run_string_impl(code, local, convert) : 

  RuntimeError: Use ** for exponentiation, not '^', which means xor

in Python, and has the wrong precedence.

Detailed traceback: 

  File "<string>", line 1, in <module>

  File "sage/structure/element.pyx", line 955, in sage.structure.element.Element.__xor__ (build/cythonized/sage/structure/element.c:9013)

    raise RuntimeError("Use ** for exponentiation, not '^', which means xor\n"+\

# Sage gives us expressions that it won't read.

# But the problem appears to be limited to the preparser :

> py_run_string("print (x**2).diff(x)")

2*x

• Write a module that wil import * from sage.all and somehow starts a REPL using the preparser.
  

This I can't even emulate well In a new session of Sage's R : 

> library(reticulate)

> use_python("/usr/local/sage/sage")

> py_run_string("from sage.all import *")

> py_run_string("preparser(True)")

> py_run_string("print 2^3")

1

# Preparser out to lunch again...

> py_run_string("print sin(x).diff(x)")

Error in py_run_string_impl(code, local, convert) : 

  NameError: name 'x' is not defined

Detailed traceback: 

  File "<string>", line 1, in <module>

# x has not been defined as the default symbolic variable. This we can fix :

> py_run_string("x=var('x')")

> py_run_string("print sin(x).diff(x)")

cos(x)

Again, reasonable results of reasonable inputs.

Let's try the REPL :

> repl_python()

Python 2.7.15 (/usr/local/sage/sage)

Reticulate 1.10 REPL -- A Python interpreter in R.

This banner can be suppressed if necessary.

>>> 2^3

1

No preparser, again...

>>> tan(x).integrate(x).trig_simplify()

-log(cos(x))

Reasonable. Let's check that we can access that from R :

>>> bar=tan(x).integrate(x).trig_simplify()

>>> exit

> py$bar

-log(cos(x))

Seems so...

So that second solution, seemingly lighter, seems promising. 

But I have two obstacles 

1. What I know I don't know : how to insert the preparser ?
   
2. What I don't know I don't know : what am I forgetting ?
   

Ideas ? Pointers ?

[Erik Bray]

Did you try my suggestion of just setting `use_python(/path/to/sage-ipython)`?

[Erik Bray]

I'm not an R expert, so take with a grain of salt, but I spent a few
minutes reading the sources for reticulate and understanding how it
works, and I don't think my above suggestion will help you much (nor
will most of the things you've tried).

It mostly just uses the interpreter path passed to use_python() to run
a little Python script that it then uses to determine a few more
things about the Python in question. Most importantly, it calls that
Python to determine the correct libpython shared library to use. It
then dlopens() the appropriate libpython, and from then on uses the
Python C API to initialize and communicate with a Python interpreter
(including bringing up the REPL, as necessary). Most Python code you
send through it is just passed directly to the Python interpreter
through the C API, so of course things like the Sage preparsing won't
work, etc.

Sure, you can drop into an interactive REPL, and call IPython.embed(),
and you'll get an IPython REPL and all the trappings thereof, but that
will only work for interactive use.

I noticed that reticulate also comes with helper functions like
use_virtualenv() and use_condaenv(). It might be useful to add a
similar use_sageenv(), which would set the appropriate environment
variables (SAGE_ROOT, etc.) and tell reticulate to use Sage's Python.
But then to get things like preparsing working you'll have to write
wrappers around most of reticulate's other APIs as well. They might
also be amenable to a patch. It might not even need to be
Sage-specific; for example maybe it would be good to have an interface
transforming inputs prior to passing to the Python interpreter (as
IPython provides, allowing us to slip in Sage's preparse()).

[Emmanuel Charpentier]

Yes, I did, with the same results : I can (more or less easily) get a working Sage (i. e. access at least to SR and associated functins), but no preparser.

[Emmanuel Charpentier]

Dear Erik,

Thank you *ery* *much*. Comments below...

Le mardi 14 août 2018 15:45:00 UTC+2, Erik Bray a écrit :

On Tue, Aug 14, 2018 at 3:21 PM Erik Bray <erik....@gmail.com> wrote:
>
> On Mon, Aug 13, 2018 at 7:07 PM Emmanuel Charpentier
> <emanuel.c...@gmail.com> wrote:
> >
> > Motivation : see this ask.sagemath.org question.
> > Tl;dr : I want to call sage from the R reticulate package in order to create mixed text/R/Sage documents.
> >
> > I am aware that Sagetex allows this, at least when using Sage's R in a \LaTeX document.
> > But this solution doesn't extend to noweb Markdown documents, which are more and more in demand (Web pages, ebooks, other interactive gadgets).
> > I have checked that the \LaTeX--> Markdown/whatever conversion currently doable with pandoc or similar tools is cumbersome to the extreme and/or loses a lot of information.
> >
> > Done so far : when used as the "Python" interpreter, Sage starts a Python session of its own interpreter (i. e. doesn't start the usual IPython session). The various environment variables are correctly defined.
> >
> > Thanks to Thierry Monteil, I have been able to create this Sage IPython session, with correct initialization (preparsing, imports, etc...). I have checked that this session can access  R objects, and that the R session can access objects created in Sage.
> >
> > But this is insufficient :
> >
> > The Sage session does not (re-)starts automatically : one has to explicitly call IPython.embed(). Not a problem when tran manually ; problematic for the intended use (creating Sage code chunks in a noweb document).
> > To get back to R, you have to exit twice : from the IPython session then from the Python session. Again not a problem in interactive use, again a serious problem for the intended use.
> >
> > So the question is : how can one *replace* the Python REPL  with Sage's ? A serious look at $SAGE_ROOT/src/sage/repl/ipython_extension.py wasn't specially enlightening...
> >
> > Any ideas ?
> >
> > [ Note that this answer is necessary to the "right" function, but not sufficient : the name of the Python object interfacing Python to the *calling* R session is, unfortunately "r". Which is the standard name we have picked for our *called* R interpreter... So some r-handling will be necessary from reticulate's side. Keeping the distinction between those may be necessary (e. g. : reusing old code...).
> > But I feel that asking for a patch has better chances if we "do our homework first", by solving *our* side of the problem *before* asking for help... ]
>
> Did you try my suggestion of just setting `use_python(/path/to/sage-ipython)`?

I'm not an R expert, so take with a grain of salt, but I spent a few
minutes reading the sources for reticulate and understanding how it
works,

That was my next step...

 

and I don't think my above suggestion will help you much (nor
will most of the things you've tried).

It mostly just uses the interpreter path passed to use_python() to run
a little Python script that it then uses to determine a few more
things about the Python in question.  Most importantly, it calls that
Python to determine the correct libpython shared library to use.  It
then dlopens() the appropriate libpython, and from then on uses the
Python C API to initialize and communicate with a Python interpreter
(including bringing up the REPL, as necessary).  Most Python code you
send through it is just passed directly to the Python interpreter
through the C API, so of course things like the Sage preparsing won't
work, etc.

Sure, you can drop into an interactive REPL, and call IPython.embed(),
and you'll get an IPython REPL and all the trappings thereof, but that
will only work for interactive use.

I noticed that reticulate also comes with helper functions like
use_virtualenv() and use_condaenv().  It might be useful to add a
similar use_sageenv(), which would set the appropriate environment
variables (SAGE_ROOT, etc.) and tell reticulate to use Sage's Python.
But then to get things like preparsing working you'll have to write
wrappers around most of reticulate's other APIs as well.

Nice idea. I'l have first to learn what is a "Python environment...".

 

They might
also be amenable to a patch.  It might not even need to be
Sage-specific; for example maybe it would be good to have an interface
transforming inputs prior to passing to the Python interpreter (as
IPython provides, allowing us to slip in Sage's preparse()).

I'll have to read (a lot), and possibly write some trial code before I can know what I don't know/understand... (in other words, I'm too ignorant to assess the magnitude of my ignorance...).

Possible alternative : would it be possible to write a dedicated wrapper library with the same interface as the standard (?) python library doing this preprocessing, thus offering to Python-enabled programs (many exist, it seems...) an interface they can use directly ? Outside reticulate, I have a few possibilities in mind (among them Pythontex, which might benefit from such an addition...).

[William Stein]

On Tue, Aug 14, 2018 at 6:57 AM, Emmanuel Charpentier
<emanuel.c...@gmail.com> wrote:
>> On Tue, Aug 14, 2018 at 3:21 PM Erik Bray <erik....@gmail.com> wrote:
>> > On Mon, Aug 13, 2018 at 7:07 PM Emmanuel Charpentier

>> They might
>> also be amenable to a patch. It might not even need to be
>> Sage-specific; for example maybe it would be good to have an interface
>> transforming inputs prior to passing to the Python interpreter (as
>> IPython provides, allowing us to slip in Sage's preparse()).
>
>
> I'll have to read (a lot), and possibly write some trial code before I can
> know what I don't know/understand... (in other words, I'm too ignorant to
> assess the magnitude of my ignorance...).
>
> Possible alternative : would it be possible to write a dedicated wrapper
> library with the same interface as the standard (?) python library doing
> this preprocessing, thus offering to Python-enabled programs (many exist, it
> seems...) an interface they can use directly ? Outside reticulate, I have a
> few possibilities in mind (among them Pythontex, which might benefit from
> such an addition...).

+1 ! Making "the Sage preparser" actually a standard Python library
(on pypi, etc.) that Sage depends on
is one of the things I've long wished would happen. It would be
fantastic to optionally provide our preparser
functionality outside of Sage. E.g., imagine somebody using sympy and
having the option to use a configured-for-sympy version of the
preparser, for easier entry of formulas.

I've cc'd Robert Bradshaw, in case he has any thoughts, since he and I
mostly wrote the preparser.

-- William

[Erik Bray]

I don't know if that would be all that useful, at least not for
Emmanuel's case of using the reticulate R package. A lot of what the
the sage preparser does is ''very'' Sage-specific.

Perhaps if it were refactored into a class and aspects of it were made
more customizable I could see how it could be useful in conjunction
with other packages. But it's still not that useful at all outside
the context of an IPython interpreter that has the necessary hooks to
transform code before passing it to the Python interpreter.

Now, if the interpreter had a *built in* hook that would allow
arbitrary preparsing on inputs that would be very powerful (and
dangerous). But short of that, any program that wants to be able to
run Python statements (i.e. in something like reticulate, or any other
program that wraps a Python interpreter) has to provide its own
interface for pre-processing inputs to the Python interpreter, and
that's fine too.

[Emmanuel Charpentier]

It would solve my use case, for one... 

Perhaps if it were refactored into a class and aspects of it were made
more customizable I could see how it could be useful in conjunction
with other packages.  But it's still not that useful at all outside
the context of an IPython interpreter that has the necessary hooks to
transform code before passing it to the Python interpreter.

Indeed, what you obtain is a "Python-like" interpreter, which could differ *considerably* fro what Python is supposed to be...

For example, we could have, say, macroes ;-)... or even explicit blocks delimited with curly braces ;-] : two non-features of Python that make me still thoink that Lisp had its points...

 

Now, if the interpreter had a *built in* hook that would allow
arbitrary preparsing on inputs that would be very powerful (and
dangerous).

With great power...

 

 But short of that, any program that wants to be able to
run Python statements (i.e. in something like reticulate, or any other
program that wraps a Python interpreter) has to provide its own
interface for pre-processing inputs to the Python interpreter, and
that's fine too.

Except that this leaves me with the task of reinventing the wheel... Given my (non-)talents, there are some non-negligible hazards of ending up with something describing a seriously hypotrochoidal orbit (and frequent axle fractures)... 

[Erik Bray]

How do you think having the Sage preparser available as a stand-alone
package would solve your use case?

As it is, you can already wrap every line of code that reticulate
passes to sage with:

from sage.repl.preparse import preparse; preparse(line)

that's more or less what's needed here. Having in a separate package
(though maybe useful for other reasons) does not help you here,
because what you really need is a reasonable interface for inserting a
pre-parser between your input and the Python interpreter.

>> Perhaps if it were refactored into a class and aspects of it were made
>> more customizable I could see how it could be useful in conjunction
>> with other packages. But it's still not that useful at all outside
>> the context of an IPython interpreter that has the necessary hooks to
>> transform code before passing it to the Python interpreter.
>
>
> Indeed, what you obtain is a "Python-like" interpreter, which could differ *considerably* fro what Python is supposed to be...
>
> For example, we could have, say, macroes ;-)... or even explicit blocks delimited with curly braces ;-] : two non-features of Python that make me still thoink that Lisp had its points...

Yes, there's nothing stopping anyone from designing a super-set of
Python (or a completely different language) and providing a transpiler
to pure Python. That's what Sage's preparse already does. See also
much of the modern JavaScript community and Babel. See also Cython
(which does NOT transpile to Python, but rather to CPython API calls
which is always another possibility, albeit much harder...)

>> Now, if the interpreter had a *built in* hook that would allow
>> arbitrary preparsing on inputs that would be very powerful (and
>> dangerous).
>
>
> With great power...
>
>>
>> But short of that, any program that wants to be able to
>> run Python statements (i.e. in something like reticulate, or any other
>> program that wraps a Python interpreter) has to provide its own
>> interface for pre-processing inputs to the Python interpreter, and
>> that's fine too.
>
>
> Except that this leaves me with the task of reinventing the wheel... Given my (non-)talents, there are some non-negligible hazards of ending up with something describing a seriously hypotrochoidal orbit (and frequent axle fractures)...

I'm not really sure what you're saying here. All I'm saying is that
if you want the Sage preparser to work in reticulate, it needs to
provide a place that you can insert a pre-processor of some kind
(which is little more than "string in, string out"). That's something
I'd suggest raising with the reticulate developers, though you can to
some extent provide it yourself.

For example, whereas reticulate provides py_run_string, you can provide

sage_run_string <- function(code, local = FALSE, convert = TRUE) {
invisible(py_run_string(sprintf("from sage.repl.preparse
import preparse; preparse('%s')", code), local, convert))
}

There are some slight subtleties with quoting that this doesn't
handle. Also, you could provide some one-time initialization code for
ensuring that `preparse` is already imported and available as a
builtin so you don't need to run the import statement each time.
Those are left as exercises to the reader.

And ofc for things like integrating the Sage REPL, or the markdown
support, things get more complicated. But still again there's nothing
stopping someone from providing their own wrappers, just as we are
ultimately doing when we plug into IPython's input processing.

[Pedro Cruz]

>>> from sage.all import *

>>> exec(preparse(r"""
... a = 2^3
... print a
... """))
8
>>> a
8
>>> exit
> py$a
8

or also:

>>> load("some.sage")  #I believe all sage power is available in load() but not all in exec(preparse()).