Move the automatic_names() feature into the Sage interpreter proper

[Erik Bray]

Paul Zimmerman pointed out to me that there's a feature of the legacy
Sage Notebook, automatic_names() [1], which turns on automatic
creation of symbolic variables and functions when they are not already
defined. For example, by default if you enter:

sage: x + y + z

you get:

NameError: name 'y' is not defined

('y', in this case, because 'x' is pre-defined as a special case).
With automatic_names(True) it inserts a shim into the Sage syntax
pre-processor that automatically creates variables from names not
already found in globals().

I see no reason this feature needs to be confined to the legacy
Notebook, as opposed to being in Sage proper. Then that feature would
be usable at the command-line, as well as in the Jupyter Notebook.

Thoughts?

[1] https://github.com/sagemath/sagenb/blob/e6910891f445e47690760966441328971d51a78d/sagenb/misc/support.py#L602

[Samuel Lelievre]

Tue 2018-03-13 17:32:11 UTC, Erik Bray:

>
> Paul Zimmerman pointed out to me that there's a feature of the legacy
> Sage Notebook, automatic_names() [1], which turns on automatic
> creation of symbolic variables and functions when they are not already

> defined. [...]

> I see no reason this feature needs to be confined to the legacy
> Notebook, as opposed to being in Sage proper. Then that feature would
> be usable at the command-line, as well as in the Jupyter Notebook.
>
> Thoughts?

+1

[Vincent Delecroix]

I remember somebdy implementing directly in Ipython at some Sage days
(there is a way to plug hooks as we do with the preparser). The hook
itself was very naive (just a while loop catching NameError in sage_eval).

I am not able to find any trace of it.

But +1 for the feature at IPython level.

[Erik Bray]

On Tue, Mar 13, 2018 at 6:36 PM, Vincent Delecroix

<20100.d...@gmail.com> wrote:
> I remember somebdy implementing directly in Ipython at some Sage days
> (there is a way to plug hooks as we do with the preparser). The hook
> itself was very naive (just a while loop catching NameError in sage_eval).

That's actually almost exactly how it works in the Sage Notebook (it's
a for loop instead of a while loop with a limit of the number of names
to attempt to convert).

There's little about how it's implemented that isn't transferrable to IPython.

> On 13/03/2018 18:32, Erik Bray wrote:
>>
>> Paul Zimmerman pointed out to me that there's a feature of the legacy
>> Sage Notebook, automatic_names() [1], which turns on automatic
>> creation of symbolic variables and functions when they are not already
>> defined. For example, by default if you enter:
>>
>> sage: x + y + z
>>
>> you get:
>>
>> NameError: name 'y' is not defined
>>
>> ('y', in this case, because 'x' is pre-defined as a special case).
>> With automatic_names(True) it inserts a shim into the Sage syntax
>> pre-processor that automatically creates variables from names not
>> already found in globals().
>>
>> I see no reason this feature needs to be confined to the legacy
>> Notebook, as opposed to being in Sage proper. Then that feature would
>> be usable at the command-line, as well as in the Jupyter Notebook.
>>
>> Thoughts?
>>
>> [1]
>> https://github.com/sagemath/sagenb/blob/e6910891f445e47690760966441328971d51a78d/sagenb/misc/support.py#L602
>>
>

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

[Travis Scrimshaw]

On Wednesday, March 14, 2018 at 3:39:14 AM UTC+10, vdelecroix wrote:

I remember somebdy implementing directly in Ipython at some Sage days
(there is a way to plug hooks as we do with the preparser). The hook
itself was very naive (just a while loop catching NameError in sage_eval).

I am not able to find any trace of it.

But +1 for the feature at IPython level.

As I remember, it was Peleg Michael at SageDays79. Also, IIRC, he had come across the automatic_names() and decided it was not worth pursuing further.

Best,
Travis

[Simon King]

Hi!

On 2018-03-13, Erik Bray <erik....@gmail.com> wrote:
> I see no reason this feature needs to be confined to the legacy
> Notebook, as opposed to being in Sage proper. Then that feature would
> be usable at the command-line, as well as in the Jupyter Notebook.

If I understand correctly, if a user switches that feature on, any
undefined variable would automatically become a symbolic expression.
But on sage-support, I have far too often seen posts of useres that
mistook symbolic expressions as something that they clearly are not: A
universal tool that is the best for every purpose.

Having that feature available would encourage the misconception
about symbolic expressions. Thus, I find it a very bad idea and wouldn't
support it.

Best regards,
Simon

[kcrisman]

I recall discussions about this quite a few years ago, and the consensus was to be careful what you wish for.  Too many typos would then mysteriously become variables.  So (even though I certainly think it is a very useful facility in many case) the potential for confusion was pretty great.  I think a nice use case of it is at the top of a worksheet prepared by an instructor who is not a 100% novice.

By the way, there was a similar method (perhaps the same one) that allowed methods to be functions, e.g. you could do foo(x) instead of x.foo() for simplicity.

Anyway, certainly it should (in principle) be *implemented* for any Sage interface, including Jupyter.  I just don't know about it becoming the default.

- kcrisman

[Erik Bray]

Certainly nobody's talking about turning it on by default--just that
the feature exists, people do use it, and there's no reason it should
be exclusive to the sage notebook.

[kcrisman]

> Anyway, certainly it should (in principle) be *implemented* for any Sage
> interface, including Jupyter.  I just don't know about it becoming the
> default.

Certainly nobody's talking about turning it on by default--just that
the feature exists, people do use it, and there's no reason it should
be exclusive to the sage notebook.

+1 

[Emmanuel Charpentier]

Such a feature might come in handy so solve a couple of cases where sage creates (or receives from its subsystems) a symbolic expression with "new" variables.

Two chronic offenders are Maxima's solver and differential equation solver(s), which routinely add new symbolic constants as required by their inputs. In these cases, getting the names of these new variables and properly declaring them for future use can become a pain in the ass.

In these cases (and other possible automatic variable creation i do dot have in mind currently), such auto_declare behaviour should be the *default*.

I am less sanguine about automatic declaration of symbolic variables in general. Having to explicitly declare my symbolic variables saved my sorry ass from more than a couple of typos (and thinkos, BTW...). However, most CASes now available do away without this mandatory declaration.

One has also to consider the case (important in Sage) of the indeterminate(s) of a polynomial. Do we risk introducing an ambiguity if allowing automatic variable declaration ?

In any case, *if* this feature is deemed useful, then +1 for including it in the Sage interpreter itself, and not as an appendix of one of its ibnterfaces.

HTH,

--

Emmanuel Charpentier

[Simon King]

Hi Emmanuel,

On 2018-03-15, Emmanuel Charpentier <emanuel.c...@gmail.com> wrote:
> One has also to consider the case (important in Sage) of the
> indeterminate(s) of a polynomial. Do we risk introducing an ambiguity if
> allowing automatic variable declaration ?

Of course. If you create a polynomial ring (explicitly or implicitly)
with variable t, but do not inject t into the global name space, and
later refer to t (say, by copy-and-pasting some output), then with
automatic variable creation you would unintentionally create t as a
symbolic variable, and would be rather surprised, since you expected
it to behave like a polynomial indeterminate.

So, better raise a NameError (clearly showing that the polynomial
indeterminate t needs to be injected into the global name space) than
implicitly define t as something that it isn't supposed to be (namely:
a symbolic variable).

> In any case, *if* this feature is deemed useful, then +1 for including it
> in the Sage interpreter itself, and not as an appendix of one of its
> ibnterfaces.

I'd prefer to totally remove that feature. But maybe that's just me:
I don't like people to have guns. But perhaps others believe that it
is totally fine to encourage people to shoot themselves into the foot.

Cheers,
Simon

[William Stein]

On Thu, Mar 15, 2018 at 7:43 AM, Emmanuel Charpentier
<emanuel.c...@gmail.com> wrote:
> [...] However, most CASes now available do away without this mandatory declaration.

And hence Sage should have automatic_names as a non-default *option*.
See the mission statement, which is to create a viable alternative to
Maple, Mathematica, etc.

I wrote automatic_names (using catching of NameError or whatever)
originally. I don't remember why I put it in sagenb only, instead of
Sage. I regret not just putting it in Sage, since it's been on the
todo list to add this to CoCalc forever, and it seems annoying to even
have to add it again. That it is missing in CoCalc Sage worksheets
(and Jupyter) does
result in a periodic stream of complaints from users.

I don't remember, but automatic_names can (or should) print a message
when it automatically creates a new symbolic variable -- this helps
deal with some people's concerns...

--
William (http://wstein.org)

[Simon King]

On 2018-03-15, William Stein <wst...@gmail.com> wrote:
> I don't remember, but automatic_names can (or should) print a message
> when it automatically creates a new symbolic variable -- this helps
> deal with some people's concerns...

Indeed. Having a warning message alerting the user of potentially
unintended behaviour would be a good thing.

Best regards,
Simon

[Sébastien Labbé]

It was not Peleg but rather proposed by Konstantin Kliakhandler. Here is the ticket:

https://trac.sagemath.org/ticket/21959

which is in needs review status but does not apply cleanly anymore.

[Nils Bruin]

On Thursday, March 15, 2018 at 7:43:45 AM UTC-7, Emmanuel Charpentier wrote:

Two chronic offenders are Maxima's solver and differential equation solver(s), which routinely add new symbolic constants as required by their inputs. In these cases, getting the names of these new variables and properly declaring them for future use can become a pain in the ass.

In these cases (and other possible automatic variable creation i do dot have in mind currently), such auto_declare behaviour should be the *default*.

These are separate issues. The option "automatic_names"  inserts bindings to SR symbols when code that is interactively input references an unbound name (and would raise a NameError). That's orthogonal to SR symbols that get created without being bound to a "global" name. The latter creation may even happen when there's no interactive namespace present in the system, and certainly no obvious reference to one would be available. That means it's not even clear where to insert the binding.

Implementing the feature in IPython is exactly the right place, because that DOES have a good notion of what the interactive namespace would be. I assume CoCalc behaves (largely) as IPython for API purposes.

[William Stein]

If you implement this in IPython, then I'll still have to implement it
again for CoCalc,
since CoCalc Sage worksheets don't use IPython. I don't really see
any way around this though.
This feature is not implemented by parsing the code, so it has to be
done by whatever eval's the
code. We could technically add via the preprocessor some ugly while
try/except around *all*
code before it is evaluated, but that would be horrendous.

Anyway, that's why I didn't implement this in IPython originally. I
wanted it to be a feature available
to the Sage notebook, since anybody who actually wants to use it is
using the Sage notebook (not
the command line). If I implemented it in IPython, then it would not
be available in the Sage notebook, since
the Sage notebook makes no use of IPython at all (or at least it
didn't at the time).

-- William

>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

--
William (http://wstein.org)

[Nicolas M. Thiery]

On Thu, Mar 15, 2018 at 08:49:02AM -0700, William Stein wrote:
> On Thu, Mar 15, 2018 at 7:43 AM, Emmanuel Charpentier
> <emanuel.c...@gmail.com> wrote:
> > [...] However, most CASes now available do away without this mandatory declaration.
>
> And hence Sage should have automatic_names as a non-default *option*.

Yeah; it's all a balance between simplicity vs risk of confusion. From
using, teaching, and witnessing people teach SageMath, there are some
contexts where you (or more likely your teacher!) know what you are
doing, there is no risk of confusion, and simplicity primes. Most of
the time we don't want it though. So +1 as well on automatic_names as
an option not set by default. Also +1 on the message when a variable
is created.

Finally, I could suggest to write it as:

%automatic_names

to highlight that this is a magic command that alters the behavior of
the interpreter.

Cheers,
Nicolas
--
Nicolas M. Thiéry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/

[TB]

This is (similar to) the behavior of the IPython's built-in magic
command 'autocall', which allows calling functions without having to
type parentheses. For example:

sage: %autocall
Automatic calling is: Smart
sage: pow 3, 4
----> pow(Integer(3), Integer(4))
81

So I find the above suggestion of automatic_names to be consistent, at
least in this regard.

This lead me to discover a bug that arrives from the combination of the
Sage preparser and Python's int(). The following works:

sage: %autocall 2
Automatic calling is: Full

But without the "%" character, which is allowed because we have
%automagic on by default, it breaks:

sage: autocall 2
[snip]
sage/local/lib/python2.7/site-packages/IPython/core/magics/auto.pyc in
autocall(self, parameter_s)
108
109 if parameter_s:
--> 110 arg = int(parameter_s)
111 else:
112 arg = 'toggle'

ValueError: invalid literal for int() with base 10: 'Integer(2)'

Obviously, it does work without the preparser, or with
sage: autocall 2r

Regards,
TB

[Emmanuel Charpentier]

To summarize the previous answers (and nitpick a bit) : automatic_names

• may come handy ;
  
• shouldn't be mandatory
  
• shouldn't be the default
  
• should print a warning when fired.
  

I also liked the idea of introducing a new magic controlling that feature. However, I have a couple of remarks :

• Some functions (mostly inherited from Maxima) *do* already create new symbolic variables, but do not inject them in the relevant namespace. In *this* case (and this case only), I'd like to have this injection done by default, but controllable via an option. Prima facie, our targets are solve and desolve + their friends. But I forget some other ones almost surely... Hints ?
• The idea of a line magic (i. e. enabling automatic_variables for the next instruction or block) should be paired with a corresponding cell magic.
• Another pair of magics could control the *silent* automatic declaration of new symbolic variables.
  

The first case might be covered by patching the current implementation of the target functions ; the control of the injection of the new variables (in fact, symbolic constants) in the relevant name space could be controlled by yet another (pair of) magic(s).

Hence the possible design of the new magics :

• %(%)auto_constants : control whether Maxima-generated constants are injected in the relevant namespace. Default : True.
  
• %(%)auto_variables : control whether unknown names trigger the creation of symbolic variables thus named. Triggers a message. Defaulf : False.
  
• %(%)silent_auto_variables : control if the message signaling the creation of an automatic variable should vbe supressed. Defaulf : False.
  

Critics ? Suggestions ? Lazzi ?

This just leaves the $10000 question : who implements this ? ;-).

--

Emmanuel Charpentier

[Emmanuel Charpentier]

Addendum to the previous answer : solve at least may generate symbolic constants with domain constraints. If we accept the idea of auto-declaring them, should we add the relevant constraints ? Example :

sage: var("x,y")

(x, y)

sage: solve([3*x+y==0,6*x==-2*y],[x,y])

[[x == -1/3*r1, y == r1]]

Similarly :

sage: solve(sin(x)==1/2, x, to_poly_solve="force")

[x == 5/6*pi + 2*pi*z37, x == 1/6*pi + 2*pi*z39]

It seems straightforward to add "assume(z39, 'integer')" to the actions taken by an hypothetic %auto_constants magic in the second case. In the first case, it is less obvious : r1 could be any complex.

Ideas ?

-

Emmanuel Charpentier

[Nils Bruin]

On Monday, March 19, 2018 at 8:04:07 AM UTC-7, Emmanuel Charpentier wrote:

• Some functions (mostly inherited from Maxima) *do* already create new symbolic variables, but do not inject them in the relevant namespace. In *this* case (and this case only), I'd like to have this injection done by default, but controllable via an option. Prima facie, our targets are solve and desolve + their friends. But I forget some other ones almost surely... Hints ?

-2. This is only possible with very comprehensive hacks: the "interactive name-space" has no special status outside of IPython. The sage library wouldn't have an idea where to insert this stuff. We'd basically require a "scratchpad" onto which all code can scribble binding suggestions and then, once control comes back to IPython, a decision could be made on iwhether to nject these bindings. 

• %(%)auto_constants : control whether Maxima-generated constants are injected in the relevant namespace. Default : True.
  

-2 for the default; -1 in general. There's no trigger event in maxima to make these bindings. If you take the sage-maxima interface (which would see symbols from maxima it hasn't seen before), you could use that as a trigger, but then the behaviour of the interface relies on lifetime history! (and the fact that the sage-maxima interface is a memory leak for its translation dictionary).

These things happen more often:

sage: matrix(2,2,[1,2,3,4]).charpoly()
x^2 - 5*x - 2

creates a polynomial over ZZ. Should IPython rebind x to ZZ['x'].gen() ? Only when x is unbound? More generally:

sage: QQ['y','z']
Multivariate Polynomial Ring in y, z over Rational Field

should that trigger y and z being bound?

[Thierry]

On Mon, Mar 19, 2018 at 08:32:23AM -0700, Nils Bruin wrote:
> On Monday, March 19, 2018 at 8:04:07 AM UTC-7, Emmanuel Charpentier wrote:
> >
> >
> >

> > - Some functions (mostly inherited from Maxima) *do* already create

> > new symbolic variables, but do not inject them in the relevant namespace.
> > In *this* case (and this case only), I'd like to have this injection done

> > by default, but controllable via an option. *Prima facie,* our targets

> > are solve and desolve + their friends. But I forget some other ones almost
> > surely... Hints ?
> >
> > -2. This is only possible with very comprehensive hacks: the "interactive
> name-space" has no special status outside of IPython. The sage library
> wouldn't have an idea where to insert this stuff. We'd basically require a
> "scratchpad" onto which all code can scribble binding suggestions and then,
> once control comes back to IPython, a decision could be made on iwhether to
> nject these bindings.
>
> >

> > - %(%)auto_constants : control whether Maxima-generated constants are

> > injected in the relevant namespace. Default : True.
> >
> > -2 for the default; -1 in general. There's no trigger event in maxima to
> make these bindings. If you take the sage-maxima interface (which would see
> symbols from maxima it hasn't seen before), you could use that as a
> trigger, but then the behaviour of the interface relies on lifetime
> history! (and the fact that the sage-maxima interface is a memory leak for
> its translation dictionary).
>
> These things happen more often:
>
> sage: matrix(2,2,[1,2,3,4]).charpoly()
> x^2 - 5*x - 2
>
> creates a polynomial over ZZ. Should IPython rebind x to ZZ['x'].gen() ?
> Only when x is unbound? More generally:
>
> sage: QQ['y','z']
> Multivariate Polynomial Ring in y, z over Rational Field
>
> should that trigger y and z being bound?

In the same vein, what about:

- d+e ?
- i+j ?
- m+n ?
- r+s ?

Shall the Euler constant, the imaginary number, the numerical
approximation and the R interpreter be locally diabled ? What about
2*i*pi*j ?

What if some user use the automatic name "plop" in her worksheet and in a
later version Sage adds "plop" into the global namspace ?

Such features might look impressive for the 5 first minutes one is using
Sage, but they are adding confusion and do not help understanding nor
teaching in the longer term ("var('y')" is another famous example).

Ciao,
Thierry

[William Stein]

If I could do things over, then none of those would be in the global
namespace by default. This is very difficult to change now, due to
the assumptions in our docstrings, among other things...

> What if some user use the automatic name "plop" in her worksheet and in a
> later version Sage adds "plop" into the global namspace ?

I regret Sage having a default globals namespace with nearly 2000
items in it. I was too strongly influenced by Magma at the time. I'm
really glad that people haven't greatly increased the global namespace
during the last few years.


--
William (http://wstein.org)

[Samuel Lelievre]

Related ticket, mentioning var and automatic_names:

- SageMath #9048: inconsistencies between the terminal version and the notebook

  https://trac.sagemath.org/ticket/9048

  

[Erik Bray]

I'm not really sure I understand the problem here, though maybe I'm
being naïve. But ISTM the implementation of this is mostly the same
regardless what the frontend is. The only difference is in how one
hooks into the frontend to perform input preprocessing.

[Erik Bray]

On Mon, Mar 19, 2018 at 2:06 PM, Nicolas M. Thiery
<Nicolas...@u-psud.fr> wrote:
> On Thu, Mar 15, 2018 at 08:49:02AM -0700, William Stein wrote:
>> On Thu, Mar 15, 2018 at 7:43 AM, Emmanuel Charpentier
>> <emanuel.c...@gmail.com> wrote:
>> > [...] However, most CASes now available do away without this mandatory declaration.
>>
>> And hence Sage should have automatic_names as a non-default *option*.
>
> Yeah; it's all a balance between simplicity vs risk of confusion. From
> using, teaching, and witnessing people teach SageMath, there are some
> contexts where you (or more likely your teacher!) know what you are
> doing, there is no risk of confusion, and simplicity primes. Most of
> the time we don't want it though. So +1 as well on automatic_names as
> an option not set by default. Also +1 on the message when a variable
> is created.
>
> Finally, I could suggest to write it as:
>
> %automatic_names

+1 I like the idea of making it a magic. I would still probably keep
the automatic_names() function for backwards-compatibility. But maybe,
at least when running in IPython, display a deprecation warning in
favor of %automatic_names.

[kcrisman]

> Shall the Euler constant, the imaginary number, the numerical
> approximation and the R interpreter be locally diabled ? What about
> 2*i*pi*j ?

If I could do things over, then none of those would be in the global
namespace by default.   This is very difficult to change now, due to
the assumptions in our docstrings, among other things...

Well, do having those in the global namespace help us fulfill the mission statement?  To my mind, they definitely do.

Example: http://reference.wolfram.com/language/tutorial/MathematicalConstants.html

They do have the difference between () and [] and between upper/lowercase that makes some of that easier (though probably also more annoying).

- kcrisman