hidding aliases from tab-completion

[Martin R]

With 10.8.beta6, several methods computing the number of things that do not follow the naming scheme `n_xxx` or `number_of_xxx` became aliases.  For example, we now have `Graph(3).n_vertices()`, and the method name `num_verts` is an alias for `n_vertices`.

IMPORTANT INTERLUDE: `num_verts` is not deprecated.

This move has some advantages, but also one disadvantage: it clutters tab-completion.

I experimented a little, and found a way to hide aliases from tab-completion, as follows:

class AlternatingSignMatrix(...):
    ...
    def __init__(self, parent, asm):
        ...
        self._matrix = asm
        self.__setattr__('number_negative_ones', self.number_of_negative_ones)
        Element.__init__(self, parent)
 
    def __dir__(self):
        return [f for f in super().__dir__() if f != "number_negative_ones"]

Is there any downside to this idea?  Of course, I would turn it into something easier to use, like a decorator.

Martin

PS: I found one "ordinary" class messing with `__dir__`, namely `Polyhedron_ZZ`, but I don't see any reason why the above approach would be problematic.

[Vincent Macri]

Aliases can be detected programmatically, that's how I removed them from the HTML docs (or more specifically, listed them as an alias with a link to the original instead of duplicating the docstring like we previously did) recently in https://github.com/sagemath/sage/pull/40753

There is a related issue to mark aliases in the interactive documentation: https://github.com/sagemath/sage/issues/40759

I like your suggestion not to pollute tab complete with aliases! But I don't think marking aliases should be a manual process or involve decorators, as that requires everyone to know about this and to consistently use it. It would also mean either a patch bomb to add the decorator to ever alias we alteady have, or many small patches adding them throughout the codebase. Both of those options are annoying.

My preference would be if whatever library/software handles tab autocomplete has a way to exclude things through its public API, which is however I removed aliases from the generated Sphinx docs. My second preference would be to programmatically detect them in __dir__ (perhaps in the SageObject __dir__ method?). I'm assuming this would have no impact on runtime performance of Sage, and __dir__ only gets called when the user tried to use tab autocomplete (if I'm wrong about how __dir__ works, then please correct me). We should definitely profile this change either way though if we are messing with magic methods of the base class of all Sage classes. I don't think tab complete performance itself is really a concern though. As long as it feels fast for users, adding even a couple milliseconds to tab autocomplete isn't an issue.

Vincent Macri (he/him)

[Martin R]

I don't think we should hide all aliases from tab-completion.  Apart from that, what my snippet does is essentially your second preference, except that removing an alias from `__dir__` is not sufficient to hide it from tab-completion.

Best wishes,

Martin

[Vincent Macri]

On 2025-10-07 11:51 a.m., 'Martin R' via sage-devel wrote:
> I don't think we should hide all aliases from tab-completion.

It's just string checks to detect aliases so you could filter to only
remove ones of a specific form such as methods of the form `num_xxx` or
`number_of_xxx` that are aliased to `n_xxx`. Personally I can't think of
a reason not to remove all aliases though (barring some edge cases such
as public methods aliased to a private method, but this can be handled),
but I don't feel too strongly either way so if you only want to apply it
to aliases of a certain form then sure.

> Apart from that, what my snippet does is essentially your second
> preference, except that removing an alias from `__dir__` is not
> sufficient to hide it from tab-completion.

It's not just removing it from `__dir__`? What else is needed?

As long as it's done automatically without programmer intervention, I
don't feel too strongly about the implementation details. I just don't
want something like adding @remove_alias_from_autocomplete to every
aliased method to become yet another quirk of Sage development that we
need to teach to every new contributor (and retroactively apply to the
codebase). Some amount of that is unavoidable in any large software
project, but the less quirks we have the better I think.

[Martin R]

> Apart from that, what my snippet does is essentially your second
> preference, except that removing an alias from `__dir__` is not
> sufficient to hide it from tab-completion.

It's not just removing it from `__dir__`? What else is needed? 

As I hinted at in the code snippet above, the alias cannot be defined in the ordinary way.  Below you can find a minimal example that shows the difference.  Both classes, `A` and `B` have a method `bad`, but in `A` this method is visible to ipython's tab-completion, in `A` it is not.

 

As long as it's done automatically without programmer intervention, I
don't feel too strongly about the implementation details. I just don't
want something like adding @remove_alias_from_autocomplete to every
aliased method to become yet another quirk of Sage development that we
need to teach to every new contributor (and retroactively apply to the
codebase).

Well, somehow you will have to tell ipython that a specific alias should not be visible to tab-completion.

Even if we want to hide all aliases: I don't know whether there is any way to distinguish an alias from a method defined as usual in python - how would you?

Best wishes,

Martin

class A:

    def good(self):
        print(17)

    bad = good

    def __dir__(self):
        return [x for x in super().__dir__() if x != "bad"]

class B:
    def __init__(self):
        self.__setattr__('bad', self.good)
   
    def good(self):
        print(42)

    def __dir__(self):
        return [x for x in super().__dir__() if x != "bad"]

[Nils Bruin]

On Tuesday, 7 October 2025 at 02:05:44 UTC-7 axio...@yahoo.de wrote:

Is there any downside to this idea?  Of course, I would turn it into something easier to use, like a decorator.

I haven't thought through all the ramifications of this, but as a general principle, being "smart" by putting exceptions in your code for particular behaviour tends to come back to haunt you later. The Zen of Python even addresses this:

Special cases aren't special enough to break the rules. 

My main problem is that __dir__ isn't solely for displaying tab completion results. It's an introspection tool. So I don't think __dir__ is an appropriate place to sanitize your tab completion lists. Tab completion is an interface job -- in this case IPython. If you think the tab completion results need sanitizing after obtaining them through introspection from python, then it needs to be happening there.

If you want to group aliases together: you can probably systematically determine if two attributes are aliases by looking at the ID they point to. If these end up being bound methods, you may have to dig into it a little bit (e.g., look at the __func__ attribute of the method instead). A fancy tab completion could group aliases together and perhaps even have a way of displaying a "preferred" one.

Once you have a protocol for tab completion sanitation, the question would be where you store the info on which attributes should be displayed (perhaps a ranking for inclusion in tab completion lists of various levels of completeness?) Should this info be distributed or should it be stored centrally? Should it be user configurable?

For the most part I think these questions end up being interface questions, not questions particular to sagemath (although the needs of sagemath might be a bit different than for other python projects), so perhaps the right level to engage with them is to contribute tools for it to IPython/jupyter. The feedback and positive or negative reception there might also be informative. There are other python-based projects with way more experience/focus on user interface than sagemath. Let's try to be compatible with that.

[Martin R]

I agree.

[Nils Bruin]

On Tuesday, 7 October 2025 at 12:49:35 UTC-7 Nils Bruin wrote:

For the most part I think these questions end up being interface questions, not questions particular to sagemath (although the needs of sagemath might be a bit different than for other python projects), so perhaps the right level to engage with them is to contribute tools for it to IPython/jupyter. The feedback and positive or negative reception there might also be informative. There are other python-based projects with way more experience/focus on user interface than sagemath. Let's try to be compatible with that.

As background for the last bit (and as indication that the sage needs may not be so different): if you've tried to use pandas by relying on tab completion (an indispensable tool to determine final grades for large courses with complicated scoring schemes), you'll quickly find that's not a nice experience either. Python attributes tend to be so plentiful and stuffed with technical things that bare tab completion is rarely concise enough to work well. Certainly the limited popups that Jupyter provides for them are way too small to get an overview. So I suspect most people learn to not rely primarily on tab completion for exploring a python library API or there is a big need for a better tab completion browser. 

[Vincent Macri]

I agree with what Nils said (what he said is what I was trying to get at when I said what my first preferred option would be). But after looking some things up it seems like modifying __dir__ is actually the recommended way of implementing this! I was surprised to learn this.

Below you can find a minimal example that shows the difference.  Both classes, `A` and `B` have a method `bad`, but in `A` this method is visible to ipython's tab-completion, in `A` it is not.

When I try your example in IPython `bad` is not visible to tab-completion for instances of `A` or instances of `B`. It is visible for `A` itself, but methods are normally called on instances, not classes so that's fine. When I try it with a regular Python prompt then it is as you said. So modifying __dir__ is sufficient for the IPython interface, and I'm happy to sacrifice alias handling for tab complete behaviour in a regular (not IPython) terminal if it means much simpler code. I'm not sure how many people use Sage in an interactive non-IPython terminal anyway.

The IPython docs do explicitly say that __dir__ is the correct way to control what shows up in tab autocomplete: https://ipython.readthedocs.io/en/stable/config/integrating.html#tab-completion

Moreover the Python docs say this about dir: https://docs.python.org/3/library/functions.html#dir

Because dir() is supplied primarily as a convenience for use at an interactive prompt, it tries to supply an interesting set of names more than it tries to supply a rigorously or consistently defined set of names, and its detailed behavior may change across releases. For example, metaclass attributes are not in the result list when the argument is a class.

So it seems that changing __dir__ is actually the correct way to implement this. I am surprised, I would have expected it to be via an IPython-specific interface. And given how the Python docs describe dir, it also seems like other introspection tools shouldn't be relying on it. Of course we'd need to try it to see for sure. Sphinx is probably the most likely thing to be relying on __dir__, so looking at how the generated docs turn out (especially with aliases) would be a good sanity check that nothing broke. Worst-case, since this wouldn't change any Sage functionality (so no deprecation policies apply), we could always revert the change if we find out later that it broke some obscure but important thing.

If we want this to apply to all Sage objects (I see no reason not to), then implementing this behaviour in __dir__ for `SageObject` seems appropriate.

Even if we want to hide all aliases: I don't know whether there is any way to distinguish an alias from a method defined as usual in python - how would you?

I did this in https://github.com/sagemath/sage/pull/40753 using the Sphinx API, but it can be done in pure Python. There are probably some edge cases the following code doesn't account for (lambdas, public aliases of private functions), but it illustrates the idea.

class A:
    def foo(self):
        return 5
    bar = foo

a = A()

a.__getattribute__('foo').__name__ == 'foo'  # True

a.__getattribute__('bar').__name__ == 'bar'  # False, because a.__getattribute__('bar').__name__ is 'foo'

So you can distinguish an alias like this. (Both 'foo' and 'bar' are in `a.__dir__()` by default, so iterate over each `s` in the super `__dir__` and check if `self.__getattribute__(s).__name__ == s`.)

And if you only want to exclude some aliases, you're working with strings so you can just write code to do whatever string processing you need to only apply it for some aliases. (So for your example, check if the alias is from num_xxx to n_xxx). Is there any reason not to filter out all public aliases of public methods though? The only thing I can think of is if the original non-alias name is something unexpected, but in that case we could just swap what's the original and what's the alias. Maybe there could be a few special cases we want to include both for (I don't think we are consistent about which of sqrt or square_root is the alias, maybe we include both in __dir__ until we refactor things to make one of them consistently the original).

Also, both Python and IPython exclude private (_foo) methods from the tab complete unless you try to tab complete on something starting with an underscore, I think this behaviour is fine and should be left alone. I'd only filter out public aliases of public methods.

[TB]

On 08/10/2025 1:24, Vincent Macri wrote:
> class A:
>     def foo(self):
>         return 5
>     bar = foo
>
> a = A()
>
> a.__getattribute__('foo').__name__ == 'foo'  # True
>
> a.__getattribute__('bar').__name__ == 'bar'  # False, because
> a.__getattribute__('bar').__name__ is 'foo'
>
> So you can distinguish an alias like this. (Both 'foo' and 'bar' are in
> `a.__dir__()` by default, so iterate over each `s` in the super
> `__dir__` and check if `self.__getattribute__(s).__name__ == s`.)

When handling inheritance, there is a disadvantage to defining an alias
with `bar = foo`. Using the same class A as above, the following may be
surprising:

sage: class B(A):
....: def foo(self):
....: return 42
....:
sage: b = B()
sage: b.foo()
42
sage: b.bar() # surprise?
5
sage: b.__getattribute__("bar").__name__
'foo'
sage: b.__getattribute__("foo").__name__
'foo'

In some places in Sage an alias `baz` will be defined as

def baz(self):
return self.foo()

and a short doc that says it is an alias, instead of the better solution
of having the full docstring of foo() with a prefix that tells this is
an alias.

Is there any idiom (or even a common library) to define aliases that
works for functions and methods, in python and in cython, with sensible
handling of docstrings and inheritance?

Regards,
TB

[Vincent Macri]

I mistakenly sent this reply directly to TB instead of the list:

On 2025-10-07 4:54 p.m., TB wrote:

> When handling inheritance, there is a disadvantage to defining an alias
> with `bar = foo`.

The advantage of course is less function call overhead than the alternative.

> instead of the better solution
> of having the full docstring of foo() with a prefix that tells this is
> an alias.

I think this could principle be handled automatically by Sphinx when
using bar = foo (this is https://github.com/sagemath/sage/issues/40759).

> sage: class B(A):
> ....:     def foo(self):
> ....:         return 42
> ....:
> sage: b = B()
> sage: b.foo()
> 42
> sage: b.bar() # surprise?
> 5
> sage: b.__getattribute__("bar").__name__
> 'foo'
> sage: b.__getattribute__("foo").__name__
> 'foo'

That's an interesting test case! But I think using __qualname__ instead
of __name__ allows these kinds of situations to be handled. I'm curious
if there is anywhere where this happens in Sage, if so that could be a
bug. I'm also wondering if this situation could be detected by linters.

[Nils Bruin]

On Tuesday, 7 October 2025 at 15:54:22 UTC-7 mathzeta2 wrote:

When handling inheritance, there is a disadvantage to defining an alias
with `bar = foo`.

[...] 

In some places in Sage an alias `baz` will be defined as

def baz(self):
return self.foo()

and a short doc that says it is an alias, instead of the better solution
of having the full docstring of foo() with a prefix that tells this is
an alias. 

 

In fact, this already points to an unintended complication with modifying the output of "dir":  Someone might be looking to subclass something and install new functionality. They check "dir" of the ancestor and find that the name they were thinking of to be unused, so they happily put their own function in. But as it happens, they have now overwritten an alias that was there for legacy reasons and now their subclassed objects fail to behave properly in other legacy contexts where other routines are relying on the aliases that are now overridden with different functionality. This could even lead to silent wrong results

[Vincent Macri]

I'm not too worried about that situation. From what the documentation for dir says, I don't think one should use dir to determine all the attributes a class has. Adding emphasis to the key parts:

Because dir() is supplied primarily as a convenience for use at an interactive prompt, it tries to supply an interesting set of names more than it tries to supply a rigorously or consistently defined set of names, and its detailed behavior may change across releases. For example, metaclass attributes are not in the result list when the argument is a class.

So:

1. The intended use of dir is for convenience at an interactive prompt
2. The list of names returned is not guarantees to be complete
3. The list of names is not guaranteed to be stable across releases

Altogether, this indicates to me that in your hypothetical the programmer made a mistake by relying on dir to provide a complete list of methods.

As for what to do, perhaps we should be relying on linters and typing.final / typing.overrides in situations like this to prevent inheritance problems.

Vincent Macri (he/him)

---

From: sage-...@googlegroups.com <sage-...@googlegroups.com> on behalf of Nils Bruin <nbr...@sfu.ca>
Sent: Tuesday, October 7, 2025 7:35:16 p.m.
To: sage-devel <sage-...@googlegroups.com>
Subject: Re: [sage-devel] hidding aliases from tab-completion

[△EXTERNAL]

[Martin R]

When I try your example in IPython `bad` is not visible to tab-completion for instances of `A` or instances of `B`. It is visible for `A` itself, but methods are normally called on instances, not classes so that's fine. When I try it with a regular Python prompt then it is as you said. So modifying __dir__ is sufficient for the IPython interface, and I'm happy to sacrifice alias handling for tab complete behaviour in a regular (not IPython) terminal if it means much simpler code. I'm not sure how many people use Sage in an interactive non-IPython terminal anyway.

That's interesting, so it possibly depends on the setup.  Here is a session on my computer:

sage: import IPython
sage: IPython.version_info
(8, 18, 1, '')
sage: ip = get_ipython()

sage: a = A()
sage: type(a)
<class '__main__.A'>
sage: ip.complete("a.b")
('a.b', ['a.bad'])
sage: b = B()
sage: type(b)
<class '__main__.B'>
sage: ip.complete("b.b")
('', [])
sage: b.bad()
42

(Ubuntu 24.04.3 LTS, SageMath 10.8.beta6, Python 3.12.3)

Martin

[TB]

On 08/10/2025 3:38, Vincent Macri wrote:
>> sage: class B(A):
>> ....:     def foo(self):
>> ....:         return 42
>> ....:
>> sage: b = B()
>> sage: b.foo()
>> 42
>> sage: b.bar() # surprise?
>> 5
>> sage: b.__getattribute__("bar").__name__
>> 'foo'
>> sage: b.__getattribute__("foo").__name__
>> 'foo'
>
> That's an interesting test case! But I think using __qualname__ instead
> of __name__ allows these kinds of situations to be handled. I'm curious
> if there is anywhere where this happens in Sage, if so that could be a
> bug. I'm also wondering if this situation could be detected by linters.
>

You are correct here, and also in your comment that typing.override can
help in certain situations. Here, a user-defined class that inherits
from a Sage class might be unaware of an alias, so also users will need
to somehow detect this.

Maybe your code for the docs that detects aliases, can later use some
magic to inspect the mro of those classes. I could not find previous
reports of such cases (in Sage or more generally in Python), though they
must have happen before, especially in large or complicated class
hierarchies.

Loosely related, I just opened #41022. The bug there, as far as I
understand, stems from not overriding a method in a derived class (but
unrelated to an alias).

Regards,
TB