remove log_html() and other log_*()

[Dima Pasechnik]

the code in src/sage/misc/log.py is quite old, and has no doctests.
As a result it has bitrotted due to an ipytthon update some years ago.

See https://trac.sagemath.org/ticket/29621

I think we should just remove this completely, as ipython nowadays has
%history magic which certainly can do the same as log_text()


Dima

[Michael Orlitzky]

On 4/30/20 11:16 AM, Dima Pasechnik wrote:
>
> I think we should just remove this completely, as ipython nowadays has
> %history magic which certainly can do the same as log_text()
>

+1

I feel the same way about functions like search_src() that badly
reimplement grep (even if they still work).

[John H Palmieri]

They're doctested, and they still work. What makes the implementation bad? They use standard Python library tools to walk a directory tree and then to do a regexp search on the files there. An advantage to this approach is that it is standard across platforms, as opposed to implementations of grep which differ on linux vs OS X (not to mention Solaris and others).

[Dima Pasechnik]

On Thu, 30 Apr 2020, 19:10 John H Palmieri, <jhpalm...@gmail.com> wrote:

On Thursday, April 30, 2020 at 9:20:00 AM UTC-7, Michael Orlitzky wrote:

On 4/30/20 11:16 AM, Dima Pasechnik wrote:
>
> I think we should just remove this completely, as ipython nowadays has
> %history magic which certainly can do the same as log_text()
>

+1

I feel the same way about functions like search_src() that badly
reimplement grep (even if they still work).

They're doctested, and they still work. What makes the implementation bad?

what do you mean by "doctested"?

yes, they are, they pass all 0 tests that are there :-)

They use standard Python library tools to walk a directory tree and then to do a regexp search on the files there. An advantage to this approach is that it is standard across platforms, as opposed to implementations of grep which differ on linux vs OS X (not to mention Solaris and others).

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/b62c455f-db74-499a-8b35-b80d24c4cb6e%40googlegroups.com.

[John H Palmieri]

On Thursday, April 30, 2020 at 11:15:31 AM UTC-7, Dima Pasechnik wrote:

On Thu, 30 Apr 2020, 19:10 John H Palmieri, <jhpalm...@gmail.com> wrote:

On Thursday, April 30, 2020 at 9:20:00 AM UTC-7, Michael Orlitzky wrote:

On 4/30/20 11:16 AM, Dima Pasechnik wrote:
>
> I think we should just remove this completely, as ipython nowadays has
> %history magic which certainly can do the same as log_text()
>

+1

I feel the same way about functions like search_src() that badly
reimplement grep (even if they still work).

They're doctested, and they still work. What makes the implementation bad?

what do you mean by "doctested"?

yes, they are, they pass all 0 tests that are there :-)

I was talking about search_src and related functions (i.e., the paragraph immediately preceding mine).

Back to the original topic, if log_html and friends are so obviously broken that they can't have been used within recent memory, I have no problems with removing them.

[Michael Orlitzky]

On 4/30/20 2:10 PM, John H Palmieri wrote:

>
> They're doctested, and they still work. What makes the implementation
> bad? They use standard Python library tools to walk a directory tree and
> then to do a regexp search on the files there. An advantage to this
> approach is that it is standard across platforms, as opposed to
> implementations of grep which differ on linux vs OS X (not to mention
> Solaris and others).

Most of the features of grep are standard:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html

The implementation in sage is,

* slow
* a weak, leaky abstraction (it's a thin wrapper around python
regex that doesn't expose all of the options and doesn't document
all of the details)
* redundant (we already require grep to build sage)
* only able to search one directory tree (not reusable knowledge
anywhere but in src/sage, unlike grep)
* a new syntax to learn on top of the syntax for grep, for most people
* code that we have to maintain forever
* code that has to be doctested 1000 times a day forever

It just doesn't need to be there. Users are better served by learning
how to use grep, and everyone would benefit from not having to
maintain/load/test it indefinitely.

[David Roe]

I understand where the criticism is coming from, but I think one of the big plusses of search_src and friends is that they're usable from the sage command line without switching context to the command line.  I don't think they should be removed.

David

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/5a14fb7f-4e12-c26c-7cf1-2590cf1e3481%40orlitzky.com.

[Michael Orlitzky]

On 4/30/20 3:11 PM, David Roe wrote:
>
> I understand where the criticism is coming from, but I think one of the
> big plusses of search_src and friends is that they're usable from the
> sage command line without switching context to the command line.  I
> don't think they should be removed.

sage: !grep
Usage: grep [OPTION]... PATTERNS [FILE]...
Try 'grep --help' for more information.

(No one else suggested removing search_*, I just think they fall into
the same category of "functions intended for people who don't know how
to use a computer but that are actually more complicated than the thing
they're supposed to replace.")

[John Cremona]

I agree with David.  I use grep 1000 times a day and I do know how to use a computer (and do not appreciate being patronised), but I also use serach_src() in the middle of Sage sessions a lot and tha seems much more useful to me than switching to a different window and navigating to wherever my Sage source code is.

[Dima Pasechnik]

On Fri, May 1, 2020 at 8:54 AM John Cremona <john.c...@gmail.com> wrote:
>
> I agree with David. I use grep 1000 times a day and I do know how to use a computer (and do not appreciate being patronised), but I also use serach_src() in the middle of Sage sessions a lot and tha seems much more useful to me than switching to a different window and navigating to wherever my Sage source code is.

You can do !grep

sage: !grep foo *

just fine, no need to leave Sage.

>
> On Thursday, April 30, 2020 at 8:55:37 PM UTC+1, Michael Orlitzky wrote:
>>
>> On 4/30/20 3:11 PM, David Roe wrote:
>> >
>> > I understand where the criticism is coming from, but I think one of the
>> > big plusses of search_src and friends is that they're usable from the
>> > sage command line without switching context to the command line. I
>> > don't think they should be removed.
>>
>> sage: !grep
>> Usage: grep [OPTION]... PATTERNS [FILE]...
>> Try 'grep --help' for more information.
>>
>> (No one else suggested removing search_*, I just think they fall into
>> the same category of "functions intended for people who don't know how
>> to use a computer but that are actually more complicated than the thing
>> they're supposed to replace.")
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/65829e6d-b7eb-4d48-a8a4-7c17d19829d5%40googlegroups.com.

[Michael Orlitzky]

On 5/1/20 3:54 AM, John Cremona wrote:
> I agree with David.  I use grep 1000 times a day and I do know how to
> use a computer (and do not appreciate being patronised)

I'm poking fun at the reasoning behind adding these functions to sage,
not the people who use them. I've been on the wrong side of a spacebar
heating[0] argument enough times to appreciate not wanting to change
what works. But we should have had some faith in people and suggested
grep from the start to avoid that problem.


[0] https://xkcd.com/1172/

[kcrisman]

I agree with David.  I use grep 1000 times a day and I do know how to use a computer (and do not appreciate being patronised), but I also use serach_src() in the middle of Sage sessions a lot and tha seems much more useful to me than switching to a different window and navigating to wherever my Sage source code is.

<nearflame>

I'm on this side, for slightly different reasons.   Having an *option* that is just a Sage function is something that should not be removed.  If it hadn't existed, maybe we don't build it, but there is no need to remove it.  Especially since we still don't (to my admittedly somewhat dated knowledge) have an easy way to search for partial words using tab-completion, so there should be a nice way to do so:

https://sagecell.sagemath.org/?q=xlwrqs

Contrast with "!grep genvecto *" which is still running many seconds later - and doesn't give me only definitions.  And that's because I used it in a directory which I knew wouldn't have many (or any).  Try this:

sage: !grep genvecto *

grep: ActiveGSLocalData: Is a directory

grep: Applications: Is a directory

<snip>

Oh, so now I have to use the -r flag ... but of course I'm in my HOME directory now.  Still waiting for the first one to show up, undoubtedly all my linear algebra materials should start cropping up.  Ooh, if I start it from that directory I get all kinds of non-Sage matches (37).

So now we have to teach the user about directory structure of the Sage bundle - after all, do we search the src/ directory or the local/ directory?  How do I find it from where I installed it?  Maybe this user has just laboriously created a symlink so they don't have to cd into the Sage directory every time, but now to search we have to.  Ugh.

By contrast, I used search_def (in my view, the most important of the search_* functions) all the time when I would have been scared off by grep.  If it is so slow, then in the documentation just put a WARNING:: block that it is slow and where to find out how to use !command.  I am always amazed by how unaware Sage developers, past and present, are of the difficulties people who are not used to command line have.

This is why people keep using things like SPSS for newcomers in intro stats - even though there are many good books using R at a similar level.  Because the interface is one people already scared of stats can at least sort of figure out.  I know of many experts in their fields who do just fine in publication, but stay as far away from non-GUI applications as they can, because (in their view) the time to learn how to use command line properly for every conceivable thing is a poor investment, as opposed to actually running experiments.  You can say they are wrong, or even mock them, or say we shouldn't have added these and given everyone a lesson in grep instead, but that is no reason to remove a function that works well enough for use.  And I'm certainly not suggesting we build something like this: https://reference.wolfram.com/language/howto/SearchForHelp.html  

As for the log functions, that is not necessarily a newbie function in any case!  And if it doesn't work anyway ... completely different situation.  I get that Python is the anti-Perl, "only one obvious way to do it".  That doesn't work so well in real life.

</nearflame>

[John Cremona]

Maybe. Does the !grep method work for non-linux systems such as Windows?

It took me a few minutes to work out what to put on the line after
!grep. I now see that to get essentially the same output as
search_src("has_cm")
I can type
!grep -r has_cm $SAGE_ROOT/src
and for search_def("has_cm"),
!grep -r "def has_cm" $SAGE_ROOT/src

I don't know whether python functions can call the ! things but I
think it would help users to keep search_src(s) as a shorthand for
!grep -r s $SAGE_ROOT/src and search_def(s) for search_src("def "+s).

John

>
>
> [0] https://xkcd.com/1172/

I'm a fan of xkcd, especially the ones that mention emacs customization!

>
> --
> You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/Q61rOkZ303M/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/824b0ba6-0743-cefe-3230-c2366ef02389%40orlitzky.com.

[kcrisman]

I don't know whether python functions can call the ! things but I
think it would help users to keep search_src(s) as a shorthand for
!grep -r s $SAGE_ROOT/src and search_def(s) for search_src("def "+s).

Thanks for more calmly and concisely making my point for me.

As for a relevant xkcd: https://xkcd.com/1168/

[Michael Orlitzky]

If you don't start sage from $SAGE_ROOT, then having an alias is nice,
but there's also a standard way to do that:

$ alias search_src="cd $SAGE_ROOT/src/sage; grep -r"

Now I can run whatever grep variant I want, easily, from anywhere:

$ search_src -il orlitzky graphs
graphs/generic_graph.py

(And pipe the result to another command, if I want). Point is, we don't
need to reimplement every fifty-year-old standard POSIX feature in
python to make it "usable." The people who know how to use grep/alias
far outnumber the people who know how to use sage's search_src(). The
people who know neither need to learn one, and they're better off
learning the reusable skill.

[Dima Pasechnik]

I would be in favour of deprecation and eventual removal of search_src()

just as we discard old implementations of maths functions in favour of maintained faster implementations in external libraries, we ought to discard outdated interfaces too.

Dima

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/c7dc3bdc-1334-3ec0-da13-e5c91a8b3510%40orlitzky.com.

[John H Palmieri]

There is a tradeoff. If I want to search the Sage source code, running "search_src(...)" in Sage is easy, and it's in our documentation, so people might find it. Running '!grep -R -n ... "$SAGE_ROOT"/src/sage' is not quite as easy; if you want the line numbers, you need -n, and it also requires knowing either that you can use $SAGE_ROOT when running Sage, or requires knowing the path to the Sage installation. Remember that some users are only accessing Sage via Jupyter notebooks or CoCalc, and maybe Unix paths and Unix commands are not at their fingertips.

So before deprecating, consider points of view of users from different backgrounds. We hope that mathematicians are using Sage, and we shouldn't require them to know about "grep". Don't get me wrong, grep is great and I use it all the time, but I bet that many of my colleagues don't know it, don't want to learn the syntax, and would view '!grep -R -n ... "$SAGE_ROOT"/src/sage' as an essentially magical incantation. I think that 'search_src(...)' is more accessible.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.

[kcrisman]

So before deprecating, consider points of view of users from different backgrounds. We hope that mathematicians are using Sage, and we shouldn't require them to know about "grep". Don't get me wrong, grep is great and I use it all the time, but I bet that many of my colleagues don't know it, don't want to learn the syntax, and would view '!grep -R -n ... "$SAGE_ROOT"/src/sage' as an essentially magical incantation. I think that 'search_src(...)' is more accessible.

Not to mention the students who we hope will grow into mathematicians and other scientists using Sage (and grep and whatever), but who when they start will not be there yet.

[Michael Orlitzky]

On 5/1/20 1:36 PM, John H Palmieri wrote:
>
> So before deprecating, consider points of view of users from different
> backgrounds. We hope that mathematicians are using Sage, and we
> shouldn't require them to know about "grep". Don't get me wrong, grep is
> great and I use it all the time, but I bet that many of my colleagues
> don't know it, don't want to learn the syntax, and would view '!grep -R
> -n ... "$SAGE_ROOT"/src/sage' as an essentially magical incantation. I
> think that 'search_src(...)' is more accessible.
>

This doesn't have to involve an incantations at all. Grep is already
there (and superior) for anyone who wants to use one, but you're free to
open up a GUI file manager, browse to the sage directory, click a few
things on a drop-down menu, and ask it to search for your string that
way. That's another relatively reusable skill that many people already
know and that solves the same problem on any platform. Grep is really
just a special case of that, "do whatever you would normally do to
search for files containing a string."

[kcrisman]

On Saturday, May 2, 2020 at 7:12:42 AM UTC-4, Michael Orlitzky wrote:

On 5/1/20 1:36 PM, John H Palmieri wrote:
>
> So before deprecating, consider points of view of users from different
> backgrounds. We hope that mathematicians are using Sage, and we
> shouldn't require them to know about "grep". Don't get me wrong, grep is
> great and I use it all the time, but I bet that many of my colleagues
> don't know it, don't want to learn the syntax, and would view '!grep -R
> -n ... "$SAGE_ROOT"/src/sage' as an essentially magical incantation. I
> think that 'search_src(...)' is more accessible.
>

This doesn't have to involve an incantations at all. Grep is already
there (and superior) for anyone who wants to use one, but you're free to
open up a GUI file manager, browse to the sage directory,

You've already lost any Sage user who doesn't understand the structure of Sage, e.g. someone using it only through a notebook.

 

click a few
things on a drop-down menu, and ask it to search for your string that
way. That's another relatively reusable skill that many people already
know and that solves the same problem on any platform. Grep is really
just a special case of that, "do whatever you would normally do to
search for files containing a string."

I'm not sure that the sort of user we are talking about would think of "search for files containing a string".  Ideally we are attracting users who don't know what a string is at all, but want to get some math done.  Eventually, yes, for power usage you would want to know that, but to start you can just ape the commands, and later learn that there is a special type of Python thingie in quotes called a string - and learn the difference between raw strings and so forth.

Also, search_def is the one I found most useful when learning Sage properly, and the GUI manager method requires people to know the "def" is the word, not "definition" or even "function" or whatever.

It's fine if there is a platform-consistent better way to implement it with some sort of alias.  But it seems perverse to remove something useful, FINDABLE BY TAB COMPLETION, and instead make people learn to use these other, more tedious or more arcane, methods, when it already exists.  It would be sort of like removing a "help()" command in some other program because there is some other way the system can find help files which a power user should know, and which we could bury in documentation somewhere.  

That's fine for a dedicated user, but one would hope Sage would continue to support even the least dedicated user.  The mission statement would indicate we want to.

[Michael Orlitzky]

On 5/2/20 9:20 AM, kcrisman wrote:
>
> Also, search_def is the one I found most useful when learning Sage
> properly, and the GUI manager method requires people to know the "def"
> is the word, not "definition" or even "function" or whatever.
>

Learning the word "def" is no harder than learning the word
"search_def," and again, the former is at least reusable knowledge.

I'll stop beating the dead horse; but I think we're catering to a user
with a highly peculiar set of disabilities.

[Sébastien Labbé]

I feel the same way about functions like search_src() that badly
reimplement grep (even if they still work).

I am fine with getting rid of the log_* functions, but I definitively want search_src(), search_def() and search_doc() to stay. Shame on me, but I use them when I need from the sage command line as well as the `sage -grep` instead of grep when I want to search the sage source from *any* directory on my computer.

For the same reason, I must also admit that I use some alias in my .bashrc file and I also define names that I can remember in .ssh/config file instead of ip adresses of servers which I just can't remember by heart :)

Sébastien

[John H Palmieri]

On Saturday, May 2, 2020 at 9:59:18 AM UTC-7, Sébastien Labbé wrote:

I feel the same way about functions like search_src() that badly
reimplement grep (even if they still work).

I am fine with getting rid of the log_* functions, but I definitively want search_src(), search_def() and search_doc() to stay. Shame on me, but I use them when I need from the sage command line as well as the `sage -grep` instead of grep when I want to search the sage source from *any* directory on my computer.

OMG, why does "sage -grep" use the "find" command?

[Matthias Koeppe]

The answer is probably something like "Solaris 7 grep does not accept -r"

[Dima Pasechnik]

HP-UX grep surely does not.

>
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/602d2d75-800c-498e-af12-2efd3980ec79%40googlegroups.com.

[Sébastien Labbé]

I don't know, but just to mention that `sage -grep` can also be called as `sage -search_src` which is the equivalent of search_src() from the command line.

$ sage -advanced

...

  -search_src <string> -- search through all the Sage library code for string
  -search_doc <string> -- search through the Sage documentation for string
  -grep <string>      -- same as -search_src
  -grepdoc <string>   -- same as -search_doc
...

 

 

[John H Palmieri]

And search_src (etc.) used to be implemented with a combination of find and grep, just like sage -grep now. It was reimplemented in Python to make it work across platforms (because of differences in grep syntax, for example) and to make it faster (true story! see #6429).

If anyone has any interest in developing Sage for Solaris or other platforms whose grep doesn't support "-r", this is another reason to keep search_src, etc.

[Michael Orlitzky]

On 5/2/20 1:55 PM, John H Palmieri wrote:
>
> OMG, why does "sage -grep" use the "find" command?
>

Others have pointed out that "-r" isn't standard, but "-r" is wrong
anyway. It's only supposed to search through python files. And having
"find" look for those files isn't any slower than having grep do it --
that's why "-r" isn't a standard flag, it's redundant.

But your OMG is justified for another reason,

find sage -print | GREP_OPTIONS= egrep '.py([xdi])?$' | xargs grep "$@"

is calling egrep on the output to find the files with pythonic
extensions, when "find -name" exists to do just that. Also, a pointless
use of xargs. Here's what it should be:

find ./ \( -name '*.py' \
-o -name '*.pyx' \
-o -name '*.pyd' \
-o -name '*.pyi' \) \
-exec grep "$@" {} +

That's all POSIX and can be run with /bin/sh and not /bin/bash.

[John H Palmieri]

And to clarify, this is what you expect users to use instead of search_src?  ;)

 

[Dima Pasechnik]

I suspect this is meant as improvement to Sage code.

>
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/53db82ce-edd9-40f8-adf7-f58958f54f44%40googlegroups.com.

[Michael Orlitzky]

On 5/6/20 11:28 PM, John H Palmieri wrote:
>
> And to clarify, this is what you expect users to use instead of
> search_src?  ;)
>

It's an improvement to sage-grep. In another message you said, "It was
reimplemented in Python to make it work across platforms... and to make
it faster." It's not a fair comparison if you don't write the find-grep
command in a portable/efficient way. The one I gave is portable (the
trailing "+" was added to POSIX around Y2K) and as fast as possible.

A script that we ship to end users has to be portable, which is why that
command is a mouthful even though most of that syntax has been in "man
find" forever. But individual users only need something that works on
their machines. So POSIX trivia aside, "do whatever you would normally
do to search a bunch of files for a string" is still the best answer,
and will be far simpler than a command that needs to work on last week's
Fedora and last decade's Solaris. In real life, I would "grep -irl" the
whole directory, because that's what I normally do and is easy for me.

[kcrisman]

A script that we ship to end users has to be portable, which is why that
command is a mouthful even though most of that syntax has been in "man
find" forever. But individual users only need something that works on
their machines. So POSIX trivia aside, "do whatever you would normally
do to search a bunch of files for a string" is still the best answer,

 

Again, I think it is not necessarily the case that users of Sage-the-software - say, in a CoCalc-provided notebook as a student - necessarily know how to "search a bunch of files for a string" or even know that there is such a thing.  On Mac most ordinary users probably just use the Spotlight function, which searches the entire system.  If you've ever searched for Sage stuff using that, you will know it is not super friendly (and then you have to wait for the system to open up Xcode to read the files, and it finds .c files, .py, .pyc, .pyx with the same string...).  

And even that doesn't work in a notebook you aren't hosting or on Sage cell server, obviously, and again many users may not even know there is a directory structure at all.   This is not a "highly peculiar set of disabilities" - rather, the skill set of people on sage-devel is a "highly peculiar set of abilities", even among people doing math on a regular basis.

[Michael Orlitzky]

On 5/7/20 8:45 AM, kcrisman wrote:
>  
> Again, I think it is not necessarily the case that users of
> Sage-the-software - say, in a CoCalc-provided notebook as a student -
> necessarily know how to "search a bunch of files for a string" or even
> know that there is such a thing.

Anyone who wants to search the source code for a string probably knows
that source code and strings both exist, and that one can be searched
for the other. I believe in the user who knows nothing, but I don't
believe that he's interested in search_src() to begin with. (This is the
sort of thing I mean by a peculiar set of disabilities.)

> On Mac most ordinary users probably
> just use the Spotlight function, which searches the entire system.  If
> you've ever searched for Sage stuff using that, you will know it is not
> super friendly (and then you have to wait for the system to open up
> Xcode to read the files, and it finds .c files, .py, .pyc, .pyx with the
> same string...).  
>
> And even that doesn't work in a notebook you aren't hosting or on Sage
> cell server, obviously, and again many users may not even know there is
> a directory structure at all.   This is not a "highly peculiar set of
> disabilities" - rather, the skill set of people on sage-devel is a
> "highly peculiar set of abilities", even among people doing math on a
> regular basis.

These are both valid concerns. I disagree only in the way we should
address them. Instead of a magic function with a non-standard name that
only kind-of works and that we have to maintain forever, I think these
two cases are better served by some documentation:

1. SageMath is an open source project, and all of its code is freely
available <link to source code>. The code can even be browsed
online <link to github/gitlab>.

2. The source code for SageMath is mostly python, and if you want to
see how something works, just search through sage's python source
files for it, and read the code.

3. Here are a few examples showing you how to search through files
on some common operating systems...

4. For advanced users, here are some incantations that will do the
job quickly from within a sage terminal or notebook...

The fact that sage is just a bunch of files in directories simplifies
the mental model, compared to a search_src() incantation that runs in a
web browser and returns something from somewhere. The ability to "do
what you would normally do" is similarly comforting to the people who
normally do it. The rest are better served by some pointers on how to
use their operating system's abilities.

[John H Palmieri]

On Thursday, May 7, 2020 at 4:40:52 AM UTC-7, Michael Orlitzky wrote:

On 5/6/20 11:28 PM, John H Palmieri wrote:
>
> And to clarify, this is what you expect users to use instead of
> search_src?  ;)
>

And to clarify, neither you nor Dima understand ";)"?

[John H Palmieri]

And in real life, I tend to use "git grep ..." so I can easily ignore large parts of the directory.

[kcrisman]

On Thursday, May 7, 2020 at 9:17:34 AM UTC-4, Michael Orlitzky wrote:

On 5/7/20 8:45 AM, kcrisman wrote:
>  
> Again, I think it is not necessarily the case that users of
> Sage-the-software - say, in a CoCalc-provided notebook as a student -
> necessarily know how to "search a bunch of files for a string" or even
> know that there is such a thing.

Anyone who wants to search the source code for a string probably knows
that source code and strings both exist, and that one can be searched
for the other. I believe in the user who knows nothing, but I don't
believe that he's interested in search_src() to begin with. (This is the
sort of thing I mean by a peculiar set of disabilities.)

We can agree to disagree on this, but "search_def()" is super-duper useful and for that one this argument does not apply.  Often that was (for me) far quicker than trying to search through the voluminous Sage documentation - and note that the online doc doesn't include *every* method.

 

These are both valid concerns. I disagree only in the way we should
address them. Instead of a magic function with a non-standard name that
only kind-of works and that we have to maintain forever, I think these
two cases are better served by some documentation:

Maybe.  In principle I agree with that, but in reality and practice (e.g. the several years I spent helping on ask.sagemath.org on a daily basis), I have found that even the most well-meaning and motivated seekers have a lot of trouble traversing our documentation.  (And to be fair, traversing other documentation, though ours is perhaps less well organized than ideal.)  If we really had this type of doc - and if it could really deal with some of the issues for GUI/notebook-only users or people who barely know how to use the command line, and was really just as easy to find - then of course it would be fine to deprecate these guys.  I'm just really unconvinced that any Sage documentation project will ever achieve that, after many attempts :-)

[kcrisman]

On a related note, that was one of the awesome things about the (now-deprecated) sagenb.  You clicked "Help" and got immediate links to very relevant Sage-specific help, including for the notebook itself. (Did I learn about search_def() from that 13 years ago?  I don't know any more.)

[Sebastian Oehms]

On Thursday, May 7, 2020 at 2:45:51 PM UTC+2, kcrisman wrote:

... and again many users may not even know there is a directory structure at all.   This is not a "highly peculiar set of disabilities" - rather, the skill set of people on sage-devel is a "highly peculiar set of abilities", even among people doing math on a regular basis.

I have never used search_src/_def/_doc. But in general, I would not deprecate a well working functionality without a real need. Surely, there are users to whom this will be annoying. But if we talk about users who are not interested in code and strings, wouldn't it be more useful for them to have a function that searches through the global name space? Do we have such a one?
The tutorial mentions Tab completion. But that only helps if you already know the beginning of the word:

 

sage: def search_names(string):
....:     g = globals()
....:     for s in g.keys():
....:         if s.find(string)>=0:
....:             print(s)

sage: search_names('poly')
bell_polynomial
bernoulli_polynomial
characteristic_polynomial
charpoly
conway_polynomial
cyclotomic_polynomial
exists_conway_polynomial
hilbert_class_polynomial
hyperbolic_polygon
hyperbolic_regular_polygon
lattice_polytope
lfsr_connection_polynomial
minimal_polynomial
minpoly
polygen
polygens
polygon
polygon2d
polygon3d
polygon_spline
polygons3d
polylog
polymake
polytopes

BTW: How do you use Tab completion on SageMathCell?

[Markus Wageringel]

Am Freitag, 8. Mai 2020 09:03:08 UTC+2 schrieb Sebastian Oehms:

But if we talk about users who are not interested in code and strings, wouldn't it be more useful for them to have a function that searches through the global name space? Do we have such a one?

We do indeed. It is an IPython feature. You can use

sage: *poly*?

which is equivalent to

sage: %psearch *poly*

The psearch magic also has options such as case-insensitivity

sage: %psearch -i *poly*

As for SageMathCell, I seem to recall that tab completion used to work there as usual, but currently it does not work for me either.

[Dima Pasechnik]

On Fri, May 8, 2020 at 10:37 AM Markus Wageringel
<markus.w...@gmail.com> wrote:
>
> Am Freitag, 8. Mai 2020 09:03:08 UTC+2 schrieb Sebastian Oehms:
>>
>> But if we talk about users who are not interested in code and strings, wouldn't it be more useful for them to have a function that searches through the global name space? Do we have such a one?
>
>
> We do indeed. It is an IPython feature. You can use
>
> sage: *poly*?
>
> which is equivalent to
>
> sage: %psearch *poly*
>
> The psearch magic also has options such as case-insensitivity
>
> sage: %psearch -i *poly*

given that iPython features came after search_src(), it'd be prudent
to move to be using them.

>
>
> As for SageMathCell, I seem to recall that tab completion used to work there as usual, but currently it does not work for me either.
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/f0b327fc-eb76-43ca-9233-7cec554ab6bc%40googlegroups.com.

[Sebastian Oehms]

On Friday, May 8, 2020 at 11:37:03 AM UTC+2, Markus Wageringel wrote:

Am Freitag, 8. Mai 2020 09:03:08 UTC+2 schrieb Sebastian Oehms:

But if we talk about users who are not interested in code and strings, wouldn't it be more useful for them to have a function that searches through the global name space? Do we have such a one?

We do indeed. It is an IPython feature. You can use

sage: *poly*?

which is equivalent to

sage: %psearch *poly*

The psearch magic also has options such as case-insensitivity

sage: %psearch -i *poly*

Nice! I didn't know that. I think that should be in the tutorials (where Tab complition is explained) and on the SageMathCell-Page.

[John H Palmieri]

On Friday, May 8, 2020 at 3:08:21 AM UTC-7, Dima Pasechnik wrote:

On Fri, May 8, 2020 at 10:37 AM Markus Wageringel
<markus.w...@gmail.com> wrote:
>
> Am Freitag, 8. Mai 2020 09:03:08 UTC+2 schrieb Sebastian Oehms:
>>
>> But if we talk about users who are not interested in code and strings, wouldn't it be more useful for them to have a function that searches through the global name space? Do we have such a one?
>
>
> We do indeed. It is an IPython feature. You can use
>
> sage: *poly*?
>
> which is equivalent to
>
> sage: %psearch *poly*
>
> The psearch magic also has options such as case-insensitivity
>
> sage: %psearch -i *poly*

given that iPython features came after search_src(), it'd be prudent
to move to be using them.

They accomplish different things: one searches the global name space and the other searches the source code.

Example: can Sage compute any fundamental groups?

sage: %psearch *fundamental_group*


sage: search_def("fundamental_group")
schemes/curves/zariski_vankampen.py:380:def fundamental_group(f, simplified=True, projective=False):
schemes/curves/projective_curve.py:1599:    def fundamental_group(self):
schemes/curves/affine_curve.py:1610:    def fundamental_group(self):
combinat/root_system/extended_affine_weyl_group.py:757:    def fundamental_group(self):
combinat/root_system/extended_affine_weyl_group.py:1387:            def to_fundamental_group(self):
combinat/root_system/extended_affine_weyl_group.py:2263:        def to_fundamental_group(self):
combinat/root_system/extended_affine_weyl_group.py:2432:        def to_fundamental_group(self):
knots/link.py:446:    def fundamental_group(self, presentation='wirtinger'):
categories/simplicial_sets.py:262:            def fundamental_group(self, simplify=True):
homology/simplicial_complex.py:3936:    def fundamental_group(self, base_point=None, simplify=True):

 

>
>
> As for SageMathCell, I seem to recall that tab completion used to work there as usual, but currently it does not work for me either.
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.

[Michael Orlitzky]

On 5/8/20 12:11 PM, John H Palmieri wrote:
>
> They accomplish different things: one searches the global name space and
> the other searches the source code.
>
> Example: can Sage compute any fundamental groups?
>

Neither approach returns the best result. Putting

site:doc.sagemath.org fundamental group

into a search engine gives you what you're looking for:

http://doc.sagemath.org/html/en/reference/combinat/sage/combinat/root_system/fundamental_group.html

[John H Palmieri]

As a topologist, I would have to say that the best results are the ones for simplicial complexes and simplicial sets, actually.

More seriously, I think that search_def should include "class" as well as "[c]?def" as parts of the regular expression forming the search. That still doesn't find the case you mentioned, because it uses "FundamentalGroup" instead of "fundamental_group". Including "class" in the definition of search_def and then calling search_def("fundamental", "group") should work.

[Michael Orlitzky]

On 5/8/20 12:59 PM, John H Palmieri wrote:
>
> As a topologist, I would have to say that the best results are the ones
> for simplicial complexes and simplicial sets, actually.
>

Good news: those are also in my top five results. Bad news (?): Google
probably knows enough about you to put them in your preferred order.

> More seriously, I think that search_def should include "class" as well
> as "[c]?def" as parts of the regular expression forming the search. That
> still doesn't find the case you mentioned, because it uses
> "FundamentalGroup" instead of "fundamental_group". Including "class" in
> the definition of search_def and then calling search_def("fundamental",
> "group") should work.

If we insist on doing this with a regular expression, the
signal-to-noise ratio gets worse as we "improve" the regex. Silly things
like

class Foo(...):
Element = SomeElementClass

will trip it up if we insist that 'class' be a part of the definition of
FooElement. And if we don't, then we'll match a million non-definitions.

There are tools that can do this correctly, but then we would have to
admit that some implementation details cannot be hidden behind a cute
interface. In lieu of that, the search engines fake it pretty well.

[Nils Bruin]

On Thursday, April 30, 2020 at 9:20:00 AM UTC-7, Michael Orlitzky wrote:

I feel the same way about functions like search_src() that badly
reimplement grep (even if they still work).

I'm definitely in favour of keeping search_src and search_def. I find it *super* convenient to not have to think how to formulate the right grep query nor what the target of my grep should be. Grep is great, but writing regexp's comes with significant cognitive load, as comes thinking of which directory to point grep at to get desired results. To me, "edit" comes in a similar category: figuring out which file to edit is a multi-step process with the general tools. Having that taken care of is a great help when you're thinking about the mathematics rather than the code primarily. These are all tools that could also be in my personal toolbox, but having them shipped and maintained with the sage distribution is something I really appreciate.

(incidentally, "edit" could use some love -- it's not as good at figuring out the appropriate source file as it was 12 years ago, due to organizational changes)

[Michael Orlitzky]

On 5/8/20 4:12 PM, Nils Bruin wrote:
> On Thursday, April 30, 2020 at 9:20:00 AM UTC-7, Michael Orlitzky wrote:
>
> I feel the same way about functions like search_src() that badly
> reimplement grep (even if they still work).
>
>
> I'm definitely in favour of keeping search_src and search_def. I find it
> *super* convenient to not have to think how to formulate the right grep
> query nor what the target of my grep should be. Grep is great, but
> writing regexp's comes with significant cognitive load,

As another recent message just showed, the pattern used by search_def()
is wrong, and the definition of "regular" means that it can never be
right. If you want to know exactly *how* it's going to get your query
wrong, the implementation details are unavoidable. The fact that python
is an interpreted language and that things can be defined on-the-fly
only makes this attempt more embarrassing.

The following is a lot closer to correct, but you still need some way to
tell sage that the whitespace/underscore is optional; in other words,
unless we build in a search engine, you have to know a bit of regex to
tell it what you mean.

sage: load('searchdefs.py')
sage: search_submodules("fundamental.?group", sage)
sage.all.fundamental_group
sage.all_cmdline.fundamental_group
sage.categories.simplicial_sets.fundamental_group
sage.combinat.cluster_complex.fundamental_group
sage.combinat.root_system.extended_affine_weyl_group.FundamentalGroupOfExtendedAffineWeylGroup
sage.combinat.root_system.extended_affine_weyl_group.fundamental_group
sage.combinat.root_system.extended_affine_weyl_group.to_fundamental_group
sage.combinat.root_system.fundamental_group.FundamentalGroupElement
sage.combinat.root_system.fundamental_group.FundamentalGroupGL
sage.combinat.root_system.fundamental_group.FundamentalGroupGLElement
sage.combinat.root_system.fundamental_group.FundamentalGroupOfExtendedAffineWeylGroup
sage.combinat.root_system.fundamental_group.FundamentalGroupOfExtendedAffineWeylGroup_Class
sage.combinat.subword_complex.fundamental_group
sage.homology.all.fundamental_group
sage.homology.delta_complex.fundamental_group
sage.homology.examples.fundamental_group
sage.homology.homology_morphism.fundamental_group
sage.homology.homology_vector_space_with_basis.fundamental_group
sage.homology.simplicial_complex.fundamental_group
sage.homology.simplicial_complex_morphism.fundamental_group
sage.homology.simplicial_set.fundamental_group
sage.knots.knot.fundamental_group
sage.knots.link.fundamental_group
sage.misc.benchmark.fundamental_group
sage.repl.ipython_kernel.all_jupyter.fundamental_group
sage.sandpiles.sandpile.fundamental_group
sage.schemes.curves.affine_curve.fundamental_group
sage.schemes.curves.constructor.fundamental_group
sage.schemes.curves.projective_curve.fundamental_group
sage.schemes.curves.zariski_vankampen.fundamental_group
sage.schemes.elliptic_curves.ell_field.fundamental_group
sage.schemes.elliptic_curves.ell_finite_field.fundamental_group
sage.schemes.elliptic_curves.ell_number_field.fundamental_group
sage.schemes.elliptic_curves.ell_padic_field.fundamental_group
sage.schemes.elliptic_curves.ell_rational_field.fundamental_group
sage.schemes.elliptic_curves.isogeny_class.fundamental_group
sage.schemes.hyperelliptic_curves.constructor.fundamental_group
sage.schemes.hyperelliptic_curves.hyperelliptic_finite_field.fundamental_group
sage.schemes.hyperelliptic_curves.hyperelliptic_padic_field.fundamental_group
sage.schemes.hyperelliptic_curves.hyperelliptic_rational_field.fundamental_group
sage.schemes.plane_conics.con_finite_field.fundamental_group
sage.schemes.plane_conics.constructor.fundamental_group
sage.schemes.toric.divisor.fundamental_group
sage.tests.benchmark.fundamental_group

There are some duplicates in there (I've wasted enough time on it), but
it matches things that a regex never will. That function is implemented
by the following code, which belongs in a third-party library and not
sage itself because it has nothing to do with mathematics:

$ cat searchdefs.py
import inspect
import pkgutil
import re
import sys

def has_name(obj):
if inspect.isclass(obj): return True
if inspect.ismethod(obj): return True
if inspect.isfunction(obj): return True
if inspect.isgenerator(obj): return True
if inspect.iscoroutine(obj): return True
return False

def search_submodules(s, module, exclude=[]):
found = []
for modinfo in pkgutil.walk_packages(module.__path__,
module.__name__ + '.'):

if isinstance(s,str):
s = re.compile(s, re.IGNORECASE)

importer, module_name, ispkg = modinfo

if ispkg or any(module_name.startswith(e) for e in exclude):
# Don't load packages, or things explicitly excluded.
continue

if module_name not in sys.modules:
loader = importer.find_module(module_name)
with warnings.catch_warnings():
warnings.simplefilter("ignore")
try:
loader.load_module(module_name)
except:
# A lot of things fail to load.
continue

members = [ (k,v) for (k,v)
in inspect.getmembers(sys.modules[module_name],
has_name)
if not k.startswith('_') ]

# Covers Element = FooElement and things like that.
maxdepth = 2
depth = 0
while len(members) > 0 and depth < maxdepth:
submembers = []
for (mname, member) in members:
if s.search(mname) is not None:
result = module_name + "." + mname
if not result in found:
found.append(result)
print(result)
try:
submembers += [ (k,v) for (k,v)
in inspect.getmembers(member,
has_name)
if not k.startswith('_') ]
except:
pass
members = submembers
depth += 1

[John H Palmieri]

On Friday, May 8, 2020 at 2:17:25 PM UTC-7, Michael Orlitzky wrote:

There are some duplicates in there (I've wasted enough time on it), but
it matches things that a regex never will. That function is implemented
by the following code, which belongs in a third-party library and not
sage itself because it has nothing to do with mathematics:

There are already lots of things in the Sage library that have to do with the Sage library rather than mathematics, like "import_statements" (in sage.misc.dev_tools) or sage.misc.superseded or sage.features.* or sage.doctest.*. Maybe they should all be separated into their own libraries, I don't know. In any case, let's not automatically discard new ideas because they have nothing to do with mathematics.

[Matthias Koeppe]

I have created https://trac.sagemath.org/ticket/29670 (Meta-ticket: Review of development tools in Sage vs. mainstream Python tools) to help convert this lively thread to action items.