redesign combinatorial statistics

[Vincent Delecroix]

Hello everyone,

I like what statistics try to do but I do not like the way it works.
Please tell me where I am wrong and where I am right. I can provide
some proof of concepts as soon as most of you agreed on the design.
Please: critics are welcome.

I think that there are two disjoint tasks:
- add semantic to Map and Morphism
- gather the list of meaningful maps

First of all, I think that most of us agree that it would be nice to
have more semantics at the level of maps (especially
injectivity/surjectivity/bijectivity, commutation relations with other
maps, etc). Note that Morphism is already a way of providing semantics
as it tells that the map preserves some structure. It will not hurt me
too much to have an alias satistics for maps from a countable set to
the non-negative integers or combinatorial_maps for maps between two
countable sets but the work has to be done at the level of maps.

Let me notice that right now, combinatoral_map is just an empty class
whose purpose is to tag some methods as being of the good kind. The
only semantic added here is "that method is actually a map".


Secondly, there is the thing at what the current FindStat is good:
gathering the set of meaningful maps from Parent A to Parent B. The
set of maps from/to somebody might be registered somewhere else in the
same way that conversion/coercions are handled. I think that such
thing must be flexible enough to be tuned by users (especially by the
FindStat project). The function to do the job might looks like
{{{
build_map_from_method(domain, range, category=X, method_name='toto',
injective=True, register=True)
}}}
In some places, a decorator might be a way to proceed.

Once that done, we can add some generic stuff at the level of parents like
{{{
sage: Partitions().known_maps_to(Permutations())
sage: Graphs().known_maps_from(Posets())
}}}
and start playing with compositions of such maps as it is done in FindStat.


That being said we do not want to register all computable maps on
startup. So we have to find the good balance between interesting maps
as "the number of descents in a permutation" and uninteresting maps as
"the polynomial map 57*n^2 - 18*n +3 from {-12, 1, 42, 2243} to the
integers".

Vincent

[Nathann Cohen]

That being said we do not want to register all computable maps on
startup. So we have to find the good balance between interesting maps
as "the number of descents in a permutation" and uninteresting maps as
"the polynomial map 57*n^2 - 18*n +3 from {-12, 1, 42, 2243} to the
integers".

If all that worries you is this, then I guess that it isn't a problem !  Why can't we have two versions of the @combinatorial_map (or whatever else) decorator in the same module, and a line that does :

if some_boolean_variable:

    combinatorial_map = real_combinatorial_map

else:

    combinatorial_map = a_decorator_that_does_absolutely_nothing

This way the information is in Sage, but at absolutely no cost for everybody when you actually want to USE Sage. And if you want to do statistics then you run Sage with a special flag that changes the definition of combinatorial_map ?

Nathann

[Viviane Pons]

That being said we do not want to register all computable maps on
startup. So we have to find the good balance between interesting maps
as "the number of descents in a permutation" and uninteresting maps as
"the polynomial map 57*n^2 - 18*n +3 from {-12, 1, 42, 2243} to the
integers".

I don't understand, are you talking about maps or statistics? There are two different things, unless you consider  statistics to be a map from the object to the integers but I'd say we want to keep two problems separated.

Right now, the combinatorial_map decorator offers already some kind of semantic information, the fact it flags the methods is in itself semantics. I agree that there should be more. Right now, you can do:

sage: p = Permutation([1,2,3])
sage: from sage.combinat.combinatorial_map import combinatorial_maps_in_class
sage: combinatorial_maps_in_class(p)
[Combinatorial map: reverse,
 Combinatorial map: complement,
 Combinatorial map: permutation poset,
 Combinatorial map: Robinson-Schensted insertion tableau,
 Combinatorial map: inverse,
 Combinatorial map: Robinson-Schensted recording tableau,
 Combinatorial map: foata_bijection,
 Combinatorial map: descent composition,
 Combinatorial map: Increasing tree,
 Combinatorial map: Binary search tree (left to right),
 Combinatorial map: Robinson-Schensted tableau shape,
 Combinatorial map: to alternating sign matrix,
 Combinatorial map: Simion-Schmidt map]

The way it works: the decorator replaces the tagged methods by instances of combinatorial map class, the functions only checks which methods are indeed instances of this class.

It is partly the information you need even so I'm not sure it's a good way to have it. And also, this function is hidden somewhere and not accessible from the object or the parent itself.

About Nathan's suggestion to have a flag, I don't know if it's such a good idea. The idea is to provide this semantic information to any user of Sage not machines.

Also, there is no need to bring Findstat into the debate. We have our own sage branch for the website. Git has made it quite easy for us to maintain it and do whatever we want in it, the concern here is indeed Sage.

The way I see this, *because* of the FindStat use case, the combinatorial_map decorator has been introduced and many Sage developers have been devoted time into adding semantic information into sage methods. Of course, we could remove this all together, but this would be a great loss for Sage as this semantic information is useful for users. And Here, I speak as a combinatorist person working with these objects every day.

So, our concern should be: how to make this information easily available to users and how could we store this without impacting efficiency of the program.

[Nathann Cohen]

> Git has made it quite easy for us to maintain
> it and do whatever we want in it, the concern here is indeed Sage.

Works for me :-P

> The way I see this, *because* of the FindStat use case, the
> combinatorial_map decorator has been introduced and many Sage developers
> have been devoted time into adding semantic information into sage methods.
> Of course, we could remove this all together, but this would be a great loss
> for Sage as this semantic information is useful for users. And Here, I speak
> as a combinatorist person working with these objects every day.
>
> So, our concern should be: how to make this information easily available to
> users and how could we store this without impacting efficiency of the
> program.

Dead right ! Is there any reason why gathering this semantic information requires this decorator to wrap the function in a combinatorial map ? Can't the information be gathered wherever we need it *without* modifying the actual function ? You could do whatever you need with a decorator that, when applied to a function f with some flags, stores a copy of this function f and works with the flags while returning the original function f, can't you ?

This would make it less intrustive, and actually not intrusive at all !

Nathann

[Viviane Pons]

Dead right ! Is there any reason why gathering this semantic information requires this decorator to wrap the function in a combinatorial map ? Can't the information be gathered wherever we need it *without* modifying the actual function ? You could do whatever you need with a decorator that, when applied to a function f with some flags, stores a copy of this function f and works with the flags while returning the original function f, can't you ?

This would make it less intrustive, and actually not intrusive at all !

 Yes I've thought of that. I'm trying to find the best way of doing it. The decorator seems the best way to flag. Before returning the orignal function, I could store the "map" itself. But I would like to create the map objects only if needed, and this I'm not yet sure how to do.

[Nathann Cohen]

> Yes I've thought of that.

Yet I was the first to publish ! One year ago ... :-P

https://groups.google.com/d/msg/sage-devel/SnPfidRM9j8/4LbWLMBf-2kJ

> I'm trying to find the best way of doing it. The decorator seems the best way to flag. Before returning the orignal function, I could store the "map" itself. But I would like to create the map objects only if needed, and this I'm not yet sure how to do.

I don't get it... What if combinatorial map just copies the list of
all its parameters AND the function into a database stored in some
module ? You have absolutely no other work to do, and you would only
create the map objects when somebody tries to ask anything about the
content of the database, don't you ?

Nathann

[Paul-Olivier Dehaye]

> The way I see this, *because* of the FindStat use case, the
> combinatorial_map decorator has been introduced and many Sage developers
> have been devoted time into adding semantic information into sage methods.
> Of course, we could remove this all together, but this would be a great loss
> for Sage as this semantic information is useful for users. And Here, I speak
> as a combinatorist person working with these objects every day.
>
> So, our concern should be: how to make this information easily available to
> users and how could we store this without impacting efficiency of the
> program.

Dead right ! Is there any reason why gathering this semantic information requires this decorator to wrap the function in a combinatorial map ? Can't the information be gathered wherever we need it *without* modifying the actual function ? You could do whatever you need with a decorator that, when applied to a function f with some flags, stores a copy of this function f and works with the flags while returning the original function f, can't you ?

This would make it less intrustive, and actually not intrusive at all !

Yes, this is a solution that I have repeated to Viviane in Montreal. The objection to that solution (while we were in Edinburgh) was that it would quickly make sense to them to add a bunch of empty classes and functions then.  Since, the transition to git has made part of that workflow better for the findstat people.

Paul

[Paul-Olivier Dehaye]

An even better way would be for them to offer to maintain the database as the findstat project, for them to implement a wrapper around that database in sage, and for something like sage-explorer (developed at Edinburgh) to do all the let's-build-a-web-service part.

There would be good symbiosis with the LMFDB project (*).

This is indeed a good solution as a first step. The only obstacle was, and is still (?) to manage to explain to others what it is that they are trying to do, and why they are trying to do it that way and not some other clever quick and dirty way. This takes time, effort, energy. Much more useful would be to get quick help in making it happen the right way right away.

Paul

(*) Not quite. But the needs of LMFDB and findstat really do align.

Paul-Olivier Dehaye

SNF Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

--
You received this message because you are subscribed to the Google Groups "sage-combinat-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-combinat-d...@googlegroups.com.
To post to this group, send email to sage-comb...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-combinat-devel.
For more options, visit https://groups.google.com/d/optout.

[Paul-Olivier Dehaye]

For anyone involved in findstat, there are several layers of complexity. This is my assessment from outside the findstat collaboration:

 - they also deal with Mathematica code

 - they also deal with user-generated information that is not code

 - they also develop some of the code and would thus 1) prefer to maintain one rather than two separate objects (code and database) 2) like to easily and flexibly be able to develop the code as they intend to use it (i.e. equipped with everything, cf. monkey patching)

 - some of the information they want is added/could easily be added by others in sage code

 - just like the LMFDB, the database term is a misnomer. They want their thing to be data and code. The code is partly their data, either directly (class dependencies) or indirectly (the code is run to produce the data). 

Paul

Paul-Olivier Dehaye

SNF Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

On Wed, May 28, 2014 at 12:13 AM, Simon King <simon...@uni-jena.de> wrote:

Hi Viviane,

On 2014-05-27, Viviane Pons <vivia...@gmail.com> wrote:
> The way it works: the decorator replaces the tagged methods by instances of
> combinatorial map class, the functions only checks which methods are indeed
> instances of this class.

And, if I understand correctly, it is this kind of things that makes
Nathann ask for removal of the decorator.

What you *want* to do is create a database that handles requests of the
type "what combinatorial maps are provided by a class C?", or perhaps even
better "what combinatorial maps are provided by a sub-category S of
Sets().Finite()?". This could very well be a (SQLAlchemy?) database that is
stored somewhere, perhaps even in a remote location.

But what you *do* is to encode this database via the *type* of methods.
The information is not put into an external database, but the
information is stored *implicitly* by changing Sage code.

I agree with Nathann: This is intrusive! And when your decorator
replaces methods by "instances of some class" then certainly the
overhead of calling these methods will increase. Applying your decorator
to a frequently (recursively?) used method may result in a slow-down, as
it seems.

Even when trying to take your perspective, I still don't understand why
you might want the decorator to work in that way.
Certainly a proper database (relying on a dedicated database engine) is
faster and easier to search and thus more useful for you than an ad-hoc
function that ultimately relies on listing attributes and determining
their types.

So, what is the rationale for *not* using a database for your database of
combinatorial maps?

> It is partly the information you need even so I'm not sure it's a good way
> to have it.

I see no reason how this could be a good way to have it.

> And also, this function is hidden somewhere and not accessible
> from the object or the parent itself.

"Hidden somewhere" is bad for a user interface.

> About Nathan's suggestion to have a flag, I don't know if it's such a good
> idea. The idea is to provide this semantic information to any user of Sage
> not machines.

It seems to me that you are argueing *against* the current decorator.
Calling a function that is hidden somewhere and relies on introspection
(listing attributes, type check) is for machines. A user of Sage should
have a database with a proper interface.

> The way I see this, *because* of the FindStat use case, the
> combinatorial_map decorator has been introduced and many Sage developers
> have been devoted time into adding semantic information into sage methods.
> Of course, we could remove this all together, but this would be a great
> loss for Sage as this semantic information is useful for users. And Here, I
> speak as a combinatorist person working with these objects every day.
>
> So, our concern should be: how to make this information easily available to
> users and how could we store this without impacting efficiency of the
> program.

Do not store information by a decorator that changes code. Migrate it to
a proper database, that may be stored in SAGE_SHARE. The decorator
should write information into the database (or perhaps even pull
information from it?) and, when done, simply return the unaltered
method. In that way, it would not be intrusive, and the database would
provide a not-so-hidden single point of entry.

Best regards,
Simon

[Viviane Pons]

Let's not mix everything here. In FindStat, we do use a database for statistics (not maps) but that's not what we're talking about here. We're just talking about flagging some object methods as maps with some semantic information.

If we were to use a sage database for those maps (I've never used one, I don't know how it works), how we would do? Are we talking about something persistent here? I was more thinking of just storing some data  when the object is created, which means where the decorator is called.

What I have in mind (and I think what Nathan had in mind) is

def map_decorator(f, args):

    # storing the info

    return f

my only question is what "storing the info" means. I understand Nathan is suggestiong I just put it in some database object somewhere, and then my permutation object would just ask this object whenever it needs to give the list of maps. Am I right here?

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.

[Paul-Olivier Dehaye]

I agree with you, but there are (at least) two questions left hanging:

1) is the database transient? i.e. repopulated once as the code is run?

2) nathann is arguing on the feature-length trac ticket that we are writing together that empty methods have no place in sage if they only exist for the decorator:

http://trac.sagemath.org/ticket/16403

The same arguments were made (by him and others) in Edinburgh, and indeed miss the point of ABCs (made in the trac ticket) and the category framework (not made in the trac ticket, because I only want to kick one can of worms at a time)

Paul

Paul-Olivier Dehaye

SNF Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

On Wed, May 28, 2014 at 12:51 AM, Simon King <simon...@uni-jena.de> wrote:

Hi Paul,

On 2014-05-27, Paul-Olivier Dehaye <paul-oliv...@math.uzh.ch> wrote:

> For anyone involved in findstat, there are several layers of complexity.
> This is my assessment from outside the findstat collaboration:
>  - they also deal with Mathematica code
>  - they also deal with user-generated information that is not code

You just provided two arguments for having an external database :)

>  - they also develop some of the code and would thus 1) prefer to maintain
> one rather than two separate objects (code and database)

That's no valid argument. Currently, they simply put a decorator in front of a
method. In future, they'll simply put a decorator in front of a method.

The important difference is that now the decorator is replacing a proper
method by an instance of some class, which is an intrusive way to encode
information that is supposed to be retrieved by a function that is hidden
somewhere. But in future the decorator will be writing to some database with
a standard database engine, without replacing the method by something else.

> 2) like to easily
> and flexibly be able to develop the code as they intend to use it (i.e.
> equipped with everything, cf. monkey patching)

Again, this seems unrelated to the question of what the decorator should
do.

>  - some of the information they want is added/could easily be added by
> others in sage code

Again, adding information is now and shall in future be possible by putting
a decorator in front of methods.

>  - just like the LMFDB, the database term is a misnomer. They want their
> thing to be data and code.

So, why don't they focus on their data and code? Why do they spend their time
with an ad-hoc implementation of a mock-database in an intrusive way? Do
not re-invent the wheel, and try to use the right tool for a job!

Let me try to re-formulate it:
- There should be a database, locally stored in SAGE_SHARE, but perhaps
  even able to talk with some remote database. This database would also
  be able to deal with external information, from Mathematica or other
  sources.
- The @combinatorial_map decorator will be one (the easiest?) way to
  register stuff in the database.
- The user should in future have additional possibilities to register stuff and
  do queries, namely by a proper database interface.

[Nathann Cohen]

> Let's not mix everything here. In FindStat, we do use a database for
> statistics (not maps) but that's not what we're talking about here. We're
> just talking about flagging some object methods as maps with some semantic
> information.

Indeed. You agreed to only talk about the aspects of this decorator which are of interest to Sage.

> my only question is what "storing the info" means. I understand Nathan is
> suggestiong I just put it in some database object somewhere, and then my
> permutation object would just ask this object whenever it needs to give the
> list of maps. Am I right here?

Let us say that when the combinatorial_map decorator is called on a function f with some parameters, it does the following

from sage.combinat.combinatorial_map import my_dictionary_database

my_dictionary_database[f] = list_of_parameters_or_anything_you_like

return f

With a design like that, any semantic information is still available if you ever need it, and the function is left unchanged.

Nathann

[Paul-Olivier Dehaye]

If this is agreeable to the findstat project, how would this fare if tomorrow me or my students:

 - start adding semantic decorators to existing classes (for instance, for groups, adding information in the groupprops database, or from categories adding information from the database of categories at the nLab)

 - start adding essentially empty classes and functions in sage that are there only so I can add the empty decorators (for instance, LMFDB)

I understand I could do this as a separate git branch, but the point is that this is meant to build on other people adding decorators, because it is easy and convenient to them and they want to help my research project, while at the same time "hey-you-never-know-how-those-semantic-annotations-might-prove-useful=so=why-not-have-it-in-sage-even-if-it-supports-just-one-guy"

I am looking for a consistent workflow to support my work and that of my students.

Paul

Paul-Olivier Dehaye

SNF Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

--

[mmarco]

A slightly unrelated problem:

Would it be possible to include the findstat functionality in sage? That is, i have a bunch of pairs (permutation, number), then call the function findtsta on them, and get a list of possible ways that the numbers can me computed from the permutations. All inside sage, without need to go to the findstat webpage.

[Nathann Cohen]

> Would it be possible to include the findstat functionality in sage?

It is possible to write the code, and desirable. As the Sage is open-source, you are also welcome to give it a try if you need the feature. Sage already queries the OEIS, though for FindStat you will probably have to design some kind of protocol to transfer the combinatorial objects with the website's owners. I am told that you can contact them at in...@findstat.org.

Nathann

[Christian Stump]

I really appreciate that so many people appear to be interested in the
FindStat functionality, being it from within Sage or from the FindStat
webpage!

--> It is almost sure that it will *never* be possible to use FindStat
without a web connection

--> Nevertheless, it is on our ToDo list (which is, as for basically
everyone, very packed -- I don't promise anything for a given time in
the future) to provide a text based query to the database to obtain
machine readable results that can be parsed into Sage. That should go
similarly to the Sage functionality for the OEIS. BUT please see that
querying FindStat is *much more complicated* than the simple query for
the OEIS: (A) the data structure is more complicated, (B) the
"following maps" is much more complicated.

--> It will not be possible (say within the next two years, except the
unlikely event that I will a lottery without participating, and then
spend some money to hire a professional programmer) to only query the
statistics data, and do the computations within your own Sage. That's
because there is (A) in additional layer of code on top of Sage that
makes that possible and (B) this layer is extra thick because we also
have stuff there that could be part of Sage but is not because there
is a *partial unwelcoming attitute* that prefers to give a "that's not
good for Sage" over "include non-perfect code, even if it would have a
strictly positive influence on the Sage user community" in such
instances.

Cheers,

Christian

[Paul-Olivier Dehaye]

Hi Mmarco,

This is a functionality many of us would want too. The question that is unclear is how to implement it in the best way. What Nathann refers to as "some kind of protocol to transfer the combinatorial objects" is a very hard problem. Fortunately, many people are starting to realise that this is a desired functionality and constructively try to find solutions. Depending on your background, you might be able to directly help, for instance by annotating some of the functions in the sage code (a very easy thing to do, for which you need only a web browser. This is a tiny step above editing documentation). 

People are actively working on this problem

Paul

Paul-Olivier Dehaye

SNF Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

--

[Paul-Olivier Dehaye]

--> It will not be possible (say within the next two years, except the
unlikely event that I will a lottery without participating, and then
spend some money to hire a professional programmer) to only query the
statistics data, and do the computations within your own Sage.

Just to make this statement more precise: it will not be possible within the next two years if Mmarco relies on your availability for this (there is no blame in that statement, it's just a clarification that others might be willing to work on this problem).

Paul

[Christian Stump]

> Just to make this statement more precise: it will not be possible within the next two years if Mmarco relies on your availability for this (there is no blame in that statement, it's just a clarification that others might be willing to work on this problem).

It is not a problem -- it's the way we set up our project! Its main
features are designed for users who do not want to do any coding or
even seeing a terminal environment. And that's good about it and one
of the reasons why people like it! I recently talked about the project
with Michelle Wachs -- she was actually asking about it! -- and she
seemed to be really impressed. I think that didn't just come from the
bare functionality, but from the easy possibility for her to interact
with it.

Anyone who is interested in working on more sophisticated features to
be done from within Sage are very welcome to talk to me and/or to
Chris Berg (though he might be a little less responsive these days)
about it to see how we can approach it. But I would not like if I find
out in 18 months from now that someone implemented a much stronger
query machine (which is certainly possible) based on our ideas, but
without collaborating with us!

Cheers, Christian

[Vincent Delecroix]

It seems that actually nobody read my initial post on that thread...
so let me repeat

There are two disjoint tasks:
- add semantic to Map and Morphism (Sage)
- gather the list of meaningful maps (Sage/FindStat)

First of all, most of us agree that we need more semantic at the level
of maps. Note that Morphism is already a way of providing semantics as

it tells that the map preserves some structure. It will not hurt me

too much to have aliases satistics and combinatorial_map for whatever
they are. But the semantic has to be implemented at the level of maps
not at the level of methods.

As Vivianne agreed: combinatoral_map is currently an empty class whose

purpose is to tag some methods as being of the good kind. The only

semantic added is "that method is actually a map".


Secondly, there is the thing at what FindStat is good: gathering the
set of meaningful maps from Parent A to Parent B. I will not develop
on the technical part underlying this question as it was already
discussed. But it would be good to have the infrastructure and some of
the information in Sage. In other words, a function which register
maps
{{{
sage: my_map = build_map_from_method(domain, range, category=X,
method_name='toto', injective=True)
sage: register_map(my_map)
}}}
In some places, a decorator might be the way to proceed.

Vincent

[Christian Stump]

> It seems that actually nobody read my initial post on that thread... so let me repeat

I did -- but didn't really have any qualified contribution...

> But the semantic has to be implemented at the level of maps not at the level of methods.

could you explain what you mean there (maybe using your example of the
number of descents of a permutation).

[Vincent Delecroix]

Hi Christian,

2014-05-28 11:32 UTC+02:00, Christian Stump <christi...@gmail.com>:

A method ( = a Python function) is not a Sage Map ( = a Python object
that model a mathematical function). I would like first to convert the
method into a map
{{{
from sage.categories.map import Map
class NumberOfDescents(Map):
def __init__(self):
Map.__init__(Permutations(), NonNegativeIntegers())

def _call_(self, p):
return Integer(p.number_of_descents())
}}}
The above example is already non-trivial since it specifies a domain
and a codomain (which is different from the parent of the image of an
element):
{{{
sage: nod = NumberOfDescents()
sage: p = Permutation([3,2,1])
sage: nod(p)
3
sage: nod(p).parent() is NN
False
}}}
The second step would be to think how to add semantic to the map. Part
of is already managed by the axioms/categories (for example a Morphism
between GradedSets preserve the grading). But there is nothing for
injectivity/surjectivity or more subtle properties.

It would make more sense to register actual maps as above.

Vincent

[Paul-Olivier Dehaye]

Vincent, I did read your email, which is very good. Thank you for posting. Unfortunately it is too focused for my taste. I would like to solve the generic problem of how to put in semantic information in sage first. Many might say let's solve one problem first, but in my mind it is already three times that this problem is appearing: 

 - LMFDB

 - findstat

 - category framework

That's why people who were at Edinburgh were invited. 

I am looking for a generic solution, and to help with this I have started writing a document that is read only here:

https://etherpad.mozilla.org/ep/pad/view/ro.Gzkl7YR-2gG/latest

I am not writing this document at this exact moment, because I have a pressing deadline until Sunday. I am willing to devote much more time for this then. This is not an empty comment, in some ways my pressing deadline of Sunday is because I am already working on this. My timing is slightly affected, because in the back of my head I knew I just had to wait for the next flare up of this discussion on the list to get another discussion started, but this time backed up with concrete elements.

Anyone who would like to help me add constructive material to the document is welcome to do so, just ask me for the link.

@Christian: a generic, better, query engine might be build outside of findstat if the findstat group is too focused on their particular use case. Believe me, findstat is as far as I can tell baby case compared to LMFDB. But it has the merits of trying a more viable approach in my mind. 

I certainly don't want to develop anything without explicitly telling findstat about it.

Paul

Paul-Olivier Dehaye

SNF Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

[Christian Stump]

> The second step would be to think how to add semantic to the map. Part
> of is already managed by the axioms/categories (for example a Morphism
> between GradedSets preserve the grading). But there is nothing for
> injectivity/surjectivity or more subtle properties.

Thanks for the clarification! I have a few more questions there:

--> Can I turn any method in Sage which mimics a mathematical function
into a map in the above sense? To put it differently: Is it right that
there are always (or almost always) parents available to use as domain
and codomain?

--> How do you think "more semantic" should be implemented? As a dict
with some conventions for keys and values?

--> What if I want to give properties that depend on input parameter
like the length of the permutation? Say I look at the map from
Permutations(n) to itself given by composition with the long cycle
(1..n), and now want to add the semantic information that it has order
n?

--> Is there a way to implement your ideas such that the lines of code
I need to touch in order to turn a method into a map is right where
the method is implemented?

Thanks, Christian

[Viviane Pons]

Vincent I had read your post but I hadn't understood that you wanted something like this. This Map class seems a really good idea to me, it looks like our combinatorial map class that we use to wrap the methods but improved.

After this discussion, the way I would see things would be:

* having a permanent database into sage containing the maps

* calling the database from the object to get create the map objects and access the information

* using the decorator to add those maps in the database

A few questions:

- how do I use the decorator to add the maps in the database, won't it cost time to check the database when I create the object?

- Where should we put the "combinatorial_maps" method to actually get the list of maps on a given object (or parent)

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.

[Nicolas M. Thiery]

Hi!

For whatever it's worth: in MuPAD, it was possible to add type
annotations on a method; typically to specify the domain and codomain:

abs := proc(x: Dom::Integer): Dom::Integer
begin
return(x)
end

We were using this extensively for:

- Dynamic type checking in debug mode
- Retrieving the domain or codomain, typically when constructing new
functions from existing ones (e.g. in module_morphism)
- ...

Python 3 will allow to add similar annotations. In the mean time we
could possibly stuff such annotations to glorify methods:

sage: def f(x): return abs(x)
sage: f.domain=ConstantFunction(ZZ)
sage: f.codomain=ConstantFunction(NN)
sage: f(-3)
3
sage: f.domain()
Integer Ring
sage: f.codomain()
Non negative integer semiring

Anyway, just food for thoughts; it's not like there is a clear plan in
this direction.

Cheers,
Nicolas
--
Nicolas M. Thiéry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/

[Christian Stump]

> Ceterum censeo: Please do not put too much semantic into an attribute of
> the map. It will be a burden for the map, will cost memory, and will be
> difficult to search.

I would personally really like to get semantic information about maps
implemented in Sage. This starts with automated concatenating maps,
and does not even end with information about a map in the Journal of
Applied Nonsense. I am sure you have reasons to vote for a centralized
system where to store such information. But how am I then convincing
someone knowing that information to add that information there? What I
like about a local system (preferably accompanying the code of the
very method) is that it makes it easy for the person having
information about a specific map to provide that information.

Of course, that information must be organized to be searchable
properly. But I don't see a problem there beside conventional namings.

[Paul-Olivier Dehaye]

Christian's point is key. The absolute hardest problem to solve is where to anchor semantic information so that others can contribute to it easily, because his project requires lots of microcontributions (cf. Nielsen's Reinventing Discovery) 

Right now, by lack of a better place, it is put in sage. According to me, this is a sensible short term solution, step 1 if you want. Step 2 is that the LMFDB starts to do the same thing putting semantic information on empty classes and methods. Step 3 is that we find a better way, armed with two use cases. I don't think we are ready to go to step 3 yet.

Paul

Paul-Olivier Dehaye

SNF Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

--
You received this message because you are subscribed to the Google Groups "sage-combinat-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-combinat-d...@googlegroups.com.
To post to this group, send email to sage-comb...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-combinat-devel.

[Paul-Olivier Dehaye]

Two comments:

 - Nathann is arguing that no empty methods/classes should be present. Which means to me that he doesn t see the value of semantic information in itself

 - They are not building a database. They are putting tags in the code that allow them at runtime to build something that amounts to a database. It's not that different to the category framework in some way, which is encoding a lot of data via code constructions (partially redundant work: cf http://nforum.mathforge.org/discussion/2490/what-would-you-want-from-a-higher-categorical-database/#Item_0 ). In any case, the way findstat does things, they have the potential to construct a database. The point is that they allow me to reuse their work to easily build my own database on top, that does something different. 

I am not arguing that what findstat does with a decorator is the best. A flagged decorator that does nothing most of the time is better than what they do. But a full-on database is, as a short term solution, heading in the wrong direction.

Paul

Paul-Olivier Dehaye

SNF Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

On Wed, May 28, 2014 at 5:23 PM, Simon King <simon...@uni-jena.de> wrote:

Hi Christian,

On 2014-05-28, Christian Stump <christi...@gmail.com> wrote:

> But how am I then convincing
> someone knowing that information to add that information there?

Ask him/her to put @combinatorial_map in front of the method in
question. In other words, there should be no need to change the interface
for those who contribute combinatorial maps. What should change is how this
contribution is processed. It should not be the case that, as a result,
the overhead of calling a method increases!

What I (an Nathann?) am trying to convince y'all: @combinatorial_map
should not wrap a method into some class instance that serves as a proxy
to the method and holds additional information purely locally. Instead,
@combinatorial_map should communicate with a (local) database before
returning the method *unwrapped*. And then, there should additionally
be tools to use the local database together with a global one.

> What I
> like about a local system (preferably accompanying the code of the
> very method) is that it makes it easy for the person having
> information about a specific map to provide that information.

To give you a drastic picture: Imagine you work for a company and want
to keep track who is your customer. You suggest to attach a little tag
to each customer, stating that (s)he is "happy customer of FooBar ltd.". It
will then be very easy to test if a given person is your customer: Just
look for the tag. And when you want to have a list of your customers in
a given town: Just do an exhaustive search in that twon and check each
person's tag!

But I somehow feel that having a database keeping track of your
customers would be better. And I do believe that tagging a method as
"combinatorial map" locally is the same as tagging your customers
locally. So, the picture applies.

> Of course, that information must be organized to be searchable
> properly. But I don't see a problem there beside conventional namings.

Hang on. Do you suggest to implicitly create a database that relies on
naming schemes??

I think one can not deny: What y'all do with the @combinatorial_map *is*
in fact creating a database---at least implicitly---, whether you want
it or not. But explicitly is better than implicitly, and one should not
re-invent the wheel. So...

Best regards,
Simon

[Paul-Olivier Dehaye]

I agree 100% with your summary, Simon, except that 

1) Nathann will have to say himself whether he agrees or not;

2) A decision should be made whether it is valid to introduce lots of empty methods and classes simply for the semantic decorators that will be added.

Paul

Paul-Olivier Dehaye

SNF Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

On Wed, May 28, 2014 at 5:32 PM, Simon King <simon...@uni-jena.de> wrote:

Hi Paul,

On 2014-05-28, Paul-Olivier Dehaye <paul-oliv...@math.uzh.ch> wrote:
> Christian's point is key. The absolute hardest problem to solve is where to
> anchor semantic information so that others can contribute to it easily,

> ...

> Right now, by lack of a better place, it is put in sage.

I think you miss the point of what Nathann and I try to say.

I think Nathann agrees that it is a reasonable idea to have a @combinatorial_map
decorator in front of a method (I do agree), provided that it gives useful
information to some people (this is the case now), and provided that it
does not have a negative impact for people who don't use this
information---and currently it *does* have a negative impact, because of
a slow-down in potentially time-critical methods.

Last attempt before leaving office:
- Currently we have a lose-lose situation: When people put
  @combinatorial_map, then it has a negative impact to some other
  people (the slow-down), and it is difficult to use the added information
  for those who want to do proper queries.
- We should better have a win-win situation: When people put
  @combinatorial_map, then it should not change the method (to not
  slow-down stuff), and the added information should be accessible to
  queries in a reasonable way.

Cheers,
Simon

[Vincent Delecroix]

Hi Paul,

2014-05-28 17:42 UTC+02:00, Paul-Olivier Dehaye
<paul-oliv...@math.uzh.ch>:

> I agree 100% with your summary, Simon, except that
> 1) Nathann will have to say himself whether he agrees or not;

Please at some point stop the flame.

> 2) A decision should be made whether it is valid to introduce lots of empty
> methods and classes simply for the semantic decorators that will be added.

It is not. Could you give an example of what you are thinking about ?
In my mind it would be seomthing like the natural injection ZZ ->
QQ... but this is already there
{{{
sage: QQ.convert_map_from(ZZ)
Natural morphism:
From: Integer Ring
To: Rational Field
}}}
So there is no need to create an empty method .as_rational() on each
integer. If you want a trivial function in your database then
implement it as a Morphism. Or better as Simon said: put in the
database what is needed to actually build the Morphism.

Vincent

[Nathann Cohen]

Yo !

> I agree 100% with your summary, Simon, except that
> 1) Nathann will have to say himself whether he agrees or not;

I think that the first time I said it was on the 20/6/2013 (one year ago):

https://groups.google.com/d/msg/sage-devel/SnPfidRM9j8/u7f4X-cLlrsJ
Quote : "Well, just returning the function as it is while registering the decorator's information somewhere seems to be nice. The decorator could even add some information to the docstring so that this information would automatically appear in the doc !"

Then, I repeated it (same day one year ago)
https://groups.google.com/d/msg/sage-devel/SnPfidRM9j8/4LbWLMBf-2kJ

I added yesterday a link toward the link above :
https://groups.google.com/d/msg/sage-devel/QRUXmy6UZVo/wiEpg-eCnYAJ

Which was right after mentionning this idea again :
https://groups.google.com/d/msg/sage-devel/QRUXmy6UZVo/BJ0WMYjmKJ0J

I mean... 

I don't know....

Yeah... I am pretty sure that I did my job and that I mentionned the idea..... What more can I do ?

Nathann

[Viviane Pons]

I think we all agree on this idea now (that the decorator should not wrap the method), so there is no point in keeping the argument.

Still, I have asked a few questions on which I'd like you guys opinions:

- what's the best way to store this information? Nathan and Simon mentioned database, would this be a run time database? Or something persistent?

- Where to put the method to get the maps, right now it is a function hidden in some folder. I would like people to be able to see it when they're on the object, but it seems silly to add it manually to the object and yet, it shouldn't be on every object...

Cheers

Viviane

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.

[Vincent Delecroix]

Hi Vivianne,

2014-05-28 19:32 UTC+02:00, Viviane Pons <vivia...@gmail.com>:

> I think we all agree on this idea now (that the decorator should not wrap
> the method), so there is no point in keeping the argument.
>
> Still, I have asked a few questions on which I'd like you guys opinions:
>
> - what's the best way to store this information? Nathan and Simon mentioned
> database, would this be a run time database? Or something persistent?

I am not sure a real database (e.g. sql or similar) would be the best.
The real question is how many maps you would like to handle ? If the
answer is below 50000 I would go for a pure Python datastructure
(dict, set and so).

> - Where to put the method to get the maps, right now it is a function
> hidden in some folder. I would like people to be able to see it when
> they're on the object, but it seems silly to add it manually to the object
> and yet, it shouldn't be on every object...

For me, it would makes sense on parents directly
{{{
sage: Permutations().known_maps_to(Partitions())
[...]
}}}
And in the methods known_maps_to and known_maps_from there should be
pointer on how to feed the database with more maps. But I would charge
the elements with that.

Vincent

[Nathann Cohen]

Hello !!!

> Is he? Sorry, then this is a point where Nathann and I do not agree.

I don't even know what I think anymore. With the turn that events take, I think I don't even need to think anymore. I think that I will think what you think I think, it is much easier for me.

> find abstract methods useful. I thought that his main point was: Don't
> be intrusive! Do not make code of some people slower just because other
> people care about the mathematical properties of some methods. And from
> my perspective, the discussion is (or should be) only about the
> question: How to care about the mathematical properties without slowing
> down other peoples code.

Yep. I also defended that "respecting other people's code" also meant "not adding method to the classes they use if the method is not useful for them". If it is only for internal stuff, a hidden (beginning with a '_' works for everybody...). But that was a long time ago and I gave up on #16403.

> Do I understand correctly: You would not like that the
> @combinatorial_map decorator constructs a database at startup time? I
> think that's a valid concern.
>
> We could try to be clever and introduce an environment variable (or
> commandline argument) such that @combinatorial_map creates a database
> when this variable is set, and otherwise does nothing at all.
>
> Hence, people who care about the implicit would-be-database encoded in
> the flags could obtain them by explicit request, and all other people
> would not be affected at all.
>
> Would this be a short-term solution? Also from Nathann's perspective?

Nathann's perspective (I just asked him) was given in this earlier message about implementing ..... a global variable to enable/disable this feature.
https://groups.google.com/d/msg/sage-devel/QRUXmy6UZVo/-A5k4K4A9QMJ

But it was answered immediately afterward :

"About Nathan's suggestion to have a flag, I don't know if it's such a good idea. The idea is to provide this semantic information to any user of Sage not machines."

Truth is, the decorator I have in mind only copies the function and the arguments to some global "list" (that's all) and I am not scared of this cost, given that it is only paid when the class is loaded.

Then, there can be a longer computation occurring the first time the database is queried, and this hardly matters for it is only paid for those who actually want to use the db information.

Nathann

[Paul-Olivier Dehaye]

Hi Vincent

The wikipedia definition of flaming has the words "hostile" and "insulting". I have certainly been hostile towards Nathann, and have explicitly explained to him early on why he was wrong and why I was being hostile. It is part of academic discourse to have heated discussions, because we are always exploring new ideas. I was hoping this would have put some sense into him and made sure to hold my ground, rather than letting the issue drop as I have done in previous discussions. His reactions have only confirmed that I was on the right track to deal with him: he has never asked for references, doesn't seem to have looked at those I pointed out, and eventually insulted me. This is completely unprofessional behaviour. I even tried to offer to discuss this in private to prevent him further harming himself. So in summary, I have been hostile and he was both hostile and insulting.

If you feel that I have been insulting somewhere, PLEASE point me to where, I will look and immediately apologise if I agree with you.

I will concede that I was waiting for the right moment to jump in on the mailing list, and that this was planned. Maybe some might be shocked at this, but again it fits in a larger context that spans at least a couple years. Again, that's a fair tactic for academics.

This is concerning my attitude towards Nathann. Towards you, I very much hope I wasn't insulting or hostile (but it is hard to pin down Nathann's arguments, it is a moving target, so you might have felt associated to some of the hostility). Your attitude has only been constructive, and in fact early on I suggested we keep the positivity to your thread. Let me blanket-apologise if I have insulted or been hostile to you. That was absolutely not my intent, quite the opposite. 

Of course, it is unpleasant for everyone, I am also sorry for that. But sometimes people need to be put in their place. 

Now back to the "content". 

I am asking Nathann's opinion because I don't want to go through the same **** again and again. 

And, reading your 2) opinion, you disagree with Simon (but there I am reading in an email he sent after yours).

Paul

Paul-Olivier Dehaye

SNF Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

[Paul-Olivier Dehaye]

@Nathann, Simon:

There is certainly a lot of misunderstandings between all of us. Both ways. The discussion was started because of a suggestion, turned into a ticket written by Nathann. Currently that ticket's description is in complete contradiction to what you seem to describe on the mailing list (which seems more sensible to me). In particular the title of the ticket is "Rename" while the last line says "remove". 

As long as this is the case, Nathann's arguments are incoherent and contradictory, and any conversation on the list will turn in circles.

My summary assessment so far is that Nathann is wrong, the implementation by findstat of their ideas is poor, and the burden is on all of us to improve it, particularly those who care the most. That's the way sage development works, as far as I understand it. Maybe I will be swayed by good arguments from Nathann, who has opened the ticket. I am genuinely keeping an open mind about implementation. I want him to improve what's there, not remove, if he cares enough.

Paul

Paul-Olivier Dehaye

SNF Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

--

[Paul-Olivier Dehaye]

>  - Nathann is arguing that no empty methods/classes should be present.

I think he is, and add Vincent too (cf. previous email of his).

Absolutely noone wants unefficient code. If there is a clear way to improve, let's do it. But burden is on all of us.

To be clear, I want to have this semantic information in the code. 

There are many reasons for this, here is one more: it provides an intermediate step for a person joining sage between a typo fix and implementing their first function, welcoming microcontributions. I think decorators would be very suitable for this, but am open to alternate suggestions.

> Would this be a short-term solution? 

Sure!

Paul

Paul-Olivier Dehaye

SNF Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

On Wed, May 28, 2014 at 8:47 PM, Simon King <simon...@uni-jena.de> wrote:

Hi Paul,

On 2014-05-28, Paul-Olivier Dehaye <paul-oliv...@math.uzh.ch> wrote:

> Two comments:
>  - Nathann is arguing that no empty methods/classes should be present.

Is he? Sorry, then this is a point where Nathann and I do not agree. I

find abstract methods useful. I thought that his main point was: Don't
be intrusive! Do not make code of some people slower just because other
people care about the mathematical properties of some methods. And from
my perspective, the discussion is (or should be) only about the
question: How to care about the mathematical properties without slowing
down other peoples code.

>  - They are not building a database. They are putting tags in the code that
> allow them at runtime to build something that amounts to a database. It's
> not that different to the category framework in some way, which is encoding
> a lot of data via code constructions

There was in fact some discussion at #10963 about the question whether it is
a good idea that the new category-with-axiom framework encodes a fully-grown
would-be-database (by this, I mean something that amounts to a database) by
purely local data and naming conventions.

I'd say: If they are putting tags in the code then they *do* create a
(would-be-)database, whether they want it or not.

> I am not arguing that what findstat does with a decorator is the best. A
> flagged decorator that does nothing most of the time is better than what
> they do. But a full-on database is, as a short term solution, heading in
> the wrong direction.

Do I understand correctly: You would not like that the
@combinatorial_map decorator constructs a database at startup time? I
think that's a valid concern.

We could try to be clever and introduce an environment variable (or
commandline argument) such that @combinatorial_map creates a database
when this variable is set, and otherwise does nothing at all.

Hence, people who care about the implicit would-be-database encoded in
the flags could obtain them by explicit request, and all other people
would not be affected at all.

Would this be a short-term solution? Also from Nathann's perspective?

Cheers,

[Nils Bruin]

On Wednesday, May 28, 2014 10:51:10 AM UTC-7, vdelecroix wrote:

For me, it would makes sense on parents directly
{{{
sage: Permutations().known_maps_to(Partitions())
[...]
}}}
And in the methods known_maps_to and known_maps_from there should be
pointer on how to feed the database with more maps. But I would charge
the elements with that.

 

You'd want to be very careful with putting cross references between (global) objects and/or storing objects in global dictionaries. Such links nearly always have memory footprint and object lifetime implications. It's terribly easy to make decisions that seem reasonable for your own usage, but lead to ever-increasing memory use (aka "memory leak") for another, entirely reasonable usage scenario. A recent example is http://trac.sagemath.org/ticket/15801, where it was found that a relatively natural thing to do: realizing, say, algebras over finite fields as living in categories "algebras over F_q" turns out to have very bad consequences: People can easily construct algebras over many different finite fields, and categories remain in memory forever (they were designed with the idea their number is uniformly bounded in time--which turns out to be a false assumption if you allow categories parametrized by base)

It may well be that for your database purposes you DO want to register things globally somewhere, but almost certainly you cannot do so by default.

[Paul-Olivier Dehaye]

At some point we will need a summary, or to use something better than mailing lists to discuss this.

Paul

Paul-Olivier Dehaye

SNF Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

[Nathann Cohen]

Yo !

> Currently that ticket's description is in complete
> contradiction to what you seem to describe on the mailing list (which seems
> more sensible to me). In particular the title of the ticket is "Rename"
> while the last line says "remove". 

Not my doing. When I posted my last comment on this ticket I set it to "positive_review" and "wontfix" because I gave up. I cannot do anything against a reviewer like you.

Afterwards Ralf changed some stuff, then you set the ticket to needs_review, then it was set to needs_work by somebody else. Honestly it is not my ticket anymore, do what you want with it.

About the difference you see between my ticket, which said "remove the function" and what I said here in my previous post :

I had not noticed before that having a hidden function was actually a nice middle-ground. You can create functions "for semantics only" if it is useful, but this is an "internal mechanism" of Sage. If you need a function which transforms a Graph into a partition of integer to this end it is good, and you can name it Graph._to_partition if you like, but Graph.to_partition means that the function is unclear and not something users should see. And even if a better name is to be found, I would still claim that this function is really not useful to users (though it may be useful for the databases you want to build), and as such has nothing to do in the (already quite crowded) list of graph functions.

So, yeah. Basically, I had not noticed that having a hidden function with a decorator on it (named to_partition if needed) was a good middle ground.

> As long as this is the case, Nathann's arguments are incoherent and
> contradictory, and any conversation on the list will turn in circles.
>
> My summary assessment so far is that Nathann is wrong, the implementation by
> findstat of their ideas is poor, and the burden is on all of us to improve
> it, particularly those who care the most. That's the way sage development
> works, as far as I understand it. Maybe I will be swayed by good arguments
> from Nathann, who has opened the ticket. I am genuinely keeping an open mind
> about implementation. I want him to improve what's there, not remove, if he
> cares enough.

Whatever.

Nathann

[Paul-Olivier Dehaye]

I think an argument that this function is badly named is very sensible. Maybe a better name would be "to_component_sizes". Why not just try changing the name, and see how much work that is for the findstat group to adapt. It might be that this does not want to be named, after all, and that simply an unnamed method needs to be registered. 

Do you feel that you own the class "Graph"? And get to decide what is interesting and what is not relating to Graphs? If so, then indeed there is a problem. What bothers you particularly? To see those methods in the code? To see it appear when you press <Tab>? Clarifying that part would help making everyone happy.

 

Paul

--

[Travis Scrimshaw]

Okay here's where I think we are, and my 2 cents in the matter.

Simon wants to have a database which stores the map information which is constructed by the decorator @combinatorial_map in order to not have it cause any slowdowns. The question is, "how to do this?"

Here's my 2 cents of a solution. The first time the object is called when we want to get the map info, have it iterate over all it's methods and call any method wrapped in @combinatorial_map. The decorator then does:

def combinatorial_map(f):

    # Check if the object's class and the method's name is in the database;

    #   this might not be the right python

    if (f.self.__class__, f.__name__) in DATABASE:

        DATABASE.add(...)

    return f

This has the advantage of being dynamic, so it shouldn't hamper startup time, and only occurs the first time the object is created, so repeated uses won't be slow. We could also go one small step further and have an (cython) ABC which does this behavior on initialization.

Best,

Travis

[Vincent Delecroix]

Hi Travis,

I agree with you and I think it was already suggested by Simon that
the maps should not be built on startup. On the other hand, the maps
that we want to register might not all come from a method.

I created ticket #16408 and drafted a working implementation in
u/vdelecroix/experimental-combinatorial_map_database. I will now
gather the ideas from this thread into the ticket description.

Best
Vincent

[mmarco]

Just to clarify, i wasn't thinking on sage sending a query to findstat, but performing the search locally (i.e. having findstat software included in Sage).

I don't know how findstat is implemented, so i don't know how wise that aproach would be. Does findstat relly on a big database that is enhanced with the new queries? Or does it compute every query from scratch?

El miércoles, 28 de mayo de 2014 10:01:07 UTC+2, Nathann Cohen escribió:

> Would it be possible to include the findstat functionality in sage?

It is possible to write the code, and desirable. As the Sage is open-source, you are also welcome to give it a try if you need the feature. Sage already queries the OEIS, though for FindStat you will probably have to design some kind of protocol to transfer the combinatorial objects with the website's owners. I am told that you can contact them at in...@findstat.org.

Nathann

[Nicolas M. Thiery]

Action time, yeah!

But gosh, it's been one year that there is a rather clear consensus
that combinatorial_map should be a noop by default, with nothing
happening. And now we managed to step on each other feet by starting
to work on it right at the same time!

Oh well :-)

Here is the description of http://trac.sagemath.org/ticket/16410,
which potentially needs review:

------------------------------------------------------------------------------
From the discussions of (TODO: add refs to the threads in sage-devel), there is a consensus that, at this point, the combinatorial_map decorator should by default return the decorated method as is, in order to have no impact on speed. On the other hand, projects built on top of Sage, like findstat are welcome to customize locally this hook to instrument the Sage code and exploit the semantic information provided by this decorator.

This ticket therefore:

Defines combinatorial_map as a no-op
Discuss the purpose of this decorator
Uses the previous implementation as an example of how to customize combinatorial_map

Note: this change is slightly backward incompatible since combinatorial_maps_in_class is not functional any more by default.
------------------------------------------------------------------------------

Vincent: maybe you can merge this ticket into yours since I put in
quite some documentation? Or maybe we want to do this in two steps, if
you rebase yours on mine? I for myself need to move on to something
else.

[Christian Stump]

> Does findstat relly on a big database that is enhanced with the new queries? Or does it compute every query from scratch?

I start with answering here since this the crucial point.

We want to use Sage in FindStat only in the background, a user
interacting with FindStat (by searching for statistics, or by
contributing statistics) should not see what's happening in the
background.

By a combinatorial statistic I mean a map from some combinatorial
collection to the integers. FindStat does not store combinatorial
statistics as maps, but only as finite explicit data sets (and finite
should be flexible in the sense that it can be very few or quite some,
say 10-2000).

Reasons:

- this gives anyone the chance to contribute to FindStat: it might
be that you computed some statistic by hand for a few values without
having any written code (think of the dimension some variety attached
to a permutation and not of something you can code super easily), or
it might be that you prefer to work in one of the Ma's and still want
to contribute your statistic.

- it is *impossible* to compute statistics on the fly. Even if you
had code to compute the statistic: what if it takes 13sec to compute
your statistic for the permutation [6,5,4,3,1,2] (just checked with
the number of decompositions into simple transpositions). In a single
database search, there are 10000th if not millions of statistics
values tested. These cannot be computed on the fly, even if you store
them temporarily.

> Just to clarify, i wasn't thinking on sage sending a query to findstat

But that will be the only possible way to go. Your question here is as
asking the same for the OEIS, see above.

> but performing the search locally (i.e. having findstat software included in Sage).

I don't know if that's maybe as well what you thought of: it might be
possible to query FindStat for the plain statistics data and do the
remaining computations within Sage. I would actually very much
appreciate if we were doing that at some point simply because I would
expect this to result in strong improvements of the search engine we
have written for FindStat so far.

BUT: this would result in code in Sage that is not useful purely
within Sage. And there are people, loud people, that say there should
not be such code in Sage.

Another problem that would need to be solved would be: Such code would
always need to interact with FindStat and would thus need to follow
predefined specifications for that interaction.

Cheers,

Christian

[Martin R]

OK, great, thanks for clarifying!

Am Mittwoch, 28. Mai 2014 20:53:36 UTC+2 schrieb Simon King:

Hi Martin,

On 2014-05-28, 'Martin R' via sage-combinat-devel <sage-comb...@googlegroups.com> wrote:
>> E.g., it should know domain and codomain, and it should know
>> what category it belongs to. I think it makes sense to let a morphism
>> know whether it is injective or surjective. However, additional
>> information that is certainly interesting to researchers (e.g.: "Was first
>> defined by John Doe in his seminal paper in Journal of Applied
>> Irrelevance") clearly should not be stored as an attribute of a morphism.
>>
>
> I do not see any reason why such information should not be stored somewhere
> within sage, possibly in form of a database.  Could you please explain?

I clearly said: "... should not be stored as an attribute OF A
MORPHISM". I also clearly said "... is certainly interesting to
researchers", hence (which I said in other parts of the post) it would
certainly be worth-while to store somewhere in Sage. I just doubt that
storing it locally as an attribute of a morphism is a good idea.

Best regards,
Simon

[Martin R]

E.g., it should know domain and codomain, and it should know
what category it belongs to. I think it makes sense to let a morphism
know whether it is injective or surjective. However, additional
information that is certainly interesting to researchers (e.g.: "Was first
defined by John Doe in his seminal paper in Journal of Applied
Irrelevance") clearly should not be stored as an attribute of a morphism.

I do not see any reason why such information should not be stored somewhere within sage, possibly in form of a database.  Could you please explain?

Martin 

[Thierry]

Hi,

On Thu, May 29, 2014 at 12:29:45PM +0200, Christian Stump wrote:

> We want to use Sage in FindStat only in the background, a user
> interacting with FindStat (by searching for statistics, or by
> contributing statistics) should not see what's happening in the
> background.

As a developper and possible future user, i do. I could not find the
source code from your wiki, papers of web search. Could you pease
provide a link to it so that we can have an idea of its complexity or
the way it could be integrated into Sage ?

> > Just to clarify, i wasn't thinking on sage sending a query to findstat
>
> But that will be the only possible way to go. Your question here is as
> asking the same for the OEIS, see above.

Actually, it is possible to install the OEIS database within Sage and do
searches offline, so i wonder why it should not be possible for
findstat:
http://www.sagemath.org/doc/reference/databases/sage/databases/sloane.html
By the way, there is a plan to merge the three OEIS-related tools within
the oeis class (online searches, offline searches, infinite OEIS
sequences provided by sage).

Ciao,
Thierry
:w

[Nicolas M. Thiery]

On Thu, May 29, 2014 at 12:29:45PM +0200, Christian Stump wrote:

> BUT: this would result in code in Sage that is not useful purely
> within Sage. And there are people, loud people, that say there should
> not be such code in Sage.

I can hear your frustration ... In similar situations, it helped me to
keep in mind that loud people are not always representative. Of course
the difficulty is to fetch the opinion from the others.

[Nicolas M. Thiery]

On Thu, May 29, 2014 at 11:42:40AM +0200, Nicolas M. Thiery wrote:
> Here is the description of http://trac.sagemath.org/ticket/16410,
> which potentially needs review:
>
> ------------------------------------------------------------------------------
> From the discussions of (TODO: add refs to the threads in sage-devel), there is a consensus that, at this point, the combinatorial_map decorator should by default return the decorated method as is, in order to have no impact on speed. On the other hand, projects built on top of Sage, like findstat are welcome to customize locally this hook to instrument the Sage code and exploit the semantic information provided by this decorator.
>
> This ticket therefore:
>
> Defines combinatorial_map as a no-op
> Discuss the purpose of this decorator
> Uses the previous implementation as an example of how to customize combinatorial_map
>
> Note: this change is slightly backward incompatible since combinatorial_maps_in_class is not functional any more by default.
> ------------------------------------------------------------------------------

Vincent on #16410:
> ...this ticket can be reviewed quickly compared to #16408 (which
> needs some design discussions). We can let this one go and then
> start discussing #16408 but I think we need more opinions on this.

Nathann, Christian, Viviane, ... do you mind briefly stating on the
ticket whether this sounds like a reasonable step forward?

[Nathann Cohen]

Yoooooooooo !!

> Nathann, Christian, Viviane, ... do you mind briefly stating on the
> ticket whether this sounds like a reasonable step forward?

Well... I don't have much to add there I think... Besides, for
political reasons, you will understand that I cannot review this
ticket :-P

For me what you do is fine. I agree with Vincent that you are actually
wiping the feature out while leaving a way to fill a database. And
this database will only be built by the find_stat guys. But to me it
is not a problem : they can use the information they like to build
their database (because they already have something running) and we
have all the time in the world to figure out what we want to do with
this information on our side.

Plus, of course, the decorator will not change anything to the
functions, which was the point :-P

Nathann

[Nathann Cohen]

> BUT: this would result in code in Sage that is not useful purely
> within Sage. And there are people, loud people, that say there should
> not be such code in Sage.

I can hear your frustration ... In similar situations, it helped me to
keep in mind that loud people are not always representative. Of course
the difficulty is to fetch the opinion from the others.

Ahahahahahahah. Well, only including in Sage code that is useful to "a Sage user" or "other developpers" does not seem that far-fetched. I still do not see what the problem is with that.

Err.... Well, assuming the obvious : that I am the "loud people" you mention.

Nathann

[Paul-Olivier Dehaye]

I might be too, I am not quite sure!

Paul

Paul-Olivier Dehaye

SNF Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

--

[Nathann Cohen]

> I might be too, I am not quite sure!

If so, I would be ashamed to steal the spotlights. The place is yours.

Nathann

[John Cremona]

On 29 May 2014 21:02, Nathann Cohen <nathan...@gmail.com> wrote:
>> > BUT: this would result in code in Sage that is not useful purely
>> > within Sage. And there are people, loud people, that say there should
>> > not be such code in Sage.

I do not know what is "code in Sage that is not useful purely within Sage".

John

>>
>> I can hear your frustration ... In similar situations, it helped me to
>> keep in mind that loud people are not always representative. Of course
>> the difficulty is to fetch the opinion from the others.
>
>
> Ahahahahahahah. Well, only including in Sage code that is useful to "a Sage
> user" or "other developpers" does not seem that far-fetched. I still do not
> see what the problem is with that.
>
> Err.... Well, assuming the obvious : that I am the "loud people" you
> mention.
>
> Nathann
>

> --
> You received this message because you are subscribed to the Google Groups

> "sage-devel" group.

> To unsubscribe from this group and stop receiving emails from it, send an

> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.

[Paul-Olivier Dehaye]

+1 for subtlety!

Paul-Olivier Dehaye

SNF Assistant Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

Nathann

[Viviane Pons]

I actually do think it would be a really good thing to have a statfinder within sage, and somehow merge findstat code with sage code. It would benefit both sage and findstat users. At the end, both FindStat and Sage have the same aim: using computers to help us doing research. I understand completely that it should have no impact on performance for other people (but it is true for every code someone adds...) but I do not understand the "it is useful only for FindStat users so it is not useful". Anyway, this not going to happen in the near future, I mostly see this as long term objective and it is only my opinion.

The short term perspectives is for "us FindStat people" to build a FindStat API that would allow query from external websites. It does not seem that complicated, it's mostly a question of taking the time to do it. Once we have that, we can start thinking of more integration, local computation, etc.

[Nicolas M. Thiery]

> On Thu, May 29, 2014 at 10:02 PM, Nathann Cohen <nathan...@gmail.com>

> > I can hear your frustration ... In similar situations, it helped me to
> > keep in mind that loud people are not always representative. Of course
> > the difficulty is to fetch the opinion from the others.
>
> Ahahahahahahah. Well, only including in Sage code that is useful to "a
> Sage user" or "other developpers" does not seem that far-fetched. I
> still do not see what the problem is with that.
> Err.... Well, assuming the obvious : that I am the "loud people" you
> mention.

Sorry, I should have clearly separated the two topics. My point was
about (1) below.

(1) When a Sage developer believes in his heart that something is the
right thing to do for Sage, (s)he should not get discouraged by
loud opposition without first reaching for the opinions of a
larger pool of developers.

Btw: there are more than one loud people :-) Actually I also put
myself in there, as the graded algebra with basis discussion from
the other month shows ...

(2) About ``only including in Sage code that is useful to "a Sage
user" or "other developpers"''. In principle, yes, sure, I
agree. Still there can be legitimate complementary point of views
and discussions about where to put the limit. E.g. when a
borderline feature is meaningful in the context of Sage, is useful
for a sister project, but does not yet have a direct use case
within Sage ...

[Paul-Olivier Dehaye]

To add on to what Viviane just said, the problem is also in using vocabulary and words such as "our side" (a constant in Nathann's prose) or 

"""

I don't even know what I think anymore. With the turn that events take, I think I don't even need to think anymore. I think that I will think what you think I think, it is much easier for me.

"""

which is a very open minded attitude, followed by 

"""

Yep. I also defended that "respecting other people's code" also meant "not adding method to the classes they use if the method is not useful for them". If it is only for internal stuff, a hidden (beginning with a '_' works for everybody...). But that was a long time ago and I gave up on #16403.

"""

(in https://groups.google.com/d/msg/sage-combinat-devel/QRUXmy6UZVo/ABg8CtDzQukJ ) 

which looks rather closed minded (but I am not sure I fully understood). 

Nathann, I understand you don't know now what you think and respect that. But did you _really_ use to think that the Graph class was yours, in the sense that you had veto rights on it? I am assuming that's the class you are referring to.  

Paul

Paul-Olivier Dehaye

SNF Assistant Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

--
You received this message because you are subscribed to the Google Groups "sage-combinat-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-combinat-d...@googlegroups.com.
To post to this group, send email to sage-comb...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-combinat-devel.

[Paul-Olivier Dehaye]

"""

E.g. when a
    borderline feature is meaningful in the context of Sage, is useful
    for a sister project, but does not yet have a direct use case
    within Sage ...

"""

Correct me if I am wrong, off the top of my head:

Assuming "the findstat people" start adding information about which maps are bijective, can't one automatically create new tests, testing two methods at once?

Assuming "the findstat people" start adding information about the ranges of maps, can't one automatically create new tests, testing that the output of a method is what it should be? (e.g. no [5,4,2] that is supposed to be a Partition but is in fact a list).

Paul

Paul-Olivier Dehaye

SNF Assistant Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

[Nathann Cohen]

Yo !

> """
> I don't even know what I think anymore. With the turn that events take, I
> think I don't even need to think anymore. I think that I will think what you
> think I think, it is much easier for me.
> """
> which is a very open minded attitude

.... and a jest, as I am sure you understood. I was answering a post which was partly about "what Nathann thinks", and I did not feel that I had anything important to contribute.

> Nathann, I understand you don't know now what you think and respect that.

I love knowing that I am respected for such effortless achievements.

> But did you _really_ use to think that the Graph class was yours, in the
> sense that you had veto rights on it? I am assuming that's the class you are
> referring to.

I have veto rights on anything that anybody does even when I am not looking. Fortunately, nobody is forced to pay attention so it is not a problem.

More seriously, this thing is open source, belongs to nobody, and there is rarely a month where I do not think "Okay I am tired of all this, I'll just stop working on Sage (or research) and do something useful with my life for a change" (last occurrence two days ago). Though you have to understand that I worked almost daily on this Graph class for the last 3-4 years. When I am not angry I care about what happens to it.

Nathann

[Paul-Olivier Dehaye]

There is definitely frustration on both "sides". My offer for you to come and talk in Zurich is serious, it's easier to talk in person. We could even invite you to a seminar to talk about mathematics. Then we go chill out somewhere and we talk about sage.

Paul

Paul-Olivier Dehaye

SNF Assistant Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye

--

[kcrisman]

There is definitely frustration on both "sides". My offer for you to come and talk in Zurich is serious, it's easier to talk in person. We could even invite you to a seminar to talk about mathematics. Then we go chill out somewhere and we talk about sage.

 

An excellent idea for all such situations, by the way, with ample previous examples in other open source projects.  With beverage of choice (mine: hot cocoa).  Teh interwebs are very poor for this kind of thing :-(

But there is a real technical disagreement here too.  It sounds like - maybe? - it is finally being resolved?  (Enough, I mean.)

- kcrisman

[Nathann Cohen]

> But there is a real technical disagreement here too.

There is no such thing. One of Paul-Olivier's last posts was about how he thought we agreed 100% together and wanted me to say it. It is part of our couple therapy.

https://groups.google.com/d/msg/sage-combinat-devel/QRUXmy6UZVo/hfPQfkVzT8MJ

Nathann

[Nicolas M. Thiery]

On Thu, May 29, 2014 at 11:28:58PM +0200, Paul-Olivier Dehaye wrote:
> """
> E.g. when a
> borderline feature is meaningful in the context of Sage, is useful
> for a sister project, but does not yet have a direct use case
> within Sage ...
> """
> Correct me if I am wrong, off the top of my head:
> Assuming "the findstat people" start adding information about which maps
> are bijective, can't one automatically create new tests, testing two
> methods at once?
> Assuming "the findstat people" start adding information about the ranges
> of maps, can't one automatically create new tests, testing that the output
> of a method is what it should be? (e.g. no [5,4,2] that is supposed to be
> a Partition but is in fact a list).

I guess we could. I am not sure to see your point though. Is it that a
potential direct use case within Sage would be to exploit this
semantic information to do more automatic testing which in turn would
contribute to make Sage more robust?

[Paul-Olivier Dehaye]

Yes, that is my point. I just mentioned this as a quick way to show a plain, clear and direct use case. I didn't want to pick a more exciting and speculative one.

Paul

Paul-Olivier Dehaye

SNF Assistant Professor of Mathematics

University of Zurich

skype: lokami_lokami (preferred)

phone: +41 76 407 57 96

chat: paulo...@gmail.com

twitter: podehaye

freenode irc: pdehaye