Re: [sage-combinat-devel] Re: Permutations... again.

143 views
Skip to first unread message

Nicolas M. Thiery

unread,
Nov 22, 2012, 11:38:40 AM11/22/12
to sage-comb...@googlegroups.com, sage-...@googlegroups.com
Hi Nathann,

On Thu, Nov 22, 2012 at 06:13:52AM -0800, Nathann Cohen wrote:
> Vincent, Travis, Dima : there's a new patch at #13742. It does nothing. It
> just checks input. It adds tests and warnings. And I lost half a day and a
> lunch because of very bad work.
> Feel free to edit the patch as you like. I just want to be sure I can
> trust permutations when I build one. Honestly, for me the only good
> solution is to remove Permutations from Sage and let everything break.
> Then say "if you need it, write it". 5 years of waiting for it to happen
> have not done much good.

Mike Hansen worked like hell back in 2007 to port 30k lines of code
from MuPAD-Combinat to Sage. Of course that was not perfect by far:
you can't achieve such a massive goal and get a clean result in such a
short time, especially when the proper infrastructure is not yet
there. And yes, it often takes more time to refactor code than to
write it from scratch once you have the proper infrastructure. But at
least we had something we could work and do our daily research work
with. Altogether Mike' work was a huge asset in the migration of our
community from MuPAD to Sage, which in the end has already led to
200k+ of code, at lot of which being quite clean.

Yes Permutations, as they are, are quite crappy; we claimed that from
day one. I totally understand your frustration from having been myself
hit hard so often by so many other crappy spots from the original Sage
code. Still, I am grateful to the people who rushed so hard to write
*something* and make Sage happen.

Granted, the refactoring of all this code is frustratingly long. But
there are a *lot* of efforts going on and things are progressing, one
step at a time (thanks everybody for that!). This will eventually get done.

And we should *not* remove that code in the meantime, because, as
crappy as it is, it still serves its purpose in waiting for something
better. If you want to add big fat warnings and some sanity checks,
sure go ahead. But please be kind and unobtrusive to those who wrote
it and those who are working hard to improve it.

Cheers,
Nicolas
--
Nicolas M. Thi�ry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/

Travis Scrimshaw

unread,
Nov 22, 2012, 12:41:28 PM11/22/12
to sage-comb...@googlegroups.com, sage-...@googlegroups.com
Hey Nathann,
   The main reason why I stalled on the permutations input was mainly trying to figure out how to handle generalized permutations (a.k.a. bi-words, two-line arrays, in bijection with N-matrices) so the RS(K) examples will still work (and as a 2fer, getting the full RSK working), and I wanted to talk in particular with Nicolas and Florent about how to structure/refactor everything. The current version of trac_8392-check_permutation-ts.patch in the sage-combinat queue currently checks to make sure input is 1 to n, however I can base that patch on the #13742 patch. I can also get back immediately into finishing the above patch if you want too.

Best,
Travis
Nicolas M. Thi�ry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/

Nathann Cohen

unread,
Nov 23, 2012, 4:44:07 AM11/23/12
to sage-comb...@googlegroups.com, sage-...@googlegroups.com
Nicolas, I know you respect his job very much and I have nothing against Mike Hansen of course.  But these days I have been using Posets and Permutations a bit, and with Florent we found several problems already, and I am pretty sure you know muuuch more of them.

This thing with the permutations. The problem with Facade = False by default on Posets, see 
https://groups.google.com/forum/?fromgroups=#!topic/sage-combinat-devel/bfWbG54sedM

There is also that bug about memory usage, 10 days ago :
https://groups.google.com/forum/?fromgroups=#!topic/sage-combinat-devel/AdErcxGPq4c

And there is also that THE INEQUALITY FOR POSETS IS BROKEN, because of "some thing you needed to add inside to have your computations go faster" (Poset inequality is only True if the cached linear extensions coincide), and "that cannot be easily changed because the code of a lot of people in Sage-combinat would be too slow otherwise".

And none of these things ever get solved. What I have a hard time accepting is that you do not feel such things as dangerous, and bad, and the need to fix them quickly. These bugs have been reported and nobody who uses these classes everyday feels like fixing them. And the bug stays. Mike's work is all very nice, but it was 5 years ago. The code should have been rewritten since just by fixing things, step by step, as the bugs appeared.

I have a very hard time accepting your development model. This Sage-combinat thing. The first time you told me about it, I think it was at the CIRM, you told me that "The goal was early integration". Ok, I can see a point to that (actually I don't, but I accept that you may, and hence I accept that answer), but look at all the problems it creates :
* To me, Sage-combinat is a *FORK* of Sage. Because you work on Sage-combinat, and hence do not feel necessary to contribute to Sage, because if a thing is fixed in Sage-combinat you do not have a problem with it anymore. It's a fork.
* You have "users", whom I call customers. I call them customers because you do a crazy amount of work for them (rebasing this damn patch queue) with the excuse that you cannot expect them to *UPDATE* their version of Sage by themselves. You told me several times that a patch could not be written because it would mess up with their code, or with its speed. Now you said that we cannot switch to GIT unless the Sage-combinat queue can be transferred to GIT too. Hell !
* You have one thousand patches there that are dying there. Last year I spent several days during the Sage Days in Cernay just trying to help write the documentation of one patch. Where is that patch ? Still in the combinat queue ? Same thing with Florent's patch on SearchForest. This is good code but I can already see how it goes : he works again on the code, improves that and that, and even though I tried to help him with the documentation I do not see the patch being submitted anytime soon. What is actually wrong, since he can already use it himself in Sage ?

This is a bad developpement model. I mean, I can show two big problems (broken equality for posets, and this awful Facade thing) that you explicitely told me cannot be fixed because of the Combinat guys. You produce patches and do not write the documentation, because you do not have to, because your development model is "put whatever you want in the combinat queue, and you don't have to write a clean Sage patch because you can use it already like that".

That's how I fix bugs in graphs : I write a patch, send it to the TRAC server, and forget about it until somebody comes to review it. Then I can totally forget about that bug. How do you propose we do the same with the constraints of Sage-combinat ? I saw three bugs, and all three have to be fixed.
No problem with Mike's code. But not fixing bugs that you know exist is actual bad work.

Some answers to points you raised in your post :


> Granted, the refactoring of all this code is frustratingly long. But
> there are a *lot* of efforts going on and things are progressing, one
> step at a time (thanks everybody for that!). This will eventually get done.

5 years. Look at it. This was done 5 years ago. It has not been fixed. Let's be honest : it will NEVER be fixed. Not this way. Not by just waiting for it to happen.


> And we should *not* remove that code in the meantime, because, as
> crappy as it is, it still serves its purpose in waiting for something
> better.  If you want to add big fat warnings and some sanity checks,
> sure go ahead.

I like to play the dumb guy, and I love stupid methods like this one. And there is a big advantage to this method : if you remove it and everything breaks, the code will be replaced in no time. That's for sure, because everybody needs it. But for as long as you have a bad code, and because you don't feel the need to rewrite it, nothing happens.


> But please be kind and unobtrusive to those who wrote
> it and those who are working hard to improve it.

Nicolas.... Honestly... I run code on my machine which you know is wrong, but it stays this way.. What do you call being unobtrusive ? Why doesn't it include to respect people by not letting them run code that you KNOW is wrong, and that they could legitimately trust a priori ?

Nathann

Andrew Mathas

unread,
Nov 23, 2012, 7:37:44 AM11/23/12
to sage-comb...@googlegroups.com, sage-...@googlegroups.com
Hiiiiiiii Nathan,

I share your frustrations. I think we all do, to some extent. It would be very nice if all of these problems, major an minor, could be fixed but the reality is that until some one cares enough about a particular problem then, as you say, it isn't going to happen. I think this is called open-source.

The absent or poor documentation is what I like least about sage. It's annoying that quite a lot of "basic" things need to be implemented and that others have been done badly. I have fixed the things that I really need, or worked around them. The responsiveness of sage-devel and sage-combinat is nice.

I switched to sage because it is the best platform that I found. Sure, many things could be better, but I don't like gap4, magma, malple, mathematica, matlab, C, ... In spite of its many warts and problems I do like writing in python and working with sage. Using sage, I have been able to compute things much more quickly and painless than before when I was writing in gap. I'm happy with that.

Does anyone have any concrete suggestions for how we could improve the "development model"? There is definitely room for improvement but short of paying someone I don't have any good suggestions. The problem is that what I want implemented is often different to what everyone else wants, but thankfully there is some overlap and so some give and take. Ideas anyone?

Andrew

P Purkayastha

unread,
Nov 23, 2012, 8:01:59 AM11/23/12
to sage-...@googlegroups.com, sage-comb...@googlegroups.com
On 11/23/2012 08:37 PM, Andrew Mathas wrote:
> Does anyone have any /concrete suggestions/ for how we could improve the
> "development model"? There is definitely room for improvement but short
> of paying someone I don't have any good suggestions. The problem is that
> what I want implemented is often different to what everyone else wants,
> but thankfully there is some overlap and so some give and take. Ideas
> anyone?
>
> Andrew

There is already a move towards a git-based development model. I hope it
will allow people to fork and experiment in their own branches quickly
instead of the current method of using queues, trac patches, or "sage
-clone", etc, which I find is a painful process.

I would say the git model might work better. For instance look at the
notebook. It is very easy to work with the development branch of any
person's fork of the notebook. At present, you can even try out out the
new UI by simply switching the notebook branch to samuela's fork. The
switch takes about a second once you have the branch in your git. You
can quickly also change back to the original notebook and resume your
usual work with that as soon as you are done experimenting. The only
hurdle that I see right now is the about 5-6 git commands one needs to
know before one can start working with the development branch(es) of the
notebook. Part of the next sage-git workshop is to create tools to make
this setup (and contributing back) easy.

Nathann Cohen

unread,
Nov 23, 2012, 8:50:06 AM11/23/12
to sage-comb...@googlegroups.com, sage-...@googlegroups.com
Hellooooooooooooooo !!


> I share your frustrations. I think we all do, to some extent. It would be
> very nice if all of these problems, major an minor, could be fixed but the
> reality is that until some one cares enough about a particular problem then,
> as you say, it isn't going to happen. I think this is called open-source.

I don't know. People usually pay attention to the code they wrote even much later. I still write to Robert Miller when I have questions on the graph backends, and I like to have his advice on Sage things from time to time. He always answers. But because it seems the combinat/ folder is full of stuff Mike Hansen wrote quickly, nobody feels responsible of them. That's also something which is quickly fixed if the classes were to be rewritten. Somebody would know them totally, and perhaps feel that they owe the community to make it a good work. I really, really hate to see on the sage-devel mailing list "is that a bug in a graph function ?", but unfortunately it happens quite often. I try to answer those things as quickly as possible with a patch. Besides, by doing that you begin to learn what users do with Sage, and you can think of those bugs you fixed when you write some new code.


> The absent or poor documentation is what I like least about sage.

Welcome to the graph world :

http://www.sagemath.org/doc/reference/sage/graphs/base/static_sparse_graph.html
http://www.sagemath.org/doc/reference/sage/graphs/pq_trees.html
http://www.sagemath.org/doc/reference/sage/graphs/graph_decompositions/graph_products.html
http://www.sagemath.org/doc/reference/sage/graphs/convexity_properties.html
http://www.sagemath.org/doc/reference/sage/graphs/distances_all_pairs.html
http://www.sagemath.org/doc/reference/sage/graphs/comparability.html
http://www.sagemath.org/doc/reference/sage/graphs/graph_decompositions/vertex_separation.html

Just to say that Sage contains tools that can let you write a lot of documentation if you need to.

One other point I really do not like about the combinat model is that, unfortunately, you all know each other. You talk with each other, you can explain stuff to each other. And so the reviewer reviews a piece of code that he already knows how to use.
Another thing I do not like much is that I find the code far, far, far too much algebraic. I have 2 examples of graph libraries I could not stand to use because of this exactly.

Boost : http://www.boost.org/doc/libs/1_52_0/libs/graph/doc/table_of_contents.html
here's their code for breadth first search : http://www.boost.org/doc/libs/1_52_0/boost/graph/breadth_first_search.hpp

Mascopt (developped in my former lab) : http://www-sop.inria.fr/mascotte/mascopt/
(I spent 1 day full-time reading the manuals, and I had no idea how to build a graph)

Well. You have the right to like abstracton and generalizations. I prefer to duplicate code to make things independent, and easy to understand. But I do not have much code to duplicate, perhaps you do. But I find your code much too algebraic by far.
Funnily, several days ago Nicolas Florent and I were talking about Sage. They wondered whether Python was a good choice of language, and whether it would remain an important language in the future.

- "It would really be an issue to have to port all of our code to another language, so much is based on Python..."
- "Do you really use Python at all ? I mean, you wrote categories and it more or less replaces class inheritance in Python, doesn't it ?"

Finally, I admit that I like to play dumb, and that for a large part I do not have to play much. But the use of categories have been explained to me one thousand times, that each time there's something I do not like and have to accept, and that after all that time I still hate the idea of having to deal with it.


> It's
> annoying that quite a lot of "basic" things need to be implemented and that
> others have been done badly. I have fixed the things that I really need, or
> worked around them. The responsiveness of sage-devel and sage-combinat is
> nice.
>
> I switched to sage because it is the best platform that I found. Sure, many
> things could be better, but I don't like gap4, magma, malple, mathematica,
> matlab, C, ... In spite of its many warts and problems I do like writing in
> python and working with sage. Using sage, I have been able to compute things
> much more quickly and painless than before when I was writing in gap. I'm
> happy with that.
>
> Does anyone have any concrete suggestions for how we could improve the
> "development model"? There is definitely room for improvement but short of
> paying someone I don't have any good suggestions. The problem is that what I
> want implemented is often different to what everyone else wants, but
> thankfully there is some overlap and so some give and take. Ideas anyone?

The development model of Sage-combinat ? Well, I do not know much about it... A few (very) practical thoughts :

1) A bugfix patch for something that is in Sage should NEVER be included in Sage-combinat. It should be sent to Sage, and sage-combinat will inherit it later, when Sage is updated. This way bugfix patches will be merged before the author's death
2) Something I discussed with Florent : everybody in Sage-combinat should have at most 2 or 3 patches in the queue. Nothing more. If you want to add another one, get your patches merged into Sage
3) Why the hell do you need to have your own fork anyway ? (just my own thought, but I still don't exactly get it)
4) Just look at the TRAC section named combinat. Each time I go look at the patches, I just dream to put them all on "need_work" state.

http://trac.sagemath.org/sage_trac/query?status=needs_review&component=combinatorics&col=id&col=summary&col=component&col=status&col=type&col=priority&col=milestone&order=priority&row=description
I do not even joke. Look at WHEN the tickets were created. In the order I get with the previous link :
Reported 3 years ago.
Reported 3 years ago.
Reported 3 years ago.
Reported 2 years ago.
Reported 2 years ago.
Reported 2 years ago.
Reported 23 months ago.
Reported 20 months ago.
Reported 16 months ago.
Reported 11 months ago.
Reported 9 months ago.
Reported 9 months ago.
Reported 9 months ago.
Reported 8 months ago.
Reported 7 months ago.
Reported 6 months ago.
Reported 5 months ago.
Reported 5 months ago.
Reported 4 months ago.
Reported 4 months ago.
Reported 3 months ago.
Reported 3 months ago.
Reported 2 months ago.
Reported 26 hours ago.
Reported 18 months ago.
Reported 13 months ago.
Reported 6 months ago.

Honestly.... Many of these tickets are not really waiting for a review. Many are waiting for the resurrection of developpers, or for their children to learn Python. Some of them have tickets that are not updated with their Sage-combinat counterpart.

So my 4) advice is : please, clean this list of dead tickets. Exchange reviews of a ticket against review of others ? We are talking of years... Years...

I just try to convince you that your working model has made you FORK from Sage. You have your own, running version of it, with your own patches. You all know how it works so you do not really need to write the documentation. You behave as if you didn't need to contribute to Sage, and you actually don't. Florent is working on a patch to distribute computations right now, and it is a very nice thing ! But for as long as he can use it without including it in Sage, he will not have to write the documentation, clean it, and send it to be reviewed.
..... Because it is already available on Sage-combinat.

5) Accept the idea that your customers may become users, have no more rights than other Sage users and developpers, update the version of Sage they use, and stop spending a lifetime managing a queue.

Nathann

Nicolas M. Thiery

unread,
Nov 23, 2012, 9:31:35 AM11/23/12
to sage-comb...@googlegroups.com, sage-...@googlegroups.com
Hi Nathann,

Would you mind taking a break and stop spreading FUD? You are not
being fair. I totally know and care about all the crap in the combinat
code. It hurts me everyday. More than you think. There is just a
bounded amount of manpower and not everything can be tackled at once.
There is a lot of Sage-Combinat code going into Sage and things *are*
progressing, including in the cleanup of old code. Just to cite a few
recent things that got merged recently or are very close to
completion:

- Tableaux cleanups coming in by Andrew
- Cleanup in the combinatorics code by Travis, ...
- Improvements to root systems by Christian, Mark, ...
- Cleanup of Symmetric Functions by Mike, Anne, ...
- Implementation of NonCommutativeSymmetricFunctions by Chris, Franco, ...
- Quivers and cluster algebras by Christian, Gregg, ...
- Category cleanup by Simon King and myself
- Lots of reviews done by Fr�d�ric Chapoton and Hugh Thomas
- Tutorials by many

All of those are fully documented. Thanks everybody for your hard work!

There is also infrastructure work coming in step by step; yes it takes
time. Sorry, those are not things one can write in an afternoon and
just post and forget.

Yes we are a community. I am proud of that. We help each other, and
all contribute our *different* points of view and man power. No
consumer there.

Categories are nothing but abstract classes. You don't need object
orientation for your applications? Great, that's all fine to me, don't
use it. But don't complain on people that *need* it and use it.

The Sage-Combinat model is certainly not perfect, and I am happy
discussing the imperfections as long as it's not just repeating things
we all know (e.g. that we are slow on reviewing).

Now if you want to discuss this further, let's do this face to
face. But not now. I have 20 hours of teaching to prepare for next
week.

Volker Braun

unread,
Nov 23, 2012, 9:33:20 AM11/23/12
to sage-...@googlegroups.com, sage-comb...@googlegroups.com
On Friday, November 23, 2012 1:50:08 PM UTC, Nathann Cohen wrote:
Well. You have the right to like abstracton and generalizations. I prefer to duplicate code to make things independent, and easy to understand.

Code duplication is a bug, plain and simple. If it doesn't fail outright then it is sure to cause somebody else grief in the future. You might think its easier to understand, but that is only because you put it there. The next person looking at it will be confused why very similar code is copy-pasted all over the place. Just say no to code duplication. 

That doesn't mean that you have to go to extremes in generic programming as Boost. But you should at least make an effort to understand the genius behind many of its modules. In fact, if your day-to-day code looks anything like Boost then you are doing something wrong. But if you design a library interface then I'd highly recommend it.

I agree with you that there is a lack of documentation of some of the abstractions, though. Without documentation, an abstraction is just gibberish to most people and make it more difficult to contribute to Sage. Like the category stuff. Is there any way to figure out what they do without reading the source code? Why are things done the way they are done? A tutorial for using them? A tutorial for when and how to write a new category?

Paul-Olivier Dehaye

unread,
Nov 23, 2012, 9:38:53 AM11/23/12
to sage-comb...@googlegroups.com, sage-...@googlegroups.com
Purkayashta: this would only address (in good or bad) a tiny part of the development issues, those that have to do with version control.
Paul





--
You received this message because you are subscribed to the Google Groups "sage-combinat-devel" group.
To post to this group, send email to sage-combinat-devel@googlegroups.com.
To unsubscribe from this group, send email to sage-combinat-devel+unsub...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sage-combinat-devel?hl=en.


John Cremona

unread,
Nov 23, 2012, 9:54:36 AM11/23/12
to SAGE devel
It's certainly going to be exciting meeting several of you in
Edinburgh in January (at
http://www.aimath.org/ARCC/workshops/onlinedata.html) !

John

On 23 November 2012 14:38, Paul-Olivier Dehaye
>> To post to this group, send email to sage-comb...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> sage-combinat-d...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/sage-combinat-devel?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To post to this group, send email to sage-...@googlegroups.com.
> To unsubscribe from this group, send email to
> sage-devel+...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel?hl=en.
>
>

Nathann Cohen

unread,
Nov 23, 2012, 10:56:01 AM11/23/12
to sage-comb...@googlegroups.com, sage-...@googlegroups.com
Helloooooooooooooo !!!


> Would you mind taking a break and stop spreading FUD?

Ok, so let's get to work. I created #13742 yesterday and it is waiting for a review :
http://trac.sagemath.org/sage_trac/ticket/13742


> You are not
> being fair.

I know. You know. Florent knows. I always speak too much.


> I totally know and care about all the crap in the combinat
> code.

Right. This Permutation thing is waiting for a review right now. Then :

1) I will try to upload a patch this very week-end that changes the dfault behaviour of Poset as mentionned there :

https://groups.google.com/forum/?fromgroups=#!topic/sage-devel/qQzzZEh-JRo

And also (as expected) in the sage-combinat fork of that thread (only Dima answered there)

https://groups.google.com/forum/?fromgroups=#!topic/sage-combinat-devel/bfWbG54sedM

It will not be backward-compatible, and I will write to Sage-devel-combinat to warn people about that.

2) I will also try to deal with this *&@#(@%(@#$ piece of code which says that two Posets may not be equal even if they have the same hasse diagram. This is *WRONG*, so it has to be changed, Wrong, and dangerous. And bad code. Two Posets are equal if and only if their hasse diagram are equal. No cached linear extension involved. Equality in Posets *HAS* to be valid.


> It hurts me everyday. More than you think. There is just a
> bounded amount of manpower and not everything can be tackled at once.

Nothing against lack of manpower, for as long as I see that everybody honestly tries.


> There is a lot of Sage-Combinat code going into Sage and things *are*
> progressing, including in the cleanup of old code.

Yeah, there I was probably guilty of bad faith, but I am a regular.


> There is also infrastructure work coming in step by step; yes it takes
> time. Sorry, those are not things one can write in an afternoon and
> just post and forget.

Some things can, I assure you. This Permutation thing above, which was my problem. Or the default Poset (facade) thing,  I do not expect it to take more than a day. Or the other Poset thing. All those are very bad things which take less than a day, and most probably less than an afternoon. Perhaps you think that many bugs you know cannot be fixed as fast as these but I can only talk for the problems I know, and I expect that those can. I am totally sure that they are not the only ones.


> Yes we are a community. I am proud of that.

Sage is also a community. Do you feel like joining us ? :-P

> We help each other,

Err... When you have a Graph problem, you know that you can knock on my door, and so you did several times. What you got the very day or the day after when the code was not already available, is a track ticket with a patch waiting for a review.
Like there : http://trac.sagemath.org/sage_trac/ticket/13664
Like there : http://trac.sagemath.org/sage_trac/ticket/12235

I told you about the three problems above. Nothing happened. Please observe that I fix them myself, because I have to.


> Categories are nothing but abstract classes. You don't need object
> orientation for your applications? Great, that's all fine to me, don't
> use it. But don't complain on people that *need* it and use it.

You are being unfair too. There is quite a gap between Object Oriented Programming and your Category cathedral. It's not just Object Oriented Programming, and it is packed with subjective decisions you made. It's not as straightforward as that.


> The Sage-Combinat model is certainly not perfect, and I am happy
> discussing the imperfections as long as it's not just repeating things
> we all know (e.g. that we are slow on reviewing).

I enumerated 4 points in my previous email. Point 3 is to me the most important, but I always try to make questions that will be ill-received look like things you can ignore. I honestly still do not see the point of Sage-combinat considering its many problems. All 3 others are (methinks) very legitimate. What do you think of them ? What can be done about that ?


> Now if you want to discuss this further, let's do this face to
> face.

Nicolas, can you tell me that there is ONE thing among those I said in my email which I have not mentionned to you before ?
I told you that you had forked Sage. I told you that I thought you needed WAYS to force you to send patches to Sage. I told you that the code was not documented enough, I even spent days in Cernay WRITING the documentation of combinat code. I told you about all this way before you read this email.


> But not now. I have 20 hours of teaching to prepare for next
> week.

Happy guy. While I have it easy, living in somebody else's place, and looking for an appartment in Paris :-P

Nathann

Nathann Cohen

unread,
Nov 23, 2012, 9:35:40 PM11/23/12
to sage-comb...@googlegroups.com, sage-...@googlegroups.com
> 1) I will try to upload a patch this very week-end that changes the dfault
> behaviour of Poset as mentionned there :

Done ! It is now patch 13747 : http://trac.sagemath.org/sage_trac/ticket/13747

And it is also very, very late... :-P

Good night everybody !

Nathann

Andrew Mathas

unread,
Nov 23, 2012, 11:14:46 PM11/23/12
to sage-...@googlegroups.com, sage-comb...@googlegroups.com
Hi Nathan,

I'm sure that I should drop this but...


I don't know. People usually pay attention to the code they wrote even much later. I still write to Robert Miller when I have questions on the graph backends, and I like to have his advice on Sage things from time to time.

When ever anything gets big enough I think this starts to break down. For example, when I whined about the lack of documentation for the pickle jar the people who put this is place didn't feel the need to step in and address this. Even though I am just a sage-combinat customer:), I posted a patch on trac and it is being reviewed (really!).

> The absent or poor documentation is what I like least about sage.

Welcome to the graph world :
 
I agree that's nice, but what I am complaining about is a general lack of documentation in sage, particularly for the metaclass framework, rather than documentation for code in sage/combinat. If you check, the code in sage/combinat has a relatively high coverage rate.

One other point I really do not like about the combinat model is that, unfortunately, you all know each other.

I am really quite amazed that you see this as a bad thing! Because the people in sage-combinat do talk to each other you can, if you want, get a lot of feedback when writing something. Moreover, in my experience, when a patch is ready to be reviewed it normally happens very quickly.

1) A bugfix patch for something that is in Sage should NEVER be included in Sage-combinat. It should be sent to Sage, and sage-combinat will inherit it later, when Sage is updated. This way bugfix patches will be merged before the author's death

 I think that was you are really complaining about are bug fixes which end up in the sage-combinat queue but are not posted to trac. I fully agree that patches should be posted to trac as soon as they are ready for review and that people shouldn't keep half finished patches lying around in the sage-combinat queue, or elsewhere. I don't see any harm, however, in a patch being both on trac and also in the sage-combinat queue if it is relevant.
 
2) Something I discussed with Florent : everybody in Sage-combinat should have at most 2 or 3 patches in the queue. Nothing more. If you want to add another one, get your patches merged into Sage
 
As Anne said, "it would be a good idea to try to integrate patches in the sage-combinat
queue more quickly."

3) Why the hell do you need to have your own fork anyway ? (just my own thought, but I still don't exactly get it)
 
For me one of the key advantages of the sage-combinst queue is that it allows code to be used and tested while it is being developed and before it is reviewed. The best way to test code it to use it, so this actually saves a lot of time and leads to better code.

4) Just look at the TRAC section named combinat. Each time I go look at the patches, I just dream to put them all on "need_work" state.

I think this is a little spurious: you see a similar picture with every component of sage, especially when you adjust for the size of the components. I think that this is a fair criticism of the sage-combiat patch queue but not of the patches on trac. Btw, many of these posts on trac come from the general sage community and in addition to patches being reviewed there are also feature requests

Let me end by saying that I thought it funny that you make such a big distinction between sage and sage-combinat as I don't really see a difference. As I see it, the development model of sage and sage-combinat, which is after all just a subset, is exactly the same. The sage-combinat patch queue makes it easier for people working on applications in algebraic combinatorics to talk with each other and to exchange ideas. The queue is a pain to maintain sometimes but I think it's infinitely better than emailing patches back and forwards or pushing half finished patches to trac.

The bottom line, which is completely independent of the sage-combinat patch queue, is that if you don't like something in sage then you have to jump in and fix it -- or convince some one to do it for you!

Cheers,
Andrew

Nathann Cohen

unread,
Nov 24, 2012, 5:21:46 AM11/24/12
to sage-comb...@googlegroups.com, sage-...@googlegroups.com
Helloooooooooo Andrew !


> I'm sure that I should drop this but...

At this level it may not hurt anymore :-)


> When ever anything gets big enough I think this starts to break down. For
> example, when I whined about the lack of documentation for the pickle jar
> the people who put this is place didn't feel the need to step in and address
> this. Even though I am just a sage-combinat customer:), I posted a patch on
> trac and it is being reviewed (really!).

HMmm..   I don't agree with that. I do not work with that many Sage developpers but I am totally convinced that any of them feels responsible about his own code to fix it when a bug is found. But you talk of documentation, and there I totally expect that it may not work anymore, as it is less "urgent and dangerous". At least for the classes I am patching now, I am pretty convinced that the explanation lies in the fact that "nobody here wrote them". I think that we are rather on the ascending path than the final explosion.


> I agree that's nice, but what I am complaining about is a general lack of
> documentation in sage, particularly for the metaclass framework, rather than
> documentation for code in sage/combinat. If you check, the code in
> sage/combinat has a relatively high coverage rate.

Well documenting the methods when it comes to categories is not really helping, it asks for a more detailed explanation. This way of writing doctests works nicely for the graph stuff, where you only want to know what the method parameters mean and what the output is quickly, but the category stuff only asks for a 10 pages explanation with paragraphs, definitions and a long list of corner-cases situations.


>> One other point I really do not like about the combinat model is that,
>> unfortunately, you all know each other.
>
> I am really quite amazed that you see this as a bad thing!

Ahhah. Sorry. I love to present as a bad thing something that is definitely good, and the opposite too. Sometimes I get convinced that it helps to think... There are many situations where you do not see the very bad sides of a good situation. "It looks so good, it just has to have good sides only !". But I insist that it is not only good in this case, especially when documenting code. Florent and I have done the same thing a couple of times : he has a patch to write and I do not know what is it, or barely, and I try to write its documentation while he explains it to me, and while I ask him questions.
This way, you know what a totally ignorant users needs to know when he first meets the class. But the same way that -- after a while -- Sage developpers are totally unable to see things through the eyes of a newcomer and take a LOT for granted, it is hard to explain to somebody new something that you have taken for granted for weeks while talking with a colleague.

Of course it is great to be able to talk together. I like this very much, even though I keep complaining about everything. Nicolas and Florent teach me many things quicker than I would have been able to teach myself otherwise(we find bugs by just doing that), but there is, in this way of working, a problem that does not exist when you cannot talk with each other : the guy who does the review will get the very same input the users will : the doc, and nothing else.


>  I think that was you are really complaining about are bug fixes which end
> up in the sage-combinat queue but are not posted to trac.

Yep


> I fully agree that
> patches should be posted to trac as soon as they are ready for review and
> that people shouldn't keep half finished patches lying around in the
> sage-combinat queue, or elsewhere. I don't see any harm, however, in a patch
> being both on trac and also in the sage-combinat queue if it is relevant.

Oh. I'm talking about the development model, and actually of... politics. If you keep adding fixes to the combinat queue, the bug will be fixed as far as the combinat guys are concerned, and they will not have any need to send it to trac. There is no need, and so it does not get done. Of course there is nothing wrong with having bugfixes in Sage-combinat. There is, for instance, no problem at all with including inside of Sage combinat a positively reviewed patch that will be merged ..... later, when Jeroen will feel like doing it. But the point is that if you insist a bugfix patch should not be there unless it is already positively reviewed, or not at all, then I am pretty sure everybody will send those patches to trac, because like everybody you need to have the bugs you meet FIXED in the code you work with.


>> 2) Something I discussed with Florent : everybody in Sage-combinat should
>> have at most 2 or 3 patches in the queue. Nothing more. If you want to add
>> another one, get your patches merged into Sage
>
>  
> As Anne said, "it would be a good idea to try to integrate patches in the
> sage-combinat
> queue more quickly."

That will not happen by just asking, though. That's what a development model is. That's what politics is all about. There are ways of doing things which infuence the way things are actually done. With the current combinat model, there is no need for the combinat developpers to send their code to Sage. That's why I call it a fork.


>> 3) Why the hell do you need to have your own fork anyway ? (just my own
>> thought, but I still don't exactly get it)
>
>  
> For me one of the key advantages of the sage-combinst queue is that it
> allows code to be used and tested while it is being developed and before it
> is reviewed. The best way to test code it to use it, so this actually saves
> a lot of time and leads to better code.

I know, I know. But look at the bad sides too ! Look at what you have to pay in exchange for that ! And did you ever really work with the REAL Sage ? Things can go pretty fast there too. And you can trust each other's code because it is fixed, and the documentation gets written too. It's not thaaaaat bad. But please, see that your development model has many good sides, like the one you mentionned, but that it comes with a load of problems that cannot be solved without changing the way you work. That's totally one of those ! This advantage is what makes combinat behave like a fork of Sage ! You are totally independent of Sage, you don't *NEED* to contribute to have your version of it evolve, and that's the problem !


>> 4) Just look at the TRAC section named combinat. Each time I go look at
>> the patches, I just dream to put them all on "need_work" state.
>
>
> I think this is a little spurious: you see a similar picture with every
> component of sage, especially when you adjust for the size of the
> components.
> I think that this is a fair criticism of the sage-combiat patch
> queue but not of the patches on trac.

Well. To me it gave an idea of how patches can be as good as dead, even though you already use them in combinat.


> Btw, many of these posts on trac come
> from the general sage community and in addition to patches being reviewed
> there are also feature requests

> Let me end by saying that I thought it funny that you make such a big
> distinction between sage and sage-combinat as I don't really see a
> difference.

That's a problem.


> As I see it, the development model of sage and sage-combinat,
> which is after all just a subset, is exactly the same.

NOooooooooooooooooooo pleaaaaaaase !! IT IS NOT ! That's my whole point ! I always say that combinat is a fork of Sage, because even if you say that patches will get merged eventually (and they sometimes do), in practice it behaves as a totally indepedent software with its developent model. You do NOT review patches internally as we do in Sage. In Sage, we actually have no way to share code unless it is clean and documented. You do NOT do that in combinat, for you work on those patches. You do not have this need, and so you work with your own copy/fork of Sage, and we don't get the bugfies, and we don't get the features that are not ready for a Sage review, etc, etc.. It's a fork, because it CAN behave so independently. Please, do not just stay convinced that "all combinat patches eventually get merged". Look at how it is in pactice ! It is much slower than that, and this is because, then again, you CAN share code without reviewing it, without writing documentation, without testing the cases the author has no interest in !


> The sage-combinat
> patch queue makes it easier for people working on applications in algebraic
> combinatorics to talk with each other and to exchange ideas.

Yes. That's the good side of your development model. That's also why you behave like if you were working on a fork.


> The queue is a
> pain to maintain sometimes but I think it's infinitely better than emailing
> patches back and forwards or pushing half finished patches to trac.

YES, but if you were working with the Sage TRAC server the patches would HAVE to be finalised. Please please, that's the whole point !


> The bottom line, which is completely independent of the sage-combinat patch
> queue, is that if you don't like something in sage then you have to jump in
> and fix it -- or convince some one to do it for you!

Ahahaha. That's why most bug reports begin with "Am I doing anything wrong or is there a problem with [...]" :-)

And I forwarded a bug report just yesterday to a french developper who wrote some code we use in Sage, to get as an answer "Nono, it's not unpleasant for a bit to getbug reports ! I am glad to know this code is useful !" :-)

Have fuuuuuuuuuuuuun !

Nathann

Nicolas M. Thiery

unread,
Nov 24, 2012, 6:10:01 AM11/24/12
to sage-comb...@googlegroups.com, sage-...@googlegroups.com
Hi all,

In very short:

> You are totally independent of Sage, you don't *NEED* to
> contribute to have your version of it evolve

Not quite true: we do pay a heavy price for each and every patch that
is not merged: we have to keep rebasing them. I can tell you that it
*hurts*. We all agree that we should get more patches in *and* we are
working on it.

Another fact: many of the patches in the Sage-Combinat queue have
strong dependencies. This not just because we screwed up our splitting
of the task; it is intrinsic to the things we are developing. In the
dependencies are things that are basically out of our control
(e.g. the patches around #715) that are super technical and that just
take time to get into Sage. Given this, there is nothing we can do but
keep those patches alive. That being said, I certainly take the blame
for not finishing faster the "more functorial patch" faster.

Last fact: many of us spent weeks, if not months, at Sage days this
year training newcomers. Don't worry, we *are* getting strong feedback
from outsiders.

Andrew Mathas

unread,
Nov 24, 2012, 6:29:10 AM11/24/12
to sage-comb...@googlegroups.com, sage-...@googlegroups.com

YES, but if you were working with the Sage TRAC server the patches would HAVE to be finalised. Please please, that's the whole point !

Nathan, I think that this is the whole point that you are not seeing: we do work with sage and we do use trac. Extensively. Personally, I want my patches posted to trac as quickly as possible and merged into sage.

Andrew

ps As I am sure you know, putting a patch up on trac in no way guarantees that it will ever be finalised.

Nathann Cohen

unread,
Nov 24, 2012, 6:43:22 AM11/24/12
to sage-comb...@googlegroups.com, sage-...@googlegroups.com
Hellooooooooooooo !!


> Nathan, I think that this is the whole point that you are not seeing: we
> do work with sage and we do use trac. Extensively. Personally, I want my
> patches posted to trac as quickly as possible and merged into sage.

Yeah, perhaps I am wrong on that. I said a lot of things about stuff I do not really understand during the last two days. I never worked with Sage-combinat because I never saw the point. What I say is actually just a result of conversations I had with Nicolas and Florent. Iar them talk abou patches, try to work with Florent to write documentation, and I see how it goes... I guess you may have as many combinat experiences as you have combinat users ^^;


> ps As I am sure you know, putting a patch up on trac in no way guarantees
> that it will ever be finalised.

Well. Indeed, but if you have no other way to share it with a colleague before, that's still a strong force to motivate you :-)

Nathann

Sébastien Labbé

unread,
Nov 24, 2012, 12:57:46 PM11/24/12
to sage-...@googlegroups.com, sage-comb...@googlegroups.com, Nicolas M. Thiery
Hi,

I have been using the sage-combinat branch from 2008 to 2010 (all of my patches were in sage-combinat before getting on the sage trac) and then stop using it. Here is my view of this. First I believe the following principle is simple but very important :

Principle #1 : The priority (ordering) of patches to be merged into Sage should be given to the next patch which is ready and have positive review.

To me the main disadvantage of Sage-Combinat queue is that it does not tend to respect Principle #1.

Here is why I claim this. First let us introduce some definitions.

Definition #2 - "Conflict". There is conflicts between two patches if they is a conflict when applying one patch after the other or if the doctest of one patch is broken by the other.

Principle #3 : When someone creates a patch in the sage-combinat queue, he or she will put the patch somewhere it does not disturb others, so somewhere it does not create conflicts with other patches. That means the patch will most probably be added after related patches susceptible to conflicts.

The consequence of Principle #3 is that the creator of a patch in the sage-combinat queue accept that *the time before inclusion of its own patch into Sage is larger than the time of inclusion of all patches it depends on*. Moreover, those patches depends on others which depends on others and so on...

Definition #4 - "Slow patch" : A patch is slow if it stays on the "needs work" or "needs review" status for a long time.

The problem is that a certain patch might depend on a slow patch, so that it can be considered as a slow patch even if it is during a period of time while you are promptly working on Sage.

I understand that a group of developpers working on similar research subject might work on similar patches/files and are suceptible to conflicts. Thus, it might be a good idea to share a common queue to "anticipate and see conflicts in advance". For example : the sage-combinat queue. But, this tends to respect Principle #3 which violates Principle #1.

Another reason which tends to violate Principle #1 is that changing the series file (changing the order of patches) takes a non negligeable amount of energy so that you might prefer to keep the ordering as it is and defined following Principle #3.

Indeed, testing if a patch commutes with others is not that fast to check : one must unapply the patches, change the series file, reapply all the patches, run sage -br (with hundreds of modified files!!!) Then, once the commutativity is proved, one must recheck that the patch still work before putting it under review (unapply patches to go back to you patch, sage -br (with hundreds of modified files again!!!!). You might after build the documentation doing sage -docbuild reference html (with again hundreds of modified files again!!!)

Hence, I believe that a development model where there exists a sane competition to be the first to be merged (creating conflicts to eventual slow patches) might be better than a system which tries to avoid conflicts (favorising slow patches).

I am looking forward to discuss with Nicolas, Florent and Nathann about it during the next meeting of Sage users in Paris region.

Cheers,

Sébastien

Robert Bradshaw

unread,
Nov 25, 2012, 1:40:06 AM11/25/12
to sage-devel
Thanks for your well thought out analysis of the situation.

> Indeed, testing if a patch commutes with others is not that fast to check :
> one must unapply the patches, change the series file, reapply all the
> patches, run sage -br (with hundreds of modified files!!!) Then, once the
> commutativity is proved, one must recheck that the patch still work before
> putting it under review (unapply patches to go back to you patch, sage -br
> (with hundreds of modified files again!!!!). You might after build the
> documentation doing sage -docbuild reference html (with again hundreds of
> modified files again!!!)

This does sound like a lot of boring manual labor to get good "ready"
code into Sage itself. It also is something the patchbot is intended
to solve--it's a pain to do manually but could totally be automated in
the background. Ideally you could mark a patch as "ready to go in" and
everything else would be automated (including seeing how far down the
stack it could commute and whether it's feasible to merge into Sage
proper.

Perhaps part of the issue is that the true dependencies are linearized
into a single queue.

> Hence, I believe that a development model where there exists a sane
> competition to be the first to be merged (creating conflicts to eventual
> slow patches) might be better than a system which tries to avoid conflicts
> (favorising slow patches).

+1

> I am looking forward to discuss with Nicolas, Florent and Nathann about it
> during the next meeting of Sage users in Paris region.
>
> Cheers,
>
> Sébastien
>

Andrew Mathas

unread,
Feb 9, 2013, 7:05:02 AM2/9/13
to sage-...@googlegroups.com, sage-comb...@googlegroups.com, Nicolas M. Thiery
Hi Sébastien,

I have been active in the sage-combinat queue for a little over a year. Here is my impression of how the sage-combinat queue works. It could be that how I think that the queue works, or how it is supposed to work, is just wrong but to summarise I think that the queue currently works in exactly the way that you would like it to -- it's just that (to my knowledge) this isn't written down clearly anywhere.


Principle #1 : The priority (ordering) of patches to be merged into Sage should be given to the next patch which is ready and have positive review.

To me the main disadvantage of Sage-Combinat queue is that it does not tend to respect Principle #1.

My understanding is that this is how the queue works: when a patch is ready for review then you move it to the top of the queue. The problem, I think, is that this is unofficial policy. Certainly, when I first started adding patches to the queue I had no idea how it worked and I was almost afraid to move things up and down in the queue least I broke something -- and, to be fair, until I understood how the queue work it was probably better for everyone that I didn't move my patches without some care.
 
Principle #3 : When someone creates a patch in the sage-combinat queue, he or she will put the patch somewhere it does not disturb others, so somewhere it does not create conflicts with other patches. That means the patch will most probably be added after related patches susceptible to conflicts.

Most of the time new patches are added to bottom of the queue, precisely so that they don't create conflicts with existing patches. Occasionally I have seen patches added higher up the queue, but they have all been patches which were expected  to be ready for review quickly.
 
The consequence of Principle #3 is that the creator of a patch in the sage-combinat queue accept that *the time before inclusion of its own patch into Sage is larger than the time of inclusion of all patches it depends on*. Moreover, those patches depends on others which depends on others and so on...

I don't think that this is a consequence of #3 and nor is it the way that the queue is supposed to work: when your patch is ready to review you should move it (close to) the top of the queue. In fact, if you look this os documented in the series file where it says:

# Patches needing review
#
# Prerequisites before putting a patch in this section:
#  - Patch with ticket on trac marked as needs review
#  - Patch uploaded or linked from trac
#  - All tests pass (at least those that were likely to get broken)

 My interpretation of this is that patches should be moved to the "Patches needing review" section of the queue/series file when they are ready.

Definition #4 - "Slow patch" : A patch is slow if it stays on the "needs work" or "needs review" status for a long time.

The problem is that a certain patch might depend on a slow patch, so that it can be considered as a slow patch even if it is during a period of time while you are promptly working on Sage.

 If your patch really does depend on a slow patch then you have to wait for it. If, on the other hand, the two patches don't commute, but you are not really relying on the patch you can move your patch past the slow patch and ask the author to rebase their patch on yours. If your patch really does depend on the other patch then you have to convince the to finalise it or -- as I did once because the author busy doing other things -- take over the patch and finish it.

I understand that a group of developpers working on similar research subject might work on similar patches/files and are suceptible to conflicts. Thus, it might be a good idea to share a common queue to "anticipate and see conflicts in advance". For example : the sage-combinat queue. But, this tends to respect Principle #3 which violates Principle #1.

As I just said, I disagree on this one. If your patch really does depend o another patch it can sometimes slow things down a little but usually it doesn't. On the other hand, when I hit true dependence it can be a hassle but it is also great that other people are writing thign which I can build on as it would be a pain to write all of this myself from scratch,
 

Another reason which tends to violate Principle #1 is that changing the series file (changing the order of patches) takes a non negligeable amount of energy so that you might prefer to keep the ordering as it is and defined following Principle #3.

Indeed, testing if a patch commutes with others is not that fast to check : one must unapply the patches, change the series file, reapply all the patches, run sage -br (with hundreds of modified files!!!) Then, once the commutativity is proved, one must recheck that the patch still work before putting it under review (unapply patches to go back to you patch, sage -br (with hundreds of modified files again!!!!). You might after build the documentation doing sage -docbuild reference html (with again hundreds of modified files again!!!)

I haven't found this to be so painful. I move my patch to the place in the queue where I want it. I then try to reapply the queue. If something beaks I'll rebase the patch if it is obvious, especially if the patch is a "slow patch" (I'm never entirely comfortable rebasing other people's patches as officially this is frowned upon but it's easy and it keeps the queue healthy I do it and hoped not be get flamed. If the rebase looks too tricky then I put a guard on the patch. Once the queue applies I check to see if sage builds and usually it does. Once or twice I have had to track down a patch that is breaking the queue, but the error messages combined with a grep of the patch directory often identifies the culprit quite quickly.

Once a horribly broke the queue. When it became clear that I couldn't fix it quickly I posted on sage-combnat for suggestions ad the issue was soon resolved.

I agree that it would be good to make this process more efficient. Some are saying that the move to git will fix this (this is not reallty clear as some of this posts make it sound like git will even write the code for us). There was also a recent suggestion to have different "themes" in the combinat queue. Perhaps this would be an improvement.

By in large, however, I think that the currently workflow is OK. If anything, once you get use to using hg, I think that the main issue is the (volunteer) review process. So far this has been ok for me (that Travis!), but I have a few patches sitting off the queue which may not be so lucky.

Cheers,
Andrew


Reply all
Reply to author
Forward
0 new messages