Plans for release

15 views
Skip to first unread message

Andy Ray Terrel

unread,
Apr 1, 2011, 4:44:07 AM4/1/11
to sympy
Are there any plans for releasing soon? It been a year and I'd like
to release my library, which only works on dev.

-- Andy

Vinzent Steinberg

unread,
Apr 1, 2011, 9:56:53 AM4/1/11
to sympy
On Apr 1, 10:44 am, Andy Ray Terrel <andy.ter...@gmail.com> wrote:
> Are there any plans for releasing soon?  It been a year and I'd like
> to release my library, which only works on dev.

I think we wanted to get the new polys in before the next release. But
we probably could also do a release before this.

Vinzent

Ronan Lamy

unread,
Apr 1, 2011, 2:26:09 PM4/1/11
to sy...@googlegroups.com
Le vendredi 01 avril 2011 � 06:56 -0700, Vinzent Steinberg a �crit :

I don't think we can do a release before the end of GSoC proposals
review, but we should probably do it as soon as possible afterwards -
hopefully with polys, but without it if it's not ready yet. Some time in
the first half of May seems reasonable, and would be a good "community
bonding" exercise for the GSoC students. What do you think, Aaron?

Mateusz Paprocki

unread,
Apr 1, 2011, 3:12:43 PM4/1/11
to sy...@googlegroups.com
Hi,

On 1 April 2011 11:26, Ronan Lamy <ronan...@gmail.com> wrote:

Le vendredi 01 avril 2011 à 06:56 -0700, Vinzent Steinberg a écrit :
> On Apr 1, 10:44 am, Andy Ray Terrel <andy.ter...@gmail.com> wrote:
> > Are there any plans for releasing soon?  It been a year and I'd like
> > to release my library, which only works on dev.
>

We should have released ages ago and it would be great to do it at some point.
 
> I think we wanted to get the new polys in before the next release. But
> we probably could also do a release before this.

I don't think we can do a release before the end of GSoC proposals
review, but we should probably do it as soon as possible afterwards -
hopefully with polys, but without it if it's not ready yet. Some time in
the first half of May seems reasonable, and would be a good "community
bonding" exercise for the GSoC students. What do you think, Aaron?

polys have been ready for three months now.
 

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To post to this group, send email to sy...@googlegroups.com.
To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sympy?hl=en.


Mateusz

Ronan Lamy

unread,
Apr 1, 2011, 3:59:46 PM4/1/11
to sy...@googlegroups.com
Le vendredi 01 avril 2011 � 12:12 -0700, Mateusz Paprocki a �crit :

> Hi,
>
> On 1 April 2011 11:26, Ronan Lamy <ronan...@gmail.com> wrote:
> Le vendredi 01 avril 2011 � 06:56 -0700, Vinzent Steinberg a
> �crit :

> > On Apr 1, 10:44 am, Andy Ray Terrel <andy.ter...@gmail.com>
> wrote:
> > > Are there any plans for releasing soon? It been a year
> and I'd like
> > > to release my library, which only works on dev.
> >
>
>
>
> We should have released ages ago and it would be great to do it at
> some point.
>
> > I think we wanted to get the new polys in before the next
> release. But
> > we probably could also do a release before this.
>
>
> I don't think we can do a release before the end of GSoC
> proposals
> review, but we should probably do it as soon as possible
> afterwards -
> hopefully with polys, but without it if it's not ready yet.
> Some time in
> the first half of May seems reasonable, and would be a good
> "community
> bonding" exercise for the GSoC students. What do you think,
> Aaron?
>
>
> polys have been ready for three months now.
>
Well, no. They've never been through review.


Aaron S. Meurer

unread,
Apr 1, 2011, 4:36:33 PM4/1/11
to sy...@googlegroups.com

On Apr 1, 2011, at 1:59 PM, Ronan Lamy wrote:

> Le vendredi 01 avril 2011 à 12:12 -0700, Mateusz Paprocki a écrit :
>> Hi,
>>
>> On 1 April 2011 11:26, Ronan Lamy <ronan...@gmail.com> wrote:

>> Le vendredi 01 avril 2011 à 06:56 -0700, Vinzent Steinberg a
>> écrit :


>>> On Apr 1, 10:44 am, Andy Ray Terrel <andy.ter...@gmail.com>
>> wrote:
>>>> Are there any plans for releasing soon? It been a year
>> and I'd like
>>>> to release my library, which only works on dev.
>>>
>>
>>
>>
>> We should have released ages ago and it would be great to do it at
>> some point.
>>
>>> I think we wanted to get the new polys in before the next
>> release. But
>>> we probably could also do a release before this.
>>
>>
>> I don't think we can do a release before the end of GSoC
>> proposals
>> review, but we should probably do it as soon as possible
>> afterwards -
>> hopefully with polys, but without it if it's not ready yet.
>> Some time in
>> the first half of May seems reasonable, and would be a good
>> "community
>> bonding" exercise for the GSoC students. What do you think,
>> Aaron?

Yes, absolutely. The GSoC stuff has pushed the release (and everything else) to the back burner, but after we accept students, things should cool down a little bit. At that point, releasing will be the top priority. And I think it would be great if we could get some of the GSoC students to help. Tt should make a good community bonding exercise. We just need help with a few critical patches, but mostly with reviewing stuff, which is something that they will need to learn to do.

By the way, we still have a few issues blocking the release. See all the Release0.7.0 issues at http://code.google.com/p/sympy/issues/list (they are sorted at the top of the list by default). Most of those are part of polys12, but a handful are not.

>>
>>
>> polys have been ready for three months now.
>>
> Well, no. They've never been through review.

Well, clearly we have a misunderstanding here. There are definitely a few things in polys12 that are blocking merging (like the sum() thing). But there have also been some other comments (mainly by Ronan) throughout your branch. For example, some people have had some problems with Pure. I think these comments are all on GitHub (correct me if I am wrong).

We have make several attempts to fix these on our own, with various success. For example, https://github.com/sympy/sympy/pull/120. It would probably be best if you could fix these things yourself, since for example, you will know best how to manage the conflicts when merging/rebasing over master. But in the absence of work from you, we have attepted to do it ourselves, so we can still release with the branch merged in at some point.

Aaron Meurer

Mateusz Paprocki

unread,
Apr 1, 2011, 5:36:28 PM4/1/11
to sy...@googlegroups.com
Hi,

I'm sure we don't understand each other at all. Lets stick to sum(). It didn't work, I made it work and didn't have the luxury to discuss the possible options for a year. The solution isn't optimal? Yes, it isn't, but it works and this is what I needed. Need a better solution? Apply a patch with a better solution on top of my branch and you are done.
 
 But there have also been some other comments (mainly by Ronan) throughout your branch. For example, some people have had some problems with Pure.  I think these comments are all on GitHub (correct me if I am wrong).

This looked to me more like a wish list. Anyway, no patches followed.
 

We have make several attempts to fix these on our own, with various success.  For example, https://github.com/sympy/sympy/pull/120.  It would probably be best if you could fix these things yourself, since for example, you will know best how to manage the conflicts when merging/rebasing  over master.  But in the absence of work from you, we have attepted to do it ourselves, so we can still release with the branch merged in at some point.

I rebased my branch twice already for a release that never happened and it seems I only wasted my time for this. Do you expect that three months after last rebase it will still work out of the box with the most recent master? I'm sure that if this branch was merged, all the tiny details would have been fixed by now. This branch is not 10 patches or something that should be supposed to work perfectly upon merge. This is a branch consists of 500+ commits touching most significant areas of SymPy and it will require time, effort and support of the community to settle in. How do you expect to have support of the community (!= two or three people) in some branch. I don't need theoretical discussions about it, I need people who will use it on daily basis and report issues and inconveniences (like you did Aaron when working on implementation of Risch algorithm, which was valuable and helpful and resulted in significant improvements to my implementation of polynomial manipulation module).
 

Aaron Meurer

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To post to this group, send email to sy...@googlegroups.com.
To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sympy?hl=en.


Mateusz

Aaron S. Meurer

unread,
Apr 1, 2011, 6:29:05 PM4/1/11
to sy...@googlegroups.com
Well, your solution is no longer even necessary, because we have completely renamed sum() to summation() in master.  That one is actually easy to fix.  We just need to revert your commit (or rebase it out).  It doesn't seem like you actually used it anywhere, so that won't cause any problems.

 
 But there have also been some other comments (mainly by Ronan) throughout your branch. For example, some people have had some problems with Pure.  I think these comments are all on GitHub (correct me if I am wrong).

This looked to me more like a wish list. Anyway, no patches followed.

Well, I do have to agree that Ronan was not the clearest about what he wanted to be done (in fact, I am not not entirely sure what needs to be done).

 

We have make several attempts to fix these on our own, with various success.  For example, https://github.com/sympy/sympy/pull/120.  It would probably be best if you could fix these things yourself, since for example, you will know best how to manage the conflicts when merging/rebasing  over master.  But in the absence of work from you, we have attepted to do it ourselves, so we can still release with the branch merged in at some point.

I rebased my branch twice already for a release that never happened and it seems I only wasted my time for this. Do you expect that three months after last rebase it will still work out of the box with the most recent master? I'm sure that if this branch was merged, all the tiny details would have been fixed by now. This branch is not 10 patches or something that should be supposed to work perfectly upon merge. This is a branch consists of 500+ commits touching most significant areas of SymPy and it will require time, effort and support of the community to settle in. How do you expect to have support of the community (!= two or three people) in some branch. I don't need theoretical discussions about it, I need people who will use it on daily basis and report issues and inconveniences (like you did Aaron when working on implementation of Risch algorithm, which was valuable and helpful and resulted in significant improvements to my implementation of polynomial manipulation module).

This is why I have suggested first fixing all review conflicts (like the sum thing) in your branch, and then merging/rebasing after that, so we can immediately merge with master.

I can definitely understand your pain here.  polys12 is a very large branch, and it is very difficult to maintain separate from master.  

So I think what we need to do is come up with a concrete list of things that have to be fixed for the merge, which are separate from theoretical things relating to the implementation (especially internal implementation, which can easily change in the future).  Personally, I am +1 on the branch except for the sum() commit and any test failures that there might be (I have reviewed each commit as they came out over the summer), but others, in particular Ronan, have had other qualms. 

What I know of is at least this:

- The commit adding the built-in functionality to sum() should be completely reverted or rebased out.  This won't even apply over the latest master anyway, because of the sum() => summation() rename.  Others' branches have shown that this doesn't actually break anything, but even if it did, it would be trivial to fix.

- All tests should pass in all Pythons (2.4-2.7) and ground types.  Of course, this needs to hold after the merge/rebase with master.  But again, this is 90% of the time very easy to do (it usually just involves changing the output in some ode test or something).

- There has been a lot of talk/work regarding Pure.  We are not sure if your implementation is the best, or if it is general enough (for example, there is only one Pure symbol).  From what I remember, this is not used in very many places, only minpoly() and maybe somewhere else too.  Would it be best to just temporarily replace Pure with a dummy symbol until we get this worked out.  Note that I don't remember everything that was wrong with Pure, so maybe others can comment on it more (or you can look through the comments on the pull requests).

- The keep_sign thing needs to be fixed, but I think this is something that we could fix after your branch gets in (it isn't necessarily blocking the merge, just the release).

- How do you feel about Chris's p12 branch (see the pull request linked to in my previous reply)?  He has basically done this all (rebased over master, rebased out the sum commit, and added some commits fixing test failures and changing Pure to be a dummy).  I think there are still some test failures there, and the commit messages need to be cleaned up, but if you are fine with it, we can finish that and get it in.

What other concrete problems do others (Ronan) have with the branch?  I really want to get this in already, because I am tired of referring people to polys12 for some functionality, and I also really want to get a release out.

Aaron Meurer

Ronan Lamy

unread,
Apr 1, 2011, 9:03:51 PM4/1/11
to sy...@googlegroups.com
Le vendredi 01 avril 2011 � 16:29 -0600, Aaron S. Meurer a �crit :

>
> On Apr 1, 2011, at 3:36 PM, Mateusz Paprocki wrote:
>
> > Hi,
> >
> > On 1 April 2011 13:36, Aaron S. Meurer <asme...@gmail.com> wrote:
> >
> >
> > On Apr 1, 2011, at 1:59 PM, Ronan Lamy wrote:
> >
> > > Le vendredi 01 avril 2011 � 12:12 -0700, Mateusz Paprocki
> > a �crit :

> > >> Hi,
> > >>
> > >> On 1 April 2011 11:26, Ronan Lamy <ronan...@gmail.com>
> > wrote:
> > >> Le vendredi 01 avril 2011 � 06:56 -0700, Vinzent
> > Steinberg a
> > >> �crit :

Standard procedure is that the submitter of the pull request fixes
things, not the reviewer. You never actually sent any pull request for
this, which is part of the problem, but you did say you wanted your
branch to be merged, so I supposed it was the same thing.

>
> Well, I do have to agree that Ronan was not the clearest about what he
> wanted to be done (in fact, I am not not entirely sure what needs to
> be done).

I am not entirely sure what needs to be done either, because the branch
is way too big to be reviewed in one go and because it has never been
possible to discuss it in a forum convenient for review (i.e. a pull
request initiated by Mateusz).

I don't understand why you never agreed to do anything that would make
the branch smaller, even though the problem is precisely that it's too
big. There are commits that could go in today.

Issue 2241 is nearly fixed. After that, it's just a matter of editing a
bunch of doctests.

> - How do you feel about Chris's p12 branch (see the pull request
> linked to in my previous reply)? He has basically done this all
> (rebased over master, rebased out the sum commit, and added some
> commits fixing test failures and changing Pure to be a dummy). I
> think there are still some test failures there, and the commit
> messages need to be cleaned up, but if you are fine with it, we can
> finish that and get it in.
>
>
> What other concrete problems do others (Ronan) have with the branch?
> I really want to get this in already, because I am tired of referring
> people to polys12 for some functionality, and I also really want to
> get a release out.

Everybody is tired of this situation, I guess. Basically, the meta-issue
is that I think that every significant change outside sympy/polys should
be discussed on its own (i.e. as a separate pull request).
Anyway, here's an incomplete list off the top of my head:
* The RootOf constructor is too complicated (cf. issue 51 and other
discussions I can't find ATM)
* Lambda.is_identity shouldn't be implemented (cf issue 2178)
* .find() and .replace() need to be discussed. The proposed interface of
the latter is too complex.
* symbols('xyz') should be deprecated, but not removed outright (I don't
think I ever mentioned that one before)
* exprtools adds an additional mess to a messy situation in Add and Mul
* Most additions to sympy.utilities.iterables reimplement functions in
itertools.


Aaron S. Meurer

unread,
Apr 1, 2011, 9:51:42 PM4/1/11
to sy...@googlegroups.com

On Apr 1, 2011, at 7:03 PM, Ronan Lamy wrote:

> Le vendredi 01 avril 2011 à 16:29 -0600, Aaron S. Meurer a écrit :
>>
>> On Apr 1, 2011, at 3:36 PM, Mateusz Paprocki wrote:
>>
>>> Hi,
>>>
>>> On 1 April 2011 13:36, Aaron S. Meurer <asme...@gmail.com> wrote:
>>>
>>>
>>> On Apr 1, 2011, at 1:59 PM, Ronan Lamy wrote:
>>>

>>>> Le vendredi 01 avril 2011 à 12:12 -0700, Mateusz Paprocki
>>> a écrit :


>>>>> Hi,
>>>>>
>>>>> On 1 April 2011 11:26, Ronan Lamy <ronan...@gmail.com>
>>> wrote:

>>>>> Le vendredi 01 avril 2011 à 06:56 -0700, Vinzent
>>> Steinberg a
>>>>> écrit :

I agree with this. Mateusz should create a pull request for polys12.

If Mateusz is willing to do something like this, I am fine. The problem is that jumbling around a huge branch like that can get messy fast, and also we need to take care not to lose any commits.

I agree, although actually I am thinking of different things than Ronan (my main issue is the use of None to get the real roots, when RootOf(real=True) would be better imo).


> * Lambda.is_identity shouldn't be implemented (cf issue 2178)
> * .find() and .replace() need to be discussed. The proposed interface of
> the latter is too complex.

What is wrong with this? I though it was just expr.replace(old, new), where it uses .match() so that it is smart if it uses Wilds. Anyway, if you don't like something, you have to be specific about what is wrong with it.

> * symbols('xyz') should be deprecated, but not removed outright (I don't
> think I ever mentioned that one before)

I don't think this will work. symbols('xyz') should create one symbol (Symbol('xyz')). Trying to deprecate this would just keep the bad interface problem there (see the discussion in issue 1977).

> * exprtools adds an additional mess to a messy situation in Add and Mul

Is this actually used anywhere? If not, we could just hold out on it pretty easily. Or we could add it as a (currently unused) prototype.

> * Most additions to sympy.utilities.iterables reimplement functions in
> itertools.

Maybe this one is being picky. I don't care either way with it.

Aaron Meurer

Mateusz Paprocki

unread,
Apr 1, 2011, 11:02:41 PM4/1/11
to sy...@googlegroups.com
Hi,

And this is my point. The more we will cut and slice this branch, the more extra work we will have.
Wouldn't be it more convenient to use the new code in master and over time converge to an API that all will like? It's easier to discuss things over and over again than implement, experiment, refine (I prefer the later). Maybe the problem is that we work on master, not some development branch where we could experiment in public and converge to widely accepted solutions. Instead we exchange many ideas (that's not wrong) and often do nothing about them (sounds like sum() issue). SymPy grown very fast because we accepted partial solutions and allowed them to grow (or to die in a natural way). If we took current approach regarding merge of my branch in which I implemented heuristic Risch algorithm, then we wound't have symbolic integrator till now and I'm convinced that Aaron wouldn't have been working on the other Risch algorithm (of course I may be wrong). Over the years (yes this implementation is quite old) I realized that heurisch() have been a huge mess (my fault in this), but it have been there and many people used it and some of them improved it (or at least suggested improvements). This is also why I would like Aaron to merge his integration3 branch as soon as possible, so that people could start using his implementation and contribute to it (even if there are very many NotImplementedError cases).
 


> * Lambda.is_identity shouldn't be implemented (cf issue 2178)
> * .find() and .replace() need to be discussed. The proposed interface of
> the latter is too complex.

What is wrong with this?  I though it was just expr.replace(old, new), where it uses .match() so that it is smart if it uses Wilds.  Anyway, if you don't like something, you have to be specific about what is wrong with it.

We agreed on this in an issue. If there are better ideas (I'm sure there are), why don't allow this to go in, and start from there?
 

> * symbols('xyz') should be deprecated, but not removed outright (I don't
> think I ever mentioned that one before)

I don't think this will work.  symbols('xyz') should create one symbol (Symbol('xyz')).  Trying to deprecate this would just keep the bad interface problem there (see the discussion in issue 1977).

Now I'm confused. Was 0.7.0 supposed to be this great release where backwards incompatible changes to the API should have been made? I'm pretty sure this was the purpose of this release. Anyway, deprecating this and keeping old syntax is a very bad idea.
 

> * exprtools adds an additional mess to a messy situation in Add and Mul

Is this actually used anywhere?  If not, we could just hold out on it pretty easily. Or we could add it as a (currently unused) prototype.

Actually it's a very important part of this branch and is used in together(), factor(), etc., to provide generalized and fast approach to manipulation of commutative expressions. This has nothing to do (directly) with Add and Mul, and if you don't understand this, please don't call this messy. I placed it in the core, because I'm sure this will be a significant part of the future core.
 

> * Most additions to sympy.utilities.iterables reimplement functions in
> itertools.

Maybe this one is being picky.  I don't care either way with it.

Exactly. Doesn't this work? It does, tests pass. Is it redundant? At least partially, but when implementing those functions I didn't have any clue  about this. Does this interfere with the merge? I don't think so and if we merged this three months ago, this could have been easily refined before the release.

So, the only real issue might be to pass the whole test suite after the merge. I will point this out again: three months ago *all* tests, doctests and documentation tests passed for most important configurations. Now they obviously won't (although Chris' branch may help here).

btw. Going through recent commits I found this:


It seems that our review machinery doesn't work that good anyway ("A" isn't a very informative commit message, unless I miss a very important point here). I pointed this out just to show that sometimes we are very picky about commits and sometimes we push this kind of stuff (I'm referring only to commit message right now, not the commit it self). Maybe average of those two extremes would be a working approach? (i.e. don't scare developers or potential developers, but also don't allow to incorrect/misleading stuff to go in).
 

Aaron Meurer

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To post to this group, send email to sy...@googlegroups.com.
To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sympy?hl=en.


Mateusz

Andy Ray Terrel

unread,
Apr 2, 2011, 2:36:33 AM4/2/11
to sy...@googlegroups.com, Mateusz Paprocki
For better or for worse, SymPy has always been a place where patches
that meet a minimum standard are pushed in. In the interest of not
going 1.5 years between releases, I suggest we create issues for what
needs to be done and push Mateusz's code.

-- Andy

Vinzent Steinberg

unread,
Apr 2, 2011, 8:30:01 AM4/2/11
to sympy
On 2 Apr., 03:03, Ronan Lamy <ronan.l...@gmail.com> wrote:
> * The RootOf constructor is too complicated (cf. issue 51 and other
> discussions I can't find ATM)
> * Lambda.is_identity shouldn't be implemented (cf issue 2178)

This is easy I guess. Are you thinking of something like 'mylambda is
S.Identity'?

> * .find() and .replace() need to be discussed. The proposed interface of
the latter is too complex.

Maybe. I think polys could work without this. But to be honest I don't
see your point. Achieving replace()'s functionality in current sympy
is counter-intuitive IMHO.

> * symbols('xyz') should be deprecated, but not removed outright (I don't
> think I ever mentioned that one before)

I disagree. It has been deprecated for a very long time in the
docstring. There was no DeprecationWarning, this was maybe a mistake.
In any case, 0.7 is intended to break compatibility. So I'm +1 on
fixing this.

> * exprtools adds an additional mess to a messy situation in Add and Mul

Please be specific.

> * Most additions to sympy.utilities.iterables reimplement functions in
itertools.

itertools depends AFAIK a lot on the specific version of Python, so
I'm fine with this, especially if we still support Python 2.4. This
can be easily fixed later.

Mateusz wrote:
> "A" isn't a very informative commit message

This was not intended, I think Aaron pushed the wrong branch by
accident.

However, I would agree that code review nowadays is pickier than in
the past, looking at some code in sympy.

> Do you expect that three months after
> last rebase it will still work out of the box with the most recent master?

Please note that the p12 [1] branch has only one failing test (if you
merge my commit trivially fixing failures), so this is currently not
really an issue.

Let's summarize anything that needs to be done - as specific as
possible - and do it.

Vinzent


[1] https://github.com/sympy/sympy/pull/120

smichr

unread,
Apr 2, 2011, 9:04:16 AM4/2/11
to sympy
Though p12 might have some kinks to work out I think it's better to
get it in place so the things that are good can get exercised and
those that are not can get excised without causing a big headache. I
also believe it will be best to get this in before the GSoC gets to
work because every bit of work that interferes with p12 makes for more
work in doing the rebase.

I've already spent about 30 hours rebasing and going over commits and
feel pretty good about the integrity of p12 as a whole. Given Ondrej's
reminder that bisection can use the --skip option which results in a
range of commits being returned as the faulty range I am less stressed
about smoothing everything out in p12. Though tests will not all pass
in each commit, all commits allow `from sympy import *` to be
executed.

I'm heartily in favor of pushing this asap and am willing to help "get
it done" as Vinzent suggests.

Chris

smichr

unread,
Apr 2, 2011, 9:07:46 AM4/2/11
to sympy
One thing that we could do as a community -- GSoC included -- would be
to get uncovered lines covered. I can repost those if there is
interest. I'm talking about lines that were modified or added by
polys12 that are uncovered, not sympy lines in general.

Chris

smichr

unread,
Apr 2, 2011, 9:18:05 AM4/2/11
to sympy
What is a good way to collaborate on the TODO list? Can we use a wiki
or git-widget?

Here's the first issue that I would raise:

TODO:
1) as_coeff_Mul is very similar to as_coeff_mul - should we name that
method .mul0() and add another named .add0()? - this isn't a show
stopper, but if we could get it cleared up before adding it I think it
would be good. There will be lag time between when we push and when
people start using things...but if this is going in for 0.7 it would
be nice to not break compatibility soon after by changing as_coeff_Mul
to something else after 0.7.

Ronan Lamy

unread,
Apr 2, 2011, 2:25:42 PM4/2/11
to sy...@googlegroups.com
Le vendredi 01 avril 2011 � 20:02 -0700, Mateusz Paprocki a �crit :

> Hi,
>
> On 1 April 2011 18:51, Aaron S. Meurer <asme...@gmail.com> wrote:
>
>
> On Apr 1, 2011, at 7:03 PM, Ronan Lamy wrote:
>
> > Le vendredi 01 avril 2011 � 16:29 -0600, Aaron S. Meurer a
> �crit :

> >>
> >> On Apr 1, 2011, at 3:36 PM, Mateusz Paprocki wrote:
> >>
> >>> Hi,
> >>>
> >>> On 1 April 2011 13:36, Aaron S. Meurer
> <asme...@gmail.com> wrote:
> >>>
> >>>
> >>> But there have also been some other comments
> (mainly by
> >>> Ronan) throughout your branch. For example, some
> people have
> >>> had some problems with Pure. I think these
> comments are all
> >>> on GitHub (correct me if I am wrong).
> >>>
> >>>
> >>> This looked to me more like a wish list. Anyway, no
> patches
> >>> followed.
> >
> > Standard procedure is that the submitter of the pull request
> fixes
> > things, not the reviewer. You never actually sent any pull
> request for
> > this, which is part of the problem, but you did say you
> wanted your
> > branch to be merged, so I supposed it was the same thing.
>
>
> I agree with this. Mateusz should create a pull request for
> polys12.
>
To be clear, I think Mateusz should create several pull requests, if
only for the reason that the GitHub interface can't deal with pull
request with more than 250 commits.

> >>
> >> I can definitely understand your pain here. polys12 is a
> very large
> >> branch, and it is very difficult to maintain separate from
> master.
> >>
> > I don't understand why you never agreed to do anything that
> would make
> > the branch smaller, even though the problem is precisely
> that it's too
> > big. There are commits that could go in today.
>
>
> If Mateusz is willing to do something like this, I am fine.
> The problem is that jumbling around a huge branch like that
> can get messy fast, and also we need to take care not to lose
> any commits.
>
>
> And this is my point. The more we will cut and slice this branch, the
> more extra work we will have.

If we push a prefix of the branch in, it won't create more work.

No, breaking APIs all the time wouldn't be convenient for our users. And
it's actually harder to explain one's choices and to make compromises
than to work on one's own. Experimenting is fine, and more than that,
it's often the best way of making progress on a difficult problem, but
it shouldn't be done in master - because we try to keep in a
quasi-releasable state, and because, as a consequence of that and of the
excessive interval since our last release, quite a few people appear to
rely on it.
What we could do is to have "official" development branches on the main
repo. These branches would be meant to be merged back into master at
some point so they would be subjected to most of the same review
requirements as master - clean commits, cleanly formatted code, (nearly)
all tests pass on (nearly) all commits for all supported platforms - but
considerations of backwards compatibility and interface design could be
relaxed.

>
>
> > * Lambda.is_identity shouldn't be implemented (cf issue
> 2178)
> > * .find() and .replace() need to be discussed. The proposed
> interface of
> > the latter is too complex.
>
>
> What is wrong with this? I though it was just
> expr.replace(old, new), where it uses .match() so that it is
> smart if it uses Wilds. Anyway, if you don't like something,
> you have to be specific about what is wrong with it.
>
>
> We agreed on this in an issue. If there are better ideas (I'm sure
> there are), why don't allow this to go in, and start from there?
>
>
> > * symbols('xyz') should be deprecated, but not removed
> outright (I don't
> > think I ever mentioned that one before)
>
>
> I don't think this will work. symbols('xyz') should create
> one symbol (Symbol('xyz')). Trying to deprecate this would
> just keep the bad interface problem there (see the discussion
> in issue 1977).
>
>
> Now I'm confused. Was 0.7.0 supposed to be this great release where
> backwards incompatible changes to the API should have been made? I'm
> pretty sure this was the purpose of this release. Anyway, deprecating
> this and keeping old syntax is a very bad idea.
>

Let's discuss it in http://code.google.com/p/sympy/issues/detail?id=1977

>
> > * exprtools adds an additional mess to a messy situation in
> Add and Mul
>
>
> Is this actually used anywhere? If not, we could just hold
> out on it pretty easily. Or we could add it as a (currently
> unused) prototype.
>
>
> Actually it's a very important part of this branch and is used in
> together(), factor(), etc., to provide generalized and fast approach
> to manipulation of commutative expressions. This has nothing to do
> (directly) with Add and Mul, and if you don't understand this, please
> don't call this messy. I placed it in the core, because I'm sure this
> will be a significant part of the future core.
>

If you discussed it and explained your plans, maybe I would understand.
For now, I don't understand and I can only disagree. In particular, I
fail to understand how handling commutative expressions can be unrelated
to Add and Mul. Aren't Add and, in the common case, Mul commutative
expressions?

>
> > * Most additions to sympy.utilities.iterables reimplement
> functions in
> > itertools.
>
>
> Maybe this one is being picky. I don't care either way with
> it.
>
>
> Exactly. Doesn't this work? It does, tests pass. Is it redundant? At
> least partially, but when implementing those functions I didn't have
> any clue about this. Does this interfere with the merge? I don't
> think so and if we merged this three months ago, this could have been
> easily refined before the release.
>

Users don't care about your state of mind. They do care about having to
learn new interfaces only for sympy. They do care when stuff added in
one release is broken in the next. But if this had been the only
problem, I probably wouldn't have minded doing the merge three months
ago, even though we already wanted to release ASAP.


>
> So, the only real issue might be to pass the whole test suite after
> the merge. I will point this out again: three months ago *all* tests,
> doctests and documentation tests passed for most important
> configurations. Now they obviously won't (although Chris' branch may
> help here).
>
>
> btw. Going through recent commits I found this:
>
>
> https://github.com/sympy/sympy/commit/7a9c9b1144306245dd65d41f1e2883c760321594
>
>
> It seems that our review machinery doesn't work that good anyway ("A"
> isn't a very informative commit message, unless I miss a very
> important point here). I pointed this out just to show that sometimes
> we are very picky about commits and sometimes we push this kind of
> stuff (I'm referring only to commit message right now, not the commit
> it self). Maybe average of those two extremes would be a working
> approach? (i.e. don't scare developers or potential developers, but
> also don't allow to incorrect/misleading stuff to go in).
>
>
> Aaron Meurer
>
>
> --
> You received this message because you are subscribed to the
> Google Groups "sympy" group.
> To post to this group, send email to sy...@googlegroups.com.
> To unsubscribe from this group, send email to sympy

> +unsub...@googlegroups.com.


> For more options, visit this group at
> http://groups.google.com/group/sympy?hl=en.
>
>
>
> Mateusz
>
> --
> You received this message because you are subscribed to the Google
> Groups "sympy" group.
> To post to this group, send email to sy...@googlegroups.com.
> To unsubscribe from this group, send email to sympy

> +unsub...@googlegroups.com.

Aaron S. Meurer

unread,
Apr 2, 2011, 4:24:17 PM4/2/11
to sy...@googlegroups.com
Well, heurisch() is slow but it has been very successful.  For example, it can solve the integral in Wikipedia that no other non-Axiom CAS can solve.   I am becoming convinced that the slowness this is just the nature of the algorithm, because it produces candidate integrals that are very large even for relatively small integrands. 

To me, there are three things that come to mind that should always be done right the first time (i.e., before any merge).  The first is the user interface, since changing this in the future breaks backward compatibility, especially if there is a release in the meanwhile. The second thing is something that is not an issue in polys12, which is that no branch should be pushed in with a known wrong result.  You have been really good at fixing any wrong result that I have reported (like when you worked on RootSum.diff), but have not been so open on considering suggestions regarding the interface for already existing functions (like the RootOf issue).  The third thing is of course test failures, which have already been talked about here.

Regarding my branch, my plan was to not merge integration3 until I finished the algorithm, so that you could have a complete implementation of the transcendental algorithm in integrate(), but I now see that this is wrong. It is better to merge it in with the NotImplementedErrors and then improve it.  That way, people will try it out.  I am more hesitant to merge integration3 because I am constantly finding bugs in it.  It needs to be tested more. N so much that it needs more tests, though this is true in some cases, but it just needs more people to try it out.  But I now see that the best way to do this is to push it out to the official repo, where everyone will be using it.

Unfortunately, integration3 is blocked pretty heavily on issue 2026 (exact or atomic substitution).  I had to completely disable algebraic substitution in exp to make the preparser algorithm work, but this makes some limit tests break.  There is no easy workaround, it just needs to be implemented.  

 


> * Lambda.is_identity shouldn't be implemented (cf issue 2178)
> * .find() and .replace() need to be discussed. The proposed interface of
> the latter is too complex.

What is wrong with this?  I though it was just expr.replace(old, new), where it uses .match() so that it is smart if it uses Wilds.  Anyway, if you don't like something, you have to be specific about what is wrong with it.

We agreed on this in an issue. If there are better ideas (I'm sure there are), why don't allow this to go in, and start from there?
 

> * symbols('xyz') should be deprecated, but not removed outright (I don't
> think I ever mentioned that one before)

I don't think this will work.  symbols('xyz') should create one symbol (Symbol('xyz')).  Trying to deprecate this would just keep the bad interface problem there (see the discussion in issue 1977).

Now I'm confused. Was 0.7.0 supposed to be this great release where backwards incompatible changes to the API should have been made? I'm pretty sure this was the purpose of this release. Anyway, deprecating this and keeping old syntax is a very bad idea.

I agree 100% here.

 

> * exprtools adds an additional mess to a messy situation in Add and Mul

Is this actually used anywhere?  If not, we could just hold out on it pretty easily. Or we could add it as a (currently unused) prototype.

Actually it's a very important part of this branch and is used in together(), factor(), etc., to provide generalized and fast approach to manipulation of commutative expressions. This has nothing to do (directly) with Add and Mul, and if you don't understand this, please don't call this messy. I placed it in the core, because I'm sure this will be a significant part of the future core.
 

> * Most additions to sympy.utilities.iterables reimplement functions in
> itertools.

Maybe this one is being picky.  I don't care either way with it.

Exactly. Doesn't this work? It does, tests pass. Is it redundant? At least partially, but when implementing those functions I didn't have any clue  about this. Does this interfere with the merge? I don't think so and if we merged this three months ago, this could have been easily refined before the release.

So, the only real issue might be to pass the whole test suite after the merge. I will point this out again: three months ago *all* tests, doctests and documentation tests passed for most important configurations. Now they obviously won't (although Chris' branch may help here).

btw. Going through recent commits I found this:


It seems that our review machinery doesn't work that good anyway ("A" isn't a very informative commit message, unless I miss a very important point here). I pointed this out just to show that sometimes we are very picky about commits and sometimes we push this kind of stuff (I'm referring only to commit message right now, not the commit it self). Maybe average of those two extremes would be a working approach? (i.e. don't scare developers or potential developers, but also don't allow to incorrect/misleading stuff to go in).

OK, so the "A" commit was my fault, and I take 100% blame for it.  I basically forgot to do a "git fetch" before "git push" (he had fixed it in his branch, but I merged the wrong revision).  So I think this is a bad example (because it was completely accidental).

But I do agree that we let things slide from time to time.  For example, we often do not keep a commit from going in if it has some spacing errors (like a==b+c instead of a == b + c), especially for new contributors.  But like I said above, the user level interface is important to get right the first time, and we do (or at least I do) block commits from going in if I see a problem there, even with the new contributors.  

There are only a few outstanding of these issues in polys12 (RootOf, Pure, Lambda).  The remainder aren't as important, because they are internal (and it seems like you are constantly rewriting the internals anyway).  

Aaron Meurer

Tim Lahey

unread,
Apr 2, 2011, 5:54:38 PM4/2/11
to sy...@googlegroups.com
On Sat, Apr 2, 2011 at 4:24 PM, Aaron S. Meurer <asme...@gmail.com> wrote:
>
> Regarding my branch, my plan was to not merge integration3 until I finished
> the algorithm, so that you could have a complete implementation of the
> transcendental algorithm in integrate(), but I now see that this is wrong.
> It is better to merge it in with the NotImplementedErrors and then improve
> it.  That way, people will try it out.  I am more hesitant to merge
> integration3 because I am constantly finding bugs in it.  It needs to be
> tested more. N so much that it needs more tests, though this is true in some
> cases, but it just needs more people to try it out.  But I now see that the
> best way to do this is to push it out to the official repo, where everyone
> will be using it.
> Unfortunately, integration3 is blocked pretty heavily on issue 2026 (exact
> or atomic substitution).  I had to completely disable algebraic substitution
> in exp to make the preparser algorithm work, but this makes some limit tests
> break.  There is no easy workaround, it just needs to be implemented.

Where can I find your branch? I have the work I started on doing an
integration test suite for Sage that I stopped when I stopped being
interested in it (because of certain changes). If I can find some
time, I can look at converting what I've done to Sympy and finishing
it. It's the Axiom test suite.

Cheers,

Tim.

--
Tim Lahey
PhD Candidate, Systems Design Engineering
University of Waterloo
http://about.me/tjlahey

Aaron S. Meurer

unread,
Apr 2, 2011, 6:03:15 PM4/2/11
to sy...@googlegroups.com
See github.com/asmeurer/sympy/tree/integration3. See also issue 2010.

The function is risch_integrate() (integrate() is the same as in master; you can use it for comparison). If it returns an unevaluated Integral, it means that the integral has been proven to be non-elementary (it will raise NotImplementedError when it hasn't been implemented yet).

Only the transcendental algorithm for exponentials and logarithms has been implemented so far, so no algebraic integrands (like with square roots) or trigonometric integrands will work. This will probably reduce to about a tenth of your test suite.

By the way, in case you didn't know, the answer from any integration function is correct up to an arbitrary constant, so you shouldn't compare the output with some standard answer. Rather, to test for correctness, you should differentiate the result, subtract it from the original, and pass it to cancel() (cancel is sufficient for the simplifications involved here) and see if that goes to 0 (i.e., cancel(expr - risch_integrate(expr, x).diff(x))).

Also, any exception that is not NotImplementedError is a bug.

Aaron Meurer

Tim Lahey

unread,
Apr 2, 2011, 6:22:26 PM4/2/11
to sy...@googlegroups.com
On Sat, Apr 2, 2011 at 6:03 PM, Aaron S. Meurer <asme...@gmail.com> wrote:
>
> By the way, in case you didn't know, the answer from any integration function is correct up to an arbitrary constant, so you shouldn't compare the output with some standard answer.  Rather, to test for correctness, you should differentiate the result, subtract it from the original, and pass it to cancel() (cancel is sufficient for the simplifications involved here) and see if that goes to 0 (i.e., cancel(expr - risch_integrate(expr, x).diff(x))).

I know. I've seen various ways that integrals are equivalent so I've
tested different ways. If one fails, I'll move on to the next one. So,
working with a standard answer is fine. Also, a standard answer allows
one to check themselves that the answer is correct since you could
display Sympy's answer and the standard one. It's also useful for
comparing timings across CAS.

Mateusz Paprocki

unread,
Apr 2, 2011, 7:31:12 PM4/2/11
to sy...@googlegroups.com
Hi,

On 2 April 2011 14:30, Vinzent Steinberg <vinzent....@googlemail.com> wrote:
On 2 Apr., 03:03, Ronan Lamy <ronan.l...@gmail.com> wrote:
> * The RootOf constructor is too complicated (cf. issue 51 and other
> discussions I can't find ATM)
> * Lambda.is_identity shouldn't be implemented (cf issue 2178)

This is easy I guess. Are you thinking of something like 'mylambda is
S.Identity'?

> * .find() and .replace() need to be discussed. The proposed interface of
the latter is too complex.

Maybe. I think polys could work without this. But to be honest I don't
see your point. Achieving replace()'s functionality in current sympy
is counter-intuitive IMHO.

polys can in 98-99% survive without this, but replace() is very handy and was a significant tool when working on the final implementation of RootOf and RootSum. Basically speaking, I don't implement tools I don't use. If the current implementation is insufficient or too complex, then this can be always fixed on top of the current work. I think that it's also useful to have this implementation in the commit history to show that this was the very beginning of this function and even if it is completely replaced with some other implementation in future, then one can always see that this path was already explored and (possibly) there is no point in going that way again (isn't this one of reasons we use an SCM system?).
 

> * symbols('xyz') should be deprecated, but not removed outright (I don't
> think I ever mentioned that one before)

I disagree. It has been deprecated for a very long time in the
docstring. There was no DeprecationWarning, this was maybe a mistake.
In any case, 0.7 is intended to break compatibility. So I'm +1 on
fixing this.

We can try to explore other alternatives here. Maybe we can add some sort of compatibility mode to SymPy, so that by default we get the new behaviour of symbols(), but user could run SymPy with some special flag to tell it that he wants the old behaviour (before he fixes all his scripts that use SymPy). This is only brainstorming, so this idea might be actually completely infeasible or simply wrong.
 

> * exprtools adds an additional mess to a messy situation in Add and Mul

Please be specific.

> * Most additions to sympy.utilities.iterables reimplement functions in
itertools.

itertools depends AFAIK a lot on the specific version of Python, so
I'm fine with this, especially if we still support Python 2.4. This
can be easily fixed later.

It might be too much to say, but I think that Python's standard libraries sometimes aren't very pleasant to work with and a thin wrapper, in this case, over itertools might be a good idea, to make the API more familiar to CAS community (of course obvious redundant code should be removed).
 

Mateusz wrote:
> "A" isn't a very informative commit message

This was not intended, I think Aaron pushed the wrong branch by
accident.

This is what I presume.
 

However, I would agree that code review nowadays is pickier than in
the past, looking at some code in sympy.

> Do you expect that three months after
> last rebase it will still work out of the box with the most recent master?

Please note that the p12 [1] branch has only one failing test (if you
merge my commit trivially fixing failures), so this is currently not
really an issue.

That's very good news. So lets follow Aaron's suggestion about improving commit messages of the new commits and we are done here?
 

Let's summarize anything that needs to be done - as specific as
possible - and do it.

Vinzent


[1] https://github.com/sympy/sympy/pull/120
--
You received this message because you are subscribed to the Google Groups "sympy" group.
To post to this group, send email to sy...@googlegroups.com.
To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sympy?hl=en.


Mateusz

Mateusz Paprocki

unread,
Apr 2, 2011, 7:56:52 PM4/2/11
to sy...@googlegroups.com
Hi,

On 2 April 2011 15:04, smichr <smi...@gmail.com> wrote:
Though p12 might have some kinks to work out I think it's better to
get it in place so the things that are good can get exercised and
those that are not can get excised without causing a big headache. I
also believe it will be best to get this in before the GSoC gets to
work because every bit of work that interferes with p12 makes for more
work in doing the rebase.

I've already spent about 30 hours rebasing and going over commits and
feel pretty good about the integrity of p12 as a whole. Given Ondrej's
reminder that bisection can use the --skip option which results in a
range of commits being returned as the faulty range I am less stressed
about smoothing everything out in p12. Though tests will not all pass
in each commit, all commits  allow `from sympy import *` to be
executed.

Thank you for your work Chris. You see, I did this already twice for this part of my branch (one should note that first part was merged during summer) and I hoped that no one else will have to go through this.

I know that it would be perfect to have all tests pass in all configurations for every commit, but unfortunately we don't live in a perfect world. What I could afford during development of this branch was to make sure that tests in core, polys and simplify (or integrate/solvers, depending on changes) work for every commit. This is because it took ages to run whole test suit on my obsolete laptop and running the test suite without caching was completely infeasible. 

In my work I assume that caching doesn't work in SymPy, so all my codes are cache independent (I always run isympy and tests with SYMPY_USE_CACHE=no) and only before doing a commit I run selected test suites (or whole test suite) with caching enabled.

The running time of the test suite for polys was kept around 30s on my laptop for GMPY ground types. Taking this opportunity I would like to remind that recently I simplified ground types so now only two configurations have to be tested: gmpy and python. Mixtures are allowed but I don't expect them to work and I don't see any real life applications for them (python ground types are now independent from sympy ground types for < Python 2.6). sympy ground types are still allowed but not encouraged in any way and won't be picked up automatically in any circumstances by polys. There is no point in further supporting sympy gorund typs because of how SymPy's core works internally (e.g. Rational(1) -> S.One). There are still there only for experimentation purpose. 
 

I'm heartily in favor of pushing this asap and am willing to help "get
it done" as Vinzent suggests.

Chris
--
You received this message because you are subscribed to the Google Groups "sympy" group.
To post to this group, send email to sy...@googlegroups.com.
To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sympy?hl=en.


Mateusz

Mateusz Paprocki

unread,
Apr 2, 2011, 8:13:51 PM4/2/11
to sy...@googlegroups.com
Hi,

This is a very good idea and I think that this could be actually done not only for my branch but for SymPy in general (e.g. SymPy code coverage day, I don't mean to do it right now, but this would certainly improve quality of our test suite). I don't know what's the current code coverage for SymPy, but last time I checked this, it wasn't very optimistic.

On my side I can say that there was a period in development of polys, when I worked with code coverage tool on daily basis and that's why some *.py files in sympy.polys in my branch have nearly 100% coverage (e.g. dense*.py, but this might have been lowered a little bit after adding root isolation code). Unfortunately I didn't have enough energy to work this way all the time.

I would like also to point out that it would be nice to have 100% code coverage in SymPy, but this won't prove in any way that the code is correct and won't assure that tests we have are of any good quality (there are some good tests in SymPy, but there are also many that could be significantly improved). But when developing software of this magnitude in an interpreted programming language, in the absence of a compiler and a static type system, this would be very helpful. 


Chris


--
You received this message because you are subscribed to the Google Groups "sympy" group.
To post to this group, send email to sy...@googlegroups.com.
To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sympy?hl=en.


Mateusz

Mateusz Paprocki

unread,
Apr 2, 2011, 11:03:17 PM4/2/11
to sy...@googlegroups.com
Hi,

Was this result ever proved to be correct? (or maybe you are talking about a different case)
 
  I am becoming convinced that the slowness this is just the nature of the algorithm, because it produces candidate integrals that are very large even for relatively small integrands. 

If I remember correctly this algorithm has exponential complexity in the number of "functional" components of an integrand and the degree of the candidate solution. The main  reason why this algorithm is only a heuristic is that currently it is only possible to get an estimate of this degree. If we take a small degree, the algorithm runs fast but it may fail to find an antiderivate. We can take a higher degree and cover more classes of integrands, but in this case the algorithm will be very slow. Currently we use original Bronstein's algorithm for this, with some tiny improvements of my own based on empirical cases (those small improvements don't change the overall behaviour of the algorithm in general). I think we can't do much about purely algorithmic issues of heurisch() (at least not me), but on the implementation level, to improve speed of this algorithm, it would be great at least to have fast matrices, possibly implemented using similar approach I took in polys (I mean multiple level structure, etc.).
 

To me, there are three things that come to mind that should always be done right the first time (i.e., before any merge).  The first is the user interface, since changing this in the future breaks backward compatibility, especially if there is a release in the meanwhile. The second thing is something that is not an issue in polys12, which is that no branch should be pushed in with a known wrong result.  You have been really good at fixing any wrong result that I have reported (like when you worked on RootSum.diff), but have not

Implementation of RootSum.diff (well RootSum itself) was a specific experience. A few years ago I discovered RootSum in Mathematica's documentation, but didn't have any idea how this work and how I could implement such a thing. When you pointed out that you need such functionality, I had already enough knowledge to figure this out and implement this. Moreover, it was great fun to work on this because at some point I realized that with new advancements in polys, we can go quite far with this, reaching almost Mathematica's level (not speaking here about speed, which isn't bad as for pure Python but could be better). After many tiring months of writing low-level code, this was finally a beautiful application of several things I implemented before (I like especially this example with exponential functions).
 
been so open on considering suggestions regarding the interface for already existing functions (like the RootOf issue).

The issue with RootOf was that I had enough work to make it actually work and that's one of reasons I wasn't very supportive in discussions about the API. Another thing is that any such discussions have to account the fact that RootOf should be a result from roots() and solve(), which is not the case right now. The API is a little over complicated to provide a way  to accommodate for this issue with roots() and solve(). So, if we want to improve RootOf we should also improve those two functions. I didn't want to do it on my own, because I think this is a big thing, because solve() is used in so many places in SymPy and it is not that trivial to fix as it was with symbols(). I don't say we can make of this two separate issues and solve them separately (at least as much as it can be done), but at least you should see that there was actually a reason.
 
 The third thing is of course test failures, which have already been talked about here.

Regarding my branch, my plan was to not merge integration3 until I finished the algorithm, so that you could have a complete implementation of the transcendental algorithm in integrate(), but I now see that this is wrong. It is better to merge it in with the NotImplementedErrors and then improve it.  That way, people will try it out.  I am more hesitant to merge integration3 because I am constantly finding bugs in it.  It needs to be tested more. N so much that it needs more tests, though this is true in some cases, but it just needs more people to try it out.  But I now see that the best way to do this is to push it out to the official repo, where everyone will be using it.

It's great you see the point. The community support is one of reasons I left at some point the discrete/concrete stuff and moved integration, because there was almost no one to help me testing the former and the later have been used and tested as a side effect all the time. I hope that at some point I will finish what I started in sympy.concrete.

There are two ways to proceed with risch_integrate(). Either use it in parallel with integrate() and advertise that there is another algorithm implemented, which is fast and can prove that antiderivatives exist, but not yet complete. Or, we can add try: risch_integrate() except NotImplementedError: (...) to integrate() and in (...) fall back to heurisch().


Unfortunately, integration3 is blocked pretty heavily on issue 2026 (exact or atomic substitution).  I had to completely disable algebraic substitution in exp to make the preparser algorithm work, but this makes some limit tests break.  There is no easy workaround, it just needs to be implemented.

Any efforts regarding this will pay off, because Risch algorithm is one of landmarks of computer algebra, so having it (among other important algorithms) shows that we are serious, and it's just better than what we have now. However, I would not forget completely about heurisch(). I think that this algorithm (when improved) will serve together with risch_integrate(), at least until we will have a reasonable implementation of G-functions approach.
 
 

 


> * Lambda.is_identity shouldn't be implemented (cf issue 2178)
> * .find() and .replace() need to be discussed. The proposed interface of
> the latter is too complex.

What is wrong with this?  I though it was just expr.replace(old, new), where it uses .match() so that it is smart if it uses Wilds.  Anyway, if you don't like something, you have to be specific about what is wrong with it.

We agreed on this in an issue. If there are better ideas (I'm sure there are), why don't allow this to go in, and start from there?
 

> * symbols('xyz') should be deprecated, but not removed outright (I don't
> think I ever mentioned that one before)

I don't think this will work.  symbols('xyz') should create one symbol (Symbol('xyz')).  Trying to deprecate this would just keep the bad interface problem there (see the discussion in issue 1977).

Now I'm confused. Was 0.7.0 supposed to be this great release where backwards incompatible changes to the API should have been made? I'm pretty sure this was the purpose of this release. Anyway, deprecating this and keeping old syntax is a very bad idea.

I agree 100% here.

 

> * exprtools adds an additional mess to a messy situation in Add and Mul

Is this actually used anywhere?  If not, we could just hold out on it pretty easily. Or we could add it as a (currently unused) prototype.

Actually it's a very important part of this branch and is used in together(), factor(), etc., to provide generalized and fast approach to manipulation of commutative expressions. This has nothing to do (directly) with Add and Mul, and if you don't understand this, please don't call this messy. I placed it in the core, because I'm sure this will be a significant part of the future core.
 

> * Most additions to sympy.utilities.iterables reimplement functions in
> itertools.

Maybe this one is being picky.  I don't care either way with it.

Exactly. Doesn't this work? It does, tests pass. Is it redundant? At least partially, but when implementing those functions I didn't have any clue  about this. Does this interfere with the merge? I don't think so and if we merged this three months ago, this could have been easily refined before the release.

So, the only real issue might be to pass the whole test suite after the merge. I will point this out again: three months ago *all* tests, doctests and documentation tests passed for most important configurations. Now they obviously won't (although Chris' branch may help here).

btw. Going through recent commits I found this:


It seems that our review machinery doesn't work that good anyway ("A" isn't a very informative commit message, unless I miss a very important point here). I pointed this out just to show that sometimes we are very picky about commits and sometimes we push this kind of stuff (I'm referring only to commit message right now, not the commit it self). Maybe average of those two extremes would be a working approach? (i.e. don't scare developers or potential developers, but also don't allow to incorrect/misleading stuff to go in).

OK, so the "A" commit was my fault, and I take 100% blame for it.  I basically forgot to do a "git fetch" before "git push" (he had fixed it in his branch, but I merged the wrong revision).  So I think this is a bad example (because it was completely accidental).

No problem with this. Simple mistakes like this happen. In this case it doesn't prove my point, but I think I could easily find one that would prove (no, I'm not going to this, there is no point in this). 
 
 

But I do agree that we let things slide from time to time.  For example, we often do not keep a commit from going in if it has some spacing errors (like a==b+c instead of a == b + c), especially for new contributors.  But like I said above, the user level interface is important to get right the first time, and we do (or at least I do) block commits from going in if I see a problem there, even with the new contributors.  

There are only a few outstanding of these issues in polys12 (RootOf, Pure, Lambda).  The remainder aren't as important, because they are internal (and it seems like you are constantly rewriting the internals anyway).

This is a very good observation. Yes, I refine internals quite often. When doing the initial implementation I try not care too much about unimportant details, but just make it work in some reasonable way and then improve it and converge to a (hopefully) good solution (it doesn't mean I don't put any thought into the initial implementation). In my branch you can observe phases when I was working on something, made it work and moved to something else. After some time, when I had seen the code in "action", I returned to this "forgotten" implementation, and improved it and entered another iteration of this loop. I think it's healthy to refactor code often because this way you can incorporate new techniques very fast into your codes. We learn all the time, so for example today I most certainly wouldn't implement 
heurisch() I did it originally. It would have been better if I actually applied this approach to heurisch().
 
 

Aaron Meurer

 

Aaron Meurer

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To post to this group, send email to sy...@googlegroups.com.
To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sympy?hl=en.


Mateusz

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To post to this group, send email to sy...@googlegroups.com.
To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sympy?hl=en.

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To post to this group, send email to sy...@googlegroups.com.
To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sympy?hl=en.

Mateusz

Ondrej Certik

unread,
Apr 2, 2011, 11:30:59 PM4/2/11
to sy...@googlegroups.com, Andy Ray Terrel, Mateusz Paprocki
On Fri, Apr 1, 2011 at 11:36 PM, Andy Ray Terrel <andy....@gmail.com> wrote:
> For better or for worse, SymPy has always been a place where patches
> that meet a minimum standard are pushed in.  In the interest of not
> going 1.5 years between releases, I suggest we create issues for what
> needs to be done and push Mateusz's code.

Given the scope of Mateusz code and the improvements that it brings, I
also vote for pushing in Mateusz branch, in the best state that we
can, and improve upon it.

As to the release, I think we can release even without the merge, in
fact, I would highly recommend it. As the merge should be tested for
some time.

Ondrej

smichr

unread,
Apr 2, 2011, 11:52:32 PM4/2/11
to sympy
The additional commits in p12 have been cleaned up (and need review).
The XFAIL test still has to be added for the failing ode.

Brian Granger

unread,
Apr 3, 2011, 12:58:24 AM4/3/11
to sy...@googlegroups.com, smichr
Hi,

I haven't followed this discussion too closely, but I think there is a
lesson to learn (which we all probably know already, but it is
relevant for the GSoC students). I am guilty as anyone on this, but
*large* branches that have many commits with complex and far reaching
changes are extremely difficult to review and merge, no matter how the
process in managed (I think sympy does a pretty good job of it...).

The solution is to not allow such branches in the first place. I know
this is completely counter to how we typically write code, but I think
it is the only real solution. Large efforts and their related
branches should be broken up into reasonably sized sub-tasks that can
be reviewed and merged on a shorter time scale. Saving it all up in
one big branch may seem easier at the time, but in the long run, it is
too painful for everyone and so much time passes, everyone has
forgotten many of the important details.

For students writing GSoC proposals, I *highly* recommend identifying
points during the summer where you will submit your code for review
and merging before continuing.

Cheers,

Brian

On Sat, Apr 2, 2011 at 8:52 PM, smichr <smi...@gmail.com> wrote:
> The additional commits in p12 have been cleaned up (and need review).
> The XFAIL test still has to be added for the failing ode.
>

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To post to this group, send email to sy...@googlegroups.com.
> To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/sympy?hl=en.
>
>

--
Brian E. Granger
Cal Poly State University, San Luis Obispo
bgra...@calpoly.edu and elli...@gmail.com

Mateusz Paprocki

unread,
Apr 3, 2011, 1:26:21 AM4/3/11
to sy...@googlegroups.com

Mateusz Paprocki

unread,
Apr 3, 2011, 2:59:27 AM4/3/11
to sy...@googlegroups.com
Hi,

On 3 April 2011 06:58, Brian Granger <elli...@gmail.com> wrote:
Hi,

I haven't followed this discussion too closely, but I think there is a
lesson to learn  (which we all probably know already, but it is
relevant for the GSoC students).  I am guilty as anyone on this, but
*large* branches that have many commits with complex and far reaching
changes are extremely difficult to review and merge, no matter how the
process in managed (I think sympy does a pretty good job of it...).

The solution is to not allow such branches in the first place.  I know
this is completely counter to how we typically write code, but I think
it is the only real solution.  Large efforts and their related
branches should be broken up into reasonably sized sub-tasks that can
be reviewed and merged on a shorter time scale.  Saving it all up in
one big branch may seem easier at the time, but in the long run, it is
too painful for everyone and so much time passes, everyone has
forgotten many of the important details.

I think sometimes large branches are unavoidable, especially when you know what's the big goal (in this case it was making polynomials manipulation algorithms fast), but you don't know exactly how to achieve it at the very beginning and you figure out a solution when gradually fixing certain issues (i.e. implementing things, committing). I don't say that we shouldn't make any plans at all, actually we should plan as much as it is in our power. However, sometimes we make a breakthrough that opens many new possibilities that we wouldn't expect to have (for me one of such breakthroughs was implementation of Wang's algorithm (multivariate factorization), which allowed me to use polynomial factorization for practical purpose especially in simplification algorithms). Enthusiasm after such event can lead additional important work, so this social aspect shouldn't overlooked (actually is should be exploited). 

Another problem is that in our community certain issues require ages of discussion before any action is taken. Does it mean I have to stop development for that time, before an issue is resolved (especially if something is important to my work)? No, I implement stuff, taking into consideration what was already discussed. Advantage is that an implementation finally is available (possibly suboptimal) and can be used as a starting point for the final implementation.

The most important thing, however, is that big branches are doable when the review process gradually follows development of a branch (this is my personal opinion). And this is how it worked with polys branch. There were two developers Aaron and Chris (lexicographic order) who reviewed this branch on regular basis and simply speaking made this branch better (I already pointed this out but it seems this is overlooked in this discussion). Forgive me if I forgot someone's else contribution. Where were other developers then? I don't want to make any accusations, I was enough happy that there was at least someone interested in my work and willing to help and I perfectly understand that SymPy is not our major concern, but school/work is (my self I can't count how many nights I spent working on SymPy). Also, there was an issue open for very long time specifically for the purpose of merging my work, where we discussed things that remained to be done, test failures, etc.

I think that the real issue is how to separate work is directly related to the problem you try to solve and what isn't that important and can be factored out to separate branches which can be developed/reviewed in parallel, but without loosing the momentum of your work. I have to admit that I failed to recognize this in a couple of places in my branch. A good example is the new syntax for symbols()/var() which could have been developed in another branch. On the other hand, the new syntax is so appealing that it was handy to have it when experimenting with polynomials in interactive sessions. 
 

For students writing GSoC proposals, I *highly* recommend identifying
points during the summer where you will submit your code for review
and merging before continuing.


btw. Sorry for the last empty message. My mouse is very annoying and it seems GMail likes to send empty e-mails.
 
Cheers,

Brian

On Sat, Apr 2, 2011 at 8:52 PM, smichr <smi...@gmail.com> wrote:
> The additional commits in p12 have been cleaned up (and need review).
> The XFAIL test still has to be added for the failing ode.
>
> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To post to this group, send email to sy...@googlegroups.com.
> To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/sympy?hl=en.
>
>



--
Brian E. Granger
Cal Poly State University, San Luis Obispo
bgra...@calpoly.edu and elli...@gmail.com

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To post to this group, send email to sy...@googlegroups.com.
To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sympy?hl=en.


Mateusz

Ondrej Certik

unread,
Apr 3, 2011, 4:54:49 AM4/3/11
to sy...@googlegroups.com
Hi Mateusz,

Well, last 2 weeks, you can also use this feature of github pull requests:

https://github.com/blog/821-mention-somebody-they-re-notified

to get the attention of a particular developer. Previously, you would
need to write an email, saying for example "Ondrej, please review
this, I need to get this in.". I don't think I got such an email (if I
did, I apologize).

In general, in my experience I would say that none of the sympy
developers wanted to "discuss things to death" (rather just provide
useful feedback), but for large branches like this one, it requires
quite a bit of work from the original author (e.g. you in this case)
to push the review forward. And it is not easy, we all know that. But
I think, given the feedback in this thread, that we all want to get
this in.

So if you would be ok to give us one last chance to get this in (so to
say:), I think that everybody will be very open to help get this in,
as I think everybody agrees we want all the features that you have
implemented.

Let us know if you have time to push this forward, or whether you
would like somebody else to do it. And then let's get it done.

Ondrej

Aaron S. Meurer

unread,
Apr 3, 2011, 4:43:47 PM4/3/11
to sy...@googlegroups.com
Yes.  See http://en.wikipedia.org/wiki/User:CRGreathouse/Risch.  And actually, it seems I got it wrong.  SymPy is the *only* CAS that can do it (including Axiom).  Note that they never tested Bronstein's original PMINT, so if heurisch is just a copy of that, it can probably do it too.

 
  I am becoming convinced that the slowness this is just the nature of the algorithm, because it produces candidate integrals that are very large even for relatively small integrands. 

If I remember correctly this algorithm has exponential complexity in the number of "functional" components of an integrand and the degree of the candidate solution. The main  reason why this algorithm is only a heuristic is that currently it is only possible to get an estimate of this degree. If we take a small degree, the algorithm runs fast but it may fail to find an antiderivate. We can take a higher degree and cover more classes of integrands, but in this case the algorithm will be very slow. Currently we use original Bronstein's algorithm for this, with some tiny improvements of my own based on empirical cases (those small improvements don't change the overall behaviour of the algorithm in general). I think we can't do much about purely algorithmic issues of heurisch() (at least not me), but on the implementation level, to improve speed of this algorithm, it would be great at least to have fast matrices, possibly implemented using similar approach I took in polys (I mean multiple level structure, etc.).

Well, Matrices should be happening this summer with GSoC (we already have a few students interested).  I have tried to suggest to the students to copy the polys model.

What we also need is a sparse implantation in the polys, so that we can use Poly in heurisch() (right now using Poly is even slower than the current Basic implementation because it takes forever to multiply two polynomials with 100 different variables).  I already did the work converting it (see polyheurisch.py in my integration3 branch), so if Poly becomes faster we can just plug it in.

 

To me, there are three things that come to mind that should always be done right the first time (i.e., before any merge).  The first is the user interface, since changing this in the future breaks backward compatibility, especially if there is a release in the meanwhile. The second thing is something that is not an issue in polys12, which is that no branch should be pushed in with a known wrong result.  You have been really good at fixing any wrong result that I have reported (like when you worked on RootSum.diff), but have not

Implementation of RootSum.diff (well RootSum itself) was a specific experience. A few years ago I discovered RootSum in Mathematica's documentation, but didn't have any idea how this work and how I could implement such a thing. When you pointed out that you need such functionality, I had already enough knowledge to figure this out and implement this. Moreover, it was great fun to work on this because at some point I realized that with new advancements in polys, we can go quite far with this, reaching almost Mathematica's level (not speaking here about speed, which isn't bad as for pure Python but could be better). After many tiring months of writing low-level code, this was finally a beautiful application of several things I implemented before (I like especially this example with exponential functions).

Well RootSum was the most deficient part of the polys that I came across for the Risch stuff.  Everything else was already implemented, except for some trivial things, so I would say that the polys are definitely strong (and I didn't even discover the RootSum deficiency for a while because it requires some very "not nice" integrals to actually return them).

 
been so open on considering suggestions regarding the interface for already existing functions (like the RootOf issue).

The issue with RootOf was that I had enough work to make it actually work and that's one of reasons I wasn't very supportive in discussions about the API. Another thing is that any such discussions have to account the fact that RootOf should be a result from roots() and solve(), which is not the case right now. The API is a little over complicated to provide a way  to accommodate for this issue with roots() and solve(). So, if we want to improve RootOf we should also improve those two functions. I didn't want to do it on my own, because I think this is a big thing, because solve() is used in so many places in SymPy and it is not that trivial to fix as it was with symbols(). I don't say we can make of this two separate issues and solve them separately (at least as much as it can be done), but at least you should see that there was actually a reason.

I see.  Well, should we at least deprecate or not allow the behavior at the user-level?

 
 The third thing is of course test failures, which have already been talked about here.

Regarding my branch, my plan was to not merge integration3 until I finished the algorithm, so that you could have a complete implementation of the transcendental algorithm in integrate(), but I now see that this is wrong. It is better to merge it in with the NotImplementedErrors and then improve it.  That way, people will try it out.  I am more hesitant to merge integration3 because I am constantly finding bugs in it.  It needs to be tested more. N so much that it needs more tests, though this is true in some cases, but it just needs more people to try it out.  But I now see that the best way to do this is to push it out to the official repo, where everyone will be using it.

It's great you see the point. The community support is one of reasons I left at some point the discrete/concrete stuff and moved integration, because there was almost no one to help me testing the former and the later have been used and tested as a side effect all the time. I hope that at some point I will finish what I started in sympy.concrete.

There are two ways to proceed with risch_integrate(). Either use it in parallel with integrate() and advertise that there is another algorithm implemented, which is fast and can prove that antiderivatives exist, but not yet complete. Or, we can add try: risch_integrate() except NotImplementedError: (...) to integrate() and in (...) fall back to heurisch().

My plan was to plug it in the current code after the table lookups (like poly integrate, rational function integrate, trigintegrate, etc.), but before heurisch() (which is done last).  The plan was to make a separate NonElementaryIntegral class so you could tell the difference between when it was proving that an integral was not elementary and when it is not implemented or returned from heurisch().  

Also, there needs to be an elementary keyword option to integrate() that when set to False (the default) tries to further integrate non-elementary functions in terms non-elementary functions (like erf).  This would currently just mean passing the integral to heurisch(), but in the future, there will be support directly in the Risch algorithm for functions like erf().




Unfortunately, integration3 is blocked pretty heavily on issue 2026 (exact or atomic substitution).  I had to completely disable algebraic substitution in exp to make the preparser algorithm work, but this makes some limit tests break.  There is no easy workaround, it just needs to be implemented.

Any efforts regarding this will pay off, because Risch algorithm is one of landmarks of computer algebra, so having it (among other important algorithms) shows that we are serious, and it's just better than what we have now. However, I would not forget completely about heurisch(). I think that this algorithm (when improved) will serve together with risch_integrate(), at least until we will have a reasonable implementation of G-functions approach.

Definitely.  Right now, it is the only thing that has a hope of handling algebraic integrals. 
Well, the constant API change is a little annoying to me, because every time I merge with your latest branch, I have to redo any changes I made to the polys.  For example, the biggest pain with the polys11 merge was redoing all of my fixes to the domains.  And I have implemented Poly.cancel(include=True), which I have had to redo two or three times because of your changes to the same function, and I see that I will have to do it again whenever I merge with the changes from polys12 (by the way, the option makes it include the coefficient with the result instead of returning it, because I have to cancel polys in the Risch code a lot to ensure that gcd(a, d) == 1 foes and to keep things from blowing up, but I don't care to carry around the coefficient separately whenever I do this).

In fact, I am pretty sure I am going to have to redo my integration3-backport branch (https://github.com/sympy/sympy/pull/131) whenever polys12 gets in (fortunately, the conflict resolutions are all in the merge commits in my branch).  And there are several polys fixing commits from my integration3 branch that I didn't even bother putting in that branch because they would have been too much of a headache to cherry-pick in (for example, my "composite option" commit, which fixes some nasty rational function integration bugs for functions with symbolic coefficients).  

The best thing would be if you could cherry-pick my fixes into your branch whenever you see me make them, so that this doesn't happen.  Would it be easier if I cherry-picked them into a branch based on polys-whatever whenever I make them and send them to you as a pull request in the future?

Basically, it is great to work on top of polys-n instead of master for my integration stuff, because I get the latest and greatest of the polys (and I don't duplicate any fixes that you make).  But there needs to be a better way to manage us both making fixes to the polys in our dev branches.

Aaron Meurer

Aaron S. Meurer

unread,
Apr 3, 2011, 5:02:39 PM4/3/11
to sy...@googlegroups.com

Nice. I was thinking one of the biggest deficiencies to GitHub was the lack of a CC field.

Now I wonder if it is possible to automatically subscribe myself to all SymPy pull request comments, even if I do not participate in that pull request…

Aaron Meurer

>
> In general, in my experience I would say that none of the sympy
> developers wanted to "discuss things to death" (rather just provide
> useful feedback), but for large branches like this one, it requires
> quite a bit of work from the original author (e.g. you in this case)
> to push the review forward. And it is not easy, we all know that. But
> I think, given the feedback in this thread, that we all want to get
> this in.
>
> So if you would be ok to give us one last chance to get this in (so to
> say:), I think that everybody will be very open to help get this in,
> as I think everybody agrees we want all the features that you have
> implemented.
>
> Let us know if you have time to push this forward, or whether you
> would like somebody else to do it. And then let's get it done.
>
> Ondrej
>

Aaron S. Meurer

unread,
Apr 3, 2011, 4:59:35 PM4/3/11
to sy...@googlegroups.com
On Apr 3, 2011, at 12:59 AM, Mateusz Paprocki wrote:

Hi,

On 3 April 2011 06:58, Brian Granger <elli...@gmail.com> wrote:
Hi,

I haven't followed this discussion too closely, but I think there is a
lesson to learn  (which we all probably know already, but it is
relevant for the GSoC students).  I am guilty as anyone on this, but
*large* branches that have many commits with complex and far reaching
changes are extremely difficult to review and merge, no matter how the
process in managed (I think sympy does a pretty good job of it...).

The solution is to not allow such branches in the first place.  I know
this is completely counter to how we typically write code, but I think
it is the only real solution.  Large efforts and their related
branches should be broken up into reasonably sized sub-tasks that can
be reviewed and merged on a shorter time scale.  Saving it all up in
one big branch may seem easier at the time, but in the long run, it is
too painful for everyone and so much time passes, everyone has
forgotten many of the important details.

Well, would you have me put in a merge request for my integration3 branch, even though the branch was entirely useless to the user until this commit (https://github.com/asmeurer/sympy/commit/e3cd5f18f86fd6377836f33f726182c8bd4dc1a0) from August 5 that added the user-level risch_integrate() function?  This is all hypothetical, because my branch is based on Mateusz polys branch, which would have had to be merged with it.

Also, as I mentioned before, risch_integrate() cannot work without this hack (https://github.com/asmeurer/sympy/commit/62f3b670963ea2fc687120fadc808cb8dbc1c532) to exp._eval_subs, which breaks the limit code (this is the commit immediately before the one above in my branch).  There are two ways I could have made this work to a mergable state.   One would have been to implement atomic or exact substitution, something that would have taken away from my time working on integration, and is something that I feel that others who have worked more in subs (like Chris or Ronan) would do a better job of implementing.  The other would have been to do a serious hack to the code to make that one case of substitution (exp) work in either case.
 

I think sometimes large branches are unavoidable, especially when you know what's the big goal (in this case it was making polynomials manipulation algorithms fast), but you don't know exactly how to achieve it at the very beginning and you figure out a solution when gradually fixing certain issues (i.e. implementing things, committing). I don't say that we shouldn't make any plans at all, actually we should plan as much as it is in our power. However, sometimes we make a breakthrough that opens many new possibilities that we wouldn't expect to have (for me one of such breakthroughs was implementation of Wang's algorithm (multivariate factorization), which allowed me to use polynomial factorization for practical purpose especially in simplification algorithms). Enthusiasm after such event can lead additional important work, so this social aspect shouldn't overlooked (actually is should be exploited). 

Another problem is that in our community certain issues require ages of discussion before any action is taken. Does it mean I have to stop development for that time, before an issue is resolved (especially if something is important to my work)? No, I implement stuff, taking into consideration what was already discussed. Advantage is that an implementation finally is available (possibly suboptimal) and can be used as a starting point for the final implementation.

The most important thing, however, is that big branches are doable when the review process gradually follows development of a branch (this is my personal opinion). And this is how it worked with polys branch. There were two developers Aaron and Chris (lexicographic order) who reviewed this branch on regular basis and simply speaking made this branch better (I already pointed this out but it seems this is overlooked in this discussion). Forgive me if I forgot someone's else contribution. Where were other developers then? I don't want to make any accusations, I was enough happy that there was at least someone interested in my work and willing to help and I perfectly understand that SymPy is not our major concern, but school/work is (my self I can't count how many nights I spent working on SymPy). Also, there was an issue open for very long time specifically for the purpose of merging my work, where we discussed things that remained to be done, test failures, etc.

I agree.  I followed religiously every commit added to Mateusz's branch, because they were all so important to me in my work (my branch is based on his for God's sake).  If I saw something I didn't like, I commented on the commit in GitHub, or on a relevant issue (to be sure, it wasn't too often that this happened).


I think that the real issue is how to separate work is directly related to the problem you try to solve and what isn't that important and can be factored out to separate branches which can be developed/reviewed in parallel, but without loosing the momentum of your work. I have to admit that I failed to recognize this in a couple of places in my branch. A good example is the new syntax for symbols()/var() which could have been developed in another branch. On the other hand, the new syntax is so appealing that it was handy to have it when experimenting with polynomials in interactive sessions. 

I agree.  In the future, you should separate the work you make to master in a separate branch.  Make it also in your branch if you need it, but it will rebase out or something in the future when it is in master. This has the additional benefit of margin your future merge/rebase with master easier, because your change will have been supported there.

I see now that it would have been easier if I had made my integration3-backport branch as I went, instead of the nightmare that it was making it retroactively.

Similarly, you and I should try to get my polys related changes into your branch, as I discussed in a previous email to this thread.

By the way, are there still relevant commits in your release0.7.0 branch?  Those should be cherry-picked into Chris's p12.

Aaron Meurer

Brian Granger

unread,
Apr 3, 2011, 6:04:30 PM4/3/11
to sy...@googlegroups.com, Aaron S. Meurer
Aaron,

I don't know enough about the state of your integration3 branch to
really know what is best in this situation. But, if parts of it are
complete, I don't think there is any harm in merging the completed
parts and going from there.

> Also, as I mentioned before, risch_integrate() cannot work without this hack
> (https://github.com/asmeurer/sympy/commit/62f3b670963ea2fc687120fadc808cb8dbc1c532)
> to exp._eval_subs, which breaks the limit code (this is the commit
> immediately before the one above in my branch).  There are two ways I could
> have made this work to a mergable state.   One would have been to implement
> atomic or exact substitution, something that would have taken away from my
> time working on integration, and is something that I feel that others who
> have worked more in subs (like Chris or Ronan) would do a better job of
> implementing.  The other would have been to do a serious hack to the code to
> make that one case of substitution (exp) work in either case.
>
>
> I think sometimes large branches are unavoidable, especially when you know
> what's the big goal (in this case it was making polynomials manipulation
> algorithms fast), but you don't know exactly how to achieve it at the very
> beginning and you figure out a solution when gradually fixing certain issues
> (i.e. implementing things, committing). I don't say that we shouldn't make
> any plans at all, actually we should plan as much as it is in our power.

I agree that at times large branches are unavoidable. But for every
one that is unavoidable, there are numerous large branches that could
have been split into smaller pieces without much difficulty (I am not
just thinking of sympy here). With this in mind, I do think it is a
good idea to encourage GSoC students to plan on merging multiple times
during the summer, rather than waiting until the very end.

> However, sometimes we make a breakthrough that opens many new possibilities
> that we wouldn't expect to have (for me one of such breakthroughs was
> implementation of Wang's algorithm (multivariate factorization), which
> allowed me to use polynomial factorization for practical purpose especially
> in simplification algorithms). Enthusiasm after such event can lead
> additional important work, so this social aspect shouldn't overlooked
> (actually is should be exploited).

Yes.

> Another problem is that in our community certain issues require ages of
> discussion before any action is taken. Does it mean I have to stop
> development for that time, before an issue is resolved (especially if
> something is important to my work)? No, I implement stuff, taking into
> consideration what was already discussed. Advantage is that an
> implementation finally is available (possibly suboptimal) and can be used as
> a starting point for the final implementation.

I have struggled with this as well during review of the quantum stuff.
While I deeply appreciated the review comments, I had external
deadlines that I wasn't about to miss while the community discussed
things (even though those discussions were important as well).

> The most important thing, however, is that big branches are doable when the
> review process gradually follows development of a branch (this is my
> personal opinion).

+10

We use this approach with IPython and it *really* helps.

Ronan Lamy

unread,
Apr 4, 2011, 12:01:48 PM4/4/11
to sy...@googlegroups.com
Le dimanche 03 avril 2011 à 14:43 -0600, Aaron S. Meurer a écrit :

> Basically, it is great to work on top of polys-n instead of master for
> my integration stuff, because I get the latest and greatest of the
> polys (and I don't duplicate any fixes that you make). But there
> needs to be a better way to manage us both making fixes to the polys
> in our dev branches.

Which is why I suggested to put the dev branch on the main repo. Once
it's there, it's frozen (no rebases) and everybody can work on it, send
pull requests, open issues against it, etc. It might force us to do a
lot of merges but that can't be worse than rebasing hundreds of commits
over and over again.


Aaron S. Meurer

unread,
Apr 4, 2011, 5:33:28 PM4/4/11
to sy...@googlegroups.com
Well, I am convinced that large branches should be frozen anyway. The larger branches have too high of a chance of rebases screwing up commits so that they don't do what the did originally. That is why I have been exclusively merging integration3 since some point in the middle of last summer (and I wish I was more enlightened at the beginning of the summer).

How would this dev branch in the official repo thing work? Who would get branches there? Who can push to them? Does the review process change with any of this? And is it really better than me just sending a pull request to Mateusz (for example)?

Aaron Meurer

Ronan Lamy

unread,
Apr 4, 2011, 8:32:11 PM4/4/11
to sy...@googlegroups.com
Le lundi 04 avril 2011 à 15:33 -0600, Aaron S. Meurer a écrit :
> Well, I am convinced that large branches should be frozen anyway. The
> larger branches have too high of a chance of rebases screwing up
> commits so that they don't do what the did originally. That is why I
> have been exclusively merging integration3 since some point in the
> middle of last summer (and I wish I was more enlightened at the
> beginning of the summer).
>
> How would this dev branch in the official repo thing work? Who would
> get branches there? Who can push to them? Does the review process
> change with any of this? And is it really better than me just sending
> a pull request to Mateusz (for example)?
>
How to decide to create new branches isn't clear. We shouldn't create
many of them - we don't want to have dozens of stale branches in the
official repo.

Pushing to them should be just like pushing to 'master': send a pull
request (to sympy:branch instead of sympy:master), get it reviewed. The
review criteria would be different, though not that much since every
commit in a dev branch is supposed to appear in master's tree in the
end.

As for it being better than sending a request to Mateusz, well, if you
had followed the latest discussions on GitHub, you would know that you
should really send it to Chris, except that Vinzent has the most recent
version so you could try with him, unless, of course, he has already
sent it back to Chris... I think it's easier to have a stable branch in
a stable location.

Ondrej Certik

unread,
Apr 4, 2011, 11:27:16 PM4/4/11
to sy...@googlegroups.com

My suggestion would be not to introduce any more "official" branches,
than the "master". It will create more confusion than benefits.
One should work hard to get the branch in, rather than keeping it
(indefinitely) alongside master. Github provides enough tools for
collaborating on such branches, e.g. sending pull requests to Mateusz,
or Chris, or the one who opened the latest pull request with the
branch.

Ondrej

Brian Granger

unread,
Apr 5, 2011, 12:11:52 AM4/5/11
to sy...@googlegroups.com, Ondrej Certik

+1


> Ondrej


>
> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To post to this group, send email to sy...@googlegroups.com.
> To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/sympy?hl=en.
>
>

--

Vinzent Steinberg

unread,
Apr 5, 2011, 10:31:02 AM4/5/11
to sympy
I agree, in this case it is better to have a central branch which
anyone can access. Now, if you want to work on p12, you have to figure
out which version is the most recent. And more importantly, anyone can
push to it.

So you suggest we don't rebase p12 anymore?

Ondrej wrote:
> Github provides enough tools for
> collaborating on such branches, e.g. sending pull requests to Mateusz,
> or Chris, or the one who opened the latest pull request with the
> branch.

I don't think this approach works well, because you create unnecessary
bottlenecks (only one person can push).

Vinzent

Mateusz Paprocki

unread,
Apr 6, 2011, 4:40:52 AM4/6/11
to sy...@googlegroups.com
Hi,

Yes, four of them in release0.7.0 (symbols requirement in integrate() and diff(), clean up of old-style exceptions (once again)) and three in polys12 from January (add order to MathML printer and allow to setup ground types and caching in isympy). I already started taking care of this.

Mateusz
Reply all
Reply to author
Forward
0 new messages