removing the old assumption system - help request

[Fabian Pedregosa]

I've been lately trying to remove the old assumption system, but this
quite a daunting task to do alone.

If you want to help with this goal, just checkout my a_removal branch:

git pull http://fseoane.net/git/sympy.git a_removal

and submit fixes to me.

Most important issue right now is that the pretty printing system is
broken (but should not depend on the assumption system). For example:

In [5]: x*y
Out[5]:
1 ___ 1 ___
\/ x *\/ y

Thanks,

fabian

[Ondrej Certik]

I think a more pressing issue is that core tests don't run:

$ bin/test sympy/core/
============================= test process starts ==============================
executable: /usr/bin/python (2.6.2-final-0)

sympy/core/tests/test_arit.py[42] ..F.F^CffF.FF.F.^C interrupted by user

________________________________________________________________________________
___________________ sympy/core/tests/test_arit.py:test_arit0 ___________________
File "/home/ondrej/repos/sympy/sympy/core/tests/test_arit.py", line
48, in test_arit0
assert e == Rational(0)
AssertionError
________________________________________________________________________________
____________________ sympy/core/tests/test_arit.py:test_pow ____________________
File "/home/ondrej/repos/sympy/sympy/core/tests/test_arit.py", line
93, in test_pow
assert e == Rational(0)
AssertionError
________________________________________________________________________________
_______________ sympy/core/tests/test_arit.py:test_pow_issue417 ________________
File "/home/ondrej/repos/sympy/sympy/core/tests/test_arit.py", line
158, in test_pow_issue417
assert 4**Rational(1, 4) == 2**Rational(1, 2)
AssertionError
________________________________________________________________________________
__________________ sympy/core/tests/test_arit.py:test_expand ___________________
File "/home/ondrej/repos/sympy/sympy/core/tests/test_arit.py", line
168, in test_expand
assert (e.expand()-a*c-b*c) == Rational(0)
AssertionError
________________________________________________________________________________
_______________ sympy/core/tests/test_arit.py:test_power_expand ________________
File "/home/ondrej/repos/sympy/sympy/core/tests/test_arit.py", line
210, in test_power_expand
assert p.expand() == a**2 + b**2 + 2*a*b
AssertionError
________________________________________________________________________________
___________________ sympy/core/tests/test_arit.py:test_ncmul ___________________
File "/home/ondrej/repos/sympy/sympy/core/tests/test_arit.py", line
235, in test_ncmul
assert (C*(A+B)).expand() == C*A+C*B
AssertionError

======== tests finished: 6 passed, 6 failed, 2 xfailed in 29.98 seconds ========
DO *NOT* COMMIT!

I had to kill it with ctrl-C. Besides that, there seem to be some
basic problems with comparisons:

In [1]: 4**Rational(1, 4)
Out[1]:
⎽⎽⎽
╲╱ 2

In [2]: 2**Rational(1, 1)
Out[2]: 2

In [3]: 2**Rational(1, 2)
Out[3]:
⎽⎽⎽
╲╱ 2

In [4]: 4**Rational(1, 4) == 2**Rational(1, 2)
Out[4]: False

So I would concentrate on this. Pretty printing is a small cosmetic issue.

Ondrej

[Ondrej Certik]

Any progress on this one? Let us know your plans with the assumptions,
as Mateusz is coming today to Reno, so we can work on that too. It is
the most pressing thing in sympy now to be fixed.

Ondrej

[Kevin Goodsell]

Ondrej Certik wrote:
>
> Any progress on this one? Let us know your plans with the assumptions,
> as Mateusz is coming today to Reno, so we can work on that too. It is
> the most pressing thing in sympy now to be fixed.
>

I've started going through the tests and trying to fix them one at a
time, but it's extremely slow going, mostly because I don't know what
I'm doing. I can't find any real documentation on the assumptions
system, and I don't know what the "old" and "new" systems are beyond a
few pieces that I've been able to put together.

I get the impression that this is fairly important, and I'd like to help
with it, but I need some guidance. Documentation would be good, but if
that doesn't exist then we're going to need to another plan.

Also, it looks like Fabian's SymPy repository has been taken down. That
combined with his lack of responses here is a bit worrying. Not wanting
to jump to any conclusions, but if Fabian is unable to continue this
work (and I understand if that's the case), I hope we have what we need
to finish it.

Any suggestions on how to proceed?

-Kevin

[Ondrej Certik]

Hi Kevin,

I think Fabian is busy with school. We should finish it ourselves.

>
> Any suggestions on how to proceed?

Essentially, the old system are assumptions attached to each Symbol()
instances, while the new system are assumptions attached in the
Assumption() instance, and that instance is then passed around into
methods like refine(). So it works just like in mathematica.

http://reference.wolfram.com/mathematica/ref/Refine.html

So what has to be done is:

* polish Fabian's branch, so that all tests pass (if you can't access
it, let me know, I should have it checked out somewhere)
* remove assumptions from the core system, and use refine() and the
new assumptions system

Then we can finally start refactoring the core, as it will become simpler.

Indeed this is an important task. Thanks a lot for looking into it.
Let me know if you have any questions -- If you have time to work on
this, I will reserve some time every day to help you out. if there are
at least 2 people working on it, we'll be able to finish it.

Ondrej

[Ondrej Certik]

I pushed Fabian's latest patches into the "a_removal" branch here:

git://github.com/certik/sympy.git

Ondrej

[Aaron S. Meurer]

Yes, thanks to the wonders of git, anyone who has pulled fabian's
repository before it went down has a full copy of it. If anyone needs
them, I have the following on my computer:

fseoane/a_removal 282bb5b some more fixes
fseoane/ask dd024f5 Documentation for the queries
module
fseoane/doc 77041ff Fix testing the rst docs under
python2.4
fseoane/logic 9b2ee4a Use efficient integer
representation across ask() method
fseoane/master 0fe6c9e Implement refine module
fseoane/py3 6ec0568 python3 fixes
fseoane/queries db2c0e4 Enable testing documentation
under doc/src/
fseoane/refine f213628 Remove code related to .is_even
from functions.elementary.complexes
fseoane/remove_assumptions 9209119 remove old assumption system
fseoane/revert aa50a12 Revert "Fix testing the rst docs
under python2.4"

But it seems that the relevant branch (a_removal) is the same HEAD as
the one Ondrej pushed.

Are there any specific tasks that need to be done? I could help
(though not with anything major, due to my own school work), but I
don't want to overlap anyone else's work. And of course, I am
available for advice or help with debugging on #sympy on IRC.

Is this just a matter or replacing .is_real type assumptions with ask
(x, Q.Real), or are there bugs that need working out?

Aaron Meurer

[Ondrej Certik]

I suggest to first get all the tests pass, so that the branch is
usable. And then simply work file by file and removing the old
assumptions stuff and make sure all the tests pass.

>
> Is this just a matter or replacing .is_real type assumptions with ask
> (x, Q.Real), or are there bugs that need working out?

I think it's just this.

Ondrej

[Ronan Lamy]

No, I think it's more complicated than that and that a significant
amount of nontrivial work is needed. For example, in a_removal (282bb5),
Or(Assume(x, 'a'), Assume(x, 'b')) raises (that's the first failure in
bin/test) because Or sorts its arguments. In master, this works thanks
to the following bit of nonsense:
(Assume(x, 'b') - Assume(x, 'a')).is_negative == False

I don't know what is the best way to proceed but I think that attempting
to solve the problem by directly removing the old assumptions will fail.
IMHO, the only way is to refactor the core and the logic modules first.

Ronan

[Ondrej Certik]

I think the only way to proceed is to take Fabian's work, and make it
usable, so that all tests pass. That's the very first step, unless we
have this starting point, we can't get nowhere. After that, we should
just remove stuff that can be removed and replace it with the new
assumptions system. If something has to be refactored, then it should
be refactored on the way.

Ondrej

[Ronan Lamy]

OK, but which work and which tests? I thought that everything he did,
except trying to remove the old assumptions, was already in and passed
all current tests.

Ronan

[Ondrej Certik]

His work in the "a_removal" branch that I pushed also to my github,
and the tests that fail, as I reported in the email in this thread
above --- my very first email:

http://groups.google.com/group/sympy/msg/c7d029939f453f17

Ondrej

[smichr]

I just tried to fix the first test in core that fails:

(a*b*c+c*b*a+b*a*c)*p-s.Rational(15)*a*b*c should equal Rational(0)

when a and c are generic symbols and b is positive and p is Rational
(5). So I tried...

>>> import sympy as s
>>> from sympy.assumptions import Assume
>>> p=s.Rational(5)
>>> s.var('a c')
(a, c)
>>> b=s.Symbol('b');Assume(b, 'positive')
Assume(b, 'positive', True)
>>> e=(a*b*c+c*b*a+b*a*c)*p-s.Rational(15)*a*b*c
>>> e
0*a**1*b**1*c**1
>>> e.evalf()
0*a**1*b**1*c**1
>>> _ == s.Rational(0)
False

I'm stuck not knowing why this thing isn't going to 0.

/c

[Ondrej Certik]

It seems like some evaluation bug. Try to debug it how the expression
is constructed. Also, for clarity, I would discourage to use global
assumptions in tests. Just use refine() for simplifying (with taking
assumptions into account).

Ondrej

[Vinzent Steinberg]

On Nov 2, 9:15 am, smichr <smi...@gmail.com> wrote:
> I just tried to fix the first test in core that fails:
>
> (a*b*c+c*b*a+b*a*c)*p-s.Rational(15)*a*b*c should equal Rational(0)
>
> when a and c are generic symbols and b is positive and p is Rational
> (5). So I tried...
>
> >>> import sympy as s
> >>> from sympy.assumptions import Assume
> >>> p=s.Rational(5)
> >>> s.var('a c')
> (a, c)
> >>> b=s.Symbol('b');Assume(b, 'positive')

This does not work, you just generate an assumption object. Use rather
global_assumptions.add(Assume(b, Q.positive)).

However you have to call global_assumptions.clear() after each test,
else you'll get a mess (this is what you get currently when just
mapping Symbol(..., **assump) to global_assumptions.add). We should
really implement local assumptions using with statements.

Vinzent

[smichr]

On Nov 2, 7:00 pm, Vinzent Steinberg

Noted...but the problem is more basic than that:

###
>>> var('b')
>>> print b*0
0*b**1
###

It appears that the _intcache values for 0, 1 and -1 have been
commented out in core/numbers.py so "Zero" is not being pulled from
the cache as something special...it's just coming out as an integer.

But...if you try to uncomment the three lines, then you run into
import problems...which again, I don't really know how to resolve. Is
there any tool that profiles your code and identifies the import
problem and offers a solution? I always feel like I'm stumbling around
in the dark when I try to fix these things.

FWIW my changes (including path fiddling) are pushed to github/
smichr's C_a_removal.

/c

[smichr]

> Is this just a matter or replacing .is_real type assumptions with ask
> (x, Q.Real), or are there bugs that need working out?
>

I ran a regex through the repo to see what appeared as assumption like
things (referred to as `used` below) and what already appeared as a
function (referred to as `defed` below). Here are the results:

>>> sorted(used.difference(defed))
['is_Add', 'is_Atom', 'is_Derivative', 'is_Function', 'is_Integer',
'is_Mul', 'is_Number', 'is_NumberSymbol', 'is_Order', 'is_Pow',
'is_Rational', 'is_Real', 'is_Singleton', 'is_Symbol', 'is_bounded',
'is_comparable', 'is_complex', 'is_composite', 'is_default',
'is_epydoc', 'is_even', 'is_finite', 'is_imaginary',
'is_infinitesimal', 'is_integer', 'is_irrational', 'is_negative',
'is_noninteger', 'is_nonnegative', 'is_nonpositive', 'is_nonzero',
'is_odd', 'is_positive', 'is_prime', 'is_rational', 'is_real',
'is_setup', 'is_unbounded', 'is_xxx', 'is_yyy']

>>> sorted(used.intersection(defed))
['is_2D', 'is_collinear', 'is_commutative', 'is_complete',
'is_concurrent', 'is_concyclic', 'is_constant', 'is_convex',
'is_dense', 'is_equilateral', 'is_homogeneous', 'is_hypergeometric',
'is_inhomogeneous', 'is_linear', 'is_lower', 'is_monic',
'is_monomial', 'is_multivariate', 'is_number', 'is_one',
'is_parallel', 'is_perpendicular', 'is_polynomial', 'is_pure',
'is_right', 'is_scalar', 'is_similar', 'is_simple', 'is_square',
'is_squarefree', 'is_symbolic', 'is_tangent', 'is_univariate',
'is_upper', 'is_zero']

>>> sorted(defed.difference(used))
['is_bold', 'is_complex_type', 'is_direct', 'is_feasible',
'is_functional', 'is_italic', 'is_on_blacklist', 'is_primitive',
'is_primitive_root', 'is_quad_residue', 'is_quasi_unit_numpy_array',
'is_rational_function', 'is_real_type', 'is_sparse']

What is the way to tell which are assumptions and which are not? e.g.
is_zero is defined as a method of polynomials but it is also an
assumption.

[Kevin Goodsell]

smichr wrote:
>
>
> It appears that the _intcache values for 0, 1 and -1 have been
> commented out in core/numbers.py so "Zero" is not being pulled from
> the cache as something special...it's just coming out as an integer.
>

In master this is no longer a problem, since it doesn't rely on the
cached instances anymore (this allows certain other things to work
properly, like Integer("0")). Rebasing a_removal onto master (which is
something I've already done in my own repro, and could push somewhere if
it will help) should fix this problem, I think.

-Kevin

[smichr]

If it works, let me know. I've got a version where I put those
_intcache's back in and Ondrej helped me get the imports right and now
it prints out 0 for 0*b as it should.

I've also made pass 1 through test_arit.py and the errors are down to

_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=

the StrictInequality error

_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=

TypeError: __nonzero__ should return bool or int, returned
StrictInequality

________________________________________________________________________________

_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=

the commutative errors

_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=

___________________ sympy\core\tests\test_arit.py:test_ncmul
___________________
File "C:\Documents and Settings\chris\sympy\sympy\core\tests
\test_arit.py", line 226, in test_ncmul
assert A*B != B*A
AssertionError
________________________________________________________________________________

_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=

the comparable error

_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=_=

_____________ sympy\core\tests\test_arit.py:test_Add_is_comparable
_____________
File "C:\Documents and Settings\chris\sympy\sympy\core\tests
\test_arit.py", line 983, in test_Add_is_comparable
assert (x+y).is_comparable == False
AssertionError


If anyone cares to take a look or a crack at fixing these, they are at
smichr's github under the C_a_removal branch.

[smichr]

Ronan did some work on the Add and Or so the StrictInequality isn't
popping up anymore. I made a helper function for the creation of
symbols with assumptions and got some failing tests to pass. I had to
push things and leave them but I *think* that refine is not necessary
to get a test to pass as in (-2)**even_int == 2**even_int...but will
have to check later.

Ronan's and my changes are pushed to C_a_removal at smichr's github

[smichr]

I would recommend that we set up an issues page dedicated to this
conversion process for this reason:

as I work through getting the tests to work, I can identify things
that should work that don't that aren't part of the file that I am
working on. Ronan, for example, got the And and Or working and that
allowed a bunch of tests in test_arit to not raise that error. And now
I've identified two more issues that need to work but someone else
could tackle these:

1)
re(x) != x (bug1() in test_arit)
I'm not sure where the logic for this will be stored; when the
comparison is made, re(x) will have an assumption of Q.real on it
while x (being generic) does not. But previously, the class re() had
an assumption of is_real on it...now I'm not sure where that
assumption will be stored since it is my impression that Assume
arguments should be (Symbol, Q val, bool).

2)
ask(exp(x), Q.bounded) == None
If x is unbounded, exp(x) may or may not be bounded. If x is
nonnegative then it is unbounded, but if x is nonpositive then it is
bounded between 1 and 0.
- what does it mean to be unbounded? that a value is in (-oo,
oo) or does (0, oo) count?
_________________________________

^ could someone answer this for me?

If we can set up an issues page then different persons can work
through the different test files (like I am on test_arit.py right now)
and if issues like the above come up, someone can work on those
without interfering with the test_* file work since the problem lies
elsewhere in sympy. The issues list could be populated with each of
the test_* files that doesn't work for starters and that way we would
know who is working on which file since a person would note that they
are working on a given test_ file. Then as problems like the above
come up, those would be blocking issues that could be processed by
someone else...perhaps even the person that identified the test, but
at least we could divide up the work in a logical way.

Is this a possibility? Does anyone have other ideas?

[Aaron S. Meurer]

In analysis, a function is bounded if there exists a real number K
such that for all x
in the domain of the function |f(x)|<=K. Therefore, both (0, oo) and
(oo, oo) are unbounded.

There is an alternative formulation, where there is bounded above (f
(x) <= K) and bounded below (f(x) >= k, K and k are real numbers).
Then a functions is called just "bounded" if it is both bounded above
and below, and unbounded if it is not bounded.

I don't know where bounded is used. Would Q.bounded_above and
Q.bounded_below be useful at all?

> If we can set up an issues page then different persons can work
> through the different test files (like I am on test_arit.py right now)
> and if issues like the above come up, someone can work on those
> without interfering with the test_* file work since the problem lies
> elsewhere in sympy. The issues list could be populated with each of
> the test_* files that doesn't work for starters and that way we would
> know who is working on which file since a person would note that they
> are working on a given test_ file. Then as problems like the above
> come up, those would be blocking issues that could be processed by
> someone else...perhaps even the person that identified the test, but
> at least we could divide up the work in a logical way.
>
> Is this a possibility? Does anyone have other ideas?
>

Also, IRC is a good place for collaboration. I may not be able to do
much coding right now because I a swamped with work, but I can offer
advice and debugging help if I am online.


Aaron Meurer
> >

[Vinzent Steinberg]

On Nov 5, 4:49 pm, "Aaron S. Meurer" <asmeu...@gmail.com> wrote:
> On Nov 5, 2009, at 3:36 AM, smichr wrote:
> > I would recommend that we set up an issues page dedicated to this
> > conversion process for this reason:
>
> > as I work through getting the tests to work, I can identify things
> > that should work that don't that aren't part of the file that I am
> > working on. Ronan, for example, got the And and Or working and that
> > allowed a bunch of tests in test_arit to not raise that error. And now
> > I've identified two more issues that need to work but someone else
> > could tackle these:

[...]

> > If we can set up an issues page then different persons can work
> > through the different test files (like I am on test_arit.py right now)
> > and if issues like the above come up, someone can work on those
> > without interfering with the test_* file work since the problem lies
> > elsewhere in sympy. The issues list could be populated with each of
> > the test_* files that doesn't work for starters and that way we would
> > know who is working on which file since a person would note that they
> > are working on a given test_ file. Then as problems like the above
> > come up, those would be blocking issues that could be processed by
> > someone else...perhaps even the person that identified the test, but
> > at least we could divide up the work in a logical way.
>
> > Is this a possibility? Does anyone have other ideas?

Just use the assumptions tag that already exists, there is no need for
a dedicated issues page.

> Also, IRC is a good place for collaboration.  I may not be able to do  
> much coding right now because I a swamped with work, but I can offer  
> advice and debugging help if I am online.

In my experience debugging is usually much more time consuming than
coding. :) Am I doing it wrong?

Vinzent

[Aaron S. Meurer]

So now that 0.6.6 is released, I think we should make this top priority. The thing Chris and I are discussing on IRC is what is the best way to collaborate on this with git so that we don't duplicate each other's work. Also, can anyone give a clear list of what needs to be done? Does anyone other than Fabian really know this?

Not if it just "hey, can you verify that it does this weird thing on your machine too". But now I am off until the end of January, so I can do real work too. :)

Aaron Meurer
>
> Vinzent
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To post to this group, send email to sy...@googlegroups.com
> To unsubscribe from this group, send email to sympy+un...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sympy?hl=en
> -~----------~----~----~----~------~----~------~--~---
>

[fab...@fseoane.net]

On Mon, 21 Dec 2009 23:04:08 -0700, "Aaron S. Meurer" <asme...@gmail.com>
wrote:

> So now that 0.6.6 is released, I think we should make this top priority.
> The thing Chris and I are discussing on IRC is what is the best way to
> collaborate on this with git so that we don't duplicate each other's
work.
> Also, can anyone give a clear list of what needs to be done? Does anyone
> other than Fabian really know this?
>

I am glad there is interest in this. I've been busy lately moving to Paris,
but now I am settled so I'll get some time to work on this.

If anyone has patches on this, please give me your git repo and I'll review
& merge. Other, the place for discussion is task 1047
(http://code.google.com/p/sympy/issues/detail?id=1047), I'll try to answer
all questions that arise and post my progress there.

> --

>
> You received this message because you are subscribed to the Google Groups
> "sympy" group.

> To post to this group, send email to sy...@googlegroups.com.

> To unsubscribe from this group, send email to

> sympy+un...@googlegroups.com.

> For more options, visit this group at

> http://groups.google.com/group/sympy?hl=en.

[Ronan Lamy]

Le mardi 22 décembre 2009 à 01:29 -0700, fab...@fseoane.net a écrit :
> On Mon, 21 Dec 2009 23:04:08 -0700, "Aaron S. Meurer" <asme...@gmail.com>
> wrote:
> > So now that 0.6.6 is released, I think we should make this top priority.
> > The thing Chris and I are discussing on IRC is what is the best way to
> > collaborate on this with git so that we don't duplicate each other's
> work.
> > Also, can anyone give a clear list of what needs to be done? Does anyone
> > other than Fabian really know this?
> >

I would say that directly swapping out the old assumption system for the
new has failed, and that we therefore need to duplicate with the new one
everything the old one does.

> I am glad there is interest in this. I've been busy lately moving to Paris,
> but now I am settled so I'll get some time to work on this.

With the holiday season, I won't be able to do much in the next 2 weeks.
I'll be in Paris next week (if I don't freeze to death while attempting
to cross the Channel), maybe some face-to-face discussion could be
useful?

Ronan

[Ondrej Certik]

Yes, you should definitely meet and work together. It's very effective.

Ondrej