Python 3.6 tests depending on external library
80 views
Skip to first unread message
Francesco Bonazzi
Jul 13, 2017, 9:22:40 AM7/13/17
to sympy
Hi,
As part of the Rubi GSoC project, the two students are importing the code from Mathematica using the MatchPy library, which provides support for Mathematica's associative and commutative pattern matchings.
So far the code hasn't been merged into SymPy, given the dependence on MatchPy, and the fact that MatchPy only supports Python 3.6.
I was wondering whether it makes sense to have a new test supporting this external dependency on Python 3.6.
I see that supporting Python 3.6 only isn't optimal and should eventually be fixed, but it could work as a temporary measure.
Can a new Travis test for the Rubi module only make sense?
What do you think?
Aaron Meurer
Jul 13, 2017, 10:25:54 AM7/13/17
to sy...@googlegroups.com
Hi.
Manuel Krebber is here at SciPy. I've talked with him a little bit
already, and will likely talk more on Saturday and Sunday during the
sprints. I'm hoping we can get something that works in all versions of
Python we support (this is important).
I discussed the possibility of a pre-compiled Python module, which
once compiled has no dependencies, so that matchpy is only required if
the rules are updated. If you have anything else you'd like to
discuss, let me know.
Regarding the question at hand, I'd like to know how far off a more
general solution is (especially one that supports all versions of
Python). But I also want Rubi to start getting into the main repo as
soon as possible. If it is far off, your method may be workable. But
I'd like to have it better at least by the time the code is released.
Aaron Meurer
> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/4f740c6c-b06c-42e7-89d0-dbd2d87a723d%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Manuel Krebber is here at SciPy. I've talked with him a little bit
already, and will likely talk more on Saturday and Sunday during the
sprints. I'm hoping we can get something that works in all versions of
Python we support (this is important).
I discussed the possibility of a pre-compiled Python module, which
once compiled has no dependencies, so that matchpy is only required if
the rules are updated. If you have anything else you'd like to
discuss, let me know.
Regarding the question at hand, I'd like to know how far off a more
general solution is (especially one that supports all versions of
Python). But I also want Rubi to start getting into the main repo as
soon as possible. If it is far off, your method may be workable. But
I'd like to have it better at least by the time the code is released.
Aaron Meurer
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/4f740c6c-b06c-42e7-89d0-dbd2d87a723d%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Francesco Bonazzi
Jul 13, 2017, 11:36:47 AM7/13/17
to sympy
Hi Aaron,
Arihant was able to use the 3to2 tool to port MatchPy to Python 2.7, and it looked like it was working. We are actually waiting for some extensions to MatchPy (support for optional wildcards), as RUBI uses them extensively. As a temporary workaround, we are generating
I also believe that a code generator (generating the decision tree) would be the best idea to solve this problem. But I don't know whether Manuel is going to add it to MatchPy.
Arihant was able to use the 3to2 tool to port MatchPy to Python 2.7, and it looked like it was working. We are actually waiting for some extensions to MatchPy (support for optional wildcards), as RUBI uses them extensively. As a temporary workaround, we are generating
I also believe that a code generator (generating the decision tree) would be the best idea to solve this problem. But I don't know whether Manuel is going to add it to MatchPy.
Francesco Bonazzi
Jul 13, 2017, 11:38:07 AM7/13/17
to sympy
On Thursday, 13 July 2017 11:36:47 UTC-4, Francesco Bonazzi wrote:
As a temporary workaround, we are generating
As a temporary workaround, we are generating the power set of all optional wildcards in a single rule, replacing them with the default matching value each time.
Aaron Meurer
Jul 13, 2017, 11:41:31 AM7/13/17
to sy...@googlegroups.com
That's great. I asked Manuel if he had heard of 3to2 and he hadn't.
That's great if it already works out of the box. I think there is
also pasteurize
http://python-future.org/automatic_conversion.html#pasteurize-py3-to-py2-3,
which is probably a better option if the goal is long term support.
He did express interest in this code generator idea. I think it can be
added, although I wasn't clear how much work it would be.
Aaron Meurer
On Thu, Jul 13, 2017 at 11:36 AM, Francesco Bonazzi
> https://groups.google.com/d/msgid/sympy/13ca03eb-8fc7-473e-8001-465b0e529bc9%40googlegroups.com.
That's great if it already works out of the box. I think there is
also pasteurize
http://python-future.org/automatic_conversion.html#pasteurize-py3-to-py2-3,
which is probably a better option if the goal is long term support.
He did express interest in this code generator idea. I think it can be
added, although I wasn't clear how much work it would be.
Aaron Meurer
On Thu, Jul 13, 2017 at 11:36 AM, Francesco Bonazzi
Francesco Bonazzi
Jul 13, 2017, 12:48:24 PM7/13/17
to sympy
I think that the current priority is to make the rules work. RUBI has a lot of utility functions that need to be translated.
So far, it looks like the rules that have been ported are much slower than on Mathematica, but this is to be expected, as Mathematica is better optimized than Python.
Arihant Parsoya
Jul 13, 2017, 12:52:24 PM7/13/17
to sympy
Hi Aaron and Francesco,
I converted most code of MatchPy to python 2. However, MatchPy has dependency on few other python3.6 libraries(such as typing). Backport version of those libraries is available online. I found that 1-2 matchpy test cases were failing in python2.7 but all were running in python3 and above.
Regarding optional wildcards, we are generating multiple patterns to account for all possible matches of a pattern. However, the number of additonal rules is very large (~150 rules in mathematica can turn out to be 26863). Total filesize of all the rubi rules can most likely exceed 100mb. Manuel said he is working to implement optional arguments in MatchPy but couldn't confirm the timeline.
Arihant
Ondřej Čertík
Jul 15, 2017, 10:29:23 AM7/15/17
to sympy
Hi Arihant,
On Thu, Jul 13, 2017 at 10:52 AM, Arihant Parsoya
<parsoya...@gmail.com> wrote:
> Hi Aaron and Francesco,
>
> I converted most code of MatchPy to python 2. However, MatchPy has
> dependency on few other python3.6 libraries(such as typing). Backport
> version of those libraries is available online. I found that 1-2 matchpy
> test cases were failing in python2.7 but all were running in python3 and
> above.
In the Python 2 version of matchpy, you said only 2 matchpy tests
fail. How about sympy tests? Do all your Rubi tests succeed in Python
2 with your setup?
If so, Then one option is to get your setup into Travis for Python 2,
and that way you have something that passes all SymPy tests in Python
2 and Python 3. I would think that's good enough as a start. Then you
go from there.
Once it is in master, other people can help improve the situation.
If the above is not sufficient to get into master, then another option
is to create a branch called "master-rubi", which will be treated just
like a master. The only difference will be that the Travis tests only
test Python 3.6 and Python 2 as above, and you ask people to help out
with that, to bring the quality of the master-rubi branch on par with
master, so that it can be merged.
Ondrej
On Thu, Jul 13, 2017 at 10:52 AM, Arihant Parsoya
<parsoya...@gmail.com> wrote:
> Hi Aaron and Francesco,
>
> I converted most code of MatchPy to python 2. However, MatchPy has
> dependency on few other python3.6 libraries(such as typing). Backport
> version of those libraries is available online. I found that 1-2 matchpy
> test cases were failing in python2.7 but all were running in python3 and
> above.
fail. How about sympy tests? Do all your Rubi tests succeed in Python
2 with your setup?
If so, Then one option is to get your setup into Travis for Python 2,
and that way you have something that passes all SymPy tests in Python
2 and Python 3. I would think that's good enough as a start. Then you
go from there.
Once it is in master, other people can help improve the situation.
If the above is not sufficient to get into master, then another option
is to create a branch called "master-rubi", which will be treated just
like a master. The only difference will be that the Travis tests only
test Python 3.6 and Python 2 as above, and you ask people to help out
with that, to bring the quality of the master-rubi branch on par with
master, so that it can be merged.
Ondrej
Francesco Bonazzi
Jul 15, 2017, 1:29:28 PM7/15/17
to sympy
On Saturday, 15 July 2017 10:29:23 UTC-4, Ondřej Čertík wrote:
Hi Arihant,
On Thu, Jul 13, 2017 at 10:52 AM, Arihant Parsoya
If so, Then one option is to get your setup into Travis for Python 2,
and that way you have something that passes all SymPy tests in Python
2 and Python 3. I would think that's good enough as a start. Then you
go from there.
I would start by creating a Python 3.6 module only. It is taking some time to actually port the RUBI tests, so I would put most of the priority on porting the rules and making RUBI work on Python 3.6.
As soon as RUBI will work on Python 3.6, we can think about how to support other Python versions.
In my opinion, we should neglect the 3to2 tool for now. We can go back to the possibility of using the 3to2 tool once the porting of the rules is complete. However, I still feel that the right way to go is by generating the decision tree out of the discrimination net in MatchPy.
Aaron Meurer
Jul 15, 2017, 5:59:53 PM7/15/17
to sy...@googlegroups.com
I would be fine with a 3.6-only dependency just for adding new rules.
That is, if we can commit a pre-compiled decision tree file that works
on 2.7-3.6 and doesn't depend on MatchPy. Then MatchPy is only needed
if the rules get updated, or if someone wants to add a custom rule at
runtime.
Aaron Meurer
That is, if we can commit a pre-compiled decision tree file that works
on 2.7-3.6 and doesn't depend on MatchPy. Then MatchPy is only needed
if the rules get updated, or if someone wants to add a custom rule at
runtime.
Aaron Meurer
> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sympy/69b61ab7-5950-416a-9076-b4869f3cfb13%40googlegroups.com.
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.
> To view this discussion on the web visit
Francesco Bonazzi
Jul 15, 2017, 9:07:21 PM7/15/17
to sympy
OK, shall we add some new Travis tests for this module only? How would you do that?
Ondřej Čertík
Jul 16, 2017, 8:25:14 AM7/16/17
to sympy
generate the decision tree.
Ondrej
Arihant Parsoya
Jul 16, 2017, 10:04:43 AM7/16/17
to sympy
Hi,
I think we should also include parser which converts rules from Mathematica Downvalues to MatchPy. If we want to add new rules, we can write it in Mathematica, and parse them using the parser.
Arihant
Aaron Meurer
Jul 16, 2017, 10:37:21 AM7/16/17
to sy...@googlegroups.com
That will be nice if Rubi updates its rules. But the other use-case I
was thinking of is if a SymPy user has an integral that SymPy cannot
do, but they know how to integrate it, they can add a rule to the
integrator. In that case, they will most likely want to write the rule
in some Python/SymPy syntax, not Mathematica.
Regarding Travis, there is already a 3.6 build. You should configure
them to only run and only import matchpy when sys.version_info >= (3,
6). The code that goes in SymPy itself should be 2.7 compatible so
that it can still be imported (even if it runs nothing).
Aaron Meurer
was thinking of is if a SymPy user has an integral that SymPy cannot
do, but they know how to integrate it, they can add a rule to the
integrator. In that case, they will most likely want to write the rule
in some Python/SymPy syntax, not Mathematica.
Regarding Travis, there is already a 3.6 build. You should configure
them to only run and only import matchpy when sys.version_info >= (3,
6). The code that goes in SymPy itself should be 2.7 compatible so
that it can still be imported (even if it runs nothing).
Aaron Meurer
> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sympy/05e60112-af23-465b-bfa1-bb5a23d2b0cd%40googlegroups.com.
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.
> To view this discussion on the web visit
Arihant Parsoya
Jul 22, 2017, 7:40:29 AM7/22/17
to sympy
Rubi test suit take hours to complete. Are we going to add the test suit in sympy code after the rubi module is complete? If we do, then all the travis test will take longer time to pass.
Aaron Meurer
Jul 22, 2017, 4:46:49 PM7/22/17
to sy...@googlegroups.com
That's pretty bad. Is it because of the number of tests or the performance of the integrator? If the performance is bad, that's something that should be looked into.
Aaron Meurer
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscribe@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at https://groups.google.com/group/sympy.
To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/9176da3a-728c-4c33-84e3-2821080ce19a%40googlegroups.com.
Arihant Parsoya
Jul 23, 2017, 11:08:53 AM7/23/17
to sy...@googlegroups.com
The slowness of tests is primarily due to:
1. We need to simplify, expand or differentiate the integrated expression in order to equate them. I used numerical evaluation at random points for few tests and it gave faster results. I haven't tried using numerical evaluation for the whole tests yet.
2. There is a Rubi utility function called `ExpandIntegrand`. It uses sympys `apart` function which is slow for large functions(Example apart(1/(x**2*(a + b*x)**10), x) takes minutes to compute).
--
You received this message because you are subscribed to a topic in the Google Groups "sympy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sympy/MyeWvJ8t7F8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sympy+unsubscribe@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at https://groups.google.com/group/sympy.
To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6Kehi3wx4gEqoeQvuN6FbGRAKFDBLOJPoaRVqgpA%3DLSAA%40mail.gmail.com.
Regards,
Arihant Parsoya,
Junior Undergraduate,
Dept of Civil Engineering,
IIT Bombay
IIT Bombay
Aaron Meurer
Jul 23, 2017, 2:36:56 PM7/23/17
to sy...@googlegroups.com
Hi
I recommend using profiling tools such as pyinstrument and snakeviz to see if there are any obvious ways we can improve the performance.
Aaron Meurer
To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/9176da3a-728c-4c33-84e3-2821080ce19a%40googlegroups.com.
--
You received this message because you are subscribed to a topic in the Google Groups "sympy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sympy/MyeWvJ8t7F8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at https://groups.google.com/group/sympy.
To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6Kehi3wx4gEqoeQvuN6FbGRAKFDBLOJPoaRVqgpA%3DLSAA%40mail.gmail.com.
--Regards,Arihant Parsoya,Junior Undergraduate,Dept of Civil Engineering,
IIT Bombay
--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at https://groups.google.com/group/sympy.
To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAJTzyWy2drf9yLOJ119%2BobTeuvQ%3DUXZqzVpjJ4PLtpUbuJ4kjA%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages