SymPEP: make MatchPy a SymPy dependency

[Francesco Bonazzi]

Hi everyone,

I have written a draft for a SymPEP (SymPy enhancement proposal) to include MatchPy as a dependency of SymPy.

https://github.com/sympy/SymPEPs/pull/3

Once SymPy depends on the MatchPy library, the bindings to MatchPy can be moved into SymPy's core.

MatchPy provides a much more powerful pattern matcher than the current one implemented in SymPy's core. In particular:

• it can match multiple patterns at the same time and very efficiently (SymPy's matcher can only process one pattern at a time),
• it can generate a decision tree in Python out of multiple patterns (SymEngine has an implementation to generate a C++ decision tree out of the same patterns).

Feel free to join the discussion either here or on the [SymPEP Pull Request](https://github.com/sympy/SymPEPs/pull/3)

[Oscar Benjamin]

On Tue, 5 Oct 2021 at 07:48, Francesco Bonazzi <franz....@gmail.com> wrote:

Hi everyone,

I have written a draft for a SymPEP (SymPy enhancement proposal) to include MatchPy as a dependency of SymPy.

https://github.com/sympy/SymPEPs/pull/3

For those less familiar to github you need to click the "files changed" tab to actually see the text of the SymPEP. A direct link is:

https://github.com/sympy/SymPEPs/pull/3/files

 

Once SymPy depends on the MatchPy library, the bindings to MatchPy can be moved into SymPy's core.

MatchPy provides a much more powerful pattern matcher than the current one implemented in SymPy's core. In particular:

• it can match multiple patterns at the same time and very efficiently (SymPy's matcher can only process one pattern at a time),
• it can generate a decision tree in Python out of multiple patterns (SymEngine has an implementation to generate a C++ decision tree out of the same patterns).

Feel free to join the discussion either here or on the [SymPEP Pull Request](https://github.com/sympy/SymPEPs/pull/3)

I think it's best to discuss this here rather than on the PR apart from discussion of minor edits.

I agree that pattern matching is a crucial part of a CAS and should really be the core of SymPy. If I was redesigning SymPy from scratch then everything would be built on top of a pattern-matching engine and almost all of the logic from the myriad Basic subclasses would just be written as pattern rules. My equivalent of SymEngine would just be a C implementation of the pattern-matching engine rather than a repeat of the object-oriented design of SymPy.

I also agree that matchpy looks like a good implementation of pattern-matching and it makes sense for it to be usable with SymPy. What the SymPEP does not address though is what benefit is gained for users by making matchpy a non-optional dependency. The examples shown look like somewhat specialised usage for which an interested user could just choose to install matchpy.

RUBI would be a good motivation but as far as I can tell RUBI does not yet work. Actually I would prefer it if RUBI was already in a separate package from SymPy - it should not have been merged until it was at least partially working. The rubi_tests folder is included for all users and includes e.g. the sympy/integrals/rubi/rubi_tests/tests/test_trinomials.py file which wastes at least 1.5MB disk space for every single SymPy user for precisely zero benefit (these tests should clearly be in a separate repo). I don't see how making matchpy a non-optional dependency would make it any easier to improve RUBI since anyone who wants to work on it can just install matchpy. In fact if RUBI was not in the main SymPy repo then it could have a hard dependency on matchpy.

The cost of making matchpy a non-optional dependency would be felt by downstream distributors of SymPy who would then have an additional dependency to include. It would also be felt by all users of SymPy with longer install time, more disk space etc. If a user does not use any of it's features then this cost comes with no benefit.

I am not saying this to disagree with the proposal but that there needs to be a clear rationale for making matchpy a hard dependency and the SymPEP does not address this at all. The SymPEP should also clearly spell out what the downsides of the proposal are.

--

Oscar

[S.Y. Lee]

Although the matchpy gives of AC1-matching capability as far as I'm aware, I just have a question that if is it really general enough. 
I'm aware that general equational matching could be an undecidable problem,
and hence there are lots of matchers or unifiers that handles some domain-specific E-matching problems
(for other cases like boolean algebra, distributivity, or some other combinations of equational identities)
and that have to be implemented separately (even there can be some distinguish for one-to-one and many matching because of efficiency)

[David Bailey]

On 05/10/2021 13:06, Oscar Benjamin wrote:

I agree that pattern matching is a crucial part of a CAS and should really be the core of SymPy. If I was redesigning SymPy from scratch then everything would be built on top of a pattern-matching engine and almost all of the logic from the myriad Basic subclasses would just be written as pattern rules. My equivalent of SymEngine would just be a C implementation of the pattern-matching engine rather than a repeat of the object-oriented design of SymPy.

I also agree that matchpy looks like a good implementation of pattern-matching and it makes sense for it to be usable with SymPy. What the SymPEP does not address though is what benefit is gained for users by making matchpy a non-optional dependency. The examples shown look like somewhat specialised usage for which an interested user could just choose to install matchpy.

It must be a little sad to discover that too late - do you have any idea how much speed up that would have produced?

RUBI would be a good motivation but as far as I can tell RUBI does not yet work. Actually I would prefer it if RUBI was already in a separate package from SymPy - it should not have been merged until it was at least partially working. The rubi_tests folder is included for all users and includes e.g. the sympy/integrals/rubi/rubi_tests/tests/test_trinomials.py file which wastes at least 1.5MB disk space for every single SymPy user for precisely zero benefit (these tests should clearly be in a separate repo). I don't see how making matchpy a non-optional dependency would make it any easier to improve RUBI since anyone who wants to work on it can just install matchpy. In fact if RUBI was not in the main SymPy repo then it could have a hard dependency on matchpy.

I think SymPy has resisted the pressure to bloat extremely well. The last time I downloaded a version of Mathematica it was something over 3GB in size! I am all in favour of not adding stuff that isn't needed just because large 'disks' are so common. Irrelevant code also makes it harder to explore the SymPy source.

David

[Aaron Meurer]

Since we are bringing up SymPEPs again, it would be helpful to agree
on the actual SymPEP process itself. There hasn't been much discussion
on https://github.com/sympy/SymPEPs/pull/2 for a while.

I was also hoping we could have a general PEP about dependencies and
conditions for when something should be added as a hard or optional
dependency.

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/01a6a980-3267-4ba6-82ca-adb06da994bdn%40googlegroups.com.

[Francesco Bonazzi]

On Tuesday, October 5, 2021 at 2:07:06 p.m. UTC+2 Oscar wrote:

I agree that pattern matching is a crucial part of a CAS and should really be the core of SymPy. If I was redesigning SymPy from scratch then everything would be built on top of a pattern-matching engine and almost all of the logic from the myriad Basic subclasses would just be written as pattern rules. My equivalent of SymEngine would just be a C implementation of the pattern-matching engine rather than a repeat of the object-oriented design of SymPy.

SymPy can still be modified in order to use a pattern matcher.

Also, MatchPy has been partly translated into C++ inside the SymEngine project: 

https://github.com/symengine/symengine/tree/329a04be017daff0362b9177da2ef5b7e5d605f7/symengine/utilities/matchpycpp

 

I also agree that matchpy looks like a good implementation of pattern-matching and it makes sense for it to be usable with SymPy. What the SymPEP does not address though is what benefit is gained for users by making matchpy a non-optional dependency. The examples shown look like somewhat specialised usage for which an interested user could just choose to install matchpy.

The benefits are mostly for SymPy development. You can start defining replacement rules to implement new algorithms.

RUBI would be a good motivation but as far as I can tell RUBI does not yet work. Actually I would prefer it if RUBI was already in a separate package from SymPy - it should not have been merged until it was at least partially working. The rubi_tests folder is included for all users and includes e.g. the sympy/integrals/rubi/rubi_tests/tests/test_trinomials.py file which wastes at least 1.5MB disk space for every single SymPy user for precisely zero benefit (these tests should clearly be in a separate repo). I don't see how making matchpy a non-optional dependency would make it any easier to improve RUBI since anyone who wants to work on it can just install matchpy. In fact if RUBI was not in the main SymPy repo then it could have a hard dependency on matchpy.

 

This can be done.

 

The cost of making matchpy a non-optional dependency would be felt by downstream distributors of SymPy who would then have an additional dependency to include. It would also be felt by all users of SymPy with longer install time, more disk space etc. If a user does not use any of it's features then this cost comes with no benefit.

Consider that it's pretty common now for libraries to depend on other libraries. An alternative would be to copy the MatchPy project as a subfolder inside SymPy (as it has been done for the multipledispatch module).

 

I am not saying this to disagree with the proposal but that there needs to be a clear rationale for making matchpy a hard dependency and the SymPEP does not address this at all. The SymPEP should also clearly spell out what the downsides of the proposal are.

The major downside is probably that MatchPy has not been extensively tested with SymPy. There is a risk factor in relying on a new library.

[Jason Moore]

I think we should relax our stringent no dependency stance for pure python dependencies. Pure python dependency management is essentially solved now for the python ecosystem with the various package managers. This dependency would also be pulled into the sympy organization, so we will have just as much control to maintain it as code in sympy's codebase.

Francesco is a long time valued contributor and if he wants/needs matchpy to help him make sympy better, then we should enable him to do that. Putting up a wall here likely only creates frustration and demotivation. There is likely more value in unleashing Francesco's matchpy improvements to sympy than any devalue adding a dependency to sympy creates.

Jason

moorepants.info
+01 530-601-9791

--

You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/003f5fd9-8137-40eb-a8a6-ca2378a08ccan%40googlegroups.com.

[Francesco Bonazzi]

On Wednesday, October 6, 2021 at 12:46:25 a.m. UTC+2 asme...@gmail.com wrote:

Since we are bringing up SymPEPs again, it would be helpful to agree
on the actual SymPEP process itself. There hasn't been much discussion
on https://github.com/sympy/SymPEPs/pull/2 for a while.

I was also hoping we could have a general PEP about dependencies and
conditions for when something should be added as a hard or optional
dependency.

The problem with SymPEP is that we will rarely use them. I mean, it's quite rare to have a proposal for a major change in SymPy.

[Francesco Bonazzi]

On Tuesday, October 5, 2021 at 2:57:41 p.m. UTC+2 syle...@gmail.com wrote:

Although the matchpy gives of AC1-matching capability as far as I'm aware, I just have a question that if is it really general enough. 
I'm aware that general equational matching could be an undecidable problem,
and hence there are lots of matchers or unifiers that handles some domain-specific E-matching problems
(for other cases like boolean algebra, distributivity, or some other combinations of equational identities)
and that have to be implemented separately (even there can be some distinguish for one-to-one and many matching because of efficiency)

 I'm not familiar with the decidability of pattern matchers.

MatchPy is simply creating a lookup data structure (I refer to the many-to-one matcher, they also have one-to-one and discrimintation nets), recurring through the target expression and lazily walk through all partitions (for associativity) or subsets (for commutativity) until a matching subexpression is found.

Their paper is here:

https://arxiv.org/abs/1710.06915

[Oscar Benjamin]

On Wed, 6 Oct 2021 at 08:52, Jason Moore <moore...@gmail.com> wrote:

I think we should relax our stringent no dependency stance for pure python dependencies. Pure python dependency management is essentially solved now for the python ecosystem with the various package managers. This dependency would also be pulled into the sympy organization, so we will have just as much control to maintain it as code in sympy's codebase.

I don't that there should be a no dependency stance. In fact I think that it was a mistake previously to not use dependencies like multipledispatch and pytest. Maintaining sympy is hard enough without needing to maintain out of date forks of otherwise independently maintained projects.

 

Francesco is a long time valued contributor and if he wants/needs matchpy to help him make sympy better, then we should enable him to do that. Putting up a wall here likely only creates frustration and demotivation. There is likely more value in unleashing Francesco's matchpy improvements to sympy than any devalue adding a dependency to sympy creates.

I did not say that I opposed the proposal just that there needs to be some justification for it in the SymPEP. The purpose of that document is to explain the rationale for changes so that a decision can be considered on its merits and also to communicate to others why a decision has been made. As it stands though the document does not address reasons for what is actually proposed which is to make matchpy a hard dependency. If there are good reasons for doing that then they should be written in the SymPEP. Partly those reasons might be speculative but they can still be listed.

--

Oscar

 

Jason

moorepants.info
+01 530-601-9791

On Wed, Oct 6, 2021 at 9:38 AM Francesco Bonazzi <franz....@gmail.com> wrote:

On Tuesday, October 5, 2021 at 2:07:06 p.m. UTC+2 Oscar wrote:

I agree that pattern matching is a crucial part of a CAS and should really be the core of SymPy. If I was redesigning SymPy from scratch then everything would be built on top of a pattern-matching engine and almost all of the logic from the myriad Basic subclasses would just be written as pattern rules. My equivalent of SymEngine would just be a C implementation of the pattern-matching engine rather than a repeat of the object-oriented design of SymPy.

SymPy can still be modified in order to use a pattern matcher.

Also, MatchPy has been partly translated into C++ inside the SymEngine project: 

https://github.com/symengine/symengine/tree/329a04be017daff0362b9177da2ef5b7e5d605f7/symengine/utilities/matchpycpp

 

I also agree that matchpy looks like a good implementation of pattern-matching and it makes sense for it to be usable with SymPy. What the SymPEP does not address though is what benefit is gained for users by making matchpy a non-optional dependency. The examples shown look like somewhat specialised usage for which an interested user could just choose to install matchpy.

The benefits are mostly for SymPy development. You can start defining replacement rules to implement new algorithms.

RUBI would be a good motivation but as far as I can tell RUBI does not yet work. Actually I would prefer it if RUBI was already in a separate package from SymPy - it should not have been merged until it was at least partially working. The rubi_tests folder is included for all users and includes e.g. the sympy/integrals/rubi/rubi_tests/tests/test_trinomials.py file which wastes at least 1.5MB disk space for every single SymPy user for precisely zero benefit (these tests should clearly be in a separate repo). I don't see how making matchpy a non-optional dependency would make it any easier to improve RUBI since anyone who wants to work on it can just install matchpy. In fact if RUBI was not in the main SymPy repo then it could have a hard dependency on matchpy.

 

This can be done.

 

The cost of making matchpy a non-optional dependency would be felt by downstream distributors of SymPy who would then have an additional dependency to include. It would also be felt by all users of SymPy with longer install time, more disk space etc. If a user does not use any of it's features then this cost comes with no benefit.

Consider that it's pretty common now for libraries to depend on other libraries. An alternative would be to copy the MatchPy project as a subfolder inside SymPy (as it has been done for the multipledispatch module).

 

I am not saying this to disagree with the proposal but that there needs to be a clear rationale for making matchpy a hard dependency and the SymPEP does not address this at all. The SymPEP should also clearly spell out what the downsides of the proposal are.

The major downside is probably that MatchPy has not been extensively tested with SymPy. There is a risk factor in relying on a new library.

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/003f5fd9-8137-40eb-a8a6-ca2378a08ccan%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AhfqzVwWoqbZosP_uXoM%2BJkw9z5LuvAtXKi0feb5zg8vQ%40mail.gmail.com.

[Aaron Meurer]

On Wed, Oct 6, 2021 at 1:52 AM Jason Moore <moore...@gmail.com> wrote:
>
> I think we should relax our stringent no dependency stance for pure python dependencies. Pure python dependency management is essentially solved now for the python ecosystem with the various package managers. This dependency would also be pulled into the sympy organization, so we will have just as much control to maintain it as code in sympy's codebase.

There's more cost to adding a dependency than just the installation
questions, which I agree are not that big of an issue like they used
to be. In fact we already have had mpmath as a dependency for a while,
and it's proven to be just fine. The sorts of questions we need to ask
when adding a hard dependency are

- Who maintains the dependency?
- What happens if they stop maintaining it? What is the risk level of
that happening?
- Does the package have a healthy codebase? For example, does the
package use idiomatic Python? Does it have tests? Does it have
documentation?
- Does the package have a healthy community? For example, does it have
a contributor guide? Does it have a code of conduct (and is it
actually followed by the community)? Are active contributors regularly
granted push access to the project?
- With the above points, keep in mind that we would almost certainly
have to upstream bug fixes at various points. How easy will that be to
do?
- How active is the development on the project? If it's not active, is
that because there aren't many active maintainers or because the
project has stabilized? if it is very active, is it stable, e.g., are
there many API breaks?
- Are there other large projects that also depend on this dependency?
- What is the license of the dependency? Would the maintainers ever
change this license to a more restrictive one? (We should not have
hard dependencies that have more restrictive licenses than SymPy
itself).
- What are the tradeoffs of having the dependency vs. not having it?
"Not having it" includes the option of writing the desired
functionality directly in SymPy.
- If we decide that the package no longer is suitable to be a
dependency for one of the above reasons, what would our options be?
- What are the recursive dependencies of the dependency, and how do
the above questions apply to those?

I like this blog post by Russ Cox, which outlines some of these same
concerns. https://research.swtch.com/deps

Any one of these could represent a reason for not including a package
as a hard dependency. I've seen a lot of projects not really consider
the above issues very carefully and end up being bitten by them.
Consider, just as an example, the pymc project, which depended on
theano, and was forced to maintain a new fork of it when the theano
developers stopped maintaining it. These are the sorts of things I
would like to formalize so that we always make sure to think about
them.

Just to be clear, I'm not saying that these things are all bad for
matchpy, nor am I saying that we shouldn't add matchpy as a
dependency. I just want to make sure we actually consider these
things. Personally, when it comes to matchpy, I am most concerned
about the size of the maintainer pool, since it only appears to have
one or two active maintainers, which are part of an academic lab,
which implies that they would likely stop maintaining it once they
leave the lab. These are the sorts of concerns that should be
addressed in the SymPEP, and weighed if they are worth it.

>
> Francesco is a long time valued contributor and if he wants/needs matchpy to help him make sympy better, then we should enable him to do that. Putting up a wall here likely only creates frustration and demotivation. There is likely more value in unleashing Francesco's matchpy improvements to sympy than any devalue adding a dependency to sympy creates.

I also don't like unnecessary friction, which is why I felt for a long
time that formal processes like SymPEP weren't necessary for SymPy.
But we need to also consider SymPy's user base. The more people and
upstream projects that are using SymPy, the more we can't just "move
fast and break things". Significant changes should be made with care.
If they aren't, we will be forced to roll them back, meaning annyong
backwards incompatibilities for users.

Aaron Meurer

>
> Jason
> moorepants.info
> +01 530-601-9791
>
>
> On Wed, Oct 6, 2021 at 9:38 AM Francesco Bonazzi <franz....@gmail.com> wrote:
>>
>>
>>
>> On Tuesday, October 5, 2021 at 2:07:06 p.m. UTC+2 Oscar wrote:
>>>
>>>
>>> I agree that pattern matching is a crucial part of a CAS and should really be the core of SymPy. If I was redesigning SymPy from scratch then everything would be built on top of a pattern-matching engine and almost all of the logic from the myriad Basic subclasses would just be written as pattern rules. My equivalent of SymEngine would just be a C implementation of the pattern-matching engine rather than a repeat of the object-oriented design of SymPy.
>>
>>
>> SymPy can still be modified in order to use a pattern matcher.
>>
>> Also, MatchPy has been partly translated into C++ inside the SymEngine project:
>> https://github.com/symengine/symengine/tree/329a04be017daff0362b9177da2ef5b7e5d605f7/symengine/utilities/matchpycpp
>>
>>>
>>>
>>> I also agree that matchpy looks like a good implementation of pattern-matching and it makes sense for it to be usable with SymPy. What the SymPEP does not address though is what benefit is gained for users by making matchpy a non-optional dependency. The examples shown look like somewhat specialised usage for which an interested user could just choose to install matchpy.
>>
>>
>> The benefits are mostly for SymPy development. You can start defining replacement rules to implement new algorithms.
>>
>>>
>>> RUBI would be a good motivation but as far as I can tell RUBI does not yet work. Actually I would prefer it if RUBI was already in a separate package from SymPy - it should not have been merged until it was at least partially working. The rubi_tests folder is included for all users and includes e.g. the sympy/integrals/rubi/rubi_tests/tests/test_trinomials.py file which wastes at least 1.5MB disk space for every single SymPy user for precisely zero benefit (these tests should clearly be in a separate repo). I don't see how making matchpy a non-optional dependency would make it any easier to improve RUBI since anyone who wants to work on it can just install matchpy. In fact if RUBI was not in the main SymPy repo then it could have a hard dependency on matchpy.
>>
>>
>> This can be done.
>>
>>>
>>> The cost of making matchpy a non-optional dependency would be felt by downstream distributors of SymPy who would then have an additional dependency to include. It would also be felt by all users of SymPy with longer install time, more disk space etc. If a user does not use any of it's features then this cost comes with no benefit.
>>
>>
>> Consider that it's pretty common now for libraries to depend on other libraries. An alternative would be to copy the MatchPy project as a subfolder inside SymPy (as it has been done for the multipledispatch module).
>>
>>>
>>> I am not saying this to disagree with the proposal but that there needs to be a clear rationale for making matchpy a hard dependency and the SymPEP does not address this at all. The SymPEP should also clearly spell out what the downsides of the proposal are.
>>
>>
>> The major downside is probably that MatchPy has not been extensively tested with SymPy. There is a risk factor in relying on a new library.
>>
>> --
>> You received this message because you are subscribed to the Google Groups "sympy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/003f5fd9-8137-40eb-a8a6-ca2378a08ccan%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AhfqzVwWoqbZosP_uXoM%2BJkw9z5LuvAtXKi0feb5zg8vQ%40mail.gmail.com.

[Aaron Meurer]

I agree SymPEPs will be rare and indeed should be rare. Such a formal
process is only needed for major changes where we should try to avoid
making bad decisions. I don't see how this is a problem though. I
would say having such a formal process only be used rarely is a good
thing.

Aaron Meurer

>
> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/e0680e96-8268-4bf8-ba9e-9afc75d42697n%40googlegroups.com.

[Francesco Bonazzi]

It would be curious if we could rewrite SymPy's core using MatchPy. I mean, there have been some development with a lot of code developed to properly handle the subclassing (or .kind property, that has been introduced now) the Add and Mul classes, especially their flattening algorithm. Multiple dispatch was an initial idea, but it's not quite optimal to use.

I would like to point out that all of these issues are practically solved if we use MatchPy. The pattern matcher does not just dispatch on the types of the expressions, but also on the expression substructure.

More MatchPy rules could be loaded in other SymPy modules (for example, if you load sympy.matrices the rules for flattening on Add and Mul for matrices will be added). Postprocessors could then be done away with.

For efficiency, MatchPy can generate a Python decision tree performing the rule matching (note: you lose the possibility to dynamically load more rules once the code has been generated).

Furthermore, there is a code generator in SymEngine targeting C++, which could potentially be used to compile algorithms written for SymPy in Python with MatchPy's rules into C++ code compatible with SymEngine (one quick note, this module is still in its early stage).

[Oscar Benjamin]

On Wed, 6 Oct 2021 at 23:36, Francesco Bonazzi <franz....@gmail.com> wrote:

It would be curious if we could rewrite SymPy's core using MatchPy. I mean, there have been some development with a lot of code developed to properly handle the subclassing (or .kind property, that has been introduced now) the Add and Mul classes, especially their flattening algorithm. Multiple dispatch was an initial idea, but it's not quite optimal to use.

I would like to point out that all of these issues are practically solved if we use MatchPy. The pattern matcher does not just dispatch on the types of the expressions, but also on the expression substructure.

I do agree and this is more or less the direction I would like to go with SymPy. To me it seems that the fundamental problem is Basic and automatic evaluation: SymPy does not provide the right data structures for these algorithms to work properly. I don't think it's possible to make the changes needed for this kind of rewrite within SymPy itself without a massive compatibility break though. In fact even if it was considered acceptable to break compatibility to the extent needed I don't think that it would be tractable to try and make the changes directly in SymPy.

My approach would be to make a new library that redefines the core of SymPy itself and that uses matchpy or equivalent. Then if that library could be made compatible with SymPy through a mechanism like sympify then it would be possible to work incrementally on it until it was relatively complete. The new library could be unencumbered by backwards compatibility constraints and could fix the known design flaws by having first-class functions, exp as pow, no automatic evaluation, kinds and kind checking built in, match-based assumptions, better handling of singular cases in basic manipulation, ...

Then SymPy itself could be made to depend on that library (and therefore indirectly on matchpy) and use it internally while still allowing users and downstream libraries to work with their old Basic subclasses. At the point when a new core works and is usable we could consider what to do with old Basic. If SymPy already gets most of the benefit of the new core by using it internally in key routines like solve, integrate (RUBI), etc we can consider to what extent it is worth maintaining the old core. We could also consider at that point whether it's worth moving the new library into the main sympy codebase or if it's better just to keep it standalone.

If the new library and its approach with matchpy turns out not to be useful then no real harm would be done (apart from potentially wasting our time). The old core and all of the rest of it would still work, we wouldn't have added a dependency for something that wasn't actually needed and we wouldn't have broken any compatibility.

Basically I think that there needs to be a plan. I don't think we should make user-visible changes like breaking compatibility or adding dependencies until we're clear what the plan is and we're ready to provide some tangible benefit to users. My vision of the SymPEP process was not so much that it should be about making decisions but about discussing and documenting what we want to do and why, especially for situations where it will take a long time to implement the proposal and would need many people to work together.

I think that what we should do right now is set up a video call to discuss this rather than going back and forth here.

Do you want to meet and discuss this?

--

Oscar

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/61ab2404-c9f7-4eaf-9255-b790d89a298fn%40googlegroups.com.

[Francesco Bonazzi]

On Thursday, October 7, 2021 at 1:57:48 a.m. UTC+2 Oscar wrote:

My approach would be to make a new library that redefines the core of SymPy itself and that uses matchpy or equivalent. Then if that library could be made compatible with SymPy through a mechanism like sympify then it would be possible to work incrementally on it until it was relatively complete. The new library could be unencumbered by backwards compatibility constraints and could fix the known design flaws by having first-class functions, exp as pow, no automatic evaluation, kinds and kind checking built in, match-based assumptions, better handling of singular cases in basic manipulation, ...

An easier way, define some new classes like AddNew, MulNew, PowNew inheriting from Expr. Then, using sympy.utilities.matchpy_connector start defining the flattening rules. I think it should already be possible. The class Replacer in sympy.utilities.matchpy_connector has some usage examples in its docstring (it's nice to get familiar with the API).

Also, remember that we have the module sympy.sandbox to put unsupported and temporary experimental code into. That is useful for experiments like this one.