Enhancing the flexibility of MatchPy

[Samith Kavishke]

Hi,

Even though, I have engaged with few chats I thought to introduce me first. I am Samith Karunathilake, a third year undergraduate from University of Moratuwa, Sri Lanka. I have an interest on contributing for the Sympy for GSOC 2024. As for now I have baisc understanding about few areas, sympy.series, matchpy_connector and parse_mathematica.

Here are the few methods that I identified about matchpy restructuring.
Since, at the moment it uses an Expression node as the root node and we can redefine it and introduce some methods to check whether it is a instance of wildcard or an operation.
Through that we can change the tree traversal method without bothering about the implementation of the Wild Card and Operations.
If Francesco Bonazzi, or any potential mentor can give their idea on this thread, it would be much helpful. I will post more about this project as a reply for this thread.

Thanks in advance for your collaboration.

Samith Karunathilake.

[Francesco Bonazzi]

The idea that I have in mind requires forking the MatchPy library. If we can find a way to modify the tree-traversal algorithms in MatchPy in order not to depend on type checks, we will make MatchPy's integration into SymPy much easier.

[Samith Kavishke]

Hi Francesco,

Thank you for the reply. How should I progress in this project? I have several issues encountered when I am going through the matchpy repository, where would I raise those issues.

[Francesco Bonazzi]

The main issue is that we don't control the MatchPy repository. It's owned by a research center. Unfortunately they don't appear to be active anymore, so I guess we'll probably have to fork the project.

Regarding SymPy, our aim is to improve the integration between MatchPy and SymPy in order to use MatchPy's pattern matching and replacement algorithms inside SymPy. So the main use case is the implementation of algorithms in SymPy using MatchPy's algorithm.

For example, can we implement an equation solver using MatchPy? There is a basic example in the documentation of "matchpy_connector".

Can MatchPy be modified and made more flexible so that we can avoid using all evil tricks used in "matchpy_connector" to get it to work?

Can MatchPy be rewritten in Rust for more efficiency?

[Samith Kavishke]

Hi,

If we are maintianing matchpy as our own repository, Is it okay for us to make the matchpy elements in more integrated way to sympy, or still we need to support the every functionalities that matchpy gave earlier. As an example,  issue 69 suggests to create neutral or identity element, but for it to make a proper meaning in XML domain we need to make it as default value, likewise there are more things that we take into consideration if we need to support this for other types as well. Because somethings that supports in current version of matchpy would break when we add those functionalities. 

[Aaron Meurer]

As far as we can tell, SymPy is the only thing that uses MatchPy,
outside of the specific research software from that research group
that it was developed for. If making MatchPy more SymPy specific eases
the development burden, we should do that, especially if we are going
to fork it anyway. I think the core of it does need to remain
independent of SymPy's types, especially if we want to try to write a
Rust backend.

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/e25976d9-682f-4379-9001-961ab21be4a7n%40googlegroups.com.

[Francesco Bonazzi]

On Wednesday, March 13, 2024 at 1:29:37 a.m. UTC+1 samithkar...@gmail.com wrote:

Hi,

If we are maintianing matchpy as our own repository, Is it okay for us to make the matchpy elements in more integrated way to sympy, or still we need to support the every functionalities that matchpy gave earlier.

Well, I would try to keep MatchPy as much generalizable as possible. That library is very useful even outside of SymPy (many projects could benefit from it).

 

As an example,  issue 69 suggests to create neutral or identity element, but for it to make a proper meaning in XML domain we need to make it as default value,

If we ever address issue 69, I believe it would be proper to raise an exception if that issue does not apply to the case. For example, if you try to construct a neutral element on an XML tree, you should not be able to use a generic identity element. This is different if nodes are additions or multiplications.

 

likewise there are more things that we take into consideration if we need to support this for other types as well. Because somethings that supports in current version of matchpy would break when we add those functionalities. 

I think we can still add new functionalities and keep MatchPy fully working, we just have to do it carefully and make sure that all unit tests of MatchPy keep working.

[Francesco Bonazzi]

On Wednesday, March 13, 2024 at 8:13:27 a.m. UTC+1 Aaron Meurer wrote:

As far as we can tell, SymPy is the only thing that uses MatchPy,
outside of the specific research software from that research group
that it was developed for.

Indeed, MatchPy is probably very underappreciated. Its developers haven't really advertised it enough.

 

If making MatchPy more SymPy specific eases
the development burden, we should do that, especially if we are going
to fork it anyway. I think the core of it does need to remain
independent of SymPy's types, especially if we want to try to write a
Rust backend.

I would still make an attempt at keeping it generic. The main issue is removing all "isinstance( ... )" and "__iter__" statements in the expression tree traversal algorithms of MatchPy and replace them with more generalizable statements. I'm confident this can be done and that we can still keep MatchPy generalizable to other libraries.

[Samith Kavishke]

Hi,
Still I am in the process of understanding the matchpy code base.
at the moment I understand that,

• Isinstance logic should replace to overridable method or any preferred way ( wrappers around classes new classes also possible without changing the existing code). 
• __iter__ logic should change due to the tree traversal must change accordingly, because it is rely on the inheritance. 
  
• replace method in the matchpy connector should change because it is replication of same code twice.
• Enhance the capability of matchpy_connector, because still does not have the capability of solving calculus, matrices and more advanced problems.

what are the other things that might affect if we add overridable method to Expressions? any suggestions for me to refer.

[Francesco Bonazzi]

I suggest to start the unit tests of MatchPy in debug mode and follow the tree traversal to get some more insight. Indeed, the code is quite complicated, but the debugger may help. MatchPy also provides a tool to dump the state into a GraphViz dot file.

The idea is that matchpy_connector should not use all those ".register( )" methods, it should not have objects subclassing MatchPy objects (currently there is Wildcard).

[Samith Kavishke]

I think, I understood the point you are making. If we are going to redefine the tree structure in matchpy using overridable methods, does that mean we have to use a parser to translate the sympy object type to matchpy or are there any better way to handle the situation.  

[Francesco Bonazzi]

No, no parsers are needed. Just a tree traversal for-loop. It's already implemented:

• iterator
  
• iterator size

As you see, in the current implementation we need to use @op_iter.register... because "op_iter" is a MatchPy function that does not exist in SymPy... With @op_iter.register we can extend it to a SymPy object. The problem is:

1. @op_iter.register is an almost hackish way to do it, it messes up with the class inheritance. We need something easier to use.
2. The .register method defines the iterator on a class. We may have cases in which we would like to have different iterators for the same class... so it shouldn't be a class method.
   

[Samith Kavishke]

Hi, 
Thank you, Francesco for your valuable insights. I have attached my current draft of GSOC 24 proposal. Can you help me with feedbacks for the proposal. 

[Samith Kavishke]

Is it worth for me to mention about the Protosym's architecture, in order to strengthen my proposal ? or do I need more contributions in sympy repositories.

[Francesco Bonazzi]

The base idea is about modifying MatchPy in order to allow easy integration of MatchPy into SymPy and into the python bindings of protosym. Additional extensions to the project idea are welcome, provided there is enough time to complete them. Trying a translation of MatchPy into Rust has the potential of speeding up the library a lot, but it should be benchmarked.

[Francesco Bonazzi]

The issue I see in the proposal is that the methods should probably be functions, not class methods. If we keep class methods we are probably still forced to use subclassing (I would like to avoid that). Anyways, it looks good, this is just a minor issue 

[Samith Kavishke]

I will go through it and try to suggest a new method if time permits. Thank you Francesco you helped a lot to achieve this amount. 

[Samith Kavishke]

I have attached the changed version of this project. Thank you!

[Samith Kavishke]

Hi,

Eventhough have not been selected, do we still have a chance to continue the project? Without the stipend.

[Aaron Meurer]

Hi Samith.

You are always free to contribute to SymPy independently of GSoC, as
it is an open source project and you can always contribute to an open
source project. We can't guarantee you the mentorship that a GSoC
contributor would receive. It would be up to Francesco and other
community members if he is interested in helping out with this
project.

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/72653f6b-d10d-4281-8fcc-c662c47d942bn%40googlegroups.com.

[Samith Kavishke]

Hi Aaron,

Thank you for the reply, I would like to give it try. In-order to contribute to this project, I need to have a fork of this repository in somewhere (Sympy org or else). So If someone can help me with that I will try to contribute.

Best Regards,
Samith Karunathilake.

[Francesco Bonazzi]

Do you mean forking MatchPy? I would suggest by creating a personal fork of the project. I would not create a fork of MatchPy on SymPy's space unless we're really sure about continuing the development of MatchPy.

[Samith Kavishke]

Hi Francesco,
yeah sure, I will proceed in that way. Thanks.