Autolev Parser

[Nikhil Pappu]

Community Bonding Period Plan:

I will get started on May 5th and work on these prerequisites until May 14th.

1. Community Bonding:

    - Interact with the community

    - Get started with the blog (check out Planet SymPy)

2. Autolev and Kane's Method:

    - Review the Autolev tutorial

    - Read DynamicsOnline: Theory and Implementation with Autolev

3. ANTLR:

    - Review The Definitive ANTLR Reference

    - Set up the work environment

4. SymPy and PyDy:

    - Go over the SymPy mechanics module (guide + documentation + code)

    - Go over the PyDy Documentation

    - Go over the SymPy and PyDy examples and models

    - Go over the Bicycle and Human Body examples

5.  Benchmarking:

     - Figure out how to run Autolev code

     - Check out the dynamics benchmarking website

[Nikhil Pappu]

The book 'DynamicsOnline: Theory and Implementation with Autolev' would be very helpful to me.

I looked up the contents and it seems to cover Dynamics, the Kane's method and the implementation part using Autolev.

It would be great if I can get a copy of it soon.

I am still not clear about running Autolev.

Should I purchase the student license for MotionGenesis or will Jason just exchange the test cases and outputs with me throughout the project?

Nikhil

[Jason Moore]

Nikhil,

I'm going to see if I can scan my copy of the book and I'll send it to you. I'll try Thursday or Friday this week.

I will also see if I can get you a copy of Autolev 4.1 educational version. It only runs on windows or via wine on linux.

Jason

moorepants.info
+01 530-601-9791

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscribe@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at https://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/67b109aa-8d7a-4078-abee-50f376e3328d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jason Moore]

Nikhil,

This might be helpful: http://www2.mae.ufl.edu/~fregly/PDFs/autolev_tutorial.pdf

moorepants.info
+01 530-601-9791

[Nikhil Pappu]

Jason,

I had already gone through that tutorial prior to writing the proposal. It was the only resource I found on Autolev. I think the guide book would be more useful as it is more elaborate and covers dynamics theory as well.

Nikhil

[Nikhil Pappu]

Ondřej, Jason,

I have written my first blog post discussing the project details and status.

Can you please go over it and provide feedback?

Also, when will you be available to discuss things?

[Ondřej Čertík]

On Fri, May 18, 2018, at 1:32 AM, Nikhil Pappu wrote:
> Ondřej, Jason,
>

> I have written my first blog post <https://wp.me/p3Sr5x-2> discussing the

> project details and status.
> Can you please go over it and provide feedback?

I think overall it looks very good. It looks like you got a good handle of ANTLR4.

I noticed you are using a listener:

https://github.com/NikhilPappu/autolev-parser/blob/f6d110ff5a8cef3aea4f6357f343491572350393/myListener.py#L260

In my codes, I ended up using a visitor --- since I needed to visit every node anyway, and recursively construct the, in your case, sympy expression probably.

One thing to consider is whether to directly use the grammar from ANTLR, which is called a concrete syntax tree (CST), and construct a sympy expression of whatever you want out of it directly. That is what you do now I think.

The alternative is to first construct an abstract syntax tree (AST) from the CST. That way the AST will be the input to the rest of your "compiler", and it won't matter that we currently use ANTLR to construct it. Later somebody can decide to write a hand parser, or some other tool. As long as it produces the AST, that's all that matters.

As it is currently, the CST depends on the exact rules as you write them in the ANTLR grammar, and everytime you change the rules, you have to rework your compiler. While if you use AST, then you only have to modify the code that converts the (modified) CST to AST, but that's it. The rest of the compiler uses AST, and so it doesn't need to change. Python, for example, uses this approach. It parses Python code into an AST, and so they are free to change the parser any way they like ---- our SymPy code that uses the Python AST doesn't need to change.

> Also, when will you be available to discuss things?

I am available over email only the next week.

Ondrej

>
> --
> You received this message because you are subscribed to the Google
> Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to sympy+un...@googlegroups.com.

> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/sympy/bc81cd14-ad9b-4503-a639-aa6f673f4c51%40googlegroups.com.

[Nikhil Pappu]

On Friday, May 18, 2018 at 4:08:29 PM UTC+5:30, Ondřej Čertík wrote:

On Fri, May 18, 2018, at 1:32 AM, Nikhil Pappu wrote:
> Ondřej, Jason,
>
> I have written my first blog post <https://wp.me/p3Sr5x-2> discussing the
> project details and status.
> Can you please go over it and provide feedback?

I think overall it looks very good. It looks like you got a good handle of ANTLR4.

I noticed you are using a listener:

 

I prefer using listeners over visitors. I feel it helps me to focus on the logic than on the tree traversal. 

One thing to consider is whether to directly use the grammar from ANTLR, which is called a concrete syntax tree (CST), and construct a sympy expression of whatever you want out of it directly. That is what you do now I think.

 

Yes, I am using the CST approach.

The alternative is to first construct an abstract syntax tree (AST) from the CST. That way the AST will be the input to the rest of your "compiler", and it won't matter that we currently use ANTLR to construct it.

 

I only have a basic idea of ASTs. I have never implemented them so I do not have any experience using them.

The ANTLR book showcased only the CST approach so I am much more comfortable with that.

I think ANTLR generates something in between a CST and an AST but it does have a lot of clutter compared to an AST. 

I was just looking at some examples of using ASTs with ANTLR in stackoverflow but most of the examples were quite trivial.

The conversion from CST to AST for the Autolev grammar wouldn't be so trivial if we want to obtain a transformation with minimal loss.

Also, I would need to change the parser code quite significantly to deal with the new tree. 

Later somebody can decide to write a hand parser, or some other tool. As long as it produces the AST, that's all that matters. 

I'm not sure why someone would want to do that if we already have a working parser.

One might rather look towards changing the actual parser code itself in my opinion, whether to fix any issues or add new functionality.

 

As it is currently, the CST depends on the exact rules as you write them in the ANTLR grammar, and everytime you change the rules, you have to rework your compiler. While if you use AST, then you only have to modify the code that converts the (modified) CST to AST, but that's it. The rest of the compiler uses AST, and so it doesn't need to change.

That is an advantage but I don't think we would have to change the grammar all that much. I have tested the grammar on a lot of inputs and it seems to parse them just fine.

 

Python, for example, uses this approach. It parses Python code into an AST, and so they are free to change the parser any way they like ---- our SymPy code that uses the Python AST doesn't need to change. 

 

I think that the AST approach would be a much better one in the case of more complex languages like Python or C.

I think this is not as important in this case as Autolev is a much simpler language. 

At its heart it mainly only consists or declarations, assignments and commands which makes direct parsing using a CST convenient enough.

[Ondřej Čertík]

Hi Nikhil,

Thanks for the answer. If I understand your points correctly, your main points are:

* Nobody would want to rewrite the parser
* Autolev language is simple enough, so AST is not worth the additional complexity / code
* Converting from CST to AST is not trivial in this case

I thought about these questions a lot and my view is the following:

Regarding the first point, people definitely might want to try some other parsers, especially since Autolev is such a simple language. I know for example that Aaron was not 100% convinced that ANTLR is the best way. One disadvantage of ANTLR is that it produces very large parser files. For example even for your simple grammar, the parser has almost 3000 lines (https://github.com/NikhilPappu/autolev-parser/blob/f6d110ff5a8cef3aea4f6357f343491572350393/AutolevParser.py) and such big files are not super easy to deal with, for example they grow the sympy git repository a lot, so we might not want to have those checked in. But then people require Java and ANTLR to generate those. And if it is possible to write a simpler parser, for example with pyparsing or something similar, then perhaps in the future we might want to go this route. Note that I personally like and recommend ANTLR, as it gets the job done pretty easily and cleanly, I really like that it keeps the grammar g4 files separate from the code.

Regarding the second point, If you look at your parser, you have to handle variables and some other stuff, and it is all tied up to how exactly ANTLR represents the nodes, as defined in your g4 file. So if I want to polish / change the g4 file, I have to understand the rest of your compiler. That's actually not so easy. As a particular example, this line https://github.com/NikhilPappu/autolev-parser/blob/f6d110ff5a8cef3aea4f6357f343491572350393/myListener.py#L205 depends on how exactly you write the rule in the g4 file, so that the 2nd Child is what you expect it to be. If you rework the g4 file, this will break. And since this is used all over the compiler, it's not easy for me to quickly figure out what needs to be changed and checked. If it used AST, then I just have to make sure my changes to the g4 file generate exactly the same AST, and don't have to worry about the rest. In practice, I just have to go to the visitor methods that deal with the old g4 rules, and modify those for the new g4 rules, to generate the same AST. So the AST approach is actually simpler, from this perspective.

Regarding the third point, I don't quite understand what you meant, as the point of AST is that you lose syntax information that you don't need. If there is some information that you need, it needs to be included in AST.

For now, I suggest you keep doing what you are doing, but I will try to send a PR with the initial stub for an AST, for this problem, so that you get an idea how it all works.

No matter however, whether AST is used or not, we will need thorough tests for the grammar, essentially every rule should be tested by tests. So that if we switch to AST later, we can be sure that we didn't break anything.

Ondrej

> > > an email to sympy+un...@googlegroups.com <javascript:>.

> > > To post to this group, send email to sy...@googlegroups.com

> > <javascript:>.

> > > Visit this group at https://groups.google.com/group/sympy.
> > > To view this discussion on the web visit
> > >
> > https://groups.google.com/d/msgid/sympy/bc81cd14-ad9b-4503-a639-aa6f673f4c51%40googlegroups.com.
> >
> > > For more options, visit https://groups.google.com/d/optout.
> >
>
> --
> You received this message because you are subscribed to the Google
> Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/sympy/d207a582-c67e-4219-8a6f-1d5c6a640ebe%40googlegroups.com.

[Ondřej Čertík]

One important point that I want to stress is that we want it to be easy for other people to contribute. The AST separates the parser from the rest of the compiler, so people can contribute to the parser, without modifying or even knowing/understanding the rest of the compiler, and vice versa, people can fix and work on the compiler (starting from AST) without knowing anything about the parser.

Ondrej

[Nikhil Pappu]

Ondrej,

Thanks for the detailed explanation. I get the significance of an AST now.

I will continue with what I am doing now as you mentioned. I will see if I can make the switch once you send that PR.

If not possible now, then I will definitely work on switching to it later.

I also plan on writing extensive test cases.

Regarding my status:

I will soon be done with the mathematical entities. I will start with the symbolic dynamics parts right after that.

 

There were some issues I faced :

1. Some output results differ between Autolev and SymPy.

For example: 

e = 4*x**2 + 7*x*y + 21*x + 4*y**2 + 21*y
factor(e, x) in Autolev
e.factor(x) in SymPy

outputs:

21*y + 4*y^2 + 4*x*(5.25 + x +1.75*y)

in Autolev
and

(7*y + 21)*x + 4*x**2 + 4*y**2 + 21*y

in SymPy

Do differences like these matter? Is there a way to get the same result in this case?

Should we try to get the exact same outputs in cases like these?

I am in the mindset that end numerical results are more important than equivalence of intermediate expressions.

2. One major issue I was facing was with dynamicsymbols.

The thing is I need to define all the Autolev variables as dynamicsymbols and constants as symbols.

I need to do this because Autolev simply uses Variables and Constants for everything and I need the variables to be defined as

dynamicsymbols for them to work with the physical entities.

Although the dynamicsymbols would work like they should with the physics entities, that doesn't seem to be the case with the 

mathematical entities of SymPy as they do not seem to be written with dynamicsymbols in mind.

In particular, I have found that dynamicsymbols do not work with dsolve, solveset.nonlinsolve and series expansions.

See this issue:

https://github.com/sympy/sympy/issues/12044

I will have to use these when I tackle the solvers after I am done with parsing the basic physics entities.

It would be great if someone could look into this.

[Aaron Meurer]

It looks like Autolev uses a different definition of factor() than
SymPy. I wouldn't in general expect Autolev and SymPy functions to do
the same thing just because they have the same name. Autolev's
factor() looks more like SymPy's horner(), although I would need to
see more examples of what it does to be sure of that.

>
> 2. One major issue I was facing was with dynamicsymbols.
> The thing is I need to define all the Autolev variables as dynamicsymbols
> and constants as symbols.
> I need to do this because Autolev simply uses Variables and Constants for
> everything and I need the variables to be defined as
> dynamicsymbols for them to work with the physical entities.
> Although the dynamicsymbols would work like they should with the physics
> entities, that doesn't seem to be the case with the
> mathematical entities of SymPy as they do not seem to be written with
> dynamicsymbols in mind.
>
> In particular, I have found that dynamicsymbols do not work with dsolve,
> solveset.nonlinsolve and series expansions.
>
> See this issue:
> https://github.com/sympy/sympy/issues/12044
>
> I will have to use these when I tackle the solvers after I am done with
> parsing the basic physics entities.
> It would be great if someone could look into this.

dynamicsymbols are just SymPy Functions of t, so anything that doesn't
work with them should be fixed.

Aaron Meurer

>
> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/sympy/2862ac25-6b9d-47a2-aec9-eae6c48d3463%40googlegroups.com.

[Nikhil Pappu]

Aaron,

Thanks for the reply. The horner method seems more appropriate indeed.

As for seeing more examples, I am currently just looking at some sample outputs from the Autolev Tutorial

as I do not have access to Autolev yet.

I will try and see if I can fix the dynamicsymbol issues.

Nikhil