SymPy Gamma: hand-written input

[Fedor Morozov]

Hello.
My name is Fedor Morozov, I'm a third year CS student at Moscow State University specializing in computer vision.

Since touchscreen devices are highly popular nowadays, there are lots of cases when it's easier to handwrite a formula then to enter it using a keyboard.
State-of-the arts computer vision approaches achieve good results in formula recognition, so it is possible to implement such a feature for SymPy Gamma.
Along with NLP parsing this would provide a very powerful user input capabilities.
There are already some similar 3rd-party applications for Wolfram Alpha, you can see an example here.
A more challenging task would be formula recognition from photos, but that's not necessary right now, I guess.

So the proposed project is to implement formula recognition capabilities for SymPy Gamma, it would also be great to make some interface for touchscreen input (web/mobile app?).
I've recently read a few articles on the topic, if you believe that the project if useful, I'll come up with a full-fledged proposal and algorithms review.
The easiest (conceptually) approach would be to select some of the recent articles and code up the algorithm.
Than some improvements can be made, for example it would be cool to use SymPy resources for formula correctness checking.

As stated above, I specialize in computer vision, so I'm familiar with basic algorithms in the area.
Now I'm researching feature descriptors, so I have some experience with shape recognition (for example, shape context), a year ago I worked on a car plate recognition task.
Also last year I participated in GSoC with computational photography project for opencv library.

Thanks for your attention, hope to hear some comments soon.

[David Li]

Hello,

This sounds like a great idea for Gamma. How familiar are you with web development? In particular, do you have any experience with JavaScript?

I would also encourage you to look at the application template and also perhaps contribute to SymPy if you haven't already (as it is a requirement).

David

[Fedor Morozov]

Hello, David.
Unfortunately I have no experience with JavaScript yet, but since the main challenge of the project is the backend part (formula recognition), I can get enough JavaScript knowledge before the coding phase starts.
I also think that it would be great to develop Sympy Gamma mobile app for the project and I have some skills for this (Java and Android programming).
Now that the idea is confirmed to be useful I'll take a look into the issues list, thank you.

среда, 12 марта 2014 г., 20:38:33 UTC+4 пользователь David Li написал:

[Matthew Rocklin]

Handwritten input sounds neat.  I use and find great value in http://detexify.kirelabs.org/classify.html

Handwritten mathematical expression recognition is general and useful enough that I'd like to see it done outside of SymPy and then imported, this way other projects can also use it.

Fedor, can you provide some background on this topic?  What other attempts have there been?  Are there other systems?  What resources have you found?  Is this even a feasible problem?  What are the hard parts?

-Matthew

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/307c7fbd-50fd-4358-a767-5d6460bb5193%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Fedor Morozov]

Hello, Matthew.
The problem is sure feasible, you can test a great implementation here: http://webdemo.visionobjects.com
Unfortunately I haven't found any opensource formula recognition tools, so maybe there is no code to adapt.
The are lots of articles on the topic even from the 80s. For example, there is a recent survey article about math expressions retrieval.
The three main steps are as follows: symbol shape classification, formula layout estimation and syntax analysis.
Most of the approaches try to parse printed formulas, in our case the task is rather different - the formula is already detected and binarized, but the shapes may significantly differ.
Robust shape estimation looks like the most challenging part, though it's the one best known to me.
For the syntax analysis it could be useful to adapt some of sympy parsing functionality.
Also if the project proves to be too difficult for one summer, even single character recognition with smart interface can provide good usability.

среда, 12 марта 2014 г., 23:03:39 UTC+4 пользователь Matthew написал:

[Gilbert Gede]

I always thought an interesting approach would be a handwriting -> LaTeX translator, and then a LaTeX -> SymPy translator (although I'm not sure if it would be LaTeX code, or an actual LaTeX document). Seemed like it would allow for a little more flexibility and would be made of smaller, more tractable subproblems.

-Gilbert

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/371a8501-c8ac-477d-9dd8-e1e4f1a957b9%40googlegroups.com.