Language model
Mathieu Blondel
discussions reminded me. That would be a nice item on the TODO list
for Tegaki 0.4 or 0.5.
Since the "smart" recognizer interface has 2 drawing boxes [*], it's
possible to write full sentences rather than isolated characters.
However, currently only the probability of individual handwritten
characters is considered during the recognition process. We could add
an option to additionally considerer the probability of character
pairs (bigram model). For example, if I input 便 then 利, the recognizer
may mis-recognize them as 使 and 刋. But if we consider also the
probability of character pairs, the recognizer may be able to figure
out that I meant to write 便 and 利 due to the fact that the probability
of (便,利) is higher than the probability of (使,刋).
This could be abstracted to work with any recognizer. The only thing
is, some recognizers like Wagomu don't return a probability for each
candidate but a distance. We need a way to convert the distances to
probabilities.
I think this represents a bit of work but it's an interesting and
feasible task. Obviously, we will need a corpus of character pair
probabilities. There may be existing ones or we may compute ours, for
example by using Wikipedia as input (although using Wikipedia will
make our probabilities biased toward encyclopedia-content).
Mathieu
[*] There's also a simple mode if you run tegaki-recognize with --simple.
Christoph Burgmer
> I think this represents a bit of work but it's an interesting and
> feasible task. Obviously, we will need a corpus of character pair
> probabilities. There may be existing ones or we may compute ours, for
> example by using Wikipedia as input (although using Wikipedia will
> make our probabilities biased toward encyclopedia-content).
If you consider all the reading based input systems (like Pinyin for Chinese)
then there should be a lot of data already in use. Not sure though if they
only map reading pairs to characters, or if they have a step inbetween making
it possible to get hold of the frequency data.
-Christoph
Mathieu Blondel
On 26 oct, 05:28, Christoph Burgmer <cburg...@ira.uka.de> wrote:
> If you consider all the reading based input systems (like Pinyin for Chinese)
> then there should be a lot of data already in use. Not sure though if they
> only map reading pairs to characters, or if they have a step inbetween making
> it possible to get hold of the frequency data.
include this kind of data too.
Another issue will be to find a good data structure for the fast
retrieval of word pair probabilities. Mecab seems to use double array,
which is derived from trie.
Mathieu