AutoLookup tool

[Brent]

Hi there,

I've been making an autolookup tool (attached) that automatically looks up possible words you're trying to spell, and shows you the best way to stroke them - as you are typing.  So, no need to break the typing flow and go to the dictionary, just type the first stroke, or fingerspell the first few letters, and it will lookup a bunch of suggestions for you.  

I wouldn't mind some feedback to know whether or not this is useful for people learning steno.

Unfortunately, I wasn't able to implement it in Python (I know you guys love python), so it's a java .jar file.  I've only tried it on Linux so far, but java is supposed to be cross-platform, so maybe someone could try it on windows or OSX and let me know if it works, or if I need to do something differently.

It works by monitoring the Plover log file, so you need to have "Log Translations" enabled in Plover's configuration to make this work.

Haven't figured out how to use github yet, but when I do, I'll put the source code up there.

Thanks,

Brent.

PS - to the programmers.  When you have plover running, whether it is turned off or on it continues to log strokes to the log file.  Perhaps it should stop logging when it is turned off?

[Hesky Fisher]

Re: logging strokes.

Plover is still listening to strokes even when it's "off" so it still is helpful to have strokes logged. How about I add logs indicating the start and stop events and your program can act accordingly?

--
You received this message because you are subscribed to the Google Groups "Plover" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ploversteno...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Snowy Wilderness]

Brent, in a word, YES...this feature would be VERY useful.

On Fri, Apr 19, 2013 at 3:21 PM, Brent <nesbit...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "Plover" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ploversteno...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Yours Truly,

Johnny Sagan aka Snowy Wilderness

[Brent]

Note, in the screenshot I have "Simplified Translations" turned on, so SKWR, for instance, is replaced with J

[Daniel Langlois]

Do you have the source files?  

[Brent]

https://github.com/brentn/StenoTray

[Daniel Langlois]

Actually, I'm a Java programmer, not quite 'conversant' with Python, but nevertheless, I'm wondering if I might be able to chip in to

convert this to Python/integrate it into Plover. 

I looked at Bag.java. It supports convenient and rapid tallies, and has a dictionary interface, except that it returns a zero count for missing items instead of raising a KeyError. In Python, a Counter might be the thing, as it is a dict subclass for counting hashable objects. 

I also looked at MinPQ, which is a kind of queue in which the elements are dequeued in priority order. The smallest key is always at the front. The implementation is with a binary heap. The classic way. And, well, usage of heapsort is rare, in general, including Python. 

The internal order of a heap-based priority queue is very non-intuitive and, one might assert, quite useless for other purposes, while a sorted list is, umm..., sorted! One might, alternatively, feel that heaps are so standard as an algorithm that they belong into the Python library, in some form, but I picture a mountain of fairly sophisticated code with an interface so rich as to be genuinely hard to learn how to use it as intended.  In the case of Python, the standard .sort is indeed good for most applications. When I want to sort a list I just use .sort(). I don't care which algorithm is used. 

This is, I guess, about the question whether to port your app to Python, and in doing so, do we care whether dictionaries are implemented using hash tables, some kind of tree structure or magic smoke.

It may be, that in using Java, you were primarily thinking about choosing your GUI programming toolkit. As, these can be hard to use. 

Plover uses wxPython, which, has been described as not nearly as bloated as Swing. I figure that if we are knocking up up quick, simple Windows GUIs, then, in theory, this can be simple, easy to write and easy to understand?

[Brent Nesbitt]

Daniel, 

Thanks for your offer to help translate StenoTray to Pyton - that would be fantastic.

The thing that makes my code work is the implementation of the dictionary as a TST (ternary search trie).  It allows for fast and efficient lookups of words with a given prefix (autocomplete).  A standard trie would also work, and might be simpler in the end (risk of using more memory though).  As for the bag - that was just a cheap way to store unordered data.

The choice of MinPQ to sort the data might be more than I need.  The key here was that I could use comparators to define my own sort orders.  I didn't want alphabetic sort, but something that sorted by stroke length.  I'm still not entirely happy with the comparator I'm using.  It would be fine to get rid of MinPQ and Bag, and use standard sort and counter.

I think StenoTray has value as a stand-alone tool, (not adding bloat or slowness to Plover) but would certainly benefit from being integrated as well.  It would have access to all the orthography/translation stuff, and probably be able to share dictionary data items as well.

My choice of java was certainly NOT for its GUI toolkit.  In fact, I started by implementing the dictionary and other data structures (which was more familiar to me), before even thinking about the GUI.  I was surprised how much work it took for this very simple GUI.  My impression was that wxWindows in Python would have been MUCH easier.  

Thanks again for your interest!

--

[Daniel Langlois]

I looked at your TST.java, which contains this:

private class Node {

private char c;                 // character

        private Node left, mid, right;  // left, middle, and right subtries

        private Value val;              // value associated with string

}

And, that can be seen as containing a letter, references to its children, and a flag (indicating, I expect, the completion of a word).

So, a single letter, a flag, and pointers..well, a single character does not represent an entire word, or key. Is it, then, a 'node'?

Or is it an 'edge'? Which, is only meant as a whimsical point. But also, I wonder if this could be realized with an array of integers. That's what the computer is doing anyways, implicitly or explicitly (memory is a big array). Suppose that you've got 400,000 of these Nodes, and it takes 15 megs of memory. At the opposite end, even  an 'int' is, like, four bytes, do we need four bytes? Maybe..but, while we're into this matter of how building a trie can take (waste) a lot of memory, consider this trie, with two words in it.

h-->e-->l-->l-->o

j-->e-->l-->l-->o

You could first build the trie, and look for duplicate branches. Also, it might be much better, if one could create a deterministic acyclic finite state automaton (as they are called), right away. I'm sure I can find a Python implementation. Although, there's still a problem, if you wind up w/this:

h-->e-->l-->l-->o

j---^

It's not possible to retrieve values associated with the words.  Since the last node in a word is shared with other words, it is not possible to store data in it.So, that means, we use an additional data structure, we can store it in a hash table. Look up a word,

run it through a hash function, which returns a number. Look at all the items in that bucket to find the data. 

We know all the keys in advance, so, we could throw away the keys of the hash table. Easy, minimal perfect (minimal, one entry for each key, perfect, no collisions) hashing, I'm, here too, sure I can find a Python implementation..

Come to think of it, there is the hash function itself, I mean, in Java, there are two methods that a class needs to override to

make objects of that class work as hash map keys. 

public int hashCode();

public boolean equals(Object o);

But, never mind that, that Node class (above) could include a Hashtable<Character, Node> children;

And, I mean, children = new Hashtable<Character, Node>();

You're using left, mid, right nodes, though I think you could just have 

children.put(letter, child);

Because, it's a hashmap. Anyways, just remembering my Java. If it goes into Python, I can see wanting to do this kind of optimization, I do think this is useful to explore, at least, as the kind of thing we might occasionally want to do. But, it might also be good to remember, that being an evil genius is more than code efficacy (true evil comes from the passion to crush the will of men). 

[Daniel Langlois]

Reading back over my post, I pause over having suggested dropping the ternary, that is, three-way, search trie, 

for what, exactly, a traditional multiway trie? The reply might be, that this modified binary search tree thing, it helps

avoid the unnecessary space needed, given that a trie has a major problem. It requires lots of nodes. 

Furthermore, if I'm suggesting managing thousands and thousands of hash maps, this is generally not a good idea, so what's that part about using a different data structure in each node, such as a hash map? Yeah, forget that part, but the part about how a lot of space can be saved--by merging identical subtrees, still makes sense to me, and I've found discussions of the 'dawg' (Directed Acyclic Word Graph).

Like this:

p-->i-->n--e

S---^   ^--g  

I've attached a dawg.py file from here: http://software.cedar-solutions.com/wordutils.html ('provides objects related to word searching and matching'). There's also a tree.py, there, and it's a ternary search tree. Apparently, directly competing alternatives?

[Daniel Langlois]

Okay, went w/Trie, but stripped out traverse--I find, that it's much simpler w/out it. I've got the trie into an array, though, so no memory problems. And, you can find as much of the key as you can, by using the longest prefix that has a value--this is the matter of allowing look up based on the longest prefix that matches. I've attached a python file, but it's basically just an algorithm, a unit test. 

One may pause a bit, here, to consider, that, okay, if your requested item is in the dictionary, with a key that hashes the same, then it acts as a more or less normal dictionary. If, on the other hand, you are looking for a string, that is similar to a key in the dictionary..then, I suppose, that the idea would be, if Plover iterated over all the keys, and found the one that was the closest match. For example, searching in a dictionary the word "motorcycle", but returns similar strings like "motorcicle". There is an easy to implement algorithm called Soundex that replaces each word by a 4-character string, such that all words that are pronounced similarly encode to the same string. The basic assumptions, are that the consonants are more important than the vowels, and that the consonants are grouped into "confusable" groups. The US census bureau uses it to locate information about a person (surnames that sound the same, but are spelled differently, are filed together, so that you can find a surname even though it may have been recorded under various spellings). 

There is perhaps a larger point, if coming up with a set of confusables for a language is not horribly tough. 

[Brent Nesbitt]

Great to hear your progress, Daniel!

Yes, I think the Trie is the right data structure to use.  Your implementation of trie (or TST) might be more generally useful as well.  I wasn't able to find a good Python implementation of either.  It will be interesting to compare the lookup speed between the java and Python versions once you are done.

I made some modifications to my (java) code on the weekend, so that the autolookup doesn't only find English words that start the same, but it also considers the stroke you type.  So that where the first version would only find littler and littlest when HREUL (little) was typed (extensions of the English translation), the new version will also find lilly and Lillian (strokes that begin with the HREUS/ chord).  This also helps for words like listen, which don't have a dictionary entry for the first syllable HREUS (lis).  

The result is still sorted by length of the English translation.

Brent

--

[Mirabai Knight]

This looks to be a fantastic tool, though I haven't been able to test it yet.

Since my last laptop went kaput, I haven't been dual-booting; Plover
runs so well on Windows now that I've gotten out of the habit. So I
tried using it on Windows, as you suggested, and can't seem to make it
work. I unzipped the StenoTray files to my Plover configuration
directory and changed stenotray.cfg so it reads:

# Location of the plover configuration file
PLOVER_CONFIG = C:\Users\MKK\AppData\Local\plover\plover\plover.cfg
# Limit the number of results (or 0 for no limit)
LIMIT_RESULTS = 0
# Use simplified steno abbreviations (such as L instead of HR)
SIMPLIFY = true
# Size of font
FONT_SIZE = 12

But when I double click StenoTray.jar, nothing happens. Am I doing
something wrong, or does it need some more fiddling to work on
Windows? I'm running Java Version 7 Update 21.

[Brent Nesbitt]

Hi Mirabai, 

Thanks for trying this on Windows.  I haven't had a windows machine to try this on yet.

I think the problem could be that the location of stenotray.cfg is hard-coded as:

System.getProperty("user.home")+"/.config/plover/stenotray.cfg"

...and I'm not sure how that translates in a windows environment.

therefore, it's probably not finding the stenotray.cfg file at all, and simply using the default location for the plover.cfg file, at:

System.getProperty("user.home")+/.config/plover/plover.cfg"

therefore it can't find the log files or dictionary either.

I need to find a more generic way to determine the default location of the Plover config directory on various systems.  Not sure how to do that yet.

I'll let you know when I figure it out - then you can try again :)

Brent.

[Rob]

This lookup is really intriguing! Nice job, Brent.


To make it even more usable (according to my taste), I made some modifications to it, see the attached file. It is scarcely tested, so it might not work on your machine at all, and as it is now, it wouldn't win a code beauty contest either.
- it should be Windows 7 compatible now
- gives suggestions for the last expression, not only the last word
- colorizes the steno (initial/ending consonant/vowel)
- option: can move asterisk to front of stroke
- emphasizes if the found word is grammatically related (it <-> it's) or if the word is the first word in an expression
- some letters were missing from simplify (e.g. lhs J)
- steno is displayed in fixed with font (Consolas)

Minor annoyances I found so far, and didn't care to do anything about them:
- when using the asterisk during fingerspelling, two characters are removed instead of one
- to start a new word with fingerspelling, an invalid entry (e.g. -RG) has to be input

[Brent]

Thanks Rob!

This is really encouraging.  I was going to say thanks mostly for figuring out how to get it working in Windows, but I realized that all your mods are pretty awesome!  Great thinking!

[Brent]

Here is a jar that should work on Windows (thanks Rob) - and merges a bunch of my recent changes with his colourization and styling.  Also, my jar is compiled with Java 6, while his is done with Java 7 - so if one doesn't work for you, try the other.

Or grab the latest code from github.

[Brent]

Is somebody able to verify whether this latest version of the auto-lookup tool works on Windows or Mac?

The config file goes in your plover config directory, but shouldn't be necessary.

[Mirabai Knight]

Works beautifully for word parts, but for some reason it's not

bringing anything up when I fingerspell. You said:

"to start a new word with fingerspelling, an invalid entry (e.g. -RG)
has to be input"

I tried that with -RG, but still nothing.

So if I write fell, I get fell, fella, feller, fellow, et cetera, but
if I write TP* *E *L *L, or -RG * TP* *E *L *L or RG TP* *E *L *L,
nothing. Is that true for the non-Windows version as well? It's too
bad, because I think fingerspelling is probably going to be the newbie
steno learner's go-to method for figuring out words.

> --
> You received this message because you are subscribed to the Google Groups
> "Plover" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ploversteno...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
Mirabai Knight, CCP, RPR, CBC
StenoKnight CART Services
917 576 4989
m...@stenoknight.com
http://stenoknight.com

[Brent Nesbitt]

Try the latest .jar I uploaded.  I think I have that fixed now.

[Brent Nesbitt]

...oh, you need to have at least 3 letters before it will do the lookup.

[Mirabai Knight]

Huh. Maybe I'm missing something, but I just downloaded the files from
your Github, and it still doesn't seem to be working. If I write:

WAOERD, I get "weird, weirdo, weirdly".

But if I write:

W*/A*/O*/*E/*R*D

I don't get anything. Tried messing around with -RG, but that doesn't
seem to have any effect either.

[Brent]

I think the -RG thing doesn't really apply.  

When you are fingerspelling, spell the English, not the steno.  So, if you want to know how to type weird, type W*/E*/EU* (for wei), and at that point you should see a list of words, including weird.

I think there might still be some issues with backspacing (*) and "glue strokes", but if you are starting with a fresh word, fingerspelling works well.

[Mirabai Knight]

Yeah, sorry, total mistake up there. Ignore. I was spelling
W*/E*/EU*/R*/TK*. No dice. Sorry about that. Just tested it again.
Can't get it to show up when fingerspelling. Anyone else have this
problem?

[Brent Nesbitt]

Oh,  you're using the new plover i bet.   It has a different log format.   I tried to make both work, but have only been testing with the old format. (haven't been able to get the new version of plover working on my main machine)

I have one machine (a netbook) running the new version, I'll test with that one tomorrow, and see if i can't figure out what 's up.

Thanks for being persistent!

[Hesky Fisher]

What kind of trouble are you having with the newer version?

[Brent]

Okay, this fixes the fingerspelling issue with the newer version.

Hesky, it's just the different log format that needs to be parsed differently.  However, I have noticed on the new version (at least in my testing), that if Plover is already running when you launch StenoTray, then sometimes the log stops recording, and you have to re-start Plover to get the log to continue recording.  Does that make any sense?

[Mirabai Knight]

Yes!! Works perfectly with fingerspelling now. Fantastic. I haven't
noticed the log bug you mentioned; I started StenoTray after Plover
was already on and had no problems. But if it happens, I'll be sure
to make a note of it.

[Brent]

Note, the latest version(s) of Plover (2.4.0)  don't work correctly with StenoTray.  

Here is a version that does (but doesn't work with some older versions)

[Emanuele Caruso]

Thank you, it works with Plover 2.4.1.

Can you please also push the updated source files to github?

Thank you!