Grammar for writing source code?

[Mark Finker]

Hi folks,

If there are any code bits floating around that might assist with dictating source code, would appreciate any pointers.

Thanks

Mark

[Charlie Tango]

Hello Mark,

A very successfully use the multi-edit command module [1]. It is
nothing more than a speech-enabled keyboard, but nevertheless it makes
technical editing and writing source code much more efficient.

Besides that, I've added several common programming symbols (such as
"==") to my vocabulary so that the spacing and capitalization is
automatically done correctly.

Best regards,
Charlie

[1] The multi-edit command module is available here:
http://dragonfly-modules.googlecode.com/svn/trunk/command-modules/documentation/mod-_multiedit.html

[Jason Veldicott]

Hi Mark,

You'll find a problem with this grammar, one which I personally find to be of annoyance, is that when you're dictating text, for example, an identifier like a.b.c, you have to wait until DNS finishes dictation (or long pause at least) before navigational or content commands can be issued, or else the commands end up as dictated text.  

Jason

[Jason Veldicott]

Also, I might mention that there appears not to be a solution around (for Dragonfly, and probably Vocola as well) for referring to existing identifiers through speech that have contracted forms like "nxtLnStr".  This means reliance on IDE pop-ups, which can be cumbersome, or typing, for those identifiers.  

If a speech solution is preferred, there is one in VoiceCode apparently.  But having looked at its code (no wonder it didn't take off) it may be easier to rewrite a similar feature than attempt reuse.  For which awareness of scope may be of benefit.

Jason

[Jason Veldicott]

Benchmarks on code dictation with Dragonfly could be of interest to people if there were any about.

[Frank Olaf Sem-jacobsen]

Hi guys,

I just thought I would chime in to say that I quite frequently and
successfully use voice code for my place and programming needs in my
daily work. Admittedly it is somewhat unstable, but as long as it
doesn't crash it works very well.

Regards,
Frank

On 23 February 2012 09:11, Jason Veldicott <jasonve...@gmail.com> wrote:
> Benchmarks on code dictation with Dragonfly could be of interest to people
> if there were any about.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Dragonfly Speech Recognition" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/dragonflyspeech/-/aK7wpKWEFH4J.
>
> To post to this group, send email to dragonf...@googlegroups.com.
> To unsubscribe from this group, send email to
> dragonflyspee...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/dragonflyspeech?hl=en.

--
Frank Olaf Sem-Jacobsen

[Jason Veldicott]

Thanks, good to know there are some voice solutions for coding about that work well in practice.

I shall post some numbers here soon.

Jason

[Jason Veldicott]

I had a try of VoiceCode, and I like its "high road" approach to coding by voice, which includes use of grammars I believe to understand code structure. And even language independence (and recognizing contracted identifiers!).  This would seem the ideal way to go. 

 

BUT, Emacs to me is like what Unix is to a Windows user, and besides that, it only supports few languages, and extension (having looked at the code) would not be straightforward.  That it is unreliable as you're saying, as it is already, would also suggest this.

The view I've taken is that the simplicity of a straightforward Dragonfly grammar or variation thereof might actually work fairly well most of the time, and for more languages, and it's easier to write, and it allows access to any IDE.  For me, as a practical solution, this seems the way to go for now.  Although having said that, I'm yet to arrive at a resoundingly successful solution using Dragonfly, but on that time will tell.

When opportunity presents though, I might dig through VoiceCode more and see if it could be useful in some way. 

Regards

Jason

On Thursday, February 23, 2012 7:19:16 PM UTC+11, Frank Olaf Sem-jacobsen wrote:

Hi guys,

I just thought I would chime in to say that I quite frequently and
successfully use voice code for my place and programming needs in my
daily work. Admittedly it is somewhat unstable, but as long as it
doesn't crash it works very well.

Regards,
Frank

On 23 February 2012 09:11, Jason Veldicott <jasonve...@gmail.com> wrote:
> Benchmarks on code dictation with Dragonfly could be of interest to people
> if there were any about.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Dragonfly Speech Recognition" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/dragonflyspeech/-/aK7wpKWEFH4J.
>

> To post to this group, send email to dragonflyspeech@googlegroups.com.

> To unsubscribe from this group, send email to

> dragonflyspeech+unsubscribe@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/dragonflyspeech?hl=en.

--
Frank Olaf Sem-Jacobsen

On Thursday, February 23, 2012 7:19:16 PM UTC+11, Frank Olaf Sem-jacobsen wrote:

Hi guys,

I just thought I would chime in to say that I quite frequently and
successfully use voice code for my place and programming needs in my
daily work. Admittedly it is somewhat unstable, but as long as it
doesn't crash it works very well.

Regards,
Frank

On 23 February 2012 09:11, Jason Veldicott <jasonve...@gmail.com> wrote:
> Benchmarks on code dictation with Dragonfly could be of interest to people
> if there were any about.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Dragonfly Speech Recognition" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/dragonflyspeech/-/aK7wpKWEFH4J.
>

> To post to this group, send email to dragonflyspeech@googlegroups.com.

> To unsubscribe from this group, send email to

> dragonflyspeech+unsubscribe@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/dragonflyspeech?hl=en.

--
Frank Olaf Sem-Jacobsen

[Mark Finker]

Thanks people.  Will check the edit module. This would seem a better option than Voicecode if it's limited in terms of language support.  And keep an eye out for any developments in this space.

Mark

[Jason Veldicott]

In case of interest to anyone, I compared the speed of writing code by voice and keyboard.  Both were used to create a copy of a selected benchmark module from the Dragonfly command modules repository:

Command-module for moving and controlling **windows**

http://dragonfly-modules.googlecode.com/svn/trunk/command-modules/documentation/mod-_winctrl.html

This module was chosen because it's reasonably long and covers different patterns of Python code. (Config code which appears at the top of the page was excluded.)

Keyboard:        45 mins

Voice (Dragonfly):  30 mins

Coding by voice was found to be 30% faster than typing*.  Even allowing for error, with the margin that voice is ahead, it could be said with certainty that speed is at least comparable, if not considerably better. *what I would consider to be a fast pace relative to typical coding speeds. 

However, some limitations: 

* This measures copying of code, not copying and editing.

* Having practiced on the benchmark module beforehand, DNS would have been slightly optimised for it in terms of recognition accuracy.

* Voice words were pre-scripted and recited rather than constructed on the fly (as familiarity is lacking as yet with the voice 'language'), and it is assumed then that in practice this construction could occur as quickly as if read from a script.

* No feature was used that would allow quick identification by voice of existing identifiers in the code (as I believe exists in VoiceCode) which might otherwise save a lot of spoken words and improve speed.

The number of words used to dictate the module as a matter of interest, which affects elapsed time and relates to efficiency of the voice description of the code, was 3934 (elsewhere 3973, so it's possible a line was left out during the timing run).  This seems quite high, but it might compare well with other voice coding systems (and considering no spoken identifiers feature as mentioned above).

Perhaps of interest would be running the same benchmark but with VoiceCode to see how it compares with Dragonfly - if there's a version somewhere that supports dictation of Python code.  Or other Dragonfly grammars that may have been cobbled together for coding.

Jason

[r...@aarden.us]

Jason,

 

Thank you very much.  I appreciate your continued support.

 

And I am really glad you provided that link.  I have not been able to find the documentation and have been wondering how progress has been made. 

 

Now maybe I can find a way to learn.  Any directions would be appreciated.

Ray 

 

--

You received this message because you are subscribed to the Google Groups "Dragonfly Speech Recognition" group.

To view this discussion on the web visit https://groups.google.com/d/msg/dragonflyspeech/-/T2B5pbfjQ7gJ.
To post to this group, send email to dragonf...@googlegroups.com.
To unsubscribe from this group, send email to dragonflyspee...@googlegroups.com.

[Jason Veldicott]

Other than the sparing documentation, the only way to find out how Dragonfly works is to go through the code, Dragonfly and the separately supplied grammars.

Here are some notes though that may assist.  Needless to say, if there are mistakes, feel free to comment.

The Grammar

A Dragonfly grammar is defined as a set of rules.  Each rule matches a pattern of words that are supplied by DNS.  When a phrase is spoken, DNS sends a sequence of recognized words, and this is matched to one of the top-level rules in the grammar.  The rule pattern therefore applies to a single complete spoken phrase.  (A rule may however contain sub-rules as part of its 'pattern', which may only match part of a spoken phrase.)

Communication between DNS and Grammar

DNS communicates with a grammar through GrammarWrapper.  As implied by "wrapper", messages are passed through the wrapper to the actual grammar in Dragonfly.  The communication really involves only two messages from DNS:

beginCallback(contextInfo)

resultsCallback(spokenWords)

DNS sends beginCallback to all grammars (the grammar wrapper in effect) when some words are spoken as a preparatory step before sending the actual words. Grammars will deactivate in response if the window context in which the words were spoken is not relevant, or activate conversely. Enablement/disablement is another related option.  DNS sends resultsCallback after the above to all grammars (wrappers) to process the results of recognition, the spoken words. 

 

Processing Spoken Words

When words are spoken and a resultsCallback message received (by GrammarWrapper), Dragonfly seeks to match the spoken words to a single top-level rule in the grammar by sending decode() to each top-level rule in the grammar.  The first rule for which decoding completes is the matching rule, which is then tasked with processing the spoken phrase though its process_recognition method.  This handling is shown in the following GrammarWrapper code:

for each active rule

                attempt to match words to rule (ie attempt decode match)

                if success (ie decoding finished):

                                tree=build a parse tree

                                rule.process_recognition(tree)

An extra step in the process, as can be seen from the code above, is the creation of a parse tree from the results of decoding. Decoding explores many paths through the grammar's element structure to match spoken words.  The parse tree describes the matching path found as a construction of relatively minimal node elements that match words to grammar elements.

Most of the work done by Dragonfly is in decoding (as can be seen in the log file with decoding output set to Debug).  The implementation of decoding can be seen in the decode() method of elements.  The decoding code is actually not straightforward to work out, but it seems to work pretty well in matching, even if it might have certain preferences (precedence for example when combining dictation, commands and optionals), and so it's likely familiarisation with it would be unnecessary.

Task of the User: Creating Grammar Rules

The task for the user of Dragonfly is to create one or more rules, each of which includes a phrase pattern description which is formed through a combination of Dragonfly elements (eg. Literal, Group, Alternative, etc), and some 'action' code in the event that the rule pattern is matched.

Easing the task for matching common kinds of word patterns are the two derived rule classes: CompoundRule and MappingRule.  The former allows the element structure of the pattern to be built somewhat automatically from a specification string e.g. "one [two|three]", although a word maybe enclosed in <> to indicate reference to another element (which is held in the variable "extras").  The latter class, MappingRule, includes not only a pattern description in terms of a set of literals, but conveniently associates actions with them as well - useful for implementing a mapping as such.  This is perhaps better suited to short actions like keystrokes, but with the use of Function actions, it can also accommodate longer actions.  

Other Bits and Pieces

Words received from DNS are represented as an array of tuples.  Each tuple contains a word and a number code.  Mostly this code is 1,000,000 which means 'dictation word', as opposed to command word.  But the code otherwise matches to, I think rules or elements or both, in the grammar, but they do not seem to be that important due to the way Dragonfly processes recognitions. 

There is a compilation part of Dragonfly that converts the Dragonfly grammar definition to one that DNS understands.  It wouldn't be a concern to the ordinary user of Dragonfly, and isn't much either to those doing more extensive work in Dragonfly, unless new elements are being added.  Even then, in terms of defining Dragonfly grammars and handling recognitions, the compiled grammar would seem almost irrelevant. Whatever matching DNS may do, Dragonfly redoes in its decoding step during recognition handling.

The long parser.py module in the root directory of Dragonfly seems to be for checking syntactic correctness of CompoundRule specification strings, although would need to recheck this. It may also be used in compilation.