Hi folks,

If there are any code bits floating around that might assist with dictating
source code, would appreciate any pointers.

Thanks

Mark

A very successfully use the multi-edit command module [1]. It is
nothing more than a speech-enabled keyboard, but nevertheless it makes
technical editing and writing source code much more efficient.

Besides that, I've added several common programming symbols (such as
"==") to my vocabulary so that the spacing and capitalization is
automatically done correctly.

Best regards,
Charlie

[1] The multi-edit command module is available here:
http://dragonfly-modules.googlecode.com/svn/trunk/command-modules/doc...

Hi Mark,

You'll find a problem with this grammar, one which I personally find to be
of annoyance, is that when you're dictating text, for example, an
identifier like a.b.c, you have to wait until DNS finishes dictation (or
long pause at least) before navigational or content commands can be issued,
or else the commands end up as dictated text.  

Jason

Also, I might mention that there appears not to be a solution around (for
Dragonfly, and probably Vocola as well) for referring to existing
identifiers through speech that have contracted forms like "nxtLnStr".
 This means reliance on IDE pop-ups, which can be cumbersome, or typing,
for those identifiers.  

If a speech solution is preferred, there is one in VoiceCode apparently.
 But having looked at its code (no wonder it didn't take off) it may be
easier to rewrite a similar feature than attempt reuse.  For which
awareness of scope may be of benefit.

Jason

Benchmarks on code dictation with Dragonfly could be of interest to people
if there were any about.

I just thought I would chime in to say that I quite frequently and
successfully use voice code for my place and programming needs in my
daily work. Admittedly it is somewhat unstable, but as long as it
doesn't crash it works very well.

Regards,
Frank

On 23 February 2012 09:11, Jason Veldicott <jasonveldic...@gmail.com> wrote:

Thanks, good to know there are some voice solutions for coding about that
work well in practice.

I shall post some numbers here soon.

Jason

I had a try of VoiceCode, and I like its "high road" approach to coding by
voice, which includes use of grammars I believe to understand code
structure. And even language independence (and recognizing contracted
identifiers!).  This would seem the ideal way to go.

BUT, Emacs to me is like what Unix is to a Windows user, and besides that,
it only supports few languages, and extension (having looked at the code)
would not be straightforward.  That it is unreliable as you're saying, as
it is already, would also suggest this.

The view I've taken is that the simplicity of a straightforward Dragonfly
grammar or variation thereof might actually work fairly well most of the
time, and for more languages, and it's easier to write, and it allows
access to any IDE.  For me, as a practical solution, this seems the way to
go for now.  Although having said that, I'm yet to arrive at a resoundingly
successful solution using Dragonfly, but on that time will tell.

When opportunity presents though, I might dig through VoiceCode more and
see if it could be useful in some way.

Regards

Jason

On Thursday, February 23, 2012 7:19:16 PM UTC+11, Frank Olaf Sem-jacobsen
wrote:

Thanks people.  Will check the edit module. This would seem a better option
than Voicecode if it's limited in terms of language support.  And keep an
eye out for any developments in this space.

Mark

In case of interest to anyone, I compared the speed of writing code by
voice and keyboard.  Both were used to create a copy of a selected
benchmark module from the Dragonfly command modules repository:

Command-module for moving and controlling **windows**

http://dragonfly-modules.googlecode.com/svn/trunk/command-modules/doc...

This module was chosen because it's reasonably long and covers different
patterns of Python code. (Config code which appears at the top of the page
was excluded.)

Keyboard:        45 mins
Voice (Dragonfly):  30 mins

Coding by voice was found to be 30% faster than typing*.  Even allowing for
error, with the margin that voice is ahead, it could be said with certainty
that speed is at least comparable, if not considerably better. *what I
would consider to be a fast pace relative to typical coding speeds.

However, some limitations:
* This measures copying of code, not copying and editing.
* Having practiced on the benchmark module beforehand, DNS would have been
slightly optimised for it in terms of recognition accuracy.
* Voice words were pre-scripted and recited rather than constructed on the
fly (as familiarity is lacking as yet with the voice 'language'), and it is
assumed then that in practice this construction could occur as quickly as
if read from a script.
* No feature was used that would allow quick identification by voice of
existing identifiers in the code (as I believe exists in VoiceCode) which
might otherwise save a lot of spoken words and improve speed.

The number of words used to dictate the module as a matter of interest,
which affects elapsed time and relates to efficiency of the voice
description of the code, was 3934 (elsewhere 3973, so it's possible a line
was left out during the timing run).  This seems quite high, but it might
compare well with other voice coding systems (and considering no spoken
identifiers feature as mentioned above).

Perhaps of interest would be running the same benchmark but with VoiceCode
to see how it compares with Dragonfly - if there's a version somewhere that
supports dictation of Python code.  Or other Dragonfly grammars that may
have been cobbled together for coding.

Jason

Other than the sparing documentation, the only way to find out how
Dragonfly works is to go through the code, Dragonfly and the separately
supplied grammars.

Here are some notes though that may assist.  Needless to say, if there are
mistakes, feel free to comment.

*The Grammar*

*
*

A Dragonfly grammar is defined as a set of rules.  Each rule matches a
pattern of words that are supplied by DNS.  When a phrase is spoken, DNS
sends a sequence of recognized words, and this is matched to one of the
top-level rules in the grammar.  The rule pattern therefore applies to a
single complete spoken phrase.  (A rule may however contain sub-rules as
part of its 'pattern', which may only match part of a spoken phrase.)

*Communication between DNS and Grammar*

*
*

DNS communicates with a grammar through GrammarWrapper.  As implied by
"wrapper", messages are passed through the wrapper to the actual grammar in
Dragonfly.  The communication really involves only two messages from DNS:

beginCallback(contextInfo)

resultsCallback(spokenWords)

DNS sends beginCallback to all grammars (the grammar wrapper in effect)
when some words are spoken as a preparatory step before sending the actual
words. Grammars will deactivate in response if the window context in which
the words were spoken is not relevant, or activate conversely.
Enablement/disablement is another related option.  DNS sends
resultsCallback after the above to all grammars (wrappers) to process the
results of recognition, the spoken words.

*Processing Spoken Words*

When words are spoken and a resultsCallback message received (by
GrammarWrapper), Dragonfly seeks to match the spoken words to a single
top-level rule in the grammar by sending decode() to each top-level rule in
the grammar.  The first rule for which decoding completes is the matching
rule, which is then tasked with processing the spoken phrase though its
process_recognition method.  This handling is shown in the following
GrammarWrapper code:

for each active rule

                attempt to match words to rule (ie attempt decode match)

                if success (ie decoding finished):

                                tree=build a parse tree

                                rule.process_recognition(tree)

An extra step in the process, as can be seen from the code above, is the
creation of a parse tree from the results of decoding. Decoding explores
many paths through the grammar's element structure to match spoken words.
The parse tree describes the matching path found as a construction of
relatively minimal node elements that match words to grammar elements.

Most of the work done by Dragonfly is in decoding (as can be seen in the
log file with decoding output set to Debug).  The implementation of
decoding can be seen in the decode() method of elements.  The decoding code
is actually not straightforward to work out, but it seems to work pretty
well in matching, even if it might have certain preferences (precedence for
example when combining dictation, commands and optionals), and so it's
likely familiarisation with it would be unnecessary.

*
*

*Task of the User: Creating Grammar Rules*

The task for the user of Dragonfly is to create one or more rules, each of
which includes a phrase pattern description which is formed through a
combination of Dragonfly elements (eg. Literal, Group, Alternative, etc),
and some 'action' code in the event that the rule pattern is matched.

Easing the task for matching common kinds of word patterns are the two
derived rule classes: CompoundRule and MappingRule.  The former allows the
element structure of the pattern to be built somewhat automatically from a
specification string e.g. "one [two|three]", although a word maybe enclosed
in <> to indicate reference to another element (which is held in the
variable "extras").  The latter class, MappingRule, includes not only a
pattern description in terms of a set of literals, but conveniently
associates actions with them as well - useful for implementing a mapping as
such.  This is perhaps better suited to short actions like keystrokes, but
with the use of Function actions, it can also accommodate longer actions.

*
*

*Other Bits and Pieces*

Words received from DNS are represented as an array of tuples.  Each tuple
contains a word and a number code.  Mostly this code is 1,000,000 which
means 'dictation word', as opposed to command word.  But the code otherwise
matches to, I think rules or elements or both, in the grammar, but they do
not seem to be that important due to the way Dragonfly processes
recognitions.

There is a compilation part of Dragonfly that converts the Dragonfly
grammar definition to one that DNS understands.  It wouldn't be a concern
to the ordinary user of Dragonfly, and isn't much either to those doing
more extensive work in Dragonfly, unless new elements are being added.
Even then, in terms of defining Dragonfly grammars and handling
recognitions, the compiled grammar would seem almost irrelevant. Whatever
matching DNS may do, Dragonfly redoes in its decoding step during
recognition handling.

The long parser.py module in the root directory of Dragonfly seems to be
for checking syntactic correctness of CompoundRule specification strings,
although would need to recheck this. It may also be used in compilation.