Rudimentary speech-to-text (ugh... 'dictation') and random stuff about the clipboard

[Gary]

Ok, guys.  For as long as I can remember, we (we being this group) have been wanting to be able to capture speech as text to use as we see fit. I've been reluctant to do this for a long time for various reasons, but now I'm starting to implement this feature since the demand is getting high.  Just want to do a brain dump to let you know what's up, so forgive the stream of crap ahead... lol

To start, what I have so far is an action to stuff a text value (including tokens) into the clipboard.  This opens up the ability to paste text values wherever you need to.  I'm not a fan of the, 'Quick Input' feature that was implemented in VA a long time back (replaced by the Recorder), which would type out individual key presses from a string of characters (I know it sounds and seems simple, but there's a lot to consider under the covers... mostly due to languages).  So, I think the ability to paste text will be faster and more reliable than individual character processing, and simplifies the user interface somewhat.

The clipboard feature was probably the easiest thing implemented in a while (of course).  However, as the user interface for VoiceAttack becomes more and more bloated, I have tended to shy away from making specific interfaces for stuff and focusing more on making building blocks that can be used for whatever they can be used (you'll notice and lament over the whole idea that there is no specific interface for adding random sounds... that will probably come later, but the point is that I could spend all my time building and maintaining an interface for one thing or make simple building blocks to support it... now you can randomize anything (not just sounds)).  The clipboard set feature is just that... a building block to stuff a value into the clipboard and that is all (also there is a new token, '{CLIP}' that will allow you to access what is in the clipboard). 

In keeping with the building block idea, I've decided to take that route with, 'dictation'.  There are currently three new actions, all considered, 'advanced' since there will be no other user interface to support them, and anything in the dictation buffer must be accessed via token.  The actions are start dictation, stop dictation and clear dictation.  The start and stop dictation actions have the option to clear what is in the dictation buffer.  The clear dictation action has the option to just remove the last thing that was said.  As mentioned, there is a dictation buffer that gets filled with each phrase you speak.  There is the usual pause that indicates the end of your statement, and whatever was said will be added to the buffer.  Nothing special here... you'll have to add your own punctuation as you normally do with the Windows speech engine.  Multiple entries in the buffer are just appended to one another when you access the values from the {DICTATION} token.  Recognized commands will be omitted from the dictation buffer if you speak them on their own (that is what I am testing now... the ability to be able to keep dictation going and still say, 'power to shields' and then resume dictation).  Since they are blocks, you can incorporate them in your own commands with whatever spoken value you want ('open quote', 'quote', 'begin dictation'... or all of those... lol).  You can incorporate your own stop and start sounds and initialize whatever variables you want to or even call a plugin.  New users to VA will probably hate this approach, but I apologize. 

The ugly side of this is now you will experience the joy of what is the dictation ability of the Windows speech engine.  It is not Dragon.  Not even close.  You will say things clearly and it will miss a lot.  It's kind of what has deterred me from implementing, 'dictation' in VA.  I can see this becoming somewhat of a support sore spot, since VA gets hit all over the web for not working right, even when it comes down to the speech engine (properly configured system or not).  I guess this would be a good time to say that I'm sorry, but it *is* eight dollar software, and this is definitely not the domain or even a main feature of VA...  might not even stay if beta testing runs afoul.

Ok... enough rambling.  Hope you are having a great day :)

Gary

[Damen Smith]

Gary, I have to say, after spending the last week trying to build this kind of functionality into my profiles, I'm very grateful that you gave us the ability to do just that. This is the most fun I've had with a piece of software since I first learned Adobe Illustrator. Again, great work, as always.

[iBuilderLive on Twitch]

let me know when you need some testing done mate.  VA rules.

[Zephy]

Thanks for working on that, Gary. You shouldn't have to apologize for Window's failure to properly recognize speech, or your customers' failure to just do the speech training like they're told. I'm not a native english speaker and I have no issue getting my commands to work.

[Aaron Goselin]

Amusingly, most non-native English speakers enunciate far better than native speakers, even if the accent is different.

[Koz Myk]

Hmmm Interesting.
I wondered why I hadn't been seeing Gary type comments lately.
I had had G+ open on a single thread ... Doh!

[Christian Simard]

I just opened a new tread about a disregard function that might apply to this !

I would be very interested to have your thoughts on it !

[nurmi]

Seems to be working pretty nicely!

One feature that would combine AWESOMELY with this would be to store the length of the last thing inserted to clipboard in a condition, and the ability to repeat a command based on the value of a condition. This would allow things like "delete last" by repeating the correct number of backspaces.

...but possibly this is veering into something that is best done in a plugin?

[Rhaedas]

Or how about just a way to get the length of the {DICTATION} variable? That makes it more flexible to use that number for however the contents need to be manipulated. Since the dictation command has a way to insert spaces, capitalize, etc, then just a count of the data should be easy, right? I have an idea on how to to streamline some commands using this, but it won't work if I don't know how long the variable is.

[Gary]

I'll look at that... should be a snap

{DICTATIONLENGTH}

Probably going to add in text compare (contains/starts with/ends with, etc) in some form to go along with condition blocks (as per multiple requests).  Hoping to do this soon... just wondering if that would help you more in your endeavor... ?

Gary

[Rhaedas]

Sounds interesting. I'm trying to create some user work arounds in the routines to make up for the Windows recognition. It's not too bad, but in a gaming setting, you can't take time to go back and edit things, so I'm thinking of ways to make corrections in real time, much like you would if you were typing it in. So keep the additions coming, they are always useful.

[Gary]

I started to add a {DICTATIONLEN} token, but I stopped because with the latest release, there is a new token, '{EXP:expression}' (in the latest beta, see the notes on the site and also in the help document).  You should be able to do this :

{EXP:Len('{DICTATION}') } or {EXP:Len('{DICTATION:LATEST}') }

This will give you a string result that can then be stuffed into a condition (new Convert Text/Token feature in 'Set Condition Value' action) or passed to a plugin or whatever (see attached).

Hope that makes sense for you!

Gary

On Sunday, March 8, 2015 at 12:39:17 PM UTC-5, Rhaedas wrote:

[Steve Rush]

Just wondering if anyone ever got this to work, and are using it in Elite.

Would be great to speak messages rather than have pre set or having to type them.

[Gary]

Can you paste text in E:D (like with CTRL + V or SHIFT + Insert) ?

If so, it should work.  I have some tweaks that I've done that will be in the next beta (hoping for getting it out today).

Gary

[Christian Simard]

Yes, CTRL + V works fine in ED text chat, 

but not SHIFT + Insert does not.

This would be a really nice command to have !!!

[Steve Rush]

yes, i can have VA press CTRL + V in the coms menu and galaxy map, it does work in Elite. i was just needing some sample commands to try, as i wasn't sure about the dictation token.

 (how to use it)

so if i change this for elite, it should work i hope.

[Steve Rush]

hey Gary, thanks to you, i finally got it working in Elite Dangerous, check this out, and feel free to use it for any thing you want.

http://plays.tv/video/5554a02430baf18ddc

[Sutex Osofar]

Ok got these below working,.....well its seem to be

/r to respond to the last ship that you received comms from
/t to send comms directly to targeted ship
/w to send comms to all wingmen
/l for local channel

Post is here

https://forums.frontier.co.uk/showthread.php?t=144958&p=2236157&viewfull=1#post2236157

[Sutex Osofar]

Have posted the reply to ship, commander and other here , using voice dictation
https://forums.frontier.co.uk/showthread.php?t=144958&p=2236157&viewfull=1#post2236157

[Charlie Blond]

I don't know when I'll use this since I play different games but this is still very cool. It was a difficult to understand but I'll just keep playing around with it.

[SnowSultan]

I just purchased Voice Attack today and it's already well worth the price just to be able to set up user-defined commands for running programs rather than relying on the ones Windows looks for. Thank you for this terrific product!

I'd really like to experiment with dictation using Voice Attack, but I'm not quite understanding what you're talking about here. Are there any examples of using the start dictation, stop dictation, and clear dictation (with the ability to paste the buffered text) that I could look at and try to figure out? I tried making my own command and although it recognizes the dictation commands, nothing gets pasted when I tell it to.

Thanks in advance for any advice, it's appreciated.

[SnowSultan]

...never mind, I got it working thanks to this post

https://forums.frontier.co.uk/showthread.php?t=120497&p=2229866&highlight=#post2229866

Amazing work, you've managed to combine a game keybinder with accurate voice interactions that works better than most dedicated software. Will definitely recommend Voice Attack in the future!

[Sylvain Martin]

Trying to adapt this for World of Tanks,

 got the beta VA, did the open channel which turns on dictation

then

 send message does

copy {DICTATION} to windows clipboard

then a ctrl-v

turn off dictation

just trying to paste in notepad and I get nothing, I tried to set copy to clipboard to "msg>>>{DICTATION}<<<msg"

and all I get is msg>>><<<msg pasted.

I wonder if DICTATION is not working in Windows 10?

 

[Gary Magenheimer]

Hi, Sylvain.  

I will check into that but it will be sometime next week (I will be out of pocket til at least Monday evening).

In the meantime, would you mind sending me the profile with the commands you are using to sup...@voiceattack.com?  I can do a quick run through and see if there's anything I can spot.

Gary

--

---
You received this message because you are subscribed to the Google Groups "VoiceAttack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to voiceattack...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gary Magenheimer]

Also, the next release of VA will allow for text to be serially output one character at a time (for the handful of supported keyboards) for those that can't use the clipboard in-game (like Star Citizen).  

Gary

On Aug 19, 2015, at 6:23 PM, Sylvain Martin <sylvain...@gmail.com> wrote:

--

[Sylvain Martin]

Thanks Gary for your support!

I figured it out!
I use ActualMultipleMonitors. It is somehow interfering with VA. while I was playing with VA, AMM crashed and VA started working properly.

then I starting digging into it and while AMM is running, I issued a VA command and the active window looses focus but still appears to have it and the focus is now set to VA. Guess I'm ditching AMM because VA is a heck of alot more useful to me 

Dictation now works and voice feedback from the commands also works with AMM not running

[Mark Bolt]

So I am having difficulties with the speech to text. I want to say something, and have it send to notepad so I want it to be like

Record

I say: Today is a good day to do stuff

Send to Notepad

End Recording

but when I get to the third step it does...NOTHING!!!...and I've tweaked and played with all kinds of settings with VA and nothing. I did fdsa, did copy/paste with ctrl v and c...works..when I say Record and do ctrl v it gives nothing..and keeps giving nothing all the way down the line of commands. I don't know what to do :( if someone could possible guide me or just "give" (I you are so kind lol) me a working program it would help me greatly.

Giving me a working copy would also allow me to study how your application of commands were.

Thx! <3 

[Gary]

Hi, Mark.

I'm wondering if this got fixed in the latest beta :  http://www.voiceattack.com/beta.aspx 

If you would, please give it a try and see if it helps any.  If not, we'll go from there.

Gary

[Mark Bolt]

Gary

Thanks for that, it seems to be working better than it did.

Another issue, not as major but something  with left mouse clicking seems to be broken.

I have a command to move mouse pointer in a place (works) but when it goes to click, it doesn't...well it does, but nothing happens...then it continues moving the mouse around where I want it

So when it comes to left mouse click it seems to be kinda not working as intended 

[Gary]

Make sure there is somewhat of a pause between your mouse down and mouse up.  DirectX games work on a polling mechanism and if events occur too quickly then the game will not pick them up.

Hope that helps!

Gary

[cmdr.m...@gmail.com]

I've been playing around with dictation but it's hard to use when there are multiple actions that might want the use dictation (although not simultaneously).

Would it be possible to have a "Dictation (single phrase)" command that waits until a phrase has been spoken, fills the buffer, then stops dictation and continues the action?  That would be a massive help to retain the flow of actions.

[cmdr.m...@gmail.com]

Having a single phrase dictation as per below would make interaction with VA in a number of situations an awful lot easier.  Is it something that could be considered?

[Gary]

I believe you can do this now by using the, 'execute command each time a dictation phrase is recognized' option in the profile option screen.  You can set up a command to turn off dictation and then do whatever you want to the buffer after the single dictation phrase is recognized.  You can also use variable flags in your dictation commands to indicate whether or not you want single-line dictation or continuous.

Hope that helps!

Gary

[cmdr.m...@gmail.com]

Thanks for the response.  Although this helps a bit it is still very painful if you have multiple commands requiring dictation as you will need to keep track of which commands are active and put the logic for handling the dictation in the single command run after the dictation phrase is recognised.

So you have:

  - partial commands that start dictation but don't handle their results

  - a requirement to set an item in the profile configuration (which, by the way, fails to be retained if the command is re-imported in to the profile)

  - a "master" command that is run after each dictation phrase and has to have complete knowledge of every command in VA that uses dictation to be able to do the right thing

If you compare the complexity and inability to extend this compared to something as simple as a single command with:

- Say 'How many light years is it to Sol?'

- Start single dictation

- [Do something with the dictation result]

Then hopefully you can see that the latter is a lot easier to write and manage.

[Gary]

I think what I need is a, 'last spoken command' token.  That might help in this case.  That way, you could turn it around and just have a single command for continuous dictation and the rest can fall into the catch-all.

I'll put this on the todo.  Hopefully in the next release.

Gary

[Gary]

Just to let you know, I started thinking about adding a, 'limit dictation to X phrases', since if I add just a single phrase, the next emails in my inbox will be, 'that's great, but can you make it do 2? 3? X?'  I ended up hitting too many walls trying to make it usable, plus there's the whole, 'what to do' when the limit is reached.  It would make that interface way more difficult to understand.  

For now, in the latest beta there is a new token called, '{LASTSPOKENCMD}'.  This returns the last spoken command (the one prior the current command {CMD}).  This can be used for different stuff, but in this case it can be used in the command that is executed by, 'Execute a command every time a dictation phrase is recognized' (or whatever it's called) on the profile options screen.  In that command, you only need to check for what spoken commands represent FULL dictation commands (not the single phrase ones that you have many of).

The command would look something like this:

Begin Text Compare : [{LASTSPOKENCMD}] Does Not Equal 'open quote'     <--- 'open quote' is the command would be the command that does the full dictation

    Stop Dictation Mode

    /* Execute some other command here to dump the dictation buffer or just do whatever inline */

End Condition

I tried it out and it seems to work.  Let me know if I've totally muddied the water o_O

Gary

On Sunday, April 17, 2016 at 8:57:43 AM UTC-5, cmdr.m...@gmail.com wrote: