Voice Coding in Blockly

[Diluka]

Hi, I’m Diluka, an undergraduate student working on my final year project. I plan to contribute to Blockly by making it more accessible to students with upper limb motor impairments through voice coding. I would appreciate any advice or thoughts you might have on this idea. Thank you!

[Christopher Allen]

Hi Diluka,

I plan to contribute to Blockly by making it more accessible to students with upper limb motor impairments through voice coding. I would appreciate any advice or thoughts you might have on this idea. Thank you!

We are very excited to hear about work being done to make block-based coding with Blockly more accessible.  We're currently doing some work in that direction ourselves, to try to improve keyboard navigation and compatibility with screen readers, so your work on voice coding would be a great complement to that.  Can you tell us a little more about what approach you are planning to take?

I don't have any specific advice for you, but we're always happy to help with any Blockly API questions you might have, and looking forward to seeing what you create.  We hope you'll share whatever you come up with—as well as your experiences—with us the community here in the forum.

Best wishes,

Christopher

[Mark Friedman]

Diluka,

  I think it's great that you're taking something like this on.  I'll mention a few things about voice coding that you might not have considered.  One is that, in addition to its accessibility benefits, voice-controlled coding could really help enable blocks-based programming on smaller form factors, especially mobile phones, where the limited screen size makes standard drag-and-drop block operations quite challenging and limited.

  Another issue to think about is what level of coding do you want to do via voice.  It could be low-level, where you are basically just giving voice equivalents to existing Blockly drag-and-drop operations, or it could be high-level, where you use A.I. to generate and edit block programs by describing what the program should do, in natural language.  And, of course, it could be both!  For the A.I. based approach, you might be interested in Evan Patton's talk at the recent Blockly Summit, where he discussed the research that MIT App Inventor (which uses Blockly) has been doing with its Aptly project.

  I'll also mention, in case you aren't aware of it, that there is an official Web Speech API (including speech recognition) although it is only fully available for Chrome and Edge and completely unavailable for Firefox.

-Mark

On Wed, Aug 7, 2024 at 3:32 AM Diluka <dilukav...@gmail.com> wrote:

Hi, I’m Diluka, an undergraduate student working on my final year project. I plan to contribute to Blockly by making it more accessible to students with upper limb motor impairments through voice coding. I would appreciate any advice or thoughts you might have on this idea. Thank you!

--
You received this message because you are subscribed to the Google Groups "Blockly" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blockly+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/blockly/c4f3b357-50b1-46f0-a1ac-75b1f8bace56n%40googlegroups.com.

[Diluka]

I aim to generate Blockly blocks using a generative AI approach. I would greatly appreciate any feedback or insights you might have on the following approach:

1. User Speech Input: The process begins with the user providing a speech input.
2. Speech-to-Text Conversion: This speech input is then converted into text using a speech-to-text API.
3. Text Input to Model: The resulting text is fed into a generative model.
4. Model Generates Block Structure: The model processes the text and generates a block structure in JSON format.
5. Blockly Rendering: Finally, the JSON object is rendered in the Blockly environment as a block.

I am keen to understand if this approach aligns with best practices or if there are any potential pitfalls or improvements that I should consider. Any guidance on optimizing this workflow or alternative suggestions would be highly valued.

Thank you for your time and feedback.

[Christopher Allen]

Hi Diluka,

I aim to generate Blockly blocks using a generative AI approach. I would greatly appreciate any feedback or insights you might have on the following approach: …

I have very little relevant experience with generative AI to create Blockly JSON, but I know that some folks have done this with a degree of success (e.g. Aptly).

Having done a little work on the reverse—getting generative AI to create human-readable summaries of what a blocks-based program does—I note that few-shot prompting (e.g. giving the LLM some example inputs and outputs in the prompt) was an easy approach that had gave a huge improvement in output quality over just giving task instructions + desired input.

Fortunately in the project I was working on any output that was reasonably understandable to a human reader was acceptable, but I note that in your fourth step:

 

4. Model Generates Block Structure: The model processes the text and generates a block structure in JSON format.

you have an additional challenge: the generated JSON must not only be valid JSON (I expect the LLM to be pretty good at that) but also a valid Blockly serialisation (I'd expect the LLM to be OK but possibly not always good enough), so you might find it useful to check for deserialization errors and have a plan to deal with them.  You might even try comparing a re-serialisation of the blocks to the original LLM output, to detect if any part of the generated JSON was ignored (and therefore might contain hallucinated properties).

Looking forward to hearing about your results,

Christopher