Try AI Integrated Blockly workspace demo - TutoriaLLM

[So Tokumaru (Soumame1090)]

Hello! I’m So Tokumaru ( https://tokumaru.work ), high school student from Japan in Malaysia, working on ways to leverage large language models (LLMs) for K-12 programming education. Recently, I built a project called "TutoriaLLM," a self-hosted application designed to support educators and students through a multi-modal LLM that offers guided tutorials.

To intergrate LLM(AI) into programming education, I have used Blockly as the framework of code editor, and intergrate LLM throughout workspace to toolbox of Blockly.

I will share these findings. 

Demo of LLM Block Suggestion

This feature allows LLM to directly suggests block to use on Blockly. Response from LLM will be parsed and If it is matched with known Blockly's Block, highlight thhe category, subcategory, and block within that particular flyout.

https://i.gyazo.com/002a9d49d1a377a4bd9c23c09e3ef826.gif

Demo of LLM Workspace highlight

This feature inherit some features from Block Suggestion. Analysing LLM's response to find specified block id of blockly from the text, and if it is matched with block id within the workspace, it appears as button and allows to highlight for user. 

https://i.gyazo.com/31abfd751e0584aacefefdede66b9997.gif

These ideas will be very useful in combining AI and CS education in the future. I have already deployed a demo of the app for you to try!

Documentation: https://tutoriallm.com

Demo: https://demo.tutoriallm.com

Repository: https://github.com/TutoriaLLM/TutoriaLLM

The app also provides functionality to run the code you create in Minecraft. You can also select a tutorial, where the AI will teach you how to create a programme. For more information, please ask the AI.

As this app is still developing, so if there is any question, ask me on this thread! 

[Aaron Dodson]

This looks really cool, thank you for sharing!

[m lz]

I know like chatgpt or claude ask question could return code,but how to make code convert to block,want to know some detail

[So Tokumaru (Soumame1090)]

As recent LLM has capability to read/write JSON, you can use the serialization system of Blockly (https://developers.google.com/blockly/guides/configure/web/serialization).

However, to convert output from LLM into Blockly's blocks,  LLM may returns inaccurate result in some time, so you have to validiate returned value.

In my project( https://github.com/TutoriaLLM/TutoriaLLM), I've set up block suggestion system instead of LLM directly writing JSON , LLM just returns block name or block id, based on prompt, to instruct user to build a code. 

2024年12月23日月曜日 6:49:58 UTC+8 freder...@gmail.com:

[m lz]

if blockly could add a bi-direction convert feature seems very useful,code conver to blocks,blocks convert to code,select code could also select relevent blocks,select blocks also highlight relevent convert codes

[ewpa...@gmail.com]

Assuming the text code is read only, that should be fairly straightforward to implement. The challenge with editable code is that usually the blocks are a subset of all of the syntax/semantics available in the target text language, so there are constructs you can express in the text representation but not the blocks. For MIT App Inventor, we specifically designed a text language called Aptly to ensure a one-to-one correspondence between the two representations for precisely this reason.

Cheers,

Evan

[So Tokumaru (Soumame1090)]

TutoriaLLM is converting blocks into javascript with javascript generator of Blockly(Currently its one-direction), so I think it is possible.

For example, Makecode has own bi-direction convert feature to convert user's blocks into Javascript or Python.

2025年1月25日土曜日 7:05:35 UTC+9 ewpa...@gmail.com: