Live speech-to-speech translation via AI hooks

[M P]

Hi All,

I did comb through earlier posts on this subject, but there is nothing recent.  Given the pace  of how AI has advanced in key technologies, it seems live speech translation should be a possibility.

First, some background:
I have created a couple Moodle plugins (local and filter) that handle bidirectional translation of all course content and asynchronous student interactions, making mixed language cohorts possible.  The results are impressive and work on all text, and asynchronous audio and video.

The next step is to target live, synchronous course activities, such as a BBB lecture.  You may have seen Google's own demo of something like this in Google Meet last year: https://youtu.be/hyXqcsWOONo.

We have used AWS bedrock services for all our other AI translation, and AWS' recent Nova Sonic model seems a well-suited, cost-effective solution for live speech-to-speech translation.

All that to say, has anyone else had a look at this? I might have a go at creating a BBB plugin for something like this, but I wanted to test the waters with experienced devs to see how feasible this would be. Thoughts?

A small caveat to the above is that language support for any kind of translation is limited to "high-resource" languages.  I have tried to target a few African languages in the Niger-Congo phylum, but they are "low resource" languages, i.e. there is insufficient AI model content to make translation reliable enough for use.

[Günther Schwienbacher]

Hello,

maybe you could have a look on the speech-to-text transcription / translation feature of BBB via gladia.io:
https://docs.bigbluebutton.org/administration/customize/#configuration-of-gladiaio

Together w/ this plugin https://github.com/bigbluebutton/bbb-plugin-live-transcription (assuming you're using BBB v3) live stt and (if configured) also live translation is possible.
If you are using LiveKit as audio gateway, use bbb-livekit-stt (https://github.com/bigbluebutton/bbb-livekit-stt) instead of bbb-transcription-controller as gladia proxy.

Don't know if this fits your need, but it's worth having a look.

(btw: there's also work going on in supporting other Realtime STT providers (e.g. Voxtral))

Best,
Guenther

[M P]

Excellent ideas, Günther.  Thanks for your feedback.

I think STT is the easiest to achieve and low-hanging fruit.  I will probably start there, but I would like to scale to speech-to-speech at some later point.

All the best,

Mathieu