BBB auto-translation (speech-to-speech)

[manish...@gmail.com]

Hi,

I need to find out how to extract the Speaker's speech in real-time during a BigBlueButton session. Any idea?

Background

Recently we implemented a new feature that would translate (Speech-to-Speech) a speaker in a live BigBlueButton session in real-time in 100+ languages. 

Here is how we have achieved it:

1. Enable auto-captioning/Start Dictation during a BigBlueButton session
2. Extract speech-to-text strings from Redis for the given meeting
3. Translate the extracted text into (say) Spanish using Google translate
4. Convert translated Spanish text into Spanish speech using Google text-to-speech
   
5. Broadcast translated Spanish speech to the browser (an html5 page with <audio/> tag) using Node/Socket.io

The Speech-to-Speech translation works accurately and in real-time. 

However, it has one limitation. 

I am not sure how BigBlueButton is converting Speech-to-text but it misses punctuation marks. The speech-to-text converted by BBB has no punctuation marks. For example, it won't have question marks or escalation marks. 

So the translated speech will lack the emotions of the speaker. 

Next Step

The solution is to not rely on BBB's speech-to-text translation but to extract speaker's audio and convert it into speech using Google's speech-to-text that supports punctuation marks. 

Hence, I am looking for a way to extract speaker's audio in real-time during a BigBlueButton session. 

Any help would be greatly appreciated!

- Manish

[Hiroshi Suga]

Not entirely answering your question, but in my understanding, BBB does not do anything about the speech recognition, it just uses the browser's function. And that's the reason why this function works only on Chrome at least for now.