📕 [New Tutorial] MiniCPM-o 4.5 : A 9B Model That Sees, Hears, and Speaks in Real Time
13 views
Skip to first unread message
Carl
Aug 12, 2026, 2:49:02 PMAug 12
to RSSC-List
FWIW, got this in an email - have not checked it out
Article:
Experiments:
Most multimodal assistants stop listening the moment they begin to answer.
That is fine for a chatbot. It is much less useful for live video, where the scene can change before the response is finished.
MiniCPM-o 4.5 takes a different approach. This 9B open-source model can keep watching a video and listening to audio while it speaks. In other words, it behaves more like a conversation partner than a turn-based assistant.
In our new article, we unpack the architecture and put its full-duplex mode through four hands-on tests:
• Real-time football commentary that speaks when something notable happens
• Retail surveillance narration that tracks behavior across a scene
• Cooking-video Q&A that handles an interruption without losing context
• Zero-shot voice cloning from a short reference clip
The key is Omni-Flow. It aligns incoming video, incoming audio, and generated speech on one shared timeline. Every second, the model decides whether to listen or speak. TAIL then keeps the response from drifting behind the live scene.
It is promising, but not magic. Proactive behavior is still basic, long sessions can produce occasional speech errors, and full-duplex deployment needs considerably more memory than a text-only benchmark suggests.
That is fine for a chatbot. It is much less useful for live video, where the scene can change before the response is finished.
MiniCPM-o 4.5 takes a different approach. This 9B open-source model can keep watching a video and listening to audio while it speaks. In other words, it behaves more like a conversation partner than a turn-based assistant.
In our new article, we unpack the architecture and put its full-duplex mode through four hands-on tests:
• Real-time football commentary that speaks when something notable happens
• Retail surveillance narration that tracks behavior across a scene
• Cooking-video Q&A that handles an interruption without losing context
• Zero-shot voice cloning from a short reference clip
The key is Omni-Flow. It aligns incoming video, incoming audio, and generated speech on one shared timeline. Every second, the model decides whether to listen or speak. TAIL then keeps the response from drifting behind the live scene.
It is promising, but not magic. Proactive behavior is still basic, long sessions can produce occasional speech errors, and full-duplex deployment needs considerably more memory than a text-only benchmark suggests.
Alan Timm
Aug 21, 2026, 9:46:04 PMAug 21
to RSSC-List
I started digging into it and don't know quite what to make of it.
Getting out of the turn-based audio interactions is intriguing. I'll get Lloyd to take a look and see if we can use it.
Thanks!
Alan
Reply all
Reply to author
Forward
0 new messages