Having Optimus learn straight from internet videos
Gmail
From LinkedIn:
Milan Kovac • 1st
VP, Head of Engineering - Optimus (Tesla...
21h • 0
One of our goals is to have Optimus learn straight from internet videos of humans doing tasks. Those are often 3rd person views captured by random cameras etc.
We recently had a significant breakthrough along that journey, and can now transfer a big chunk of the learning directly from human videos to the bots (1st person views for now).
This allows us to bootstrap new tasks much faster compared to teleoperated bot data alone (heavier operationally).
Many new skills are emerging through this process, are called for via natural language (voice/text), and are run by a single neural network on the bot (multi-tasking).
Next: expand to 3rd person video transfer (aka random internet), and push reliability via self-play (RL) in the real-, and/or synthetic-(sim / world models) world.