Real-time object recognition with recent gaze

[Jesse Weisberg]

Hi Pupil Community!

I made a Capture plugin so that you can run real-time object recognition on the world view and identify which object is closest to your gaze in real-time.  I also overlaid visualizations on the world view to show these things in real-time.  Here's a short video demonstration: https://youtu.be/MqqQnCbzryA

I hope some of you find this useful in your Pupil endeavors!  You can view/grab the code here:  https://github.com/jesseweisberg/pupil

Also, I'd love to hear suggestions on how you think I can make this plugin better or what applications you think can stem from this. 

-Jesse

**Note: I'm using Tensorflow with only CPU here, so it slows my capture down to about 10-15 FPS.  If you have a GPU it should work much faster.

[aar...@stanford.edu]

Great job, Jesse!  I noticed you also had a project controlling a robotic arm with ROS via an Arduino.  I'm starting a project that's kind of a hybrid of what you've done: I want to use Pupil data to control a couple of servos via Arduino.  I was wondering if you'd be around this forum to provide advice as needed?

Thanks,
Aaron

[Jesse Weisberg]

Hi Aaron– yes I'll be around, ask away!