LLaVA - This Open Source Model Can SEE Just like GPT-4-V

Alan Timm

unread,

Oct 11, 2023, 3:30:45 PM10/11/23

to RSSC-List

This is amazing and would fit in well in an llm-driven robotic solution.

I wonder how it would work out to hook this into camera video feed as a datasource for a robot? Just let it spin full time?

I think a vision pipeline using yolo for realtime object classification and image segmenting, followed by a llava description could be useful. the llava step is computationally expensive but only needs to be done once per identified image segment.

https://www.youtube.com/watch?v=doD7mPuqovU

Project Page: https://llava-vl.github.io/

arXiv Paper: https://arxiv.org/abs/2304.08485

Online Demo: https://llava.hliu.cc/

Prompt example:

"What is in this photo? json format.

{
"people": [
{
"name": "Man on left",
"age": "30s",
"occupation": "Engineer",
"clothing": "Brown shirt",
"accessories": "Watch",
"action": "Talking"
},
{
"name": "Man on right",
"age": "50s",
"occupation": "Engineer",
"clothing": "Black shirt",
"accessories": "Glasses",
"action": "Listening"
}
],
"location": "Warehouse",
"items": [
{
"name": "Robot",
"description": "A large metal robot with a blue arm"
},
{
"name": "Boxes",
"description": "Numerous boxes on shelves and in the background"
}
],
"conversation": "Two engineers discussing a project in a warehouse"
}

Jim DiNunzio

unread,

Oct 18, 2023, 1:43:59 AM10/18/23

to Alan Timm, RSSC-List

A group of people gathered around a table with various electronic equipment, such as laptops, keyboards, and a droid, could provide an opportunity to learn about different aspects of electronics, technology, and robotics. You could gain insights into the design, construction, and programming of electronic devices, as well as the practical applications of these technologies. Additionally, you might learn about the latest trends and advancements in the field, and engage in discussions with experts and enthusiasts who can share their knowledge and experiences. This gathering could also serve as a platform for networking and collaboration, allowing you to connect with like-minded individuals and potentially explore new projects or opportunities in the field.

Wow, that is amazing!

From: rssc...@googlegroups.com <rssc...@googlegroups.com> On Behalf Of Alan Timm
Sent: Wednesday, October 11, 2023 12:31 PM
To: RSSC-List <rssc...@googlegroups.com>
Subject: [RSSC-List] LLaVA - This Open Source Model Can SEE Just like GPT-4-V