ROS2/Nav2 and AI integration

94 views
Skip to first unread message

Sergei Grichine

unread,
Sep 5, 2026, 1:30:13 PMSep 5
to hbrob...@googlegroups.com
Hi All,

There was a lot of discussion during our Tuesday ROS SIG meetings about controlling ROS2 robots via voice commands. While it is possible to generate ROS actions or trigger pre-programmed scripts using prompts, a more architecturally sound approach is to use an Model Context Protocol (MCP) server.

There are already some great existing packages that handle this well.

Here are my (well, the Overlords') notes on the topic:


Comments and feedback are welcome!

Best Regards,
-- Sergei

Michael Wimble

unread,
Sep 5, 2026, 1:43:36 PMSep 5
to hbrob...@googlegroups.com
Thanks. I was just watching a YouTube on this. Good work. 

On Sep 5, 2026, at 10:30 AM, Sergei Grichine <vital...@gmail.com> wrote:


--
You received this message because you are subscribed to the Google Groups "HomeBrew Robotics Club" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hbrobotics+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hbrobotics/CA%2BKVXVPRg9qKTD5zEaAwDi9pn5Cq4ruUqNXd%3DAQzk-tDiutc8Q%40mail.gmail.com.

James H Phelan

unread,
Sep 5, 2026, 6:09:35 PMSep 5
to hbrob...@googlegroups.com

YouTube link?

James H Phelan
"Nihil est sine ratione cur potius sit quam non sit"
Leibniz

Michael Wimble

unread,
Sep 5, 2026, 6:33:38 PMSep 5
to hbrob...@googlegroups.com

James H Phelan

unread,
Sep 6, 2026, 11:51:05 AMSep 6
to hbrob...@googlegroups.com

Very interesting.

There appear to be LOTS of MCPs out there or you can make your own.

How do you choose or how do you make one for your robot?

I should think Titania would need more than Stormy or Tablebot.

So, of course, I Googled it: "ROS2 MCP server"

(I had to fill in some links that failed to copy)

The Model Context Protocol (MCP) has rapidly gained adoption within the robotics ecosystem, enabling large language models (LLMs) and AI assistants (like Claude, ChatGPT, or Cursor) to natively interface with ROS 2 architectures. Instead of requiring complex hardcoded scripts, an MCP server exposes the ROS 2 graph directly to an AI agent as structured tools. [1, 2, 3]

Here is an overview of the primary open-source projects providing ROS 2 MCP servers:

1. Full-Scale ROS 2 Interaction Bridge

These packages turn the AI agent into an active node on the ROS 2 graph, giving it the ability to introspect, publish, and subscribe.[https://discourse.openrobotics.org/t/amazing-ros-2-mcp-native-ros-2-mcp-server-for-ai-assisted-robotics-agents/55270]
  • Amazing ROS 2 MCP: https://discourse.openrobotics.org/t/amazing-ros-2-mcp-native-ros-2-mcp-server-for-ai-assisted-robotics-agents/55270: A native, rclpy-driven server that operates without needing rosbridge or WebSockets, offering single-digit millisecond latency.
    • Key Features: Full graph introspection (topics, services, actions, nodes); auto workspace discovery for custom messages; built-in image capture support for vision LLMs; and physical safety guardrails like velocity clamping and dry-run modes. [2, 4]
  • robotmcp / ros-mcp-server: https://github.com/robotmcp/ros-mcp-server A popular framework providing bi-directional communication between LLMs and robots. It provides deep ROS environment understanding out-of-the-box and works seamlessly across ROS 1 and ROS 2 distros (such as Humble and Jazzy). [https://github.com/LCAS/ros2_mcp]
  • wise-vision / ros2_mcp: A highly automated Python implementation listed in the official MCP registry.
    • Key Features: Automatic type discovery for finding available topics, nested field support for complex custom message types, and native Docker configurations. [6, 7]
2. Specialized Navigation & Debugging Servers

If you don't need a broad network bridge, specialized servers focus on specific aspects of the ROS 2 stack:
Basic Architecture Overview

When configuring a typical deployment (such as combining Claude Desktop with a localized ROS 2 simulation), the connection architecture generally follows this pipeline:  
[Claude / IDE Client] ──(via SSE / Stdin)──> [ROS2 MCP Server] ──(via rclpy/DDS)──> [ROS2 Nodes / Robot]  [3, 5]

For external or remote AI clients, the connection is typically exposed securely over an SSE interface utilizing proxies like mcp-proxy. [5]

Are you looking to set up an MCP server to control a physical or simulated robot in real-time, or are you looking to use AI to debug existing ROS 2 log/bag files? Let me know your specific use case, and I can provide configuration or code snippets.
AI responses may include mistakes.

"Please compare the 3 full-scale bridges"

While all three of these frameworks implement the Model Context Protocol (MCP) to expose a robot’s interfaces to AI clients (like Claude or Cursor), they vary significantly in underlying architecture, safety controls, and target use cases. [1, 2, 3]
The primary structural differences between Amazing ROS 2 MCP, robotmcp / ros-mcp-server, and wise-vision / ros2_mcp highlight how each bridges the gap between LLMs and robotics:

Direct Comparison Overview

Feature / Attribute Amazing ROS 2 MCP robotmcp / ros-mcp-server wise-vision / ros2_mcp
Communication Layer Native (rclpy with background spin-thread) Bridge-dependent (rosbridge / WebSockets) Native (rclpy over standard stdio transport)
Supported ROS Versions ROS 2 only (Humble, Jazzy, etc.) ROS 1 and ROS 2 ROS 2 only
Latency / Performance Single-digit millisecond latency (DDS speed) Moderate (limited by WebSocket translation layer) Low/Standard (DDS via direct stdio transport)
Built-in Safety Layer High (Velocity clamping, dry-run mode, blocked topics) Low (Relies on client-side prompt constraints) Low/Medium (Auto-QoS matching to prevent crashes)
Cloud/Remote Access Local focus (requires proxy for remote access) Native Cloud Tunnels (via Cloudflare + OAuth) Local focus (optional InfluxDB plugin)
Specialized Toolkits Native Nav2 integration out-of-the-box General robot control submodules General data analysis & debugging

Detailed Analysis of Each Bridge

1. Amazing ROS 2 MCP


  • The Architecture: It operates as a true native ROS 2 node. Instead of converting messages through an intermediate web server, it embeds an asynchronous rclpy node using a background spin-thread pattern. It talks directly to the standard ROS 2 middleware layer (DDS). [3, 4]
  • Biggest Strength: Hardware Safety and Performance. Giving an LLM direct execution power over hardware can be unpredictable. This project is the only one featuring native safety guardrails—such as velocity clamping on /cmd_vel to prevent runaway robots, hard-coded topic blocklists, and a safe "dry-run" simulation mode. [3, 4]
  • Best For: Real-time hardware control, low-latency applications, and mobile robots leveraging the ROS 2 Navigation Stack (Nav2). [3, 4]

2. robotmcp / ros-mcp-server


  • The Architecture: Unlike the others, it relies on a rosbridge WebSocket connection to link to the robot. The MCP server functions as an external wrapper talking to that web bridge rather than spinning up its own ROS node. [5, 6]
  • Biggest Strength: Legacy Compatibility and Cloud Infrastructure. Because it relies on rosbridge, it can cross the generational gap and control both ROS 1 and ROS 2 systems natively without modification. Furthermore, the robotmcp ecosystem includes ready-made enterprise features like Cloudflare tunneling (to safely control a robot remotely over the internet) and OAuth 2.1 user access control. [1, 7, 8]
  • Best For: Heterogeneous fleets containing legacy ROS 1 robots, or deployments requiring cloud-to-robot control over remote networks. [1, 8]

3. wise-vision / ros2_mcp

  • The Architecture: A streamlined Python implementation focused entirely on the Standard I/O (stdio) transport protocol preferred by native desktop clients like Claude Desktop or Cursor.
  • Biggest Strength: Developer UX and Introspection. This server specializes in flattening complex, nested custom message types so an LLM can easily map fields without crashing. It features a robust Auto-QoS (Quality of Service) selection engine that detects the publisher's QoS settings dynamically, resolving a common headache where an AI agent fails to echo data due to mismatched transient/reliable policies.
  • Best For: Rapid local prototyping, AI-assisted code generation in IDEs like Cursor, and natural language data-querying of sensor fields. [2, 6]
To help tailor a recommendation, which of the following closely matches your architecture?
  1. Are you working on a local machine/simulation or a remote physical robot?
  2. Do you require the LLM to actively drive a robot around, or are you primarily using it to inspect telemetry data?

"With Amazing ROS 2 MCP what kind of proxy for remote access?"

Because Amazing ROS 2 MCP runs as a native, local rclpy node, it natively communicates via standard input/output (stdio) JSON-RPC payloads. It does not bundle a web server or a built-in cloud network. [1]
To bridge an external AI client (like a remote cloud LLM or an external tool) to a robot running Amazing ROS 2 MCP, two primary types of proxies are utilized:

1. Transport-Layer Proxies (MCP Specific)

Most native desktop AI clients (like Claude Desktop or Cursor) require a local stdio subprocess to execute an MCP server. If your robot is in a lab and your AI development client is on a different machine, you use an MCP Proxy to bridge across the network: [1, 2]

  • mcp-remote / mcp-proxy: These utility shims run locally on your development machine. They intercept the stdio from your AI client and convert the traffic into Streamable HTTP or Server-Sent Events (SSE), forwarding it over the network to the robot host. [2, 3, 4, 5]
  • Reverse Proxies (Caddy, Nginx, or OAuth2-Proxy): On the robot machine side, you use a standard web server proxy to accept those incoming SSE or HTTP requests and pipe them straight down into the Amazing ROS 2 MCP process's stdio. Adding OAuth2-Proxy is highly recommended here to prevent unauthenticated access to your physical hardware. [6, 7]

2. Networking-Layer Proxies (ROS 2 Specific)

Instead of exposing the AI-to-MCP transport layer, a common alternative approach in robotics is to run the MCP server on your remote development machine and use a Robotics Network Proxy / Tunnel to stretch the ROS 2 graph to the machine where the LLM is. [8, 9]

  • Zenoh (zenohd): This is the modern industry standard for remote ROS 2 communication. You run a Zenoh router endpoint on both your robot and your local workstation. It bridges the DDS network safely across routers, allowing the Amazing ROS 2 MCP node running on your desk to see and talk to the robot's topics as if it were plugged in via ethernet. [9]
  • Mesh VPNs (Tailscale / ZeroTier): You can put both your robot and your AI development machine on a virtual private mesh network. This secures and wraps the ROS 2 DDS middleware (via CycloneDDS or FastDDS) inside an encrypted WireGuard tunnel so the local node can query telemetry data seamlessly. [10, 11]

Depending on your project setup, which side are you trying to keep local?

  • Do you want to run the MCP server directly on the robot and have a remote AI connect to it?
  • Or would you prefer to run the MCP server on your laptop and tunnel the ROS 2 data from the robot?

"I'm running NAV2 on the robot and want to be able to access LLMs on the web for possible voice command, image identification, navigation, manipulation, etc."
To connect a web-based AI or a cloud LLM to Amazing ROS 2 MCP running locally on your robot (which handles Nav2, vision, and manipulation), you need a proxy pipeline that bridges incoming cloud network traffic into the local stdio (Standard I/O) process of the MCP server.
Because you are dealing with critical robotics functions like navigation and manipulation, security and low latency are paramount. The standard, recommended proxy architecture for this specific use case is an SSE (Server-Sent Events) Gateway.

The Recommended Architecture: SSE + Secure Tunnel

Cloud LLMs cannot directly execute a local terminal process on your robot. Instead, they communicate over the web via HTTP. You can bridge this gap using the official Model Context Protocol (MCP) SSE Gateway combined with a secure network tunnel.
[Web/Cloud LLM] ──(HTTP POST/SSE)──> [Secure Tunnel / Reverse Proxy] ──> [MCP SSE Gateway] ──(stdio)──> [Amazing ROS2 MCP] ──(DDS)──> [Nav2 / MoveIt / Vision Nodes]

1. The Gateway: @modelcontextprotocol/server-sse

The official MCP team provides an npm package designed exactly for this. It runs on the robot alongside Amazing ROS 2 MCP. It creates a lightweight web server that listens for incoming HTTP POST requests from the cloud, translates those requests into stdio format, and pipes them directly into the Amazing ROS 2 MCP python node. It then streams the robot's responses back to the cloud using Server-Sent Events (SSE).

2. The Secure Tunnel: Tailscale Funnel or Cloudflare Tunnels

Since your robot is likely behind a home or local lab router (NAT), the cloud LLM cannot see its local IP address. You need a secure proxy tunnel to expose the SSE Gateway to the internet:
  • Tailscale Funnel: If you want a quick, highly secure way to expose a local port to the public internet with built-in TLS encryption.
  • Cloudflare Tunnels (cloudflared): An industry standard that routes traffic from a public domain name (e.g., ://yourdomain.com) directly to the local port of your SSE gateway without opening any ports on your router.

Step-by-Step Implementation Guide

Here is how to set up the proxy stack on your robot:

Step 1: Install the MCP SSE Wrapper

You can wrap the Amazing ROS 2 MCP server using the official Node-based SSE gateway wrapper. On your robot, run:
# Ensure you have Node.js installed on the robot
npm install -g @modelcontextprotocol/server-sse

Step 2: Launch the SSE Gateway with Amazing ROS 2 MCP

You configure the SSE gateway to launch the ROS 2 launch file or Python script for Amazing ROS 2 MCP as its underlying sub-process.
# Source your ROS2 workspace first
source /opt/ros/humble/setup.bash
source ~/your_robot_ws/install/setup.bash

# Run the SSE server on port 3000, passing the command to launch your MCP server
mcp-server-sse --port 3000 -- ros2 run amazing_ros2_mcp mcp_server_node
(Note: Replace ros2 run... with the exact execution command specified by your Amazing ROS 2 MCP workspace setup).

Step 3: Tunnel the Port to the Web

Expose port 3000 to the internet so your cloud voice/image application can hit it. If using Cloudflare Tunnels:
cloudflared tunnel --url http://localhost:3000
This will generate a temporary public URL (or a permanent one if configured in your Cloudflare dashboard) that your web application can point to.

How Your Multi-Modal Workflow Will Operate

Once this proxy is live, your web-based LLM application can leverage Amazing ROS 2 MCP’s native features to handle your specific goals:
  • Voice Commands to Nav2: The web app captures user voice, transcribes it to text via an API (like Whisper), and passes it to the LLM. The LLM calls the Nav2 tool via the proxy, which Amazing ROS 2 MCP translates into a nav2_msgs/action/NavigateToPose goal.
  • Image Identification: Amazing ROS 2 MCP has built-in image capture capabilities. The LLM can request a snapshot. The proxy streams the compressed image frame up to the cloud LLM (like GPT-4o or Claude 3.5 Sonnet) for visual grounding, object detection, or safety checks.
  • Manipulation: The LLM can send high-level coordinates or target object names. The server forwards these to your manipulation nodes (e.g., MoveIt 2 tasks) to calculate inverse kinematics and command the arm, keeping safety clamping active so the arm doesn't move at dangerous velocities.
To help fine-tune this setup, let me know:
  • What cloud platform or frontend interface are you planning to use to host your web LLM / voice application?
  • Do you already have a vision node publishing camera frames to a specific ROS 2 topic (like /camera/image_raw) that the LLM will need to look at?

That is all the information my feeble human brain can digest for now, thanks


James H Phelan
"Nihil est sine ratione cur potius sit quam non sit"
Leibniz

Sergei Grichine

unread,
Sep 6, 2026, 12:13:27 PMSep 6
to hbrob...@googlegroups.com
Good stuff, James!

I was actually thinking along the same lines, and was actively updating my notes, added chapters on how to choose the right MCP Server.

Here is the current state, including some safety considerations.

Best Regards,
-- Sergei


To view this discussion visit https://groups.google.com/d/msgid/hbrobotics/62852d56-b090-4d9f-a079-386b6c0ce03c%40hal-pc.org.

Chris Albertson

unread,
Sep 6, 2026, 5:33:09 PMSep 6
to hbrob...@googlegroups.com
I see AI/robot integration as a wide spectrum.     There are two ends and a wide range of middle ground between them.

Top-level: There is a robot with hard-coded behaviors.  Perhaps it can do actions like “Go to location X,Y.” The AI accepts voice commands and outputs top-level commands like “go to location(x,y)”.  The AI can only issue “top level” commands to the robot.  The AI can be very slow as new top-level commands only need to be sent on the order of every minute or so.

Bottom-Level: This is what they call an end-to-end model.  The AI directly accepts pixels from a video camera and sound from a microphone and outputs voltage levels to a set of motors. There is no hand-coded or “if this then that” logic in the robot.  Here, the AI is sending very low-level commands to directly control voltage to a motor.    The AI must be very fast because these voltage commands need to be sent 30 to 200 times per second.

We can find examples of both kinds and maybe even more examples for the middle ground.    Famously, Tesla claims their cars use end-to-end AI.  They claim the cameras directly control the steering and speed of the car.  Dynamically balanced, human-like walking is always done by an AI that directly controls motor torques in real time.    

In an interview, an engineer from Toyota described a parallel system to be used in upcoming cars.   There is an algorithmic anti-collision system that will take over and prevent anyone, humans or AI, from crashing into other cars or pedestrians, and then the AI system drives the car, but the algorithm system is still watching.   I don’t think this fits on that above spectrum.  It is kind of like giving the person in the passenger seat a brake pedal or kill switch.  The robot car has an independent safety driver.


One other trend I notice, mostly with hobby systems, is the overuse of LLMs.   Look instead at how Apple’s system works.   I have a battery-powered wristwatch with a tiny computer and microphone.  I can say “Siri, set an 11-minute timer for pasta” and then I hear “OK, set timer for 11 minutes”.   No data was sent from my watch; the tiny computer did this locally.    I also tested my watch by asking “Siri, I forgot the proof of the unique factorization theorem, can you outline it for me?”  The watch fails to make sense of the request and sends it to Apple’s cloud-based Siri; that system also fails and then asks me if it can forward the request to ChatGPT.   I answer yes, and it does.   The answer is very good.  (I lied when I said I had forgotten.)

What Apple’s system does is minimize response time and resource use.  Simple things are done directly on the wristwatch and are nearly instantaneous, but complex queries go back to a large data center and run on rack-mounted servers that generate kilowatts of heat.  Apple is not the only one to work like this.  All their competitors do the same.       At the hobby level, we save a lot of development work by going straight to an LLM.

Over time I expect that most robots that are designed for economic work (housekeeping, unskilled construction labor,…) will use all of the above techniques. I think they will be end-to-end AI running in real time but with an LLM in control of the top level and independent algorithm “safety drives” and it will have both tiny computers and AI-computers and a cloud interface too.   The winning answer is "all of the above”.     It will be fun to watch how this evolves over the next 10 to 20 years.

My interest right now is in the end-to-end area.   I’d like to see an AI robot that is as smart as a dog, if that is possible. 



James H Phelan

unread,
Sep 6, 2026, 9:26:26 PMSep 6
to hbrob...@googlegroups.com

Chris,

I like the practicality of layered systems. Local LLMs (or MedLM or SmLM) seem stupid because we try to cram ChatGPT into a shoebox and ask them to be too generalized. A Raspberry Pi with a speech recognition system trained on its owner and robot appropriate vocabulary, not Shakespeare, should be directable to 'Kitchen'. A local SmLM or better VisualLanguageAction model trained on robot appropriate material instead of the entire WWW and run on a slightly bigger CPU/GPU should be able to do household tasks as several of our colleagues are beginning to do. A challenge may to create the robot appropriate data sets to train our bots. But they're coming. 

The possibility of continuing education is interesting. Snapshots taken and labeled by YOLO, which are often wrong, could be uploaded to ChatGPT while the robot 'sleeps', correctly labeled, and fed back to update YOLOs model.


James H Phelan
"Nihil est sine ratione cur potius sit quam non sit"
Leibniz

Sergei Grichine

unread,
Sep 7, 2026, 2:47:06 PMSep 7
to hbrob...@googlegroups.com
Chris, James:

There was an interesting Q&A exchange at GTC 2024 in San Jose, in March 2024 - it struck me then, although it's hard to find a video now. It was about the eventual possibility of Full Neural Rendering -  replacing Rasterization and Ray Tracing, eliminating Game Engines.

https://music.youtube.com/watch?v=7mte6s5HzJM - Bilawal Sidhu's long explanation of Physical AI

At that time I was thinking that NNs could ideally generate optimized binary code directly, bypassing tool chains (language compilers, linkers, game engines and even the OS itself, just bare metal) - Nvidia's idea was even better:

Current game:

Player input
     ↓
Game engine
     ↓
Game logic / physics
     ↓
3D scene
     ↓
Renderer
     ↓
Pixels

versus the future Huang was describing:

Player input + game state
          ↓
       AI World model
          ↓
   generated frames
          ↓
       screen

Bilawal Sidhu asked, "How far do you think we are in this world where every pixel is generated at real-time frame rates? And what is your vision for gaming and non-gaming experiences in that new paradigm?"

Jensen Huang, NVIDIA's founder and CEO, answered:
"I think with almost almost everything in technology, the S curve is not longer than a decade once it becomes true, once it becomes practical and better. And, of course, ChatGPT is not only practical; in most cases, it's better. I think it's less than ten years away.

In ten year's time you're at the other end of that S curve. In five years from now, you're probably right in the middle where everything is changing in real-time, and everybody's going, 'Oh, look at that, this is happening.' And so you just got to decide, are we two years into it, into that ten years? Probably, we're probably already two years into it. And so I would say that within the next five to ten years, somewhere in between, it's largely the case."

Currently, in 2026, neural rendering has progressed dramatically - DLSS 5 can generate the final displayed appearance using a neural model - but it still sits on top of a conventional game-engine rendering pipeline. We are still a long way from the more radical architecture in which the game itself is effectively a real-time generative model.

Driving robot's actuators directly (well, via cmd_vel) is akin to Full Neural Rendering. I have no reason to assume that Tesla's production vehicle is simply "camera → neural network → motors.". I'd guess they are very, very layered under the hood. I am even more skeptical about us, hobby roboticists, ever laying our hands on hardware and services that could enable this architecture (fully and safely).

That's why my research is focused on finding something to play with in my lab with the toys I can afford. You might have slightly better toys, but not Tesla level power.

Hopefully, LLMs can do strategy, hardware like HuskyLens 2 can provide a bit of "visual intelligence", and ROS2 - a body safe enough to drive around the house or outdoors, based on a set of pre-defined behaviors with certain level of flexibility.

              LLM / AI
                 │ 
           strategy / goals 
                 │ 
                 ▼
        ┌─────────────────┐
        │ behavior layer  │
        │ "go to kitchen" │
        └────────┬────────┘
                 │
                 ▼
             Nav2 / ROS2
          planning + control
                 │
                 ▼
        local safety layer
        obstacle avoidance
        emergency stop
                 │
                 ▼
        motor controller
                 │
                 ▼
               wheels

--------------------------------------

        CAMERAS / LIDAR / IMU
                 │
                 ▼
        ┌────────────────────┐
        │     PERCEPTION     │
        │ objects            │
        │ distances          │
        │ people             │
        │ free space         │
        │ landmarks          │
        │ scene semantics    │
        └─────────┬──────────┘
                  │
                  ├──────────► LLM
                  │
                  ├──────────► Behavior
                  │
                  ├──────────► Nav2
                  │
                  └──────────► Safety/Guard

How about "AI is a passenger who can tell the robot where it would like to go, while the robot remains responsible for figuring out how to get there safely"?


So, what actual choice do we have - build better robots now or wait for better AI (and further re-confirm the validity of Sergei's Law - my sincere gratitude to those who try)?

Best Regards,
-- Sergei


To view this discussion visit https://groups.google.com/d/msgid/hbrobotics/a73f5b04-142e-45dd-8ab6-748bac422916%40hal-pc.org.

Chris Albertson

unread,
Sep 7, 2026, 9:26:52 PMSep 7
to hbrob...@googlegroups.com

No, it does not work that way.  We do not output cmd_vel.    We directly output the torque of each motor, then the speed of the motor is determined by the resistance on the shaft. The output of the AI is CAN bus commands in real time.

I placed a “hello world video” on YouTube.   It is not great, but I stopped to record the video just as soon as the software (and robot) stopped crashing, just to verify I could get the software stack to run.     In this video, the motor torques are updated 50 Hz.

Yes, we do use cmd_vel, but that is the input to the model.


I am pretty sure I can actually lay my hands on real hardware.    I would build a robo-dog, not a humanoid, because I don’t want to spend $10K.   A robo-dog that has very high performance, can jump over fences and do a backflip is about $1,500 for the motors and controllers.   Dogs have four legs, and each is 3-DOF, so 12 motors.  Motors cost $100 each with integrated CAN bus controllers

The way the full, 100% neural network controller works is that you build a very basic “multi-layer perceptron”. This is the simplistic kind of network with four fully interconected layers and is more than 1,000 times smaller than a basic LLM and runs on an Apple Mac mini, with room left over.

The input of the network is basically cmd-vel but with both X and Y populated and also cmd_pose to command body leans and rolls.    The output is a vector of 12 torque values, one for each motor.  The network is run 50 times per second.

So, a very powerful robo-dog needs only a computer as powerful as a MacMini and 12 motors at $100 each, plus structure and batteries.     I think where most people overestimate the computer requirements for AI is because they equate AI with ChatGPT.   The robot dog only needs 100 million parameters, that is a tenth of a gigabyte.

There is simulation software running but it is NOT in the robot’s control loop, it is only used to provide feedback dyurring the trining process.   Once the robot is trained the only thing running is ther nueral netwok,   Software used is Pytorch and the MuJoCo simulator.


My “hello world video” is linked below.   The robot lives in a simulated 2D world, so there is no left or right for him.  It is unreal, but the point is to get the software stack running and show it learns anything other than face-planting on the floor.  It was done on an Apple M2-Pro Mini and runs many times faster than real time entirely on the CPU, not yet using GPU or NPU.     The model size is tiny, maybe under one million parameters.
 

PS.  A real humanoid would need a much more complex nueral network but still under a billion parameters and runs at aiut 100Hz on resonable hardware.

Sergei Grichine

unread,
Sep 8, 2026, 8:16:29 PMSep 8
to hbrob...@googlegroups.com
Chris,
"No, it does not work that way. We do not output cmd_vel. We directly output the torque of each motor..."
- what you are describing is a closed architecture for a robodog (or any "base" in ROS terminology). While the RL model layer allows the robot to balance and move around - possibly commanded by cmd_vel from higher layers in the hierarchy - this layer alone won't bring you a beer.
To interact with humans, perceive surroundings, and navigate, you need a high-level framework that ultimately issues those cmd_vel commands (as you noted, "we do use cmd_vel, but that is the input to the model").
This thread isn't focused on the "bases" themselves, even complex ones like your planned robodog. Instead, it is about organizing the higher layers (LLM, VLM, Navigation, etc.) into a manageable architecture that won't give us downstream integration headaches.
Even in cutting-edge setups, we often see a hybrid approach: a VLM/LLM breaks down a task, a Navigation stack (like Nav2) handles path planning and collision avoidance to generate safe trajectories, and those trajectories are finally fed as cmd_vel target states into a high-frequency RL model controlling the joint torques. But for my use case I am trying to build the LLM->Legs bridge, not reinvent the legs.
Your future robodog project is highly interesting and deserves its own dedicated thread. I would be genuinely interested to see your repository, Wiki, videos, and notes whenever you are ready to share them with this group.
Meanwhile I will be working further on my notes: https://github.com/slgrobotics/articubot_one/wiki/ROS2-and-AI-integration
Best regards,
Sergei


--
You received this message because you are subscribed to the Google Groups "HomeBrew Robotics Club" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hbrobotics+...@googlegroups.com.

Sergei Grichine

unread,
Sep 9, 2026, 8:57:06 PMSep 9
to hbrob...@googlegroups.com
I've got Seggy moving under ChatGPT control: 


Work in progress.

Best Regards,
-- Sergei

James H Phelan

unread,
Sep 9, 2026, 10:37:05 PMSep 9
to hbrob...@googlegroups.com

Sergei,

Cool! Hope to see a demo video at the Challenge! (or sooner??)

James H Phelan
"Nihil est sine ratione cur potius sit quam non sit"
Leibniz

Sergei Grichine

unread,
Sep 10, 2026, 1:08:22 PM (14 days ago) Sep 10
to hbrob...@googlegroups.com
I received my  HUSKYLENS 2 Plus Kit, it takes a bit of effort to set up.

Here are my observations so far:


Best Regards,
-- Sergei


To view this discussion visit https://groups.google.com/d/msgid/hbrobotics/eec144de-9437-453f-8390-8f06f2d4828a%40hal-pc.org.

Sergei Grichine

unread,
Sep 14, 2026, 9:28:31 PM (9 days ago) Sep 14
to hbrob...@googlegroups.com
For fans and owners of HuskyLens 2, here is my ROS 2 package:


This is still work in progress, soon I hope to use Depth Anything V2 with the extracted image - which will provide estimated distances to detected object.

Best Regards,
-- Sergei

Reply all
Reply to author
Forward
0 new messages