Visual SLAM with RTAB‐Map (ROS2)

[Sergei Grichine]

Hi All,

I experimented a bit with RTAB-Map, and here are my findings so far. This is still a work in progress.

https://github.com/slgrobotics/articubot_one/wiki/Visual-SLAM-with-RTAB%E2%80%90Map

I'd love to hear about your experience using Visual Odometry, Visual SLAM, or similar technologies in ROS 2—whether indoors or outdoors.

Best Regards,

-- Sergei

[Sergei Grichine]

If you are interested in playing with RTAB-Map in simulation check out this section:

https://github.com/slgrobotics/articubot_one/wiki/Visual-SLAM-with-RTAB%E2%80%90Map#running-gazebo-simulation-dev-branch

Seggy has simulated OAK-D lite camera now (in RGBD mode) and works with RTAB-map in sim.

Best Regards,

-- Sergei

On Sun, Jul 19, 2026 at 7:15 PM James H Phelan <jhph...@hal-pc.org> wrote:

intriguing!  will follow with interest!

Wonder if this will help avoid 'things lidar and pointcloud can't see' like shoes on the floor, coffee table crossbars?

James H Phelan
"Nihil est sine ratione cur potius sit quam non sit"
Leibniz

--
You received this message because you are subscribed to the Google Groups "HomeBrew Robotics Club" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hbrobotics+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hbrobotics/CA%2BKVXVMxTm2ntM5NRdwMDjuMcCO64j%2Bh%3DJMvZLr9vVtLnOB-RA%40mail.gmail.com.

[Chris Albertson]

I think the advantage of LIDAR over video was always that the LIDAR was easier to process not better, but easier because it is already in the form of a point cloud and it takes considerable processing to turn video into points.     But now we have depth cameras and bigger computers.      Video starts to win because it is already a 2D field of view, whereas LIDAR has to be mechanically scanned.   (I hate to say that Elon Musk was right about this.)

I’d like video to win only because I can buy decent cameras for cheap even if it takes a bigger computer.

One thing I’m wanting to explore, and will once I have a better simulation platform, is using AI to navigate.   The trouble today is the vast amount of computing that is needed for AI while LIDAR is at least 100,000 times less computation.        But then I see open-source V-JEPA 2.1 is a free download.     People are using this and then with just under 200 hours of fine-tuning policy on a real robot.

I think this is going to be the best solution.  The robot just looks around to know where it is because it sees familiar objects.

The robot could be dropped into a new house and look around and be familiar instantly.    But for now, SLAM is easier because it is mature.

I am somewhat encouraged because even the “gigantic” version of V-JEPA 2.1 has “only” 2 billion parameters.  This is smaller by a factor of 200 than a full-size LLM.    2B can run on almost any small computer.    And I doubt we need even this.   The base model is only 0.1B parameters.

[Dave Everett]

On Fri, 31 Jul 2026 at 06:33, Chris Albertson <alberts...@gmail.com> wrote:

I think the advantage of LIDAR over video was always that the LIDAR was easier to process not better, but easier because it is already in the form of a point cloud and it takes considerable processing to turn video into points.     But now we have depth cameras and bigger computers.      Video starts to win because it is already a 2D field of view, whereas LIDAR has to be mechanically scanned.   (I hate to say that Elon Musk was right about this.)

I think it's more likely he has been convinced by people who know about this stuff rather than Musk working it out himself. So now you don't have to feel bad as you are complimenting those who actually know :)

It's always been the goal to use a passive sensor like a camera, this has been a known issue for many decades. The problem is the same one as radio signals, if you have multiple signals on the same frequency, they will interfere. The same issue applies to lidar, the lidar is relying on either determining the time from send to receiver the phase of a coded signal. If another source f light at the same rough frequency is shone towards the lidar, how can it tell the difference? If the light is encoded, you just mess up the coding and it will not receive correctly. Some years ago I was talking to a researcher at a local university, he was part of a self-driving car project using lidar. I mentioned this problem, He said it was not an issue, nut when I pressed him on it, he could not explain how.

I’d like video to win only because I can buy decent cameras for cheap even if it takes a bigger computer.

Video has to win, I' banking on the same. 

One thing I’m wanting to explore, and will once I have a better simulation platform, is using AI to navigate.   The trouble today is the vast amount of computing that is needed for AI while LIDAR is at least 100,000 times less computation.        But then I see open-source V-JEPA 2.1 is a free download.     People are using this and then with just under 200 hours of fine-tuning policy on a real robot.

This is an area I need to research more. I have avoided it because it's a black box component. You will never know how it really works or how it's going to fail, and you can't just change a few numbers to fix issues, you usually have to retrain all or much of the network. 

I think this is going to be the best solution.  The robot just looks around to know where it is because it sees familiar objects.

There wasd a paper some time ago that used semantic recognition not only to quickly map an unknown environment, but record the location of items it could recognise. The robot could then be commanded to  find an object and it would know where it was and also how to get there. From memory they were using a Boston Dynamics dog and going to AirBNBs so they had random locations. I'll try and find the paper this weekend.

Dave

[Chris Albertson]

On Jul 30, 2026, at 2:46 PM, Dave Everett <daveev...@gmail.com> wrote:

One thing I’m wanting to explore, and will once I have a better simulation platform, is using AI to navigate.   The trouble today is the vast amount of computing that is needed for AI while LIDAR is at least 100,000 times less computation.        But then I see open-source V-JEPA 2.1 is a free download.     People are using this and then with just under 200 hours of fine-tuning policy on a real robot.

This is an area I need to research more. I have avoided it because it's a black box component. You will never know how it really works or how it's going to fail, and you can't just change a few numbers to fix issues, you usually have to retrain all or much of the network. 

This video will convince you quickly.  Are you really going to hand-code a control system that does these two-wheel balancing acts and flips?   While you are busy coding some basic movments, the competition is doing this:

Training takes time but at least this a not an LLM with 200 billion parameters.  These have under 1 billion and could in theory run on a cell phone.   

So as to prevent a misunderstanding,    I see robot control as having three layer, even if not exactly designed that way we have three.  

1) Balance and locomtion, This allows normal walking and this acrobatics and high speed rolling.   The skill is just moving and not tipping over

2) Some specifi skill. like a flip or picking up a coffee mg or opening a door

3) Making high level plans like clearing dishes from a table and wsing them by stringing together a series ov level 2 skills.

The small few-million parameter models ar for #1, wile a large LLM might be needed for #3.   v-jepa is 100 times smaller then an LLM and might work for #2 or #3 but it is not yet so common and only a few years new.

The small

[Dave Everett]

On Fri, 31 Jul 2026 at 15:55, Chris Albertson <alberts...@gmail.com> wrote:

On Jul 30, 2026, at 2:46 PM, Dave Everett <daveev...@gmail.com> wrote:

This is an area I need to research more. I have avoided it because it's a black box component. You will never know how it really works or how it's going to fail, and you can't just change a few numbers to fix issues, you usually have to retrain all or much of the network. 

This video will convince you quickly.  Are you really going to hand-code a control system that does these two-wheel balancing acts and flips?   While you are busy coding some basic movments, the competition is doing this:

Competition? I'm not in a business race. Good for them. I will state categorically, I have never needed a robot to do flips, in fact I would consider that a flaw if my robot started doing flips. It would spill my  caviar.

Training takes time but at least this a not an LLM with 200 billion parameters.  These have under 1 billion and could in theory run on a cell phone.   

So as to prevent a misunderstanding,    I see robot control as having three layer, even if not exactly designed that way we have three.  

1) Balance and locomtion, This allows normal walking and this acrobatics and high speed rolling.   The skill is just moving and not tipping over

2) Some specifi skill. like a flip or picking up a coffee mg or opening a door

3) Making high level plans like clearing dishes from a table and wsing them by stringing together a series ov level 2 skills.

Other than the flips and acrobotics in general, this is a fair list. 

The small few-million parameter models ar for #1, wile a large LLM might be needed for #3.   v-jepa is 100 times smaller then an LLM and might work for #2 or #3 but it is not yet so common and only a few years new.

As mentioned, the only issue I have with these black boxes is the inability to demonstrate the control method. This has come up in work I have done in the past and I continue to see it in other ventures. For example, I have watched the series of videos from Vulcan Robotics, best viewed through the youtube channel "Nick Builds". They are very unclear about the processes, they are having a go and I applaud that, but for a business model hey are skipping a lot of important steps. They did not understand that if you change the gearing on a servo, the trained model, it will not work the same, and they have trained a model to fold laundry items, this is fine, but I do not see any work on the robot getting to the laundry area. More to the point, every test shows a piece of laundry already on the table, no doubt a human has placed it there. Nothing wrong with dealing with the process in stages, but they are claiming they are ready to go to market with this.

I can say from my own experience that a simple task usually requires many more harder tasks to be "perfected" first in order to get that simple task working. Some of that has been black box, like recognition of soft drink cans.

The detection of these cans was identified as a difficult problem at the start. All drink cans are the same size and shape, only the colours and logos differ. We have tried tensorflow, yolo etc and the detection confidence is very poor. I came up with a script that moves the neck and torso through every position in increments of 10 degrees and takes a picture with the very camera that will be used for detection later on. We ended up with about 1100 images, processed for about 20hrs on a dedicated GPU machine. With 100  test and validation images, we could not get reliable detection, if any detection it would always be below 40%. These systems can detect elephants, trucks and bicycles very well, but we have no need to manipulate any of those objects. I think back to the system I built about 10 years ago now, a garbage camera and hand designed filters that did a 90% reliable job. I will probably go back to that. I know that is not sexy, but I just want some reliable detection.

Dave

[Chris Albertson]

Generally, specialized tasks are easier.   Finding a soda can is easy, but what if the task is “picking up stuff my owner would not want on the floor”?  I know your task was soda cans.   But in general, a home-robot would need to handle the general case.       Even a self-driving car needs to handle the case where “don’t drive over stuff that can damage the car.”  Rocks are bad, but paper and leaves are OK, animals are bad, and so are the larger potholes.      But now there is an infinite variety of all of these and grey areas.  In general, you can’t hard-code a solution because with a million cars, your software will literally see the entire “one in a million case” every day.

Or not just folding laundry but deciding to postpone that task because there are more urgent things to do first, and then I will get back to laundry while the bread is in the oven.  

Also, there could be a dozen reasons you can determine the model did not work.  Perhaps it was overfitting and failing to generalize?  It seems like maybe that was if you were only using good images and no images of “not-a-can,” then I think that’s why.   Fix that with some serious data augmentation and regularization, basically adding a ton of noise to the data.   This is easy to detect as there will be a loss divergence between better training and validation datasets.   Did you check for this?  What’s happening is that the model is memorizing pixels rather than learning a general description of a soda can.   It is very important to watch for this.

As for not knowing this, the algorithm is simple.  It’s an element-wise dot product followed by some activation function like RELU, then do that again, for each layer.  The result is the output layer.  It is a mistake to see the model as a black box.  It is not.  It is a set of high-dimensional vectors, and training is an optimization process for finding the best values for those vectors.  You have a dozen or so knobs to turn during that process.

--
You received this message because you are subscribed to the Google Groups "HomeBrew Robotics Club" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hbrobotics+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/hbrobotics/CAG61pbf89aCO1tswT2Cc3fTURCsm8CTacu%2Bo41%3D4fZ-J_RMXeg%40mail.gmail.com.

[Alan Downing]

FYI, here's Nvidia's augmentation to VSLAM that I find very interesting:

Navigating and understanding complex environments over extended periods of time is a significant challenge for robots. People interacting with the robot may want to ask questions like where something happened, when it occurred, or how long ago it took place, which would require the robot to reason over a long history of their deployment. To address this problem, we introduce a Retrieval-augmented Memory for Embodied Robots, or ReMEmbR, a system designed for long-horizon video question answering for robot navigation. To evaluate ReMEmbR, we introduce the NaVQA dataset where we annotate spatial, temporal, and descriptive questions to long-horizon robot navigation videos. ReMEmbR employs a structured approach involving a memory building and a querying phase, leveraging temporal information, spatial information, and images to efficiently handle continuously growing robot histories. Our experiments demonstrate that ReMEmbR outperforms LLM and VLM baselines, allowing ReMEmbR to achieve effective long-horizon reasoning with low latency. Additionally, we deploy ReMEmbR on a robot and show that our approach can handle diverse queries.

https://nvidia-ai-iot.github.io/remembr/

To view this discussion visit https://groups.google.com/d/msgid/hbrobotics/FAB70EB4-A094-403D-9AFA-89D18BD83039%40gmail.com.

[Sergei Grichine]

Alan - thanks for that link to "ReMEmbR: Building and Reasoning..." paper.

I originally thought that RTAB-Map uses similar approach, but it turns out that's not the case:

https://share.google/aimode/RIcyzumm59YR9BrBH

I'd guess that the ReMEmbR would require a real machine on board (i.e. NN-capable), while RTAB-Map seems to run comfortably on my RPi5 next to the rest of the robot's full Nav stack.

Best Regards,

-- Sergei

To view this discussion visit https://groups.google.com/d/msgid/hbrobotics/CAAvYDnGjfPfDdjRxNe3F-ze_hp13dDvxE15TY8G8oFxKcay1UQ%40mail.gmail.com.

[Dave Everett]

On Fri, Jul 31, 2026 at 10:51 AM Chris Albertson <alberts...@gmail.com> wrote:

Generally, specialized tasks are easier.   Finding a soda can is easy, but what if the task is “picking up stuff my owner would not want on the floor”? 

Yes it does need to be generalisable. I know you are characterising my process as hards-coding, but it is not that hard coded. The camera must move, the robot must move the arm must move, these actions occur in response to external and internal sensors. I am not hard-coding, if the can is 20deg to the left then turn 20deg. I am moving the camera and torso in small increments, this is not even servoing, and in the end ALL of that stuff works GREAT. The camer centering, the torso centering, the can distance calculation, the arm servoing, the gripping, all great. What doesn't work is the black box part, and it is a black box. At any moment, you cannot determine what the box is doing, the "decisions" it is making, you get to see the input and the output only. No one can look at a billion floating point numbers and understand what is happening right then and there.

Also, there could be a dozen reasons you can determine the model did not work. 

I'm not convinced anyone can determine why it fails so bad. All the cans were within the range the camera could reasonably resolve, they were in all orientations the camera might see it, from up high, low and everywhere in between, left and right of the camera and in between, and many images where the can was not in frame at all.

As far as training goes, you keep training until the error rises again. I forget the specific numbers but you are looking for (1). we got down to well below 3 before the error started climbing again, so we went back to an earlier checkpoint. We tried training with various parameters and it still identified a microscope as a can, and saw multiple cans when there weren't any.

Again, who can confidentially say how the process is working? Is colour a factor? it should be. Is relative size a factor? it should be. But it almost certainly is not in these networks.

My early hypothesis was that cans vary so little that image numbers do not help. Elephant do vary, dogs vary, bicycles vary enough that a network appears to converge, but drink cans don't vary enough.

Our plan was to load different networks in order to locate different objects, I wanted to start with the can as it is something that could demonstrate a useful task.

Oddly, we had better performance with a network trained on 80 images. It was terrible, but still better.

Dave