Adaptive noise models for visual factors in visual SLAM backend

[Barnwal Tejal | Robotics Engineer]

Hi GTSAM community,

I’m building a visual SLAM backend in GTSAM with:

• Wheel Odometry factors between consecutive frames (Similarity3), using tight sigmas

  • rotation σ ≈ 0.01

  • translation σ ≈ 0.01

  • scale σ ≈ 0.01

• Visual factors between the current frame and a (possibly distant) keyframe, using looser sigmas

  • rotation σ ≈ 0.2

  • translation σ ≈ 0.3

  • scale σ ≈ 0.1

Problem

Even when visual tracking quality is poor, the visual factors sometimes pull the trajectory more than the high-confidence odometry (because we use the same noise sigma for visual factors regardless of tracking quality or reprojection error). In some datasets, the optimiser appears to place too much weight on visual constraints, depending on how the noise is set.

My expectation

• Odometry should dominate locally (short-term motion).

• Visual constraints should strongly influence the solution, mainly for true loop closures.

• Poor-quality visual matches should not override strong odometry.

Main Questions

What are the recommended patterns for adapting visual factor noise based on measurement quality?

For example, scaling sigmas based on:

• number of inliers,

• average reprojection error,

• baseline/parallax.

In practice, do people:

• Manually scale diagonal sigmas (e.g., proportional to 1/√inliers or reprojection error)?

• Use any more principled approach?

If anyone has implemented adaptive visual factor weighting in an open-source GTSAM project, I’d really appreciate pointers or high-level guidance.

Thanks a lot!

[Dellaert, Frank]

Hi !

All of these strategies are great ideas. This is actually one of the hardest problems in a VIO pipeline. At very least, you should have aggressive outlier filtering and possibly even robust error models - although they will only work really well if you have good initial estimates, which can be the case in a visual slam situation. Still, any outlier that makes it through something like Ransac will have the potential to completely break the pipeline - and hence, robust (Huber, Geman -McClure - both built into GTSAM) is a good backstop for that. 

I think this is a great topic for a discussion and I know that there's many people on this list that have tackled issues like this. So it would be great to get some community wisdom :-)

Best
Frank

--
You received this message because you are subscribed to the Google Groups "gtsam users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtsam-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/gtsam-users/5c643f19-3b76-4fd3-85d2-ab03da11ac07n%40googlegroups.com.

[Brett Downing]

This is a tricky one to get right.

I was using gtsam in an additive manufacturing context, and we used visual features to perform surface reconstruction and CAD parameter fitting. The whole pipeline becomes quite sensitive to correctness of uncertainties

We ended up using very simple factors so that things like parallax were accouned for by the error equations, and all of our uncertainties could be expressed in the same units as the original measurements.

We had some success with adaptive corner detection by estimating the uncertainty of the feature location based how blurry the local region is. The brute force equation looks like fitting a Gaussian to the DFT, and inverting the estimated spectral sigma to recover the spatial one.

Robust factors were extremely important for our pipelines, but had a nasty habit of producing a significantly non-convex whole-graph error functions.  Initialisation got scary and we started using global Lipschitz solver methods to find the correct convergence basin. Heavy on compute but saved a lot of messing around

Cheers, 

Brett 

To view this discussion visit https://groups.google.com/d/msgid/gtsam-users/PH0PR07MB8653480C7ED50445F569631ECF6CA%40PH0PR07MB8653.namprd07.prod.outlook.com.

[Dellaert, Frank]

Ah, yes - we do now have MIT’s GNC (Graduated at non-convexity) optimizer - maybe similar to what you're alluding to - Incidentally, we're also using that in a different project and it's a lifesaver. 

Frank

From: gtsam...@googlegroups.com <gtsam...@googlegroups.com> on behalf of Brett Downing <brett.d...@gmail.com>
Date: Monday, February 16, 2026 at 5:16 PM
To: gtsam users <gtsam...@googlegroups.com>

Subject: Re: [GTSAM] Adaptive noise models for visual factors in visual SLAM backend

You don't often get email from brett.d...@gmail.com. Learn why this is important

To view this discussion visit https://groups.google.com/d/msgid/gtsam-users/CAGzEurw1%2B9_URqSD%2BXhFpsw4Yu9Uar4P258QW-pCAQcKDeT2gQ%40mail.gmail.com.