Impact of 2D Keypoint Noise on SFM Optimization in GTSAM

[Avinash Ayite]

Problem Overview

I'm currently analyzing how 2D keypoint detection noise influences the quality of Structure from Motion (SfM) reconstruction using GTSAM’s optimization framework. Specifically, I'm varying the standard deviation of keypoint noise from 0.001 to 3.0 pixels, and evaluating its effect on:

• Overall graph optimization error

• Camera pose translation accuracy

• Camera pose rotation accuracy

• 3D landmark position accuracy

Initial Findings
When using ground-truth keypoints, the optimization performs extremely well—yielding accurate reconstructions for both camera poses and landmark locations. However, as I introduce increasing levels of keypoint noise, I observe a roughly linear degradation in both pose and landmark accuracy relative to the noise level.

Question
I'm seeking insights on how GTSAM’s optimization handles noisy keypoint initializations. Specifically: Does the optimization framework explicitly model or account for the uncertainty introduced by noisy keypoint observations?

I’ve attached a modified SfM example along with the results obtained under various keypoint noise conditions for your reference.