Refining known 3D poses with bundle adjustment

[Kevin Cain]

Consider the case of reasonably accurate known extrinsics and intrinsics for a group of images, but no 2D features, matches, or triangulated 3D points. Is there a clever way to refine the poses (e.g., via bundle adjustment) with these limitations, as these camera poses aren't accurate enough for pixel reprojection / depth estimation?

I can successfully take this group of images through an uncalibrated SfM pipeline like Colmap, BoofCV, Agisoft, and get reasonable convergence. Because we're running on a mobile device, though, the full sparse SfM pipeline is a bit heavy.

If we had RGB-D rather than RGB, we could use the depth to refine pose in different ways. We could use SLAM methods, too, but even very good methods aren't yet able to compete with full BA.

Might there be a sparser way to run sparse reconstruction, to have minimal inputs for BA?