Dealing with 3D landmarks moving behind camera during bundle adjustment

[asr]

I've been trying to perform pose optimization with a stereo camera by doing local bundle adjustment (BA) over the last N camera poses and their associated triangulated landmarks.

However, I've noticed that a lot of the time I'll see a few landmarks move behind their associated camera during the optimization process . My current naïve approach to deal with these landmarks is to remove them from any keyframes they belong to once the current local BA process has completed. This should prevent them from affecting the next local BA process that occurs when the optimization window shifts one frame forward. However, this isn't a great solution since these landmarks would have already affected the optimization results for the current optimization window.

As such, I wanted to ask if there's a better way to deal with landmarks moving behind the camera during the optimization process. Is this a common problem, and if so are there ways that GTSAM can help deal with these landmarks?

Ideally, I'd like to have GTSAM check if any landmark moves behind a camera, while its performing its Levenberg-Marquardt iterations, and automatically remove these landmarks from the factor graph (and have some way of telling the user which points were removed) while continuing to optimize. Is something like this possible? Thanks very much.

[Yotam Stern]

Hey Alex, projection factors in gtsam can raise chirality exception if a point moves behind the camera its projected to, by default when this happens the factor silently ignores the error and returns zero error and jacobian for the projection of that point. so in practice it does not effect the result of the BA it is part of, the optimizer is ignoring the projection that that point to the camera it fell behind.

Alternatively you can turn on the error raising and instead of silently ignoring the chirality exception it'll raise it and the optimization will fail.