Assume I have a SFM problem where I fixed X0 with a strong prior and some strong priors for all the landmarks anchored on X0, then I am start tracking the landmark 2d features by optical flow. This could be a monocular VO pipeline where the system bootstaped by some known 3d points.
So my factor graph would look like this:
prior--> X0 - X1 - X2 - X3 - X4 - X5
prior-> P0, [many landmarks]
If I just use a fixlagsmoother (batch or isam2), do I need to keep track of the oldest pose and the points originated from it, so I will need to keep adding prior to them to keep locking the scale?
A batch fix lag smoother seems will drop the prior factors when removing a variable since they are not connected to other nodes. Any comments are welcome.