It depends on the structure of the problem.
"While iSAM2 is capable of constant-time updates when the new factors only affect the upper cliques of the Bayes tree, the inclusion of landmark states breaks this assumption, as the same landmark may be seen over many frames." (1)
If you're keeping your landmarks on the graph, it's certainly not going to get *faster* when you add more data. Depending on what you are trying to accomplish, you can maybe avoid explicitly handling landmarks on the graph by using "smart factors" (2). Generally speaking, it makes sense that larger problems usually take longer to solve; smart factors are one way to slow down the rate of growth in the solving time.
"GTSAM can easily perform recursive estimation, where only a subset of the poses are kept in the factor graph, while the remaining poses are marginalized out. In all examples above we explicitly optimize for all variables using all available measurements, which is called Smoothing because the trajectory is “smoothed” out, and this is where GTSAM got its name (GT Smoothing and Mapping). When instead only the last few poses are kept in the graph, one speaks of Fixed-lag Smoothing. Finally, when only the single most recent poses is kept, one speaks of Filtering ..." (3)
In general, vision problems are difficult to solve with filtering because marginalization is immediate and the linearization point cannot be revisited. Smoothing really is a great approach for vision stuff because because the problems aren't linear, and smoothing *doesn't* marginalize, so the errors associated with bad linearlization points aren't compounded.
You can try fixed-lag smoothing (there are several examples for different ways to do fixed-lag smoothing in the GTSAM code). Depending on who you ask, it's either a good compromise between speed and correctness or a horrible bargain that combines the worst problems from smoothing and filtering.
Good luck!
Mike Sheffler
----------
(1) "Robust vision-aided navigation using Sliding-Window Factor graphs", Chiu, Han-Pang; Williams, Stephen; Dellaert, Frank; Samarasekera, Supun; Kumar, Rakesh; ICRA Proceedings 2013
(2) "Eliminating Conditionally Independent Sets in Factor Graphs: A Unifying Perspective Based on Smart Factors",
Carlone, Luca; Kira, Zsolt; Beall, Chris; Indelman, Vadim; Dellaert, Frank; ICRA Proceedings 2014
(3) "Factor Graphs and GTSAM : A Hands-on Introduction", Dellaert, Frank; (just search for it)