Projection Factor Marginals Crash (point along focus of expansion / collinear with motion) + code to reproduce

[Harry]

Hello,

I am using GenericProjectionFactorCal3_S2 to solve a bundle adjustment problem with GTSAM. I have noticed that this optimizes correctly, but will occasionally crash during the computation of marginals. Examining the points at fault revealed that they are often along the direction of travel.

It makes sense that this system would be indeterminant (triangulation is impossible), but it also makes sense that one would expect to include these points in a typical bundle adjustment problem (after all, they are still useful for resolving rotations). Is there a better factor to use, or a way to automatically remove the offending point or points from the marginals computation?

Please see the attached Python script to recreate this error.

Thanks!

[Harry]

I received your reply Frank but I think you may not have replied all so it never showed up this forum. Following up here just in case anyone else needs this info.

We were able to get things running thanks to Frank's suggestion and some additional logic checking. He suggested adding a sufficient number of priors to our landmark locations, and so we just added priors (with a large enough uncertainty band) to every landmark and gtsam is happy with it. We also went ahead and removed keypoints (before reaching the bundle adjustment step) that were close to the focus of expansion and at far enough depth because those are notoriously difficult to estimate.

[Yotam Stern]

Hi Harry, this is interesting, can you perhaps include a code example of the cleanups you did?

Did you try linearizing the graph and looking at the jacobians of certain landmarks to try and see which of them is singular?

[Harry]

Nope. I followed Frank's suggestion and basically wrote the quick and dirty solution of adding a large prior to every single landmark and GTSAM was happy (results weren't too bad either). I made some changes to the script and attached it here. There are several better ways to go about solving this. I tried for several weeks and ultimately did get smart projection factors to work, but not at their fullest potential. This was months ago now, but if I recall the problems there are that the smart factors operate sort of 'outside' of the graph object (unlike most of the other factors defined in GTSAM), which made it difficult to use on real-time applications where we did not have all of the information ahead of time. They are a tremendously useful set of factors but I don't yet have good criticism/feedback on how to make them Python- and CPP-friendly.

Anyways, see the attached script for the quick and dirty example. Hope it helps!

[Yotam Stern]

Adding large priors to all landmark seems to be a nice trick indeed and solved the problem on most cases, it also had minor impact on the covariance of the interest poses in the graph.

However I have had some extreme cases where I had to add a very tight prior on some landmark for the marginalization to succeed which is not subtle at all.

I wonder if there is some relatively simple way to try and detect those landmarks before trying marginalization which can be expensive on a large problem, I had partial success with aggregation the jacobians relating to the specific landmark from all of the projection factors relating to it, so I get a block of (2xn,3) for each landmark then using svd to see if it has a very small singular value which can indicate that this landmark is under determined.

This approach worked mostly well but still I had some special cases where all the singular values were non negligible but marginalization of the entire graph still complained on that specific landmark.

im trying to solve huge mapping problems by partitioning it to small chunks that are densely solved with visual Slam then exctrating the information from the small chunks to be merged into the large scale solution.

--
You received this message because you are subscribed to a topic in the Google Groups "gtsam users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gtsam-users/CciNzw_PC8I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gtsam-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gtsam-users/68f42d24-e7af-44ad-a5bd-379dee3ac618n%40googlegroups.com.

[Yotam Stern]

Hi all, after some more tinkering around I found a stable solution (at least for my typical graphs), looking at the smallest singular value was not important enough, the key was looking at the ratio between the smallest and the second smallest values. This ratio gives a very good prediction whether a specific landmark is going to crash the marginalization, anything below the threshold for valid landmarks seems to pass somewhere between the range of 4e-4 to 5e-4, all landmarks below it always crash, all those above it always succeed, all the ones in betweens depend on the general graph, so i set the threshold in my code to 5e-4 and after testing on a large data (over 200k images split into several data sets) i haven't had any case of marginals crashing after filtering landmarks this way. there is no need to add priors this way, ofcourse another alternative if you don't want to remove the ill-posed landmarks from the graph is to add priors on them, and thanks to the above rule it's possible to calculate the maximum possible covariance that will help keep the points from being degenerate while having the minimal effect possible on the graph marginalization.

i attached a modified version of the above script to demonstrate the calculation

[Dellaert, Frank]

Wow, that’s amazing! Yes, in effect, that is their condition number. I’m wondering whether somehow we can build in those type of protections and or warnings into GTSAM. 

Thanks for this investigation!  Your application seems to really push the boundaries in terms of scale. 

Best!

Frank

---

From: gtsam...@googlegroups.com <gtsam...@googlegroups.com> on behalf of Yotam Stern <sht...@gmail.com>
Sent: Monday, January 22, 2024 12:10:20 AM
To: gtsam users <gtsam...@googlegroups.com>
Subject: Re: [GTSAM] Re: Projection Factor Marginals Crash (point along focus of expansion / collinear with motion) + code to reproduce

 

You received this message because you are subscribed to the Google Groups "gtsam users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtsam-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gtsam-users/e673bd62-a565-4a78-a480-9ef42418381en%40googlegroups.com.