Find correspondences of points of intereset with known pixel coordinates across multiple camera views

[Ekansh Sharma]

I am currently working on a project where I have nine points of interest identified through Hough lines intersection in each camera view. I have a total of three different views: left, mid, and right. While I have the pixel coordinates of these points, the correspondences between them across the three views are unknown. The cameras are calibrated, so the intrinsic and extrinsic parameters are known. My goal is to find the correspondences of these points across the three views to understand where a point in the left camera view appears in the right camera view or the mid camera view.

The idea is to determine the optimal correspondences among all three camera views. Here's how it works: I compute all possible 3-line intersections along with their corresponding distances, resulting in a total of 9^3 intersections. Then, I frame the problem as a 3-dimensional Assignment Problem. The goal is to minimize the cost of intersection distances, thereby finding the optimal assignment.

Is there an existing example which I can follow for this in gtsam repo? or do you have any suggestion on how can I use FG for this?

[Ekansh Sharma]

 Here's a plot illustrating the rays from the three camera views: