Modelling nearest distance between two 3D meshes.

[Helen]

I'm having trouble solving a 3D geometry problem, and I'd like to understand if it's solvable in a performant way using Ceres.

To simplify to a similar - but easier to understand - problem:

Imagine I have a mesh "A" that I want to scale and rotate (with scale + rotation represented as variables!) to be as close to a mesh "B" as possible.

"Closeness" is defined as the sum of....

• For each vertex in A, find the distance to the nearest vertex in B
• For each vertex in B, find the distance to the nearest vertex in A

Now that first clause is super easy to model.

We can add a residual block for each vertex in "A", deform the vertex within the Cost Function using our variables, and work out their nearest neighbor in "B" - great.

But what about that second clause?

Do we have to do it in a single residual block? Deform every vertex in "A" in the residual block and compare to every vertex in "B"?

Do we create a residual block for every vertex in "B", and then have to re-deform every vertex in "A" within each residual block?

Is there something better that we can do to break down the problem??

Any help would be truly appreciated - I've spent hours thinking about it and reading the Ceres documentation - it feels like there must be something I'm missing...

[jkfl...@gmail.com]

Hi Helen

It sounds to me like you will need to recalculate the correspondences on every iteration, since the closest vertex in the other mesh will probably change after each step. What you're then describing is the Iterative Closest Points algorithm (ICP), and not a non-linear gradient descent with a global continuous cost function, which is what Ceres is more oriented towards.

The actual code to calculate the required rotation/translation given all of the correspondences is very simple, you first calculate and subtract the mean correspondence difference (which is the translation), then do an SVD on the moments of the correspondence differences to get the rotation matrix and scale.

There are a multitude of ICP libraries available, just off the top of my head KISS-ICP was recently open sourced: https://github.com/PRBonn/kiss-icp

Best

Julian

[Helen]

Thank you for your time, and apologies, perhaps I over-simplified the problem-space!

So for full context; I'm working to implement the following paper, which uses Dogleg minimization:
https://people.eecs.berkeley.edu/~kanazawa/papers/cvpr17_menagerie_camready.pdf

The problem has a number of other facets which makes it solvable using Gradient-Descent - for example, both Meshes "A" and "B" additionally have marked keypoints that should be aligned.

The part I'm struggling with is how to represent the term 'Es2m' using Ceres - which is to say that we should deform Mesh "A" such that we minimize the distance of vertices in Mesh "B" to the nearest vertices in deformed Mesh "A". If you have a moment, do you have any suggestions for tackling the term in Ceres (the other terms work great!)?