Obtaining shape and expression coefficients from BU-4DFE

[nataniel...@gmail.com]

Hello all,

I am looking to do the following:

- Load the Basel 2017 model

- Fit the model to the depth data contained in the BU-4DFE dataset (which is identical in format to the BU-3DFE dataset used in basel-face-pipeline)

I was wondering if anyone could give me any guidance on which functions I should be modifying since I'm not very familiar with Scala.

Some initial thoughts I had were to replace the statistical model used in Registration.scala with the BFM2017 model, although the objects are of different type. I have also been looking at the registration function in Registration.scala which might give the shape and expression coefficients that I am looking for?

Thank you for the help

Nataniel

[Andreas Forster]

Hi Nataniel,

The entry point of the registration is in the file src/main/scala/registration/Registration.scala. The main method is at the bottom of the file in the object Registration. There you can see the two things you need to change:

1. The BU3DDataProvider handling all loading, saving as well as lists of IDs or Experssions.

2. The method Registration.run() in the same file.

The run method is on line 194:

- The method iterates first over the different expressions.  Then the expression specific model is loaded. Here you need to change that the BFM is loaded. However, as you mentioned they are of different types: MoMoExpress and StatisticalMeshModel (discussion further down).

- Next the method iterates over the identities. It loads the data for identity and expression. There you would need to make sure that the data provider loads the right data.

- Then the target is aligned with the model and the method Registration.registration() is called. In there I do not expect that you need to change something.

Model: MoMo vs. StatisticalMeshModel

The main difference between the models is that MoMo has a separate shape and expression basis. The SMM has only one basis. The easiest solution would be to stack the two bases to one. For most applications, such a basis should work even that it is no longer orthogonal.

If you want to learn more, there is currently the future learn course running dedicated to shape modeling (see http://shapemodelling.cs.unibas.ch/). The course is also based on scalismo. If you successfully complete the shape modeling course, you can visit us in Basel for the following summer school where we usually have a few projects proposed but you could also work on your own project, e.g. this registration.

I hope this information helps you to proceed and I am curious to hear about your experience and maybe future problems.

Best, Andreas

--
You received this message because you are subscribed to the Google Groups "scalismo-faces" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalismo-face...@googlegroups.com.
To post to this group, send email to scalism...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scalismo-faces/f7ca5b25-d22d-464c-934f-7d5f979279c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

*****************************************

Dr. Andreas Morel-Forster
Departement Mathematik und Informatik
Spiegelgasse 1
CH-4051 Basel
PHONE: +41 61 207 05 52

MAIL: Andreas.Forster@unibas.ch
*****************************************

[nataniel...@gmail.com]

Hi Andreas,

Thank you so much for your reply. I have parsed the code and I believe I can make the changes such that the code uses the Basel face model instead of the StatisticalMeshModel and such that we learn the (shape + expression) basis. The only conceptual problem that I have left is that in this specific application we want the identity (shape and color coefficients) to remain constant for an individual. To recap one important thing, in the BU-4DFE dataset there are short videos of a person performing an expression. The idea would be to obtain the shape coefficients for that person and then to fit the expression coefficients for all of the frames.

Does this make any sense? Is there any way of doing this?

Some ideas could be enforcing some of the coefficients to stay constant and to only infer the shape (identity) coefficients in the neutral frame which occurs at the beginning of the videos.

Again thank you very much.

MAIL: Andreas...@unibas.ch
*****************************************

[Andreas Forster]

Hi Nataniel,

Good to here that you made progress. I think what you sketch in the end of your mail is a good idea:

1. Infer the identity from the neutral state there you can use only the neutral model.

2. Then you could build a person specific model with the mean of the statistical model as the infered identity. As basis you would then use only the expression part.

3. Use the personalized model to register all the following frames with expression.

Let me know if you need more input about how to build these different models.

Best, Andreas

To view this discussion on the web visit https://groups.google.com/d/msgid/scalismo-faces/df505b8c-e155-4a87-b491-1059d15118b9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

*****************************************

Dr. Andreas Morel-Forster
Departement Mathematik und Informatik
Spiegelgasse 1
CH-4051 Basel
PHONE: +41 61 207 05 52

MAIL: Andreas.Forster@unibas.ch
*****************************************

[nataniel...@gmail.com]

Hi Andreas,

Again thanks for the help.

I have successfully navigated some of the first steps, and had to actually make a bunch of changes to registration and registrationForLevel (since we are not using a DenseVector but are using MoMoCoefficients).

The only thing I haven't figured out is, in line 172, we have the line:

val config = RegistrationConfiguration[_3D, GaussianProcessTransformationSpace[_3D]](

optimizer = LBFGSOptimizer(numIterations = numberOfIterations),

metric = HuberDistanceMetric[_3D](optimizationPointSampler),

transformationSpace = GaussianProcessTransformationSpace(reducedGPModel.gp.interpolateNearestNeighbor),

regularizer = L2Regularizer,

regularizationWeight = regularizationWeight)

Except, our model (let's say reducedGPModel) does not have the gp attribute since it's a MoMo and not a StatisticalMeshModel.

How do I find a suitable transformation space using the Basel MoMo?

Thank you!

On Wednesday, March 27, 2019 at 4:28:00 AM UTC-4, Andreas Forster wrote:

Hi Nataniel,

Good to here that you made progress. I think what you sketch in the end of your mail is a good idea:

1. Infer the identity from the neutral state there you can use only the neutral model.

2. Then you could build a person specific model with the mean of the statistical model as the infered identity. As basis you would then use only the expression part.

3. Use the personalized model to register all the following frames with expression.

Let me know if you need more input about how to build these different models.

Best, Andreas

To unsubscribe from this group and stop receiving emails from it, send an email to scalism...@googlegroups.com.

To post to this group, send email to scalism...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scalismo-faces/df505b8c-e155-4a87-b491-1059d15118b9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

*****************************************

Dr. Andreas Morel-Forster
Departement Mathematik und Informatik
Spiegelgasse 1
CH-4051 Basel
PHONE: +41 61 207 05 52

MAIL: Andreas...@unibas.ch
*****************************************

[nataniel...@gmail.com]

Hi Andreas, just a quick bump on this thread because I haven't been able to figure out a way around this roadblock. Thanks :)

[Bernhard Egger]

Hi Nataniel,

I'm not sure if I fully got your problem. But the MoMo has gp members too. Actually it has two of them. One for the color and one for the shape.

so momo.shape is basically the gp shape part of the MoMo.

Perhaps you can try if that is what is missing?

Best

Bernhard

[nataniel...@gmail.com]

Basically my question can boil down to, how do I write this line with a MoMo instead of a StatisticalMeshModel (which reducedGPModel is a member of):

transformationSpace = GaussianProcessTransformationSpace(reducedGPModel.gp.interpolateNearestNeighbor),

On Thursday, April 11, 2019 at 5:45:31 PM UTC-4, natanie...@gmail.com wrote:

Thank you for the answer. I will investigate this tomorrow morning and report ASAP.

[nataniel...@gmail.com]

Also thank you for the answer Bernhard.

[nataniel...@gmail.com]

Hi Bernhard,

It didn't work. As far as I can see the MoMo does not have a member shape so momo.shape fails.

I am looking to write this line:

transformationSpace = GaussianProcessTransformationSpace(reducedGPModel.gp.interpolateNearestNeighbor),

Using a MoMo instead of a StatisticalMeshModel. reducedGPModel.gp is of class DiscreteLowRankGaussianProcess. Does a MoMo have something comparable?

[Bernhard Egger]

Hi Nataniel,

a MoMo contains 3 Gaussian process models:

val bfm = MoMoIO.read(new File("model2017-1_bfm_nomouth.h5")).get

bfm.neutralModel.shape.gpModel //shapeGP
bfm.neutralModel.color.gpModel //colorGP
bfm.expressionModel.get.expression.gpModel // expressionGP

Best

Bernhard

[nataniel...@gmail.com]

Actually I found that MoMoBasic does have .shape and if I do:

MoMoBasic.shape.gpModel.interpolateNearestNeighbors 

I am almost there. Except there is a final issue. The resulting object is of class:

scalismo.statisticalmodel.LowRankGaussianProcess[scalismo.geometry._3D,scalismo.geometry.Point[scalismo.geometry._3D]]

and we need an object of class

scalismo.statisticalmodel.LowRankGaussianProcess[scalismo.geometry._3D,scalismo.geometry.Vector[scalismo.geometry._3D]]

How to change from Point to Vector?

[nataniel...@gmail.com]

Thank you.

I am finally able to fit the parameters. I am fitting to the neutral faces and will see if I can extend it to the expressive faces (by fitting the expression parameters and keeping the shape parameters constant within a same identity)

Thanks so much for all of the help. The codebase is beautifully written.

[Andreas Forster]

Hi Nataniel,

I am glad we could help you get started.

And thank you for the positive feedback. It's good to hear that our software is usable.

Best, Andreas

--

You received this message because you are subscribed to the Google Groups "scalismo-faces" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalismo-face...@googlegroups.com.
To post to this group, send email to scalism...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scalismo-faces/8ba8f14b-d99e-4598-9e82-57aa3c1c2313%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

*****************************************

Dr. Andreas Morel-Forster
Departement Mathematik und Informatik
Spiegelgasse 1
CH-4051 Basel
PHONE: +41 61 207 05 52

MAIL: Andreas.Forster@unibas.ch
*****************************************