Incorporating more landmarks into the fitting

[Jerry Liu]

In my studies, I'm trying to apply this shape model to 3D reconstruct faces from 2D images to aid craniofacial surgery, similar to the applications in this course. So far, I've been able to use the model to perform fitting on a 2D image. Looking at our results so far, the faces modelled have a great match for nose, & eyes but the cheeks look too thin or too wide.

We think this is because the fitting is only using the default 12 landmarks around the eyes, nose, mouth and single chin point. Would you/anyone else be able to provide some insight on how I could add more jawline landmarks? I am not sure how to define more landmarks because each one has a specified name.

The 12 I currently use are: center.chin.tip, center.nose.tip, right.nose.wing.tip, left.nose.wing.tip, right.eye.corner_outer, right.eye.corner_inner, left.eye.corner_inner, left.eye.corner_outer, right.lips.corner, center.lips.upper.outer, left.lips.corner, center.lips.lower.outer.

I found out that there were more, summing up to a total of 40 landmarks, which includes the eyebrows, more of the nose and mouth, etc, but I see that there is still very little for the jawline and cheeks. Would there be more landmarks that I am missing or do you have a suggestion for how I could add more?

Any help is greatly appreciated!

Thank you, 
Jerry

[Bernhard Egger]

Hi Jerry,

you are raising an interesting question.
We did not include landmarks on the jawline since they are not really nicely defined. Correspondence in those regions is much less certain than in the regions landmarks are typically located. However a lot of people use landmarks in those regions.
To include such landmarks and use them with our model you need to integrate them into the model.
You can actually use scalismoLab (or scalismo-UI) to click those landmarks. You load the model reference and choose the LM button on top to click the landmarks. You can then save the landmarks to a file and load them within your software.

The landmarks in the model are stored in the h5 files (with their 3D coordinates, not the id's). You can actually look at it with e.g. HDFView. The easiest way to add new landmarks is via loading the landmarks in scalismo faces and adding them using the .withLandmarks Function of the MoMo to add new landmarks to a loaded model in scalismo-faces (to avoid numerical inexactness you can also use the findclosestpoint on the referenceMesh on the model first - not sure if this will be needed).

Those steps are a little bit coarse - let me know if you get stuck somewhere

Best
Bernhard

[Jerry Liu]

Hi Bernhard,

Thank you for your help, what you said makes sense. 

I did as you said and loaded the landmarks into scalismo faces and added the new ones, and then created a new momo with the .withLandmarks function. What I'm unsure about how is how to export the momo successfully back into an h5 file so that it can be used in the script like the original. What I'm finding now with doing MoMoIO.write(newMoMo, newFile) is that it creates the file properly but the dataset is now listed as a json and not as a string like the original, causing the script to give an error relating to java.util.NoSuchElementException: None.get when looking for the landmark ids.

Would you have any suggestions?

Jerry

[Bernhard Egger]

Hi Jerry,

the model can hold different kind of landmarks as indicated in readLandmarks in scalismo/faces/io/MoMoIO.scala.

Can you check if the model has landmarks after loading and print them out (use the hasLandmarks function)?

Can you write and read again the original model?

There is no test for MoMoIO so it is possible that it does not work properly

Best
Bernhard

[Jerry Liu]

Hi Bernhard,

I haven't looked into the different kinds of landmarks, I will do that now.

I used the hasLandmarks function on my new momo and it returns true, and I can do newMoMo.landmarks to print the new landmarks as well. 

I tried reading the original momo and doing MoMoIO.write(momo, newFile) and it also gives the same error as with the newMoMo.

Do you know if there's a way to edit the contents of the h5 file directly without needing to create a new file?

Jerry

[Jerry Liu]

Just a note, I also notice that between reading the file and writing it, the file size decreases by about 25% from 170 879 kb to 136 036 kb. Here is the code I tried for just basic reading and rewriting (ignoring imports)

  val momoFile = new File("data/model2017-1_face12_nomouth.h5")

  val momoURI = momoFile.toURI

  val momo = MoMoIO.read(momoURI).get.neutralModel

  val modelRenderer = MoMoRenderer(momo)

  MoMoIO.write(momo, new File("data/model2017-1_face12_nomouth_copy.h5"))

[Jerry Liu]

Hi Bernhard,

Sorry, I think I might have miscommunicated something in my last two messages. So I believe that adding the landmarks to the .h5 file works properly, and the script is able to execute properly.

This means that I am able to load both the original model (momo) and the new one (newMoMo) with additional landmarks and run the original script where I am using 24 landmarks.

What goes wrong is when I try to access or use any of the new landmarks that I added into the newMoMo. For example, I added a landmark called right.cheek.eye using this line:

  var newLandmarks = landmarks + ("right.cheek.eye" -> Landmark("right.cheek.eye",Point3D(-70.25, 30.19, 47.24),None,None))

and then I saved the new momo using:

  val newMoMo = momo.withLandmarks(newLandmarks)

  MoMoIO.write(newMoMo, new File("data/model2017-1_face12_nomouth_new.h5"))

I believe this was saved properly into the newMoMo because the original landmarks can be accessed properly. However, I also added a line in the .tlms file with right.cheek.eye in the same format as the other landmarks, but it causes the script to produce an error (Exception in thread "main" java.util.NoSuchElementException: None.get). Would there possibly be another part of the code that I need to change for everything to work together? My understanding of this error is that it tried to search for something matching the id right.cheek.eye but it couldn't find anything.

Please let me know if any of this doesn't make sense, and thanks again for helping me out!

Jerry

[Bernhard Egger]

Hi Jerry,

I ran your code and it does what it should (with val landmarks = momo.landmarks).
The model is getting smaller because you only saved the neutral model.
  val momo = MoMoIO.read(momoURI).get.neutralModel //this removes the expression part of the model.

Can you provide the trace of your error? Which method/call causes this error?
Are you using an IDE or are you working in one of our tutorials?
Can you check if you did not add any ugly characters to the tlms file?
Does it still work if you remove your additional landmark again?
Also a wrong newline at the end of the line could cause a problem.

Best
Bernhard

[Jerry Liu]

Hi Bernhard,

I see, thank you for making me aware of only reading in the neutral model. Is the expressionModel used for datasets with facial expressions like smiling, frowning, etc?

The trace is:

Exception in thread "main" java.util.NoSuchElementException: None.get

at scala.None$.get(Option.scala:349)

at scala.None$.get(Option.scala:347)

at scalismo.faces.sampling.face.evaluators.LandmarkPointEvaluator.logValue(LandmarkPointEvaluator.scala:36)

at scalismo.faces.sampling.face.evaluators.LandmarkPointEvaluator.logValue(LandmarkPointEvaluator.scala:30)

at scalismo.sampling.evaluators.ProductEvaluator.$anonfun$logValue$1(ProductEvaluator.scala:29)

at scalismo.sampling.evaluators.ProductEvaluator.$anonfun$logValue$1$adapted(ProductEvaluator.scala:29)

at scala.collection.Iterator$$anon$10.next(Iterator.scala:448)

at scala.collection.Iterator.foreach(Iterator.scala:929)

at scala.collection.Iterator.foreach$(Iterator.scala:929)

at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)

at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:157)

at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:155)

at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1417)

at scala.collection.TraversableOnce.sum(TraversableOnce.scala:216)

at scala.collection.TraversableOnce.sum$(TraversableOnce.scala:216)

at scala.collection.AbstractIterator.sum(Iterator.scala:1417)

at scalismo.sampling.evaluators.ProductEvaluator.logValue(ProductEvaluator.scala:29)

at scalismo.sampling.algorithms.MetropolisHastings.next(Metropolis.scala:77)

at scalismo.sampling.algorithms.MetropolisHastings.next(Metropolis.scala:97)

at scalismo.sampling.MarkovChain.$anonfun$iterator$1(MarkovChain.scala:26)

at scala.collection.Iterator$$anon$7.next(Iterator.scala:128)

at scala.collection.Iterator$SliceIterator.next(Iterator.scala:259)

at scala.collection.Iterator.foreach(Iterator.scala:929)

at scala.collection.Iterator.foreach$(Iterator.scala:929)

at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)

at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:59)

at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:50)

at scala.collection.immutable.VectorBuilder.$plus$plus$eq(Vector.scala:658)

at scala.collection.immutable.VectorBuilder.$plus$plus$eq(Vector.scala:635)

at scala.collection.TraversableOnce.to(TraversableOnce.scala:310)

at scala.collection.TraversableOnce.to$(TraversableOnce.scala:308)

at scala.collection.AbstractIterator.to(Iterator.scala:1417)

at scala.collection.TraversableOnce.toIndexedSeq(TraversableOnce.scala:300)

at scala.collection.TraversableOnce.toIndexedSeq$(TraversableOnce.scala:300)

at scala.collection.AbstractIterator.toIndexedSeq(Iterator.scala:1417)\

The line in my code that it is crashing at is:

val initLMSamples: IndexedSeq[RenderParameter] = poseFitter.iterator(init10).take(1000).toIndexedSeq

I believe this line is the one that is used to initialize the position of the momo on the image based on the positions of the landmarks. 

If I go into the error trace line:

at scalismo.faces.sampling.face.evaluators.LandmarkPointEvaluator.logValue(LandmarkPointEvaluator.scala:36)

it looks like val lmLocation: TLMSLandmark2D = landmarksRenderer.renderLandmark(targetLandmark.id, sample).get

so I would guess that it is trying to find a targetLandmark.id that it thinks does not exist, although I am fairly certain it exists and is implemented properly in the h5. 

I am now doing this in the IntelliJ IDEA IDE but I believe I have all the required libraries and imports needed for it to run.

I don't believe I'm adding any ugly characters because I am using code to write all the landmarks (your original ones + my additional ones) and they are all in the same format but only my additional ones don't work.

When I remove my additional landmark, the original landmarks still work but calling my new ones causes the same error as before.

Please let me know if you disagree with anything I said or your thoughts in general, I think they would be extremely helpful!

[Bernhard Egger]

Hi Jerry Liu,

the facial expression model is built on anger,disgust, fear, happy, sad and surprise (see https://arxiv.org/pdf/1709.08398.pdf for details) - if you work on neutral facial expressions it is totally ok, to remove the expression part of the model.

I can not reproduce what you describe. I execute the following code:

object JerryLiu extends App{

  scalismo.initialize()

  val momoFile = new File("/afs/csail.mit.edu/u/e/egger/unibas/model2015/model2017-1_bfm_nomouth.h5")

  val momoURI = momoFile.toURI
  val momo = MoMoIO.read(momoURI).get.neutralModel

  println(momo.landmarks.contains("right.cheek.eye"))
  MoMoIO.write(momo, new File("/tmp/model2017-1_face12_nomouth_new.h5"))
  val readmomo = MoMoIO.read(new File("/tmp/model2017-1_face12_nomouth_new.h5")).get

  println("-")
  println(readmomo.landmarks.contains("right.cheek.eye"))
  val modelRenderer = MoMoRenderer(momo)

  val landmarks = momo.landmarks

  var newLandmarks = landmarks + ("right.cheek.eye" -> Landmark("right.cheek.eye",Point3D(-70.25, 30.19, 47.24),None,None))

  val newmomo = momo.withLandmarks(newLandmarks)

  MoMoIO.write(newmomo, new File("/tmp/model2017-1_face12_nomouth_new2.h5"))
  println("-")
  println(newmomo.landmarks.contains("right.cheek.eye"))

  val readnewmomo = MoMoIO.read(new File("/tmp/model2017-1_face12_nomouth_new2.h5")).get
  println("-")
  println(readnewmomo.landmarks.contains("right.cheek.eye"))
}

And the output on the console is:
false
-
false
-
true
-
true

Can you please execute the code and check your output?

Perhaps the landmarkPoint is not on the surface? Can you check if that one helps?

var newLandmarks = landmarks + ("right.cheek.eye" -> Landmark("right.cheek.eye",momo.referenceMesh.pointSet.findClosestPoint(Point3D(-70.25, 30.19, 47.24)).point,None,None))
During rendering it takes the id of that point from the reference - that would fail if the point is not on the reference.

If that was not it, could it be possible that you added the landmark twice? This could cause strange errors (?)

Best
Bernhard

Am Donnerstag, 8. März 2018 17:17:28 UTC-5 schrieb Jerry Liu:

[Jerry Liu]

Hi Bernhard,

I executed your code and my output is identical to yours. The initial momoFile you used was the full face model2017-1_bfm_nomouth.h5 so I tried it again with model2017-1_face12_nomouth.h5 and got the same result. 

However, I tried your find closest point and it solved the issue! I think you're right, since the point was not exactly one of the points on the mesh, it was unable to be used properly.

Thank you so much for all your help, I really appreciate it and I will be able to move forward with my work now. I will let you know if any further issues relating to this come up later.

Jerry

[Bernhard Egger]

Hi Jerry,

perfect. Let us know if there are more issues - I'm happy to help.
I hope the additional landmarks improve your results. You can also play around with the uncertainty of the landmarks and adapt them to the image size at hand and the uncertainty of the landmarks. Perhaps it even makes sense to use different uncertainties for the newly added points by using two different landmark evaluators.

Best
Bernhard

[Jerry Liu]

Hi Bernhard,

The new landmarks are working well! I am now interested in testing the facial expression model, but I noticed that is should be implemented differently (I initially thought I could just replace the call of neutralModel with expressionModel). While the neutralModel returns a MoMoBasic, the expressionModel is something completely different like an Option. Would you have any examples on how to invoke the expressionModel instead of or in addition to the neutralModel for model fitting? 

Thank you,

Jerry

[Bernhard Egger]

Hi Jerry,

the expression and neutral model are very similar from the usage.
Just remove the .neutralmodel.
You will get an Option always when loading a model - that is from the IO:
val momo = MoMoIO.read(momoURI).get.neutralModel gives you the neutral Model
val momo = MoMoIO.read(momoURI).get gives you the model with expressions
The .get is necessary since something could go wrong in the IO-process (e.g. wrong file name). The Option means it is either a MoMo or None - to handle the failure case you can use .getOrElse

Our state of the art StandardFitScript which is part of the Basel Face Pipeline provides a boolean-switch if you want to use the expression model with expression proposals or not:
https://github.com/unibas-gravis/basel-face-pipeline/blob/master/src/main/scala/fitting/StandardFitScript.scala

Best
Bernhard