To calculate compactness, generalization and specificity of a model in statismo

[Vipul Raikar]

I was wondering what is the right approach to evaluate a model for the above parameters using statismo. 

[Marcel Luethi]

Hi Vipul,

Statismo does not provide these methods out of the box. It is, however, easy to implement them yourself. You can find some example code here:
https://github.com/statismo/shape-challenge-2014  .

As this example was written for a challenge where we had a number of test datasets on which we evaluated the metrics, you will see that the code establishes correspondence before computing the generalization score. If you have datasets that are already in correspondence, you can use cross-validation instead, as shown here:
https://github.com/statismo/statismo/blob/master/modules/VTK/examples/vtkCrossValidationExample.cxx

I hope this helps.

Best regards,

Marcel

On Thu, Feb 9, 2017 at 5:09 PM, Vipul Raikar <vipul...@gmail.com> wrote:

I was wondering what is the right approach to evaluate a model for the above parameters using statismo. 

--
You received this message because you are subscribed to the Google Groups "statismo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to statismo-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vipul Raikar]

Thank you very much for you response, Marcel. I will look through the code. 

I also happen to be familiar with scalismo and wrote a quick script to calculate generalization and specificity. I had a few questions:

1. When calculating genralization, am I correct in assuming that the test set used is not the one to train the model and similarly for specificity, the test set used is the one that was used to train the model.

2. Having read through Styner, Martin A., et al. "Evaluation of 3D correspondence methods for model building." Information processing in medical imaging. Springer Berlin Heidelberg, 2003, it seems the method to calculate compactness involves calculating the cumulative variance of the model described as a function of the modes or principle components. I was wondering if you could provide an explanation as to how to go about this?

Thank you very much!

Vipul  

To unsubscribe from this group and stop receiving emails from it, send an email to statismo-user...@googlegroups.com.

[Marcel Luethi]

Hi Vipul,

You are correct that when computing the generalization for a model, the test set should not be in the model (as otherwise it can be explained perfectly). Scalismo automatically does that for you. In Statismo you could use the cross-validation functionality of the data manager for that. For computing the specificity it is not necessary to do a leave one out cross-validation (as the comparison is with respect to randomly drawn samples, and not the best fit).

Regarding the compactness: The paper you mention defines compactness (and if I am not mistaken also generalization and specificity) as a function of the number of PCA components. For comparing two models quantitatively you will have to fix how many components you want in your model. We often simply choose all the components (assuming that there is not much noise represented in the model).   Another alternative is to compute compactness for all the components and then plot the values, as they do it in the paper.

Best regards,

Marcel

To unsubscribe from this group and stop receiving emails from it, send an email to statismo-users+unsubscribe@googlegroups.com.