Hi Chiara,
The gl.assign.mahalanobis script was written as a part of a series of scripts for population assignment. It is used in an exploratory sense to assign the most likely population membership, together with other indicators. It was not specifically designed to identify outliers, but I guess it could be.
It is a bit tricky because it standardizes on the axes, such that a confidence ellipse in multivariate space is transformed to a confidence sphere. Outliers are points outside that sphere. The problem with that of course when you apply it to ordinated space is that the deeper noise dimensions are given equal weight to the higher informative dimensions, or at least that is my understanding (please correct me if I am wrong someone). So you should apply it to a restricted space of the higher dimensions I guess. The script allows you to do that with the dim.limit parameter. If dim.limit = 2, there is not much value add over plotting confidence ellipses in your PCA or PCoA plots and looking where your suspect values lie. The
gl.assign.mahalanobis will give you a probability.
If you are looking for outliers in deeper dimensional space, then the script will handle that. Points that are not outliers in a 2 dimensional plot can still be outliers in comparison with a three dimensional confidence ellipse for example.
Maybe have a play and let us know how it goes.
Arthur