populations.fst_summary, how are these calculated and can I assess significance?

[Natalia Bayona Vásquez]

Hi everyone,

I was trying to use the values from populations.fst_summary output from populations. However I would like to have the statistical significance of these values.

For what I understand these Fst are different from Weir-Cockerham and Nei's Fst values (and F'st), according to the formula in this post, but also according to results that I get from hierfstat and mmod when I tried to calculate an array of these estimates and all of them are different to the values presented in the Stacks output.

Is there a way for me to check for the significance of Fst values between population pairs in Stacks?

Thanks!

[Catchen, Julian]

Hi Natalia,

The Fst values in the fst_summary output file are just simple averages
of the AMOVA-based Fst point values that are calculated for each
polymorphic site. That is, if you take, for a population pair, the AMOVA
Fst values from the populations.fst_Y-Z.tsv output file, and average
them, you will get the value in the fst_summary file.

We take an Analysis of Molecular Variance (AMOVA) approach to partition
genetic variance between and within populations. This is analogous to a
statistical ANOVA.

As the manual states, this is defined in Weir's book:

Analysis of Molecular Variance Fst calculation. Derived from Weir,
Genetic Data Analysis II, chapter 5, "F Statistics," pp166-167.

This is also the approach taken by Excoffier and used in Arlequin. You
can read about it from him at:

Excoffier, L., Smouse, P. E., & Quattro, J. M. (1992). Analysis of
molecular variance inferred from metric distances among DNA haplotypes:
application to human mitochondrial DNA restriction data. Genetics,
131:479-491.

We use the AMOVA approach in both SNP and haplotpye cases.

I find Bird's paper very useful in explanation of the different
calculations:

Bird, et al. Detecting and measuring genetic differentiation. (2011).
Detecting and measuring genetic differentiation. In Phylogeography and
Population Genetics in Crustacea (pp. 31–55). CRC Press.

Hope this helps.

julian


Natalia Bayona Vásquez wrote on 3/7/19 1:47 PM:

> Hi everyone,
>
> I was trying to use the values from populations.fst_summary output from
> populations. However I would like to have the statistical significance
> of these values.
> For what I understand these Fst are different from Weir-Cockerham and
> Nei's Fst values (and F'st), according to the formula in this

> <https://groups.google.com/forum/#!searchin/stacks-users/fst$20calculation%7Csort:date/stacks-users/BLcDbgSGVyQ/hh5v5LxJBwAJ>

[Natalia Bayona]

This is very informative!

Thanks, Julian.

--

Natalia J Bayona-Vásquez, PhD 
Environmental Health Science | Postdoctoral Research Associate 
150 E Green St | Athens, GA 30602

e: njba...@uga.edu 
w: http://njbayona7.wixsite.com