bin evaluation

[KC soh]

Hi 

I would like to compare the results of BinSanity to other binning algorithms and i see that there is a bin_evaluation script in the utils folder and was wondering how to use it?

Do we have to map the bins from BinSanity to the reference genomes first?

Thank you.

[edgraham]

Hello,

So the Bin_evaluation script was written to calculate the adjusted rand index, precision, and recall which are described in the BinSanity paper. For these metrics you have to have a priori knowledge of the contig assignment. So for example if you were testing methods on the infant Gut Metagenome as we did in the paper you would assume that the genomes published in Sharon et al. (2013) were the proper contig assignments. To use the script to compare a binning method to these known contig assignments you would need one directory that contained the known reference genomes (in this case the genomes pulled from Sharon et al. 2013) that would be called as -r /Directory/Containing/Reference/Genomes and another directory containing the putative genomes generating by the clustering method you are evaulating, which would be called as -b /Directory/Containing/Putative/Genomes.In essence the precision defines whethere each cluster contains members of a single class, a score of 1 meaning all the contigs in a bin are from a single source (As calculated in reference to the known contig assignments). Then recall considers whether each member of a class is assigned to the same bin, 1 being that all contigs in a bin are from one source organism. The rand index uses the harmonic mean or these statistical calculations and then adjusts for chance.

If in contrast you want to say evaluate multiple methods performance on a metagenome you yourself have generated and do not know the actual contig assignments you could use something like Anvi'o to visually compare how each methods is clustering your contigs. 

Regards,

Elaina