Hi ben,
The "vegdist" function from the vegan package is usefull in order to calculate pairwise community distance between samples (e.g., euclidean, sorensen, bray-curtis...). You can use the OTU table to compute that kind of distance using "vegdist" function.
The Unifrac matrix is by itself a distance matrix, like a bray-curtis matrix, so I don't clearly understand why you are trying to use the vegdist function on the unifrac matrix!
You can use metaMDS function as an ordinattion method to depict community variations across samples...or capscale function to understand community variations as a function of environnmental variables.
Hope this help!
Flo