Chris,
You might want to check out the package, geometry, It has a vignette for the convhulln function, which can calculate hull areas/volumes.
In terms of overlap, although maybe a little inelegant, one can calculate the area of two hulls (in two dimensions), plus one hull that combines the points of the two hulls. If there is little or no overlap, the one hull will have a much larger area than either of the two hulls, individually. If there is considerable overlap, the area of the one hull will resemble the areas of the two hulls. A good way to evaluate this is compare AC, the area of the third hull comprising points of both groups to AA + AB, the sum of the areas of the hulls of individual groups; i.e., AC / (AA +AB). This value will start to exceed 1 if the hulls do not overlap, or do not overlap much, but will be considerably less than 1 if they overlap substantially.
I have to also warn against conclusions made in two-dimension projections. Imagine two discoidal clouds of points with no overlap at all in a third PC dimension, viewed from a two-dimensional projection of PC1 and PC2. One might conclude that the groups share the same space, when they don’t. Luckily, convhulln can handle volumes of hulls in n dimensions.
Hope that helps,